Skip to main content


Semantic segmentation is the process of assigning labels to specific regions in an image. This richer understanding of the environment enables a variety of creative AR features. For example, an AR pet could identify ground to run along, AR planets could fill the sky, the real-world ground could turn into AR lava, and more!

Image of doty masked with the sky channel

What's new?

ARDK 3 provides this unqiue feature through the new XRSemanticsSubsystem class. The ARSemanticSegmentationManager makes this subsystem's data available as a MonoBehaviour and manages the subsystem's lifecyle. Through these classes, ARDK serves semantic predictions in two forms:

  1. A buffer of unsigned integers for each pixel in the viewport, referred to as "packed semantic channels." The 32 bits of each integer correspond to a semantic channel and are either enabled (value is 1) or disabled (value is 0) depending on whether an object in that channel exists at that pixel. A pixel can have more than one label, e.g. both ground and natural_ground.
Buffer Diagram
  1. For each semantic channel, there is a buffer of normalized (between 0 and 1) float values for each pixel in the viewport. These floats show the probability that their pixel should be classified as the specified semantic channel.

Unity Scene Integration

By default, the ARSemanticSegmentationManager is simple, with only one exposed parameter for framerate settings in its Inspector window.

ARSemanticSegmentationManager Inspector

While ARSemanticSegmentationManager itself is uncomplicated, there are features in other Lightship components that require one to be present and active in the scene. These include:

Available Semantic Channels

The following table lists the current set of semantic channels. Because the ordering of channels in this list may change with new versions of ARDK, we recommend that you use names rather than index values in your app. Use the XRSemanticsSubsystem.TryGetChannelNames method or ARSemanticSegmentationManager.ChannelNames property to verify names at runtime.

Because channel names are read from a neural network model, there will be slight delays when the semantics subsystem starts, while the model is initialized, and before channel names are available. This delay can be reduced by downloading the model in advance. See Neural Network Model Preloading for more information.

Click to expand the table of semantic channels
IndexChannel NameNotes
[0]skyIncludes clouds. Does not include fog.
[1]groundIncludes everything in natural_ground and artificial_ground. Ground may be more reliable than the combination of the two where there is ambiguity about natural vs artificial.
[2]natural_groundIncludes dirt, grass, sand, mud, and other organic / natural ground. Ground with heavy vegetation or foliage may be detected as foliage instead.
[3]artificial_groundIncludes roads, sidewalks, tracks, carpets, rugs, flooring, paths, gravel, and some playing fields.
[4]waterIncludes rivers, seas, ponds, lakes, pools, waterfalls, some puddles. Does not include drinking water, water in cups, bowls, sinks, baths. Water with strong reflections may be detected as what is in the reflection instead of as water.
[5]personIncludes body parts, hair, and clothes worn. Does not include accessories or carried objects. Does not distinguish between individuals. Mannequins, toys, statues, paintings, and other artistic expressions of a person are not considered a “person”, though some detections may occur. Photorealistic images of a person are detected as a “person”. Model performance may suffer when a person is partially visible in the image or is in an unusual body posture, such as when crouching or widely spreading their arms.
[6]buildingIncludes residential and commercial buildings, both modern and traditional. Should not be considered synonymous with walls.
[7]foliageIncludes bushes, shrubs, leafy parts and trunks of trees, potted plants, and flowers.
[8]grassGrassy ground, e.g. lawns, rather than tall grass.

Experimental Semantic Channels

Lightship also offers a set of experimental semantic channels. Because they are experimental, these channels may need accuracy improvements; if you find an issue with them, please provide feedback in the Lightship Developer Community.


Experimental features are not recommended for use in production-level applications!

Click to expand the experimental channels
IndexChannel NameNotes
[9]flower_experimentalThe part of a plant that has brightly colored petals and provides pollen. Does not include any green parts (stems, leaves).
[10]tree_trunk_experimentalThe part of a tree that connects the leafy crown and branches with its roots. Does not include leaves, branches, flowers, or processed/cut wood (logs).
[11]pet_experimentalOnly dogs and cats. Not designed to work on soft toys of dogs and cats.
[12]sand_experimentalFinely divided rock and mineral particles. Sand has various compositions but is defined by its small grain size. Includes all types of sandy/gravelly surfaces found on the floor/ground, wet sand, and sandcastles/sand art. Does not include dirt, soil.
[13]tv_experimentalScreen portion of any digital device. Includes both screens turned on and screens turned off. Excludes the frame (“bezel”) around the screen, and stands.
[14]dirt_experimentalIncludes mud and dust covering the ground. May overlap significantly with sand. Includes dirt roads.
[15]vehicle_experimentalincludes all components of vehicles. Covers all types of cars and trucks. Intended to also work on (but accuracy may vary) motorcycles, bicycles, rickshaws, horse carriages, tuk-tuks, trains/trams, boats/ships, airplanes/helicopters/hovercrafts.
[16]food_experimentalAnything that can be consumed as “food” and not “drink” by human beings, including salt & pepper, and boxes/packaging likely to contain food or ingredients. Does not include live animals, pet food, or plates/dishes the food is served on.
[17]loungeable_experimentalAll objects intended to be used to sit on. Includes beds (including the legs and bases), and legs and bases for chairs or sofas. Does not include pet beds/blankets, or any empty spaces/gaps in chairs or sofas.
[18]snow_experimentalAtmospheric water vapor frozen into ice crystals and falling in light white flakes or covering the ground as a white layer. Includes clean and dirty snow. Includes icebergs and icicles, ice/snow sculptures, igloos/snow houses. Does not include ice in drinks, snow in the sky.

Learn More