Skip to main content

class ARSemanticSegmentationManager

(Niantic.Lightship.AR.Semantics.ARSemanticSegmentationManager)

Overview​

The ARSemanticSegmentationManager controls the XRSemanticsSubsystem and updates the semantics textures on each Update loop. Textures and XRCpuImages are available for confidence maps of individual semantic segmentation channels and a bit array indicating which semantic channels have surpassed the chosen confidence threshold per pixel. For cases where a semantic segmentation texture is overlaid on the screen, utilities are provided to read semantic properties at a given point on the screen.

    class ARSemanticSegmentationManager: SubsystemLifecycleManager< XRSemanticsSubsystem, XRSemanticsSubsystemDescriptor, XRSemanticsSubsystem.Provider > {
    public:
        // fields
    
         IReadOnlyList<string> ChannelNames => _readOnlyChannelNames;
         IReadOnlyDictionary<string, int> ChannelIndices => _readOnlyChannelNamesToIndices;

        // properties
    
        uint??? TargetFrameRate;
        bool IsMetadataAvailable;
        Action<ARSemanticSegmentationModelEventArgs> MetadataInitialized;

        // events
    
        event FrameReceived();

        // methods
    
        void Update();
    
        Texture2D GetSemanticChannelTexture(
            string channelName,
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        Texture2D GetPackedSemanticsChannelsTexture(
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        Texture2D GetSuppressionMaskTexture(
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        bool TryAcquireSemanticChannelCpuImage(
            string channel,
            out XRCpuImage cpuImage,
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        bool TryAcquirePackedSemanticChannelsCpuImage(
            out XRCpuImage cpuImage,
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        bool TryAcquireSuppressionMaskCpuImage(
            out XRCpuImage cpuImage,
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        int GetChannelIndex(string channelName);
    
        uint GetSemantics(
            int viewportX,
            int viewportY,
            XRCameraParams? cameraParams = null
        );
    
        List<int> GetChannelIndicesAt(
            int viewportX,
            int viewportY,
            XRCameraParams? cameraParams = null
        );
    
        List<string> GetChannelNamesAt(
            int viewportX,
            int viewportY,
            XRCameraParams? cameraParams = null
        );
    
        bool DoesChannelExistAt(
            int viewportX,
            int viewportY,
            string channelName,
            XRCameraParams? cameraParams = null
        );
    
        bool DoesChannelExistAt(
            int viewportX,
            int viewportY,
            int channelIndex,
            XRCameraParams? cameraParams = null
        );
    
        bool TrySetChannelConfidenceThresholds(Dictionary<string, float> channelConfidenceThresholds);
        bool TryResetChannelConfidenceThresholds();

    protected:
        // methods
    
        override void OnBeforeStart();
        override void OnDisable();
    };

Detailed Documentation​

The ARSemanticSegmentationManager controls the XRSemanticsSubsystem and updates the semantics textures on each Update loop. Textures and XRCpuImages are available for confidence maps of individual semantic segmentation channels and a bit array indicating which semantic channels have surpassed the chosen confidence threshold per pixel. For cases where a semantic segmentation texture is overlaid on the screen, utilities are provided to read semantic properties at a given point on the screen.

Fields​

ChannelNames​

IReadOnlyList<string> ChannelNames => _readOnlyChannelNames

The names of the semantic channels that the current model is able to detect.

ChannelIndices​

IReadOnlyDictionary<string, int> ChannelIndices => _readOnlyChannelNamesToIndices

The indices of the semantic channels that the current model is able to detect.

Properties​

TargetFrameRate​

uint??? TargetFrameRate

Frame rate that semantic segmentation inference will aim to run at.

IsMetadataAvailable​

bool IsMetadataAvailable

True if the underlying subsystem has finished initialization.

MetadataInitialized​

Action<ARSemanticSegmentationModelEventArgs> MetadataInitialized

An event which fires when the underlying subsystem has finished initializing.

Methods​

OnBeforeStart​

override void OnBeforeStart()

Callback before the subsystem is started (but after it is created).

OnDisable​

override void OnDisable()

Callback when the manager is being disabled.

Update​

void Update()

Callback as the manager is being updated.

GetSemanticChannelTexture​

Texture2D GetSemanticChannelTexture(
        string channelName,
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Returns semantic segmentation texture for the specified semantic channel.

    Parameters:

    channelName - The semantic channel to acquire.

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    The texture for the specified semantic channel, if any. Otherwise, null.

GetPackedSemanticsChannelsTexture​

Texture2D GetPackedSemanticsChannelsTexture(
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Retrieves the texture of semantic data where each pixel can be interpreted as a uint with bits corresponding to different classifications.

    Parameters:

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    The packed semantics texture, owned by the manager, if any. Otherwise, null.

GetSuppressionMaskTexture​

Texture2D GetSuppressionMaskTexture(
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Retrieves the suppression mask texture, where each pixel contains a uint which can be used to interpolate between the predicted depth and the far field depth of the scene. This is useful for enabling smooth occlusion suppression.

    Parameters:

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    The suppression mask texture, owned by the manager, if any. Otherwise, null.

TryAcquireSemanticChannelCpuImage​

bool TryAcquireSemanticChannelCpuImage(
        string channel,
        out XRCpuImage cpuImage,
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Attempt to acquire the latest semantic segmentation XRCpuImage for the specified semantic class. This provides direct access to the raw pixel data.

The XRCpuImage must be disposed to avoid resource leaks.

    Parameters:

    channel - The semantic channel to acquire.

    cpuImage - If this method returns true, an acquired XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    True if the CPU image was acquired. Otherwise, false

TryAcquirePackedSemanticChannelsCpuImage​

bool TryAcquirePackedSemanticChannelsCpuImage(
        out XRCpuImage cpuImage,
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Tries to acquire the latest packed semantic channels XRCpuImage. Each element of the XRCpuImage is a bit field indicating which semantic channels have surpassed their respective detection confidence thresholds for that pixel. (See GetChannelIndex)

The utility GetChannelNamesAt can be used for reading semantic channel names at a viewport location.

    Parameters:

    cpuImage - If this method returns true, an acquired XRCpuImage. The CPU image must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    True if the CPU image was acquired. Otherwise, false

TryAcquireSuppressionMaskCpuImage​

bool TryAcquireSuppressionMaskCpuImage(
        out XRCpuImage cpuImage,
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Tries to acquire the latest suppression mask XRCpuImage. Each element of the XRCpuImage is a uint32 value which can be used to interpolate between instantaneous depth and far field depth.

    Parameters:

    cpuImage - If this method returns true, an acquired XRCpuImage. The CPU image must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    True if the CPU image was acquired. Otherwise, false

GetChannelIndex​

int GetChannelIndex(string channelName)

Get the channel index of a specified semantic class. This corresponds to a bit position in the packed semantics buffer, with index 0 being the most-significant bit.

    Parameters:

    channelName - The name of the semantic class.

    Returns:

    The index of the specified semantic class, or -1 if the channel does not exist.

GetSemantics​

uint GetSemantics(
        int viewportX,
        int viewportY,
        XRCameraParams? cameraParams = null
    )

Returns the semantics at the specified pixel on screen.

    Parameters:

    viewportX - Horizontal coordinate in viewport space.

    viewportY - Vertical coordinate in viewport space.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    A 32-bit packed unsigned integer where each bit is a binary indicator for a class, and the most-significant bit corresponds to the channel that is the 0th element of the ChannelNames list.

GetChannelIndicesAt​

List<int> GetChannelIndicesAt(
        int viewportX,
        int viewportY,
        XRCameraParams? cameraParams = null
    )

Returns an array of channel indices that are present at the specified pixel onscreen.

This query allocates garbage.

    Parameters:

    viewportX - Horizontal coordinate in viewport space.

    viewportY - Vertical coordinate in viewport space.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    An array of channel indices present for the pixel.

GetChannelNamesAt​

List<string> GetChannelNamesAt(
        int viewportX,
        int viewportY,
        XRCameraParams? cameraParams = null
    )

Returns an array of channel names that are present for the specified pixel onscreen.

This query allocates garbage.

    Parameters:

    viewportX - Horizontal coordinate in viewport space.

    viewportY - Vertical coordinate in viewport space.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    An array of channel names present for the pixel.

DoesChannelExistAt​

bool DoesChannelExistAt(
        int viewportX,
        int viewportY,
        string channelName,
        XRCameraParams? cameraParams = null
    )

Check if a semantic class is detected at the specified location in screen space, based on the confidence threshold set for this channel. (See TrySetChannelConfidenceThresholds)

    Parameters:

    viewportX - Horizontal coordinate in viewport space.

    viewportY - Vertical coordinate in viewport space.

    channelName - Name of the semantic class to look for.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    True if the semantic class exists at the given coordinates.

DoesChannelExistAt​

bool DoesChannelExistAt(
        int viewportX,
        int viewportY,
        int channelIndex,
        XRCameraParams? cameraParams = null
    )

Check if a semantic class is detected at the specified location in screen space, based on the confidence threshold set for this channel. (See TrySetChannelConfidenceThresholds)

    Parameters:

    viewportX - Horizontal coordinate in viewport space.

    viewportY - Vertical coordinate in viewport space.

    channelIndex - Index of the semantic class to look for in the ChannelNames list.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    True if the semantic class exists at the given coordinates.

TrySetChannelConfidenceThresholds​

bool TrySetChannelConfidenceThresholds(Dictionary<string, float> channelConfidenceThresholds)

Sets the confidence threshold for including the specified semantic channel in the packed semantic channel buffer.

Each semantic channel will use its default threshold value chosen by the model until a new value is set by this function during the AR session. Changes to the semantic segmentation thresholds are undone by either restarting the subsystem or by calling TryResetChannelConfidenceThresholds.

    Parameters:

    channelConfidenceThresholds - A dictionary consisting of keys specifying the name of the semantics channel that is needed and values between 0 and 1, inclusive, that set the threshold above which the platform will include the specified channel in the packed semantics buffer. The key must be a semantic channel name present in the list returned by TryGetChannelNames.

    System.NotSupportedException - Thrown when setting confidence thresholds is not supported by the implementation.

    Returns:

    True if the thresholds were set. Otherwise, false.

TryResetChannelConfidenceThresholds​

bool TryResetChannelConfidenceThresholds()

Resets the confidence thresholds for all semantic channels to the default values from the current model.

This reverts any changes made with TrySetChannelConfidenceThresholds.

    Parameters:

    System.NotSupportedException - Thrown when resetting confidence thresholds is not supported by the implementation.

    Returns:

    True if the thresholds were reset. Otherwise, false.