Skip to main content

class XRSemanticsSubsystem

(Niantic.Lightship.AR.XRSubsystems.XRSemanticsSubsystem)

Overview​

Defines an interface for interacting with semantic segmentation functionality.

class XRSemanticsSubsystem:
        SubsystemWithProvider< XRSemanticsSubsystem, XRSemanticsSubsystemDescriptor, XRSemanticsSubsystem.Provider >,
        ISubsystemWithModelMetadata {
    public:
    
        class Provider;

        // properties
    
        uint TargetFrameRate;
        List<string> SuppressionMaskChannels;
        uint? LatestFrameId;
        bool IsMetadataAvailable;

        // methods
    
        XRSemanticsSubsystem();
    
        bool TryGetSemanticChannel(
            string channelName,
            out XRTextureDescriptor semanticsChannelDescriptor,
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        bool TryAcquireSemanticChannelCpuImage(
            string channelName,
            out XRCpuImage cpuImage,
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        bool TryGetPackedSemanticChannels(
            out XRTextureDescriptor packedSemanticsDescriptor,
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        bool TryAcquirePackedSemanticChannelsCpuImage(
            out XRCpuImage cpuImage,
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        bool TryGetSuppressionMaskTexture(
            out XRTextureDescriptor suppressionMaskDescriptor,
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        bool TryAcquireSuppressionMaskCpuImage(
            out XRCpuImage cpuImage,
            out Matrix4x4 samplerMatrix,
            XRCameraParams? cameraParams = null
        );
    
        bool TryGetChannelNames(out IReadOnlyList<string> names);
        bool TrySetChannelConfidenceThresholds(Dictionary<string, float> channelConfidenceThresholds);
        bool TryResetChannelConfidenceThresholds();
        static bool Register(XRSemanticsSubsystemCinfo semanticsSubsystemCinfo);
    };

Detailed Documentation​

Defines an interface for interacting with semantic segmentation functionality.

Properties​

TargetFrameRate​

uint TargetFrameRate

Specifies the target frame rate for the platform to target running semantic segmentation inference at.

The target frame rate.

    Parameters:

    System.NotSupportedException - Thrown frame rate configuration is not supported.

LatestFrameId​

uint? LatestFrameId

Returns the frame id of the most recent semantic segmentation prediction.

The frame id.

    Parameters:

    System.NotSupportedException - Thrown if getting frame id is not supported.

IsMetadataAvailable​

bool IsMetadataAvailable

Is true if metadata has been downloaded and decrypted on the current device. Only if this value is true can the semantic segmentation label names or inference results be acquired.

If metadata is available.

    Parameters:

    System.NotSupportedException - Thrown frame rate configuration is not supported.

Methods​

XRSemanticsSubsystem​

XRSemanticsSubsystem()

Construct the subsystem by creating the functionality provider.

TryGetSemanticChannel​

bool TryGetSemanticChannel(
        string channelName,
        out XRTextureDescriptor semanticsChannelDescriptor,
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Gets a semantics channel texture descriptor and a matrix used to fit the texture to the viewport.

    Parameters:

    channelName - The string description of the semantics channel that is needed.

    semanticsChannelDescriptor - The semantics channel texture descriptor to be populated, if available from the provider.

    samplerMatrix - Converts from normalized viewport coordinates to normalized texture coordinates.

    cameraParams - Describes the viewport.

    System.NotSupportedException - Thrown if the implementation does not support semantics channel texture.

    Returns:

    true if the semantics channel texture descriptor is available and is returned. Otherwise, false.

TryAcquireSemanticChannelCpuImage​

bool TryAcquireSemanticChannelCpuImage(
        string channelName,
        out XRCpuImage cpuImage,
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Tries to acquire the latest semantics channel XRCpuImage.

    Parameters:

    channelName - The string description of the semantics channel that is needed.

    cpuImage - If this method returns true, an acquired XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Params of the viewport to sample with

    Returns:

    Returns true if an XRCpuImage was successfully acquired. Returns false otherwise.

TryGetPackedSemanticChannels​

bool TryGetPackedSemanticChannels(
        out XRTextureDescriptor packedSemanticsDescriptor,
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Gets a packed semantics texture descriptor.

    Parameters:

    packedSemanticsDescriptor - The packed semantics texture descriptor to be populated, if available from the provider.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Describes the viewport.

    System.NotSupportedException - Thrown if the implementation does not support packed semantics texture.

    Returns:

    true if the packed semantics texture descriptor is available and is returned. Otherwise, false.

TryAcquirePackedSemanticChannelsCpuImage​

bool TryAcquirePackedSemanticChannelsCpuImage(
        out XRCpuImage cpuImage,
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Tries to acquire the latest packed semantic channels CPU image.

    Parameters:

    cpuImage - If this method returns true, an acquired XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Describes the viewport.

    Returns:

TryGetSuppressionMaskTexture​

bool TryGetSuppressionMaskTexture(
        out XRTextureDescriptor suppressionMaskDescriptor,
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Tries to generate a semantic suppression mask texture descriptor from the latest semantics.

    Parameters:

    suppressionMaskDescriptor - The semantic suppression mask texture descriptor to be populated, if available from the provider.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Describes the viewport.

    System.NotSupportedException - Thrown if the implementation does not support semantic suppression mask textures.

    Returns:

    true if the suppression mask texture descriptor is available and is returned. Otherwise, false.

TryAcquireSuppressionMaskCpuImage​

bool TryAcquireSuppressionMaskCpuImage(
        out XRCpuImage cpuImage,
        out Matrix4x4 samplerMatrix,
        XRCameraParams? cameraParams = null
    )

Tries to generate a suppression mask XRCpuImage from the latest semantics.

    Parameters:

    cpuImage - If this method returns true, an acquired XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Describes the viewport.

    Returns:

TryGetChannelNames​

bool TryGetChannelNames(out IReadOnlyList<string> names)

Get a list of the semantic channel names for the current semantic model.

    Parameters:

    System.NotSupportedException - Thrown when reading the channel names is not supported by the implementation.

    Returns:

    A list of semantic category labels. The list will be empty if metadata has not yet become available.

TrySetChannelConfidenceThresholds​

bool TrySetChannelConfidenceThresholds(Dictionary<string, float> channelConfidenceThresholds)

Sets the confidence thresholds used for including the specified semantic channels in the packed semantic channel buffer.

Each semantic channel will use its default threshold value chosen by the model until a new value is set by this function during the AR session.

    Parameters:

    channelConfidenceThresholds - A dictionary consisting of keys specifying the name of the semantics channel that is needed and values between 0 and 1, inclusive, that set the threshold above which the platform will include the specified channel in the packed semantics buffer. The key must be a semantic channel name present in the list returned by TryGetChannelNames.

    System.NotSupportedException - Thrown when setting confidence thresholds is not supported by the implementation.

    Returns:

    True if the threshold was set. Otherwise, false.

TryResetChannelConfidenceThresholds​

bool TryResetChannelConfidenceThresholds()

Resets the confidence thresholds for all semantic channels to the default values from the current model.

This reverts any changes made with TrySetChannelConfidenceThresholds.

    Parameters:

    System.NotSupportedException - Thrown when resetting confidence thresholds is not supported by the implementation.

    Returns:

    True if the thresholds were reset. Otherwise, false.

Register​

static bool Register(XRSemanticsSubsystemCinfo semanticsSubsystemCinfo)

Register the descriptor for the semantics subsystem implementation.

    Parameters:

    semanticsSubsystemCinfo - The semantics subsystem implementation construction information.

    Returns:

    true if the descriptor was registered. Otherwise, false.