class XRSemanticsSubsystem

(Niantic.Lightship.AR.XRSubsystems.XRSemanticsSubsystem)

Overview

Defines an interface for interacting with semantic segmentation functionality.

class XRSemanticsSubsystem:
	    SubsystemWithProvider< XRSemanticsSubsystem, XRSemanticsSubsystemDescriptor, XRSemanticsSubsystem.Provider >,
	    ISubsystemWithModelMetadata {
	public:
	
		class Provider;

		// properties
	
		uint TargetFrameRate;
		HashSet<string> SuppressionMaskChannels;
		uint? LatestFrameId;
		bool IsMetadataAvailable;

		// methods
	
		XRSemanticsSubsystem();
	
		bool TryGetSemanticChannel(
			string channelName,
			out XRTextureDescriptor semanticsChannelDescriptor,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquireSemanticChannelCpuImage(
			string channelName,
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryGetPackedSemanticChannels(
			out XRTextureDescriptor packedSemanticsDescriptor,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquirePackedSemanticChannelsCpuImage(
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryGetSuppressionMaskTexture(
			out XRTextureDescriptor suppressionMaskDescriptor,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquireSuppressionMaskCpuImage(
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquireSuppressionMaskCpuImage(
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams,
			Matrix4x4? referencePose
		);
	
		bool TryGetChannelNames(out IReadOnlyList<string> names);
		bool TrySetChannelConfidenceThresholds(Dictionary<string, float> channelConfidenceThresholds);
		bool TryResetChannelConfidenceThresholds();
		static bool Register(XRSemanticsSubsystemCinfo semanticsSubsystemCinfo);

	protected:
		// methods
	
		override void OnStop();
	};

Detailed Documentation

Defines an interface for interacting with semantic segmentation functionality.

Properties

TargetFrameRate

uint TargetFrameRate

Specifies the target frame rate for the platform to target running semantic segmentation inference at.

The target frame rate.

    Parameters:

    System.NotSupportedException - Thrown frame rate configuration is not supported.

LatestFrameId

uint? LatestFrameId

Returns the frame id of the most recent semantic segmentation prediction.

The frame id.

    Parameters:

    System.NotSupportedException - Thrown if getting frame id is not supported.

IsMetadataAvailable

bool IsMetadataAvailable

Is true if metadata has been downloaded and decrypted on the current device. Only if this value is true can the semantic segmentation label names or inference results be acquired.

If metadata is available.

    Parameters:

    System.NotSupportedException - Thrown frame rate configuration is not supported.

Methods

XRSemanticsSubsystem

XRSemanticsSubsystem()

Construct the subsystem by creating the functionality provider.

TryGetSemanticChannel

bool TryGetSemanticChannel(
		string channelName,
		out XRTextureDescriptor semanticsChannelDescriptor,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Gets a semantics channel texture descriptor and a matrix used to fit the texture to the viewport.

    Parameters:

    channelName - The string description of the semantics channel that is needed.

    semanticsChannelDescriptor - The semantics channel texture descriptor to be populated, if available from the provider.

    samplerMatrix - Converts from normalized viewport coordinates to normalized texture coordinates.

    cameraParams - Describes the viewport.

    System.NotSupportedException - Thrown if the implementation does not support semantics channel texture.

    Returns:

    true if the semantics channel texture descriptor is available and is returned. Otherwise, false.

TryAcquireSemanticChannelCpuImage

bool TryAcquireSemanticChannelCpuImage(
		string channelName,
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Tries to acquire the latest semantics channel XRCpuImage.

    Parameters:

    channelName - The string description of the semantics channel that is needed.

    cpuImage - If this method returns true, an acquired XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Params of the viewport to sample with

    Returns:

    Returns true if an XRCpuImage was successfully acquired. Returns false otherwise.

TryGetPackedSemanticChannels

bool TryGetPackedSemanticChannels(
		out XRTextureDescriptor packedSemanticsDescriptor,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Gets a packed semantics texture descriptor.

    Parameters:

    packedSemanticsDescriptor - The packed semantics texture descriptor to be populated, if available from the provider.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Describes the viewport.

    System.NotSupportedException - Thrown if the implementation does not support packed semantics texture.

    Returns:

    true if the packed semantics texture descriptor is available and is returned. Otherwise, false.

TryAcquirePackedSemanticChannelsCpuImage

bool TryAcquirePackedSemanticChannelsCpuImage(
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Tries to acquire the latest packed semantic channels CPU image.

    Parameters:

    cpuImage - If this method returns true, an acquired XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Describes the viewport.

    Returns:

    

TryGetSuppressionMaskTexture

bool TryGetSuppressionMaskTexture(
		out XRTextureDescriptor suppressionMaskDescriptor,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Tries to generate a semantic suppression mask texture descriptor from the latest semantics.

    Parameters:

    suppressionMaskDescriptor - The semantic suppression mask texture descriptor to be populated, if available from the provider.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Describes the viewport.

    System.NotSupportedException - Thrown if the implementation does not support semantic suppression mask textures.

    Returns:

    true if the suppression mask texture descriptor is available and is returned. Otherwise, false.

TryAcquireSuppressionMaskCpuImage

bool TryAcquireSuppressionMaskCpuImage(
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Tries to generate a suppression mask XRCpuImage from the latest semantics.

    Parameters:

    cpuImage - If this method returns true, an acquired XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Describes the viewport.

TryAcquireSuppressionMaskCpuImage

bool TryAcquireSuppressionMaskCpuImage(
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams,
		Matrix4x4? referencePose
	)

Tries to generate a suppression mask XRCpuImage from the latest semantics.

    Parameters:

    cpuImage - If this method returns true, an acquired XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to texture coordinates.

    cameraParams - Describes the viewport.

    referencePose - The pose to calculate the samplerMatrix for.

TryGetChannelNames

bool TryGetChannelNames(out IReadOnlyList<string> names)

Get a list of the semantic channel names for the current semantic model.

    Parameters:

    System.NotSupportedException - Thrown when reading the channel names is not supported by the implementation.

    Returns:

    A list of semantic category labels. The list will be empty if metadata has not yet become available.

TrySetChannelConfidenceThresholds

bool TrySetChannelConfidenceThresholds(Dictionary<string, float> channelConfidenceThresholds)

Sets the confidence thresholds used for including the specified semantic channels in the packed semantic channel buffer.

Each semantic channel will use its default threshold value chosen by the model until a new value is set by this function during the AR session.

    Parameters:

    channelConfidenceThresholds - A dictionary consisting of keys specifying the name of the semantics channel that is needed and values between 0 and 1, inclusive, that set the threshold above which the platform will include the specified channel in the packed semantics buffer. The key must be a semantic channel name present in the list returned by TryGetChannelNames.

    System.NotSupportedException - Thrown when setting confidence thresholds is not supported by the implementation.

    Returns:

    True if the threshold was set. Otherwise, false.

TryResetChannelConfidenceThresholds

bool TryResetChannelConfidenceThresholds()

Resets the confidence thresholds for all semantic channels to the default values from the current model.

This reverts any changes made with TrySetChannelConfidenceThresholds.

    Parameters:

    System.NotSupportedException - Thrown when resetting confidence thresholds is not supported by the implementation.

    Returns:

    True if the thresholds were reset. Otherwise, false.

Register

static bool Register(XRSemanticsSubsystemCinfo semanticsSubsystemCinfo)

Register the descriptor for the semantics subsystem implementation.

    Parameters:

    semanticsSubsystemCinfo - The semantics subsystem implementation construction information.

    Returns:

    true if the descriptor was registered. Otherwise, false.

OnStop

override void OnStop()

Invoked when the subsystem is being stopped.