class XRSemanticsSubsystem

(Niantic.Lightship.AR.XRSubsystems.XRSemanticsSubsystem)

Overview

Defines an interface for interacting with semantic segmentation functionality.

class XRSemanticsSubsystem:
	    SubsystemWithProvider< XRSemanticsSubsystem, XRSemanticsSubsystemDescriptor, XRSemanticsSubsystem.Provider >,
	    ISubsystemWithModelMetadata {
	public:
	
		class Provider;

		// properties
	
		uint TargetFrameRate;
		HashSet<string> SuppressionMaskChannels;
		uint? LatestFrameId;
		bool IsMetadataAvailable;

		// methods
	
		XRSemanticsSubsystem();
	
		bool TryGetSemanticChannel(
			string channelName,
			out XRTextureDescriptor semanticsChannelDescriptor,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquireSemanticChannelCpuImage(
			string channelName,
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryGetPackedSemanticChannels(
			out XRTextureDescriptor packedSemanticsDescriptor,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquirePackedSemanticChannelsCpuImage(
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryGetSuppressionMaskTexture(
			out XRTextureDescriptor suppressionMaskDescriptor,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquireSuppressionMaskCpuImage(
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquireSuppressionMaskCpuImage(
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams,
			Matrix4x4? targetPose
		);
	
		bool TryGetChannelNames(out IReadOnlyList<string> names);
		bool TrySetChannelConfidenceThresholds(Dictionary<string, float> channelConfidenceThresholds);
		bool TryResetChannelConfidenceThresholds();
		static bool Register(XRSemanticsSubsystemCinfo semanticsSubsystemCinfo);

	protected:
		// methods
	
		override void OnStop();
	};

Detailed Documentation

Defines an interface for interacting with semantic segmentation functionality.

Properties

TargetFrameRate

uint TargetFrameRate

Specifies the target frame rate for the platform to target running semantic segmentation inference at.

The target frame rate.

    Parameters:

    System.NotSupportedException - Thrown frame rate configuration is not supported.

LatestFrameId

uint? LatestFrameId

Returns the frame id of the most recent semantic segmentation prediction.

The frame id.

    Parameters:

    System.NotSupportedException - Thrown if getting frame id is not supported.

IsMetadataAvailable

bool IsMetadataAvailable

Is true if metadata has been downloaded and decrypted on the current device. Only if this value is true can the semantic segmentation label names or inference results be acquired.

If metadata is available.

    Parameters:

    System.NotSupportedException - Thrown frame rate configuration is not supported.

Methods

XRSemanticsSubsystem

XRSemanticsSubsystem()

Construct the subsystem by creating the functionality provider.

TryGetSemanticChannel

bool TryGetSemanticChannel(
		string channelName,
		out XRTextureDescriptor semanticsChannelDescriptor,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Get the XRTextureDescriptor for the specified semantic channel.

    Parameters:

    channelName - The string description of the semantics channel to acquire.

    semanticsChannelDescriptor - The resulting semantic channel texture descriptor, if available.

    samplerMatrix - The matrix that transforms from normalized viewport coordinates to normalized texture coordinates.

    cameraParams - Describes the viewport the texture is to be displayed on.

    System.NotSupportedException - Thrown if the implementation does not support semantics channel texture.

    Returns:

    true if the semantic channel texture descriptor is available and is returned. Otherwise, false.

TryAcquireSemanticChannelCpuImage

bool TryAcquireSemanticChannelCpuImage(
		string channelName,
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Acquire the latest semantic channel CPU image.

    Parameters:

    channelName - The string description of the semantics channel to acquire.

    cpuImage - The resulting XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - The matrix that transforms from normalized viewport coordinates to normalized image coordinates.

    cameraParams - Describes the viewport the image is to be displayed on.

    System.NotSupportedException - Thrown if the implementation does not support semantic channels CPU images.

    Returns:

    Returns true if an XRCpuImage was successfully acquired. Returns false otherwise.

TryGetPackedSemanticChannels

bool TryGetPackedSemanticChannels(
		out XRTextureDescriptor packedSemanticsDescriptor,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Get the packed semantics texture descriptor.

    Parameters:

    packedSemanticsDescriptor - The resulting semantic channel texture descriptor, if available.

    samplerMatrix - The matrix that transforms from normalized viewport coordinates to normalized texture coordinates.

    cameraParams - Describes the viewport the texture is to be displayed on.

    System.NotSupportedException - Thrown if the implementation does not support packed semantics texture.

    Returns:

    true if the packed semantics texture descriptor is available and is returned. Otherwise, false.

TryAcquirePackedSemanticChannelsCpuImage

bool TryAcquirePackedSemanticChannelsCpuImage(
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Acquire the latest packed semantic channels XRCpuImage.

    Parameters:

    cpuImage - The resulting XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - The matrix that transforms from normalized viewport coordinates to normalized image coordinates.

    cameraParams - Describes the viewport the image is to be displayed on.

    Returns:

    Returns true if an XRCpuImage was successfully acquired. Returns false otherwise.

TryGetSuppressionMaskTexture

bool TryGetSuppressionMaskTexture(
		out XRTextureDescriptor suppressionMaskDescriptor,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Get a semantic suppression texture descriptor.

    Parameters:

    suppressionMaskDescriptor - The resulting semantic suppression mask texture descriptor, if available.

    samplerMatrix - The matrix that transforms from normalized viewport coordinates to normalized texture coordinates.

    cameraParams - Describes the viewport the texture is to be displayed on.

    System.NotSupportedException - Thrown if the implementation does not support suppression mask texture.

    Returns:

    true if the suppression mask texture descriptor is available and is returned. Otherwise, false.

TryAcquireSuppressionMaskCpuImage

bool TryAcquireSuppressionMaskCpuImage(
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Acquire the latest suppression mask XRCpuImage.

    Parameters:

    cpuImage - The resulting XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - The matrix that transforms from normalized viewport coordinates to normalized image coordinates.

    cameraParams - Describes the viewport the image is to be displayed on.

    Returns:

    Returns true if an XRCpuImage was successfully acquired. Returns false otherwise.

TryAcquireSuppressionMaskCpuImage

bool TryAcquireSuppressionMaskCpuImage(
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams,
		Matrix4x4? targetPose
	)

Acquire the latest suppression mask XRCpuImage.

    Parameters:

    cpuImage - The resulting XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - The matrix that transforms from normalized viewport coordinates to normalized image coordinates.

    cameraParams - Describes the viewport the image is to be displayed on.

    targetPose - Any image acquired has been captured in the past. The target pose argument defines the pose the image needs to synchronize with. If the image can be synchronized, the samplerMatrix will be calibrated to warp the image as if it was taken from the target pose. If this argument is null, the image will not be warped.

    Returns:

    Returns true if an XRCpuImage was successfully acquired. Returns false otherwise.

TryGetChannelNames

bool TryGetChannelNames(out IReadOnlyList<string> names)

Get a list of the semantic channel names for the current semantic model.

    Parameters:

    System.NotSupportedException - Thrown when reading the channel names is not supported by the implementation.

    Returns:

    A list of semantic category labels. The list will be empty if metadata has not yet become available.

TrySetChannelConfidenceThresholds

bool TrySetChannelConfidenceThresholds(Dictionary<string, float> channelConfidenceThresholds)

Sets the confidence thresholds used for including the specified semantic channels in the packed semantic channel buffer.

Each semantic channel will use its default threshold value chosen by the model until a new value is set by this function during the AR session.

    Parameters:

    channelConfidenceThresholds - A dictionary consisting of keys specifying the name of the semantics channel that is needed and values between 0 and 1, inclusive, that set the threshold above which the platform will include the specified channel in the packed semantics buffer. The key must be a semantic channel name present in the list returned by TryGetChannelNames.

    System.NotSupportedException - Thrown when setting confidence thresholds is not supported by the implementation.

    Returns:

    True if the threshold was set. Otherwise, false.

TryResetChannelConfidenceThresholds

bool TryResetChannelConfidenceThresholds()

Resets the confidence thresholds for all semantic channels to the default values from the current model.

This reverts any changes made with TrySetChannelConfidenceThresholds.

    Parameters:

    System.NotSupportedException - Thrown when resetting confidence thresholds is not supported by the implementation.

    Returns:

    True if the thresholds were reset. Otherwise, false.

Register

static bool Register(XRSemanticsSubsystemCinfo semanticsSubsystemCinfo)

Register the descriptor for the semantics subsystem implementation.

    Parameters:

    semanticsSubsystemCinfo - The semantics subsystem implementation construction information.

    Returns:

    true if the descriptor was registered. Otherwise, false.

OnStop

override void OnStop()

Invoked when the subsystem is being stopped.