class ARSemanticSegmentationManager

(Niantic.Lightship.AR.Semantics.ARSemanticSegmentationManager)

Overview

The ARSemanticSegmentationManager controls the XRSemanticsSubsystem and updates the semantics textures on each Update loop. Textures and XRCpuImages are available for confidence maps of individual semantic segmentation channels and a bit array indicating which semantic channels have surpassed the chosen confidence threshold per pixel. For cases where a semantic segmentation texture is overlaid on the screen, utilities are provided to read semantic properties at a given point on the screen.

	class ARSemanticSegmentationManager: SubsystemLifecycleManager< XRSemanticsSubsystem, XRSemanticsSubsystemDescriptor, XRSemanticsSubsystem.Provider > {
	public:
		// fields
	
		 IReadOnlyList<string> ChannelNames => _readOnlyChannelNames;
		 IReadOnlyDictionary<string, int> ChannelIndices => _readOnlyChannelNamesToIndices;

		// properties
	
		uint??? TargetFrameRate;
		bool IsMetadataAvailable;
		Action<ARSemanticSegmentationModelEventArgs> MetadataInitialized;

		// events
	
		event FrameReceived();

		// methods
	
		void Update();
	
		Texture2D GetSemanticChannelTexture(
			string channelName,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		Texture2D GetPackedSemanticsChannelsTexture(
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		Texture2D GetSuppressionMaskTexture(
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquireSemanticChannelCpuImage(
			string channel,
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquirePackedSemanticChannelsCpuImage(
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		bool TryAcquireSuppressionMaskCpuImage(
			out XRCpuImage cpuImage,
			out Matrix4x4 samplerMatrix,
			XRCameraParams? cameraParams = null
		);
	
		int GetChannelIndex(string channelName);
	
		uint GetSemantics(
			int viewportX,
			int viewportY,
			XRCameraParams? cameraParams = null
		);
	
		List<int> GetChannelIndicesAt(
			int viewportX,
			int viewportY,
			XRCameraParams? cameraParams = null
		);
	
		List<string> GetChannelNamesAt(
			int viewportX,
			int viewportY,
			XRCameraParams? cameraParams = null
		);
	
		bool DoesChannelExistAt(
			int viewportX,
			int viewportY,
			string channelName,
			XRCameraParams? cameraParams = null
		);
	
		bool DoesChannelExistAt(
			int viewportX,
			int viewportY,
			int channelIndex,
			XRCameraParams? cameraParams = null
		);
	
		bool TrySetChannelConfidenceThresholds(Dictionary<string, float> channelConfidenceThresholds);
		bool TryResetChannelConfidenceThresholds();

	protected:
		// methods
	
		override void OnBeforeStart();
		override void OnDisable();
	};

Detailed Documentation

The ARSemanticSegmentationManager controls the XRSemanticsSubsystem and updates the semantics textures on each Update loop. Textures and XRCpuImages are available for confidence maps of individual semantic segmentation channels and a bit array indicating which semantic channels have surpassed the chosen confidence threshold per pixel. For cases where a semantic segmentation texture is overlaid on the screen, utilities are provided to read semantic properties at a given point on the screen.

Fields

ChannelNames

IReadOnlyList<string> ChannelNames => _readOnlyChannelNames

The names of the semantic channels that the current model is able to detect.

ChannelIndices

IReadOnlyDictionary<string, int> ChannelIndices => _readOnlyChannelNamesToIndices

The indices of the semantic channels that the current model is able to detect.

Properties

TargetFrameRate

uint??? TargetFrameRate

Frame rate that semantic segmentation inference will aim to run at.

IsMetadataAvailable

bool IsMetadataAvailable

True if the underlying subsystem has finished initialization.

MetadataInitialized

Action<ARSemanticSegmentationModelEventArgs> MetadataInitialized

An event which fires when the underlying subsystem has finished initializing.

Methods

OnBeforeStart

override void OnBeforeStart()

Callback before the subsystem is started (but after it is created).

OnDisable

override void OnDisable()

Callback when the manager is being disabled.

Update

void Update()

Callback as the manager is being updated.

GetSemanticChannelTexture

Texture2D GetSemanticChannelTexture(
		string channelName,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Returns semantic segmentation texture for the specified semantic channel.

    Parameters:

    channelName - The semantic channel to acquire.

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    The texture for the specified semantic channel, if any. Otherwise, null.

GetPackedSemanticsChannelsTexture

Texture2D GetPackedSemanticsChannelsTexture(
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Retrieves the texture of semantic data where each pixel can be interpreted as a uint with bits corresponding to different classifications.

    Parameters:

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    The packed semantics texture, owned by the manager, if any. Otherwise, null.

GetSuppressionMaskTexture

Texture2D GetSuppressionMaskTexture(
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Retrieves the suppression mask texture, where each pixel contains a uint which can be used to interpolate between the predicted depth and the far field depth of the scene. This is useful for enabling smooth occlusion suppression.

    Parameters:

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    The suppression mask texture, owned by the manager, if any. Otherwise, null.

TryAcquireSemanticChannelCpuImage

bool TryAcquireSemanticChannelCpuImage(
		string channel,
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Attempt to acquire the latest semantic segmentation XRCpuImage for the specified semantic class. This provides direct access to the raw pixel data.

The XRCpuImage must be disposed to avoid resource leaks.

    Parameters:

    channel - The semantic channel to acquire.

    cpuImage - If this method returns true, an acquired XRCpuImage. The XRCpuImage must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    True if the CPU image was acquired. Otherwise, false

TryAcquirePackedSemanticChannelsCpuImage

bool TryAcquirePackedSemanticChannelsCpuImage(
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Tries to acquire the latest packed semantic channels XRCpuImage. Each element of the XRCpuImage is a bit field indicating which semantic channels have surpassed their respective detection confidence thresholds for that pixel. (See GetChannelIndex)

The utility GetChannelNamesAt can be used for reading semantic channel names at a viewport location.

    Parameters:

    cpuImage - If this method returns true, an acquired XRCpuImage. The CPU image must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    True if the CPU image was acquired. Otherwise, false

TryAcquireSuppressionMaskCpuImage

bool TryAcquireSuppressionMaskCpuImage(
		out XRCpuImage cpuImage,
		out Matrix4x4 samplerMatrix,
		XRCameraParams? cameraParams = null
	)

Tries to acquire the latest suppression mask XRCpuImage. Each element of the XRCpuImage is a uint32 value which can be used to interpolate between instantaneous depth and far field depth.

    Parameters:

    cpuImage - If this method returns true, an acquired XRCpuImage. The CPU image must be disposed by the caller.

    samplerMatrix - A matrix that converts from viewport to image coordinates according to the latest pose.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    True if the CPU image was acquired. Otherwise, false

GetChannelIndex

int GetChannelIndex(string channelName)

Get the channel index of a specified semantic class. This corresponds to a bit position in the packed semantics buffer, with index 0 being the most-significant bit.

    Parameters:

    channelName - The name of the semantic class.

    Returns:

    The index of the specified semantic class, or -1 if the channel does not exist.

GetSemantics

uint GetSemantics(
		int viewportX,
		int viewportY,
		XRCameraParams? cameraParams = null
	)

Returns the semantics at the specified pixel on screen.

    Parameters:

    viewportX - Horizontal coordinate in viewport space.

    viewportY - Vertical coordinate in viewport space.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    A 32-bit packed unsigned integer where each bit is a binary indicator for a class, and the most-significant bit corresponds to the channel that is the 0th element of the ChannelNames list.

GetChannelIndicesAt

List<int> GetChannelIndicesAt(
		int viewportX,
		int viewportY,
		XRCameraParams? cameraParams = null
	)

Returns an array of channel indices that are present at the specified pixel onscreen.

This query allocates garbage.

    Parameters:

    viewportX - Horizontal coordinate in viewport space.

    viewportY - Vertical coordinate in viewport space.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    An array of channel indices present for the pixel.

GetChannelNamesAt

List<string> GetChannelNamesAt(
		int viewportX,
		int viewportY,
		XRCameraParams? cameraParams = null
	)

Returns an array of channel names that are present for the specified pixel onscreen.

This query allocates garbage.

    Parameters:

    viewportX - Horizontal coordinate in viewport space.

    viewportY - Vertical coordinate in viewport space.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    An array of channel names present for the pixel.

DoesChannelExistAt

bool DoesChannelExistAt(
		int viewportX,
		int viewportY,
		string channelName,
		XRCameraParams? cameraParams = null
	)

Check if a semantic class is detected at the specified location in screen space, based on the confidence threshold set for this channel. (See TrySetChannelConfidenceThresholds)

    Parameters:

    viewportX - Horizontal coordinate in viewport space.

    viewportY - Vertical coordinate in viewport space.

    channelName - Name of the semantic class to look for.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    True if the semantic class exists at the given coordinates.

DoesChannelExistAt

bool DoesChannelExistAt(
		int viewportX,
		int viewportY,
		int channelIndex,
		XRCameraParams? cameraParams = null
	)

Check if a semantic class is detected at the specified location in screen space, based on the confidence threshold set for this channel. (See TrySetChannelConfidenceThresholds)

    Parameters:

    viewportX - Horizontal coordinate in viewport space.

    viewportY - Vertical coordinate in viewport space.

    channelIndex - Index of the semantic class to look for in the ChannelNames list.

    cameraParams - Params of the viewport to sample with. Defaults to current screen dimensions if null.

    Returns:

    True if the semantic class exists at the given coordinates.

TrySetChannelConfidenceThresholds

bool TrySetChannelConfidenceThresholds(Dictionary<string, float> channelConfidenceThresholds)

Sets the confidence threshold for including the specified semantic channel in the packed semantic channel buffer.

Each semantic channel will use its default threshold value chosen by the model until a new value is set by this function during the AR session. Changes to the semantic segmentation thresholds are undone by either restarting the subsystem or by calling TryResetChannelConfidenceThresholds.

    Parameters:

    channelConfidenceThresholds - A dictionary consisting of keys specifying the name of the semantics channel that is needed and values between 0 and 1, inclusive, that set the threshold above which the platform will include the specified channel in the packed semantics buffer. The key must be a semantic channel name present in the list returned by TryGetChannelNames.

    System.NotSupportedException - Thrown when setting confidence thresholds is not supported by the implementation.

    Returns:

    True if the thresholds were set. Otherwise, false.

TryResetChannelConfidenceThresholds

bool TryResetChannelConfidenceThresholds()

Resets the confidence thresholds for all semantic channels to the default values from the current model.

This reverts any changes made with TrySetChannelConfidenceThresholds.

    Parameters:

    System.NotSupportedException - Thrown when resetting confidence thresholds is not supported by the implementation.

    Returns:

    True if the thresholds were reset. Otherwise, false.