How to Use Neural Network Model Preloading

Awareness features such as Meshing and Depth use neural network models to know how to draw each pixel in an AR session. To improve runtime performance, model preloading allows you to download model files ahead of time, frontloading the installation process and avoiding just-in-time loading delays that can negatively impact the user experience.

Prerequisites

• You will need a Unity project with Lightship AR enabled. For more information, see Installing ARDK 3.0.

Neural Network Performance Modes

Lightship awareness features support a range of performance modes, allowing developers to select for performance, quality, or a balanced mix. These modes are analogous to the Unity EnvironmentDepthMode values used in the XROcclusionSubsystem.

For example, Fast optimizes for faster processing time at the expense of quality, while Smooth takes more time to deliver the best graphics. A Custom mode is also available for developers who want to supply their own neural network model file without overriding one of the predefined Lightship models. Each mode has its own neural network model file that can be preloaded before using AR features.

To request a feature mode, specify the feature and mode before starting the awareness feature:

DownloadModel(DepthMode.Medium);
DownloadModel(SemanticsMode.Smooth);

See Runtime/Utilities/Preloading/Feature.cs for the definitive list of feature modes.

Register a Local Model File

To avoid performing the expensive download process when starting an AR session, you may want to download the model files beforehand. To support this, ModelPreloader allows you to register local model files for any feature mode.

To register a local model file with ARDK:

1. Ensure that the model file is in a location that your application can access.
2. Create an empty GameObject in the scene where preloading should occur:
   1. In the Hierarchy, right-click the scene, then select Create Empty. Name the new GameObject ModelRegister.
3. Add a new script component to the empty:
   1. Select the empty GameObject in the Hierarchy, then, in the Inspector, click Add Component, then add a new script.
4. Determine which model you would like to preload based on your application's performance and quality needs.
5. Decide when preloading will occur. Preloading can happen any time after AR Foundation has initialized the Lightship loader (when XRGeneralSettings.Instance.Manager.isInitializationComplete is true) and before awareness features have started. For example, if your AR script is a MonoBehaviour, preloading can happen during Start().

note

Registering a model for a feature mode will override that model for the rest of the AR session. If you want to avoid this, use the Custom mode.

6. Instantiate a ModelPreloader object with the ModelPreloaderFactory:

private bool TryInitializePreloader()
{
    IModelPreloader preloader = ModelPreloaderFactory.Create();
    if (null == preloader)
    {
        // Need to wait for XR to initialize before we can preload.
        return false;
    }
}

7. Call RegisterModel(Feature, FeatureMode, filepath) to register the model with ARDK. This file must be accessible by the application and must remain in place for the duration of the AR session.

var statusCode = preloader.RegisterModel(DepthMode.Medium, "/example/path/to/model_file");
if (statusCode is PreloaderStatusCode.Success or
                              PreloaderStatusCode.RequestInProgress or
                              PreloaderStatusCode.FileExistsInCache)
{
    // The model is now registered to the cache, or the model was already in the cache.
    // This will be the model used for Lightship's EnvironmentDepthMode.Medium moving forward.
}

Preload a Lightship Neural Network Model

To preload a neural network model file before starting an awareness feature:

1. Create an empty GameObject in the scene where preloading should occur:
   1. In the Hierarchy, right-click the scene, then select Create Empty. Name the new GameObject ModelPreloader.
2. Add a new script component to the empty:
   1. Select ModelPreloader in the Hierarchy, then, in the Inspector, click Add Component, then add a new script to it.
3. Determine which model you would like to preload based on your application's performance and quality needs.
4. Decide when preloading will occur. Preloading can happen any time after AR Foundation has initialized the Lightship loader (when XRGeneralSettings.Instance.Manager.isInitializationComplete is true) and before awareness features have started. For example, if your AR script is a MonoBehaviour, preloading can happen during Start().

Important!

Model preloading APIs are only available during runtime after a Lightship AR session has started. If a model is not preloaded, awareness features will automatically choose one when starting.

5. Instantiate a ModelPreloader object with the ModelPreloaderFactory:

private void Start()
{
    IModelPreloader preloader = ModelPreloaderFactory.Create();

    if (null == preloader)
    {
        // Need to wait for XR to initialize before we can preload.
        // Defer preloader instantiation to a later function.
        return;
    }
}

6. If you are also using local neural network model files, call ExistsInCache() to check if the model already resides in the cache.

if (preloader.ExistsInCache(DepthMode.Medium))
{
    // The model is already in the cache. No download is required.
}

7. Call DownloadModel(Feature, FeatureMode) to start the download:

preloader.DownloadModel(DepthMode.Medium);

8. If you want to display download progress to the user, periodically call CurrentProgress(). When progress is 1.0, model preloading is complete. For example:

private void Update()
{
    var statusCode = preloader.CurrentProgress(DepthMode.Medium, out float progress)
    if (Mathf.Approximately(progress, 1.0f))
    {
        // The download has completed successfully, or the model was already in the cache
    }
}

Example Model Preloading Script

Click here to expand the example script

using System;
using System.Collections.Generic;
using System.Linq;
using Niantic.Lightship.AR.Utilities.Preloading;

using UnityEngine;
using UnityEngine.Serialization;
using UnityEngine.UI;
using UnityEngine.XR.Management;

public class PreloaderTestManager : MonoBehaviour
{
    [SerializeField]
    private Button _depthButton;

    [SerializeField]
    private Dropdown _depthDropdown;

    [SerializeField]
    private Button _semanticsButton;

    [SerializeField]
    private Dropdown _semanticsDropdown;

    [SerializeField]
    private Text _depthStatusText;

    [SerializeField]
    private Text _semanticsStatusText;

    [SerializeField]
    private Button _clearCacheButton;

    [SerializeField]
    private Text _preloadStatusText;

    [SerializeField]
    private Slider _percentageSlider;

    [SerializeField]
    private Text _percentageText;

    [SerializeField]
    private Text _cacheStatusText;

    [SerializeField]
    private string _localModelPath;

    private IModelPreloader _preloader;

    private DepthMode _depthMode;
    private SemanticsMode _semanticsMode;

    private void Start()
    {
        TryInitializePreloader();
        InitializeDropdownNames<DepthMode>(_depthDropdown);
        InitializeDropdownNames<SemanticsMode>(_semanticsDropdown);
    }

    private void OnDisable()
    {
        _preloader?.Dispose();
        _preloader = null;
    }

    private void InitializeDropdownNames<T>(Dropdown dropdown)
    {
        // Initialize the feature mode names in the dropdown menu.
        SetFeatureModeNames<T>(dropdown);
        dropdown.value = dropdown.options.Select(option => option.text).ToList().IndexOf("Medium");
    }

    private void SetFeatureModeNames<T>(Dropdown dropdown)
    {
        List<string> modeNames = new();
        foreach (var i in Enum.GetValues(typeof(T)))
        {
            var modeName = Enum.GetName(typeof(T), i);
            if (modeName != "Unspecified")
                modeNames.Add(modeName);
        }
        dropdown.AddOptions(modeNames);
    }

    private void Update()
    {
        if (_preloader == null && !TryInitializePreloader())
            return;

        UpdateDownloadProgress();
        UpdateCacheStatusText();
    }

    private bool TryInitializePreloader()
    {
        if (!XRGeneralSettings.Instance.Manager.isInitializationComplete)
        {
            // Need to wait for XR to initialize before we can preload
            return false;
        }

        _depthMode = DepthMode.Medium;
        _semanticsMode = SemanticsMode.Medium;

        _preloader = ModelPreloaderFactory.Create();

        if (null == _preloader)
        {
            // Need to wait for XR to initialize before we can preload
            return false;
        }

        _depthButton.onClick.AddListener
        (
            () =>
            {
                _depthMode = ParseFeatureMode<DepthMode>(_depthDropdown);
                PreloaderStatusCode result;

                if (_depthMode == DepthMode.Custom)
                {
                    if (_localModelPath.Length == 0)
                    {
                        _preloadStatusText.text = "Depth file registration: Please specify a local path to a model file in PreloaderTestManager";
                        return;
                    }

                    result =  _preloader.RegisterModel(_depthMode, _localModelPath);
                    _preloadStatusText.text = "Depth file registration: " + (IsError(result) ? "failed" : "success");
                    return;
                }

                result = _preloader.DownloadModel(_depthMode);
                _preloadStatusText.text = "Depth download started: " + (IsError(result) ? "failed" : "success");
            }
        );

        _semanticsButton.onClick.AddListener
        (
            () =>
            {
                _semanticsMode = ParseFeatureMode<SemanticsMode>(_semanticsDropdown);
                PreloaderStatusCode result;

                if (_semanticsMode == SemanticsMode.Custom)
                {
                    if (_localModelPath.Length == 0)
                    {
                        _preloadStatusText.text = "Semantics file registration: Please specify a local path to a model file in PreloaderTestManager";
                        return;
                    }

                    result =  _preloader.RegisterModel(_semanticsMode, _localModelPath);
                    _preloadStatusText.text = "Semantics file registration: " + (IsError(result) ? "failed" : "success");
                    return;
                }

                result = _preloader.DownloadModel(_semanticsMode);
                _preloadStatusText.text = "Semantics download started: " + (IsError(result) ? "failed" : "success");
            }
        );

        _clearCacheButton.onClick.AddListener
        (
            () =>
            {
                int successes = 0;
                if (_preloader.ClearFromCache(_depthMode))
                    successes++;

                if (_preloader.ClearFromCache(_semanticsMode))
                    successes++;

                _preloadStatusText.text = "Clear cache: " + successes + " successes";
            }
        );

        _depthDropdown.onValueChanged.AddListener
        (
            (int val) =>
            {
                var mode = val + 1; // We skip the 'unspecified' mode
                _depthMode = (DepthMode) mode;
            }
        );

        _semanticsDropdown.onValueChanged.AddListener
        (
            (int val) =>
            {
                var mode = val + 1; // We skip the 'unspecified' mode
                _semanticsMode = (SemanticsMode) mode;
            }
        );

        return true;
    }

    private static T ParseFeatureMode<T>(Dropdown dropdown)
    {
        var modeName = dropdown.options[dropdown.value].text;

        T mode = (T) Enum.Parse(typeof(T), modeName);
        return mode;
    }

    private void UpdateDownloadProgress()
    {
        // Display the progress of the feature modes that are selected with the dropdown menus
        var depthStatus = _preloader.CurrentProgress(_depthMode, out var selectedDepthProgress);
        if (IsError(depthStatus))
        {
            _depthStatusText.text = depthStatus == PreloaderStatusCode.RequestNotFound ?
                "0%" : "Failure: " + depthStatus.ToString();
        }
        else
        {
            _depthStatusText.text = (selectedDepthProgress * 100).ToString("0") + "%";
        }

        var semanticsStatus = _preloader.CurrentProgress(_semanticsMode, out var selectedSemanticsProgress);
        if (IsError(semanticsStatus))
        {
            _semanticsStatusText.text = semanticsStatus == PreloaderStatusCode.RequestNotFound ?
                "0%" : "Failure: " + semanticsStatus.ToString();
        }
        else
        {
            _semanticsStatusText.text = (selectedSemanticsProgress * 100).ToString("0") + "%";
        }

        // Summarize their download progress
        float combinedProgress = 0, activeDownloads = 0;
        if (selectedDepthProgress > 0 && selectedDepthProgress < 1)
        {
            combinedProgress += selectedDepthProgress;
            activeDownloads++;
        }

        if (selectedSemanticsProgress > 0 && selectedSemanticsProgress < 1)
        {
            combinedProgress += selectedSemanticsProgress;
            activeDownloads++;
        }

        float totalProgress = activeDownloads > 0 ? combinedProgress / activeDownloads : 0;
        _percentageText.text = (totalProgress * 100).ToString("0") + "%";
        _percentageSlider.value = totalProgress;
    }

    private void UpdateCacheStatusText()
    {
        // Cache status
        List<string> modeNames = new();
        DepthMode depthMode = new DepthMode();
        SemanticsMode semanticsMode = new SemanticsMode();
        string cacheStatusText = "Model files preloaded in cache: " + System.Environment.NewLine;

        // Update the cache status for all depth modes
        foreach (DepthMode i in Enum.GetValues(typeof(DepthMode)))
        {
            if (i == DepthMode.Unspecified)
                continue;

            depthMode = i;
            var present = _preloader.ExistsInCache(depthMode);

            if (present)
                modeNames.Add(Enum.GetName(typeof(DepthMode), i) + " Depth");
        }

        // Update the cache status for all semantics modes
        foreach (SemanticsMode i in Enum.GetValues(typeof(SemanticsMode)))
        {
            if (i == SemanticsMode.Unspecified)
                continue;

            semanticsMode = i;
            var present = _preloader.ExistsInCache(semanticsMode);

            if (present)
                modeNames.Add(Enum.GetName(typeof(SemanticsMode), i) + " Semantics");
        }


        // Summarize cache status
        for (int i = 0; i < modeNames.Count; i++)
        {
            cacheStatusText += modeNames[i] + (i < modeNames.Count - 1 ? ", " : "");
        }

        _cacheStatusText.text = cacheStatusText;
    }

    private static bool IsError(PreloaderStatusCode statusCode)
    {
        if (statusCode is PreloaderStatusCode.Success or
                            PreloaderStatusCode.RequestInProgress or
                            PreloaderStatusCode.FileExistsInCache)
            return false;

        return true;
    }
}

More Information

• Neural Network Model Preloading Feature page