Skip to main content
Version: 3.4

How to Convert a Screen Point to Real-World Position Using Depth

ARDK 3.0's depth map output allows for dynamically placing objects in an AR scene without the use of planes or a mesh. This example covers how to use depth output to choose a point on the screen and place an object there.

Prerequisites​

You will need a Unity project with Lightship AR enabled. For more information, see Installing ARDK 3.

tip

If this is your first time using depth, Accessing and Displaying Depth Information provides a simpler use case for depth and is easier to start with.

Steps​

If the main scene is not AR-ready, set it up:

  1. Remove the Main Camera.

  2. Add an ARSession and XROrigin to the Hierarchy, then add an AR Occlusion Manager Component to either of them.

    AR Session and XR Origin AR Occlusion Manager

  3. Create a script that will handle depth picking and placing prefabs. Name it Depth_ScreenToWorldPosition, then add the corresponding code to it.

Click to show the Depth_ScreenToWorldPosition script
using Niantic.Lightship.AR.Utilities;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
public class Depth_ScreenToWorldPosition : MonoBehaviour
{
    public AROcclusionManager _occMan;
    public ARCameraManager _arCameraManager;
    public Camera _camera;
    public GameObject _prefabToSpawn;
    private Matrix4x4 m_DisplayMatrix;
    XRCpuImage? depthimage;
    private void OnEnable()
    {
        _arCameraManager.frameReceived += OnCameraFrameEventReceived;
    }
    private void OnDisable()
    {
        _arCameraManager.frameReceived -= OnCameraFrameEventReceived;
    }
    private void OnCameraFrameEventReceived(ARCameraFrameEventArgs args)
    {
        // Cache the screen to image transform
        if (args.displayMatrix.HasValue)
        {
#if UNITY_IOS
            m_DisplayMatrix = args.displayMatrix.Value.transpose;
#else
        m_DisplayMatrix = args.displayMatrix.Value;
#endif
        }
    }
    void Update()
    {
        if (!_occMan.subsystem.running)
        {
            return;
        }
        if (_occMan.TryAcquireEnvironmentDepthCpuImage(out
            var image))
        {
            depthimage?.Dispose();
            depthimage = image;
        }
        else
        {
            return;
        }
#if UNITY_EDITOR
        if (Input.GetMouseButtonDown(0))
        {
            var screenPosition = new Vector2(Input.mousePosition.x, Input.mousePosition.y);
#else
        if (Input.touches.Length > 0) {
          var screenPosition = Input.GetTouch(0).position;
#endif
            if (depthimage.HasValue)
            {
                // Sample eye depth
                var uv = new Vector2(screenPosition.x / Screen.width, screenPosition.y / Screen.height);
                var eyeDepth = depthimage.Value.Sample<float>(uv, m_DisplayMatrix);
                // Get world position
                var worldPosition =
                  _camera.ScreenToWorldPoint(new Vector3(screenPosition.x, screenPosition.y, eyeDepth));
                //spawn a thing on the depth map
                Instantiate(_prefabToSpawn, worldPosition, Quaternion.identity);
            }
        }
    }
}
  1. Add the Depth_ScreenToWorldPosition script as a Component of the XROrigin in the Hierarchy:
    1. In the Hierarchy window, select the XROrigin, then click Add Component in the Inspector.
    2. Search for the Depth_ScreenToWorldPosition script, then select it.
  2. Create a Cube to use as the object that will spawn into the scene:
    1. In the Hierarchy, right-click, then, in the Create menu, mouse over 3D Object and select Cube.
    2. Drag the new Cube object from the Hierarchy to the Assets window to create a prefab of it, then delete it from the Hierarchy. (The Cube in the Assets window should remain.)
  3. Assign the fields in the Depth_ScreenToWorldPosition script:
    1. In the Hierarchy window, select the XROrigin, then expand the Depth_ScreenToWorldPosition Component in the Inspector window.
    2. Assign the XROrigin to the Occ Man field.
    3. Assign the Main Camera to the Camera field.
    4. Assign the Main Camera to the _arCameraManager field.
    5. Assign your new Cube prefab to the Prefab to Spawn field.
  4. Open Build Settings, then click Build and Run to build to device and try it out.

More Information​