Establishing Localization

This document describes the basic steps to setting up AR localization using the ARDK. These steps ensure that multiple clients can interact in a shared AR environment.

These steps assume you’ve already added code to your AR experience that uses ARDK networking APIs to create a ARNetworking session that clients can join. See Starting a Shared Experience and Creating a Multiplayer Experience for more information on creating and joining ARNetworking sessions.

There are two basic approaches: have clients scan and synchronize using objects in their environment, or have clients scan and synchronize using a marker, like a QR-code.

Synchronize Using the Environment

Synchronizing using the environment works as follows:

  • The client designated as the host scans a static object in the environment and successfully uploads at least one initial scan to Niantic’s AR Backend.

  • Non-host clients receive the initial scan data from Niantic’s AR Backend, scan the same object, and then use all of this to synchronize and co-locate within the shared environment.

  • All clients now share the same coordinate space (the host’s) and can share co-located Pose information so that every client can locate other clients in the (shared) world.

Scan the Environment as Host

First, the client designated as the host should scan the environment to collect mapping information that can be shared for synchronizing with other clients. Configure and run an ARSession to collect the mapping data. You can use an ARNetworkingManager or enable IsSharedExperienceEnabled in your ARSession, as described in “Starting a Shared Experience” in Creating Shared AR Experiences.

As soon as IsSharedExperienceEnabled is enabled, the session starts collecting mapping information. The client designated as the host will start scanning the environment and send mapping data back to Niantic’s AR Backend. The host must send at least one map back to Niantic’s AR Backend for the shared session to work, although the host will regularly send updates to Niantic’s AR Backend throughout the lifetime of the session.

To ensure that the host user can scan a good initial map, we recommend providing clear instructions in your game during this scanning process, with the following guidelines communicated to the user:

  • Find and scan a stationary object in your environment. Look for an object with many sharp features (e.g. edges with high contrast, non-symmetrical/repeating patterns) and no shiny or reflective surfaces. One example might be an object with distinct features and colors, like a shoe.

  • Stand 3+ feet away from the object you are mapping.

  • Have your device camera looking straight at the object, not from a high or low angle. Avoid looking at the floor. We want to understand as much of the 3D scene as possible, and wide, flat surfaces don’t help. You may need to position the object on a raised surface, like a table.

  • Keeping your camera looking at the object, slowly move a few steps in a circular direction around the object, both to the left and right. Avoid moving in a direction towards or away from the object. Avoid swiveling the camera (i.e. only rotating around a single point without any side-to-side or up-and-down movement).

  • Continue moving left and right until the scanning completes. This should take approximately 15 seconds, but may take up to several minutes depending on the complexity of the environment and the capabilities of your device.

The following video demonstrates what mapping and localizing can look like in-game:

Note: If you’re testing the scanning process, make sure not to have a debugger attached, as this may negatively impact performance.

Throughout the scanning/mapping process, provide feedback to the user indicating that scanning is progressing. You can render the point cloud data from IARFrame.RawFeaturePoints using ARFeaturePointRenderer to show the user how the scan is progressing.

The host will remain in the WaitingForLocalizationData PeerState until the first scan completes. Once the first scan has completed, the host PeerState will be updated to Stable. You can communicate to the user that a successful scan has been uploaded and that other users can join the session and start to localize. If you want to display the scanned map, you can use ARMapVisualizationRenderer.

Synchronize the Environment as Non-Host Client

When other non-host clients join the session with IsSharedExperienceEnabled enabled, they will need to localize by scanning the same environment and synchronizing with the mapping data uploaded by the host.

When a client joins the session, Niantic’s AR Backend will immediately start sending it any mapping data that was uploaded by the host. You can track the progress by checking the PeerState for the client.

  • If the PeerState is Unknown, the client has not connected to Niantic’s AR Backend yet. You can tell the user that the game is waiting for a connection to Niantic’s AR Backend.

  • If the PeerState is WaitingForLocalizationData, this means the host hasn’t completed uploading an initial map yet, or that there are network issues. You can indicate to the user that the client is waiting for the host to upload data.

  • If the PeerState is Localizing, this means the client has received mapping data from Niantic’s AR Backend. At this point the user should start scanning the same environment as the host. Use the same user guidelines as listed for the host scan process. Users should scan the same object/environment that the host scanned, at the same distance and device orientation.

  • If the PeerState is Stable, this means ARDK has synchronized the user’s scan with the downloaded data from Niantic’s AR Backend and the client is now localized in the shared environment.

  • If the PeerState is Failed, this means the host never successfully uploaded an initial map and left the session. The client should also leave the session and try to connect to a new session.

The following basic code example subscribes to PeerState updates and checks the PeerState:

using Niantic.ARDK.AR.Networking;
using Niantic.ARDK.AR.Networking.ARNetworkingEventArgs;

IARNetworking _arNetworking;

void SubscribeToPeerStateUpdates()
{
 _arNetworking.PeerStateReceived += OnPeerStateReceived;
}
void OnPeerStateReceived(PeerStateReceivedArgs args)
{
  var peer = args.Peer;
  var peerState = args.State;
  // In this example, ignore update if it's not for the local device
  if (_arNetworking.Networking.Self.Identifier != peer.Identifier)
    return;
  // Leave session (or do something else) if localization failed
  if (peerState == PeerState.Failed)
    _arNetworking.Networking.Leave();
  // ...process other PeerStates accordingly...
}

Alternatively, the up-to-date localization states of all peers can be retrieved through the IARNetworking interface.

var allPeerStates = arNetworking.LatestPeerStates;
var myPeerState = arNetworking.LocalPeerState;

See the ARNetworking sample in ARDK-examples and the AR Voyage sample on ARDK Downloads for ways to scan localization data and communicate the process to the user.

See “Peer Poses” in Creating Shared AR Experiences for how to share client positions and orientations in the shared AR space once all clients are localized.

Synchronize Using Markers

ARDK provides an alternate way to localize using markers, like QR Codes. Scanning markers instead of objects in the environment is often easier for users, and works in situations where users can’t find good objects to scan. Additionally markers can contain session information so clients can scan a marker to both join a specific session and synchronize with that session.

Synchronizing using markers is generally less accurate than synchronizing using the environment and can incur more drift between peer positions and orientations. Use marker synchronization in spaces with few scannable objects or features, such as outdoor spaces.

Synchronizing using markers works as follows:

  • Host generates marker data

  • Host calls ARNetworking.InitializeForMarkerScanning and displays the marker on screen

  • Non-host clients use ARNetworking.ScanForMarker and scan the marker displayed on the host device. Clients use results to join session and/or localize in the shared session. A client PeerState of Stabilizing indicates the client has successfully localized in the shared environment using markersync.

See Marker Sync and the MarkerSyncExample scene in ARDK-examples for more details on this process.

Note that the ARDK can collect synchronization information for both environment synchronization and marker synchronization at the same time. Currently, the ARDK collects environment mapping data in the background even while marker synchronization is in progress. This allows the possibility of starting with marker synchronization to quickly synchronize all clients, and then attempt environment synchronization for better accuracy.

To do this, first synchronize using marker synchronization so that all clients are in Stablizing PeerState. Then, instruct players to scan the environment and try to localize for a reasonable amount of time. If the host moves to Stable it means the environment was scanned. If all clients move to Stable it means they were able to localize using the environment. If the host or clients don’t move to Stable within a reasonable amount of time, you can choose to proceed with the experience anyway, as the devices are already synchronized using marker sync.

See Also