Detecting Images

Search for images in the real world and use them for a variety of AR effects.

Overview

An ARSession can be configured to search for specified images in the real world and surface them as image anchors. This anchor can then be used for a variety of effects such as:

  • Making a character seem to turn 3D and jump out of the image.

  • Making a treasure chest appear in a specific real world location near a landmark (for example, in a mural).

  • Making a static image “animate” by creating a virtual overlay.

An ARSession will update an image anchor over time if its initial estimate is wrong or if the image moves, but these updates won’t necessarily be smooth or consistent because the algorithm is designed for detection, not tracking.

Enabling Image Detection

Creating the Image Set

The ARSession needs to be told up front what images to detect. This is done by creating an IARReferenceImage for each image and including them in the ARWorldTrackingConfiguration.DetectionImages that is passed to IARSession.Run.

ARReferenceImage s are created by calling ARReferenceImageFactory.Create, which has overrides allowing a reference image to be created from various sources.

private IARSession _session;  // A session that is initialized elsewhere.

public void RunSessionWithImageDetection
  (IARWorldTrackingConfiguration config)
{
  float imageSize = 0.3f;  // 30 centimeters.

  var imageSet = new HashSet<IARReferenceImage>();

  string jpegFilePath = "path/to/your/file.jpg";
  var referenceImage1 = ARReferenceImageFactory.Create("image1", jpegFilePath, imageSize);
  if (referenceImage1 != null)
    imageSet.Add(referenceImage1);

  byte[] jpegFileContents = new byte[0];  // Load jpeg from a file, or receive the bytes from the network.
  var referenceImage2 = ARReferenceImageFactory.Create("image2", jpegFileContents, jpegFileContents.Length, imageSize);
  if (referenceImage2 != null)
    imageSet.Add(referenceImage2);

  // The raw RGBA values of an image, possibly loaded from a file like a PNG, received from the network, or extracted from the camera.
  // In this case they're in ARGB format.
  byte[] rawImage = new byte[0];
  int rawImageWidth = 300;
  int rawImageHeight = 300;
  var referenceImage3 = ARReferenceImageFactory.Create("image3", rawImage,
    rawImageWidth, rawImageHeight, ByteOrderInfo.big32, AlphaInfo.First, 4, imageSize);
  if (referenceImage3 != null)
    imageSet.Add(referenceImage3);

  config.DetectionImages = imageSet.AsArdkReadOnly();

  session.Run(config);
}

Creating the Image Set Asynchronously

Loading and processing images can take enough time that ARDK provides asynchronous versions of all the ARReferenceImageFactory functionality, enabling other UI elements to continue updating while your images load.

private IARSession _session;  // A session that is initialized elsewhere.

public IEnumerator RunSessionWithImageDetectionAsynchronously
  (IARWorldTrackingConfiguration config)
{
  float imageSize = 0.3f;  // 30 centimeters.

  var imageSet = new HashSet<IARReferenceImage>();

  var imageProcessed = false;
  Action<IARReferenceImage> addImageToSet = image =>
  {
    if (image != null)
      imageSet.Add(image);

    imageProcessed = true;
  };

  string jpegFilePath = "path/to/your/file.jpg";
  ARReferenceImageFactory.CreateAsync("image1", jpegFilePath, imageSize, addImageToSet);

  while (!imageProcessed)
    yield return null;

  imageProcessed = false;

  byte[]
    jpegFileContents =
      new byte[0]; // Load jpeg from a file, or receive the bytes from the network.

  ARReferenceImageFactory.CreateAsync
    ("image2", jpegFileContents, jpegFileContents.Length, imageSize, addImageToSet);

  while (!imageProcessed)
    yield return null;

  imageProcessed = false;

  // The raw RGBA values of an image, possibly loaded from a file like a PNG, received from the network, or extracted from the camera.
  // In this case they're in ARGB format.
  byte[] rawImage = new byte[0];
  int rawImageWidth = 300;
  int rawImageHeight = 300;
  ARReferenceImageFactory.CreateAsync
  (
    "image3",
    rawImage,
    rawImageWidth,
    rawImageHeight,
    ByteOrderInfo.big32,
    AlphaInfo.First,
    4,
    imageSize,
    addImageToSet
  );

  while (!imageProcessed)
    yield return null;

  imageProcessed = false;

  config.SetDetectionImagesAsync
  (
    imageSet.AsArdkReadOnly(),
    delegate { session.Run(config); }
  );
}

Using ARImageDetectionManager

To simplify the process of enabling image detection and creating GameObjects to track the real world images, we provide a Manager you can add to your scene. The API reference and in-code comments/tool tips for ARImageDetectionManager explains how to use it.

Picking the Best Images

To help an ARSession surface image anchors with optimal speed and accuracy, use reference images that have lots of non-repetitive visual features. On Android there is a strict requirement that the image be at least 300x300 pixels.

Using Image Anchors

Subscribing to Anchor Events

Once an image has been detected in the environment, an ARImageAnchor will be created for it. It will first show up in the ARSession.AnchorsAdded callback, and then in future ARSession.AnchorsUpdated callbacks.

This is an example of attaching a game object to detected images, either directly attached to the image anchor or attached to a new anchor based on what image is found.

private IARSession _session;  // A session that is initialized elsewhere.

private HashSet<string> _imagesToAttachTo = new HashSet<string>(
new string[]{
  "imageToAttachTo",
  "anotherImageToAttachTo"
});

private bool ShouldAttachToImage(IARReferenceImage image)
{
  return _imagesToAttachTo.Contains(image.Name);
}

Dictionary<Guid, GameObject> _attachedAnchors = new Dictionary<Guid, GameObject>();

Private void SetupCallbacks()
{
  _session.AnchorsAdded += OnAnchorsAdded;
  _session.AnchorsUpdated += OnAnchorsUpdated;
}

private void OnAnchorsAdded(AnchorsArgs args)
{
  foreach (var anchor in args.Anchors)
  {
    if (anchor.AnchorType != AnchorType.Image)
      continue;

    var imageAnchor = anchor as IARImageAnchor;
    if (imageAnchor == null)
      continue;

    IARAnchor anchorToAttachTo;
    if (ShouldAttachToImage())
    {
      anchorToAttachTo = anchor;
    }
    else
    {
      anchorToAttachTo = _session.AddAnchor(anchor.Transform, 1.0f);
    }

    var attachedObject = Instantiate();  // Create whatever game object should be attached.
    _attachedAnchors[anchor.Identifier] = attachedObject;
    attachedObject.transform.position = anchor.Transform.ToPosition();
    attachedObject.transform.rotation = anchor.Transform.ToRotation();
  }
}

private void OnAnchorsUpdated(AnchorsArgs args)
{
  foreach (var anchor in args.Anchors)
  {
    if (!_attachedAnchors.ContainsKey(anchor.Identifier))
      continue;

    _detectedImages[anchor.Identifier].transform.position = anchor.Transform.ToPosition();
    _detectedImages[anchor.Identifier].transform.rotation = anchor.Transform.ToRotation();
  }
}

Mock Image Anchors

If you want to iterate on an image detection experience in the Unity Editor, refer to the instructions for adding mock image anchors on the Mock Mode page.

Best Practices

Images To Use

The quality of image tracking is highly dependant on the image being tracked. What makes an easily tracked image isn’t immediately intuitive. You’re looking for an image that:

  • Has many image features. Areas of high visual contrast. Something solid color or with a gradient won’t have the features you need.

  • Doesn’t have repeating patterns or features. A chess board has sharp visual features at the intersections of squares, but the visual features repeat constantly making them less useful for placing the image.

ARCore and ARKit both provide tools for determining the quality of a given image. You should use those tools if you need a way to verify an assumption about whether an image will be good or bad for tracking.

Placing Virtual Content

Image detection does not cleanly track moving images, so the anchor is prone to changing its position suddenly. It updates in discrete jumps rather than continuously updating from its old position to its new one, and the angle of the image plane can change more dramatically than plane anchors tend to, making the orientation especially inconsistent. These issues are prominent whenever an image moves, but can also cause problem for stationary images as errors in the image detection are corrected. Depending on the type of virtual content you’re placing, there are ways to work around these issues.

  • For detachable 3D content, like an astronaut who jumps out of a picture and then floats around, it is best to create a new ARAnchor at the image anchor’s position and attach the virtual object to that anchor instead. That anchor should update its pose more smoothly and consistently than the image anchor.

  • Similarly, add a new ARAnchor for content that isn’t near the image but uses it as a landmark. For example, once the ARSession detects a movie poster, place an ARAnchor a meter in front of the image anchor and attach your virtual content to that.

  • For flat 2D content that is directly tied to the image, like a virtual picture frame, the image anchor should be used directly. In this case, being close to the most up-to-date detected location of the image is more important than being consistent to where the virtual object was last frame. This will look poorly tracked if the image moves, but should fix itself once the image stops moving.

  • 3D content that is directly tied to the image, like a virtual character who reaches out of the image but is still connected directly to it, will most spotlight the weaknesses of image detection. However, creative techniques can mitigate these issues.

    • Any small changes in the image anchor’s orientation will cause the character reaching out of it to move a lot, as anything far from the image’s center is on a long lever arm that turns small rotational changes into large translational changes. So, if the image is known to be on a wall, attach the virtual object to the image anchor’s position but ignore its orientation.

    • If it’s important that the virtual object seamlessly mesh with the image, consider entirely occluding the real world image with a virtual copy of it.