> ## AI Agent Context
> Get the complete documentation index at: https://aistudio.google.com/llms.txt
> Use this file to discover all available pages.
> For the full documentation in a single file, fetch: https://aistudio.google.com/llms-full.txt

# Spatial reasoning

Gemini Robotics ER models can point to objects, track them in video, detect
them with bounding boxes, and generate movement trajectories.

For full runnable code, see the
[Robotics cookbook](https://github.com/google-gemini/robotics-samples/blob/main/Getting%20Started/gemini_robotics_er.ipynb).

## Point to objects  {% id="pointing-to-objects" %}

The following example finds specific objects in an image and returns their
normalized `[y, x]` coordinates:

{% tabs %}
  {% tab filename="robotics_spatial.py" label="Python" %}
  ```python
  from google import genai
  
  PROMPT = """
            Point to no more than 10 items in the image. The label returned
            should be an identifying name for the object detected.
            The answer should follow the json format: [{"point": <point>,
            "label": <label1>}, ...]. The points are in [y, x] format
            normalized to 0-1000.
          """
  client = genai.Client()
  
  uploaded_file = client.files.upload(file="my-image.png")
  
  image_response = client.interactions.create(
      model="gemini-robotics-er-2-preview",
      input=[
          {
              "type": "image",
              "uri": uploaded_file.uri,
              "mime_type": uploaded_file.mime_type
          },
          {"type": "text", "text": PROMPT}
      ],
      generation_config={"thinking_level": "high"},
  )
  
  print(image_response.output_text)
  ```

  {% /tab %}

  {% tab filename="robotics_spatial.sh" label="REST" %}
  ```bash
  # First, ensure you have the image file locally.
  # Encode the image to base64
  IMAGE_BASE64=$(base64 -w 0 my-image.png)
  
  curl -X POST \
    "https://generativelanguage.googleapis.com/v1beta/interactions" \
    -H "x-goog-api-key: $GEMINI_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "gemini-robotics-er-2-preview",
      "input": {
        "parts": [
          {
            "inlineData": {
              "mimeType": "image/png",
              "data": "'"${IMAGE_BASE64}"'"
            }
          },
          {
            "text": "Point to no more than 10 items in the image. The label returned should be an identifying name for the object detected. The answer should follow the json format: [{\"point\": [y, x], \"label\": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000."
          }
        ]
      },
      "generation_config": {
        "thinking_config": {
          "thinking_level": "high"
        }
      }
    }'
  ```

  {% /tab %}

{% /tabs %}

The output will be a JSON array containing objects, each with a `point`
(normalized `[y, x]` coordinates) and a `label` identifying the object.

{% tabs %}
  {% tab filename="output.json" label="JSON" %}
  ```json
  [
    {"point": [376, 508], "label": "small banana"},
    {"point": [287, 609], "label": "larger banana"},
    {"point": [223, 303], "label": "pink starfruit"},
    {"point": [435, 172], "label": "paper bag"},
    {"point": [270, 786], "label": "green plastic bowl"},
    {"point": [488, 775], "label": "metal measuring cup"},
    {"point": [673, 580], "label": "dark blue bowl"},
    {"point": [471, 353], "label": "light blue bowl"},
    {"point": [492, 497], "label": "bread"},
    {"point": [525, 429], "label": "lime"}
  ]
  ```

  {% /tab %}

{% /tabs %}

The following image is an example of how these points can be displayed:

![](https://www.gstatic.com/cloud-cms/9520553bf3464dfdca8d4bf4b4da2951.png)

## Tracking objects in a video  {% id="object-tracking-video" %}

Gemini Robotics ER 2 can also analyze video frames to track objects
over time. See [Video inputs](/docs/video-understanding#supported-formats)
for a list of supported video formats.

{% tabs %}
  {% tab filename="robotics_tracking.py" label="Python" %}
  ```python
  from google import genai
  
  client = genai.Client()
  
  uploaded_file = client.files.upload(file="my-video.mp4")
  
  prompt = """
            Point to the red ball in every frame where it appears.
            The answer should follow the json format: [{"point": [y, x],
            "label": <label>}, ...]. The points are in [y, x] format
            normalized to 0-1000. Return one entry per frame that contains
            the object.
          """
  
  image_response = client.interactions.create(
    model="gemini-robotics-er-2-preview",
    input=[
      {
          "type": "video",
          "uri": uploaded_file.uri,
          "mime_type": uploaded_file.mime_type
      },
      {"type": "text", "text": prompt}
    ],
  )
  
  print(image_response.output_text)
  ```

  {% /tab %}

{% /tabs %}

## Object detection and bounding boxes  {% id="object-detection-bounding-boxes" %}

In addition to points, you can prompt the model to return 2D bounding boxes,
which provide more spatial detail for detected objects.

{% tabs %}
  {% tab filename="robotics_bounding_boxes.py" label="Python" %}
  ```python
  from google import genai
  
  client = genai.Client()
  
  uploaded_file = client.files.upload(file="my-image.png")
  
  prompt = """
            Detect all objects in this image and return bounding boxes.
            The answer should follow the JSON format:
            [{"label": <label>, "y": <y_min>, "x": <x_min>,
              "y2": <y_max>, "x2": <x_max>}, ...]
            where coordinates are normalized to 0-1000.
          """
  
  image_response = client.interactions.create(
    model="gemini-robotics-er-2-preview",
    input=[
      {
          "type": "image",
          "uri": uploaded_file.uri,
          "mime_type": uploaded_file.mime_type
      },
      {"type": "text", "text": prompt}
    ],
  )
  
  print(image_response.output_text)
  ```

  {% /tab %}

{% /tabs %}

## Trajectories  {% id="trajectories" %}

Gemini Robotics ER 2 can generate sequences of points that define a
trajectory, useful for guiding robot movement.

This example requests a trajectory to move a red pen to an organizer, including
an estimate of the intermediate waypoints. The code has been reduced to show
only the prompt.

{% tabs %}
  {% tab filename="robotics_trajectories.py" label="Python" %}
  ```python
  prompt = """
            Generate a trajectory for the robotic arm to pick up the red pen
            and place it in the organizer. Return a list of waypoints as JSON:
            [{"step": <int>, "point": [y, x], "action": <description>}, ...]
            where coordinates are normalized to 0-1000.
          """
  ```

  {% /tab %}

{% /tabs %}

## Making room for a laptop  {% id="making-room-laptop" %}

This example shows how Gemini Robotics ER can reason about a space. The prompt
asks the model to identify which object needs to be moved to create
space for another item.

{% tabs %}
  {% tab filename="robotics_spatial_reasoning.py" label="Python" %}
  ```python
  from google import genai
  
  client = genai.Client()
  
  uploaded_file = client.files.upload(file="path/to/image-with-objects.jpg")
  
  prompt = """
            Point to the object that I need to remove to make room for my laptop
            The answer should follow the JSON format: [{"point": <point>,
            "label": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000.
          """
  
  image_response = client.interactions.create(
    model="gemini-robotics-er-2-preview",
    input=[
      {
          "type": "image",
          "uri": uploaded_file.uri,
          "mime_type": uploaded_file.mime_type
      },
      {"type": "text", "text": prompt}
    ],
  )
  
  print(image_response.output_text)
  ```

  {% /tab %}

{% /tabs %}

The response contains a 2D coordinate of the object that answers the user's
question, in this case, the object that should move to make room for a laptop.

```json
[
  {"point": [672, 301], "label": "The object that I need to remove to make room for my laptop"}
]
```

![](https://www.gstatic.com/cloud-cms/2e93adec0bc100a1db910fd7ea48e758.png)

## Packing a lunch  {% id="packing-lunch" %}

The model can also provide instructions for multi-step tasks and point to
relevant objects for each step. This example shows how the model plans a series
of steps to pack a lunch bag.

{% tabs %}
  {% tab filename="robotics_packing.py" label="Python" %}
  ```python
  from google import genai
  
  client = genai.Client()
  
  uploaded_file = client.files.upload(file="path/to/image-of-lunch.jpg")
  
  prompt = """
            Explain how to pack the lunch box and lunch bag. Point to each
            object that you refer to. Each point should be in the format:
            [{"point": [y, x], "label": }], where the coordinates are
            normalized between 0-1000.
          """
  
  image_response = client.interactions.create(
    model="gemini-robotics-er-2-preview",
    input=[
      {
          "type": "image",
          "uri": uploaded_file.uri,
          "mime_type": uploaded_file.mime_type
      },
      {"type": "text", "text": prompt}
    ],
  )
  
  print(image_response.output_text)
  ```

  {% /tab %}

{% /tabs %}

The response of this prompt is a set of step by step instructions on how to pack
a lunch bag from the image input.

**Input image**

![](https://www.gstatic.com/cloud-cms/ad80224a995418224640d876f08c5e1e.png)

**Model output**

```none
Based on the image, here is a plan to pack the lunch box and lunch bag:

1.  **Pack the fruit into the lunch box.** Place the [apple](apple), [banana](banana), [red grapes](red grapes), and [green grapes](green grapes) into the [blue lunch box](blue lunch box).
2.  **Add the spoon to the lunch box.** Put the [blue spoon](blue spoon) inside the lunch box as well.
3.  **Close the lunch box.** Secure the lid on the [blue lunch box](blue lunch box).
4.  **Place the lunch box inside the lunch bag.** Put the closed [blue lunch box](blue lunch box) into the [brown lunch bag](brown lunch bag).
5.  **Pack the remaining items into the lunch bag.** Place the [blue snack bar](blue snack bar) and the [brown snack bar](brown snack bar) into the [brown lunch bag](brown lunch bag).
```

## What's next  {% id="whats-next" %}

*   [Agentic capabilities](/docs/robotics-agentic) — code execution, instrument reading, image annotation.
*   [Task orchestration](/docs/robotics-orchestration) — long-horizon tasks with custom robot APIs.
*   [Robotics with streaming](/docs/robotics-streaming) — real-time bidirectional streaming (Gemini Robotics ER 2 only).
*   [Video understanding](/docs/robotics-video-progress) — moment finding and progress classification (Gemini Robotics ER 2 only).