Add Camera and Sensors

Isaac Sim provides a variety of sensors that can be used to sense the environment and robot’s state. This tutorial guides you through attaching a camera sensor to a mock robot, a process that can be generalized to other sensors. Details regarding the camera and other types of sensors can be found in our Advanced Tutorials and Sensor Extensions under Manuals.

Learning Objectives

This tutorial details how to:

  • Add cameras

  • Attach cameras to geometries

Getting Started

Prerequisites

Note

We recommend starting this tutorial using the Isaac/Samples/Rigging/MockRobot/mock_robot_rigged.usd file provided to have a standardized setup.

Adding Camera

To add a camera:

  1. Go to the Menu Bar and select Create > Camera. A camera appears on the stage tree, and a grey wireframe representing the camera’s view appears on the stage.

  2. You can move and rotate the camera’s transform just like any other objects on the stage.

You can also add a camera by moving the current view in the viewport to a view of your choosing, and then go to the Camera button on the upper left hand corner of the viewport display, and select Camera > Create Camera from View. A new camera appears on the Stage tree, and the list of cameras that can be selected in the Camera button is provided.

Camera Inspector Extension

After the cameras are added in the scene, you can use the Camera Inspector Extension to manage the cameras. The Camera Inspector Extension allows you to:

  • Create multiple viewports for each camera

  • Check camera coverage

  • Get and set camera poses in the desired frames

Launching Extension

To open the Camera Inspector extension:

  1. Go to the Menu Bar.

  2. Select Isaac Utils > Workflows > Camera Inspector.

  3. After launching the extension, verify that you can see your camera already selected in the dropdown.

  4. When adding a new camera, you must click the Refresh button to ensure that the extension finds this new camera.

Camera State Textbox

../_images/tutorial_gui_camera_sensors_camera_status_textbox.png

The Camera State textbox near the top of the extension provides a convenient way to copy the position and orientation of your camera directly into code. Click the copy icon on the right of the textbox to copy to your clipboard.

Creating a Viewport

With the camera selected, you can create a new viewport for your camera.

  1. Click on the ‘Create Viewport’ button to the right of the camera dropdown menu.

By default, this creates a new viewport and assigns the current selected Camera to it.

  1. Assign different cameras to different viewports using the two dropdown menus and buttons in the extension:

    ../_images/tutorial_gui_camera_sensors_create_viewport.png

  2. After launching your viewport, you can change the resolution using the menu in the top left and going to Viewport.

Note

When changing the resolution, Omniverse Kit only supports square pixels. This means that the resolution aspect ratio must be the same as the aperture ratio.

../_images/tutorial_gui_camera_sensors_viewport_config.png

Camera Status Panel

../_images/tutorial_gui_camera_sensors_camera_status_panel.png

Below the camera and viewport dropdown menus is the camera status panel.

This panel displays the location and orientation of the camera in the local coordinate system and also the world coordinate system. By default, it displays the coordinates in the World Axis. However, using the Camera Axes dropdown, this can be changed to be in the USD Axis or the ROS Axis.

The camera status panel updates automatically, if the camera is moved in the stage using the gizmo. It also updates if the location or orientation of the camera is updated in the property panel.

Attach a Camera to Robot

  1. Rename the newly added camera to car_camera.

  2. It is easier to place the camera if you can see the desired camera input stream and where it is relative to the robot from an outside camera. Open up a second viewport window by going to the Menu Bar and click Window > Viewport > Viewport 2. A new viewport appears; dock it wherever you’d like.

  3. Keep one of the viewports in Perspective camera view, and change the other one to car_camera view. Find the Cameras menu on the top edge of the viewport, and switch to Camera > car_camera.

  4. Validate that you have a view of the onboard camera and an overview of the scene.

  5. Attach the camera to the robot’s body by dragging the prim under body. The camera moves together with the body. You may need to switch the camera view for the viewport again.

  6. Point the camera slightly down and make it face forward so you can see the car and the ground. Set the camera transform translation to x=-0.6,y=0,z=2.2, orientation to x=0,y=-80,z=-90, and scale to x=1,y=1,z=1.

  7. Verify that you see the viewport showing the onboard camera view splitting the window between the robot’s body and the ground and the relative position and orientation of the camera to the robot in the Perspective camera viewport.

  8. Press Play. The camera onboard the robot moves with the robot.

A similar strategy is used to apply other onboard sensors.

Important

If the view of the camera is moved while displaying, it changes the camera’s properties. Instead, affix a prim to the parent with the correct offset and affix the camera to that new prim. Then, if the camera position is accidentally moved, it can be reset by zeroing all its position and orientation parameters relative to the prim, which cannot be easily changed.

Summary

In this tutorial, you learned how to use the Camera Inspector Extension. Additionally, you also learned how to add a camera to the robot.

Next Steps

Further Reading