Writer Control

Overview

Omni.Replicator.Agent (ORA) provides built-in writers as well as the ability for you to create your own custom writers.

Built-in Writers

Tao Writer

Overview

TaoWriter is a writer that publishes aligned images with detected character data. It produces RGB images from camera, corresponding labels with 2d bounding box, 3d bounding box and segmentation data on people. The labels from TaoWriter will have bounding box coordinates for each character in the scene and follow Kitti 3D annotations format.

Parameters

Tao Writer expose the following parameters:

  • rgb: control whether RGB annotator is output

  • bbox: control whether character data is output, including 2d bounding box, 3d bounding box, and joint position character data

  • semantic_segmentation: control whether semantic segmentation is output

  • video: control whether to generate a video from the RGB images (ffmpeg must be installed on the system)

Output

  • Rgb: Current Frame’s render product from camera’s viewport

  • Colored Semantic Segmentation: every character is colored with a distinguishing color

  • Character Information

    For each character that passes the width/height threshold checking, TAO Writer outputs the following data:

    • Semantic label: character’s semantic label

    • 2d tight bounding box: x_max, x_min, y_max, y_min

    • 2d loose bounding box: x_max, x_min, y_max, y_min

    • Character 3d position: character’s position in the Isaac Sim Omniverse space

    • Character joint information

      • Joint Order: [‘Pelvis’, ‘Head’, ‘Left_Shoulder’, ‘Left_Elbow’, ‘Left_Hand’, ‘Right_Shoulder’, ‘Right_Elbow’, ‘Right_Hand’, ‘Left_Thigh’, ‘Left_Knee’, ‘Left_Foot’, ‘Left_Toe’, ‘Right_Thigh’, ‘Right_Knee’, ‘Right_Foot’, ‘Right_Toe’]

      • 3d_joint_position: for each joint output its translate in x, y, z position

      • 2d_joint_position: for each joint output its 2d position on the screen

    • Character 3d bounding_box information

      • 3d bounding box scale: scale value in x, y, z

      • 3d bounding box camera space location: bounding box location in camera space

      • 3d bounding box rotation in degree in camera space

      • 3d bounding box’s vertexes’ 2d projection on screen

  • Video: If the the video option is checked, it uses ffmpeg to generate a video from the output images

Objectron Writer

Overview

Objectron writer outputs a 3D label file in JSON format. This ground truth file is useful for training 3D object detection models or 6-DoF pose estimation. Each file records the camera poses, as well as the 3D bounding boxes of the objects of interest, both in the 2D image plane and the 3D camera space. Additionally, it includes the pose and class of each object.

Parameters

Objectron Writer provides the following parameters:

  • rgb: controls whether RGB annotator is output

  • bbox: controls whether character data is output, including 3d bounding boxes’ scale, camera space rotation, and camera space translation

  • semantic_segmentation: controls whether semantic segmentation is output

  • distance_to_camera: controls whether the depth image is output

Output

  • Rgb: The current frame’s render product from the camera’s viewport

  • Colored Semantic Segmentation: every character is colored with distinguish color

  • Distance to the Camera: Depth

  • Camera Information:

    • Camera Projection Matrix: camera projection matrix

    • Camera View: camera view matrix

    • Camera Intrinsic Data cx, cy, fx, fy

    • Viewport Width/Height

    • Camera’s tranlation: x, y, z in Isaac Sim coordinates

    • Camera’s rotation in quaternion xyzw

  • Character Information

    For each character that passes the width/height threshold checking, the TaAO Writer outputs the following data:

    • Semantic label: character’s semantic label

    • Character_3d_bounding_box information:

      • 3d bounding box camera space location: bounding box location in camera space

      • 3d bounding box rotation in camera space: quaternion xyzw format

      • 3d bounding box’s vertexes’ 2d projection on screen

      • 3d bounding box’s vertexes’ 3d position in camera space

RTSP Writer

Overview

RTSPWriter is a custom writer that publishes annotations of attached render products to an RTSP server. It tracks a dictionary of render products (HydraTexture) by the combo of the annotator name and the render product’s prim path. Each render product is recorded as an instance of RTSPCamera. The published RTSP URL of each RTSPCamera instance is constructed by appending the render product’s camera prim path and the annotator name to the base output directory.

Annotators

These annotators are supported:

  • LdrColor / rgb

  • semantic_segmentation

  • instance_id_segmentation

  • instance_segmentation

  • DiffuseAlbedo

  • Roughness

  • EmissionAndForegroundMask

  • distance_to_camera

  • distance_to_image_plane

  • DepthLinearized

  • HdrColor

Parameters

RTSPWriter accepts the following parameters:

  • annotator: a string variable that specifies the annotator of all attached render products to be streamed. The accepted value is LdrColor / rgb, semantic_segmentation, instance_id_segmentation, instance_segmentation, HdrColor, distance_to_camera, or distance_to_image_plane. The string value must be exact.

  • output_dir: a string variable that specifies the base RTSP URL in the form of rtsp://<RTSP server hostname>:8554/<base topic name>. Given a render product and the annotator, the full RTSP URL is formatted as rtsp://<RTSP server hostname>:8554/<base topic name>_<camera prim path>_<annotator>. For example, if the RTSP server hostname is my_rtsp_server.com; the base topic name is RTSPWriter; the camera prim path is /World/Cameras/Camera_01; the annotator is rgb, then the full RTSP URL is rtsp://my_rtsp_server.com:8554/RTSPWriter_World_Cameras_Camera_01_LdrColor.

Instructions

Before RTSPWriter can stream camera viewports in Omniverse, the following steps are required:

  1. Install and start an RTSP server

    The Writer sends all streams to the same RTSP server. The server can be local. One candidate of RTSP server is MediaMTX. To install and start a Linux version of such a server:

    1. Download and extract a standalone binary from the release page

      mkdir mediamtx; cd mediamtx
wget https://github.com/bluenviron/mediamtx/releases/download/v1.1.1/mediamtx_v1.1.1_linux_amd64.tar.gz
tar xvzf mediamtx_v1.1.1_linux_amd64.tar.gz

    2. Start the server: ./mediamtx

  2. Install FFmpeg

    Install FFmpeg on the same machine where RTSPWriter is executed from. sudo apt update && sudo apt install -y ffmpeg

  3. Register and initialize RTSPWriter

    As an example, the following code snippet prepares to send the rgb or equivalently LdrColor annotators of the render products to rtsp://<RTSP server hostname>:8554. The base topic name is RTSPWriter. The full topic name of the RTSP stream is constructed as RTSPWriter_<camera prim path>_<annotator>. If the annotator data format is supported by NVENC, the raw frame buffers encoding of the render product is performed on one dedicated GPU device. If there are 8 render products and 4 visible GPU devices, every GPU encodes 2 render product frame buffers.

    import omni.replicator.core as rep
...
render_products = []
render_products.append(rep.create.render_product(...))
...
render_products.append(rep.create.render_product(...))
...
writer = rep.WriterRegistry.get("RTSPWriter")
writer.initialize(annotator="rgb", output_dir="rtsp://<RTSP server hostname>:8554/RTSPWriter")
writer.attach(render_products)

Assume the camera prim path of a render product is /World/Cameras/Camera_01, the full topic name of the RTSP stream is RTSPWriter_World_Camera_01_LdrColor.

Custom Writers

ORA supports using custom writers created by you, through the UI or config file directly.

  • To enable it from UI, select Custom from the dropdown of the Replicator Setting panel, enter the writer’s name and its input parameters in the text boxes.

  • To enable it from config file, put the writer’s name and parameters in the replicator section.

ORA obtains the writer using the writer’s name from the Replicator extension, so writers are expected to be registered in the Replicator beforehand. For writers mentioned above (provided by ORA), they are registered when ORA is loaded. Other custom writers must be registered by you. Follow the Custom Writer documentation to create and register custom writers.

Notes

Width/Height Threshold Checking

To make sure characters are not mostly occluded, we apply the following test to select valid characters.

We define two conditions for whether a character should be labeled, visibility in height and visibility in width. See the images for more details.

  • For occluded characters (characters that are blocked by an object within the camera frame), characters must satisfy both the visibility in height and width requirements.

  • For truncated characters (characters that are cut off by the camera frame), characters must satisfy visibility for height OR the visibility for width.

  • Visibility in height

    • If the head is visible and 20% of the height is visible then this condition is met

    ../../_images/omni-replicator-character-writers-notes-1.png

    • If the head is not visible and 60% of the height is visible, this condition is met

    ../../_images/omni-replicator-character-writers-notes-2.png

  • Visibility in width

    • If more than 60% body width is visible, then this condition is met

    ../../_images/omni-replicator-character-writers-notes-3.png