Omniverse Common Extension

Introduction

The Common extension is the one extension that every sensor extension depends on. It provides a set of common functionalities/headers that are used by other extensions.

GenericModelOutput

The GenericModelOutput is a common data structure that is used by all sensor extensions to represent the output of the sensor. The output is defined according to ISO855 sensor frame, which means the following:

  • Angles are in degrees from [-180, 180] for azimuth and [-90, 90] for elevation in a right-handed coordinate system.

  • Front is +x, left is +y, up is +z.

Structure Members

  • magicNumberuint32_t

    • A unique identifier for the output. Should reflect MAGIC_NUMBER_GMO, which is the ASCII for “NGMO”.

  • majorVersionuint32_t

    • The major version number of the model output.

  • minorVersionuint32_t

    • The minor version number of the model output.

  • patchVersionuint32_t

    • The patch version number of the model output.

  • numElementsuint32_t

    • The number of elements in the array members of the model output.

  • frameOfReferenceFrameOfReference

    • The frame of reference for the model output. The default value is FrameOfReference::SENSOR.

  • frameIduint64_t

    • The model (simulation) frame ID of the model output.

  • timestampNsuint64_t

    • The timestamp of the model output in nanoseconds.

  • coordsTypeCoordsType

    • The type of coordinates used in the model output. The default value is CoordsType::SPHERICAL.

  • outputTypeOutputType

    • The type of output. The default value is OutputType::POINTCLOUD.

  • modelToAppTransformfloat[16]

    • A transformation matrix that transforms from the model’s coordinate system to the application’s coordinate system.

  • frameStartFrameAtTime

    • The start frame of the model output. It transforms from the model’s coordinate system to the global coordinate system at frame start time.

  • frameEndFrameAtTime

    • The end frame of the model output. It transforms from the model’s coordinate system to the global coordinate system at frame end time. See below for more information.

  • auxTypeAuxType

    • The modality specific type of auxiliary data. The default value is AuxType::NONE. See below for more information.

  • paddinguint8_t[4]

    • Padding to align the structure to a multiple of 8 bytes.

  • elementsBasicElements

    • The basic elements of the model output. See below for more information.

  • auxiliaryDatavoid*

    • A pointer to the auxiliary data. This may not be filled. See below for more information.

BasicElements

Represents the basic elements of the model output.

  • timeOffsetNsint32_t*

    • Time offset from the start of the point cloud.

  • xfloat*

    • Azimuth in degree [-180,180] or cartesian x in m.

  • yfloat*

    • Elevation in degree or cartesian y in m.

  • zfloat*

    • Distance in m or cartesian z in m.

  • scalarfloat*

    • Sensor specific scalar.

  • flagsuint8_t*

    • Sensor specific flags representing ElementFlags enum values. For instance, for probing if the element is valid.

FrameAtTime

Represents a frame at a specific time.

  • timestampNsuint64_t

    • The timestamp of the frame in nanoseconds. The default value is 0UL.

  • orientationfloat4

    • The orientation of the frame.

  • posMfloat3

    • The position of the frame.

  • paddinguint8_t[4]

    • Padding to align the structure to a multiple of 8 bytes.

FrameOfReference

Represents the frame of reference for the model output.

  • SENSOR

  • PARENT

  • GLOBAL

  • SENSOR_MOTION_COMPENSATED

  • PARENT_MOTION_COMPENSATED

CoordsType

Represents the type of coordinates used in the model output.

  • CARTESIAN

  • SPHERICAL : x,y,z of BasicPoints contains: azimuth, elevation, distance

  • UNDEFINED : Pixels?

ElementFlags

Represents flags for elements.

  • FLAG_1

  • FLAG_2

  • FLAG_3

  • FLAG_4

  • FLAG_5

  • FLAG_5

  • FLAG_6

  • FLAG_7

  • VALID

OutputType

Represents the type of output.

  • POINTCLOUD

AuxType

Represents the type of auxiliary data.

  • NONE

  • LIDAR

  • RADAR

  • USS

  • IDS

AuxiliaryData

Every modality has its specific auxiliary data. See AuxType enum for a list of available auxiliary data types. Below are the structures for the supported modalities:

LidarAuxiliaryData

The LidarAuxiliaryData structure contains auxiliary data for a Lidar sensor.

  • uint32_t scanComplete

    • Whether the scan is complete.

  • float azimuthOffset

    • The offset to +x in radians for specific sensors.

  • LidarAuxHas filledAuxMembers

    • Which auxiliary data is filled.

  • uint32_t* emitterId

    • The emitter ID.

  • uint32_t* channelId

    • The channel ID.

  • uint8_t* echoId

    • The echo ID.

  • uint32_t* matId

    • The material ID.

  • uint32_t* objId

    • The object ID.

  • uint32_t* tickId

    • The tick ID.

  • uint8_t* tickStates

    • The tick states.

  • float* hitNormals

    • The hit normals.

  • float* velocities

    • The velocities.

The LidarAuxHas enum class is used to specify which auxiliary data is filled for a Lidar sensor.

  • NONE

    • No auxiliary data is filled.

  • EMITTER_ID

    • The emitter ID is filled.

  • CHANNEL_ID

    • The channel ID is filled.

  • ECHO_ID

    • The echo ID is filled.

  • MAT_ID

    • The material ID is filled.

  • OBJ_ID

    • The object ID is filled.

  • TICK_ID

    • The tick ID is filled.

  • TICK_STATES

    • The tick states are filled.

  • HIT_NORMALS

    • The hit normals are filled.

  • VELOCITIES

    • The velocities are filled.

USSAuxiliaryData

The USSAuxiliaryData structure contains auxiliary data for an USS sensor.

  • uint32_t numSgws

    • The number of SGWs.

  • uint32_t numSamplesPerSgw

    • The number of samples per SGW.

RadarAuxiliaryData

The RadarAuxiliaryData structure contains auxiliary data for a Radar sensor.

  • uint8_t sensorID

    • The ID of the sensor that generated the scan.

  • uint8_t scanIdx

    • The scan index for sensors with multi-scan support.

  • uint64_t timeStampNS

    • The scan timestamp in nanoseconds.

  • uint64_t cycleCnt

    • The scan cycle count (unique per scan index).

  • float maxRangeM

    • The maximum unambiguous range for the scan.

  • float minVelMps

    • The minimum unambiguous velocity for the scan.

  • float maxVelMps

    • The maximum unambiguous velocity for the scan.

  • float minAzRad

    • The minimum unambiguous azimuth for the scan.

  • float maxAzRad

    • The maximum unambiguous azimuth for the scan.

  • float minElRad

    • The minimum unambiguous elevation for the scan.

  • float maxElRad

    • The maximum unambiguous elevation for the scan.

  • uint32_t numDetections

    • The number of detections.

  • RadarAuxHas filledAuxMembers

    • Which auxiliary data is filled.

  • float* rv_ms

    • The radial velocity (m/s), always filled.

  • uint32_t* semId

    • The SEM ID, optional.

  • uint32_t* matId

    • The material ID, optional.

  • uint32_t* objId

    • The object ID, optional.

The RadarAuxHas enum class is used to specify which auxiliary data is filled for a Radar sensor.

  • NONE

    • No auxiliary data is filled.

  • SEM_ID

    • The semantic ID is filled.

  • MAT_ID

    • The material ID is filled.

  • OBJ_ID

    • The object ID is filled.

IDSAuxiliaryData

The IDSAuxiliaryData structure contains auxiliary data for an IDS (Idealized Depth Sensor) sensor.

  • uint32_t numRows

    • The number of rows in the sensor’s field of view.

  • uint32_t numCols

    • The number of columns in the sensor’s field of view.

  • float minColUnit

    • The minimum column unit of the specified field of view.

  • float maxColUnit

    • The maximum column unit of the specified field of view.

  • float minRowUnit

    • The minimum row unit of the specified field of view.

  • float maxRowUnit

    • The maximum row unit of the specified field of view.

  • float emitterCfgOriginX

    • The x-coordinate of the origin of the emitter configuration.

  • float emitterCfgOriginY

    • The y-coordinate of the origin of the emitter configuration.

  • float emitterCfgOriginZ

    • The z-coordinate of the origin of the emitter configuration.

  • int emitterGenType

    • The type of the emitter generation.

  • float elementSize

    • The maximum size of the spatial element to generate.

  • float radius

    • The radius at which the element size is defined.

  • float* originX

    • The x-coordinates of the origins.

  • float* originY

    • The y-coordinates of the origins.

  • float* originZ

    • The z-coordinates of the origins.

  • uint32_t* objectId

    • The object IDs.

Utility functions

Additionally, the GenericModelOutput provides utility functions, e.g, for building the structure out of a contiguous memory block, or for copying the structure. Below, is a list of the most important utility functions:

sizeInBytes

The sizeInBytes function is a part of the omni::sensors namespace. It calculates the total size of the GenericModelOutput structure and its elements.

Function Signature

NV_HOSTDEVICE
inline size_t sizeInBytes(const GenericModelOutput& output)

Parameters

  • outputconst GenericModelOutput&

    • The GenericModelOutput structure for which to calculate the size.

Returns

  • size_t

    • The total size of the GenericModelOutput structure and its elements.

Description

This function calculates the total size of the GenericModelOutput structure and its elements.

getModelOutputPtrFromBuffer

The getModelOutputPtrFromBuffer function is a part of the omni::sensors namespace. It retrieves a pointer to a GenericModelOutput structure from a buffer.

Function Signature

NV_HOSTDEVICE
inline GenericModelOutput* getModelOutputPtrFromBuffer(void* inData)

Parameters

  • inDatavoid*

    • The input data buffer from which to retrieve the GenericModelOutput structure.

Returns

  • GenericModelOutput*

    • A pointer to the GenericModelOutput structure retrieved from the buffer.

Description

This function first checks if the magic number of the GenericModelOutput structure is correct. If it is, the function then checks if the version of the GenericModelOutput structure is supported. If the version is supported, the function sets the basic elements and auxiliary data of the GenericModelOutput structure. If the magic number or version is not correct, the function prints an error message.

cpyGMOToBuffer

The cpyGMOToBuffer function is a part of the omni::sensors namespace. It copies a GenericModelOutput structure to a buffer.

Function Signature

inline void cpyGMOToBuffer(uint8_t* buffer,
                           const omni::sensors::GenericModelOutput* gpc,
                           const bool bufferOnHost = true, // not needed for cpu only
                           const bool pointerOnHost = true, // not needed for cpu only
                           const int32_t cudaDevice = -1, // not needed for cpu only
                           const cudaStream_t stream = 0) // not needed for cpu only

Parameters

  • bufferuint8_t*

    • The buffer to which to copy the GenericModelOutput structure.

  • gpcconst omni::sensors::GenericModelOutput*

    • The GenericModelOutput structure to copy.

  • bufferOnHostbool

    • Whether the buffer is on the host. Default is true.

  • pointerOnHostbool

    • Whether the pointer is on the host. Default is true.

  • cudaDeviceint32_t

    • The CUDA device to use. Default is -1.

  • streamcudaStream_t

    • The CUDA stream to use. Default is 0.

Description

This function first checks if the magic number of the GenericModelOutput structure is correct. If it is, the function then checks if the version of the GenericModelOutput structure is supported. If the version is supported, the function sets the copy kind based on whether the buffer and pointer are on the host or device. If the magic number or version is not correct, the function prints an error message. The function then copies the GenericModelOutput structure to the buffer using the appropriate copy kind.

Plugins

IGenericModelOutputIO

The IGenericModelOutputIO class is an interface for generic model output I/O operations. It writes and reas into a hdf5 file.

Methods

  • void init(const GMOIOConfig& cfg)

    • Initializes the IGenericModelOutputIO interface with the given configuration.

  • void writeModelOutput(const GenericModelOutput& modelOutput)

    • Writes the given model output.

  • GenericModelOutput readModelOutput(const omni::string& sensorName = "", const int frameId = -1)

    • Reads a model output. If a sensor name is given, it reads the model output for that sensor. If a frame ID is given, it reads the specific frame. If no arguments are given, it reads frames in order.

  • void addPacket(void* packet, const size_t packetSize)

    • Adds a packet with the given size.

Initialization

To get an object of the IGenericModelOutputIO interface, you need to call the carb::getFramework()->acquireInterface<omni::sensors::IGenericModelOutputIOFactory>()->createInstance() function. To initialize the IGenericModelOutputIO interface, you need to create a GMOIOConfig structure and call the init method.

GMOIOConfig

The GMOIOConfig structure is used to configure the IGenericModelOutputIO interface.

  • AccessType accessType

    • The access type for the IGenericModelOutputIO interface. Default is AccessType::RECORD_FULL.

  • bool onlyValid

    • Whether to only consider valid data. Default is false.

  • bool loop

    • Whether to loop the data. Default is false. Only needed for AccessType::READ.

  • uint32_t maxPoints

    • The maximum number of points. Default is 0. Only needed for AccessType::READ.

  • omni::string fileName

    • The name of the file. Default is an empty string.

  • omni::string sensorName

    • The name of the sensor. Default is an empty string.

IProfileReader

The IProfileReader reads the parameter of the modality specific profile from a JSON file. The interface is defined in (include/omni/sensors/IProfileReader.h).

Users can create objects that implement this interface by acquiring the IProfileReaderFactory (include/omni/sensors/IProfileReader.h) carbonite interface and using the createInstance() method.

Instantiation

To instantiate the IProfileReader the user needs to acquire the carbonite IProfileReaderFactory interface, and use the create instance method:

auto profileReader = carb::getFramework()->acquireInterface<omni::sensors::IProfileReaderFactory>()->createInstance();

Initialization

The user has to initialize the profile reader with the JSON string:

const auto json = profileReader->getProfileJsonAtPaths(profileName.c_str(), ProfileType::LIDAR);
profileReader->init(json.c_str(),ProfileType::LIDAR); // Currently, supprted: LIDAR, RADAR, USS, IDS (and variants)

Filling a Profile object

The user can get the data size object of the profile (for data dependent sizes, e.g, for LIDAR) and has to allocate the memory for the profile objectId before the profile reader can fill the profile object:

auto dataSize = profileReader->dataSizeProfile();
// can be done with std::vector or cudaMalloc etc, depending on the need of the user
void* profile{nullptr};
profile = malloc(dataSize);
profileReader->update(profile);
//

Pythong Bindings

The common extension provides a multitude of python bindings, e.g., for the GenericModelOutput structure, the IGenericModelOutputIO and the IProfileReader interface.