RTX Sensor Visual Materials

There are 21 sensor materials that are rendered in the visual spectrum, and more can not be added at this time. Their properties are stored in json files by the same name, located in the ./data/material_files/ folder.

Index

Sensor Material Type

0

Default

1

AsphaltStandard

2

AsphaltWeathered

3

VegetationGrass

4

WaterStandard

5

GlassStandard

6

FiberGlass

7

MetalAlloy

8

MetalAluminum

9

MetalAluminumOxidized

10

PlasticStandard

11

RetroMarkings

12

RetroSign

13

RubberStandard

14

SoilClay

15

ConcreteRough

16

ConcreteSmooth

17

OakTreeBark

18

FabricStandard

19

PlexiGlassStandard

20

MetalSilver

31

INVALID

How to use sensor materials

To use sensor materials, Isaac Sim must know how to map indices (seen in the Material ID return) to the sensor material type in the table above. This is done by setting the following carb settings in ./exts/omni.isaac.sensor/config/extension.toml [settings] section.

rtx.materialDb.rtSensorNameToIdMap="DefaultMaterial:0;AsphaltStandardMaterial:1;AsphaltWeatheredMaterial:2;VegetationGrassMaterial:3;WaterStandardMaterial:4;GlassStandardMaterial:5;FiberGlassMaterial:6;MetalAlloyMaterial:7;MetalAluminumMaterial:8;MetalAluminumOxidizedMaterial:9;PlasticStandardMaterial:10;RetroMarkingsMaterial:11;RetroSignMaterial:12;RubberStandardMaterial:13;SoilClayMaterial:14;ConcreteRoughMaterial:15;ConcreteSmoothMaterial:16;OakTreeBarkMaterial:17;FabricStandardMaterial:18;PlexiGlassStandardMaterial:19;MetalSilverMaterial:20"

With the rtx.materialDb.rtSensorNameToIdMap set, you must edit a csv file to map exact material name tokens to sensor material types. This csv file is located at ./kit/rendering-data/runtime/RtxSensorMaterialMap.csv.

The RtxSensorMaterialMap.csv file contains a material prim partial names to sensor material type pairs. The ones that come with Isaac Sim by default can be deleted as they may clash with names you wish to set. There is only one csv file. It controls the material mapping for all of the content. It is read at Isaac Sim startup and any changes to it require a restart to show up.

As an example, consider this scene:

../../_images/isaac_sim_sensors_rtx_material_map.png

The /Root/SM_floor29/SM_floor02/SM_floor02 prim, has a material prim assigned to it whose path is /Root/SM_floor29/Looks/MI_Floor_02b. If you want to add an entry to the csv file so that the SM_floor02 prim looks like rough concrete to the RTX sensors, you would add the entry:

mi_floor_02b,ConcreteRoughMaterial

Note that in the csv mapping file, the first token after the first appearance of /Looks/ in the material prim name attached to the mesh is used, and it must always be lowercase in the csv file, no matter what the case is on the stage. Also note how the word Material is concatenated onto the sensor material type from the table above.

RTX Sensor Non-Visual Materials

The RTX Sensor extensions now include support for rendering non-visual materials. To enable non-visual material rendering, you will need to run Isaac Sim from a terminal with additional settings.

From a Linux terminal:

$ ./isaac-sim.sh --/rtx/materialDb/nonVisualMaterialCSV/enabled=false

From a Windows command prompt:

> .\isaac-sim.bat --/rtx/materialDb/nonVisualMaterialCSV/enabled=false

You can also enable the setting when running Isaac Sim with Python using:

import carb
carb.settings.get_settings().set_bool("/rtx/materialDb/nonVisualMaterialCSV/enabled", False)

To run the scene, open Isaac Sim using two additional settings:

From a Linux terminal:

$ ./isaac-sim.sh \
--/rtx/materialDb/nonVisualMaterialCSV/enabled=false \
--/app/rtxSensorSample/contentMaterialTest/enabled=true \
--/rtx/rtxsensor/coordinateFrameQuaternion="0.5,-0.5,-0.5,-0.5"

From a Windows command prompt:

> .\isaac-sim.bat ^
--/rtx/materialDb/nonVisualMaterialCSV/enabled=false ^
--/app/rtxSensorSample/contentMaterialTest/enabled=true ^
--/rtx/rtxsensor/coordinateFrameQuaternion="0.5,-0.5,-0.5,-0.5"

Note

By default, /rtx/rtxsensor/coordinateFrameQuaternion is set to "1.0,0.0,0.0,0.0". This specific example was built using the legacy quaternion, hence the changed setting.

Use the content browser to navigate to Isaac/Samples/Examples/NonVisualMaterials/RtxSensorSample.usda, then open the scene. Select the LegacyRtxSensorSample camera view. You should see a view like the following:

Non-visual materials rendered by legacy RTX sensor.

Now select the RtxSensorSample camera view. You should see a view like the following:

Non-visual materials rendered by current RTX sensor.

Note the change in colors and reflections due to non-visual material attributes defined in the .usda file. An example portion of the file is provided below:

def Material "GreenAlbedo"
{
    # ... additional attributes
    custom string inputs:nonvisual:attributes = "emissive"
    custom string inputs:nonvisual:coating = "none"
    custom string inputs:nonvisual:coating = "rubber"
    # ... additional attributes
}

The full set of non-visual material attributes is described in the Omniverse Non-Visual Materials Extension documentation. Attributes may be added to materials from the UI by right-clicking the material in the Stage window, then selecting Add > Attribute. This will open a new window like the one below, enabling you to specify custom non-visual attributes.

Adding a non-visual material attribute.

After adding the new attribute, it will appear in the material’s properties, at which point it can be populated:

Populating a new non-visual material attribute.

Debugging

The carb parameter:

[settings]
rtx.materialDb.rtSensorMaterialLogs=true

can help. If set to true it will output a list of all the materials in the scene that are NOT mapped to a sensor material. This list outputs to the terminal and the log at Isaac Sim startup.