Semantic Labeling

Semantic labeling refers to embedded metadata that describes the properties of an asset. When applied, it allows a simulation to understand what it is “seeing” within a given stage.

But in order for semantic labels to be useful, it is important to pick a taxonomy before you begin and use that naming process to define all of the assets in your datasets. Without this consistency in naming, it will cause problems for configuring the simulation environment to get accurate results. One of the biggest challenges for semantic labeling is that it needs to be considered and defined up front, and no two groups or users will necessarily refer to the same object in the same manner. Should an object be labeled as a “car”, “automobile”, “sedan”, “coupe”, or “vehicle”?

Technically, it can be any or multiple of these and have a similar meaning. But no one interpretation is a singularly correct answer. As such, it makes little sense to try and force one way of labeling as part of the broad SimReady specification. Each of you will have your own ways of semantically labeling your 3D assets, and it’s important to respect those efforts.

We’ve opted to encourage the use of a system that abstracts the semantic label away from any particular language: by leveraging the open-source database WikiData (https://wikidata.org). This database has more than 100 million objects, brands, geographic locations, and other things that can be searched for via their website and through API. Each item has a unique, alpha-numeric identifier (called a Q-Code) that defines each type of object within its feature graph.

WikiData also provides a list of synonyms and subtrees within its taxonomy, as well as translations of Q-Code items into a number of languages. All of this information can be used (and stored) within a SimReady asset if desired. Also, because the Wikidata database is open-source, users can add new entries if an object type or class is missing and add to this very strong classification system with a new and unique Q-Code identifier.

SimReady Taxonomy for Semantic Labeling

Once semantic labeling is applied, output imagery generated from SimReady 3D assets can be queried for specific objects. These queries can be viewed in different forms as annotated outlines to indicate where in a given frame the object in question exists. The SimReady specification captures and stores the following baseline metadata for its semantic labeling schema:

Image Annotations derived from semantic labels (2D bounding box / instance segmentation / semantic segmentation / depth)

Label Name	Description
class	This label is a human-readable, baseline classification. For example, this object is a “cup”, or a ”truck”.
hierarchy	The hierarchy semantic label defines an ordered syntax structure that allows for typical relational navigation of the asset in search. For example, a chosen hierarchy might be represented as “machine/vehicle/emergency_vehicle/fire_engine”. The WikiData tree structure provides this information, but you are welcome to define your own hierarchies.
qcode	This is the alpha-numeric identifier pulled from WikiData. Given the number of legitimate uses of words within many languages, choosing the correct Q-Code identifier is left up to the user.

Please see the SimReady specification document to learn more about how the semantic labels are generated and stored within a SimReady asset.