Nucleus on Google Cloud

../../_images/cloud_gcp_header.jpg

Introduction

This guide focuses on installing and configuring an Omniverse Enterprise Nucleus Server on Google Cloud. The requirements for a successful deployment are outlined along with the technical steps below.

Depending on requirements and/or internal policies, there may be some settings and/or options that are set differently than what is covered within this documentation. This document focuses on using the Cloud Shell launched from within the Google Cloud console, however standing up individual Virtual Machines manually using the web-based interface is also supported.

Solution Overview

The following steps outline a reference solution that implements the primary components of a Nucleus deployment. To handle communication from end users, a Global Cloud Load Balancer is configured to direct and manage traffic to the Nucleus server. The Nucleus server runs on a Google Cloud instance with no external IP Address and only accepts traffic from the load balancer. To communicate with the NVIDIA Container Registry, a Cloud NAT is configured so the Nucleus server can access the Internet.

../../_images/gcp_network_diagram.jpg

Attention

Steps within this document require network configuration changes to allow remote connectivity. Please consult your IT/Information Security team for proper advisement.

Prerequisites

  • A Google Cloud account to deploy the necessary Virtual Machines to run an Enterprise Nucleus Server.

  • This tutorial uses gcloud and gsutil commands, which can run from a Cloud Shell instance launched from the Console. If it’s preferred to use gcloud and gsutil from a local workstation, install the Google Cloud CLI. The documentation explains how to run commands using the in-browser Cloud Shell; if it’s preferred to use the gcloud CLI from a workstation, adjust the instructions accordingly.

  • In the Google Cloud console, on the project selector page, select an existing or create a Google Cloud project.

  • Ensure a billing account is enabled for your new Google Cloud project. Click here to learn how to check if billing is enabled for your project.

  • Enable the Compute Engine API.

  • A Fully Qualified Domain Name (FQDN) that you plan to use for your Nucleus environment (e.g., nucleus.mydomain.com).

  • The ability to perform DNS changes (Creation of an A record.)

Attention

When copying the BASH and YAML code from this document to your workstation, use a text editor such as Notepad++ (Windows) or BBEdit (macOS) that supports Unix EOL (End of Line) conversion so the file is saved correctly. Some text editors add additional non-visible line breaks that may cause parsing issues.

Variable Legend

The following variables are used throughout this document:

Variable
Description
SERVER_NAME
The name for the Nucleus server
GROUP_NAME
The name for the instance group
ARTIFACT_NAME
The file name of the Nucleus artifact
ZONE
The desired zone for the server
SIZE
The size of the boot disk in GB
PROJECT
The name of the project
ADDRESS_NAME
The name for the IP Address resource, not the IP Address itself
CERTIFICATE_NAME
The name of the certificate
DOMAIN
The chosen FQDN for the Nucleus server
REGION
The zone where you created the Nucleus server
SOURCE
The path to the YAML file
PROJECT_ID
The Project ID (not the Project Name) of your project

Note

It’s very helpful to make a list of the values for the variables above in a text file on your workstation as they will be used often within this documentation.

Provisioning your Environment

Preparing your Cloud Shell environment

Using Cloud Shell run the following commands to prepare your environment.

  1. Set your project:

gcloud config set project [PROJECT_ID]

In the Console, you can locate your Project ID in the IAM & Admin Settings.

  1. Ensure you have a default VPC network. If your project does not already have a default VPC network, create one:

gcloud compute networks create default \
  --subnet-mode=auto

Creating the Nucleus server instance

The Nucleus server instance is created from a base Ubuntu Linux 22.04 LTS image. This example uses an e2-standard-8 (8 vCPU’s, 32GB RAM) machine type. Choose the sizing that best meets your performance requirements.

Using Cloud Shell, create the instance:

gcloud compute instances create [SERVER_NAME] \
  --zone=[ZONE] \
  --machine-type=e2-standard-8 \
  --network-interface=subnet=default,no-address \
  --tags=nucleus-server,allow-iap \
  --maintenance-policy=MIGRATE \
  --provisioning-model=STANDARD \
  --scopes=https://www.googleapis.com/auth/cloud-platform \
  --create-disk=\
auto-delete=yes,\
boot=yes,\
image-project=ubuntu-os-cloud,\
image-family=ubuntu-2204-lts,\
mode=rw,\
size=[SIZE],\
type=projects/[PROJECT]/zones/[ZONE]/diskTypes/pd-ssd \
  --shielded-secure-boot \
  --shielded-vtpm \
  --shielded-integrity-monitoring \
  --reservation-affinity=any

Note

When choosing the size of the instance, consider both the current and expected future growth.

Creating an unmanaged instance group

In order for the Nucleus server to serve as the backend to a load balancer, it must be part of an instance group. The following details creating of the instance group and adding the Nucleus server to it.

  1. Using Cloud Shell, create the instance group:

    gcloud compute instance-groups unmanaged create [GROUP_NAME] --zone=[ZONE]
    
  2. Add port mapping to the instance group to facilitate Nucleus server traffic:

    gcloud compute instance-groups unmanaged set-named-ports \
    [GROUP_NAME] \
      --zone=[ZONE] \
      --named-ports=\
    api:3019,\
    auth:3100,\
    auth-login:3180,\
    discovery:3333,\
    lft:3030,\
    search:3400,\
    tagging:3020,\
    web:80
    
  3. Add the instance to the instance group:

    gcloud compute instance-groups unmanaged add-instances [GROUP_NAME] \
      --instances=[SERVER_NAME] --zone=[ZONE]
    

Creating a static IP Address

Create a static external IP Address for the load balancer. This IP Address will also be used to map the server to your domain.

  1. Using Cloud Shell, create a static external IP Address:

    gcloud compute addresses create [ADDRESS_NAME] \
      --global \
      --ip-version IPV4 \
      --network-tier=PREMIUM
    

Take note of the resulting address in the Console for the next step.

  1. Using a DNS service such as domains.google.com, map the reserved external IP Address you created to a fully qualified domain name (FQDN) using an A record. (e.g., nucleus.mydomain.com)

Creating a certificate

Create a certificate to provide a secure TLS/SSL connection to the Nucleus server.

  1. Using Cloud Shell, create the certificate:

    gcloud beta compute ssl-certificates create [CERTIFICATE_NAME] \
      --global \
      --domains=[DOMAIN]
    

Creating a firewall rules

In this example, you create the nucleus-ports firewall rule. This is an ingress rule that allows traffic from the Google Cloud load balancer to your Nucleus server over specified ports. Google Cloud load balancers always use IP ranges, 130.211.0.0/22 and 35.191.0.0/16, so there is no need to change these values. This example uses the target tag nucleus-server to identify the VM.

  1. Using Cloud Shell, create a firewall rule:

    gcloud compute firewall-rules create nucleus-ports \
      --direction=INGRESS \
      --priority=1000 \
      --network=default \
      --action=ALLOW \
      --rules=\
    tcp:80,tcp:3009,tcp:3010,tcp:3019,tcp:3020,\
    tcp:3030,tcp:3100,tcp:3180,tcp:3333,tcp:3400 \
      --source-ranges=130.211.0.0/22,35.191.0.0/16 \
      --target-tags=nucleus-server
    
  2. Create a health check and specify TCP Port 3333 to use for it:

    gcloud compute health-checks create http http-basic-check \
      --request-path="/healthcheck" \
      --port 3333
    
  3. As the Nucleus server has no external IP Address, connect to it using Identity-Aware Proxy (IAP). Create a firewall rule to allow IAP traffic to your Nucleus server:

    gcloud compute firewall-rules create allow-ssh-from-iap \
    --source-ranges 35.235.240.0/20 \
    --target-tags allow-iap \
    --allow tcp:22
    

Creating a Cloud NAT

Your Nucleus server will not have an external IP Address, so cannot reach the Internet without a Cloud NAT and Cloud Router.

  1. Create a Cloud Router:

    gcloud compute routers create nat-router \
      --network default \
      --region [REGION]
    
  2. Add a configuration to the router:

    gcloud compute routers nats create nat-config \
      --router-region [REGION] \
      --router nat-router \
      --nat-all-subnet-ip-ranges \
      --auto-allocate-nat-external-ips
    

Creating the HTTPS load balancer

You create an HTTPS Load Balancer to serve as a front end for the Nucleus server. An HTTPS Load Balancer provides a single endpoint to connect to the Nucleus server, while at the same time providing added security and traffic management.

Creating the backend services

Omniverse Nucleus requires different traffic types to be routed in different ways. There are eight different backend services to create.

Copy this BASH script to a text file on your local workstation, edit it using a text editor of your choice, and ensure that the file ends with the extension of .sh. Then upload the file into your Cloud Shell environment.

#! /bin/bash

GROUP_NAME=[GROUP_NAME]
ZONE=[ZONE]
BACKENDS="api auth auth-login discovery lft search tagging web"

# Iterate over all backends.
for BACKEND in $BACKENDS; do

  # Create backend service.
  gcloud compute backend-services create nucleus-$BACKEND \
    --load-balancing-scheme=EXTERNAL_MANAGED \
    --protocol=HTTP \
    --port-name=$BACKEND \
    --health-checks=http-basic-check \
    --enable-cdn \
    --enable-logging \
    --timeout=30 \
    --global

  # Add backend service to instance group.
  gcloud compute backend-services add-backend nucleus-$BACKEND \
    --instance-group=$GROUP_NAME \
    --instance-group-zone=$ZONE \
    --balancing-mode "UTILIZATION" \
    --capacity-scaler=1 \
    --max-utilization=1 \
    --global

done

Wait until all backend services are created before proceeding to the next step.

Creating the URL map

This step creates what is referred to as the load balancer resource itself. Using Cloud Shell, create a URL map to route the incoming requests to the default backend service:

gcloud compute url-maps create nucleus-lb \
  --default-service nucleus-web

Creating the HTTPS proxy

The proxy is the portion of the load balancer that holds the TLS/SSL certificate for an HTTPS load balancer. You will also load your certificate in this step.

  1. Using Cloud Shell, create a target HTTPS proxy to route requests to the load balancer:

    gcloud compute target-https-proxies create nucleus-https-lb-proxy \
      --url-map=nucleus-lb \
      --ssl-certificates=[CERTIFICATE_NAME]
    
  2. Create a global forwarding rule to route incoming requests to the proxy:

    gcloud compute forwarding-rules create nucleus-https-443 \
      --load-balancing-scheme=EXTERNAL_MANAGED \
      --network-tier=PREMIUM \
      --address=[ADDRESS_NAME] \
      --global \
      --target-https-proxy=nucleus-https-lb-proxy \
      --ports=443
    
  3. Create a global TLS/SSL policy and attach it to the HTTPS proxy:

    gcloud compute ssl-policies create nucleus-ssl-policy \
      --profile MODERN \
      --min-tls-version 1.0
    
  4. Attach the TLS/SSL policy to the global target HTTPS proxy:

    gcloud compute target-https-proxies update nucleus-https-lb-proxy \
      --ssl-policy=nucleus-ssl-policy
    

Importing the URL map configuration file

  1. In Google Cloud, a URL map routes incoming web requests to backend services. You configure the mapping using YAML syntax. Copy this YAML code to a text file on your local workstation, edit it using the editor of your choice, and ensure that the file ends with the extension of .yaml. Then upload the file into your Cloud Shell environment.

Replace PROJECT with the Project ID of your Google Cloud project and DOMAIN with the FQDN of your chosen domain (e.g., nucleus.mydomain.com).

defaultUrlRedirect:
  hostRedirect: [DOMAIN]
  httpsRedirect: true
  prefixRedirect: /omni/web3/
  redirectResponseCode: FOUND
hostRules:
- hosts:
  - [DOMAIN]
  pathMatcher: path-matcher
kind: compute#urlMap
name: nucleus-lb
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-web
  name: path-matcher
  pathRules:
  - paths:
    - /
    urlRedirect:
      httpsRedirect: false
      pathRedirect: /omni/web3/
      stripQuery: false
  - paths:
    - /omni/web3/*
    routeAction:
      urlRewrite:
        pathPrefixRewrite: /
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-web
  - paths:
    - /omni/api
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-api
  - paths:
    - /omni/auth
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-auth
  - paths:
    - /omni/auth/login/*
    routeAction:
      urlRewrite:
        pathPrefixRewrite: /
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-auth-login
  - paths:
    - /omni/discovery/healthcheck
    routeAction:
      urlRewrite:
        pathPrefixRewrite: /healthcheck
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-discovery
  - paths:
    - /omni/discovery/*
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-discovery
  - paths:
    - /omni/discovery
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-discovery
  - paths:
    - /omni/lft/*
    routeAction:
      urlRewrite:
        pathPrefixRewrite: /
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-lft
  - paths:
    - /omni/search3
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-search
  - paths:
    - /omni/tagging3
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-tagging
  - paths:
    - /omniverse://*
    routeAction:
      urlRewrite:
        pathPrefixRewrite: /
    service: https://www.googleapis.com/compute/v1/projects/[PROJECT]/global/backendServices/nucleus-web
  1. Import the URL map configuration file to the load balancer:

    gcloud compute url-maps import nucleus-lb \
      --source=[SOURCE] \
      --global
    

Creating the HTTP load balancer

  1. In Google Cloud, you create a second HTTP load balancer to forward requests to the HTTPS load balancer. Copy this YAML code to a text file on your local workstation, edit it using the editor of your choice, and ensure that the file ends with the extension of .yaml. Then upload the file into your Cloud Shell environment.

    kind: compute#urlMap
    name: nucleus-lb-redirect
    defaultUrlRedirect:
      redirectResponseCode: MOVED_PERMANENTLY_DEFAULT
      httpsRedirect: True
    
  2. Using Cloud Shell, import the URL map configuration file to the load balancer:

    gcloud compute url-maps import nucleus-lb-redirect \
      --source=[SOURCE] \
      --global
    
  3. Create a target HTTP proxy to route requests to the load balancer, using the URL map you created:

    gcloud compute target-http-proxies create nucleus-http-proxy \
      --url-map=nucleus-lb-redirect
    
  4. Create a global forwarding rule to route incoming requests to the proxy:

    gcloud compute forwarding-rules create nucleus-http-80 \
      --load-balancing-scheme=EXTERNAL_MANAGED \
      --network-tier=PREMIUM \
      --address=[ADDRESS_NAME] \
      --global \
      --target-http-proxy=nucleus-http-proxy \
      --ports=80
    

Installing Nucleus

  1. Locate the Enterprise Nucleus Server artifact on your local workstation and upload the file into your Cloud Shell environment.

  2. Using Cloud Shell, copy the artifact to your instance using the following command:

    gcloud compute scp [ARTIFACT_NAME] [SERVER_NAME]:/tmp --zone=[ZONE]
    
  3. Log into your instance using SSH using the following command:

    gcloud compute ssh [SERVER_NAME] --zone=[ZONE]
    
  4. See the Additional Reading and Reference Links at the bottom of this document for a walk-through of installing and configuring an Enterprise Nucleus Server.

Troubleshooting

When initially creating the instance, the following error may appear:

../../_images/cloud_gcp_setting_project_id.jpg

If this error appears, locate the Project ID within the GCP project selector. (Use the Project ID not the name.) Then use the following command to set the project:

gcloud config set project [PROJECT_ID]