Integrate Vector Log Processor with Kit Container#

Configure Vector with a Custom Entrypoint Script#

Create a entrypoint_vector_dev.sh in your working directory:

touch entrypoint_vector_dev.sh

Script Components Overview#

The custom entrypoint script orchestrates Vector configuration and application startup. It handles multiple critical functions:

Environment Setup:
- User environment configuration for Kit application
- Debug logging for troubleshooting
- Environment variable validation
Vector OTel Processing Control:
- Checks VECTOR_OTEL_ACTIVE flag to enable/disable Vector processing
- Falls back to standard Kit execution if Vector is disabled
Configuration Management:
- Supports both custom configuration VECTOR_CONF_B64 and static configuration
- Decodes base64-encoded Vector configuration
- Replaces OTel endpoint placeholders with actual values
- Creates log file directories /tmp/kit_structured_logs.log
Validation & Testing:
- Vector configuration syntax validation
- OTel endpoint connectivity testing
- Network connectivity verification to env var OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
Process Orchestration:
- Starts Vector log processor in background
- Launches Kit application with log capture
- Pipes Kit output to log files for Vector processing

Add the Script Content#

Copy the following in the entrypoint script:

nano entrypoint_vector_dev.sh

#!/bin/bash
echo "[entrypoint_vector_dev.sh] Starting container..."
# Set required environment variables for Kit
export USER="ubuntu"
export LOGNAME="ubuntu"
# Check if Vector OTEL processing is enabled
if [ "$VECTOR_OTEL_ACTIVE" = "TRUE" ]; then
    echo "[Vector] Vector OTEL processing is ENABLED (VECTOR_OTEL_ACTIVE=TRUE)"
    # Create log files if they don't exist
    echo "[Vector] Setting up log files..."
    touch /tmp/kit_structured_logs.log
    chmod 666 /tmp/kit_structured_logs.log
    # Validate OTEL endpoint
    if [ -z "$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT" ]; then
        echo "[Vector] ERROR: OTEL_EXPORTER_OTLP_LOGS_ENDPOINT is not set!"
        exit 1
    fi
    if [[ ! "$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT" =~ ^https?:// ]]; then
        echo "[Vector] ERROR: Invalid OTEL endpoint format. Must start with http:// or https://"
        exit 1
    fi
    echo "[Vector] Using OTEL endpoint: $OTEL_EXPORTER_OTLP_LOGS_ENDPOINT"
    # Determine which Vector configuration to use
    if [ ! -z "$VECTOR_CONF_B64" ]; then
        echo "[Vector] Custom Vector configuration provided via VECTOR_CONF_B64"
        echo "[Vector] Decoding and using customer-provided configuration..."
        # Decode Vector config
        echo "$VECTOR_CONF_B64" | base64 -d > /tmp/vector_raw.toml
        # Replace OTEL endpoint
        sed "s|PLACEHOLDER_OTEL_ENDPOINT|$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT|g" /tmp/vector_raw.toml > /tmp/vector.toml
        echo "[Vector] Using CUSTOM Vector configuration (from VECTOR_CONF_B64)"
    else
        echo "[Vector] No custom configuration provided. Using static/default Vector configuration..."
        # Copy static configuration and replace OTEL endpoint
        cp /opt/vector/static_config.toml /tmp/vector_raw.toml
        sed "s|PLACEHOLDER_OTEL_ENDPOINT|$OTEL_EXPORTER_OTLP_LOGS_ENDPOINT|g" /tmp/vector_raw.toml > /tmp/vector.toml
        echo "[Vector] Using STATIC Vector configuration (from /opt/vector/static_config.toml)"
    fi
    # Show the first few lines of the config for debug
    echo "[Vector] First 10 lines of /tmp/vector.toml:" && head -n 10 /tmp/vector.toml
    # Validate Vector config
    echo "[Vector] Verifying Vector configuration..."
    if [ -x "/opt/vector/bin/vector" ]; then
        /opt/vector/bin/vector validate /tmp/vector.toml
        if [ $? -ne 0 ]; then
            echo "[Vector] ERROR: Vector configuration validation failed!"
            exit 1
        fi
    fi
    echo "[Vector] Starting Kit with real-time log forwarding..."
    # Start Vector reading from the log file
    /opt/vector/bin/vector --config /tmp/vector.toml &
    VECTOR_PID=$!
    # Start Kit and pipe its output to the log file
    stdbuf -oL /entrypoint.sh 2>&1 | stdbuf -oL tee -a /tmp/kit_structured_logs.log &
    PIPELINE_PID=$!
    # Wait for the pipeline (which includes Kit) to complete
    wait $PIPELINE_PID
    echo "[Vector] Kit pipeline has ended"
    echo "[Vector] Cleaning up Vector process..."
    # Kill Vector
    kill -TERM $VECTOR_PID 2>/dev/null || true
    echo "[Vector] Container exiting"
    exit 0
else
    echo "[Vector] Vector OTEL processing is DISABLED (VECTOR_OTEL_ACTIVE=FALSE or not set)"
    echo "[Vector] Running Kit without log processing."
    exec /entrypoint.sh
fi

Vector OTel Transform#

The Vector OTel Transform is responsible for converting the Kit application’s plaintext logs into OpenTelemetry (OTel) format required by NVCF. This transformation ensures your logs are properly structured and compliant with OTel standards.

Vector follows the Vector Remap Language (VRL) to interact with the log processing. You are encouraged to extend Vector’s capability as required.

Transformation Process#

Input: Kit application plaintext logs (stdout/stderr)
Processing: Vector Remap Language (VRL) transformation rules
Output: OpenTelemetry-compliant JSON logs

For the purpose of this document, we are using verified VRL syntax to work appropriately with the log processing pipeline.

The vector.toml file being created as part of this step, will serve as the static configuration file. This step is provided as a reference to how the VRL works. This is a required step as part of the configuration.

To pass a custom vector configuration, use the VECTOR_CONF_B64 environment variable.

Create the Vector Configuration File#

touch vector.toml
nano vector.toml

Copy the Vector Configuration#

[sources.kit_logs]
 type = "file"
 include = ["/tmp/kit_structured_logs.log"]
 read_from = "beginning"
 max_line_bytes = 1024
 ignore_older_secs = 86400  # Ignore logs older than 24 hours
 remove_after_secs = 604800  # Remove processed files after 7 days

 [transforms.otel_transforms]
 type = "remap"
 inputs = ["kit_logs"]
 source = '''

 # Extract log level from message
     level = "INFO"
     message = to_string(.message) ?? ""
     if message != "" {
         if match(message, r'\[Error\]') {
             level = "ERROR"
     } else if match(message, r'\[Warning\]') {
         level = "WARN"
     } else if match(message, r'\[Debug\]') {
         level = "DEBUG"
     }
 }

 # Set severity number based on level
 severity_number = 9
 if level == "ERROR" {
     severity_number = 17
 } else if level == "WARN" {
     severity_number = 13
 } else if level == "DEBUG" {
     severity_number = 5
 }
 .resourceLogs = [{
     "resource": {"attributes": [{
         "key": "service.name",
         "value": {"stringValue": "kit-vector-app"}
     }]
     },
     "scopeLogs": [{
         "scope": {
         "name": "kitvector.log",
         "version": "1.0.0"
         },
         "logRecords": [{
             "timeUnixNano": to_string(to_unix_timestamp(now(), unit:
 "nanoseconds")),
             "body": {"stringValue": .message},
             "severityText": level,
             "severityNumber": severity_number
         }]
     }]
 }]
 '''

 [sinks.otel_collector]
 type = "http"
 inputs = ["otel_transforms"]
 uri = "${OTEL_EXPORTER_OTLP_LOGS_ENDPOINT}"
 encoding.codec = "json"
 method = "post"
 request.headers.Content-Type = "application/json"
 framing.method = "newline_delimited"
 batch.max_events = 1
 batch.timeout_secs = 1
 request.retry_attempts = 3
 request.retry_initial_backoff_secs = 1
 request.retry_max_duration_secs = 10

 [sinks.console_debug]
 type = "console"
 inputs = ["otel_transforms"]
 encoding.codec = "json"
 encoding.timestamp_format = "rfc3339"

This configuration allows kit logs to be read from thestdout/stderr redirected log file /tmp/kit_structured_logs.log.

The OTel transformed logs are directed to the NVCF’s OTEL_EXPORTER_OTLP_LOGS_ENDPOINT & to the console. As a result, you can observe the kit application logs on the NVCF UI and your observability backend.