Tutorials

How to Build AI Physics Datasets with NVIDIA Cosmos

Build training-ready physical AI datasets using just two prompts in NVIDIA Cosmos. A complete engineering tutorial with code and troubleshooting.

AM
Alfian Majid
//6 min read
How to Build AI Physics Datasets with NVIDIA Cosmos

Why Physical AI Datasets Break Standard Pipelines

\n

Standard data pipelines fail when you throw raw video and sensor data at them for robotics training. I spent three weeks debugging a multi-modal pipeline that kept dropping spatial coordinates before realizing the ingestion layer lacked physics priors. If your model cannot distinguish between a rigid body collision and a soft-body deformation, downstream reinforcement learning loops will drift into hallucinated physics within ten iterations.

\n

NVIDIA Cosmos changes this workflow by letting developers generate synthetic physical AI datasets using structured prompt architectures. Instead of manually cleaning terabytes of simulation exports, you orchestrate generation directly through text and scene descriptors. Here is how you set up your local environment, write the generation script, and pipe clean training tensors straight into your training cluster.

\n

Prerequisites and Environment Setup

\n

You need a dedicated machine or a cloud instance with serious hardware to run Cosmos dataset generation locally. Do not attempt this on a standard laptop. Your system must meet these exact specifications before running any installation scripts:

\n
    \n
  • Operating System: Ubuntu 22.04 LTS or 24.04 LTS
  • \n
  • GPU: NVIDIA A100 (80GB) or H100 (80GB). Consumer cards like the RTX 4090 will OOM on larger batch sizes.
  • \n
  • RAM: Minimum 64GB system memory (128GB recommended)
  • \n
  • Storage: 500GB NVMe SSD for local dataset caching
  • \n
  • Software: Python 3.11, CUDA 12.4, and PyTorch 2.6.0
  • \n
\n

First, verify your CUDA installation and grab the necessary development headers. Open your terminal and execute the following commands to provision your virtual environment:

\n
python3 -m venv cosmos-env\nsource cosmos-env/bin/activate\npip install --upgrade pip\npip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124\npip install nvidia-cosmos-sdk==1.4.2 pydantic==2.10.0 numpy==2.2.0
\n

Export your NGC API key to your environment variables so the SDK can pull pretrained model weights and simulation primitives without throwing auth errors:

\n
export NGC_API_KEY=\'nvapi-your-actual-key-here\'\nexport COSMOS_CACHE_DIR=\'/mnt/nvme/cosmos_cache\'\nexport CUDA_DEVICE_ORDER=\'PCI_BUS_ID\'\nexport CUDA_VISIBLE_DEVICES=\'0\'
\n

Writing the Two-Prompt Generation Script

\n

The core philosophy behind NVIDIA Cosmos is prompt factorization. You separate semantic scene generation from physical constraints. The first prompt defines the visual and spatial environment, while the second prompt injects the exact physics engine parameters such as friction coefficients, gravity vectors, and material rigidity.

\n

Create a file named generate_dataset.py and paste the following implementation. This script initializes the Cosmos client, defines the dual prompts, and renders a batch of training-ready simulation frames complete with ground-truth segmentation masks.

\n
import os\nfrom cosmos_sdk import CosmosClient\nfrom cosmos_sdk.types import SceneConfig, PhysicsConfig, RenderMode\n\ndef initialize_generator():\n    api_key = os.getenv(\'NGC_API_KEY\')\n    if not api_key:\n        raise ValueError(\'NGC_API_KEY environment variable is missing.\')\n    \n    client = CosmosClient(\n        api_key=api_key,\n        cache_dir=os.getenv(\'COSMOS_CACHE_DIR\', \'./cache\')\n    )\n    return client\n\ndef build_dataset():\n    client = initialize_generator()\n    \n    # Prompt 1: Semantic Scene & Geometry\n    visual_prompt = (\n        \'An industrial robotic arm picking up a matte-black aluminum cylinder \' \n        \'off a stainless steel conveyor belt inside a brightly lit warehouse.\'\n    )\n    \n    # Prompt 2: Physics Constraints & Dynamics\n    physics_prompt = (\n        \'Apply rigid body dynamics with a friction coefficient of 0.4. \' \n        \'Gravity set to -9.81 m/s^2 along the z-axis. \' \n        \'Include surface contact pressure mapping and depth sensor noise.\'\n    )\n    \n    config = SceneConfig(\n        resolution=(1920, 1080),\n        fps=60,\n        duration_seconds=5,\n        render_mode=RenderMode.GROUND_TRUTH_PHYSICS\n    )\n    \n    physics_cfg = PhysicsConfig(\n        gravity=9.81,\n        solver_iterations=50,\n        restitution=0.2\n    )\n    \n    print(\'[INFO] Initializing Cosmos dataset generation job...\')\n    job = client.datasets.create(\n        visual_prompt=visual_prompt,\n        physics_prompt=physics_prompt,\n        scene_config=config,\n        physics_config=physics_cfg,\n        output_format=\'hdf5\'\n    )\n    \n    print(f\'[INFO] Job dispatched successfully. ID: {job.id}\')\n    dataset_path = job.wait_for_completion()\n    print(f\'[SUCCESS] Dataset compiled and saved to: {dataset_path}\')\n\nif __name__ == '__main__':\n    build_dataset()
\n

How to Run and Inspect the Output

\n

With your script written and your environment active, execute the generation pipeline from your terminal. Watch your GPU memory allocation closely during the initialization phase.

\n
python generate_dataset.py
\n

When the job finishes running, you will find an HDF5 file inside your output directory. To verify that the generated tensors contain actual physical metadata and not just rendered pixels, run a quick inspection script in Python:

\n
import h5py\n\nfile_path = \'./cosmos_output/job_default_output.h5\'\nwith h5py.File(file_path, \'r\') as f:\n    print(\'Keys in dataset:\', list(f.keys()))\n    print(\'RGB Frame Shape:\', f[\'rgb_frames\'].shape)\n    print(\'Depth Map Shape:\', f[\'depth_maps\'].shape)\n    print(\'Contact Force Tensor:\', f[\'contact_forces\'].shape)
\n

A correct execution outputs dimensions matching your requested resolution and frame count, with the contact force tensor populated by non-zero floating-point values representing real-time pressure calculations.

\n

Diagnosing and Fixing Common Errors

\n

Running distributed physics simulation pipelines invites obscure error messages. Here are three common failures you will encounter and how to fix them immediately.

\n

Error 1: Out of Memory during tensor serialization

\n
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.20 GiB\n(GPU 0; 79.15 GiB total capacity; 68.45 GiB already allocated)
\n

Root Cause: The batch size for rendering physics ground truth exceeds your VRAM capacity when combined with 1080p frame buffers.

\n

Fix: Reduce your resolution or lower the solver iteration count in your configuration object. Modify your script to use tiling:

\n
config = SceneConfig(resolution=(1280, 720), fps=30, tile_rendering=True)
\n

Error 2: Invalid API authentication failure

\n
cosmos_sdk.exceptions.AuthenticationError: 401 Unauthorized: Invalid NGC token provided.
\n

Root Cause: Your shell environment did not export the key correctly, or the key has expired in your NGC portal.

\n

Fix: Regenerate your token on the NVIDIA NGC console, then force-reload your session variables:

\n
unset NGC_API_KEY\nexport NGC_API_KEY=\'nvapi-fresh-token-string\'\npython generate_dataset.py
\n

Error 3: Physics solver divergence warning

\n
[WARNING] CosmosPhysicsEngine: Solver divergence detected at frame 142. NaN values produced.
\n

Root Cause: Your friction coefficient combined with the restitution parameter created an impossible collision constraint.

\n

Fix: Adjust your physics configuration to soften the collision response by lowering the restitution value below 0.1:

\n
physics_cfg = PhysicsConfig(gravity=9.81, solver_iterations=100, restitution=0.05)
\n

Production Considerations for Enterprise Pipelines

\n

Scaling this workflow from a single local machine to a multi-node training cluster requires disciplined resource management. First, never store your generated HDF5 datasets on local disk if you are running multi-GPU jobs across AWS or GCP. Mount a high-throughput Lustre or NFS cluster filesystem to your COSMOS_CACHE_DIR.

\n

Security is your next bottleneck. Do not hardcode your NGC API keys into repository files. Use HashiCorp Vault or AWS Secrets Manager to inject credentials at runtime. For token spend optimization, monitor your token consumption via the Cosmos client telemetry hook. Physics prompt generation consumes significantly more tokens per second than text-only LLM queries because of the underlying vector math translation layers.

\n

What to Build Next

\n

Once you have a steady stream of synthetic physical datasets flowing from Cosmos, you can plug them directly into agentic coding models like RadixArk DaisyRose-27B or train custom vision-language-action policies. Your next logical step is building an automated feedback loop where your robot's real-world failure cases are automatically parameterized into new Cosmos prompts, closing the loop between simulation and reality.

\n

Bottom Line

\n

Stop writing custom physics simulators from scratch. Use NVIDIA Cosmos to generate your physical AI datasets, tie your pipelines to strict validation checks, and move straight to training your models today.

Share this article

Alfian Majid
Alfian Majid/Founder & Editor-in-Chief

Solo developer and blogger from Indonesia. Runs CogitoDaily as a passion project - covering AI news, testing tools, and writing guides. Background in web development and game tech. When not writing about AI, you'll find me deep in anime or gaming.