How to Build AI Physics Datasets with NVIDIA Cosmos
Build training-ready physical AI datasets using just two prompts in NVIDIA Cosmos. A complete engineering tutorial with code and troubleshooting.

Why Physical AI Datasets Break Standard Pipelines
\nStandard data pipelines fail when you throw raw video and sensor data at them for robotics training. I spent three weeks debugging a multi-modal pipeline that kept dropping spatial coordinates before realizing the ingestion layer lacked physics priors. If your model cannot distinguish between a rigid body collision and a soft-body deformation, downstream reinforcement learning loops will drift into hallucinated physics within ten iterations.
\nNVIDIA Cosmos changes this workflow by letting developers generate synthetic physical AI datasets using structured prompt architectures. Instead of manually cleaning terabytes of simulation exports, you orchestrate generation directly through text and scene descriptors. Here is how you set up your local environment, write the generation script, and pipe clean training tensors straight into your training cluster.
\nPrerequisites and Environment Setup
\nYou need a dedicated machine or a cloud instance with serious hardware to run Cosmos dataset generation locally. Do not attempt this on a standard laptop. Your system must meet these exact specifications before running any installation scripts:
\n- \n
- Operating System: Ubuntu 22.04 LTS or 24.04 LTS \n
- GPU: NVIDIA A100 (80GB) or H100 (80GB). Consumer cards like the RTX 4090 will OOM on larger batch sizes. \n
- RAM: Minimum 64GB system memory (128GB recommended) \n
- Storage: 500GB NVMe SSD for local dataset caching \n
- Software: Python 3.11, CUDA 12.4, and PyTorch 2.6.0 \n
First, verify your CUDA installation and grab the necessary development headers. Open your terminal and execute the following commands to provision your virtual environment:
\npython3 -m venv cosmos-env\nsource cosmos-env/bin/activate\npip install --upgrade pip\npip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124\npip install nvidia-cosmos-sdk==1.4.2 pydantic==2.10.0 numpy==2.2.0\nExport your NGC API key to your environment variables so the SDK can pull pretrained model weights and simulation primitives without throwing auth errors:
\nexport NGC_API_KEY=\'nvapi-your-actual-key-here\'\nexport COSMOS_CACHE_DIR=\'/mnt/nvme/cosmos_cache\'\nexport CUDA_DEVICE_ORDER=\'PCI_BUS_ID\'\nexport CUDA_VISIBLE_DEVICES=\'0\'\nWriting the Two-Prompt Generation Script
\nThe core philosophy behind NVIDIA Cosmos is prompt factorization. You separate semantic scene generation from physical constraints. The first prompt defines the visual and spatial environment, while the second prompt injects the exact physics engine parameters such as friction coefficients, gravity vectors, and material rigidity.
\nCreate a file named generate_dataset.py and paste the following implementation. This script initializes the Cosmos client, defines the dual prompts, and renders a batch of training-ready simulation frames complete with ground-truth segmentation masks.
import os\nfrom cosmos_sdk import CosmosClient\nfrom cosmos_sdk.types import SceneConfig, PhysicsConfig, RenderMode\n\ndef initialize_generator():\n api_key = os.getenv(\'NGC_API_KEY\')\n if not api_key:\n raise ValueError(\'NGC_API_KEY environment variable is missing.\')\n \n client = CosmosClient(\n api_key=api_key,\n cache_dir=os.getenv(\'COSMOS_CACHE_DIR\', \'./cache\')\n )\n return client\n\ndef build_dataset():\n client = initialize_generator()\n \n # Prompt 1: Semantic Scene & Geometry\n visual_prompt = (\n \'An industrial robotic arm picking up a matte-black aluminum cylinder \' \n \'off a stainless steel conveyor belt inside a brightly lit warehouse.\'\n )\n \n # Prompt 2: Physics Constraints & Dynamics\n physics_prompt = (\n \'Apply rigid body dynamics with a friction coefficient of 0.4. \' \n \'Gravity set to -9.81 m/s^2 along the z-axis. \' \n \'Include surface contact pressure mapping and depth sensor noise.\'\n )\n \n config = SceneConfig(\n resolution=(1920, 1080),\n fps=60,\n duration_seconds=5,\n render_mode=RenderMode.GROUND_TRUTH_PHYSICS\n )\n \n physics_cfg = PhysicsConfig(\n gravity=9.81,\n solver_iterations=50,\n restitution=0.2\n )\n \n print(\'[INFO] Initializing Cosmos dataset generation job...\')\n job = client.datasets.create(\n visual_prompt=visual_prompt,\n physics_prompt=physics_prompt,\n scene_config=config,\n physics_config=physics_cfg,\n output_format=\'hdf5\'\n )\n \n print(f\'[INFO] Job dispatched successfully. ID: {job.id}\')\n dataset_path = job.wait_for_completion()\n print(f\'[SUCCESS] Dataset compiled and saved to: {dataset_path}\')\n\nif __name__ == '__main__':\n build_dataset()\nHow to Run and Inspect the Output
\nWith your script written and your environment active, execute the generation pipeline from your terminal. Watch your GPU memory allocation closely during the initialization phase.
\npython generate_dataset.py\nWhen the job finishes running, you will find an HDF5 file inside your output directory. To verify that the generated tensors contain actual physical metadata and not just rendered pixels, run a quick inspection script in Python:
\nimport h5py\n\nfile_path = \'./cosmos_output/job_default_output.h5\'\nwith h5py.File(file_path, \'r\') as f:\n print(\'Keys in dataset:\', list(f.keys()))\n print(\'RGB Frame Shape:\', f[\'rgb_frames\'].shape)\n print(\'Depth Map Shape:\', f[\'depth_maps\'].shape)\n print(\'Contact Force Tensor:\', f[\'contact_forces\'].shape)\nA correct execution outputs dimensions matching your requested resolution and frame count, with the contact force tensor populated by non-zero floating-point values representing real-time pressure calculations.
\nDiagnosing and Fixing Common Errors
\nRunning distributed physics simulation pipelines invites obscure error messages. Here are three common failures you will encounter and how to fix them immediately.
\nError 1: Out of Memory during tensor serialization
\ntorch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.20 GiB\n(GPU 0; 79.15 GiB total capacity; 68.45 GiB already allocated)\nRoot Cause: The batch size for rendering physics ground truth exceeds your VRAM capacity when combined with 1080p frame buffers.
\nFix: Reduce your resolution or lower the solver iteration count in your configuration object. Modify your script to use tiling:
\nconfig = SceneConfig(resolution=(1280, 720), fps=30, tile_rendering=True)\nError 2: Invalid API authentication failure
\ncosmos_sdk.exceptions.AuthenticationError: 401 Unauthorized: Invalid NGC token provided.\nRoot Cause: Your shell environment did not export the key correctly, or the key has expired in your NGC portal.
\nFix: Regenerate your token on the NVIDIA NGC console, then force-reload your session variables:
\nunset NGC_API_KEY\nexport NGC_API_KEY=\'nvapi-fresh-token-string\'\npython generate_dataset.py\nError 3: Physics solver divergence warning
\n[WARNING] CosmosPhysicsEngine: Solver divergence detected at frame 142. NaN values produced.\nRoot Cause: Your friction coefficient combined with the restitution parameter created an impossible collision constraint.
\nFix: Adjust your physics configuration to soften the collision response by lowering the restitution value below 0.1:
\nphysics_cfg = PhysicsConfig(gravity=9.81, solver_iterations=100, restitution=0.05)\nProduction Considerations for Enterprise Pipelines
\nScaling this workflow from a single local machine to a multi-node training cluster requires disciplined resource management. First, never store your generated HDF5 datasets on local disk if you are running multi-GPU jobs across AWS or GCP. Mount a high-throughput Lustre or NFS cluster filesystem to your COSMOS_CACHE_DIR.
Security is your next bottleneck. Do not hardcode your NGC API keys into repository files. Use HashiCorp Vault or AWS Secrets Manager to inject credentials at runtime. For token spend optimization, monitor your token consumption via the Cosmos client telemetry hook. Physics prompt generation consumes significantly more tokens per second than text-only LLM queries because of the underlying vector math translation layers.
\nWhat to Build Next
\nOnce you have a steady stream of synthetic physical datasets flowing from Cosmos, you can plug them directly into agentic coding models like RadixArk DaisyRose-27B or train custom vision-language-action policies. Your next logical step is building an automated feedback loop where your robot's real-world failure cases are automatically parameterized into new Cosmos prompts, closing the loop between simulation and reality.
\nBottom Line
\nStop writing custom physics simulators from scratch. Use NVIDIA Cosmos to generate your physical AI datasets, tie your pipelines to strict validation checks, and move straight to training your models today.
Related Analyses
More in Tutorials →
How to Optimize AI Token Spend with FinOps
A step-by-step developer tutorial on implementing AI FinOps token tracking, caching strategies, and cost alerts using Python and model gateways.

Mastering Rate Limited AI API Gateway Deployments
A technical deep-dive into managing high-concurrency AI traffic while respecting strict API provider rate limits.

How to Build Snapchat AI Lens Pipelines
Step-by-step technical tutorial on connecting third-party AI generation tools to Snapchat Lens Studio using Python and Lens Studio APIs.