Training data is costly
High-quality and diverse data is essential for AI model training. Yet obtaining such data is both expensive and labor-intensive.
Game render extraction demo
A universal real-time pipeline for modern game engines
Camera, metric depth, and scene information captured live from commercial games, for large-scale AI model training.
Why extract render data from games, and why it has to work across engines.
High-quality and diverse data is essential for AI model training. Yet obtaining such data is both expensive and labor-intensive.
Games, originally made to entertain players with immersive worlds and convincing physics, are good sources of synthetic training data, as they cover many carefully designed scenes.[1][2]
However, creating tools to extract diverse game data requires massive reverse engineering work and dedicated modding tools.
Project goalCreate a universal render data extraction system that needs minimal reverse engineering work, for rendering analysis.
This project aims to build a universal, real-time render data extraction pipeline on modern game engines. Instead of integrating with each engine separately, it works at the graphics API level, so games built on very different engines share a single extraction path. The data it collects is intended for large-scale AI model training.
One Graphics API Proxy sits between the engine and D3D12, covering Messiah, Snowdrop, Anvil, BlackSpace, RAGE and Unreal titles.
Data is copied every frame while the game runs, at a cost of tens of microseconds per frame.
Camera pose, metric depth, and object geometric and material information, captured together for each frame.
The same pipeline extracts different kinds of render data. Switch between the camera trajectory demos and the object information demo below.
Each demo pairs gameplay with the camera trajectory extracted from it. The two videos share a roughly synchronized timeline, so they can be watched side by side.
The game execution environment and the visualization environment are kept separate. The pipeline sends the extracted data to Blender over UDP, and Blender renders the camera trajectory. Because UDP gives no delivery guarantees, some latency and packet loss in the trajectory view is expected.
Every engine renders through the same graphics API. The Graphics API Proxy intercepts those calls, forwards them to D3D12, and uses D3D12 to retrieve the camera, metric depth, and object geometric and material information.
Measured on two commercial games. Note that Where Winds Meet is an MMORPG with built-in anti-cheat.
| Game | Camera pose | Metric depth (1920×1080) |
|---|---|---|
| Avatar: Frontiers of Pandora | 21.493 µs / frame | 73.45 µs / frame |
| Where Winds MeetMMORPG · built-in anti-cheat | 10.629 µs / frame | 30.427 µs / frame |
| Game | Camera pose | Metric depth | ||
|---|---|---|---|---|
| Mean | Max | Mean | Max | |
| Avatar: Frontiers of Pandora | 1 | 1 | 2.038 | 5 |
| Where Winds MeetMMORPG · built-in anti-cheat | 2.610 | 4 | 2.482 | 3 |
Latency is measured in frames. Lower is better for every value on this page.