Real-world character reactions
Characters that move in response to what happens around them, driven by generative motion models. Text-guided streaming motion generation is integrated as a first step.
Portfolio demo
A real-time D3D12 engine that runs AI models inside its render graph
A low-latency rendering pipeline that integrates heterogeneous AI models into rendering, to make real-time scenes more immersive.
This project builds a low-latency rendering pipeline in which AI models are first-class parts of the frame. The goal is to show how a range of models can enhance immersion, and to record the challenges and insights of making them run alongside rendering. So far, a text-guided streaming motion generation model is integrated, as a first step toward real-world character reactions.
Characters that move in response to what happens around them, driven by generative motion models. Text-guided streaming motion generation is integrated as a first step.
DLSS-style upscaling that reconstructs a high-resolution frame from a cheaper render.
Particle systems whose behavior and appearance come from a learned model.
Models predict the next geometric state of fluids and particles, which then feeds the shadow and geometry passes.
Learned approximations of rigid and soft body dynamics that run alongside rendering.
Material appearance evaluated by a compact network instead of hand-authored shading code.
A streaming text-to-motion model drives a skinned character as the text prompt changes over time. Drag the timeline or pick a chapter to jump to an annotated moment.
The renderer is a deferred shading render graph. AI models act as intermediate nodes that receive shading and compute resources from earlier passes and produce resources for later ones, all the way to the final render target. A model can even run before the first pass, as particle and fluid prediction does, supplying the geometric state the rest of the frame renders. Developers can then combine models and shading passes to customize the output style, character animation, particle effects, and fluid or physics prediction.
Each pass declares the resources it reads and writes, and the graph resolves barriers. That contract is what lets an AI model sit between passes like any other node.
Primitives outside the view are skipped before any constants, descriptors, or draws are recorded for them.
Shadows for the directional light, split into cascades so near and far geometry both get enough resolution.
Point and spot lights are baked into a lightmap atlas, leaving the frame budget for dynamic lighting and model inference.
An AI model's inference is an ordered set of operators. DirectML executes those operators as D3D12 compute work and uses D3D12's own resources and queues. ONNX gives heterogeneous models one portable description, and ONNX Runtime with the DirectML execution provider maps that description onto DirectML.
Models built in PyTorch, TensorFlow, or JAX are exported to ONNX, one framework-neutral graph of ordered operators.
The session shares the engine's D3D12 device and runs on a dedicated compute queue, so model work never stalls the graphics queue.
ONNX Runtime assigns supported operators to DirectML and, when input shapes are fixed, fuses each partition into a single compiled DirectML operator. Unsupported operators fall back to the CPU.
For models that consume frame data, resources from earlier render graph passes are bound as model inputs, DirectML records its dispatches on the compute queue, and a fence orders the result before the pass that reads it. The text-to-motion model needs no frame data, so its small inputs and outputs stay in host memory.
Two problems stand between a research model and a model that can share a frame with rendering.
Most generative models optimize for output quality, not inference speed. Their run time is dominated by many denoising steps, full-precision floating-point math, and layers that are not designed for throughput. Without inference optimization, the model competes with rendering for the same GPU.
Models are built in PyTorch, TensorFlow, or JAX, so each one has to be checked and converted to ONNX before the engine can run it. ONNX expresses the inference pipeline as ordered, standard operators, but its documentation is incomplete and hides caveats in compute graph compilation and optimization, which makes every conversion a cost of its own.
What made the difference between a model that runs and a model that fits in a frame.
Exported models default to dynamic input shapes, which keeps ONNX Runtime from fully optimizing the compiled graph. With the DirectML execution provider, fixed shapes are what allow a partition to be fused into one compiled operator instead of being dispatched operator by operator. Pinning each input's shape is necessary for good performance: in this project it collapsed 1,226 DirectML dispatches, each with its own resource barrier, into a single fused dispatch, and cut the mean denoising time from 42.45 ms to 5.54 ms.
Distilling a model to use fewer denoising steps, and analyzing where floating-point precision can be lowered, is essential for low-latency integration that does not interfere with rendering performance. Streaming models are the exception for step count: a streaming denoiser rolls out a little more motion every frame instead of producing a whole sequence in one pass, so each frame's denoising cost stays roughly constant however many sampling steps the model uses. Reducing steps is then not needed to stay within the frame budget; precision and per-call cost remain the levers. See the measurements below.
Analyze its invocation frequency, input and output shapes, and model size first:
Cost of running text-guided streaming motion generation inside the renderer, and the effect of pinning input shapes, measured on a single RTX 5080.
| Measurement | Mean | Worst |
|---|---|---|
| Without stream motion generation | 2.45 ms | 3.4 ms |
| With stream motion generation | 2.87 ms | 6.01 ms |
| Added by stream motion generation | +0.42 ms | +2.61 ms |
| Measurement | Mean | Worst |
|---|---|---|
| Denoise (10 steps), dynamic shapes1,226 DirectML dispatches, each with its own resource barrier | 42.45 ms | 98.43 ms |
| Denoise (10 steps), shapes pinned1 fused DirectML dispatch | 5.54 ms | 6.42 ms |
| Denoise speed-up from pinning shapes | 7.7 × | 15.3 × |
| Decode, dynamic shapesRuns on the CPU | 2.29 ms | 3.13 ms |
| Decode, shapes pinnedRuns on the CPU | 2.39 ms | 3.48 ms |
FloodDiffusion is a streaming motion generation model, so the number of sampling steps barely changes what each denoising call costs. It changes how soon the first frame of motion is generated and how quickly a new text prompt is reflected in the motion. In the current pipeline, a new prompt shows up in the motion after about 1.5 seconds at either step count.
| Sampling steps | Denoise mean | Denoise worst | New-prompt latency |
|---|---|---|---|
| 10 stepsShapes pinned | 5.54 ms | 6.42 ms | ~1500 ms |
| 50 stepsShapes pinned | 5.63 ms | 6.51 ms | ~1500 ms |
Measured on a single NVIDIA RTX 5080 at 1920×1080 in the Bistro scene with vsync off, after 30 seconds of streaming generation. Lower is better for every time.
Third-party model and assets used in this project.
The streaming text-to-motion model that generates the character's motion. Its text encoder and denoiser run through ONNX Runtime with DirectML.
github.com/ShandaAI/FloodDiffusionSource of the skinned demo character that the generated motion is retargeted onto.
mixamo.comSource of the Bistro scene used in the demo and in every performance measurement.
github.com/zeux/niagara_bistro