Portfolio demo

AI-Integrated Renderer

A real-time D3D12 engine that runs AI models inside its render graph

A low-latency rendering pipeline that integrates heterogeneous AI models into rendering, to make real-time scenes more immersive.

D3D12deferred render graph
DirectMLvia ONNX Runtime
1model integrated: text-guided streaming motion
+0.42 msmean frame time added by stream motion generation

Overview

This project builds a low-latency rendering pipeline in which AI models are first-class parts of the frame. The goal is to show how a range of models can enhance immersion, and to record the challenges and insights of making them run alongside rendering. So far, a text-guided streaming motion generation model is integrated, as a first step toward real-world character reactions.

In progress

Real-world character reactions

Characters that move in response to what happens around them, driven by generative motion models. Text-guided streaming motion generation is integrated as a first step.

Planned

Super resolution

DLSS-style upscaling that reconstructs a high-resolution frame from a cheaper render.

Planned

Particle effects

Particle systems whose behavior and appearance come from a learned model.

Planned

Next-frame physics / fluid prediction

Models predict the next geometric state of fluids and particles, which then feeds the shadow and geometry passes.

Planned

Physics

Learned approximations of rigid and soft body dynamics that run alongside rendering.

Planned

Neural materials

Material appearance evaluated by a compact network instead of hand-authored shading code.

Demo

A streaming text-to-motion model drives a skinned character as the text prompt changes over time. Drag the timeline or pick a chapter to jump to an annotated moment.

AI-integrated renderer demo↗
0:00 / 0:00

Rendering architecture

The renderer is a deferred shading render graph. AI models act as intermediate nodes that receive shading and compute resources from earlier passes and produce resources for later ones, all the way to the final render target. A model can even run before the first pass, as particle and fluid prediction does, supplying the geometric state the rest of the frame renders. Developers can then combine models and shading passes to customize the output style, character animation, particle effects, and fluid or physics prediction.

Current render graph with its integrated AI nodeThe CPU culls the scene, then the graphics queue runs the shadow pass, G-buffer, deferred lighting, and tonemap passes. A streaming text-to-motion model runs on a compute queue through DirectML, its output is decoded and retargeted on the CPU, and the pose drives the skinned character in the G-buffer pass.CPUGRAPHICS QUEUERender graph passesAI NODEIntegratedFrustum cullingShadow passCascaded shadow mapsG-BufferSkinned + static geometryDeferred lighting+ baked lightmapsTonemap · UIDisplay spaceFinal targetBack bufferText promptChanges over timeText encoder + denoiserDirectML · compute queueVAE decode + retargetCPUJOINT POSES → SKINNED CHARACTER
Text-guided streaming motion generation: the text encoder and denoiser run on a dedicated compute queue, decoding and retargeting run on the CPU, and the resulting pose drives the skinned character in the G-Buffer pass.
01

Deferred shading render graph

Each pass declares the resources it reads and writes, and the graph resolves barriers. That contract is what lets an AI model sit between passes like any other node.

02

CPU frustum culling

Primitives outside the view are skipped before any constants, descriptors, or draws are recorded for them.

03

Cascaded shadow maps

Shadows for the directional light, split into cascades so near and far geometry both get enough resolution.

04

Baked lightmaps

Point and spot lights are baked into a lightmap atlas, leaving the frame budget for dynamic lighting and model inference.

Running models on D3D12

An AI model's inference is an ordered set of operators. DirectML executes those operators as D3D12 compute work and uses D3D12's own resources and queues. ONNX gives heterogeneous models one portable description, and ONNX Runtime with the DirectML execution provider maps that description onto DirectML.

  1. 1

    Export to ONNX

    Models built in PyTorch, TensorFlow, or JAX are exported to ONNX, one framework-neutral graph of ordered operators.

  2. 2

    Create an ONNX Runtime session with the DirectML execution provider

    The session shares the engine's D3D12 device and runs on a dedicated compute queue, so model work never stalls the graphics queue.

  3. 3

    Fuse and compile

    ONNX Runtime assigns supported operators to DirectML and, when input shapes are fixed, fuses each partition into a single compiled DirectML operator. Unsupported operators fall back to the CPU.

  4. 4

    Bind, dispatch, and synchronize

    For models that consume frame data, resources from earlier render graph passes are bound as model inputs, DirectML records its dispatches on the compute queue, and a fence orders the result before the pass that reads it. The text-to-motion model needs no frame data, so its small inputs and outputs stay in host memory.

Challenges

Two problems stand between a research model and a model that can share a frame with rendering.

01

Inference speed is often neglected in AI model development

Most generative models optimize for output quality, not inference speed. Their run time is dominated by many denoising steps, full-precision floating-point math, and layers that are not designed for throughput. Without inference optimization, the model competes with rendering for the same GPU.

02

Development and inference environments differ

Models are built in PyTorch, TensorFlow, or JAX, so each one has to be checked and converted to ONNX before the engine can run it. ONNX expresses the inference pipeline as ordered, standard operators, but its documentation is incomplete and hides caveats in compute graph compilation and optimization, which makes every conversion a cost of its own.

Insights

What made the difference between a model that runs and a model that fits in a frame.

  1. Pin every input shape

    Exported models default to dynamic input shapes, which keeps ONNX Runtime from fully optimizing the compiled graph. With the DirectML execution provider, fixed shapes are what allow a partition to be fused into one compiled operator instead of being dispatched operator by operator. Pinning each input's shape is necessary for good performance: in this project it collapsed 1,226 DirectML dispatches, each with its own resource barrier, into a single fused dispatch, and cut the mean denoising time from 42.45 ms to 5.54 ms.

  2. Fewer steps, lower precision

    Distilling a model to use fewer denoising steps, and analyzing where floating-point precision can be lowered, is essential for low-latency integration that does not interfere with rendering performance. Streaming models are the exception for step count: a streaming denoiser rolls out a little more motion every frame instead of producing a whole sequence in one pass, so each frame's denoising cost stays roughly constant however many sampling steps the model uses. Reducing steps is then not needed to stay within the frame budget; precision and per-call cost remain the levers. See the measurements below.

  3. Profile the model before integrating it

    Analyze its invocation frequency, input and output shapes, and model size first:

    • Invocation frequency. Models that generate in a latent space and decode to data space can generate every N frames and decode one result across several frames. Pacing the decoder to the playhead has a catch: a prompt change has to wait for every already-finalized latent ahead of the decoder, all of them on the old prompt. Fewer sampling steps let the sampler outrun the paced decoder further, so that backlog grows and must be handled when the prompt changes.
    • Model size. A large model competes with rendering for resources and may not be suitable to ship to players, unless it is distilled into a more compact one.
    • Input and output size. When inputs and outputs are small, sending them to and from the model over IPC can be the pragmatic choice. It avoids debugging where an ONNX conversion or execution failed.

Performance

Cost of running text-guided streaming motion generation inside the renderer, and the effect of pinning input shapes, measured on a single RTX 5080.

2.87 msmean frame time with stream motion generation
+0.42 msmean cost of stream motion generation per frame
6.01 msworst frame with stream motion generation
+2.61 msworst-case cost of stream motion generation

Rendering frame time

MeasurementMeanWorst
Without stream motion generation2.45 ms3.4 ms
With stream motion generation2.87 ms6.01 ms
Added by stream motion generation+0.42 ms+2.61 ms

Stream motion generation

MeasurementMeanWorst
Denoise (10 steps), dynamic shapes1,226 DirectML dispatches, each with its own resource barrier42.45 ms98.43 ms
Denoise (10 steps), shapes pinned1 fused DirectML dispatch5.54 ms6.42 ms
Denoise speed-up from pinning shapes7.7 ×15.3 ×
Decode, dynamic shapesRuns on the CPU2.29 ms3.13 ms
Decode, shapes pinnedRuns on the CPU2.39 ms3.48 ms

Sampling steps and responsiveness

FloodDiffusion is a streaming motion generation model, so the number of sampling steps barely changes what each denoising call costs. It changes how soon the first frame of motion is generated and how quickly a new text prompt is reflected in the motion. In the current pipeline, a new prompt shows up in the motion after about 1.5 seconds at either step count.

Sampling stepsDenoise meanDenoise worstNew-prompt latency
10 stepsShapes pinned5.54 ms6.42 ms~1500 ms
50 stepsShapes pinned5.63 ms6.51 ms~1500 ms

Measured on a single NVIDIA RTX 5080 at 1920×1080 in the Bistro scene with vsync off, after 30 seconds of streaming generation. Lower is better for every time.

References

Third-party model and assets used in this project.

Model

FloodDiffusion ↗

The streaming text-to-motion model that generates the character's motion. Its text encoder and denoiser run through ONNX Runtime with DirectML.

github.com/ShandaAI/FloodDiffusion
Character

Mixamo ↗

Source of the skinned demo character that the generated motion is retargeted onto.

mixamo.com
Scene

niagara_bistro ↗

Source of the Bistro scene used in the demo and in every performance measurement.

github.com/zeux/niagara_bistro