How we made H3 so cheap

To understand how we got there, it helps to first understand why diffusion video generation is slow in the first place. The diffusion model starts with noise in a latent video representation and repeatedly runs a large transformer to turn that noise into the final video. Base H3 does 49 dense transformer forwards for a single generation, which is very slow.

Hao AI Lab distilled H3 from 49 dense forward passes down to 4 and used Video Sparse Attention (VSA), which divides the video sequence into blocks and only computes attention for the most important blocks. This makes attention about 90% sparse. Using this approach, their FastH3 model generated a 14.37 second video in 12.88 seconds on 8 B200s.

Critically, their fastest sparse attention path uses a custom kernel built specifically for Blackwell. Hopper GPUs like the H100 and H200 instead fall back to a slower Triton implementation which erases all of the speedup so we started from FastH3 and wanted to see whether we could get the same model running in realtime on Hopper.

FastH3's published setup was taking us 23.8 seconds on 8 H200s. We got that down to 14.1 seconds on the exact same H200 node, a 1.69x speedup, and then got the final version to 13.5 seconds on 8 H100 80GBs. We did this without reducing the number of diffusion steps or increasing sparsity.

The main thing we discovered was that once Hao AI Lab made the diffusion process this fast, the model computation itself was no longer actually the biggest bottleneck.

FastH3 was moving its Variational Autoencoder (VAE) from the CPU every request, moving a roughly 67GB text encoder over PCIe every request, and passing a roughly 4GB fp32 frame buffer between processes before encoding the video. The VAE is what converts the latent representation generated by the diffusion model into the actual video you can watch.

VAE decoding alone was taking around 7 seconds. The text encoder had a similar problem. We kept both resident in GPU memory instead, which cut VAE decoding from 7.0 seconds to 2.5 seconds and text encoding from 1.8 seconds to almost nothing.

We also found that after the VAE decoded the video, FastH3 was passing around a roughly 4GB fp32 frame buffer before encoding it. We converted the frames to uint8 earlier, which meant there was much less data to move between processes. We then parallelized the final video encoding, which got our total processing time below the duration of the video.

One result that surprised us was how little FP8 quantization helped performance. FastVideo dynamically quantizes activations on every call, so a lot of the theoretical compute savings get eaten by the extra memory reads and writes. In our case, FP8 only reduced clock time by 0.24 seconds. The more useful part of FP8 was that it reduced memory usage enough for us to get rid of FSDP.

We were also surprised by how close 8xH100s got to 8xB200s here. FastH3 reports 12.88 seconds on 8 B200s, while we got 13.51 seconds on 8 H100s. The B200s have much more compute, FlashAttention 4, and a sparse attention kernel that Hopper does not have, but in this benchmark they are only about 5% faster.

Why is the gap so small? Once you get to 8 GPUs, the model itself is no longer dominating runtime. A lot of the remaining time is communication, decoding, encoding, moving tensors around, and other serial work that does not become twice as fast just because your tensor cores are twice as fast. We think this makes older compute pretty promising even if it's not as powerful as the latest cards.

In total, about 93% of our speedup over the FastH3 setup came from outside the model.

The main result can be summarized as the following: Hao AI Lab showed that H3 could generate video in realtime on Blackwell. We found that you do not actually need Blackwell to do it. 8xH100s are already fast enough.