Using “near-real-time” hardware rendered imagery in traditional high-end animation and visual effects pipelines.

By Terrence Masson; Digital Fauxtography Inc.

May, 2003

Let me start by saying that I am excited like I haven’t been for many years about the near future of computer graphics. Time was when every single year there was something new and exciting to anticipate at SIGGRAPH. There were always some brand new techniques like morphing and particle systems, the always unique and annual(!) Pixar shorts and breakthrough new applications in film from ILM like the Abyss and T2. … stuff to make you fairly drool for that electronic theater show. More than the singular show experience you would also get ready for those tools to migrate down and out of the R&D into available commercial applications. Who can remember “playing with” Elastic Reality and Dynamation (Willy!) for the very first time? With very few exceptions (like more recent flocking / crowd simulators and 2d/3d tracking programs) the progress of the last ten years has been more incremental than the decade before. Granted this should be expected given the youth of our industry. We needed time to digest these new and wonderful tools in pursuit of “photorealism” and recreating natural phenomenon. All this for sound economic reasons to be sure, but unfortunately all too often at the expense of pure artistic creativity; again with some wonderful exceptions such as Yoichiro Kawaguchi and the omnipresent Charles Csuri.

I create this historical backdrop for a reason, to illustrate the dramatic new frontier for us all by posing a question to the “users” out there : What new process or application is there to “get us all jazzed” moving forward? My contention is that it is hardware rendering; specifically programmable shading languages for GPU driven graphics cards. The reader should notice that I have not explicitly mentioned “video games” yet; this is for several reasons. Firstly the migration of feature film talent and tools to video game production is definitely a recent and wonderful trend that everyone should feel good about. In the last two years many feature film visual effects artists have made the switch to video games, most notably to Electronic Arts (EA), the worlds largest (and perhaps the most forward thinking) game publishing companies. I am however not talking about using new talent to “kick it up a notch” for in-game and pre-rendered quality in video game production. This pipeline has broad applications in every corner of computer graphics, animation and film production. My second main reason for not mentioning video games is much more pedantic; it is to avoid this reaction:

Producer: “Video game graphics in my movie? Never.”

So to put my fellow visual effects supervisors and producers at peace we’ll get down to clarifying. The topic of this article is something that I have been speaking about for several years now, namely using “video game” hardware rendering technology to benefit traditional software rendering production pipeline of high end feature film visual effects and animation. “High-End” is also a key phrase that focuses our topic apart from other application of real-time (or near real-time) applications of “video game” hardware rendering technology.

Image01: Real-time hardware rendering graphics from Robocop2

Historical SIDE BAR: Jay Jay the Jet Plane produced by Modern Cartoons in Los Angeles is an excellent recent example of a successful “non video game use” of real-time hardware rendering. Historically, deGraf/Wahrman’s work in Robocop2 (1990) was the first feature film use, based on their experience with “Mike the Talking Head” in 1988. PDI and Henson also created in 1988 “Waldo C. Graphic” for the television Muppet Show, all of which utilized the hardware rendering of the new SGI 4D workstations.

Image02: Real-time hardware rendering Waldo C. Graphic from Jim Henson

Since my focus is on using hardware rendering for “high end” visuals as opposed to “real time video games”, we should define what does one exactly mean by “high end” anyway? In feature film visual effects and animation, quality (as opposed to frame rate) is the number one priority of the technical artist. You can always crank up those samples (or your “spec mod” or whatever) and throw another “render farm weekend” at a shot. Massively (usually TOO massivly) complex geometry coated with hundreds of huge textures, driven by articulated rigging and thickly layered intricate shaders under many (usually way TOO may) lights. Now before all the pixel-tweaking snobs start screaming “You can’t do that in a video game!” Am I saying that we now can do all this in hardware in realtime and throw away all our software renderers? In a word: no. The key is to get outside of thinking in REAL-time (30 frames per second for instance) and get into thinking NEAR-real-time. If we don’t have to rely on the myriad of limitations in any given game engine that nessisarily restricts quality in the favor of frame rate, we can then “crank the quality up to 11” so to speak. Pull out all the stops, add all the functionality and quality you want to. Your frame rate will drop from 30 to 10fps or to 2 or even 1, but chances are you may still be getting a 100-fold advantage over an equivalent software renderer. Saved to disc the hardware rendered images are simply utilized in a traditional multi-layered compositing pipeline.

Image03: Real-time hardware rendering Dawn from nVIDIA

So now that all the light bulbs have gone on over some heads and the epiphanies are settling in, you want to implement these tools into your production pipeline right? You can either hire an outside contractor or “roll your own”. Thankfully the financial bar for getting into this is extremely low, even by today’s plummeting tool cost standards. For example, configuring a new Dell workstation with a FX1000 card running Cg shaders would be your basic standard TD tool set for well under $3000. Today’s latest hardware rendering is best exemplified by Nvidia’s Quadro FX series and by the ATI Fire GL-X1. Both provide 8X AGP, 128 bit graphics pipeline with full IEEE 32 bit floating point precision. Just replace your graphics card, download your choice of high level shading language, upgrade to DX9 and you’re good to go. (See web links below for many more details.)

For software, Nvidia’s Cg programming language and ATI’s RenderMonkey shader tool offer fully programmable vertex and per-pixel shaders. For example, operating as a layer above OpenGL and DirectX, Nvidia’s Cg provides open source for Windows, Linux, Xbox and (soon) MacOSX; is being used actively by hundreds of developers worldwide and is endorsed by Alias|Wavefront Maya, Discreet 3Dmax and Softimage|XSI.

Image04: Real-time hardware rendering Ogre from nVIDIA and Spellcraft Studios.

Basically speaking there is not much in a given image that you cannot create equally well rendered in hardware rather than software; including realistic hair and lighting effects. Along with the advantages come most of the same challenges still found in traditional software renderers.

Image05: Real-time hardware rendering chimp from ATI

Image06: Real-time hardware rendering bacteria from ATI

Highly realistic natural phenomenon (fire, smoke, flowing water) continue to be some of the most difficult and computationally complex images one must create. Thankfully excellent production quality 4X anisotropic anti-aliasing is now possible, but motion blur continues to be an elusive challenge in hardware rendering. This and other realistic optical effects are clearly the largest remaining challenge. Even so, in camera depth of field has been recently illustrated in demos on both the Nvidia and ATI web sites, so the door has been opened.

Image07: Real-time hardware rendering in-camera depth of field from nVIDIA

We can create all this great imagery, in a fraction of the time, but I can hear you out there saying “But are we limited to video res?!” Not at all, in fact the only real restriction to pixel resolution (at least with the Nvidia’s NC30 based Cg language) is an inherent 128 meg memory limitation, which is hardly limiting. To illustrate an example : a 1920 x 1080, 32 bit per pixel “high-def” frame contains some 2,073,600 pixels, at 4 bytes each gives us about 7.9 megs per frame. Which leaves us plenty of room for process overhead while cranking these images out.

There of course will be some particular “icing on the cake” rendering effects particularly suited for software rendering. Right now HDR (high dynamic range) is all the rage and not as yet fully implemented on hardware for instance. Rather than technically incapable, this and other implementations are simply waiting for time and attention to be given by the right team of software engineers and R&D production TDs. It is therefore my hope that more companies will create new, flexible, next generation production pipelines that combine the best elements of both these tool sets. This has been just the briefest of introductions to what will prove to be an exciting and challenging new stage of computer graphics. Look to VFXpro.com for follow up articles later this summer with specific feature film shot by shot examples utilizing programmable hardware shading and rendering

Valuable web reference links:

1) Nvidia demo page : http://www.nvidia.com/view.asp?PAGE=power_demos

2) The Cg Tutorial: The Definitive Guide to Programmable Real-Time Graphics by Fernando and Kilgard from Addison-Wesley. ISBN 0321194969

3) Nvidia, “Integrating the Cg language into the professional workflow”: http://www.nvidia.com/docs/lo/2674/SUPP/QuadroFX-Cg.pdf

4) CG Shaders resource on the web http://www.cgshaders.org/

5) ATI demo page : http://mirror.ati.com/developer/demos/r9800.html

6) ATi RenderMonkey Toolsuite: http://mirror.ati.com/developer/sdk/radeonSDK/html/Tools/RenderMonkey.html

7) Waldo C. Graphic bio : http://www.henson.com/fun/fcreature/waldo_fcbts.html

8) Spellcraft Studios Ogre : www.yeahthemovie.de

9) For a detailed history of computer graphics read CG 101: A Computer Graphics Industry Reference by Terrence Masson from NewRiders. www.cg101.com