So everyone gets what VFX is. And PDX is a no brainer for anyone who’s ever gotten tired of writing out P-O-R-T-L-A-N-D and is fine enough with an airport acronym, and a cool one at that (what with it’s iconic X). But DPX? Who the what now? Have I gone and scrambled my brains at the Driftwood Room again prior to posting?
DPX is my favorite file image format. And oh how I missed it.
You’ll have to excuse me for a sec while I geek out. But we’re never gonna survive unless we get a little geeky.
Pulling a DPX into Nuke used to mean you were working on a film. Hell yeah. This is something I took for granted; that those tasty dpx’s would always be sitting there, waiting for me every morning.
I’m not sure quite sure when they went away, but at some point along the way, along came EXR’s, and things got all complicated.
Now at my brand new gig, I’m the “end of the line.” Has a nice ring to it! And guess what? DPX is back in vogue. Allow me to sing it’s praises.
For live action footage, there is no better. Nuke absolutely eats a DPX for breakfast.
And I’m not talking Portland brunch… where you cruise into Tin Shed on Alberta, put your name down, grab yourself a cuppa the “free” Joe only it’s not really free because you’ll be billed for it in the event that you wait long enough to finally sit down and your server sees that you have the cup and marks you down for 1 stumptown special, but let’s not even go there yet because you’re still sitting there waiting but you stick it out because it’s like you brought your dog and Tin Shed is like the only place that really allows dogs because they have the outdoor seating and all that but it’s like sunny and so it’s crowded but hey it is SUNday after all so like whatever I’ll just sit here for what time is it? is that even morning anymore? and wait until they call my name and there’s not really much else within biking distance except hey maybe Juniors is my backup plan but I’d have to tie the dog up out front and the food is good and all but it was kind of dirty in there last time I was there or at least my cup had spots on it and my water was kind of, I don’t know, dusty? so like I should probably just wait it out but there’s like 3 other 2-tops ahead of me on the list, like when did this place turn into Gravy anyway I thought Tin Shed was like normal but whatever it is like the best thing Alberta has so I better just suck it up.
Ok, that diatribe you just went through? That was an EXR. DPX would have you out the door and smiling already while EXR is still loading in scanlines. If you want to talk in brunch terms, this is like Gravy vs. Equinox – am I right? Ok, maybe I should explain that ref for any non P-towners who aren’t familiar with our obsessive brunch scene. Gravy over on Mississippi Ave is hands down one of the best brunches in town. The best french toast in the city – maybe the galaxy. But you wait for it. It’s a mob scene even on a weekday. In other words, it’s great but it’s the SLOW boat. However, right around the corner from Gravy is Equinox. Which is a damn good brunch – up on par with Gravy for sure. The difference being it’s tucked around the corner, just enough off the beaten path where you can walk right in and sit down immediately. You’re done and moving on with the rest of your day while the Gravy folks are still standing outside, tapping feet, shooting glances to watches, pondering how much longer before they just… bail.
Don’t be afraid to bail on EXR when the circumstances permit. Proof? You want proof? Get this – for a plain Jane little 3 node comp render of 50 some odd frames in Nuke (in > blur > out) with footage shot originally with a RED Epic @ 4k, here are the stats:
EXR (zip scanline) – 2 minutes 57 seconds
DPX – 10 seconds.
No @$#% kidding. 10 seconds. I mean, I had a feeling DPX would be faster, just from working with them time and time again… but I’ll admit that had my head spinning. I’m not even sure how to explain this because it doesn’t add up – there is nothing apparently different about this machine I’m on… there’s no DPX equivalent of a Red Rocket card or anything, and no GPU accelerated nodes in my script. I’m scratching my head – how is this possible? This box does have an SSD but I was sure to clear the caches before rendering, so it should be apples to apples. I ran it twice and got similar render times. The sys admin wizards at Weta Digital might have some sort of caching enabled on Linux here that is somehow locally stashing the DPXs in RAM, but not the EXRs. Hmmm. That gives me more questions than answers! Mental note: never ever leave Linux.
Anyway, a 2nd opinion is in order, good doctor. Let’s try something a bit more scientific and I’ll shoot this to the renderfarm. I’ve switched shots and have a 2k shot that originated on film, not that it really matters once you’ve made a DPX and an EXR. Kicking these off now and will wait for that juicy stats email… looking for total time elapsed and I’ll get you a per frame CPU time (minus batch load time).
EXR (piz): 6m 53s – 7 secs
EXR (scanline): 6m 38s – 6 secs
DPX: 4m 10s – 4 secs
This was done on a Sunday – as low traffic and open of a farm as Weta Digital gets. Different versions were rendered within minutes of each other, and those Nuke renders should have had the blades all to themselves. The bottom line: 33% faster rendering.
You can talk all you want about how Nuke uses EXRs as it’s internal intermediary format and is floating point across the board. Bla bla bla. Look at the results. That adds up to a heck of a lot of saved time over the course of the day, not to mention the interactive boost while working. As long as your capture format is less than 16 bit, and if they are shooting digital (or even film), it is most definitely less than 16 bit – you simply cannot beat what I’ll call “integer love.” The CPU just chews through it. For grained live action footage shot with most cameras, DPX. There is no substitute.
The RED Epic, with it’s claimed 11 stops of RAW dynamic range? No problem. The Canon 5D and it’s 14 bit sensor? Easy money. 10bit Log DPX handles as much dynamic range as 16bit linear because, kind of like an mp3 still sounds damn good (or maybe FLAC is a better comparison), log space will put all of the image goodness where it counts. Your highlights? Don’t worry, they’re unscathed and intact.
File size you say? Network traffic? Also not a problem here. 10bit log DPX was 13mb per beautiful grainy frame in this example (2k live action plate). EXR/Piz could only come up with 14mb. That varies, but the time spent unpacking them doesn’t so often a small filesize advantage for Piz can be ignored in favor of performance.
Some people will scoff at a raw DPX and it’s log encoding, saying it looks washed out and unviewable. First of all, who are these people and why are you letting them anywhere near a vfx pipe decision? Amateurs! Ha! The truth is, an EXR in linear colorspace is just as strange to view in it’s dark raw state. You’re never going to view either of them raw and are gonna have to chuck a LUT on it either way, so this is a non-issue. Worry not, Photoshop can still open it correctly.
Don’t get me wrong, DPX is not for everything. There’s the whole alpha channel thing. The dpx spec can handle it (as well as rendering up to 16 bit linear) but many of the software, After Effects comes to mind, still seems to be stuck on the old Cineon spec. Cineon was what DPX was before it was dpx – .cin – and Cineon had no alpha channel support and was locked to 10 bit log.
CG renders especially will use all of the extra range goodness that EXR can offer up. Over in EXR land, Piz compressed is the way to go for live action plates over slow networks, or large feature film projects that need oodles of storage. If you can afford larger filesizes, Nuke likes a good scanline zipped EXR much better. Single scanlines for live action, and often shops will write tiled for CG out of the renderer and then convert to ZipS as a post process.
Really, the big revelation with EXR was embedding multiple passes into one file, and knocking things down to “half-float” for all of the calculation and storage advantages. Finally, floating point was worth the price of admission. Historically, Tiff could always do float precision but no one used it because the extra usable range came at such a rendering and disk space cost that it was overkill 99.997% of the time. In fact, at Weta prior to the EXR revolution, we were using liff (essentially a log IFF) as our primary format for CG renders, for many of the same reasons that DPX works so well.
When EXR 2.0 is finally released, it will up the ante and you’re going to start seeing more standardization of the deep image formatting for CG without holdouts or layering problems that useless per pixel Z passes always had.
But for live action plates (in most cases*), viva la DPX!