SYDNEY, AUSTRALIA — When many studios are limited to one facility and a small bank of computers, Australia’s Cutting Edge was sitting on a gold mine – two big VFX studios and all the hardware that came with them. Two times the resources doesn’t mean two times the efficiency, though, especially when your farms don’t talk to each other. To create that bridge, the team invested in PipelineFX’s Qube!, a render farm management tool tailored to the needs of big VFX projects.
Since 1992, Cutting Edge has built a string of studios that employs over 150 artists and reaches from Brisbane to Sydney. Like most professional shops they can do it all: films (Predestination, The Age of Adaline); commercials and TV series, including Powers, PlayStation Network’s first scripted show that’s based on Marvel’s beloved graphic novel series. What they couldn’t do was run all render requests off a single dispatcher, and the minutes kept adding up.
Plans to link the farms in Brisbane and Sydney started with a test. The idea was to put render farm managers against each other in a multi-site battle, utilizing both live and offsite workers. This became the proving ground for key needs like: global log files tracking, scripting, platform compatibility and project mirroring. After tests totaling 10,000 jobs, Qube! was declared the winner, in large part, because of how well it could handle the logs.
“With other render management tools, reading log files for machines that aren't local was problematic, and reading log files is an incredibly important part of seeing what's going on and troubleshooting,” says Rangi Sutton, VFX supervisor at Cutting Edge. “Our TDs and support crew can now dial up logs for a given job through GUIs or the command line tools supplied on any workstation. It’s the only way to see how everything's going across the sites.”
By running off a single dispatcher, artists can submit jobs onsite or remotely through VPN and get access to the power of two farms. With this single point of control, Cutting Edge can now count the software licenses for applications like NUKE for all users over the group. Previously, license allocations needed to be predefined between Sydney and Brisbane. During the day the farms consist of dedicated slave units, but after 7pm, all idle workstations join the pool, offering a 20-30% boost in rendering power every night.
Another key feature that drew the team was Qube!’s native compatibility with OSX and Linux, and the Unix-like design. “We already have in-house tools to manage our environment for our major applications; Maya, C4D, NUKE, Houdini, so we can control the specifics of the software used per project and even per shot. While Qube provides a lot of built-in job types that would help smaller studios kick things off, they don’t get in our way when we choose to ignore them. Essentially, the way we initialize session at the workstation on the terminal, is exactly the same as what happens on the render farm. We don’t run into new sets of problems for farm-rendered jobs such as user permissions or application versions.”
Running simulations in Houdini proved to be another source of time-savings thanks to per-frame dependencies. In the past, the team found their system was dependent on stages completing before the render could move on. With Qube! that all changed.
“Now we can send stages to the Supervisor and tell Qube! how to perform the operations: pre-processing, simulations, post-processing, followed by rendering. We can configure the granularity of these dependencies; simulations will require all the pre-processed geometry to be available before commencing, whereas a Mantra render need only wait for all the geometry for its given frame.
Using the Qube! Python API, we can describe all these stages from inside of Houdini's Python interpreter, and submit the whole dependency tree in one go,” said Sutton. “Retakes of big effects scenes can be re-simulated and re-rendered through on a single-button click, without any unnecessary idle render nodes or wrangler intervention.”
In the near future, Cutting Edge will also be able to do something that was impossible before: track farm data for every project. “We had access to resource tracking – how long a crew has been working on individual shots, different storage stats – but never a stat like comparing CPU cycles against project size. With 50,000 render jobs over the last six months, it’s the aggregate data that Qube! is recording in its data warehouse that will help us make a lot of important strategic decisions. Now we can apply cost-accounting to the render usage, as well as make software license decisions based on the actual usage over time.”