Most of you may not realize it, but you’ve got a secret habit that you’ve been keeping from your friends and family for years. But not to worry, odds are your loved ones are also guilty of the same behavior. I’m of course talking about digital hoarding. The good news is, digital hoarding is not fatal and there is a cure for it — compression.
So how did we get to the point where we’re acquiring more and more data without ever deleting it? You can blame that on modern cloud platforms — serving up bottomless buckets of capacity. This new lifestyle allows us to store with reckless abandon and we rarely feel the direct impact. Often when I’m presenting to an audience, I like to ask how many of them have ever deleted an email on their free cloud email platform. Often it is just one person — it must have been pretty motivating for him/her to press that delete button! For the most part, email, media and documents just go into the infinite bit bucket because it’s easiest and we know that we can use modern search to fish out what we really care about.
Modern operating systems such as Windows 10 can have file compression turned on by clicking a checkbox, trading .ZIP file archives across the cloud is completely normal, and computers decompress most ordinary workloads in realtime.
In addition to our inadvertent hoarding, there are a couple of factors driving the adoption of compression specifically in the media and entertainment industry today:
IT’S COST EFFECTIVE
The costs of networking and storage capacity have decreased, but not as quickly as the cost of computational power, so the case of compressing and decompressing video frames in order to get realtime 4K, 8K, and 12K resolutions makes economic sense. Historically, traditional uncompressed digital workflows from SD to 2K have benefitted from sufficient bandwidth and capacity on the network and storage side by using commodity hardware. With the introduction of 4K resolution workflows, the increased pressure to deploy 40 gigabit Ethernet networks has been met with resistance from departments keen on maximizing existing infrastructure they just recently purchased. Since many productions have transitioned to 4K and are eyeing 8K and beyond, many creative workstation and render node systems have increased from around 8-cores in 2008 to 24-cores and beyond systems in 2016.
We use these cores effectively by leveraging new file formats that store the original lossless data in a compressed fashion – often saving 70 percent of storage footprint and network capacity. Additionally, it’s important to note that lossless compression is the key to ensuring content quality is not sacrificed throughout the creative process. The other side of the image compression coin is ‘lossy’ compression, which throws away image detail in order to shrink image sizes. In many creative workflows, it’s lossless or nothing.
THE MILLENNIAL MINDSET
Surprisingly, the average age of the creative members of a production team can also encourage an increase of compression technology adoption. For this example, let’s call historical technical directors and editors, special effects artists and colorists aged 40 or older, “craftsman builders.” This group is used to an expensive technical infrastructure enabling craftsmanship over quantity. Until recently, they have associated their work similar to handling linear film and tape, so these craftsman builders would take extra care and effort to do everything right the first time – mistakes and changes were perceived to be very expensive. This generation would often spend lots of time to anticipate the needs of the director and would only keep the version that captured the essence of what the director wanted.
Now let’s analyze the new bunch, the creative members of the production team aged 30 and under. This group has existed in an all-digital world, complete with large amounts of storage, compute and technical resources that are seemingly infinite and easily accessible. Their work approach differs considerably from the older, seemingly slower craftsman builders. They never had the limitation of expensive film cuts or limited technology time, so they rapidly create and iterate multiple versions with a similar outcome. Whereas the craftsmen have been taught to deliver exactly what the director has asked for, the younger artists are anticipating the possible different whims of the director and creating 10 different versions of a particular scene for him/her. If we didn’t effectively compress these workloads, productions might be hitting storage constraints much quicker than the old guard, so to speak! Compression has enabled a different way of doing work and provides greater opportunities for more creative experimentation. Technology innovation begets new and interesting processes that will drive further artistic experiences.
At this rate, uncompressed workflows will become increasingly obsolete. Adoption of higher resolution content that is already compressed will become the de facto standard, and production companies will increasingly look to drive down costs by using compression whenever possible. This applies to both active workloads and historic archival content.
Pixspan’s PXZ image file format is one of the most compelling formats that push this fact. They have been able to prove that simply adding their file format into Flame allows people to run native 4K lossless content over existing 10 gigabit Ethernet networks without having to copy data over. This is a great bonus: managing content in place on the network, lowering the throughput requirement and freeing up storage capacity for creative workflows. Now both generations can get what they want: quality and quantity, whew!
Combine this trend with the increase in global digital collaboration, and follow the sun architectures, it’s starting to make sense. You will see more support for lossless compression in diverse workflows, including Pixar RenderMan, Blackmagic Design DaVinci Resolve and Adobe Premiere, and will likely be using compression as the standard for collaboration rather than assuming the difficult task of managing exponentially growing uncompressed workflows that some people still fear today. Compression is the only way the industry will be able to scale up the higher resolution and deeper colors that are pushing the next generation.
We’ve all become digital hoarders, but the true test is managing our data more intelligently.
Ryan Sayre is the CTO at Large for Dell EMC.