At the 2013 HPA Retreat Andrew Setos and Jim DeFilippis gave some fascinating (or frightening) numbers indicating the growing size of video production projects. As a result of the increase in digital production, higher frame rates in content capture and the move to raw content resolutions at 4K and even higher the industry faces challenges on storage challenges, archiving the production data and dealing with the proliferation (even anarchy) in file formats.
DeFilippis and Setos at the 2013 HPA Retreat, photo by Steve Weiner
According to the presentation a TV show in HD generates about 30 hours of media, which at 200 Mb/s results in 2.7 TB of content. Over a season this generated about 60 TB. If the content is in 3D the storage capacity requirements double. If 3D content is 4K rather than 2K (HD) increases the storage capacity by 4X (500 GB)!
Movies in DCI format generate about 1-2 PB per title. With 3D this increases by more than 2X to 5-6 TB. Going to higher frame rates (e.g. 48 or 60 fps) can double or triple those numbers to 10-15 TB. Also remember that modern cinema camera operate up to 120 fps so these numbers could climb even higher.
Each new on-line distributor requires their own file formats, usually at 1-4 Mb/s. For a 1,000 hour library of active content this represents 0.5 to 2 PB for each distribution deal. As the resolution of the content increases these numbers will increase as well.
The presenters projected the estimated growth of TV and movie file generation out to 2016. By 2016 they estimate that both TV and Movie file requirements could total close to 1 Exabyte each. This will be over an 18X increase from 2012 to 2016. Such large file requirements will drive significant demand for active file storage, content archiving and finding and using this increasing tsunami of content.
Storing this volume of storage will have implications for the growth of many different storage technologies including hard disk drives, magnetic tape, optical discs, flash memory and storage in the cloud (which is actually data stored in large data centers on various storage technologies).
The rapid developments in all types of storage technology make for a dynamic market where the best solution for a particular digital storage or archiving problem will change and evolve rapidly over time. Setos and DeFilippis give a good summary comparison of the available storage technologies and where they fit in media storage markets. These technologies included solid state memory, HDDs, optical discs, magnetic data tape and the Cloud.
There are many efforts inside and outside the media and entertainment industry to deal with long-term content preservation. Some of the questions to ask is whether there is a true archival storage for digital content akin to stone tablet for analog content. Even if we create media that can last 100 or even 1,000 years it will probably be obsolete in terms of storage capacity or device format and interfaces within 50 years and probably even less time. For modern digital preservation archives migration needs to be built into any true content preservation plans. In many cases archives are active systems that are regularly tapped to access long tail content. This trend will likely drive the adoption of these more active archives at the expense of traditional store and forget passive archives.
The presenters provided some insight into the plethora of digital formats used in various aspects of production, post-production and content distribution.. It is difficult to imagine having only one common file format but it may be possible to minimize the number of total file formats and to create links between these formats making it easier to find content between formats and converting content from one format to another.
Digital video projects are increasing in the amount of content shot for a project as well as the resolution and frame rate of content capture. This is swelling the entire volume of content for video production, post-production processing and distributing and monetizing this content. While content storage for active projects increases the total storage capacity of on-going archival storage will experience a true data tsunami that must be dealt with by developing storage technology as well as increasing storage efficiencies offered, for example, in cloud storage. In addition to larger overall storage and archiving requirements handling such as huge volume of files and such large files requires that our video (and audio) formats from content acquisition through post-production and distribution need to do a better job or allowing the flow of content from one stage to another and finally into archiving and even to support the reverse flow for finding and using long tail archived content.