Hulking storage, metadata hash, big pipes and Terabyte cameras

Posted By Tom Coughlin on November 02, 2009 12:00 am | Permalink
In general increasing storage capacity and performance as well as faster networks and processing power are enabling richer content and the long term storage of content. At the same time many media and entertainment professionals are not making sure that their content is being protected for the long term. Revenues expected for digital storage, including services for the professional media and entertainment market could exceed $10 B by 2014.

For more information on these trends please see the 2009 SMPTE conference paper, 2009 Survey of Digital Storage in Professional Media and Entertainment, by Thomas M. Coughlin. More information on the annual report on digital storage for professional media and entertainment can be found on the tech papers page of www.tomcoughlin.com.

In the same session, Oliver Morgan from Metaglue Corp. gave an interesting talk about using the MXF wrapper to contain content quality control data that he called “self-propelled data” using hashing algorithms to tie the metadata to the content essence that offers a scalable architecture within metadata for event tracks for time-dependent metadata as well as multi-track files. He said that this hashing could use the CRC-32 Castagnoli algorithm used by iSCSI. This algorithm requires low compute costs making it fairly easy to implement. The QC data can be used in AS02 and AS03 files and can include “Generation UID” that allows tracking who makes changes in the essence and when those changes are made. An interesting comment that he made is that there is an increasing need for partial restore of content, making segmented hashing very useful.

There was a entire session today with presentations focused on 3 Gbps infrastructure used in new production facilities. Christopher Bauer from ABC discussed the difficulties and opportunities of installing a fiber optic HD central switching facility supporting 3 Gbps data routing. He showed fascinating pictures of the differences including showing that a single fiber optic connection could replace up to 96 copper connections used previously. Other papers in this session discussed Harris’s 3 Gbps infrastructure to support 1080p and 3D production work. Another paper from Aspera discussed internet (cloud) based reliable content delivery for file-based transfers comparing commercially available as well as academic (laboratory) transport schemes in term of bandwidth utilization, network efficiency and transfer time. Yesterday I wrote of developments expected soon for 40 Gbps Ethernet transport—subjects for the 2010 SMPTE conference!

As in many such events it is possible to learn as much outside the sessions chatting with knowledgeable attendees as in the sessions themselves. In one of these conversations with a member of a well-known professional video camera manufacturer I found some interesting developments in data capture. These cameras run over $200k each, without lenses and there are maybe 300-400 units total worldwide. Most of these are rented as needed vs. owned by a facility. True 4k cameras are going to be available soon and 8k cameras may be available in the next few years to support Ultra-HD formats. The conversion to Ultra-HD may take about 10 years, which is significantly faster than the HD conversion due to the increased rate of change that current digital facilities offer. Perhaps the biggest issue in the introduction of new technology will be amortization of current investments. One interesting observation is that although currently digital tape is the most common high-end camera recording media, solid state storage devices with storage capacities as high as 1 TB and very fast data rates may soon be available making higher resolution content capture and off-loading even easier.

40 Gb Ethernet pipes in production facilities and 1 TB solid state cameras, it seems that the professional media and entertainment market will be making and using more content capacity than ever before. Storing, distributing, finding and using that content will demand increasing capabilities in digital storage. Looks like there are a lot of new factors to include in the next survey (and report) on digital storage in professional media and entertainment!