Everything worth keeping, ends up in a digital archive. As a consequence, there is more content in archives than in any other part of the media and entertainment workflow. In this article we will talk about trends in storage for archiving and digital preservation based upon the 2019 Digital Storage in Media and Entertainment Report and the 2019 IBC in Amsterdam.
Traditional storage capacity for archiving are removable media such as tape and optical discs that can be put on a shelf or in an automated library system until needed. However, there is a growing trend to use hard disks for long-term storage because access to the content is faster and because these devices are increasing in storage capacity.
As our 2019 Digital Storage for Media Professionals surveys show (see below), the highest percentage of true archived content is kept on hard disk drives and digital magnetic tape (see the pie chart below). Hard disk drives are becoming dominant for archiving, particularly for smaller facilities. If we include local network storage (mostly HDD-based) with external HDDs the total HDD archive storage is probably over 60%.
HDD Archives
There were a number of HDD archiving products on display at the 2019 IBC including products from large companies as well as smaller companies (many based in Europe). Let’s look at some of these products.
The figure below shows Dell media data center products, including their MAM and HDD-based archiving products.
At the Toshiba booth there were a few European storage startup companies. One was selling their Datainer modular interlocking units for stacking inactive HDDs (either individual drives or drives that are part of a RAID set. While this is not a recommended method for a long-term archive—or at least not the best repository for the only copy of your valuable content, it may be a useful and inexpensive way for a small facility to keep inactive content that will be used again in the near future. Next to the stack of Datainer HDD containers is a QSAN XCubeNAS Network Attached Storage (NAS) box. Of course, all of these storage devices featured Toshiba HDDs.
Disk Archive is a company out of the UK that is offering hard disk drive-based archive products. They have been at several IBC and even NAB shows. At the 2019 IBC they were showing their ALTO ARX product, advertising a fully functional 200 TB product for $19,900 in a 4U box, as shown below.
The ALTO ARX keeps 2 copies of each piece of data, uses Ethernet connectivity and only turns HDDs on when data is accessed to save power. They say the HDDs are back on line within 30 seconds. The company says that they run their HDDs for 30 days to eliminate infant mortality before putting these HDDs in their systems, to increase system reliability and longevity. Alto says that a disk removed from any ALTO will be recognized by any other ALTO and a disk can be opened independently with any standard computer.
A company called RNT (Rausch Netzwerktechnik ) out of Germany was showing their storage systems, including the BigFoot XXLarge shown below.
The BigFoot Storage XXL, based on Open-E JovianDSS, offers high performing soft- ware-defined storage. BigFoot Storage XXL of- fers 6 SSDs and 48 disk spinners in a single chassis, which gives you more bandwidth and IOPS. RAM can be expanded up to 512 GB. Customers can reach higher capacities and even further increase speed through caching. Among the applications advertised are large storage pools, private clouds and backup.
Magnetic Tape
Digital tape cartridges are available today with native storage capacities as high as 20 TB (IBM 3592 digital tape version introduced in 2019). The most popular magnetic tape format is the LTO format (83% of all tape used for archiving according to our 2019 M&E professional survey). Generation 8 LTO has a native cartridgecapacity of 12 TB.
Spectra Logic introduced StorCycle, its storage management software solution designed for data-driven organizations that need a modern storage lifecycle management workflow. StorCycle allows organizations to create a new Perpetual Tier of storage, reducing the overall cost of storing data by up to 70 percent, while giving users full access to their data. StorCycle can be implemented as standalone software using public cloud storage or existing network-attached storage, or combined with Spectra Logic storage hardware to create a complete storage solution.
The image below shows how StorCycle helps bring data to an archiving (perpetual storage) environment.
With StorCycle’s unique Project Archive feature, users can tag and move entire project data sets to a Perpetual Storage Tier. This allows the complete project to be accessed for further analysis, categorization, and comparison, all while being securely preserved for as long as needed. This makes StorCycle ideal for preserving data in fields such as computational and seismic research, oil and gas studies, semiconductor designs, genomics, media and entertainment, weather forecasting, autonomous vehicle research, and other fields where large amounts of machine-generated data are created.
Various companies were showing their LTO tape products—including at the LTO Program booth at the 2019 IBC.
Below is an HPE branded LTO drive and LTO-7 cartridge as well as a Network LTO Appliance from gblabs.
A company called Metus (out of Turkey) was selling an ArchiveBox that combined a Media Asset Management (MAM) system with a 1RU, one drive LTO-8 archive system with an 84 TB library capacity and unlimited off-line capacity. The company says that its UI provides web access to assets on tape. The content can be viewed as a proxy.
IBM storage was showcasing its media solutions at the 2019 IBC. IBM is a maker of magnetic tape solutions, both for its Enterprise Tape products and the LTO format. IBM magnetic tape solutions were on display as well as their flash to tape Flape solutions.
IBM was also showing a new media manager that can search metadata from multiple data sources using their Watson AI applied to media content. The company said that it displayed how automatic metadata tagging of video content can fuel faster search and time to air, as well as make content more engaging for viewers. In fact, the company said that AI-powered automation makes content creation 50%-70% faster than before, freeing up time for in-house video teams to work on more creative and complex projects.
The Sony booth at the 2019 IBC included their optical Disc Archiving System, but storage capacity of the shipping discs are still only 300 GB (although they show 500 GB discs.
Cloud Archives
Archives can be offline, as when digital tape, optical disks or hard disk drives are placed on a shelf in a controlled environment, or near-line on a networked library or storage array where they can be accessed more readily. In addition, a networked library may be available to remote users. Content stored in a connected library available remotely is said to be in the “cloud.” This cloud storage can be owned and controlled by the content owner, and is then a private cloud, or this content could be stored on storage resources owned and managed by others, and is then a public cloud.
There are many permutations of cloud storage that include private and public cloud resources and there are several offerings that combine local near-line storage with some type of cloud storage. In general cloud storage is often either a form of near-line storage or object storage.
The figure below shows the 2019 report projections for cloud archive storage (near-line, object and off-line) versus local archives
Xendata partnered with Wasabi at the IBC. While showing their onsite magnetic tape and optical disc archive solutions they were focusing on their Cloud Storage Solutions. The company says they can offer cloud computing without having to modify existing file-based applications. As shown in the figure below, virtual machines access cloud object storage accounts by running XenData Cloud File Gateway (CFG) software. Each object is presented as a file in the file system, allowing you to run your existing file-based applications in the cloud. The CFG manages caching to VM disks, overcoming object storage performance limitations.
StorageDNA was showing their Free Storage Decision Tools for calculating the cost of on-premise and in the cloud data storage. This tool helps to make the best decision on which cloud storage service to use.
Iron Mountain and Qumulo joined the Active Archive Alliance just before the 2019 IBC. Iron Mountain also said that it completed the development of nearly four megawatts of turn-key data center capacity in Amsterdam and London, increasing the company’s presence in Europe. According to the company the data centers are powered by 1000% renewable energy.
Object Matrix (from Wales) launched the latest version of its media focused object storage solution, MatrixStore. MatrixStore 4.1 features compatibility with more hardware and operating systems, as well as updates to MatrixStore Vision, web admin and monitoring. MatrixStore is a media focused private and hybrid cloud solution that is built on object storage technology from Object Matrix. It enables media companies to focus on activities that bring value rather than managing legacy storage platforms.
The latest version includes the company’s recently launched monitoring and analytics tool, Sense. It enables the collection of real-time statistics, real-time monitoring of hardware and analytics over all storage systems. Sense can work stand-alone or with alternative dashboards to view and analyse how media assets are being used.
MatrixStore 4.1 also includes certification of the latest Supermicro platform based on scalable processors, as well as certification of the Dell storage server. Already compatible with Cisco’s storage server, customers can now benefit from flexible hardware costings, different hardware support levels and a variety of customizable DELL EMC equipment.
Wrap Up
Archiving is an important element in all professional media and entertainment workflows. But before data can be digitally archived it has to be in a digital format. There were surprisingly large number of digital conversion systems and services on display at the IBC. The image below shows a system advertised to convert film content to 8K digital video.
Whether captured digitally or converted to digital formats, video content in modern archives are easier to find and monetize than ever before. As a result, archiving is more than just store and forget. Active archiving, where content is frequently accessed, is playing an important role in modern archiving and multiple types of storage devices are used in archiving both the data and analyzing that data to generate metadata making it easier to manage and access that data.
The 2019 Digital Storage in Media and Entertainment Report
Many of the figures in this article are from our most recent report on the state of digital storage in the Media and Entertainment industry. The 2019 Digital Storage for Media and Entertainment Report, from Coughlin Associates, provides 253 pages of in-depth analysis of the role of digital storage in all aspects of professional media and entertainment. Projections are given out to 2024 for digital storage demand in content capture, post-production, content distribution and content archiving are provided in 63 tables and 128 figures.
The fifteenth annual report includes results from a 2019 survey of Coughlin Associates, Digital Production Buzz, HPA and SMPTE members on their digital storage needs in these target segments (comparing the results to similar 2009, 2010 and 2012-2019 surveys). These surveys were used to refine the current report analysis from previous editions and track industry trends.
About the Author
Tom Coughlin, President, Coughlin Associates is a digital storage analyst as well as business and technology consultant. He has over 37 years in the data storage industry with multiple corporate engineering and management positions.
Dr. Coughlin has many publications and six patents to his credit. Tom is also the author of Digital Storage in Consumer Electronics: The Essential Guide, which is in its second edition with Springer. Coughlin Associates provides market and technology analysis as well as Data Storage Technical and Business Consulting services. Tom publishes the Digital Storage Technology Newsletter, the Media and Entertainment Storage Report, the Emerging Non-Volatile Memory Report and other industry reports. Tom is also a regular contributor on digital storage for Forbes.com and other blogs.
Tom is active with SMPTE, SNIA, the IEEE (he is President of IEEE-USA and active in the Consumer Electronics Society where he is chairman of the Future Directions Committee) and other professional organizations. Tom is the founder and organizer of the Storage Visions Conferences (www.storagevisions.com as well as the Creative Storage Conferences (www.creativestorage.org). He was the general chairman of the annual Flash Memory Summit, the world’s largest independent storage event, for 10 years. He is a Fellow of the IEEE and a member of the Consultants Network of Silicon Valley (CNSV). For more information on Tom Coughlin and his publications and activities go to www.tomcoughlin.com.