Storage: Scale Out vs. Scale Up

Media formats grow larger and more complex with every incremental upgrade release of a camera or application. What options exist for production companies when it comes to getting work done, given this amount of change? Post production environments must invest in platform upgrades just to keep pace with the surge in media formats, giving proof to Moore’s Law of computing power that doubles every two years, just so we can take on these new projects, formats and creative capabilities. Over the past 10 years, the number of new media formats has increased to the point that the sheer variety of data types challenge everyone trying to keep up with change.

Each step our industry takes to support new filters and looks, effects and storylines has also placed pressure on the facility to do more with what it already has. Any system admin with a history in this business can look at what they’ve managed over time and the most painful point of the business has been a migration event — the time when older technology could no longer be upgraded enough to keep up with new formats and performance requirements.

When computers get faster and post facilities replace the old with the new, it can be described as scale-up. This means that portions of individual components improve (scale-up), but the next performance bottleneck is likely to be exposed. Ultimately, migrations are a result of investments in scale-up products such as computers and storage — the very core components of modern digital production. These technologies can only be improved along one or two vectors of their design paradigm, such as CPU performance or disk capacity. At a certain point they must be either replaced or work alongside a newly-deployed, updated technology, since the original equipment lacks the ability to scale in all the necessary areas and stay in use as the primary production platform.

In the shared-access storage market, the three common scale vectors are throughput, capacity and parallel access. Most storage in the market is designed to scale-up one or two of these three vectors. A typical example occurs when expanding a SAN with more disks/arrays, resulting in more capacity and more throughput. However, the improvements are generally only experienced if the number of storage clients stays the same — the storage is scaled-up to enhance the experience of an existing number of users.

A scale-out storage platform is engineered to support all three vectors of scale — not only throughput and capacity, but the complete end-to-end scale of adding more users and more powerful applications and media management systems that insist on touching more assets and providing status and availability at the same time.

The constant challenge with business growth is taking on new productions while delivering existing ones — a scale-out scenario. The addition of more users drives organizational migration events, since most shared-access storage struggles to support greater amounts of parallel user access compared to the initial design specification. Add an asset management system or two, and everyone will agree that overall storage I/O is not only about high performance — it’s about more aggregate performance for more users accessing more files in parallel than ever before.

To scale-out the metadata portion of storage requires adding compute within the storage architecture in order to handle the increase in parallel file access. Most scale-up storage has a controller server acting as the traffic cop, with a backup controller server acting as the failover. High availability is achieved with this traditional design, but surprisingly, most storage vendors have attempted little when it comes to scaling the compute within the controller part of the architecture. This is the concept of clustering, and it’s embraced in the virtualization market when running an application across multiple computers at the same time. As certain storage vendors add scale-out via clustering to traditional scale-up performance and throughput design that’s critical to realtime workflows, such as video editing and streaming playback, digital production environments can tap into modern scale-out storage that expands easily while also addressing all three of the scale vectors.

Avoiding the pain of data migration is a clear goal that should be expected from storage technology, and scale-out offers a way to overcome this challenge. A modern scale-out architecture can avoid the painful scheduled outages due to data migrations in addition to adding capacity on the fly to meet ever-increasing production needs. Maintaining availability of the data while it is migrated from older-generation platforms to newer ones is a core tenet of scale-out storage, and a final benefit to scaling out all three vectors of shared storage architecture.