Those new 4TB drives that are coming, there's more than just one more Terabyte to be excited about.
We've all watched over the years as storage and memory capacities have continued to grow in conjunction with Moore's law. It seems true that Intel and other manufacturers have figured out how to overcome barrier after physical barrier to double the number of transistors on a chip (or double storage density on a platter) and continue to overwhelm the industry with more space, cores and storage than we know what to do with! (I myself date back to the age of full length, full height, 25pin SCSI devices that didn't quite get to 1GB. I also got to repair more than a few of those washing machine sized "tubs" that you would lower spindles into with a handle and a few reel to reel tape devices as well.)
So what's all the excitement about these new 4TB drives? It's not really very much of an increase now is it?
What's exciting to Small Tree is that the vendors are now implementing "Advanced Format." This is a new drive standard that changes how data is stored on the platter. On previous devices - going back even to my storage "tub" days - data was stored on the platter in 512byte increments. You could "not" write data to the device in chunks smaller than 512bytes. That was the size of a sector. Each sector required a Disk Address Mark, some ECC or error correction bits and a gap to separate it from the next sector. Much like TCP and the idea of jumbo frames, the more things you have to look up and decode, the longer and more cpu intensive it is.
The new Advanced Format implemented in Hitachi's 4TB drives moves the minimum sector size to 4K. This allows for fewer DAM's to look up, less overall gap space and fewer ECC decodes to run. With this change, the smallest size a file can theoretically be is 4k. Given the larger size of files today and the relative size of the drives by comparison, this will hardly matter.
So, will this new format be supported by RAID card vendors? Most definitely yes. However, they have not all jumped on the bandwagon just yet. For those that have not, these new drives implement a spec called "512e." This allows the drives to continue to accept, write and read requests for smaller increments to maintain compatibility. These requests are handled by doing a "read-modify-write" cycle to the drive. To write 512bytes, the 4K sector containing that data is read in, the 512bytes is modified within the sector, and then the entire sector is written back out - RAIDS have used this technique for a long time to write less than a full stripe.
Overall, we've seen this technology offer a 33% increase in the number of streams Small Tree can support from a single storage array. We expect to see continued improvements as vendors begin to adopt the technology and hone their OS tuning to take advantage.
Steve Modica is CTO of Small Tree