Back to Blog

Yuri Rabover

The Challenge of Efficiency in Storage Management: IOPS

Keanu Reeves trying to calculate how many IOPS in The MatrixStorage Efficiency: How many IOPS are in Terabyte?

Earlier we looked at the challenges on maintaining efficiency in compute fabric related to standard building blocks of workload and servers. The more standardization you have on both sides the easier is to plan and control. But in real life the workload doesn’t look that uniform and the underlying hardware also doesn’t seem to originate from the same mold.

Storage management - maximizing the efficiency of storage - has its own specifics. On one hand, it is also tempting to build it from the standard components and then increase efficiency by sharing it as much as possible across all pieces of the workloads. But in storage the efficiency has 2 sides: performance related and space related and these sides behave quite differently.

In traditional storage based on hard disk drives rotational latency and the I/O blender effect pose a serious limit how much sharing you can accomplish before you start experiencing visible performance degradation. So the efficiency is quite often sacrificed to performance guarantees by partitioning the storage into smaller LUNs. However, the space handling adds another dimension. Even if you limited the number of concurrent loads to a small number, these loads can consume space in different patterns. Even if a VM uses little IOPS it could consume huge amount of disk space or introduce massive block changes which may cause reduced deduplication or compression ratios or cause snapshot space overflow. So you could be not only less efficient on IOPS capacity but may lose space efficiency. So the so called density may become even more evasive than in compute: space density and performance density could be quite different. Then how would you define efficiency? Some storage vendors try to come up with fancy metrics of the amount of IOPS per terabyte which is even more difficult to compute.

Storage technology is experiencing tremendous changes, the traditional drives are rapidly becoming a commodity and now you can build cheap SANs by using abundant local disk drives instead of expensive disk arrays. On one hand this could seriously improve storage efficiency but then it will create another challenge in maintaining performance in that environment. The total amount of storage you need didn’t change, it is now distributed across hundreds of rack mounted servers connected by network fabric. You no longer have a central place where you keep all your Vms, instead you need to solve even a bigger puzzle to in placing them closer to the compute nodes they run on but also making sure you have enough space and enough network bandwidth. Computing efficiency there will likely become much more complex.

Another storage advance feature is rapid progress with solid stated drives (SSD) based on flash memory. Many performance related problems just disappear – you could get hundreds of thousand of IOPS with sub-millisecond latency. But space efficiency of the underlying SSD drives is very important – flash memory is still expensive and many flash vendors implement their own techniques to increase SSD density. It is now hidden from you but you still need to know how much load to place where and again, if you don’t know workload demand, maintaining efficiency is becoming a bigger challenge.

As in compute fabric, it is practically impossible to maintain efficiency without controlling performance and the other way around. And as storage becomes more distributed other technology silos start impacting these decisions. So you need to be workload demand aware across all the silos. Do you have a solution for that?

Image source: Keanu Reeves trying to calculate how many IOPS in the IT classic, The Matrix