Back to Blog

Yuri Rabover

Storage Management Challenges created by Datacenter Heterogenity

Matt Damon struggling with a certain kind of storage management in ContagionWhat’s in the store for me?

Let’s continue examining importance of heterogeneity in the modern datacenter. This time we will look into storage. When you plan the datacenter from scratch it is fairly rare you start with multiple vendors. It is so much easier to deal with a reliable and reputable big storage vendor like EMC or IBM. Many companies started with them and have been pretty happy. But let’s look at some details of these decisions to see how effective they are.

In traditional storage environment it is quite common to set up a storage area network, (SAN) based on fiber channel connectivity (FC). FC setup requires FC switches, FC arrays and FC HBAs in every host. It is practically a real network, just based on a different technology than your IP network. And at the time when IP networks could not deliver multi-gigabit bandwidth, FC SANs were the only game in town for high-performing storage.

Now 10-40GB networks are a common and popular technology. And very often you can run your storage entirely over IP networks. Instead of FC SCSI you can switch to iSCSI and then you won’t need any expensive HBA or FC switches, you could use the standard NIC cards and l2/l3 switches you would need anyway. And there are many vendors which ship storage devices supporting iSCSI. In addition to iSCSI you can also use file storage like NFS or CIFS which also runs on top of IP but uses network file system protocols. It is somewhat easier to manage but it is a different technology and requires special support in storage devices.

Different storage vendors reacted to such changes differently. Some vendors like NetApp have been traditionally strong with networked storage and provide excellent support for iSCSI and NFS arrays; whereas EMC has been slower and often added this support to their devices as an afterthought. In VNX file and block storage are 2 entirely different subsystems, managed completely differently.

So now you may face a storage management challenge: you want to move to the new, more efficient technology but your existing storage infrastructure isn’t going to go away overnight, the amortization schedule can be several years. During these years you may have to use devices from different vendors. Moreover, your existing vendors may start upgrading your devices to support the new technology which will just make your life more complicated.

So the fact of life is that heterogeneity in storage is reality. It means that you should be able to make a decision how much storage capacity from every storage vendor to use and to buy. Which load to place where. Which load to move across if demand changes. Devices from different vendors provide various space and IOPS capacity as well as mobility. The decision making space is large and complex and if you look at management software vendors you rarely find solutions which control heterogeneous storage. Very likely you will find “plugins” for different storage vendors that show you how much space and IOPS you use and alert you when you experience a bottleneck in a particular vendor's storage array.

But you need a solution which will allow you to control your load across all this to ensure performance is adequate and you are not locked in into a single vendor because of your maintenance contracts. Let’s see if you have a solution for that.

Image Source: Matt Damon struggling with a certain kind of storage management in Contagion.