VMdamentals.com

I do not often write to specific implementations of a vendor. This time however I focus on EMC’s FAST-cache technology, and we will be playing a little “follow the I/O” to see what it actually does, where it helps and where it might not.

Spoiler Alert

First of all: Spoiler alert! I work for EMC as a vSpecialist, so the upcoming post comes from an EMC vGeek. However, this blog is about deep dives on cool technology, any cool technology. FAST-cache is proving more and more to be able to cope with huge I/O loads and make best use of EFD drives (EFDs or Enterprise Flash Drives as EMC calls them are based on SLC technology where drives designated as SSDs are usually based on cheaper, slower and less reliable MLC technology). EMC uses flash technology exactly where it counts: Have them deliver I/Os but not the required capacity.


Why Solid State Drives matter

I’m sure you heard the story before: CPUs got faster and faster, drives did not. I won’t be repeating all the boring graphs on this. When Solid State Drives came around things changed a lot. SSDs are basically the opposite from spinning drives: Extremely fast seek times, but at less capacity. SSDs definitely have a use; it is just not efficient to run your entire workload from them in most cases.


“Solid State Drives are dirt-cheap”

Yes, it is true. Solid State Drives are dirt cheap, if you look at them the right way. Let me share the story on how I got an SSD inside my company laptop (and this is true story: Thanks Bertrand!!).

At VMworld in Copenhagen 2011 I joined breakfast with my bosses boss. He was considering getting a Mac book for his wife, and was thinking about an SSD inside. As the discussion went along, I asked him why my corporate Mac book pro hadn’t come with an SSD. The answer was simple: SSDs are expensive devices. So I looked at him as surprised as I could, and told him that wasn’t true at all:

“Look at it like this. An average hard drive in your laptop will cost you around 70 euros, that is about one euro for every I/O (as 7200rpm SATA drives deliver around 70 IOPS). Now look at an average SSD: It will cost you around 150 euros, and deliver around 5.000 IOPS (and that is very conservative). So an SSD will cost you around three cents for every I/O! Capacity does not make my laptop run smooth, performance does. So SSDs really are dirt cheap!”

He just smiled, and said he did not mind if I ordered an SSD. Yeah!

This clearly demonstrates that SSD is extremely cheap, with one big caveat: You need to draw IOPS from it, not capacity. How do you do that? Well, FAST-cache seems like a very powerful solution for this, so we’ll dig deeper into the technology.


The idea behind FAST-cache

I have said this often: “Cache is King”. And that really is true. What EMC has done with FAST-cache (when looking at it from a distance), is actually expanding the cache of the array with SSDs, for both reads and writes (and especially the writes is where it REALLY counts!).

You could state that SSDs are very much slower over DRAM cache. On the other hand, it is way faster than a spinning drive. DRAM cache is an example of even more IOPS at an even lower capacity, at a higher price. As SSD neatly fits between DRAM and spinning drives, it makes sense to deploy it there; and that is just what FAST-cache does.

The theory behind FAST-cache is not all that complex: Divide the array’s storage up in 64KB blocks, count the number of hits on any of those blocks, and create a cache page on the FAST-cache EFDs if there have been three hits (reads or writes) onto that block. In case FAST-cache fills up, the array starts to seek pages in these EFDs that will render a full stripe write to the spinning disks in the array, and force flush out to spinning disks. Why are full-stripe writes so cool? Read about it in “RAID5 DeepDive and Full-Stripe Nerdvana”


A closer look at FAST-cache

Looking closer at FAST-cache, it is actually not an extension to the DRAM cache; it is a layer that floats between the SAN’s DRAM cache and the spinning disks (right where you’d expect EFDs).

The really cool part is, that when you size FAST-cache properly and the workload is suited for FAST-cache, you may never need to force flush any data from FAST-cache, making the array really FLY. The main difference from most competitors? The arrays FLIES for both reads and WRITES. And especially the ability to cache writes is what makes FAST-cache so cool, especially in VDI environments.


Follow the I/O

Knowing the idea behind FAST-cache, we can now play a little game called “Follow the I/O!”. We will shoot some I/O’s to an imaginary array, and see where the I/O will go.

Let’s say we shoot a write to the array to block X. The write is captured in the DRAM cache (which is mirrored across controllers), so the array responds right back with a write acknowledge. At some point this write may be flushed to the spinning disks in the backend. The DRAM cache algorithm will also try to find full-stripe writes by the way.

But now we imagine that the same block is read twice after that initial write. As long as the write was still stored in DRAM cache, the array will return the block from DRAM cache. But something else happens: As the block was hit three times, the FAST-cache is now primed to cache this block into FAST-cache as well. If we need to flush the DRAM cache at some stage, we can flush the data from DRAM cache but it will still be available in FAST-cache, which is not as fast as the DRAM cache but still a factor 1000 faster over spinning disks. Mission accomplished.

And this is where the magic begins: As the DRAM cache is usually too small to serve all (or almost all) the I/Os, FAST-cache acts as an effective second level cache. At a factor 1000 of the speed of spinning disks, this is still a very acceptable speed. But it gets better: As we are able to size FAST-cache from 100GB up to several TBs (in the largest VNX array), we should be able to put in enough FAST-cache to deliver over 90% of the active data from FAST-cache. This will greatly reduce the overall response time of the array.


Three strikes OUT

The “three strikes” algorithm of FAST-cache is a pretty smart idea. If you’d be moving large amounts of data, the FAST-cache algorithm would not activate. And that is by design: Cache would not help in large copy actions.

But random hits on active blocks will eventually cache these blocks into FAST-cache. The 64KB granularity helps here. As most workloads perform I/Os below 64KB (unless you are copying large amounts of data), there is a significant chance that even though a workload performs 4KB reads and writes to different blocks, they’ll still hit the same 64KB FAST-cache block which will result in promotion into FAST-cache. Brilliant!


A prime example of FAST-cache effectiveness: VDI workloads

A very strong use case of FAST-cache is the latest Reference Archtiecture that both EMC and VMware put out:

EMC infrastructure for VMware View 5.0 Reference Architecture (EMC)
EMC infrastructure for VMware View 5.0 Proven Solutions Guide (EMC)
EMC infrastructure for VMware View 5.0 Reference Architecture (VMware)

I’ll just drop all of coolness on you right away: In this RA, EMC runs 1000 View desktops (consuming 9.8 IOPS each) on 15x 15K SAS drives with only TWO 100GB EFDs configured as FAST-cache in front of them.

Yes, that is ONE THOUSAND vDesktops out of 15 spinning disks and two small EFDs!


Looking closer, you could even get away with the same config running from 10x 10K SAS drives with two EFDs in front as the spinning disks were doing only 40 IOPS a piece (and you can get enough capacity out of the 10 drives). Awesome stuff, and it shows in the greatest detail how powerful FAST-cache really is. As Chad Sakac likes to say “It is face melting stuff!”.

In another post I have done a mathematical deepdive on how the VNX manages things under the VDI covers which you can read here: Backwards VDI math: Putting numbers to the 1000 user RA

One Response to “EMC FAST-cache and “Follow the I/O””