VMdamentals.com

Thanks to Veeam’s Happy Holidays gift, I now have a license for several Veeam products. The one I really wanted to try in my home lab was Veeam Backup and Replication.

In this blogpost, I will try various ways to connect the Veeam appliance to my Iomega IX2-200 NAS box. This setup is very tiny indeed, but it clearly shows the options you have and how they perform compared to each other.


Ways of connecting to the IX2-200

I could come up with four different ways of connecting a Windows VM (where Veeam runs) to an IX2-200:

1. CIFS: Send backups directly to a CIFS share on the IX2-200 via the network;
2. NFS: Mount the IX2-200 to vSphere via NFS and create a virtual disk (vmdk) on the share;
3. iSCSI: Mount the IX2-200 to vSphere via iSCSI and create a virtual disk (vmdk) on the share;
4. local-iSCSI: Install the Microsoft iSCSI initiator within the VM and connect to the IX2-200 directly via the network.

Since I have always been a big fan of PHDvirtual Backup, I could start off by stating that Veeam needs a Windows VM (and license) to run, and PHDvirtual does not. I know a lot of people would love to see a “shootout” between these two products. That blogpost might come, but today I’ll just be setting up Veeam, checking out some of its features and tune the target storage.


Setting up the Veeam VM

I started off by creating a Windows 2003R2 x64 enterprise edition VM, with a single vCPU and 512MB of memory. I used a 10GB disk as the operating system disk, added a second 10GB disk for the Veeam cache and even added a third virtual disk as a Windows swap disk. After patching of the server installation, I installed Veeam Backup & Replication 5.01 using the local SQL express database. I also installed the Veeam Enterprise manager on the same VM (you need to install IIS as well in this case) together with the Application Item Restore Wizards (because you can).

Initial configuration was remarkably easy. I added vCenter, created a backup job, selected an 8GB running test VM, and I started the job manually in virtual appliance mode (the mode where virtual disks are hot-added to the Veeam VM for backup purposes). I used the LAN target setting as the target I was backing up to was a LAN target. As this target, I selected my Iomega IX2-200 directly through a CIFS share (all on a Gbit link). The backup performed rendered me a backup speed of 3MB/s. Needless to say I was not overly impressed with it.


Finding the 3MB/s bottleneck

The bottleneck was not too hard to find. I was convinced the IX2-200 was able to do more. Networking was not saturated by far, even when I accidentally might have 100Mbit in the path. When looking at the Veeam VM, I noticed it was consuming 95% of CPU load during backup – Bottleneck found.

The Veeam appliance is doing it all – it reads from a virtual disk, calculates checksums, compares to its dedup database, and compresses and writes when needed. All of this is pretty CPU consuming, and a single core on my home lab servers just wasn’t going to cut it.


Some nasty theory on running a dual vCPU VM on a dualcore vSphere node

My home lab consists of two older dual core AMD X2 4000+ servers running ESX4.1 (not ESXi). Not the greatest performers. I was reluctant to add a second vCPU to the Veeam VM (not exactly the best practice on a dual core system), but I had to get CPU power from somewhere.

Since all other VMs on the ESX servers were very quiet anyway, I decided to try and give the Veeam VM a dual vCPU. I figured that the Veeam VM would be able to use all CPU power available minus the time consumed by the Service Console times two. This is because the Service Console uses core0, and during that time the Veeam VM cannot run due to its dual vCPU. The “gap” that is created this way on core1 can be filled with all other VMs. Nice for theory, but would it work?

After reconfiguing the Veeam VM with a second vCPU and a memory upgrade to 1GB, the first backup made with two vCPUs was still specified as “LAN target”. Now I ran another full backup of the test VM (I did not want to have any influence from CBT just yet). This time the backup speed settled at 7 [MB/sec], more than twice the throughput! This time the CPU usage settled around 60-80%. And yes, ready time boosts up to an average of 85[ms] per core. That is definitely a problem, but still I was pleased seeing a more than doubled backup rate.

Because I basically wanted to determine the most effective way of connecting to the backup target, I used this VM configuration as a basis. From there, I tried various ways of connecting the IX2-200 to the Veeam VM.


Connecting to the IX2-200 using vSphere iSCSI

I enabled iSCSI on the IX2-200, and connected a 50GB iSCSI volume to vSphere using the software iSCSI initiator. I then created a 49GB virtual disk (vmdk) and mounted it to the Veeam VM.

I ran a backup job with exaclty the same settings as before, but now to the virtual disk via vSphere iSCSI. This job uses just a tadd less CPU power, but it was hardly noticeable. Backup speed was a steady 8MB/sec,
some faster than the direct CIFS backup made previously. And this at a slightly lower CPU overhead!


Connecting to the IX2-200 using vSphere NFS

Next target type: NFS through vSphere. I connected a share from the IX2-200 via NFS to vSphere, and created a 50GB virtual disk (vmdk) on the NFS share. The 50GB virtual disk was added to the Veeam VM and used as a backup target. Once again the same job settings were used (just a different target type). To no surprise, this job performs just like the iSCSI job: a steady 8MB/sec with the exact same CPU usage pattern.


Connecting to the IX2-200 using MS iSCSI initiator

The fourth and final way to connect the IX2-200 to the Veeam VM is a direct ISCSI connection. In this option I use the Microsoft iSCSI initiator, which connects directly to the IX2-200. Using this target, Veeam manages to backup at a rate of 8MB/s, so nothing new here.

Finally a graph comparing the different options tested:

So which way of connecting is the fastest?

In the end, all four ways of connecting Veeam to an IX2-200 produced about the same backup speeds. So what to choose? Several things to consider:

– The Direct CIFS option is handy, because backup files land directly on the IX2-200. You can see and read them using Windows CIFS directly;
– Both iSCSI based options have one issue: You have to reserve the projected backup size directly on the IX2-200. No thin options here.
– When connecting through vSphere, you will always be looking at a vmfs and/or a vmdk file in which your backups reside. Not the biggest issue, because it is very simple to add the virtual disk to any vSphere environment. It can be a problem though when you plan to rsync your files off site; in this case the direct CIFS option might be better for you.

In my setup it does not really matter what I choose. This is because I use the IX2-200 solely as a backup target, and I always have vSphere hanging around somewhere 🙂


Choosing between the local-lan-wan target setting

So now on to tuning Veeam to get the most out of my setup. First thing to look at is the “storage – optimize for” option. I have consistently set this value to “LAN target” for all backup tests up to now (since my backup target was a LAN target). As Veeam states, these settings not only impact bandwidth used, but also CPU power consumed. And since my CPUs are not the fastest around, it might be smart to try various settings on this option.

First I tried to set this value to “local target”. CPU load remained the same, but the backup rate was slightly higher to the direct CIFS target at 8MB/s. This is probably due to the fact that the CPU was somewhat less loaded. Ready times were the same, but wait times of the vCPUS were just a tad lower, possibly enabling a slightly faster transfer rate. the resulting backup file was a a little larger (but nothing extreme- 4,396GB vs 4,411GB), possibly because in LAN-target mode the deduplication is less effective.

I also tested the “WAN target” setting. Backup speed this time was back to 7MB/s. I also performed a test by setting the compression to “best”. This sunk the performance down to 4MB/s, not something to try with too little CPU power as could have been expected.


Remaining bottlenecks – where to look next

So now we have seen that all ways of connecting to the IX2 delivers performance that is very comparable.
I think it is safe to assume that the limits I see here are a limitation of any of the unchanged factors. So it could be several things, like:

– Limitations of the CPU performance / high readytimes in my home lab;
– Limitations of the IX2-200;
– Limitations of the network to the IX2-200.

Not much I can do right now about the CPU or the IX2-200 itself. So maybe I need to take a look at the network that connects my vSphere environment to the IX2-200…? Check out part 2 of my blogpost on Veeam, where I configure jumbo frames and we’ll go and see if that helps!


Conclusion

The conclusion for now is, that Veeam is pretty CPU intensive. Don’t try to give the Windows server on which you run Veeam a single vCPU, at least try to add a second vCPU, even on a limited dual core system.

The way you connect to your target storage does not seem to matter that much – direct CIFS or iSCSI, NFS or iSCSI through VMware, it all worked pretty well and all about at the same speed.

4 Responses to “Veeam Backup part 1- Optimizing IX2-200 backup speeds”