Snapshot Consolidation needed – Which with my luck… fails
As I am testing several third party backup tools, this morning I stumbled upon a failed backup. No snapshot present on the VM which could not be backed up – but a yellow mention in the VI client: “Configuration Issues – Virtual Machine disks consolidation is needed“. And my luck was, that selecting “consolidate” ended in that one brilliant error: Unable to access file
since it is locked . Great. Here’s what was wrong!
Root cause
So what is causing this “needs consolidation” warning? Looking at VMware KB Article 2003638, a need for consolidation is now visible (the need has always been there π ). When this warning pops up, the snapshot is already removed from the VM, but the snapshot files could not be consolidated back into the base disk.
How can you find out if any of your VMs needs snapshots to be consolidated? You can simply add the “consolidation” tab to the VI client, find any VMs that require consolidation and kick it off:
Of course, consolidating the snapshot did not work in my case. Lucky me, I got this error:
So why is this occuring? A little history: As I was testing a third party solution of “Brand X”, this error occured. I will not go into details which backup product this was though π
So the backup solution of Brand X was trying to backup a VM. A VMware-level snapshot was made, the base disk was hot-added to the VM that handles the backup. Then the backup was made.
After the backup was made however, the backup solution somehow did not manage to hot-remove the disks from the backup VM. This meant the base disks of the VM being backed up where still locked, hence the failure when trying to consolidate.
How it was fixed
The solution was, as it is most of the time, simple: I manually removed the base disks from the backup VM (without deleting them from disk of course!), then I retried the consolidation. This time: success!
All’s well that ends well!
Dude thank you. That error was making m mental!! I found you through Google and bingo my system is back. Mind you that this VM is my 5.1 vCenter server so that added a little more stress. The up side is it’s only a lab environment.
Thank you again and happy holidays!
Storage VMotion to another datastore. The delta files follow the directory, but now you can successfully Snapshot -> Consolidate, and the delta files will go away.
Note: We could not see the snapshots in the snapshot manager. Running ESXi 5.0u1.
Thank you for the storage vmotion idea. I had went through just about every vmware kb article on consolidation I could find, and nothing was solving this. That worked perfectly!
I have also had success by restarting the management services of the ESXi host in which the VM sits. Restarting the management service unlocks the file and allows the disk consolidation. To do this just access the CLI and type “services.sh restart” Once that is complete the consolidation should work.
services.sh restart from cli worked for me as well.
Thank you! This post saved me a lot of time!
Erik,
So you shut down the server first, manually removed the base disks (backup disks), ran the consolidate, added back the original server vmdk disk files and fired the server back up?
I wanted to make sure that is how you did it before I proceed.
Thanks!
Actually in my case there was no ESXi reboot required at all. The base disk was simply still connected to the backup proxy VM, and for that reason the base disk was locked aka the snapshot could not be consolidated (as the base disk was locked). Removing the base disk from the backup appliance VM was enough to remove the lock, after which consolidation was possible, and problem solved.
Can someone confirm the steps for me as I’ve been encountering this lately.
1. power down VM
2. remove virtual disks temporarily (which removes the lock I assume)
3. run consolidate
4. add the virtual disks back to the VM
5. Power on
Is this correct–anyone?
Clarification: I use Veeam B&R on a physical blade with direct SAN/datastore access (Fiber channel) so this backup server is NOT a VM nor does it mount the disks of the VMs to where I can ‘remove them’. This might happen under the hood but it doesn’t work the same way when it is a physical server. Yes, when my Veeam was a VM appliance mode system I could go to this server and see disks mounted from the VMs being backed up but not any more. After re-reading I realized you were saying remove the mounted disks from the backup system–not the VM itself. This may not solve my issue although it’s exactly what is happening. Nothing I have tried has worked and the solutions per VM affected have varied…it’s very strange. The only 100% solution was to restore from my Veeam backup but I can’t do this easily on every server especially DB servers under frequent changes. This might not be the solution for me but if anyone can confirm I would appreciate it….I know this post is old and may be abandoned at this point…cheers!
Ben’s solution of performing a storage vMotion (whether the machine be off or on), then running Consolidate worked perfectly. The machine’s snapshot files were all removed and the machine was using its original vmdk file once again.
In my case, a storage vMotion cleared up all issues. I didn’t even need to consolidate after the fact (vSphere 5.5).
Hi,
In my case vSphereDataProtection was holding the disk.
Thank you.
Best regards,
daco