{"id":633,"date":"2010-10-13T17:46:11","date_gmt":"2010-10-13T16:46:11","guid":{"rendered":"http:\/\/vmdamentals.com\/?p=633"},"modified":"2011-12-29T11:41:19","modified_gmt":"2011-12-29T10:41:19","slug":"vmotion-fails-with-source-detected-that-destination-failed-to-resume","status":"publish","type":"post","link":"https:\/\/www.vmdamentals.com\/?p=633","title":{"rendered":"VMotion fails with “Source detected that destination failed to resume”"},"content":{"rendered":"

I recently came across a problem where VMotions would fail at 82%. The failure would be “A general system error occurred: Source detected that destination failed to resume”. So what’s wrong here?<\/p><\/blockquote>\n




\nWhat’s wrong?<\/strong>
\nThe environment is based on NFS storage. Looking closer in the logs, I found that the VMotion process actually appears to try to VMotion a VM from one datastore to another datastore, which is kind of impossible unless we were doing a storage VMotion.<\/p>\n

Now for the weirder part: Looking at vCenter, all NFS mounts appear to be mounted in exactly the same fashion. But looking at the NFS mounts through the command line revealed the problem: Using esxcfg-nas -l<\/em> the output of the nodes showed different mount paths! The trouble lies in the name used for the NFS storage device during the addition of the shares to ESX: IP address, shortname, FQDN, even using different capitalization causes the issue (!).
\n


\nIts Lab Time!<\/strong>
\nI’m not sure this is a bug or a feature within VMware. Fact is, that from vCenter I have found no way to spot this issue (NAS mounts do not show any device ID anywhere in the config). Funny, vCenter sees the differences in the NFS box name somehow on the various hosts, and changes all<\/strong><\/em> NAS box names to the one you most recently messed with in vCenter! This is very dirty indeed. <\/p>\n

Potentially you could spot this situation: If you perform a refresh on the storage of an ESX node (under configuration->storage), all other nodes will follow to display the mount name used by the recently refreshed host! If you then proceed to refresh another ESX host’s storage (which uses a different name to mount to the NFS box), the mount names of all other ESX nodes in the cluster will change again to the name the host last refreshed has as a mount name! Only when you use the commandline (vdf -h<\/em> or esxcfg-nas -l<\/em> you can spot the difference more easily. <\/p>\n

In vCenter:<\/p>\n

\"vCenter<\/p>\n

As you can see, both of my homelab hosts are happily connected to exactly the same nas device, namely “xhd-nas01” (by the way ‘xhd’ comes from http:\/\/.www.xhd.nl<\/a>, my other hobby \ud83d\ude42 and yes, my two ESX nodes are called “Terrance” and “Phillip” \ud83d\ude42 ).<\/p>\n

But when looking from the console, I see this:<\/p>\n

[root@Terrance ~]# esxcfg-nas -l
\nnas01_nfs is \/nfs\/nas01_nfs from xhd-nas01<\/font><\/strong> mounted<\/code><\/p>\n

[root@Phillip ~]# esxcfg-nas -l
\nnas01_nfs is \/nfs\/nas01_nfs from XHD-NAS01<\/font><\/strong> mounted
\n<\/code>\n<\/p><\/blockquote>\n

As you can see, I used the shortname for the NAS, but I used capitals for one of the mounts. This in turn is enough to change the device ID of the NAS share:<\/p>\n

[root@Phillip ~]# vdf -h \/vmfs\/volumes\/nas01_nfs\/
\nFilesystem Size Used Avail Use% Mounted on
\n\/vmfs\/volumes\/eeb146ca-55f664e9<\/font><\/strong>
\n922G 780G 141G 84% \/vmfs\/volumes\/nas01_nfs
\n<\/code><\/p>\n

[root@Terrance ~]# vdf -h \/vmfs\/volumes\/nas01_nfs\/
\nFilesystem Size Used Avail Use% Mounted on
\n\/vmfs\/volumes\/22b2b3d6-c1bff8b0<\/font><\/strong>
\n922G 780G 141G 84% \/vmfs\/volumes\/nas01_nfs
\n<\/code>\n<\/p><\/blockquote>\n

So in fact, the Device IDs really ARE different! Everything works, right until you try to perform a VMotion. Then you encounter this error:<\/p>\n

\"vCenter
\n


\nSo how to solve the problem?<\/strong>
\nIf you have the luxury of being able to shut your VMs: Shut them, remove the VMs from inventory, remove the NAS mounts from your hosts, then reconnect the NAS mounts (and this time using the exact same mounthost name!). After that, you can add the VMs back to your environment and start them again.<\/p>\n

If shutting your VMs is not that easy, you could consider creating new NAS mounts (or find some other storage space), and use storage VMotion to get your VMs off the impacted NAS mounts. Then remove and re-add your NFS stores and use storage VMotion to move the VMs back again.<\/p>\n

In my testlab I created an additional NFS mount, this time on both nodes using the same name (FQDN in this case):<\/p>\n

[root@Terrance ~]# esxcfg-nas -l
\nnas01_nfs is \/nfs\/nas01_nfs from xhd-nas01 mounted
\nnas01_nfs02 is \/nfs\/nas01_nfs02 from xhd-nas01.xhd.local<\/font> mounted<\/p>\n

[root@Phillip ~]# esxcfg-nas -l
\nnas01_nfs is \/nfs\/nas01_nfs from XHD-NAS01 mounted
\nnas01_nfs02 is \/nfs\/nas01_nfs02 from xhd-nas01.xhd.local<\/font> mounted<\/code>\n<\/p><\/blockquote>\n

And you can see this time the device IDs really ARE equal:<\/p>\n

[root@Terrance ~]# vdf -h \/vmfs\/volumes\/nas01_nfs*
\nFilesystem Size Used Avail Use% Mounted on
\n\/vmfs\/volumes\/22b2b3d6-c1bff8b0
\n 922G 780G 141G 84% \/vmfs\/volumes\/nas01_nfs
\n\/vmfs\/volumes\/5f2b773f-ab469de6<\/font>
\n 922G 780G 141G 84% \/vmfs\/volumes\/nas01_nfs02<\/p>\n

[root@Phillip ~]# vdf -h \/vmfs\/volumes\/nas01_nfs*
\nFilesystem Size Used Avail Use% Mounted on
\n\/vmfs\/volumes\/eeb146ca-55f664e9
\n 922G 780G 141G 84% \/vmfs\/volumes\/nas01_nfs
\n\/vmfs\/volumes\/5f2b773f-ab469de6<\/font>
\n 922G 780G 141G 84% \/vmfs\/volumes\/nas01_nfs02<\/code>\n<\/p><\/blockquote>\n

After a storage VMotion to this new NFS store the VM VMotioned without issue… Problem solved!<\/p>\n","protected":false},"excerpt":{"rendered":"

I recently came across a problem where VMotions would fail at 82%. The failure would be “A general system error occurred: Source detected that destination failed to resume”. It turned out to be related to NFS mounts and the names used for reaching the NFS box…<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-633","post","type-post","status-publish","format-standard","hentry","category-vmware"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.vmdamentals.com\/index.php?rest_route=\/wp\/v2\/posts\/633","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.vmdamentals.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.vmdamentals.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.vmdamentals.com\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.vmdamentals.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=633"}],"version-history":[{"count":43,"href":"https:\/\/www.vmdamentals.com\/index.php?rest_route=\/wp\/v2\/posts\/633\/revisions"}],"predecessor-version":[{"id":3578,"href":"https:\/\/www.vmdamentals.com\/index.php?rest_route=\/wp\/v2\/posts\/633\/revisions\/3578"}],"wp:attachment":[{"href":"https:\/\/www.vmdamentals.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=633"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.vmdamentals.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=633"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.vmdamentals.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=633"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}