Alarme de latence Read ou Write pour datastore partagé NFS par cluster, yellow 300ms/5min et red 500ms/3min (mailto:admin@vmware.local) :

foreach ($vicluster in Get-View -ViewType ClusterComputeResource) {foreach ($dsname in ($vicluster.Datastore|%{Get-View $_}|?{$_.summary.Accessible -and $_.summary.MultipleHostAccess -and $_.info.nas})){if (!((Get-View AlarmManager).GetAlarm($vicluster.moref)|?{(get-view $_).info.name -match "Datastore $($dsname.name) datastore RW latency"})){(Get-View AlarmManager).CreateAlarm($vicluster.moref,(New-Object VMware.Vim.AlarmSpec -Property @{Name = "Datastore $($dsname.name) datastore RW latency in $($vicluster.name)";Description = "Custom alarm to monitor NFS $($dsname.name) ($($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)) read or write latency in $($vicluster.name)";Enabled = $true;expression = (New-Object VMware.Vim.OrAlarmExpression -Property @{expression = @((New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "datastore"}|?{$_.nameinfo.key -match "totalReadLatency"}|?{$_.RollupType -match "average"}).key;Instance = ($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)});Operator = "isAbove";Yellow = "300";YellowInterval = "300";Red = "500";RedInterval = "180";Type = "HostSystem"});(New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "datastore"}|?{$_.nameinfo.key -match "totalWriteLatency"}|?{$_.RollupType -match "average"}).key;Instance = ($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)});Operator = "isAbove";Yellow = "300";YellowInterval = "300";Red = "500";RedInterval = "180";Type = "HostSystem"}))});Action=(New-Object VMware.Vim.GroupAlarmAction -Property @{Action= (New-Object VMware.Vim.AlarmTriggeringAction -Property @{Action = (New-Object VMware.Vim.SendEmailAction -Property @{ToList = "admin@vmware.local";Subject = "[vAlarm] $($dsname.name) RW latency on {targetName}";CcList = "";Body = ""});TransitionSpecs = @((New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "green";FinalState = "yellow";Repeats = $false});(New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "yellow";FinalState = "red";Repeats = $false}))})});ActionFrequency = "1800"}))}}}

Alarme de latence Read ou Write pour datastore partagé VMFS par cluster, yellow 30ms/5min et red 40ms/3min (mailto:admin@vmware.local) :

foreach ($vicluster in Get-View -ViewType ClusterComputeResource) {foreach ($dsname in ($vicluster.Datastore|%{Get-View $_}|?{$_.summary.Accessible -and $_.summary.MultipleHostAccess -and $_.info.vmfs})){if (!((Get-View AlarmManager).GetAlarm($vicluster.moref)|?{(get-view $_).info.name -match "Datastore $($dsname.name) RW latency in $($vicluster.name)"})){(Get-View AlarmManager).CreateAlarm($vicluster.moref,(New-Object VMware.Vim.AlarmSpec -Property @{Name = "Datastore $($dsname.name) RW latency in $($vicluster.name)";Description = "Custom alarm to monitor VMFS $($dsname.name) ($($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)) read or write latency in $($vicluster.name)";Enabled = $true;expression = (New-Object VMware.Vim.OrAlarmExpression -Property @{expression = @((New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "datastore"}|?{$_.nameinfo.key -match "totalReadLatency"}|?{$_.RollupType -match "average"}).key;Instance = ($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)});Operator = "isAbove";Yellow = "30";YellowInterval = "300";Red = "40";RedInterval = "180";Type = "HostSystem"});(New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "datastore"}|?{$_.nameinfo.key -match "totalWriteLatency"}|?{$_.RollupType -match "average"}).key;Instance = ($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)});Operator = "isAbove";Yellow = "30";YellowInterval = "300";Red = "40";RedInterval = "180";Type = "HostSystem"}))});Action=(New-Object VMware.Vim.GroupAlarmAction -Property @{Action= (New-Object VMware.Vim.AlarmTriggeringAction -Property @{Action = (New-Object VMware.Vim.SendEmailAction -Property @{ToList = "admin@vmware.local";Subject = "[vAlarm] $($dsname.name) RW latency on {targetName}";CcList = "";Body = ""});TransitionSpecs = @((New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "green";FinalState = "yellow";Repeats = $false});(New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "yellow";FinalState = "red";Repeats = $false}))})});ActionFrequency = "1800"}))}}}

Alarme de latence SIOC Normalized Latency pour datastore partagé par cluster, yellow 30ms/5min et red 40ms/3min (mailto:admin@vmware.local) :

foreach ($vicluster in Get-View -ViewType ClusterComputeResource) {foreach ($dsname in ($vicluster.Datastore|%{Get-View $_}|?{$_.summary.Accessible -and $_.summary.MultipleHostAccess -and $_.IormConfiguration.Enabled})){if (!((Get-View AlarmManager).GetAlarm($vicluster.moref)|?{(get-view $_).info.name -match "Datastore $($dsname.name) SIOC Normalized Latency in $($vicluster.name)"})){(Get-View AlarmManager).CreateAlarm($vicluster.moref,(New-Object VMware.Vim.AlarmSpec -Property @{Name = "Datastore $($dsname.name) SIOC Normalized Latency in $($vicluster.name)";Description = "Custom alarm to monitor $($dsname.name) ($($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)) datastore SIOC Normalized Latency in $($vicluster.name)";Enabled = $true;expression = (New-Object VMware.Vim.OrAlarmExpression -Property @{expression = @((New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "datastore"}|?{$_.nameinfo.key -match "sizeNormalizedDatastoreLatency"}|?{$_.RollupType -match "average"}).key;Instance = ($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)});Operator = "isAbove";Yellow = "30000";YellowInterval = "300";Red = "40000";RedInterval = "180";Type = "HostSystem"}))});Action=(New-Object VMware.Vim.GroupAlarmAction -Property @{Action= (New-Object VMware.Vim.AlarmTriggeringAction -Property @{Action = (New-Object VMware.Vim.SendEmailAction -Property @{ToList = "admin@vmware.local";Subject = "[vAlarm] $($dsname.name) SIOC Normalized Latency on {targetName}";CcList = "";Body = ""});TransitionSpecs = @((New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "green";FinalState = "yellow";Repeats = $false});(New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "yellow";FinalState = "red";Repeats = $false}))})});ActionFrequency = "1800"}))}}}

Alarme de Disk Command aborts pour datastore VMFS partagé par cluster, en cas de défaillance/saturation DAS/SAN/iSCSI (mailto:admin@vmware.local) :

foreach ($vicluster in Get-View -ViewType ClusterComputeResource) {foreach ($diskname in ($vicluster.Datastore|%{Get-View $_}|?{$_.info.vmfs -and $_.summary.Accessible -and $_.summary.MultipleHostAccess}|%{$_.info.vmfs.extent}|%{$_.diskname})){if (!((Get-View AlarmManager).GetAlarm($vicluster.moref)|?{(get-view $_).info.name -match "Disk $diskname command aborts"})){(Get-View AlarmManager).CreateAlarm($vicluster.moref,(New-Object VMware.Vim.AlarmSpec -Property @{Name = "Disk $diskname command aborts in $($vicluster.name)";Description = "Custom alarm to monitor $diskname disk command aborts in $($vicluster.name)";Enabled = $true;expression = (New-Object VMware.Vim.OrAlarmExpression -Property @{expression = (New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "disk"}|?{$_.nameinfo.key -match "commandsAborted"}|?{$_.GroupInfo.Key -ne "virtualDisk"}|?{$_.RollupType -match "summation"}).key;Instance = $diskname});Operator = "isAbove";Yellow = "1";Red = "2";Type = "HostSystem"})});Action=(New-Object VMware.Vim.GroupAlarmAction -Property @{Action= (New-Object VMware.Vim.AlarmTriggeringAction -Property @{Action = (New-Object VMware.Vim.SendEmailAction -Property @{ToList = "admin@vmware.local";Subject = "[vAlarm] command aborts on disk $diskname on {targetName}";CcList = "";Body = ""});TransitionSpecs = @((New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "green";FinalState = "yellow";Repeats = $false});(New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "yellow";FinalState = "red";Repeats = $true}))})});ActionFrequency = "1800"}))}}}

Ceux qui, comme nous, espéraient la voir arriver avec vSphere 5 ont été déçu. L’alarme de latence de datastore n’existe pas encore, pas en GUI du moins. Et pour cause, les compteurs de latence ne sont disponibles qu’au travers des objets VirtualMachine ou HostSystem (et HostSystem uniquement pour les compteurs SIOC). Nous vous avions proposé une solution de contournement, en PowerCLI évidement, grâce au compteur maxTotalLatency mais il restait encore à trouver quel datastore avait déclenché l’alarme.

Aujourd’hui nous vous proposons une série de oneliner qui vont créer (si elles n’existent pas déjà, ce qui veut dire que vous pouvez les utiliser en tache planifiée) des alarmes pour chaque datastore (instance = UUID) qui se déclencheront si les seuils de latence fixés sont atteint sur un ESX. Attention ça pique.

Alarme de latence Read ou Write pour datastore partagé NFS, yellow 300ms/5min et red 500ms/3min (mailto:admin@vmware.local) :

foreach ($dsname in (Get-View -ViewType datastore|?{$_.summary.Accessible -and $_.summary.MultipleHostAccess -and $_.info.nas})){if (!((Get-View AlarmManager).GetAlarm((Get-Folder -NoRecursion |Get-View).moref)|?{(get-view $_).info.name -match "$($dsname.name) datastore RW latency"})){(Get-View AlarmManager).CreateAlarm((Get-Folder -NoRecursion |Get-View).MoRef,(New-Object VMware.Vim.AlarmSpec -Property @{Name = "$($dsname.name) datastore RW latency";Description = "Custom alarm to monitor NFS $($dsname.name) ($($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)) read or write latency";Enabled = $true;expression = (New-Object VMware.Vim.OrAlarmExpression -Property @{expression = @((New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "datastore"}|?{$_.nameinfo.key -match "totalReadLatency"}|?{$_.RollupType -match "average"}).key;Instance = ($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)});Operator = "isAbove";Yellow = "300";YellowInterval = "300";Red = "500";RedInterval = "180";Type = "HostSystem"});(New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "datastore"}|?{$_.nameinfo.key -match "totalWriteLatency"}|?{$_.RollupType -match "average"}).key;Instance = ($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)});Operator = "isAbove";Yellow = "300";YellowInterval = "300";Red = "500";RedInterval = "180";Type = "HostSystem"}))});Action=(New-Object VMware.Vim.GroupAlarmAction -Property @{Action= (New-Object VMware.Vim.AlarmTriggeringAction -Property @{Action = (New-Object VMware.Vim.SendEmailAction -Property @{ToList = "admin@vmware.local";Subject = "$($dsname.name) RW latency";CcList = "";Body = ""});TransitionSpecs = @((New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "green";FinalState = "yellow";Repeats = $false});(New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "yellow";FinalState = "red";Repeats = $false}))})});ActionFrequency = "1800"}))}}

Alarme de latence Read ou Write pour datastore partagé VMFS, yellow 30ms/5min et red 40ms/3min (mailto:admin@vmware.local) :

foreach ($dsname in (Get-View -ViewType datastore|?{$_.summary.Accessible -and $_.summary.MultipleHostAccess -and $_.info.vmfs})){if (!((Get-View AlarmManager).GetAlarm((Get-Folder -NoRecursion |Get-View).moref)|?{(get-view $_).info.name -match "$($dsname.name) datastore RW latency"})){(Get-View AlarmManager).CreateAlarm((Get-Folder -NoRecursion |Get-View).MoRef,(New-Object VMware.Vim.AlarmSpec -Property @{Name = "$($dsname.name) datastore RW latency";Description = "Custom alarm to monitor VMFS $($dsname.name) ($($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)) read or write latency";Enabled = $true;expression = (New-Object VMware.Vim.OrAlarmExpression -Property @{expression = @((New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "datastore"}|?{$_.nameinfo.key -match "totalReadLatency"}|?{$_.RollupType -match "average"}).key;Instance = ($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)});Operator = "isAbove";Yellow = "30";YellowInterval = "300";Red = "40";RedInterval = "180";Type = "HostSystem"});(New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "datastore"}|?{$_.nameinfo.key -match "totalWriteLatency"}|?{$_.RollupType -match "average"}).key;Instance = ($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)});Operator = "isAbove";Yellow = "30";YellowInterval = "300";Red = "40";RedInterval = "180";Type = "HostSystem"}))});Action=(New-Object VMware.Vim.GroupAlarmAction -Property @{Action= (New-Object VMware.Vim.AlarmTriggeringAction -Property @{Action = (New-Object VMware.Vim.SendEmailAction -Property @{ToList = "admin@vmware.local";Subject = "$($dsname.name) RW latency";CcList = "";Body = ""});TransitionSpecs = @((New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "green";FinalState = "yellow";Repeats = $false});(New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "yellow";FinalState = "red";Repeats = $false}))})});ActionFrequency = "1800"}))}}

Alarme de latence SIOC Normalized Latency pour datastore partagé, yellow 30ms/5min et red 40ms/3min (mailto:admin@vmware.local) :

foreach ($dsname in (Get-View -ViewType datastore|?{$_.summary.Accessible -and $_.summary.MultipleHostAccess -and $_.IormConfiguration.Enabled})){if (!((Get-View AlarmManager).GetAlarm((Get-Folder -NoRecursion |Get-View).moref)|?{(get-view $_).info.name -match "$($dsname.name) datastore SIOC Normalized Latency"})){(Get-View AlarmManager).CreateAlarm((Get-Folder -NoRecursion |Get-View).MoRef,(New-Object VMware.Vim.AlarmSpec -Property @{Name = "$($dsname.name) datastore SIOC Normalized Latency";Description = "Custom alarm to monitor $($dsname.name) ($($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)) datastore SIOC Normalized Latency";Enabled = $true;expression = (New-Object VMware.Vim.OrAlarmExpression -Property @{expression = @((New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "datastore"}|?{$_.nameinfo.key -match "sizeNormalizedDatastoreLatency"}|?{$_.RollupType -match "average"}).key;Instance = ($dsname|%{$_.Host}|%{$_.MountInfo.Path.split("/")[-1]}|sort -unique)});Operator = "isAbove";Yellow = "30000";YellowInterval = "300";Red = "40000";RedInterval = "180";Type = "HostSystem"}))});Action=(New-Object VMware.Vim.GroupAlarmAction -Property @{Action= (New-Object VMware.Vim.AlarmTriggeringAction -Property @{Action = (New-Object VMware.Vim.SendEmailAction -Property @{ToList = "admin@vmware.local";Subject = "$($dsname.name) SIOC Normalized Latency";CcList = "";Body = ""});TransitionSpecs = @((New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "green";FinalState = "yellow";Repeats = $false});(New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "yellow";FinalState = "red";Repeats = $false}))})});ActionFrequency = "1800"}))}}

Bonus : Alarme de Disk Command aborts pour datastore VMFS partagé, en cas de défaillance/saturation DAS/SAN/iSCSI (mailto:admin@vmware.local) :

foreach ($diskname in (Get-View -ViewType datastore|?{$_.info.vmfs -and $_.summary.Accessible -and $_.summary.MultipleHostAccess}|%{$_.info.vmfs.extent}|%{$_.diskname})){if (!((Get-View AlarmManager).GetAlarm((Get-Folder -NoRecursion |Get-View).moref)|?{(get-view $_).info.name -match "$diskname Disk Command aborts"})){(Get-View AlarmManager).CreateAlarm((Get-Folder -NoRecursion |Get-View).MoRef,(New-Object VMware.Vim.AlarmSpec -Property @{Name = "$diskname Disk Command aborts";Description = "Custom alarm to monitor $diskname Disk Command aborts";Enabled = $true;expression = (New-Object VMware.Vim.OrAlarmExpression -Property @{expression = (New-Object VMware.Vim.MetricAlarmExpression -Property @{Metric = (New-Object VMware.Vim.PerfMetricId -Property @{CounterId = ((Get-View (Get-View ServiceINstance).Content.PerfManager).PerfCounter|?{$_.groupinfo.key -match "disk"}|?{$_.nameinfo.key -match "commandsAborted"}|?{$_.GroupInfo.Key -ne "virtualDisk"}|?{$_.RollupType -match "summation"}).key;Instance = $diskname});Operator = "isAbove";Yellow = "1";Red = "2";Type = "HostSystem"})});Action=(New-Object VMware.Vim.GroupAlarmAction -Property @{Action= (New-Object VMware.Vim.AlarmTriggeringAction -Property @{Action = (New-Object VMware.Vim.SendEmailAction -Property @{ToList = "admin@vmware.local";Subject = "$diskname Disk Command aborts";CcList = "";Body = ""});TransitionSpecs = @((New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "green";FinalState = "yellow";Repeats = $false});(New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec -Property @{StartState = "yellow";FinalState = "red";Repeats = $true}))})});ActionFrequency = "1800"}))}}

MAJ 30/10/2012 : Mise à jours des scripts pour n’exclure que les vm “invalid” ou “orphaned”.

MAJ 06/07/2012 : Mise à jour des scripts pour exclure “*/hostCache/*” pour ne pas remonter les fichiers vswp de la fonction Host Cache d’ESXi 5.

MAJ 05/07/2012 : Mise à jour des scripts pour intégrer les fichiers vswp.xxxxxx, merci à Larus Hjartarson pour l’idée.

MAJ 21/03/2012 : Mise à jour des scripts pour exclure “var/tmp/cache” dans le cas d’une redirection de la partition scratch sur un datastore.

MAJ 06/03/2012 : Mise à jour des scripts pour le support des multi-datacenter/multi-vcenter, des snapshot NetApp et des fichiers zdump qui trainent.

MAJ 28/02/2012 : Mise à jour des scripts suite à une erreur dans le “SideIndicator”

MAJ 24/02/2012 : Et pour ceux qui ont *VRAIMENT* confiance, voici le script pour supprimer les fichiers vswp orphelins (et si vraiment vous en avez, vous pouvez changer le filtrage au niveau du |?{$_.lostfile -match “vswp”}) :

foreach ($lost in (compare -passthru -property path $(foreach ($ds in ((get-view -ViewType datastore|sort name -unique|?{$_.summary.accessible}|%{(get-view $_.browser).SearchDatastoreSubFolders($("[" + $_.Name + "]"),(New-Object VMware.Vim.HostDatastoreBrowserSearchSpec -property @{matchPattern=("*zdump*","*.xml","*.vmsn","*.vmsd","*.vswp*", "*.vmx","*.vmdk","*.vmss","*.nvram","*.vmxf"); details=(New-Object VMware.Vim.FileQueryFlags -property @{filesize=$true;modification=$true})}))})|?{$_.file})){foreach ($files in $ds.file){$files|?{$_.Path -notmatch "-ctk.vmdk|esxconsole.vmdk|esxconsole-flat.vmdk"}|select @{n="path";e={$ds.FolderPath + $files.path|%{$_ -replace " ",""}}},FileSize,Modification|?{$_.path -notmatch ".zfs|.snapshot|var/tmp/cache|/hostCache/|.lck"}}}) $(get-view -ViewType virtualmachine|?{$_.Runtime.ConnectionState -notmatch "invalid|orphaned"}|%{$_.layoutex.file}|?{$_.type -ne "log"}|%{$_.name}|%{$_ -replace " ",""}|sort -unique|select @{n="path";e={$_}})|?{$_.SideIndicator -eq "<="}|select @{n="LostFile";e={$_.Path -replace "]","] "}}, @{n="FileSize";e={"{0:N0}" -f ($_.FileSize/1MB)}}, Modification|?{$_.lostfile -match "vswp"})){(get-view (Get-View ServiceInstance|select -first 1).content.filemanager|select -first 1).DeleteDatastoreFile($lost.LostFile,(Get-View -ViewType datacenter|?{$_.Datastore -match (get-datastore $lost.LostFile.split("[]")[1]|get-view).moref}|select -first 1).moref)}

MAJ 24/02/2012 : Mise à jour du script pour afficher la taille en MegaBytes et la date de modification des fichiers.

Très fortement inspiré du célèbre Raiders of the Lost VMX (mais aussi du post Orphaned files and folders – Spring cleaning) de LucD, nous avons enfin fini de finioler un oneliner PowerCLI qui va vous lister les fichiers .xml, .vmsn, .vmsd, .vswp, .vmx, .vmdk, .vmss, .nvram et .vmxf qui traînent sur vos datastores et qui ne sont pas rattachés à une quelconque vm du vcenter depuis lequel vous avez exécuté le script.

compare -passthru -property path $(foreach ($ds in ((get-view -ViewType datastore|sort name -unique|?{$_.summary.accessible}|%{(get-view $_.browser).SearchDatastoreSubFolders($("[" + $_.Name + "]"),(New-Object VMware.Vim.HostDatastoreBrowserSearchSpec -property @{matchPattern=("*zdump*","*.xml","*.vmsn","*.vmsd","*.vswp*", "*.vmx","*.vmdk","*.vmss","*.nvram","*.vmxf"); details=(New-Object VMware.Vim.FileQueryFlags -property @{filesize=$true;modification=$true})}))})|?{$_.file})){foreach ($files in $ds.file){$files|?{$_.Path -notmatch "-ctk.vmdk|esxconsole.vmdk|esxconsole-flat.vmdk"}|select @{n="path";e={$ds.FolderPath + $files.path|%{$_ -replace " ",""}}},FileSize,Modification|?{$_.path -notmatch ".zfs|.snapshot|var/tmp/cache|/hostCache/|.lck"}}}) $(get-view -ViewType virtualmachine|?{$_.Runtime.ConnectionState -notmatch "invalid|orphaned"}|%{$_.layoutex.file}|?{$_.type -ne "log"}|%{$_.name}|%{$_ -replace " ",""}|sort -unique|select @{n="path";e={$_}})|?{$_.SideIndicator -eq "<="}|select @{n="LostFile";e={$_.Path -replace "]","] "}}, @{n="FileSize";e={"{0:N0}" -f ($_.FileSize/1MB)}}, Modification|ft -autosize

Les snapshots ZFS (.zfs/snapshot), les fichiers -ctk.vmdk (fichiers CBT) et les vmdk des COS d’ESX classic (esxconsole-) sont filtrés, n’hésitez pas à nous remonter d’autres faux positifs que vous pourriez rencontrer. Prochaine étape : ménage automatique…

Voici donc un petit one-liner qui permet de combiner les actions de rescan et refresh :

Get-View -ViewType HostSystem -Property ConfigManager.StorageSystem -Filter @{"Name"="^vmvisor"}|%{Get-View $_.ConfigManager.StorageSystem|%{$_.RescanAllHba();$_.RefreshStorageSystem()}}

L’équivalent en moins rapide mais en plus simple :

Get-VMHost vmvisor*|Get-VMHostStorage -RescanAllHba -Refresh

Vous aurez remarqué que nous n’avons pas inclus de RescanVmfs car selon le vSphere API Reference, la méthode RefreshStorageSystem l’intègre :

The RefreshStorageSystem method obtains information from the host platform and it performs a rescan of VMFS volumes. It does not look for new devices.

Bonus : la même commande pour la partie réseau, dont le refresh n’est aussi disponible qu’individuellement sur chaque host :

Get-View -ViewType HostSystem -Property ConfigManager.NetworkSystem -Filter @{"Name"="^vmvisor"}|%{Get-View $_.ConfigManager.NetworkSystem|%{$_.RefreshNetworkSystem()}}

Inutile de vous rappeler qu’il vous faudra ajuster le filtrage de la commande Get-View à votre nomenclature.

N’importe lequel de ces messages vous mènera aux forums VMware, sur des blogs ou même à cette kb qui précise justement ce que nous allons prendre à contre pied :

Because a virtual machine can be moved between hosts, the host where the virtual machine is currently registered may not be the host maintaining the file lock. The lock must be released by the ESX host that owns the lock

Le point commun de ces troubleshooting est l’objectif de faire sauter le lock du ou des fichiers, ce qui parait tout à fait censé et que nous avons d’ailleurs déjà suivi par le passé en essayant aussi de redémarrer le service hostd en masse grâce à plink (ou esxi.txt est la liste des fqdn) avant de redémarrer l’ESX en dernier recours :

@echo off
FOR /F %%i in (esxi.txt) do call :plink %%i
:plink
"C:\Program Files (x86)\PuTTY\plink.exe" -pw "password" root@%1 "/sbin/services.sh restart"
GOTO End
:Last
:End

Cette dernière étape est particulièrement longue et fastidieuse alors que, sauf cas exceptionnel, la solution “idéale” est de trouver le dernier ESX ayant fait tourné la vm (via le vmware.log ou grâce à export régulier des infos du vCenter), de passer DRS en manuel pour être prompté au démarrage de la VM et de selectionner le bon ESX :

C’est tout simple mais ça marche très bien

Function not implemented

La solution était caché dans une kb VMware sur le troubleshooting d’erreur lors de la creation d’un datastore :

Note: If you are using VMware ESXi or the <identifier> you are specifying is located within the /vmfs/ directory, you may need to append the conv=notrunc parameter to the dd command. Not doing so may result in a Function not implemented error.

Alleluia !

Cette update apporte majoritairement du support matériel supplémentaire mais 2 nouveautés intéressantes font leur apparition :

• La limite de vCPU par core passe de 8 (ou 11 pour du VDI) à 20
• vmfs-undelete, un outil de restauration de fichiers supprimés pour VMFS

Bien que vmfs-undelete ne soit encore qu’experimental et non disponible sur ESXi, il sera très probablement plus que bien accueilli par la communauté VMware