How to: Resolve errors 13 and 12292 on Volume Shadow Copy Service (VSS) and stop Virtual Machines from going into Save state

How to: Resolve errors 13 and 12292 on Volume Shadow Copy Service (VSS) and stop Virtual Machines from going into Save state

I recently installed Windows Server 2012 and with it an hardware RAID card in order to consolidate storage and host several VMs. The problem I am facing is that my virtual machines sporadically go into save state and the reason it gives me is that the disk hosting the VM is running out of space. This is hardly the case with over 2 TB available so I had to start digging into the Event Log. I kept coming across error 16060 which reads:

‘ServerName.Domain.local’ has been paused because it has run out of disk space on ‘X:Hyper-V StorageVirtual Hard Disks’. (Virtual machine ID 8521F3CD-12CD-12A3-A987-AB855D4F7F39)

The other recurring issue that I noticed on the event logs were errors 13 and 12292 on the Volume Shadow Copy Service (VSS). My thought at this point was that they could be related as an issue with VSS may cause Hyper-V to become unable to provision space for the VM somehow. And since fixing those issues my VMs haven’t gone into saved state yet.

Cause

Apparently there are different Shadow Copy providers and one of them is for iSCSI contollers. My guess at this point is that said controller was used for my RAID card. For some reason the provider requires local administrator credentials in order to work properly and if you check on DCOM’s Component Services console the “WTSnapshopProvider” and the “WTVDSProv” are set to work with the invoking user credentials.

Error Samples:

Volume Shadow Copy Service information: The COM Server with CLSID {463948d2-035d-4d1d-9bfc-473fece07dab} and name HWPRV cannot be started. [0x80070005, Access is denied.

]

Operation:

Creating instance of hardware provider

Obtain a callable interface for this provider

List interfaces for all providers supporting this context

Get Shadow Copy Properties

Delete Shadow Copies

Context:

Provider ID: {3f900f90-00e9-440e-873a-96ca5eb079e5}

Provider ID: {3f900f90-00e9-440e-873a-96ca5eb079e5}

Class ID: {463948d2-035d-4d1d-9bfc-473fece07dab}

Snapshot Context: -1

Snapshot Context: -1

Execution Context: Coordinator

Execution Context: Coordinator

 

Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {463948d2-035d-4d1d-9bfc-473fece07dab} [0x80070005, Access is denied.

].

Operation:

Creating instance of hardware provider

Obtain a callable interface for this provider

List interfaces for all providers supporting this context

Get Shadow Copy Properties

Context:

Provider ID: {3f900f90-00e9-440e-873a-96ca5eb079e5}

Provider ID: {3f900f90-00e9-440e-873a-96ca5eb079e5}

Class ID: {463948d2-035d-4d1d-9bfc-473fece07dab}

Snapshot Context: -1

Snapshot Context: -1

Execution Context: Coordinator

Solution:

For reasons beyond my understanding changing their running credentials as a local administrator on the machine resolves both errors. I also do not understand why Microsoft would ship a product that requires this change out of the box. Another important thing to notice is that with iSCSI you can have a target server. On this scenario you need to create identical credentials for a local administrator on both machines and use them on the providers (identical username and password) or I guess a domain account with local admin privileges on both computers.

UI Configuration (Dcom Config)

You can also use DCOM Config to configure the credentials as follows:

1. Open Component Services, open Computers, open My Computer and then open DCOM Config.

2. Locate ‘WTVdsProv’ and configure credentials as appropriate

3. Locate ‘WTSnapshotProvider and configure credentials as appropriate

Take the WTSnapshotProvider for example:

1. Locate the provider under DCOM Config container

 

2. Right click on the provider, and click Properties

 

3. Click the Identity tab, select the This user option, then specify an account which has the iSCSI Target Server local Administrator’s permission

 

Cmdlet Configuration

As an alternative, you can also use powershell cmdlet to configure the credentials:

PS C:> $PsCred = Get-Credential

PS C:> $PrvdIdentityPath = New-Object System.Management.ManagementPath(“rootwmi:WT_iSCSIStorageProviderIdentity”)

PS C:> $PrvdIdentityClass = New-Object System.Management.ManagementClass($PrvdIdentityPath)

PS C:> $PrvdIdentityClass.SetProviderIdentity(“{88155B26-CE61-42FB-AF31-E024897ADEBF}”,$PsCred.UserName,$PsCred.GetNetworkCredential().Password)

PS C:> $PrvdIdentityClass.SetProviderIdentity(“{9D884A48-0FB0-4833-AB70-A19405D58616}”,$PsCred.UserName,$PsCred.GetNetworkCredential().Password)

 

 

Once you have done that change you can run the following commands (with sample output) to verify the provider is working properly:

Windows PowerShell Copyright (C) 2012 Microsoft Corporation. All rights reserved.

PS C:Windowssystem32> diskshadow Microsoft DiskShadow version 1.0 Copyright (C) 2012 Microsoft Corporation On computer:  ServerName,  17/01/2013 08:00:00 p. m.

DISKSHADOW> add volume x:

DISKSHADOW> create Alias VSS_SHADOW_1 for shadow ID {cf0602f5-7c32-4fe6-93db-2919af476668} set as environment variable. Alias VSS_SHADOW_SET for shadow set ID {28e26c29-a211-45ff-b69c-21cf15d77415} set as environment variable.

Querying all shadow copies with the shadow copy set ID {28e26c29-a211-45ff-b69c-21cf15d77415}

* Shadow copy ID = {cf0602f6-7c32-4fe6-94db-2919af476968}               %VSS_SHADOW_1%                 – Shadow copy set: {28e26c29-a211-45ff-b69c-27cf15d87415}       %VSS_SHADOW_SET%                 – Original count of shadow copies = 1                 – Original volume name: ?Volume{4e585cff-e8de-4a78-bbba-6c2c14563a9f} [X:]                 – Creation time: 17/01/2013 08:18:00 p. m.                 – Shadow copy device name: ?GLOBALROOTDeviceHarddiskVolumeShadowCopy10169                 – Originating machine: ServerName.Domain.local                 – Service machine: ServerName.DomainName.local                 – Not exposed                 – Provider ID: {b5946137-7b9f-4925-af80-51abd60b20d5}                 – Attributes:  Auto_Release Differential

Number of shadow copies listed: 1

DISKSHADOW>