Troubleshooting VMM: Resolving Error 2606 During New Virtual Machine Job Startup

Table of Contents

System Center Virtual Machine Manager (VMM) is a powerful management solution for virtualized datacenters, enabling administrators to configure, manage, and optimize virtual machines across multiple hypervisors. However, like any complex system, VMM can encounter issues that disrupt its operations. One such problem that can impede the efficient management of virtualized resources is Error 2606, which typically arises when attempting to initiate new jobs, particularly those involving virtual machine creation or modification.

This error signals a critical conflict within the VMM environment, preventing the execution of essential tasks. Understanding the underlying mechanisms of VMM’s job management and object locking system is crucial for effective troubleshooting. This article will delve into the symptoms of Error 2606, explain its root cause, and provide a comprehensive resolution strategy to restore VMM’s operational integrity.

Symptoms of Error 2606¶

When administrators attempt to start a new job within System Center Virtual Machine Manager, they may encounter a specific error message that halts the process. This error is consistently identified as Error 2606, indicating a fundamental conflict in resource access. The full error message typically presents as follows, directing the administrator towards a potential resolution path:

Error (2606)
Unable to perform the job because one or more of the selected objects are locked by another job.

Recommended Action
To find out which job is locking the object, in the Jobs view, group by Status, and find the running or canceling job for the object. When the job is complete, try again.

This message clearly points to an object locking mechanism as the source of the problem. It implies that the virtual machine, host, or other VMM object that the new job intends to interact with is currently reserved or in use by another ongoing or stuck operation. The VMM console attempts to guide the user towards identifying the conflicting job, often suggesting a review of the “Jobs” view.

Further investigation into the VMM trace logs provides more granular detail about the precise nature of the lock conflict. A typical trace entry associated with Error 2606 might look like this:

48412,04:24:10.457 02-10-2014,0x097C,0x05BC,4,CarmineObjectLock.cs,822,0x00000000,CarmineObjectLock; Task Failed Acquiring Delete lock on VM object with ID

This trace log entry is highly informative for experienced VMM administrators. It indicates that Task ID 1 attempted to acquire a “Delete lock” on a specific “VM object with ID

Understanding VMM Job Management and Object Locking¶

System Center Virtual Machine Manager relies on a robust job management and object locking system to ensure data integrity and prevent conflicts during concurrent operations within a virtualized environment. Every action performed in VMM, from creating a new virtual machine to migrating a host, is encapsulated as a “job.” These jobs often interact with various VMM objects, such as virtual machines, hosts, storage, and networks. To maintain consistency and avoid corruption, VMM employs an object-locking mechanism.

When a job needs to modify an object, it first attempts to acquire a lock on that object. This lock acts like a reservation, preventing other jobs from making conflicting changes simultaneously. There are different types of locks, such as read locks, write locks, and delete locks, each granting specific permissions and restricting others. For instance, a “write lock” prevents other jobs from modifying or deleting the object until it’s released, ensuring that only one job can perform a significant change at a time. This concurrency control is fundamental to the stability and reliability of the VMM environment.

The necessity of object locking becomes apparent when considering scenarios like simultaneous VM migrations or configuration changes. Without locks, two independent jobs could attempt to move the same VM or modify its properties, leading to an inconsistent state or potential data loss. VMM’s architecture is designed to handle these complexities by strictly enforcing these locking protocols. However, under certain circumstances, these locks may not be released correctly.

This failure to release a lock can occur if a VMM job crashes unexpectedly, the VMM service is restarted prematurely, or if there are underlying database issues that prevent the proper cleanup of job-related records. When a lock becomes orphaned or stuck, it persists in the VMM database even though the job that created it is no longer active or relevant. This orphaned lock then prevents any subsequent, legitimate jobs from interacting with the locked object, leading directly to Error 2606. The VMM database, specifically the tbl_VMM_Lock table, serves as the central repository for tracking these active object locks, making it a critical component in understanding and resolving such issues.

Cause of Error 2606¶

The fundamental cause of Error 2606 in System Center Virtual Machine Manager is the presence of locked records in the VMM database. This occurs when an object, such as a virtual machine, host, or other VMM resource, has an active lock associated with it that prevents new jobs from interacting with it. While object locking is an intentional and necessary design feature of VMM to maintain data integrity, the error arises when these locks become orphaned or stuck.

An orphaned lock typically results from a job that failed to complete cleanly or was interrupted. For example, if a virtual machine creation job crashes midway, or a migration job times out, the lock it placed on the VM object or host resource might not be released as intended. Similarly, an unexpected shutdown or restart of the VMM service while jobs are in progress can leave locks in a persistent state within the database. Network connectivity issues between the VMM server and managed hosts or storage can also contribute to job failures that lead to unreleased locks.

These lingering locks prevent any subsequent jobs from acquiring the necessary access to the affected objects. The new job, upon attempting to obtain a lock, finds that the object is already reserved by a non-existent or inactive task. This creates a deadlock situation where legitimate operations are blocked, leading to the display of Error 2606. Essentially, the VMM database incorrectly perceives the object as being actively used, even though no valid job is currently holding it. Identifying and correctly removing these orphaned lock entries from the database is the key to resolving this specific error and restoring VMM’s ability to execute new tasks.

Resolution for Error 2606¶

Resolving Error 2606 involves directly interacting with the VMM database to identify and remove the problematic object locks. This process requires careful execution and adherence to best practices, as direct database manipulation carries inherent risks. Before proceeding with any database changes, it is absolutely critical to create a full backup of your VirtualManagerDB database. This ensures that you can revert to a previous state if any unintended issues arise during the resolution process.

The tbl_VMM_Lock table within the VMM database is where all active object locks are recorded. Our resolution strategy focuses on querying this table to find the orphaned locks and then removing them.

Step-by-Step Resolution Guide:¶

1. Access SQL Server Management Studio (SSMS):
   Open SQL Server Management Studio and connect to the SQL Server instance hosting your VMM database (typically named VirtualManagerDB). You will need appropriate permissions to query and modify this database.

2. Identify Existing Locks:
   To see what object locks currently exist in your VMM database, execute the following SQL query:

   SELECT * FROM [VirtualManagerDB].[dbo].[tbl_VMM_Lock];
   

   This query will display all entries in the tbl_VMM_Lock table. Review the output carefully. Look for entries that correspond to objects you are trying to manage and tasks that you know have failed or are no longer active in the VMM console’s “Jobs” view. The TaskID column is crucial here, as it represents the unique identifier of the job that placed the lock. Other columns like ObjectID (the ID of the locked VMM object) and LockType (e.g., WriteLock, DeleteLock) can help confirm if the lock is indeed related to your problem.

   Example Output Structure for tbl_VMM_Lock:

   LockID (GUID)	ObjectID (GUID)	TaskID (GUID)	LockType (int)	LockTime (datetime)	LockedBy (string)	Description (string)
   {GUID-1}	{VM-ID-A}	{Task-ID-X}	1 (WriteLock)	2023-10-27 10:30:00	VMMService	VM creation
   {GUID-2}	{Host-ID-B}	{Task-ID-Y}	2 (DeleteLock)	2023-10-27 11:15:00	VMMService	Host maintenance
   {GUID-3}	{VM-ID-C}	{Task-ID-Z}	1 (WriteLock)	2023-10-27 12:00:00	VMMService	VM property update

   Note: LockType values are typically integers, with 1 often representing a Write Lock and 2 a Delete Lock, though specific interpretations might vary slightly.

3. Confirm the Problematic Task ID:
   If you have a specific Task ID from the VMM trace log (as shown in the symptoms section, e.g., Task ID 2 holding the Write lock), you can use a more targeted query to verify its presence:

   SELECT * FROM [VirtualManagerDB].[dbo].[tbl_VMM_Lock] WHERE TaskID = 'Your_Problematic_Task_GUID';
   

   Replace 'Your_Problematic_Task_GUID' with the actual GUID from your VMM trace or from the output of the previous SELECT * query. This helps isolate the exact lock causing the issue.

4. Perform Database Backup:
   As mentioned, this step is non-negotiable. Right-click on your VirtualManagerDB in SSMS, go to Tasks -> Backup, and create a full backup to a secure location.

5. Delete the Orphaned Lock:
   Once you have positively identified the TaskID associated with the orphaned lock that is causing Error 2606, you can proceed to delete the record from the tbl_VMM_Lock table. Execute the following DELETE command, ensuring you replace 'Your_Problematic_Task_GUID' with the correct TaskID:

   DELETE FROM [VirtualManagerDB].[dbo].[tbl_VMM_Lock] WHERE TaskID = 'Your_Problematic_Task_GUID';
   

   Example: If the problematic TaskID was {E8D1C4D2-9A0F-4B7C-8D5E-1F2A3B4C5D6E}, the command would be:

   DELETE FROM [VirtualManagerDB].[dbo].[tbl_VMM_Lock] WHERE TaskID = '{E8D1C4D2-9A0F-4B7C-8D5E-1F2A3B4C5D6E}';
   

   Execute this command carefully. It will remove the specific lock entry from the database.

6. Verify Deletion (Optional but Recommended):
   You can run the SELECT query from Step 3 again to confirm that the record has been successfully removed.

7. Restart VMM Services:
   After making direct changes to the VMM database, it is often a good practice to restart the System Center Virtual Machine Manager service on your VMM server. This ensures that VMM re-reads its configuration and current state from the database and clears any cached information about locks.

   • Open services.msc.
   • Locate “System Center Virtual Machine Manager”.
   • Right-click and select “Restart”.

8. Re-attempt the VMM Job:
   Now, go back to your VMM console and attempt to start the new job that was previously failing with Error 2606. It should now proceed without encountering the object lock conflict.

Advanced Troubleshooting and Preventative Measures¶

While deleting orphaned locks resolves the immediate issue, it’s also important to consider why these locks became orphaned in the first place. Addressing the root cause can prevent recurrence of Error 2606.

• Review VMM Job History: Regularly check the VMM job history for recurring failures or jobs that frequently get stuck in a “Running” or “Canceling” state. This can indicate underlying issues with specific hosts, storage, or network configurations.
• Monitor VMM Server Health: Ensure the VMM server itself has sufficient resources (CPU, RAM, disk I/O) and is stable. Performance bottlenecks on the VMM server can lead to job timeouts and failures.
• Check SQL Server Performance: The VMM database is a critical component. Ensure your SQL Server instance is healthy, performing well, and has adequate resources. Slow database operations can contribute to job failures.
• Network and Storage Stability: Unreliable network connectivity between the VMM server, hosts, and storage arrays can disrupt VMM jobs, leading to incomplete operations and orphaned locks. Verify network integrity and storage array responsiveness.
• VMM Update Status: Keep your System Center Virtual Machine Manager environment updated with the latest patches and hotfixes. Microsoft frequently releases updates that address known issues and improve stability.
• Scheduled Database Maintenance: Implement regular SQL Server database maintenance plans for VirtualManagerDB, including index rebuilding/reorganizing and statistics updates, to ensure optimal performance and health.

Understanding the interaction between VMM jobs, object locking, and the underlying database is key to maintaining a robust virtualized infrastructure. By following these steps and implementing preventative measures, administrators can effectively mitigate the impact of Error 2606 and ensure the smooth operation of their VMM environment.

---

We hope this detailed guide helps you resolve Error 2606 in your System Center Virtual Machine Manager environment. Have you encountered this issue before? What was your experience, or do you have any additional tips to share with the community? Feel free to leave your comments and insights below!