System VMs does not start after upgrade to CloudStack 4.2

Folks from CloudStack thought it was fun to release CloudStack 4.2 with incomplete/incorrect documentation, so after you update CloudStack to 4.2, all your system VMs are not going to work, once you restart them…

BUG described here: with some proposed solutions

A full solutions is very well explained here:

In case the URL is down, here is a very much copy/paste from the original author, all credits goes to him.:


Step 1): Mount your secondary storage to your management server with the following:

mount -t nfs {ip_of_storage_server}:[path_to_secondary_storage] /mnt

Step 2): Download the latest version of the templates:

/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /mnt -u -h kvm -F

Step 3): Find the name of the old template in the database:

USE cloud
SELECT install_path FROM template_host_ref WHERE template_id=3;

Sample output:


Step 4): write down the name of the .qcow2 file that your given in the previous step.

Step 5): from the management server locate the new template on the mounted secondary storage:

cd /mnt/template/tmpl/1/3/

Step 6): rename the .qcow2 file in that folder to the name we copied from the database.

Step 7): edit the file in the same folder and change both instances of the old name to the new one.

Step 8): we need to reset the cached template in the database:

UPDATE template_spool_ref SET download_pct='0',download_state='NOT_DOWNLOADED',state='NULL',local_path='NULL',install_path='NULL',template_size='0' WHERE template_id='3';

Step 9): Unmount your secondary storage from the management server:

umount /mnt

Step 10): disable the zone from the management UI.

Step 11): update the database records for your system VMs to be ‘Stopped’. You will need to do this for both the Secondary Storage VM and the Console Proxy. The ID of the system VM is the number in it’s name, for example; s-34-VM,, would have an ID of ’34′.

UPDATE vm_instance SET state='Stopped' where id='{id_of_system_vm}';

Step 12): From the management UI, destroy both the system VMs.

Step 13): Once both system VMs have been destroyed, re-enable the zone.

Step 14): Tail the management log and watch for the VMs to start.

tail -f /var/log/cloudstack/management/management-server.log


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: