You want to move a VM between cluster A and cluster B, or standalone host A to standalone host B?
As of PVE 7.3:
qm remote-migrate
Now, this is documented here, but the documentation on how to specify the API token argument is honestly not at all clear. So here’s a quick howto page with how to use the command and some gotchas you might trip over on the way like I did.
Make an API token on the target host:
Datacenter -> Permissions -> API Tokens
I have not researched what the minimum privileges are, and so here am going with a non privilege seperated root api key.
“Token ID” is just any text string. I have just used TOKENID in this screenshot to match my notes below.
You should consider setting an expiry date so that you can’t accidentally leave a ‘forever key’ with full root access enabled.
Proxmox will then present you with what I guess you would call your full token ID (user and Token ID from the previous screen combined together) and secret:
Then on the shell of the source host, you do the below
NB: the “apitoken” value you have to provide to the CLI tool should take the form of an entire HTTP header for whatever reason, and you combine the user, token id and secret as shown below (Bits in bold are what you need to set for your environment – I have constructed the APIToken below to match the screenshot above so you can see where the elements came from).
qm remote-migrate SOURCEVMID TARGETVMID apitoken='Authorization: PVEAPIToken=root@pam!TOKENID=c362d275-5e68-4482-a4c1-a0114c2ea408',host=TARGETIP,port=8006,fingerprint='HOSTFINGERPRINT' --target-bridge=vmbr0 --target-storage=ZFS-Mirror --online=true
NB; I am doing online migration because as of writing, in Proxmox 8.2.2;
- Ceph source does not support offline migration, period (
ERROR: no export formats for 'SOURCESTORAGE:vm-SOURCEVMID-disk-0' - check storage plugin support!
), and
- when I tried instead to use an NFS volume as the source, I found that a ZFS target does not support offline migration unless it is from a ZFS source: (
ERROR: migration aborted (duration 00:00:01): error - tunnel command '{"export_formats":"qcow2+size","format":"qcow2","volname":"vm-SOURCEVMID-disk-0.qcow2","migration_snapshot":0,"storage":"local-lvm","allow_rename":"1","with_snapshots":1,"cmd":"disk-import"}' failed - failed to handle 'disk-import' command - unsupported format 'qcow2' for storage type lvmthin
)
- while I suppose I probably could have exported from NFS to LVM or another NFS store on the target, my default local-lvm storage volume on the target host does not have enough storage for the VM to go on there, all of the storage is tied up in the ZFS-Mirror volume.
Then I hit a new problem:
ERROR: online migrate failure - error - tunnel command '{"migrate_opts":{"remote_node":"TARGETVMHOST","type":"websocket","spice_ticket":null,"migratedfrom":"SOURCEVMHOST","nbd":{"scsi0":{"drivestr":"TARGETSTORAGE:vm-TARGETVMID-disk-0,aio=native,cache=writeback,discard=on,format=raw,iothread=1,size=32G,ssd=1","volid":"TARGETSTORAGE:vm-TARGETVMID-disk-0","success":true}},"storagemap":{"default":"TARGETSTORAGE"},"network":null,"nbd_proto_version":1},"start_params":{"skiplock":1,"forcemachine":"pc-i440fx-9.0+pve0","statefile":"unix","forcecpu":null},"cmd":"start"}' failed - failed to handle 'start' command - start failed: QEMU exited with code 1
Not very descriptive, but fortunately you can get the /reason/ qemu exited with code 1 from the target host – checking the output of the ‘qmtunnel’ task in the task history on the target host, I see that the combination of disk options which are set on my source ceph storage are not valid with the target ZFS storage, and so the task is aborting:
QEMU: kvm: -drive file=/dev/zvol/TARGETSTORAGE/vm-TARGETVMID-disk-0,if=none,id=drive-scsi0,cache=writeback,aio=native,discard=on,format=raw,detect-zeroes=unmap: aio=native was specified, but it requires cache.direct=on, which was not specified.
To fix this, I switched aio back to the default io_uring on the source, restarted the VM to make that take effect, and then restarted the migrate.
You see the output of the copy progress on the shell window as it runs, but it also shows up in the proxmox web ui as a regular migrate task which you can montior through the task viewer as normal:
Once the task is done, if you did not specify --delete=true
then you will need to issue qm unlock SOURCEVMID
to unlock the VM on the source host in order to be able to delete it.