Proxmox appears to crash HA

davidtja · January 11, 2021, 5:15pm

I have a Proxmox server for my HA install which I built through following Whiskerz install (so I’m no whizz at this).
Everything has been running fine then I was running out of disk space so I simply increased the space. All seemed fine. In the past week however after a Proxmox backup my HA is’nt running. Looking at the log in Proxmox I get the following:

INFO: starting new backup job: vzdump 100 --storage local --quiet 1 --mailnotification always --mailto [email protected] --mode snapshot --compress lzo
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-01-11 00:00:03
INFO: status = running
INFO: VM Name: hassosova-3.12
INFO: include disk 'sata0' 'local-lvm:vm-100-disk-1' 96G
INFO: include disk 'efidisk0' 'local-lvm:vm-100-disk-0' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-100-2021_01_11-00_00_03.vma.lzo'
INFO: started backup task 'b10e2ebd-3bd1-4c89-9318-c1423db0c376'
INFO: resuming VM again
INFO:   0% (83.2 MiB of 96.0 GiB) in  3s, read: 27.7 MiB/s, write: 13.1 MiB/s
INFO:   1% (989.5 MiB of 96.0 GiB) in 23s, read: 45.3 MiB/s, write: 27.4 MiB/s
INFO:   2% (2.0 GiB of 96.0 GiB) in 47s, read: 42.2 MiB/s, write: 42.0 MiB/s
INFO:   3% (2.9 GiB of 96.0 GiB) in  1m 20s, read: 30.2 MiB/s, write: 28.8 MiB/s
INFO:   4% (3.8 GiB of 96.0 GiB) in  1m 43s, read: 40.8 MiB/s, write: 38.2 MiB/s
INFO:   5% (4.8 GiB of 96.0 GiB) in  2m 15s, read: 31.2 MiB/s, write: 31.1 MiB/s
INFO:   6% (5.8 GiB of 96.0 GiB) in  2m 43s, read: 34.4 MiB/s, write: 34.4 MiB/s
INFO:   7% (6.8 GiB of 96.0 GiB) in  3m 30s, read: 22.2 MiB/s, write: 22.0 MiB/s
INFO:   8% (7.7 GiB of 96.0 GiB) in  4m  5s, read: 27.7 MiB/s, write: 27.5 MiB/s
INFO:   9% (8.6 GiB of 96.0 GiB) in  4m 17s, read: 78.1 MiB/s, write: 69.8 MiB/s
INFO:  10% (9.6 GiB of 96.0 GiB) in  4m 39s, read: 46.2 MiB/s, write: 45.5 MiB/s
INFO:  11% (10.6 GiB of 96.0 GiB) in  5m  5s, read: 37.0 MiB/s, write: 36.6 MiB/s
INFO:  12% (11.6 GiB of 96.0 GiB) in  5m 34s, read: 35.1 MiB/s, write: 34.1 MiB/s
INFO:  13% (12.5 GiB of 96.0 GiB) in  5m 59s, read: 39.5 MiB/s, write: 39.2 MiB/s
INFO:  14% (13.4 GiB of 96.0 GiB) in  6m 24s, read: 37.5 MiB/s, write: 35.3 MiB/s
INFO:  15% (14.4 GiB of 96.0 GiB) in  6m 48s, read: 41.0 MiB/s, write: 40.5 MiB/s
INFO:  16% (15.4 GiB of 96.0 GiB) in  7m 13s, read: 41.3 MiB/s, write: 40.2 MiB/s
INFO:  17% (16.3 GiB of 96.0 GiB) in  7m 41s, read: 33.3 MiB/s, write: 33.1 MiB/s
INFO:  18% (17.3 GiB of 96.0 GiB) in  8m  7s, read: 37.7 MiB/s, write: 36.4 MiB/s
INFO:  19% (18.3 GiB of 96.0 GiB) in  8m 34s, read: 36.7 MiB/s, write: 36.2 MiB/s
INFO:  20% (19.2 GiB of 96.0 GiB) in  9m 10s, read: 27.0 MiB/s, write: 26.3 MiB/s
INFO:  21% (20.2 GiB of 96.0 GiB) in  9m 35s, read: 40.4 MiB/s, write: 39.8 MiB/s
INFO:  22% (21.1 GiB of 96.0 GiB) in 10m  7s, read: 30.5 MiB/s, write: 29.6 MiB/s
INFO:  23% (22.1 GiB of 96.0 GiB) in 10m 33s, read: 37.4 MiB/s, write: 37.2 MiB/s
INFO:  24% (23.0 GiB of 96.0 GiB) in 10m 58s, read: 39.1 MiB/s, write: 37.6 MiB/s
INFO:  25% (24.0 GiB of 96.0 GiB) in 11m 23s, read: 39.9 MiB/s, write: 39.6 MiB/s
INFO:  26% (25.0 GiB of 96.0 GiB) in 12m  0s, read: 27.1 MiB/s, write: 26.3 MiB/s
INFO:  27% (25.9 GiB of 96.0 GiB) in 12m 26s, read: 37.4 MiB/s, write: 36.4 MiB/s
INFO:  28% (26.9 GiB of 96.0 GiB) in 12m 55s, read: 32.9 MiB/s, write: 31.7 MiB/s
INFO:  29% (27.9 GiB of 96.0 GiB) in 13m 18s, read: 44.7 MiB/s, write: 43.6 MiB/s
INFO:  30% (28.8 GiB of 96.0 GiB) in 13m 47s, read: 33.8 MiB/s, write: 32.3 MiB/s
INFO:  31% (29.8 GiB of 96.0 GiB) in 14m 18s, read: 31.2 MiB/s, write: 31.1 MiB/s
INFO:  32% (30.8 GiB of 96.0 GiB) in 14m 51s, read: 31.3 MiB/s, write: 30.8 MiB/s
INFO:  33% (31.7 GiB of 96.0 GiB) in 15m 24s, read: 29.1 MiB/s, write: 28.8 MiB/s
INFO:  34% (32.7 GiB of 96.0 GiB) in 15m 49s, read: 37.8 MiB/s, write: 36.5 MiB/s
INFO:  35% (33.6 GiB of 96.0 GiB) in 16m 10s, read: 47.9 MiB/s, write: 41.5 MiB/s
INFO:  36% (34.6 GiB of 96.0 GiB) in 16m 33s, read: 42.7 MiB/s, write: 42.0 MiB/s
INFO:  37% (35.5 GiB of 96.0 GiB) in 16m 51s, read: 53.7 MiB/s, write: 47.1 MiB/s
INFO:  38% (36.6 GiB of 96.0 GiB) in 17m 16s, read: 42.7 MiB/s, write: 42.2 MiB/s
INFO:  39% (37.5 GiB of 96.0 GiB) in 17m 32s, read: 57.7 MiB/s, write: 50.1 MiB/s
INFO:  40% (38.4 GiB of 96.0 GiB) in 18m  1s, read: 32.8 MiB/s, write: 32.6 MiB/s
INFO:  41% (39.4 GiB of 96.0 GiB) in 18m 24s, read: 43.5 MiB/s, write: 37.2 MiB/s
INFO:  42% (40.4 GiB of 96.0 GiB) in 19m  2s, read: 27.2 MiB/s, write: 26.9 MiB/s
INFO:  43% (41.3 GiB of 96.0 GiB) in 19m 29s, read: 35.2 MiB/s, write: 30.8 MiB/s
INFO:  44% (42.3 GiB of 96.0 GiB) in 20m  9s, read: 25.0 MiB/s, write: 25.0 MiB/s
INFO:  45% (43.2 GiB of 96.0 GiB) in 20m 35s, read: 35.5 MiB/s, write: 30.9 MiB/s
INFO:  46% (44.2 GiB of 96.0 GiB) in 21m  5s, read: 33.5 MiB/s, write: 32.4 MiB/s
INFO:  47% (45.1 GiB of 96.0 GiB) in 21m 32s, read: 35.7 MiB/s, write: 30.3 MiB/s
lzop: No space left on device: <stdout>
ERROR: VM 100 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job
ERROR: VM 100 qmp command 'backup-cancel' failed - unable to connect to VM 100 qmp socket - timeout after 5974 retries
ERROR: Backup of VM 100 failed - VM 100 qmp command 'query-backup' failed - got timeout
INFO: Failed at 2021-01-11 00:42:00
INFO: Backup job finished with errors

TASK ERROR: job errors

I assume it says I have no space, but I already increased the VM space so have plenty. I must be missing something. Should this kill HA? Advice really appreciated.

Burningstone · January 11, 2021, 6:45pm

Where are you storing the backup? On the same machine?
Also your backup is quite huge with 96gig, a complete backup of my VM running HA doesn’t even use 20gig.

davidtja · January 11, 2021, 7:34pm

Yes am storing on same machine. The 96 gig is what I expanded the drive to when it said it was full up. It was at 32 gig. Im sure my backup is less than 10gig.
I’ve got a feeling that although I expanded it - I may also need to resize it. I’m not familiar with this system though I merely followed an online tutorial which is fine I guess until you get into problems further down the road.

Burningstone · January 11, 2021, 7:46pm

First of all you should NOT store the backup on the same machine, otherwise you’ll lose everything when the machine crashes. Would be way better to store it on an external disk pr on a NAS.
How did you configure the backup? My VM has 128GB of space allocated, still the backup is smaller than 20GB.

davidtja · January 11, 2021, 7:53pm

I do store backups on other machines as well. I only have one VM on this nuc with HA on it. I just set up a besic backup through the UI to backup twice a week at midnight.

piotr · January 11, 2021, 7:55pm

What I have done:

Create NFS shares on my Synology NAS and NFS as storage in Proxmox under DataCenter > Storage > NFS. Set type to VZDump back file. Use this to store the backups/snapshots. I have set all my vm’s en lxc’s to use NFS.

davidtja · January 11, 2021, 8:05pm

Yeah but issue isn’t about how to do backups, I may well have done it wrong. My issue is my HA appears to be crashing after a backup. Given my backup is 20gb I think the problem stems from my just extending the drive size but I suspect now I also should have resized it after. I have no idea how to fix this so any help greatly appreciated. Thanks

Burningstone · January 11, 2021, 8:10pm

https://pve.proxmox.com/wiki/Resize_disks

davidtja · January 11, 2021, 8:19pm

Yes I have seen that which is where I realised I also have to resize. I wasn’t sure if this was old information as it seems strange that you can do part of this in the GUI but the rest at the shell.
So assuming I did this and increased it from 32gig to 96gb (I think thats what I did…) Then the command line would be "qm resize <vmid> <disk> +64G" however i’m not sure what the vmid &disk are called. Where would I see that. Thanks for your advice.

Burningstone · January 12, 2021, 6:48am

VM id is the number of your VM, should be visible in the proxmox GUI on the leftvhand side where all your VMs are. About the disk I don’t know, don’t have access to my machine to check currently.

piotr · January 12, 2021, 6:51am

What I meant: you should also put your vm’s on NFS. Local disk only for Proxmox VE.

piotr · January 12, 2021, 6:52am

After resizing the qm you must also resize the disk in the guest os.

Shaad · January 12, 2021, 8:06am

+1
Do not forget this step.

But still, how can you get a backup of 96Go for your HA VM ? What do you store in your HA VM ?
My HA VM runs fine on a 15Go VM.

davidtja · January 12, 2021, 8:42am

Firstly many thanks for all your assistance. Its enough getting my head around HA without Proxmox, I had hoped to run more VM’s but havent got that far yet hence went the Proxmox route.
So my Vm I believe is called 100.

So is it:
qm resize 100 vm-100-disk-1 +64G
OR
qm resize 100 PVE +64G

THanks for help

piotr · January 12, 2021, 2:30pm

Click on the VM (hassosova-3.12) > select Hardware > select Hard Disk > click the “resize” button on top of the frame.

But again: I wouldn’t run VM’s from the same disk as Proxmox. Create some NFS shares on your NAS.

davidtja · January 12, 2021, 2:37pm

I had already done the resizing bit as you described but still ran into trouble a few weeks later when doing the backup. Its as though the resizing didn’t really stick. If that makes sense. I was on 32gb and increased to 96gb (I have a 250gb SSD in there) and only use for HA.
I take your point about not using the VM’s from the same disk but at this stage it seems the easiest thing to do. Once I fix this I can experiment with using NFS share.
I believe I have now worked out my disk is called sata0 and going through the process of running the command line which I was struggling to find which is:
qm resize 100 sata0 +64G
is exactly the same as doing it through the GUI.
I believe what I actually need to do is enlarge the partition.
Thanks for advice so far.