zsysctl state remove does not work on @autozsys snapshots

[pjgoodall]

Describe the bug

Once again, my system updates are failing because the zfs bpool associated with zsysctl is too full. My system is failing to update - which is a major problem . I cannot remove any of the state to fix the space problem. Last time I had to remove all snapshots using the config file. I have made no user saves of state since - in the hope that removing state would then work - it does not.

The documentation in ZFS focus on Ubuntu 20.04 LTS: ZSys commands for state management in the section: Removing state says:

zsysctl state remove STATE_NAME ...
for automated state save. anything after autozsys_ (

The online man page is much too terse: man zsysctl-state-remove

To Reproduce

comment: It would be very useful to have a reporting format output with one record per line. Like the lxc list options.

zsysctl show | head -n 15
Name:           rpool/ROOT/ubuntu_xu2lsl
ZSys:           true
Last Used:      current
History:        
  - Name:       rpool/ROOT/ubuntu_kxfody
    Created on: 2021-07-22 11:13:06
  - Name:       rpool/ROOT/ubuntu_0fqcdr
    Created on: 2021-07-22 10:28:20
  - Name:       rpool/ROOT/ubuntu_g3snm5
    Created on: 2021-07-22 09:18:06
  - Name:       rpool/ROOT/ubuntu_g3snm5@autozsys_e0beh0
    Created on: 2021-07-21 13:46:03
  - Name:       rpool/ROOT/ubuntu_g3snm5@autozsys_hzhsg1
    Created on: 2021-07-21 13:41:33
  - Name:       rpool/ROOT/ubuntu_g3snm5@autozsys_lx3tdn

I would like to delete the following named state:

  - Name:       rpool/ROOT/ubuntu_g3snm5@autozsys_hzhsg1
    Created on: 2021-07-21 13:41:33

following the instructions I get:

% sudo zsysctl state remove hzhsg1
ERROR couldn't remove user state hzhsg1: Couldn't find state: no matching state for hzhsg1

I also tried, for want of better documentation:

% sudo zsysctl state remove 'rpool/ROOT/ubuntu_g3snm5@autozsys_hzhsg1'
ERROR couldn't remove user state rpool/ROOT/ubuntu_g3snm5@autozsys_hzhsg1: Couldn't find state: no matching state for rpool/ROOT/ubuntu_g3snm5@autozsys_hzhsg1 

Expected behavior
I expect sudo zsysctl state remove hzhsg1 to complete successfully, and for the named state not to appear next time I execute: zsysctl show

For ubuntu users, please run and copy the following:

I f you need the bug report I will email it on request. I don't want to put the content on a public forum.

1. ubuntu-bug zsys --save=/tmp/report
2. Copy paste below /tmp/report content:

Installed versions:

% cat '/etc/os-release'
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
...

% zsysctl version 
zsysctl 0.4.8
zsysd   0.4.8

Additional context
related issues #204 #208

[ideologysec]

does sudo zsysctl state remove --system hzhsg1 work?

(I have somewhat of the opposite problem; --system should remove the root and identically named user states, but does not)

[pjgoodall]

The particular snapshot I mention seems to have gone. I now have keeplast: 0 in my zsys.conf file.
I tried sudo zsysctl state remove --system yud2qo. This worked
I'm suspecting that I did the documentation/blog as written, and missed the --system parameter.

Is there a way of resetting the configuration to its installation defaults?

[dxcSithLord]

I had this same problem - not enough room in bpool to create a snapshot,but upgrades were working.
The problem was that zfs keeps the old kernels in the snapshots, but bpool, by default is not big enough to handle the default of 20 snapshots of different upgrades for the kernel. This can be resolved by creating /etc/zsys.conf and entering the history settings, available from this github, or the blog at https://didrocks.fr/2020/06/04/zfs-focus-on-ubuntu-20.04-lts-zsys-state-collection/.
In my case I had 18 snapshots, so the garbage collection didn't do anything,because the default is to keep the last 20. As no more could be created, this is a stable state (which is not a good state).
By reducing the keeplast down to 10, then running
sudo zsysctl service gc
This resolved my space problem.
To identify the problem, I used:
zpool list -r bpool -t snap
which lists how much space each snapshot used and also how much data it refers to.
zpool list bpool
will show the space available, which does not include snapshots.
I had 18 snapshots:
NAME USED AVAIL REFER MOUNTPOINT
bpool/BOOT/ubuntu_l3g9y7@autozsys_8mgn83 80K - 283M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_z80dj8 80K - 283M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_izpa76 80K - 283M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_wp8u2n 163M - 379M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_kb7rwz 72K - 191M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_ysxh5t 72K - 191M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_dfirvr 0B - 191M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_f0pzbi 0B - 191M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_wr5zkm 0B - 191M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_o18m6m 72K - 191M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_500lim 0B - 191M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_tcpl14 0B - 191M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_f4n8uo 252M - 287M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_873rci 167M - 384M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_wyysrc 96K - 193M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_88t86h 88K - 289M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_bnx720 0B - 386M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_8onazk 0B - 386M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_2plbti 0B - 193M -
bpool/BOOT/ubuntu_l3g9y7@autozsys_dzjytk 0B - 193M -

Above 10 the garbage collection will be able to act, but this does not prevent having more than 10. With more than 10, the history definitions take effect, so it kept the oldest snapshots, but dropped snapshots that exceeded the number per bucket.

I went from:
zpool list bpool
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1.88G 1.59G 287M - - 28% 85% 1.00x ONLINE -

to

bpool 1.88G 1.43G 451M - - 32% 76% 1.00x ONLINE -

ZSYS defaults appear to be better suited to server configurations (where most expected to be used), but does allow tuning. Perhaps the zsys install could put the /etc/zsys.conf file in place with comments showing the defaults and suggestions for machines that are not on 24/7.

[ngoeddel-openi]

While doing an apt upgrade I got different error messages because initramfs-tools could not be build and it also was not possible to create system snapshots.
So I started googling and found this open issue from 3 years ago.

I have the same issue and I am not able to delete snapshots. I somehow have 38 user snapshots and 7 private and 7 root snapshots. I tried to delete the oldest ones using zsysctl state remove iy00xb and zsysctl state remove --system iy00xb but it didn't work. Without --system it told me that there is no such state. With --system it did something and updated the GRUB menu but the state was still there afterwards. And the only thing that worked was to use sudo zsysctl state remove iy00xb.

However my bpool remained untouched and was the main reason why the system state could not be saved because of too less available space. I don't know how to fix that right now.

More and more I am thinking that it was a bad idea to use ZFS for my Ubuntu installation. I may should consider completely reinstalling it with good old ext4...

[dxcSithLord]

See my comment above regards bpool, which should help.

[ngoeddel-openi]

I had to remove some older kernels and remove old snapshots in the bpool using zfs destroy bpool/BOOT/ubuntu_ppjin2@autozsys_XXXXXX.
So I listed my snapshots with this command: zfs list -t snapshot -o name,creation -s creation -r bpool
Then I deleted the oldest ones until I had enough space free on /boot again. Then I deleted some old kernels, then I deleted the older snapshots again, then I fixed broken installs and if that did not work because of too less space available I deleted even more snapshots.
After while back and forth I finally was able to install everything properly again that was broken. Apparently a bpool of 2,1 GiB is not enough for just a few kernels and 6 snapshots. Well. I can not change anything about that easily again, since I also have encrypted everything.