Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addition of granular bcachefs formatting and multi-disk support #961

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ProjectInitiative
Copy link
Contributor

These changes provide advanced usage of bcachefs formatting options. Because bcachefs provides
a one-stop-shop for wiping, partitioning, formatting and mounting with unique features for
each step, it has been broken out into it's own type if users wish to go beyond a single
disk setup.

examples/bcachefs.nix has been updated to showcase this advanced usage and some of the
available options.

Because all three processes take place via the type object, requiring "disks" attribute set
is not longer a hard requirement if a bcachefs type is provided. A check has been added to
account for that use case.

I will note that I have not gone through the official testing process called out in the contributing guidelines quite yet. I will looking into this soon, just wanted to get this PR out there for visibility :)

I have tested provisioning a VM with a 3 disk pool with this branch using nixos-anywhere, and also running the disko tool manually on another, already provisioned VM.

@ProjectInitiative ProjectInitiative force-pushed the master branch 2 times, most recently from b2b7b6d to 9dabcb4 Compare February 8, 2025 16:14
@ProjectInitiative
Copy link
Contributor Author

I am planning on re-working this to follow the mdadm and zfs arch with mult-drive support, it is proving to be a good challenge.

@Mic92
Copy link
Member

Mic92 commented Feb 12, 2025

Looking forward to it.

@ProjectInitiative
Copy link
Contributor Author

@Lassulus You mentioned wanting to take a look at the current state. This is very rough, but this is what I currently have.

I added in some markers in the individual _create scripts (BCACHEFS POSITION and BCACHEFS_MEMBER POSITION) within bcachefs.nix and bcachefs_member.nix to track the order in which they get collated. When I load it into repl and look at the top level _create function, I see them out of order (member should come first). I am running into an order of ops issue (I think) where I am not properly tracking that disks tagged with bcachefs_member need to be processed first.

 nixosConfigurations.testmachine.config.disko.devices._create
"set -efux\n\ndisko_devices_dir=$(mktemp -d)\ntrap 'rm -rf \"$disko_devices_dir\"' EXIT\nmkdir -p \"$disko_devices_dir\"\n\n( # bcachefs pool1   /mnt/pool #\n  declare -a formatOptions=('--compression=zstd')\n  declare -a mountOptions=(verbose degraded)\n  mountpoint=/mnt/pool\n  name=pool1\n  type=bcachefs\n  echo \"Creating bcachefs device: $device\" >&2\n  \n  echo BCACHEFS POSITION\n
...
...
...

Curious your thoughts. Thanks

@ProjectInitiative
Copy link
Contributor Author

ProjectInitiative commented Feb 23, 2025

Linking the following for visibility:
koverstreet/bcachefs-tools#308
systemd/systemd#8234

@ProjectInitiative
Copy link
Contributor Author

Some additional pitfalls discovered while testing with real hardware:

  • Using nixos-anywhere fails because when installing on systems with kernels < 6.13, UUID installations fail. A "nofail" mount would need to be added to all installs which seems hacky and not suitable for people running older kernels.
  • Encryption won't work out of the box as I have not found (yet) a good way to implement the interactive passphrase required at setup even though mounting options allow for key-files and passphrase files. Not sure why initial provisioning does not support this.

Successful deployments are possible, but require additional "helpers" external to this project. I am not sure this is a feasible implementation at this time. Might be good as an experimental feature with a write up on what is needed to supplement within a user's nixos config.

This addition attempts to integrate with the current arch of supporting partitioning.

examples/bcachefs-multi-disk.nix has been added to showcase this advanced usage and some of the
available options.

Current limitations and disko changes to allow for compatibility:
* lib/types/gpt.nix was updated to generate a deterministic UUID based on the partition name if one is not provided. Throughout testing, partition labels were not creating correctly.
  * This should not have an effect on other, other than changing the default behavior to favor UUID over partition labels
* Testing mostly succeeds. Multi-disk root setups are not working due to some bcachefs and systemd issues that are currently open. These issues are linked under _config of lib/types/bcachefs.nix
  * Because of this, the fileSystems nix config is not working on reboots due to the FS UUID presenting as available even if not all disks are initialized yet. A systemd-mount alternative has been provided.
  * The final reboot test might need to be omitted.
* Bcachefs works the best with the latest (or kernel 6.13 or new) boot.kernelPackages = config.pkgs.linuxPackages_latest; should be added to the _config section for systems, but pkgs is not provided in this module.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants