Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking: systemd hardening in NixOS #377827

Open
18 tasks
LordGrimmauld opened this issue Jan 29, 2025 · 7 comments
Open
18 tasks

Tracking: systemd hardening in NixOS #377827

LordGrimmauld opened this issue Jan 29, 2025 · 7 comments
Assignees
Labels
1.severity: security Issues which raise a security issue, or PRs that fix one 5. scope: tracking Long-lived issue tracking long-term fixes or multiple sub-problems 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: systemd

Comments

@LordGrimmauld
Copy link
Contributor

LordGrimmauld commented Jan 29, 2025

Idea

Systemd allows for hardening services using service config options documented in man systemd.exec

Currently, these hardening features are barely used, with even a minimal installation of NixOS having many many services running completely without confinement. This issue is my attempt to track progress in this effort, make related issues and PRs searchable, and generally improve the situation.

Methodology

  • Install NixOS with few system services
  • run systemd-analyze security
  • select an important service that has bad confinement
  • improve it by setting systemd.services.<name>.serviceConfig.<hardening options>
  • if possible and feasible, run (and potentially add) nixos tests to ensure correct functionality
  • open PRs adding these improvements to the service modules already existing within nixos.

The goal is to have these hardening features sensible and stable enough to "just work". There should be no need for an opt-in toggle that is as inaccessible as the weirdness with current hardened profile. NixOS should be (more) secure by default.

Help wanted

Testing these things is hard, and some of these changes are likely to cause regressions by accident. While ideally all the PRs see testing before a merge, it is naive to assume no mistakes will happen. Please report regressions (potentially linking this tracking issue) if bad confinements make their way into unstable.

To help prevent bad changes ending up in unstable, please test and review PRs linked in this list. Yes, these will be module update PRs. However, testing is relatively simple: it is sufficient to just add the hardening options to systemd.services.<name>.serviceConfig explicitly on a local config.

Progress and related PRs/Issues

This list is intended to be updated as more PRs and Issues pop up.

@LordGrimmauld LordGrimmauld added 1.severity: security Issues which raise a security issue, or PRs that fix one 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: systemd labels Jan 29, 2025
@LordGrimmauld LordGrimmauld self-assigned this Jan 29, 2025
@FliegendeWurst FliegendeWurst added the 5. scope: tracking Long-lived issue tracking long-term fixes or multiple sub-problems label Jan 29, 2025
@newAM
Copy link
Member

newAM commented Jan 30, 2025

I have a lot of local hardening that I need to upstream. Here's a few of easier ones.

@0x4A6F 0x4A6F changed the title Tracking: Systemd hardening in Nixos Tracking: systemd hardening in NixOS Jan 30, 2025
@dasJ
Copy link
Member

dasJ commented Jan 30, 2025

We have this downstream module that may make things easier. It sets the most strict variant per-unit in a versioned manner so more security options can be added later without breaking existing units. These options can be reverted by just assigning them, for example serviceConfig.PrivateNetwork = false.

{ config, lib, ... }:
let
  # mkDefault is 1000
  # assignment is 100
  # This ensures that service defaults are overwritten, while
  # giving the administrator a chance to just override by assigning.
  mkDefaulter = lib.mkOverride 999;
in
{
  options = {
    systemd.services = lib.mkOption {
      type = lib.types.attrsOf (
        lib.types.submodule (
          { config, ... }:
          {
            options = {
              sandbox = lib.mkOption {
                description = ''
                  Level of confinement for this particular unit.

                  0 is no confinement at all.
                  2 is the current state of confinement. Requires systemd >= 245.
                '';
                default = 0;
                type = lib.types.enum [
                  0
                  2
                ];
              };

              allowImplicitRoot = lib.mkOption {
                description = "Whether or not the fact that this unit is running as root should be ignored.";
                type = lib.types.bool;
                default = false;
                example = true;
              };
            };

            config = lib.mkMerge [
              (lib.mkIf (config.sandbox >= 2) {
                serviceConfig = lib.mapAttrs (_: mkDefaulter) {
                  # Filesystem stuff
                  ProtectSystem = "strict"; # Prevent writing to most of /
                  ProtectHome = true; # Prevent accessing /home and /root
                  PrivateTmp = true; # Give an own directory under /tmp
                  PrivateDevices = true; # Deny access to most of /dev
                  ProtectKernelTunables = true; # Protect some parts of /sys
                  ProtectControlGroups = true; # Remount cgroups read-only
                  RestrictSUIDSGID = true; # Prevent creating SETUID/SETGID files
                  PrivateMounts = true; # Give an own mount namespace
                  RemoveIPC = true;
                  UMask = "0077";

                  # Capabilities
                  CapabilityBoundingSet = ""; # Allow no capabilities at all
                  NoNewPrivileges = true; # Disallow getting more capabilities. This is also implied by other options.

                  # Kernel stuff
                  ProtectKernelModules = true; # Prevent loading of kernel modules
                  SystemCallArchitectures = "native"; # Usually no need to disable this
                  ProtectKernelLogs = true; # Prevent access to kernel logs
                  ProtectClock = true; # Prevent setting the RTC

                  # Networking
                  RestrictAddressFamilies = ""; # Example: "AF_UNIX AF_INET AF_INET6"
                  PrivateNetwork = true; # Isolate the entire network

                  # Misc
                  LockPersonality = true; # Prevent change of the personality
                  ProtectHostname = true; # Give an own UTS namespace
                  RestrictRealtime = true; # Prevent switching to RT scheduling
                  MemoryDenyWriteExecute = true; # Maybe disable this for interpreters like python
                  PrivateUsers = true; # If anything randomly breaks, it's mostly because of this
                  RestrictNamespaces = true;
                };
              })
            ];
          }
        )
      );
    };
  };

  config.warnings =
    lib.mapAttrsToList
      (k: _: "${k}.service is running as implicit root, this should not be done in production.")
      (
        lib.filterAttrs (
          _: v:
          v.sandbox != 0
          && !v.allowImplicitRoot
          && !(v.serviceConfig ? User)
          && ((v.serviceConfig ? DynamicUser) -> !v.serviceConfig.DynamicUser)
        ) config.systemd.services
      )
    ++ lib.mapAttrsToList (k: _: "${k}.service has no system call filter.") (
      lib.filterAttrs (
        _: v: v.sandbox != 0 && !(v.serviceConfig ? SystemCallFilter)
      ) config.systemd.services
    );
}

Feel free to use the module, it's public domain now.

@LordGrimmauld
Copy link
Contributor Author

@dasJ this is actually an interesting input for #377958, versioning the hardening options. I'll have a think about it and might incorporate a similar idea!

@symphorien
Copy link
Member

Hi,
if you intend to push a large scale effort to generalize systemd hardening, can we agree on some guidelines like discouraging SystemCallFilter ? it is very very error prone. A quick search suggests it has caused countless issues:

#197443 is interesting: the value of SystemCallFilter was correct until a go update, and it broke. It seems very challenging to determine the correct value for SystemCallFilter.

There may be other frequently problematic options, but I mostly noticed this with systemcallfilter.

@LordGrimmauld
Copy link
Contributor Author

LordGrimmauld commented Feb 2, 2025

Hi, if you intend to push a large scale effort to generalize systemd hardening, can we agree on some guidelines like discouraging SystemCallFilter ? it is very very error prone. A quick search suggests it has caused countless issues:

* [nixos.matrix-appservice-irc: systemd hardening breaks ExecStartPre script #273929](https://github.com/NixOS/nixpkgs/issues/273929)

* [nixos.photoprism: Rotating images fails. #249120](https://github.com/NixOS/nixpkgs/issues/249120)

* [Grafana (with protocol = socket) dies after a few seconds #207824](https://github.com/NixOS/nixpkgs/issues/207824)

* [botamusique: too strict systemd hardening #205702](https://github.com/NixOS/nixpkgs/issues/205702)

* [sourcehut: go binaries (metasrht-api etc.) crash #199778](https://github.com/NixOS/nixpkgs/issues/199778)

* [nixos/paperless-ng: Unable to consume documents when classification model is present #164615](https://github.com/NixOS/nixpkgs/issues/164615)

* [github-runner: systemd hardening too restrictive #181573](https://github.com/NixOS/nixpkgs/issues/181573)

* [Go 1.19 binaries that use `@resources` SystemCallFilter crashing on startup due to SECCOMP failure #197443](https://github.com/NixOS/nixpkgs/issues/197443)

* [gitea startup is broken #74849](https://github.com/NixOS/nixpkgs/issues/74849)

* [Gitea v1.17.3: "159 bad syscall" when running systemd pre-exec script #199234](https://github.com/NixOS/nixpkgs/issues/199234)

* [logrotate: Crashes #345602](https://github.com/NixOS/nixpkgs/issues/345602)

#197443 is interesting: the value of SystemCallFilter was correct until a go update, and it broke. It seems very challenging to determine the correct value for SystemCallFilter.

There may be other frequently problematic options, but I mostly noticed this with systemcallfilter.

I very much disagree with this take. Yes, hardening will break stuff by accident. Either at introduction, or at some random update. But i don't think we should compromise on security for convenience here, if fixing these things is trivial.

You might persuade me to limit system call filters to groups (@system-service, @mount, @privileged), but not doing system call filters at all is a dangerous limitation.

That being said: the breakage should be obvious. Hence it makes sense to set SystemCallErrorNumber when doing system call filtering, as is proposed in https://github.com/NixOS/nixpkgs/pull/377958/files#diff-4d95b0a47116c34d3f39bfa6a87e2dc776bb8007858ae4fee861f605dbf61edaR109

Keep in mind: any update of a package that adds systemcalls is changing ABI and shouldn't really make it to stable in the first place. And any update to the hardening is potentially breaking and shouldn't make it to backports. So realistically, this should only ever affect unstable, and ideally the failures should be made loud. But compromising here is not really an option imo.

@06kellyjac
Copy link
Member

I think just a policy of doing these changes alongside the relevant nixpkgs package/module maintainers should be enough. People get some time to trial the changes before merging & then know to be on the look out for issues or PRs with fixes in the coming weeks.

And probably pausing this initiative before ZHF.

Also I'd say any additional hardening should probably be shared upstream so some if it could be included across more distros/releases and also gives an opportunity to get upstream maintainer input on potential issues before merging the change.

@Guanran928
Copy link
Contributor

Guanran928 commented Feb 4, 2025

nixos/sing-box: harden systemd service Would be happy for it to be reviewed :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1.severity: security Issues which raise a security issue, or PRs that fix one 5. scope: tracking Long-lived issue tracking long-term fixes or multiple sub-problems 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: systemd
Projects
None yet
Development

No branches or pull requests

7 participants