A major part of dma.space's infrastructure stack are the NixOS boxes.

As the name describes, these boxes run NixOS, but they're also managed using GitOps. Their code lives in dma/infra/nixos. Any commits pushed to main will automatically get deployed within 10 seconds to all machines.

• home-assistant-one
• nixos-aux-1
• dreamflasher

• nixos/
  • machines/ - NixOS machine configurations.
  • modules/ - Reusable NixOS modules.
    • sanity.nix - Sane defaults for all NixOS machines. For now, these are server-oriented, so if you're using a NixOS box for IoT-adjacent purposes, you won't need to import this module.
  • Justfile - recipe file for common operations (see Common Operations).

Currently, these are the major steps to onboarding yourself to manage these machines:

1. Gain access to these machines over the network, ideally over Tailscale. Currently, the point of contact for Tailscale is @diamond, but this may change once we have our own Headscale instance.
2. Be added into ./modules/ssh/public_keys.txt. This also requires a re-deployment from an existing administrator.
3. Set up your local environment. See Setting Up Your Local Environment.

You will need at least Nix installed on your local machine. Optionally, install Direnv as well for easier environment variable management.

If you're just using Nix, you can enter a shell with all dependencies by running:

nix develop

If you're using Direnv, create a .envrc file under the nixos directory (this directory) with the following content:

use flake

Tip:

You may have a need to save secret environment variables such as Bitwarden session keys in your .envrc. For this reason, consider adding it to your local gitignore by running:

echo ".envrc" >> "$(git rev-parse --show-toplevel)/.git/info/exclude"

Most, if not all, common operations can be performed via the just command. To see a list of available recipes, run:

just

A few notable recipes include:

• just deploy <machine> - Deploy the given machine. Machines are assumed to have the hostname of <machine> but this may change in the future.
• just ssh <machine> - SSH into the given machine. Machine hostnames follow the above rule.
• just sync-bitwarden-secrets <machine> - Sync Bitwarden secrets for the given machine. See Adding Bitwarden Secrets to a Machine.

This section contains useful playbooks for managing these machines.

This section will not go into the detailed steps of installing NixOS on a new machine, but rather the recommendations for setting it up:

Create a machines/<machine>/configuration.nix manually. For most machines (especially common Thinkcentre models), it should be enough to just import nixos-hardware directly.

After creating this file, add it to the local flake.nix, under .nixosConfigurations.<machine>. Use self.lib.nixosSystem unless you have a good reason not to.

Create a machines/<machine>/disko.nix containing the disk layout using [disko][disko].

It is strongly recommended to use full disk encryption via LUKS + Btrfs, unless you are absolutely sure the machine will not store any secrets or run any secure workloads. You may refer to existing NixOS machines for disko examples.

Don't forget to add the file to the local flake.nix, under .diskoConfigurations.<machine> as well. Prefer self.lib.diskoConfiguration for this.

Install NixOS directly by flashing it onto the hard drive using a SATA-to-USB or NVME-to-USB adapter. This will save you the trouble of having to transfer this repository over to the machine's live environment. These two commands will be very handy:

nix run 'nixpkgs#disko' -- --mode destroy,format,mount --flake '.#<machine>'
nixos-install --no-channel-copy --flake '.#<machine>'

For convenience, import the few core modules: network, sanity, and ssh. See existing configuration.nix files for reference.

After installation, set up Secure Boot and TPM2 unlocking as soon as you can. The secureboot module should aid you in this process, but refer to [Lanzaboote][lanzaboote]'s existing documentation for more details:

• Prepare Your System
• Enable Secure Boot

If you're here, chances are you're already added into the administration group for Bitwarden users. If not, please reach out to an existing administrator first to be added before continuing.

Before continuing, it is strongly advised that you have Nix and Direnv installed. This will allow you to store your Bitwarden secrets much easier.

First, set the BITWARDENCLI_APPDATA_DIR environment variable to prevent the CLI from using or overriding your personal Bitwarden configuration. It is strongly recommended to set this in .envrc so you don't forget:

$ cat .envrc
export BITWARDENCLI_APPDATA_DIR="$HOME/.config/bitwarden-dma-space"
...

Then, you'll need to obtain your Bitwarden API key. For this, follow the Bitwarden official documentation. You may choose to add these as environment variables as well:

$ cat .envrc
export BW_CLIENTID="your-client-id"
export BW_CLIENTSECRET="your-client-secret"

Now, proceed to run bw unlock. This will prompt you for your master password and return a session key which contains the unlocked vault. You may choose to add this key as well:

$ cat .envrc
export BW_SESSION="your-session-key"

Doing this is not recommended however, since it exposes your session key (and therefore essentially your entire vault) in plaintext on disk, and the key will never expire. Instead, prefer exporting it manually in the current shell session.

Head to the Bitwarden web app or extension, then navigate to the Server Credentials/NixOS Machines collection. Here, you will find 1 secret item per machine.

To add a secret to a machine, open the corresponding item, then add a new hidden field with the name being the SOPS path you want to store the secret at relative to /run/secrets.

For example, do add a secret at /run/secrets/authentik/secret_key, you would add a new hidden field with the name authentik/secret_key and the value being the value of the secret.

This step only needs to be done once per machine. To validate that a machine is ready for SOPS, ensure it has the sops.* options in its configuration.nix.

If not, start by referring to sops-nix's step 3 of Usage Examples. Essentially, you need to do the following steps:

1. Grab the machine's age key from its host SSH key using ssh-to-age.
2. Add it to vars.nix under <machine>.sops.hostPubKey.
3. Find the Bitwarden secret ID. There are 2 ways to do this:
   • Using just get-bitwarden-secret-id <machine>, which will match a secret with the exact name given, but this is not guaranteed to be in the correct collection.
   • Using the &itemId=<UUID> value when you click on the secret item in the Bitwarden web app.
4. Add it to vars.nix under <machine>.sops.bitwardenSecretID.

Then, add the boilerplate snippet to the machine's configuration.nix:

{
  sops = {
    defaultSopsFile = ./secrets.bitwarden.yaml;
    age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
  };
}

First, make sure that the local Bitwarden vault is up to date by running bw sync.

Then, run just sync-bitwarden-secrets <machine> to synchronize the secrets from Bitwarden to the ./machines/<machine>/secrets.bitwarden.yaml file. The SOPS file will automatically be generated with the host SSH key being the only decrypting recipient.

Having more than just the host's recipient key is not recommended. Instead, prefer regenerating the secret file from source Bitwarden if needed. This way, the secrets are always up to date with Bitwarden.

You may use the secrets in your machine like any other sops-nix secrets. For example:

sops.secrets."authentik/secret_key" = {
  owner = "authentik";
};

This will place the secret at /run/secrets/authentik/secret_key with the owner being the authentik user.

Deploy the machine using just deploy <machine> to push the updated secrets to the machine.