A major part of dma.space's infrastructure stack are the NixOS boxes.

As the name describes, these boxes run NixOS, but they're also managed using GitOps. Their code lives in dma/infra/nixos. Any commits pushed to main will automatically get deployed within 10 seconds to all machines.

• home-assistant-one
• nixos-aux-1
• dreamflasher

• nixos/
  • machines/ - NixOS machine configurations.
  • modules/ - Reusable NixOS modules.
    • sanity.nix - Sane defaults for all NixOS machines. For now, these are server-oriented, so if you're using a NixOS box for IoT-adjacent purposes, you won't need to import this module.
  • Justfile - recipe file for common operations (see Common Operations).

Currently, these are the major steps to onboarding yourself to manage these machines:

1. Gain access to these machines over the network, ideally over Tailscale. Currently, the point of contact for Tailscale is @diamond, but this may change once we have our own Headscale instance.
2. Be added into ./modules/ssh/public_keys.txt. This also requires a re-deployment from an existing administrator.
3. Set up your local environment. See Setting Up Your Local Environment.

You will need at least [Nix][nix] installed on your local machine. Optionally, install [Direnv][direnv] as well for easier environment variable management.

If you're just using Nix, you can enter a shell with all dependencies by running:

nix develop

If you're using Direnv, create a .envrc file under the nixos directory (this directory) with the following content:

use flake

Tip:

You may have a need to save secret environment variables such as Bitwarden session keys in your .envrc. For this reason, consider adding it to your local gitignore by running:

echo ".envrc" >> "$(git rev-parse --show-toplevel)/.git/info/exclude"

All NixOS boxes are automatically deployed with GitOps practices. The following diagram briefly explains its architecture:

Whenever a new commit is pushed to the main branch, all machines race to deploy this new commit. This way, we achieve GitOps without requiring an actual CI/CD pipeline, simplifying the deployment pipeline by a lot.

To see how caught up the deployments are on the NixOS fleet, run just watch. This spawns a dashboard of machines and their commits which refreshes every 5 seconds. Use this immediately after pushing to comprehensively view how the machines are going along.

comin, the GitOps tool we use, supports using testing branches to try out configurations without persisting them to the next reboot.

To use this, simply push a new commit to a new testing-{machine} branch, where {machine} is the name of the machine. For example, a commit that updates home-assistant-one's configuration would be pushed to testing-home-assistant-one.

Once everything is tested to be working, you can simply git checkout main then git rebase testing-{machine} to add your new commits on top. Then, push to main as usual.

Most, if not all, common operations can be performed via the just command. To see a list of available recipes, run:

just

This section contains useful playbooks for managing these machines.

This section will not go into the detailed steps of installing NixOS on a new machine, but rather the recommendations for setting it up:

Create a machines/<machine>/configuration.nix manually. For most machines (especially common Thinkcentre models), it should be enough to just import nixos-hardware directly.

After creating this file, add it to the local flake.nix, under .nixosConfigurations.<machine>. Use self.lib.nixosSystem unless you have a good reason not to.

Create a machines/<machine>/disko.nix containing the disk layout using [disko][disko].

It is strongly recommended to use full disk encryption via LUKS + Btrfs, unless you are absolutely sure the machine will not store any secrets or run any secure workloads. You may refer to existing NixOS machines for disko examples.

Don't forget to add the file to the local flake.nix, under .diskoConfigurations.<machine> as well. Prefer self.lib.diskoConfiguration for this.

Install NixOS directly by flashing it onto the hard drive using a SATA-to-USB or NVME-to-USB adapter. This will save you the trouble of having to transfer this repository over to the machine's live environment.

If you're inside the Nix develop environment, you can run:

sudo disko-install \
  --mode format \
  --disk '<machine-disk>' '/dev/sdXY' \
  --flake "path://$PWD#<machine>"

Note: Use path://$PWD rather than . since disko has buggy Flake subdirectory support.

Note: This doesn't set the root password. If you're one of the infra admins, you should have your SSH keys inside ./modules/ssh/public_keys.txt and SSH directly into the box's LAN IP to continue provisioning.

If you're doing this from the NixOS live environment, and you want to install directly into the same machine, you can run:

nix run 'nixpkgs#disko' -- --mode destroy,format,mount --flake '.#<machine>'
nixos-install --no-channel-copy --flake '.#<machine>'

For convenience, import the few core modules: network, sanity, and ssh. See existing configuration.nix files for reference.

After installation, set up Secure Boot and TPM2 unlocking as soon as you can. The secureboot module should aid you in this process, but refer to [Lanzaboote][lanzaboote]'s existing documentation for more details:

• Prepare Your System
• Enable Secure Boot

Moved to a separate playbook.