This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. # NixOS Boxes A major part of dma.space's infrastructure stack are the NixOS boxes. As the name describes, these boxes run NixOS, but they're also managed using GitOps. Their code lives in [dma/infra/nixos](https://codeberg.org/dma/infra/tree/main/nixos). Any commits pushed to `main` will automatically get deployed within 10 seconds to all machines. ## Pages ```dokuwiki <catlist infrastructure:nixos-boxes: -noHead> ``` ## Structure - `nixos/` - `machines/` - NixOS machine configurations. - `modules/` - Reusable NixOS modules. - `sanity.nix` - Sane defaults for all NixOS machines. For now, these are server-oriented, so if you're using a NixOS box for IoT-adjacent purposes, you won't need to import this module. - `Justfile` - recipe file for common operations (see [Common Operations](#common-operations)). ## Onboarding Yourself Currently, these are the major steps to onboarding yourself to manage these machines: 1. Gain access to these machines over the network, ideally over Tailscale. Currently, the point of contact for Tailscale is @diamond, but this may change once we have our own Headscale instance. 2. Be added into `./modules/ssh/public_keys.txt`. This also requires a re-deployment from an existing administrator. 3. Set up your local environment. See [Setting Up Your Local Environment](#setting-up-your-local-environment). ### Setting Up Your Local Environment You will need at least [Nix][nix] installed on your local machine. Optionally, install [Direnv][direnv] as well for easier environment variable management. If you're just using Nix, you can enter a shell with all dependencies by running: ```sh nix develop ``` If you're using Direnv, create a `.envrc` file under the `nixos` directory (this directory) with the following content: ```sh use flake ``` > **Tip:** > > You may have a need to save secret environment variables such as Bitwarden > session keys in your `.envrc`. For this reason, consider adding it to your > local gitignore by running: > > `echo ".envrc" >> "$(git rev-parse --show-toplevel)/.git/info/exclude"` ## Deployment All NixOS boxes are automatically deployed with GitOps practices. The following diagram briefly explains its architecture: ```dokuwiki <div centeralign> <mermaid> flowchart TD codeberg[dma/infra @ main] codeberg --[pull every 15s]--> comin_a codeberg --[pull every 15s]--> comin_b subgraph nixos_machine_a[NixOS Machine A] nixos_rebuild_a[nixos-rebuild] comin_a[comin.service] --> nixos_rebuild_a end subgraph nixos_machine_b[NixOS Machine B] nixos_rebuild_b[nixos-rebuild] comin_b[comin.service] --> nixos_rebuild_b end </mermaid> </div> ``` Whenever a new commit is pushed to the `main` branch, all machines race to deploy this new commit. This way, we achieve GitOps without requiring an actual CI/CD pipeline, simplifying the deployment pipeline by a lot. To see how caught up the deployments are on the NixOS fleet, run `just watch`. This spawns a dashboard of machines and their commits which refreshes every 5 seconds. Use this immediately after pushing to comprehensively view how the machines are going along. ### Testing Branches [comin](https://github.com/nlewo/comin), the GitOps tool we use, supports using testing branches to try out configurations without persisting them to the next reboot. To use this, simply push a new commit to a new `testing-{machine}` branch, where `{machine}` is the name of the machine. For example, a commit that updates `home-assistant-one`'s configuration would be pushed to `testing-home-assistant-one`. Once everything is tested to be working, you can simply `git checkout main` then `git rebase testing-{machine}` to add your new commits on top. Then, push to main as usual. ## Common Operations Most, if not all, common operations can be performed via the `just` command. To see a list of available recipes, run: ```sh just ``` ## Playbooks This section contains useful playbooks for managing these machines. ### Setting Up a New Machine This section will not go into the detailed steps of installing NixOS on a new machine, but rather the recommendations for setting it up: #### Creating a configuration.nix Create a `machines/<machine>/configuration.nix` manually. For most machines (especially common Thinkcentre models), it should be enough to just import `nixos-hardware` directly. After creating this file, add it to the local `flake.nix`, under `.nixosConfigurations.<machine>`. **Use `self.lib.nixosSystem`** unless you have a good reason not to. #### Disk Partitioning Create a `machines/<machine>/disko.nix` containing the disk layout using [disko][disko]. **It is strongly recommended to use full disk encryption via LUKS + Btrfs**, unless you are absolutely sure the machine will not store any secrets or run any secure workloads. You may refer to existing NixOS machines for disko examples. Don't forget to add the file to the local `flake.nix`, under `.diskoConfigurations.<machine>` as well. Prefer `self.lib.diskoConfiguration` for this. #### Installing NixOS Quickly Install NixOS directly by flashing it onto the hard drive using a SATA-to-USB or NVME-to-USB adapter. This will save you the trouble of having to transfer this repository over to the machine's live environment. If you're inside the Nix develop environment, you can run: ```sh sudo disko-install \ --mode format \ --disk '<machine-disk>' '/dev/sdXY' \ --flake "path://$PWD#<machine>" ``` > **Note:** Use `path://$PWD` rather than `.` since `disko` has buggy Flake subdirectory support. > **Note:** This doesn't set the root password. If you're one of the infra admins, you should have your SSH keys inside `./modules/ssh/public_keys.txt` and SSH directly into the box's LAN IP to continue provisioning. If you're doing this from the NixOS live environment, and you want to install directly into the same machine, you can run: ```sh nix run 'nixpkgs#disko' -- --mode destroy,format,mount --flake '.#<machine>' nixos-install --no-channel-copy --flake '.#<machine>' ``` #### Re-using Modules For convenience, import the few core modules: `network`, `sanity`, and `ssh`. See existing `configuration.nix` files for reference. #### Secure Boot After installation, set up Secure Boot and TPM2 unlocking as soon as you can. The `secureboot` module should aid you in this process, but refer to [Lanzaboote][lanzaboote]'s existing documentation for more details: - [Prepare Your System](https://github.com/nix-community/lanzaboote/blob/master/docs/getting-started/prepare-your-system.md) - [Enable Secure Boot](https://github.com/nix-community/lanzaboote/blob/master/docs/getting-started/enable-secure-boot.md) ### Adding Bitwarden Secrets to a Machine Moved to [a separate playbook](/infrastructure/playbook/bitwarden-sops-nix).