Differences

This shows you the differences between two versions of the page.

--- infrastructure:nixos-boxes [2025/12/21 07:44] – created diamond
+++ infrastructure:nixos-boxes [2026/01/08 09:12] (current) – move bitwarden-sops out into a page diamond
@@ Line 1: / Line 1: @@
 # NixOS Boxes
+A major part of dma.space's infrastructure stack are the NixOS boxes.
+As the name describes, these boxes run NixOS, but they're also managed using GitOps. Their code lives in [dma/infra/nixos](https://codeberg.org/dma/infra/tree/main/nixos). Any commits pushed to `main` will automatically get deployed within 10 seconds to all machines.
+## Pages
+```dokuwiki
+<catlist infrastructure:nixos-boxes: -noHead>
+```
+## Structure
+- `nixos/`
+  - `machines/` - NixOS machine configurations.
+  - `modules/` - Reusable NixOS modules.
+    - `sanity.nix` - Sane defaults for all NixOS machines. For now, these are
+      server-oriented, so if you're using a NixOS box for IoT-adjacent purposes,
+      you won't need to import this module.
+  - `Justfile` - recipe file for common operations (see
+    [Common Operations](#common-operations)).
+## Onboarding Yourself
+Currently, these are the major steps to onboarding yourself to manage these
+machines:
+. Gain access to these machines over the network, ideally over Tailscale.
+   Currently, the point of contact for Tailscale is @diamond, but this may
+   change once we have our own Headscale instance.
+. Be added into `./modules/ssh/public_keys.txt`. This also requires a
+   re-deployment from an existing administrator.
+. Set up your local environment. See
+   [Setting Up Your Local Environment](#setting-up-your-local-environment).
+### Setting Up Your Local Environment
+You will need at least [Nix][nix] installed on your local machine. Optionally,
+install [Direnv][direnv] as well for easier environment variable management.
+If you're just using Nix, you can enter a shell with all dependencies by
+running:
+```sh
+nix develop
+```
+If you're using Direnv, create a `.envrc` file under the `nixos` directory (this
+directory) with the following content:
+```sh
+use flake
+```
+> **Tip:**
+>
+> You may have a need to save secret environment variables such as Bitwarden
+> session keys in your `.envrc`. For this reason, consider adding it to your
+> local gitignore by running:
+>
+> `echo ".envrc" >> "$(git rev-parse --show-toplevel)/.git/info/exclude"`
+## Deployment
+All NixOS boxes are automatically deployed with GitOps practices. The following diagram briefly explains its architecture:
+```dokuwiki
+<div centeralign>
+<mermaid>
+    flowchart TD
+        codeberg[dma/infra @ main]
+        codeberg --[pull every 15s]--> comin_a
+        codeberg --[pull every 15s]--> comin_b
+        subgraph nixos_machine_a[NixOS Machine A]
+            nixos_rebuild_a[nixos-rebuild]
+            comin_a[comin.service] --> nixos_rebuild_a
+        end
+        subgraph nixos_machine_b[NixOS Machine B]
+            nixos_rebuild_b[nixos-rebuild]
+            comin_b[comin.service] --> nixos_rebuild_b
+        end
+</mermaid>
+</div>
+```
+Whenever a new commit is pushed to the `main` branch, all machines race to deploy this new commit. This way, we achieve GitOps without requiring an actual CI/CD pipeline, simplifying the deployment pipeline by a lot.
+To see how caught up the deployments are on the NixOS fleet, run `just watch`. This spawns a dashboard of machines and their commits which refreshes every 5 seconds. Use this immediately after pushing to comprehensively view how the machines are going along.
+### Testing Branches
+[comin](https://github.com/nlewo/comin), the GitOps tool we use, supports using testing branches to try out configurations without persisting them to the next reboot.
+To use this, simply push a new commit to a new `testing-{machine}` branch, where `{machine}` is the name of the machine. For example, a commit that updates `home-assistant-one`'s configuration would be pushed to `testing-home-assistant-one`.
+Once everything is tested to be working, you can simply `git checkout main` then `git rebase testing-{machine}` to add your new commits on top. Then, push to main as usual.
+## Common Operations
+Most, if not all, common operations can be performed via the `just` command. To
+see a list of available recipes, run:
+```sh
+just
+```
+## Playbooks
+This section contains useful playbooks for managing these machines.
+### Setting Up a New Machine
+This section will not go into the detailed steps of installing NixOS on a new
+machine, but rather the recommendations for setting it up:
+#### Creating a configuration.nix
+Create a `machines/<machine>/configuration.nix` manually. For most machines
+(especially common Thinkcentre models), it should be enough to just import
+`nixos-hardware` directly.
+After creating this file, add it to the local `flake.nix`, under
+`.nixosConfigurations.<machine>`. **Use `self.lib.nixosSystem`** unless you
+have a good reason not to.
+#### Disk Partitioning
+Create a `machines/<machine>/disko.nix` containing the disk layout using
+[disko][disko].
+**It is strongly recommended to use full disk encryption via LUKS + Btrfs**,
+unless you are absolutely sure the machine will not store any secrets or run
+any secure workloads. You may refer to existing NixOS machines for disko
+examples.
+Don't forget to add the file to the local `flake.nix`, under
+`.diskoConfigurations.<machine>` as well. Prefer `self.lib.diskoConfiguration`
+for this.
+#### Installing NixOS Quickly
+Install NixOS directly by flashing it onto the hard drive using a SATA-to-USB or NVME-to-USB adapter. This will save you the trouble of having to transfer this repository over to the machine's live environment.
+If you're inside the Nix develop environment, you can run:
+```sh
+sudo disko-install \
+  --mode format \
+  --disk '<machine-disk>' '/dev/sdXY' \
+  --flake "path://$PWD#<machine>"
+```
+> **Note:** Use `path://$PWD` rather than `.` since `disko` has buggy Flake subdirectory support.
+> **Note:** This doesn't set the root password. If you're one of the infra admins, you should have your SSH keys inside `./modules/ssh/public_keys.txt` and SSH directly into the box's LAN IP to continue provisioning.
+If you're doing this from the NixOS live environment, and you want to install directly into the same machine, you can run:
+```sh
+nix run 'nixpkgs#disko' -- --mode destroy,format,mount --flake '.#<machine>'
+nixos-install --no-channel-copy --flake '.#<machine>'
+```
+#### Re-using Modules
+For convenience, import the few core modules: `network`, `sanity`, and `ssh`.
+See existing `configuration.nix` files for reference.
+#### Secure Boot
+After installation, set up Secure Boot and TPM2 unlocking as soon as you can.
+The `secureboot` module should aid you in this process, but refer to
+[Lanzaboote][lanzaboote]'s existing documentation for more details:
+- [Prepare Your System](https://github.com/nix-community/lanzaboote/blob/master/docs/getting-started/prepare-your-system.md)
+- [Enable Secure Boot](https://github.com/nix-community/lanzaboote/blob/master/docs/getting-started/enable-secure-boot.md)
+### Adding Bitwarden Secrets to a Machine
+Moved to [a separate playbook](/infrastructure/playbook/bitwarden-sops-nix).