Update all stable non-major dependencies

A better guide can be found in the new docs.

README.md

@@ -44,22 +44,6 @@ Ein paar der Hosts haben den selben Namen, was es etwas schwieriger macht, das P
 ansible-playbook playbooks/deploy.yaml --inventory inventories/chaosknoten/hosts.yaml --limit public-reverse-proxy
 ```
 
-## Neuen Web-Service hinzufügen
-
-Wir deployen Web-Services auf dem Chaosknoten hinter dem `public-reverse-proxy`. Der Service-Name `service.hamburg.ccc.de` sollte ein CNAME für `public-reverse-proxy.hamburg.ccc.de` und die Service-VM unter `service-intern.hamburg.ccc.de` mit einer `172.31.17.x`-Adresse erreichbar sein.
-
-Im Ansible-Repo müssen diese Sachen hinzugefügt werden:
-* `inventories/chaosknoten/hosts.yaml`: 
-    * SSH-Config für die neue VM unter all/children/debian_12/hosts
-    * Einträge für die Rollen, die dieser Host haben soll, min. `certbot_hosts`, `nginx_hosts`, ggf. `docker_compose_hosts`
-* `inventories/chaosknoten/host_vars/`*host*`.yaml`: config vars für den neuen Host
-* `resources/chaosknoten/public-reverse-proxy/nginx/acme_challenge.conf`: Liste der Hostnamen um den neuen Host erweitern, die hinter dem Reverse-Proxy stehen
-* `resources/chaosknoten/public-reverse-proxy/nginx/nginx.conf`: Liste der Hostnamen um den neuen Host erweitern, die hinter dem Reverse-Proxy stehen
-* `resources/chaosknoten/`*host*`/nginx/`*host*`.hamburg.ccc.de.conf`: Server auf dem Service-Host
-    * Individuelle Config für den Service. Wenn Docker Compose, hier weiterleiten auf den eigentlichen Dienst in Compose.
-    * Cert-Dateinamen anpassen
-* `resources/chaosknoten/`*host*`/docker_compose/compose.yaml.j2`: Config für Docker Compose (wenn verwendet)
-
 ## License
 
 This CCCHH ansible-ccchh repository is licensed under the [MIT License](./LICENSE).  

docs/guides/create-new-web-service-vm.md

@@ -1,119 +0,0 @@
-# How to create all necessary entries for new (web service) VM
-
-Let's assume that you want to add a new web service `example.hamburg.ccc.de` which is going to be hosted on the VM `example` on chaosknoten. These are the steps that you need to take to create the VM and add it to the Ansible repo.
-
-## IP, DNS, VM
-
-1. Allocate a fresh [IPv6 in Netbox in the 2a00:14b0:42:102::/64 net](https://netbox.hamburg.ccc.de/ipam/prefixes/47/ip-addresses/). This will be the management address for the VM. 
-2. On `ns-intern`:
-   1. Add an entry `example.hosts.hamburg.ccc.de` as an AAAA pointing to the allocated IP.
-      To do this, edit the zonefile in `/etc/bind/master/hamburg.ccc.de.zone`.
-   2. Add an entry `example.hamburg.ccc.de` as a CNAME for `public-reverse-proxy` to the same zone.
-   3. Commit and reload the zone by running `sudo make` while inside `/etc/bind/master`.
-3. On Chaosknoten:
-   1. Create a new VM, for example by cloning the Debian template 9023.  
-      Give it the name `example`.
-   2. Edit the ethernet interface to be connected to `vmbr0`, VLAN tag `2`.
-      Good $genders also enable the proxmox firewall.
-   3. Configure the IPv6 address in the Cloud-Init section. Leave IPv4 set to DHCP.
-   4. Make sure the VM is started at boot (options).
-   5. Adjust any other VM parameters as needed.
-   6. Boot the VM.
-4. Add the [VM to Netbox](https://netbox.hamburg.ccc.de/virtualization/virtual-machines/).
-    - Make sure to enter the VM ID.
-    - Add an Ethernet interface to the VM; we typically use `eth0` as a name.
-    - Add IP for that interface, then choose "Assign IP" and search for the IP you've created. Make it the primary IP of that interface.
-
-## Ansible Basics
-
-As the first step, we need to make the host known to Ansible.
-
-1. In `.sops.yaml`, add an entry for the host. Follow the other entries there.
-   1. `keys.hosts.chaosknoten.age` needs an age public key (must be generated; the private key gets added later in the host-specific YAML)
-   2. `creation_rules` needs an entry for the host, referencing the age key.
-   3. Re-encrypt existing files with the new key (manly `group_var/all.sops.yaml`): `find inventories -name "*.sops.*" | xargs sops updatekeys --yes`
-2. In `inventories/chaosknoten/hosts.yaml`:
-   1. Configure basic connection info:
-        ```yaml
-        example:
-            ansible_host: example.hosts.hamburg.ccc.de
-            ansible_user: chaos
-            ansible_ssh_common_args: -J ssh://chaos@router.hamburg.ccc.de
-        ```
-        You typically will want to use router as a jump host so that you can run Ansible on an IPv4 only connection.
-    2. Add the host to the desired roles.
-       1. As a minimum, you'll want the following roles:
-          - `base_config_hosts`
-          - `infrastructure_authorized_keys_hosts`
-          - `ansible_pull_hosts`
-       2. For a typical web service based on Docker Compose, you'll also want:
-          - `docker_compose_hosts`
-          - `nginx_hosts`
-          - `certbot_hosts`.
- 3. In the directory `inventories/chaosknoten/host_var/`:
-    1. A file `inventories/chaosknoten/host_var/example.yaml` with the host/service specific configuration.
-    2. A file `inventories/chaosknoten/host_var/example.sops.yaml` with the encrypted secrets for the host/service. Run `sops inventories/chaosknoten/host_var/example.yaml` to edit/create that file. Entries here should generally be prefixed with `secret__` to make it easier to see where that variable is coming from in templates etc.
-       * Add an entry `ansible_pull__age_private_key` with the age private key you generated above.
-
-## Ansible-Pull Configuration
-
-Since you added your host to the `ansible_pull_hosts`, you also need to follow [SOPS: New Host](./sops-new-host.md) before continuing.
-
-## Service-specific config
-
-From here, we go into the details of the web service that you want to configure. For a typical web service with Docker Compose, you will likely want to configure the following.
-
-Make `inventories/chaosknoten/host_var/example.yaml` look like this:
-```yaml
-certbot__acme_account_email_address: le-admin@hamburg.ccc.de
-certbot__certificate_domains:
-  - "example.hamburg.ccc.de"
-certbot__new_cert_commands:
-  - "systemctl reload nginx.service"
-
-docker_compose__compose_file_content: "{{ lookup('ansible.builtin.template', 'resources/chaosknoten/example/docker_compose/compose.yaml.j2') }}"
-
-nginx__version_spec: ""
-nginx__configurations:
-  - name: example.hamburg.ccc.de
-    content: "{{ lookup('ansible.builtin.file', 'resources/chaosknoten/spaceapiccc/nginx/example.hamburg.ccc.de.conf') }}"
-```
-
-This will create `compose.yaml` from the template `resources/chaosknoten/example/docker_compose/compose.yaml.j2'`, and the nginx config from `resources/chaosknoten/spaceapiccc/nginx/example.hamburg.ccc.de.conf`. Of course, depending on your service, you might need additional entries. See the other hosts and the roles for more info.
-
-## First Ansible run
-
-Before you can run Ansible successfully, you will want to make sure you can connect to the VM, and that the host key has been added to your known hosts:
-* `ssh chaos@example.hosts.hamburg.ccc.de`
-* `ssh -J chaos@router.hamburg.ccc.de chaos@example.hosts.hamburg.ccc.de`
-
-Then run Ansible for `public-reverse-proxy` to add the necessary entries:
-
-```sh
-ansible-playbook playbooks/deploy.yaml --inventory inventories/chaosknoten/hosts.yaml --limit public-reverse-proxy
-```
-
-Finally run Ansible for the new host:
-
-```sh
-ansible-playbook playbooks/deploy.yaml --inventory inventories/chaosknoten/hosts.yaml --limit example
-```
-
-# Commit your changes
-
-Do not forget to commit your changes, whether it's a new host or you are making changes to an existing host.
-
-And always `git pull` before you run Ansible so avoid reverting anything!
-
-# Monitoring
-
-## Gatus (`status.hamburg.ccc.de`)
-
-After you configured a new service or website, add it to our status and uptime monitoring.  
-Take a look at the configuration in `resources/external/status/docker_compose/config` and extend it to cover the newly added service or website. The configuration should probably happen in either `services-chaosknoten.yaml` or `websites.yaml`. Taking the existing configuration as a reference should give guidance on how to configure new checks. Additionally there's also the comprehensive [Gatus Documentation](https://github.com/TwiN/gatus?tab=readme-ov-file#table-of-contents).
-
-After you've added some checks, the configuration can be deployed using:
-
-```sh
-ansible-playbook playbooks/deploy.yaml --inventory inventories/external --limit status 
-```

docs/guides/monitoring-gatus.md

@@ -0,0 +1,18 @@
+---
+title: "Monitoring: Gatus"
+summary: How to Setup Monitoring Using Gatus
+---
+
+# Monitoring: Gatus
+
+After configuring a new service or website, it should be added to our status and uptime monitoring available at [status.hamburg.ccc.de](https://status.hamburg.ccc.de).
+
+Take a look at the configuration in `resources/external/status/docker_compose/config` and extend it to cover the newly added service or website. The configuration should probably happen in either `services-chaosknoten.yaml` or `websites.yaml`.  
+Taking the existing configuration as a reference should give guidance on how to configure new checks.  
+Additionally, there's the comprehensive [Gatus Documentation](https://github.com/TwiN/gatus?tab=readme-ov-file#table-of-contents).
+
+After you've added some checks, the configuration can be deployed using:
+
+```
+ansible-playbook playbooks/deploy.yaml -i inventories/external -l status -t docker_compose
+```

docs/guides/new-chaosknoten-vm.md

@@ -0,0 +1,99 @@
+---
+title: New Chaosknoten VM
+summary: How to Create a New VM on Chaosknoten
+---
+
+# New Chaosknoten VM
+
+This guide outlines the steps to take for creating a new VM (`myservice`) on Chaosknoten. It might also act as a useful reference for other environments however.
+
+## IP and DNS
+
+Decide on what kind of network configuration the VM needs. In this guide we assume a public v6 and a non-public v4 in the v4-NAT network. If you have special requirements, refer to TODO for more information on the different kinds of networks available on Chaosknoten.  
+
+1. Allocate a fresh [v6 in NetBox in the v4-NAT v6-network](https://netbox.hamburg.ccc.de/ipam/prefixes/47/ip-addresses/).
+    - The _hostname_ should be the full FQDN, like: `myservice.hosts.hamburg.ccc.de`
+2. Add a new `AAAA` record pointing to the chosen v6 to the [`hamburg.ccc.de` zone](https://git.hamburg.ccc.de/CCCHH/ansible-infra/src/branch/main/resources/chaosknoten/auth-dns/zones/hamburg.ccc.de.zone).  
+  It should look something like this:
+  ```
+  myservice.hosts IN AAAA 2a00:14b0:42:102::ff
+  ```
+3. Run the Ansible playbook for the `auth-dns` host to publish the new DNS entry:  
+  ```
+  ansible-playbook -i inventories/chaosknoten playbooks/deploy.yaml -l auth-dns -t knot
+  ```
+
+## VM Creation
+
+Now that we have a v6, let's create a new VM on Chaosknoten.
+
+1. Clone the `chaos-debian-XX-tmpl-YYYY-MM-DD` template with ID `9023`.
+    - The _Name_ should be the hostname used earlier, e.g. `myservice`.
+    - The _Mode_ should be set to _Full Clone_ and the _Target Storage_ to _nvme0_.
+    - All other settings should be left as is.
+2. Once cloned, edit the VM settings as follows:
+    - The network interface for the VM should already be configured to use bridge _vmbr0_ with VLAN tag _2_, which is the correct configuration for the v4-NAT network used in this example.  
+      For other networks, other bridges or VLANs should be used.
+    - If there isn't a good reason to not do so, enable the Proxmox firewall for the VM. Firewall security groups exist for various common use cases to make the firewall configuration fast and easy. 
+    - In the _Cloud-init_ section, set the chosen static v6 with its gateway. As we're using the v4-NAT network in this example, the gateway would be `2a00:14b0:42:102::1`.  
+      For other networks, different network configuration must be set.
+    - In the _Options_ section, set _Start at Boot_ to _Yes_.
+    - Optionally configure other VM parameters.  
+      Especially the _Memory_, _Processors_ and _Hard Disk_ options might be of interest.
+3. Next start up the VM.
+    - Once it has an IP, you should be able to SSH into it using the hostname used in the DNS entry earlier (e.g. `myservice.hosts.hamburg.ccc.de`):
+      ```
+      ssh chaos@myservice.hosts.hamburg.ccc.de
+      ```
+      Or if no v6 connectivity is available:
+      ```
+      ssh -J chaos@router.hamburg.ccc.de chaos@myservice.hosts.hamburg.ccc.de
+      ```
+4. Finally add the VM to NetBox
+    1. First [add the VM itself](https://netbox.hamburg.ccc.de/virtualization/virtual-machines/).
+        - The _Name_ should be the hostname of the VM, so e.g. `myservice`.
+        - _Site_ and _Cluster_ should be set. For Chaosknoten that would be _IRZ42_ and _chaosknoten_.
+        - The _Proxmox VM ID_ should be set to the ID of the VM in Proxmox.
+        - All other options should be left as is.
+    2. Then add the interface of the VM to the newly created VM in NetBox by adding a component.
+        - The _Name_ should be the name of the interface of the VM. Most likely _net0_.
+        - The _Primary MAC Address_ should be set to the MAC address generated by Proxmox, which can be found in the _Network Device_ configuration in Proxmox.
+        - _802.1Q Switching_ should be set appropriately. As we're using the v4-NAT network in this example, the _802.1Q Mode_ would be _Access_ with the _VLAN group_ being _IRZ42 Intern_ and the _Untagged VLAN_ being _v4-NAT_.
+        - All other options should be left as is.
+    4. Finally assign the IP address created earlier in NetBox to the just created interface by going into the _Assignment_ section of the IP address configuration.
+        - Also check _Make this the primary IP for the device/VM_ while doing so.
+
+## Ansible
+
+Next up the VM should be added to the Ansible configuration management. While Ansible can and should be used to configure various services and configuration options, this section focuses on the basic setup to be done.
+
+1. [Setup SOPS for the new host](./sops-new-host.md).
+2. Add the VM to the inventory (e.g. `inventories/chaosknoten/hosts.yaml`):
+    1. First add basic connection info. For our host in the v4-NAT network, this would look like this:
+       ```yaml
+       myservice:
+         ansible_host: myservice.hosts.hamburg.ccc.de
+         ansible_user: chaos
+         ansible_ssh_common_args: -J ssh://chaos@router.hamburg.ccc.de
+       ```
+       Adding the router as a jump host enables us to run Ansible on a v4-only connection.
+    2. Add the host to a base set of inventory groups for a base configuration:
+        - `base_config_hosts`
+        - `infrastructure_authorized_keys_hosts`
+        - `ansible_pull_hosts`
+3. Run a first Ansible playbook against the new host to apply the basic configuration:
+   ```
+   ansible-playbook -i inventories/chaosknoten playbooks/deploy.yaml -l myservice
+   ```
+
+Feel free to commit at this point before continuing with further configuration.
+
+## Further Setup
+
+With the basic configuration now done, the VM can be set up further. Here are some helpful pointers:
+
+- Inventory variables for the host can be set in its host vars file, e.g. `inventories/chaosknoten/host_vars/myservice.yaml`.
+- Secrets should be stored using SOPS, see: [SOPS: Storing Secrets](./sops-storing-secrets.md)
+- Guidance on setting up a web service can be found in the [Web Service](./web-service.md) guide.
+
+Afterward you should also take care of setting up monitoring for the newly configured service, see: [Monitoring: Gatus](./monitoring-gatus.md)

docs/guides/web-service-setup.md

@@ -0,0 +1,68 @@
+---
+title: Web Service
+summary: How to Setup a Web Service
+---
+
+# Web Service
+
+This guide assumes you followed [New Chaosknoten VM](./new-chaosknoten-vm.md) for setting up a VM (`myservice`) in the v4-NAT network. It continues of that to set up a web service `mywebservice` behind our `public-reverse-proxy`.
+
+1. First, the `public-reverse-proxy` should be configured to make the upcoming web service reachable and have certbot work.
+    1. Start out by adding an entry to the `map` in the `stream` section in `playbooks/resources/chaosknoten/public-reverse-proxy/nginx/nginx.conf`, e.g.:
+    ```
+    stream {
+        # ...
+        map {
+            # ...
+            mywebservice.hamburg.ccc.de myservice.hosts.hamburg.ccc.de:8443;
+        }
+    }
+    ```
+    2. Next add an entry to the `map` in `playbooks/resources/chaosknoten/public-reverse-proxy/nginx/acme_challenge.conf`, e.g.:
+    ```
+    map $host $upstream_acme_challenge_host {
+        # ...
+        mywebservice.hamburg.ccc.de myservice.hosts.hamburg.ccc.de:31820;
+        default "";
+    }
+    ```
+    3. Finally apply the configuration by running the Ansible playbook for the `public-reverse-proxy`:
+    ```
+    ansible-playbook playbooks/deploy.yaml -i inventories/chaosknoten/hosts.yaml -l public-reverse-proxy -t public_reverse_proxy
+    ```
+2. Add a CNAME for the FQDN of the service pointing to the `public-reverse-proxy`. For a service `myservice` under `hamburg.ccc.de`, this would need an entry in the [`hamburg.ccc.de` zone](https://git.hamburg.ccc.de/CCCHH/ansible-infra/src/branch/main/resources/chaosknoten/auth-dns/zones/hamburg.ccc.de.zone) like this:
+   ```
+   mywebservice IN CNAME public-reverse-proxy
+   ```
+3. Next add the VM to the relevant inventory groups. For a web service running on docker compose, the following groups would be needed:
+    - `docker_compose_hosts`
+    - `nginx_hosts`
+    - `certbot_hosts`.
+4. Then configuration for the VM hosting the web service needs to be provided, which should look something like this:
+  ```yaml
+  # inventories/chaosknoten/host_vars/myservice.yaml
+  certbot__acme_account_email_address: le-admin@hamburg.ccc.de
+  certbot__certificate_domains:
+    - "mywebservice.hamburg.ccc.de"
+  certbot__new_cert_commands:
+    - "systemctl reload nginx.service"
+
+  docker_compose__compose_file_content: "{{ lookup('ansible.builtin.template', 'resources/chaosknoten/myservice/docker_compose/compose.yaml.j2') }}"
+
+  nginx__version_spec: ""
+  nginx__configurations:
+    - name: mywebservice.hamburg.ccc.de
+      content: "{{ lookup('ansible.builtin.file', 'resources/chaosknoten/myservice/nginx/mywebservice.hamburg.ccc.de.conf') }}"
+  ```
+  This would create a `compose.yaml` on the host from the template `resources/chaosknoten/example/docker_compose/compose.yaml.j2'` and an nginx configuration from `resources/chaosknoten/myservice/nginx/mywebservice.hamburg.ccc.de.conf`, so both files need to be filled accordingly.
+  Of course, depending on your service, you might need additional or different configuration.  
+  Generally you should look at the configuration of existing hosts and the provided roles for guidance.
+5. Finally configure the web service on the new host by running the Ansible playbook for it:
+   ```
+   ansible-playbook playbooks/deploy.yaml -i inventories/chaosknoten/hosts.yaml -l myservice 
+   ```
+
+## Additional Resources
+
+- For storing secrets using SOPS, see: [SOPS: Storing Secrets](./sops-storing-secrets.md)
+- After setting up the web service, you should also take care of setting up monitoring it, see: [Monitoring: Gatus](./monitoring-gatus.md)