add option to netbox role for custom pipeline code for OIDC mapping

Add option to netbox role for ensuring custom pipeline code for OIDC
group and role mapping is either present or not.

The custom pipeline code is licensed under the Creative Commons: CC
BY-SA 4.0 license.

See:
https://github.com/goauthentik/authentik/blob/main/LICENSE
https://github.com/goauthentik/authentik/blob/main/website/integrations/services/netbox/index.md
https://docs.goauthentik.io/integrations/services/netbox/
5676b1a468

README.md

@@ -45,3 +45,8 @@ Im Ansible-Repo müssen diese Sachen hinzugefügt werden:
     * Individuelle Config für den Service. Wenn Docker Compose, hier weiterleiten auf den eigentlichen Dienst in Compose.
     * Cert-Dateinamen anpassen
 * `resources/chaosknoten/`*host*`/docker_compose/compose.yaml.j2`: Config für Docker Compose (wenn verwendet)
+
+## License
+
+This CCCHH ansible-ccchh repository is licensed under the [MIT License](./LICENSE).  
+[`custom_pipeline_oidc_group_and_role_mapping.py`](./roles/netbox/files/custom_pipeline_oidc_group_and_role_mapping.py) is licensed under the Creative Commons: CC BY-SA 4.0 license.

resources/chaosknoten/grafana/docker_compose/prometheus_alerts.rules.yaml

@@ -166,7 +166,7 @@ groups:
       # Longer intervals to account for disk intensive hypervisor tasks (backups, moving VMs, etc.).
       - alert: HypervisorHostUnusualDiskReadRate
         expr: (sum by (instance) (rate(node_disk_read_bytes_total[2m])) / 1024 / 1024 > 50) * on(instance) group_left (nodename) node_uname_info{nodename="chaosknoten"}
-        for: 60m
+        for: 90m
         labels:
           severity: warning
         annotations:
@@ -174,7 +174,7 @@ groups:
           description: "Disk is probably reading too much data (> 50 MB/s)\n  VALUE = {{ $value }}"
       - alert: HypervisorHostUnusualDiskWriteRate
         expr: (sum by (instance) (rate(node_disk_written_bytes_total[2m])) / 1024 / 1024 > 50) * on(instance) group_left (nodename) node_uname_info{nodename="chaosknoten"}
-        for: 60m
+        for: 90m
         labels:
           severity: warning
         annotations:
@@ -256,7 +256,7 @@ groups:
       # Since hard disks on the hypervisor can easily have their IO saturated by hypervisor tasks (backups, moving VMs, etc.), alert when the IO is above the regular threshold for a very long time.
       - alert: HypervisorHostUnusualHardDiskIo
         expr: (rate(node_disk_io_time_seconds_total{device=~"s.+"}[1m]) > 0.5) * on(instance) group_left (nodename) node_uname_info{nodename="chaosknoten"}
-        for: 50m
+        for: 90m
         labels:
           severity: warning
         annotations:

resources/chaosknoten/tickets/nginx/tickets.hamburg.ccc.de.conf

@@ -38,7 +38,7 @@ server {
 
     location = / {
         #return 302 https://wiki.hamburg.ccc.de/infrastructure:service-overview#tickets_pretix;
-        return 302 https://tickets.hamburg.ccc.de/hackertours/38c3/;
+        return 302 https://tickets.hamburg.ccc.de/hackertours/eh22/;
     }
 
     location / {

roles/netbox/README.md

@@ -0,0 +1,88 @@
+# `netbox` role
+
+A role for setting up NetBox.  
+It automatically pulls in all required dependencies like Redis and PostgreSQL, deploys the provided systemd services and gunicorn config and sets up a PostgreSQL database named `netbox` with an owner named `netbox` and the specified password.
+However providing the [NetBox configuration](#netbox-configuration), [setting up a web server like nginx to proxy to gunicorn](#web-server-setup) and tasks like creating users, etc. you have to do yourself.
+
+## Supported Distributions
+
+Should work on Debian-based distributions.
+
+## Required Arguments
+
+- `netbox__version`: The NetBox version to deploy.
+- `netbox__db_password`: The password to use for connection to the database.
+  This is required since the upgrade script runs as root and therefore peer authentication doesn't work.
+- `netbox__config`: The NetBox config to deploy.
+  See [NetBox Configuration](#netbox-configuration) for more infos.
+
+## Optional Arguments
+
+- `netbox__custom_pipeline_oidc_group_and_role_mapping`: Whether or not to have custom pipeline code for OIDC group and role mapping present.
+  See [Custom Pipeline Code for OIDC Group and Role Mapping](#custom-pipeline-code-for-oidc-group-and-role-mapping) for more infos.  
+  Defaults to `false`.
+
+## NetBox Configuration
+
+The NetBox configuration should include a connection to Redis as well as a connection to PostgreSQL.  
+Configuration for the Redis connection:
+
+```python
+REDIS = {
+    "tasks": {
+      "HOST": "localhost",
+      "PORT": 6379,
+      "USERNAME": "",
+      "PASSWORD": "",
+      "DATABASE": 0,
+      "SSL": False,
+    },
+    "caching": {
+      "HOST": "localhost",
+      "PORT": 6379,
+      "USERNAME": "",
+      "PASSWORD": "",
+      "DATABASE": 1,
+      "SSL": False,
+    },
+}
+```
+
+Configuration for the PostgreSQL connection:
+
+```python
+DATABASE = {
+  "HOST": "localhost",
+  "NAME": "netbox",
+  "USER": "netbox",
+  "PASSWORD": "<same as netbox__db_password>",
+}
+```
+
+Further configuration should take place. Some relevant resources can be found here:
+
+- Installation guide configuration docs: <https://netboxlabs.com/docs/netbox/en/stable/installation/3-netbox/#configuration>
+- Configuration docs: <https://netboxlabs.com/docs/netbox/en/stable/configuration/>
+- Example configuration: <https://github.com/netbox-community/netbox/blob/main/netbox/netbox/configuration_example.py>
+
+## Web Server Setup
+
+As this role just sets up gunicorn, but doesn't set up a web server, you need to do that yourself.  
+The relevant documentation on how to do that can be found here:
+
+- Web server setup docs: <https://netboxlabs.com/docs/netbox/en/stable/installation/5-http-server/>
+- Example base nginx config: <https://github.com/netbox-community/netbox/blob/main/contrib/nginx.conf>
+
+## Custom Pipeline Code for OIDC Group and Role Mapping
+
+Setting the option `netbox__custom_pipeline_oidc_group_and_role_mapping` to `true` makes this role ensure custom pipeline code for OIDC group and role mapping is present.
+Note that this role uses code for NetBox >= 4.0.0.  
+The code is available in `files/custom_pipeline_oidc_group_and_role_mapping.py`, licensed under the CC BY-SA 4.0 license and taken from [this authentik NetBox documentation](https://docs.goauthentik.io/integrations/services/netbox/).
+The documentation also shows how to use the pipeline code by defining a custom `SOCIAL_AUTH_PIPELINE`, which you also need to do, as the configuration isn't provided by this role.
+However instead of under `netbox.custom_pipeline.` the functions are available under `netbox.custom_pipeline_oidc_mapping.` with this role.
+See also [the default settings.py](https://github.com/netbox-community/netbox/blob/main/netbox/netbox/settings.py) for the default `SOCIAL_AUTH_PIPELINE`.
+
+## Links & Resources
+
+- The NetBox Git Repo: <https://github.com/netbox-community/netbox>
+- The NetBox installation docs: <https://netboxlabs.com/docs/netbox/en/stable/installation/>

roles/netbox/defaults/main.yaml

@@ -0,0 +1 @@
+netbox__custom_pipeline_oidc_group_and_role_mapping: false

roles/netbox/files/custom_pipeline_oidc_group_and_role_mapping.py

@@ -0,0 +1,55 @@
+# Licensed under Creative Commons: CC BY-SA 4.0 license.
+# https://github.com/goauthentik/authentik/blob/main/LICENSE
+# https://github.com/goauthentik/authentik/blob/main/website/integrations/services/netbox/index.md
+# https://docs.goauthentik.io/integrations/services/netbox/
+from netbox.authentication import Group
+
+class AuthFailed(Exception):
+    pass
+
+def add_groups(response, user, backend, *args, **kwargs):
+    try:
+        groups = response['groups']
+    except KeyError:
+        pass
+
+    # Add all groups from oAuth token
+    for group in groups:
+        group, created = Group.objects.get_or_create(name=group)
+        user.groups.add(group)
+
+def remove_groups(response, user, backend, *args, **kwargs):
+    try:
+        groups = response['groups']
+    except KeyError:
+        # Remove all groups if no groups in oAuth token
+        user.groups.clear()
+        pass
+
+    # Get all groups of user
+    user_groups = [item.name for item in user.groups.all()]
+    # Get groups of user which are not part of oAuth token
+    delete_groups = list(set(user_groups) - set(groups))
+
+    # Delete non oAuth token groups
+    for delete_group in delete_groups:
+        group = Group.objects.get(name=delete_group)
+        user.groups.remove(group)
+
+
+def set_roles(response, user, backend, *args, **kwargs):
+    # Remove Roles temporary
+    user.is_superuser = False
+    user.is_staff = False
+    try:
+        groups = response['groups']
+    except KeyError:
+        # When no groups are set
+        # save the user without Roles
+        user.save()
+        pass
+
+    # Set roles is role (superuser or staff) is in groups
+    user.is_superuser = True if 'superusers' in groups else False
+    user.is_staff = True if 'staff' in groups else False
+    user.save()

roles/netbox/handlers/main.yaml

@@ -0,0 +1,24 @@
+- name: Run upgrade script
+  ansible.builtin.command: /opt/netbox/upgrade.sh
+  become: true
+  # When it runs, this should always report changed.
+  changed_when: true
+
+- name: Ensure netbox systemd services are set up and up-to-date
+  ansible.builtin.systemd_service:
+    daemon_reload: true
+    name: "{{ item }}"
+    enabled: true
+    state: restarted
+  become: true
+  loop:
+    - "netbox.service"
+    - "netbox-rq.service"
+
+- name: Ensure netbox housekeeping timer is set up and up-to-date
+  ansible.builtin.systemd_service:
+    daemon_reload: true
+    name: "netbox-housekeeping.timer"
+    enabled: true
+    state: restarted
+  become: true

roles/netbox/meta/argument_specs.yaml

@@ -0,0 +1,16 @@
+argument_specs:
+  main:
+    options:
+      netbox__version:
+        type: str
+        required: true
+      netbox__db_password:
+        type: str
+        required: true
+      netbox__config:
+        type: str
+        required: true
+      netbox__custom_pipeline_oidc_group_and_role_mapping:
+        type: bool
+        required: false
+        default: false

roles/netbox/meta/main.yaml

@@ -0,0 +1,11 @@
+---
+dependencies:
+  - role: redis
+  - role: postgresql
+    vars:
+      postgresql__dbs:
+        - name: netbox
+          owner: netbox
+      postgresql__users:
+        - name: netbox
+          password: "{{ netbox__db_password }}"

roles/netbox/tasks/main.yaml

@@ -0,0 +1,124 @@
+- name: Ensure all dependencies are installed
+  ansible.builtin.apt:
+    name:
+      - python3
+      - python3-pip
+      - python3-venv
+      - python3-dev
+      - build-essential
+      - libxml2-dev
+      - libxslt1-dev
+      - libffi-dev
+      - libpq-dev
+      - libssl-dev
+      - zlib1g-dev
+      - git
+  become: true
+
+- name: Ensure NetBox source is present
+  ansible.builtin.git:
+    repo: https://github.com/netbox-community/netbox.git
+    dest: /opt/netbox/
+    version: "{{ netbox__version }}"
+  become: true
+  notify:
+    - Run upgrade script
+    - Ensure netbox systemd services are set up and up-to-date
+
+- name: Ensures custom pipeline code for OIDC group and role mapping is present
+  ansible.builtin.copy:
+    src: custom_pipeline_oidc_group_and_role_mapping.py
+    dest: /opt/netbox/netbox/netbox/custom_pipeline_oidc_mapping.py
+    mode: "0644"
+    owner: root
+    group: root
+  when: netbox__custom_pipeline_oidc_group_and_role_mapping
+  become: true
+  notify:
+    - Ensure netbox systemd services are set up and up-to-date
+
+- name: Ensures custom pipeline code for OIDC group and role mapping is not present
+  ansible.builtin.file:
+    path: /opt/netbox/netbox/netbox/custom_pipeline_oidc_mapping.py
+    state: absent
+  when: not netbox__custom_pipeline_oidc_group_and_role_mapping
+  become: true
+  notify:
+    - Ensure netbox systemd services are set up and up-to-date
+
+- name: Ensure netbox user
+  block:
+    - name: Ensure netbox group exists
+      ansible.builtin.group:
+        name: netbox
+        system: true
+      become: true
+
+    - name: Ensure netbox user exists
+      ansible.builtin.user:
+        name: netbox
+        group: netbox
+        password: '!'
+        system: true
+      become: true
+
+- name: Ensure relevant directories are owned by netbox user
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: netbox
+    recurse: true
+  become: true
+  loop:
+    - "/opt/netbox/netbox/media/"
+    - "/opt/netbox/netbox/reports/"
+    - "/opt/netbox/netbox/scripts/"
+
+- name: Deploy configuration.py
+  ansible.builtin.copy:
+    content: "{{ netbox__config }}"
+    dest: "/opt/netbox/netbox/netbox/configuration.py"
+    mode: "0644"
+    owner: root
+    group: root
+  become: true
+  notify: Ensure netbox systemd services are set up and up-to-date
+
+- name: Ensure provided gunicorn config is copied
+  ansible.builtin.copy:
+    remote_src: true
+    src: "/opt/netbox/contrib/gunicorn.py"
+    dest: "/opt/netbox/gunicorn.py"
+    mode: "0644"
+    owner: root
+    group: root
+  become: true
+  notify: Ensure netbox systemd services are set up and up-to-date
+
+- name: Ensure provided netbox systemd service files are copied
+  ansible.builtin.copy:
+    remote_src: true
+    src: "/opt/netbox/contrib/{{ item }}"
+    dest: "/etc/systemd/system/{{ item }}"
+    mode: "0644"
+    owner: root
+    group: root
+  become: true
+  loop:
+    - "netbox.service"
+    - "netbox-rq.service"
+  notify: Ensure netbox systemd services are set up and up-to-date
+
+- name: Ensure provided housekeeping systemd service and timer are copied
+  ansible.builtin.copy:
+    remote_src: true
+    src: "/opt/netbox/contrib/{{ item }}"
+    dest: "/etc/systemd/system/{{ item }}"
+    mode: "0644"
+    owner: root
+    group: root
+  become: true
+  loop:
+    - "netbox-housekeeping.service"
+    - "netbox-housekeeping.timer"
+  notify: Ensure netbox housekeeping timer is set up and up-to-date

roles/postgresql/README.md

@@ -0,0 +1,37 @@
+# Role `postgresql`
+
+Ensures `postgresql` is installed by installing the distributions package.  
+Also ensures the optionally given databases and users are set up as specified.
+
+## Supported Distributions
+
+Should work on Debian-based distributions.
+
+## Required Arguments
+
+None.
+
+## Optional Arguments
+
+- `postgresql__dbs`: List of databases with their owner to ensure are set up.  
+- `postgresql__dbs.*.name`: Name of the database.
+- `postgresql__dbs.*.owner`: Owner of the database.
+- `postgresql__users`: List of users to ensure are set up.
+- `postgresql__users.*.name`: Name of the user.
+- `postgresql__users.*.password`: Optional password for the user.
+   If left unset, the user will have no password set, but can still connect using [peer authentication](https://www.postgresql.org/docs/current/auth-peer.html) on the local system.
+   (Peer authentication works when a password is set as well.)
+
+## Example Arguments
+
+```yaml
+postgresql__dbs:
+  - name: netbox
+    owner: netbox
+  - name: foo
+    owner: bar
+postgresql__users:
+  - name: netbox
+    password: super_secret
+  - name: bar
+```

roles/postgresql/defaults/main.yaml

@@ -0,0 +1,2 @@
+postgresql__dbs: [ ]
+postgresql__users: [ ]

roles/postgresql/meta/argument_specs.yaml

@@ -0,0 +1,28 @@
+argument_specs:
+  main:
+    options:
+      postgresql__dbs:
+        type: list
+        elements: dict
+        required: false
+        default: [ ]
+        options:
+          name:
+            type: str
+            required: true
+          owner:
+            type: str
+            required: true
+      postgresql__users:
+        type: list
+        elements: dict
+        required: false
+        default: [ ]
+        options:
+          name:
+            type: str
+            required: true
+          password:
+            type: str
+            required: false
+            default: ""

roles/postgresql/tasks/main.yaml

@@ -0,0 +1,28 @@
+- name: Ensure postgresql is installed
+  ansible.builtin.apt:
+    name:
+      - postgresql
+  become: true
+
+- name: Ensure Python library for community.postgresql is installed if needed
+  ansible.builtin.apt:
+    name:
+      - python3-psycopg
+  become: true
+  when: postgresql__dbs != [ ] or postgresql__users != [ ]
+
+- name: Ensure users
+  community.postgresql.postgresql_user:
+    name: "{{ item.name }}"
+    password: "{{ item.password | default('') }}"
+  become: true
+  become_user: postgres
+  loop: "{{ postgresql__users }}"
+
+- name: Ensure dbs with owners
+  community.postgresql.postgresql_db:
+    name: "{{ item.name }}"
+    owner: "{{ item.owner }}"
+  become: true
+  become_user: postgres
+  loop: "{{ postgresql__dbs }}"

roles/redis/README.md

@@ -0,0 +1,15 @@
+# Role `redis`
+
+Ensures `redis` is installed by installing the distributions package.
+
+## Supported Distributions
+
+Should work on Debian-based distributions.
+
+## Required Arguments
+
+None.
+
+## Optional Arguments
+
+None.

roles/redis/tasks/main.yaml

@@ -0,0 +1,5 @@
+- name: Ensure redis is installed
+  ansible.builtin.apt:
+    name:
+      - redis
+  become: true