Skip to content

Monitoring

Current State

Deployed via Docker on two VMs, plus a native (non-Docker) node-exporter on dell-mini, and Caddy metrics from certs-01. The Infrastructure Overview dashboard has been rebuilt; an updated repository dashboard JSON now includes a proposed Caddy row, but its import/rendering in live Grafana has not yet been verified. Related work is tracked in issues #3–#8, #43 for dell-mini specifically.

Actual Stack

graph TD
    subgraph "gha-general-01 (192.168.10.189)"
        NE1[node-exporter]
        CB1[cAdvisor :8080]
        AL1[Grafana Alloy :12345]
        R1[2x GHA runner containers\ndfair / kc]
        BK[Buildkit builder]
        EX[Runner job exporter :9119]
    end

    subgraph "monitor-01 (192.168.10.158)"
        NE2[node-exporter]
        CB2[cAdvisor :8080]
        AL2[Grafana Alloy :12345]
        PROM[Prometheus :9090]
        LOKI[Loki :3100]
        GF[Grafana :3000]
        KUMA[Uptime Kuma :3001]
    end

    subgraph "certs-01 (LXC 106; 192.168.10.50)"
        CADDY[Caddy :443 / metrics :9180]
    end

    subgraph "dell-mini (192.168.10.117)"
        NE3[node-exporter\nnative systemd, no Docker]
    end

    NE1 --> AL1
    CB1 --> AL1
    EX -->|direct scrape :9119| PROM
    AL1 -->|remote_write| PROM
    NE2 --> PROM
    CB2 --> PROM
    NE3 -->|direct scrape| PROM
    CADDY -->|metrics scrape :9180| PROM
    CADDY -->|HTTPS proxy| GF
    CADDY -->|HTTPS proxy| KUMA
    AL2 --> LOKI
    PROM --> GF
    LOKI --> GF

Lenovo CI runner observability (deployed 2026-11-03)

The physical lenovo-lap (192.168.10.193, VLAN 10) has two working Docker-based GitHub Actions runners. The independent monitoring deployment is documented under lenovo-observability/. It includes native node-exporter (:9100), cAdvisor (:8080) and a read-only runner job/container observer (:9119) to be scraped by the existing Prometheus on monitor-01 (192.168.10.158). The Grafana panels cover host CPU/RAM/disk, runner container state and current inferred job. Uptime Kuma includes an SSH/TCP port 22 check and separate HTTP keyword checks for both runner containers; Telegram delivery still needs a test.

Deployment verified by operator on 2026-11-03: the dedicated nftables metrics firewall was loaded on Lenovo; native node-exporter was installed and reached from monitor-01, the Python GitHub runner exporter returned gha_runner_container_running=1 and observer metrics for both runners, and cAdvisor served container CPU metrics. Prometheus on monitor-01 was validated with promtool, reloaded using HUP, and confirmed the three new jobs (lenovo-lap-node, lenovo-lap-containers, lenovo-lap-runners) UP, bringing the observed total from ten to thirteen active scrape targets. The user also configured an SSH/TCP 22 monitor in Uptime Kuma.

2026-11-03 exporter verification: Following an exporter-only restart, gha_runner_container_stats_up=1 for both kc_backend and kc_infra. Observed host-relative CPU was 0.00000% / 0.03333% and host-relative RAM 1.06468% / 1.82642%, respectively. The updated Grafana panels were reported to display host-percentage values. The exporter remained active on TCP 9119 and Prometheus scraped it every 15 seconds with HTTP 200. These are point-in-time readings, not workload benchmarks.

Operator reports: the revised Grafana dashboard layout appears correct and the host-percentage runner panels show data. Two additional Uptime Kuma HTTP keyword checks were added, one per runner; their heartbeat status and Telegram notifications have not been independently verified.

Not yet verified: persistent nftables rules after a reboot, negative tests from unauthorized Lab clients, and Telegram DOWN/recovery notification delivery on the new Kuma monitors. Do not equate a healthy container with authoritative GitHub runner online status. Deployment and testing guidance: Lenovo runbook. Versioned dashboard: Infrastructure Overview.

Monitoring validation closure — issue #104 (2026-11-03)

Operator-reported complete: The operator confirmed the full #104 acceptance checklist after the Lenovo observability deployment in PRs #113 and #114. The following checklist records that confirmation; it is not a fresh independent capture of service configuration or alert logs.

  • [x] Grafana 13 Scenes/V2 Infrastructure Overview dashboard imported and rendered; Lenovo, Caddy, and GPU panels reviewed. Lenovo's backend/infra CPU and memory panels display percentages of host capacity.
  • [x] Active GitHub Actions job observations checked: running jobs appear, and completed jobs clear back to idle in the log-derived observer and Grafana. This does not prove GitHub API-backed queued/success/failed status or durations; #7 remains separate.
  • [x] NAS node-exporter added to monitoring and its Prometheus scrape verified, per operator. The NAS endpoint, scrape-job name, and live configuration diff were not supplied in this session and should be captured during a future configuration export; do not invent them here.
  • [x] Certificate-expiration monitoring and GPU outage/fallback alerting configured and tested, per operator. Exact expressions, alert thresholds, and delivery routing were not independently recorded here; retain the live rules/configuration as the authoritative source until exported.
  • [x] Uptime Kuma HTTPS monitors validated with the homelab internal CA, without bypassing TLS validation, per operator. Earlier LLM endpoint monitors accepted HTTP 401 as evidence of reachability/auth enforcement, not inference correctness.
  • [x] Telegram DOWN/recovery notifications exercised for the monitoring checks, per operator. The Lenovo SSH check and the separate backend/infra HTTP-keyword checks were configured earlier; none checks authoritative GitHub runner registration.

Evidence boundary: Node/cAdvisor/runner metrics, Prometheus scrape health (13 targets before the subsequently reported NAS addition), and host-relative Lenovo CPU/RAM measurements were observed directly in terminal output earlier on 2026-11-03. The remaining checklist items above are reported completed by the operator, with no raw validation artifacts supplied for this documentation change. Avoid asserting a new total scrape-target count, NAS address, alert-rule definitions, or notification timestamps without live output.

Components (as deployed)

Container Host Port Role
grafana monitor-01 3000 Visualization — Infrastructure Overview dashboard has been rebuilt
prometheus monitor-01 9090 Metrics store. Not 3001 — that's Kuma (see below, was documented wrong)
uptime-kuma monitor-01 3001 2.5.7; health checks, CA trust, Telegram DOWN/recovery notification integration via dedicated Mullvad Tinyproxy. See Telegram proxy.
loki monitor-01 3100 Log aggregation
alloy monitor-01 + gha-general-01 12345 Unified telemetry collector — ships node-exporter/cAdvisor data to Prometheus/Loki
cadvisor monitor-01 + gha-general-01 8080 Per-container CPU/mem/network metrics
node-exporter monitor-01 + gha-general-01 — Host-level CPU/mem/disk metrics
node_exporter (native, not a container) dell-mini 9100 Host-level CPU/mem/disk metrics — scraped directly by Prometheus, no Alloy (dell-mini doesn't run Docker)
dfair-gha-runner-local, kc-gha-runner-local gha-general-01 — The 2 active self-hosted GitHub Actions runners; PL retired 2026-10-22
gha-runner-job-exporter.service (systemd/Python) gha-general-01 9119 Read-only Docker runner log observer; exported metrics for current job names
buildx_buildkit_builder-* gha-general-01 — Buildkit builder for the runners

Fixed (was issue #3): all scrape targets were still pointed at pre-migration 192.168.1.x IPs (proxmox's target was literally still 192.168.1.197, its old flat-network address) — a direct side effect of the VLAN migration earlier this session. Fixed by updating /opt/monitoring/prometheus/prometheus.yml on monitor-01 to the current 192.168.10.x addresses and restarting the prometheus container. All 7 configured targets (prometheus, monitor-01-node/-containers, gha-general-01-node/-containers, proxmox-node, dell-mini-node) are now healthy.

Caddy and internal HTTPS monitoring (2026-10-20)

Caddy runs on certs-01 (LXC 106, 192.168.10.50); full configuration and CA handling are documented in Internal HTTPS / Caddy. Prometheus runs in Docker on monitor-01 and reads /opt/monitoring/prometheus/prometheus.yml (bind-mounted read-only to /etc/prometheus/prometheus.yml). Added one scrape job alongside existing jobs:

  - job_name: "caddy"
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - "192.168.10.50:9180"

promtool check config passed, the lifecycle reload succeeded, and the target API reported health: up with up{job="caddy"} = 1 and no scrape error. Existing seven targets remain configured; eight targets are now configured including Caddy. Caddy exposes upstream health and HTTP metrics. The Prometheus API was rechecked on 2026-10-20: up{job="caddy"} = 1, and both caddy_reverse_proxy_upstreams_healthy series (192.168.10.158:3000 and 192.168.10.158:3001) returned 1. The values show successful scraping and Caddy-reported upstream state, not verified end-to-end application health.

Uptime Kuma 2.5.7 runs in Docker Compose at /opt/monitoring/compose.yml on monitor-01; upgraded from 1.23.17 on 2026-10-25 with existing monitors/history migrated and the persistent named uptime-kuma-data volume retained. A checked pre-migration backup is on monitor-01 at /opt/monitoring/backups/kuma-v1-20261025-130228/uptime-kuma-v1-data.tar.gz (plus Compose backup). Image is pinned to louislam/uptime-kuma:2.5.7, not latest.

  uptime-kuma:
    image: louislam/uptime-kuma:2.5.7
    dns:
      - 192.168.10.1
    environment:
      NODE_EXTRA_CA_CERTS: /etc/homelab-ca/root.crt
      NOTIFICATION_PROXY: http://192.168.10.171:3128
    volumes:
      - uptime-kuma-data:/app/data
      - /opt/monitoring/certs/homelab-root-ca.crt:/etc/homelab-ca/root.crt:ro

Telegram notification provider Homelab Telegram Alerts is configured and applied to existing monitors; the user received a successful Telegram Test and actual downtime/recovery testing succeeded once the test monitor's notification was explicitly enabled. This notification proxy is a dedicated HTTPS CONNECT service on CT 107 with Mullvad egress; monitor probes do not use that proxy simply because NOTIFICATION_PROXY is set. New monitors still need the notification enabled, or the provider's Default Enabled option where available. See Telegram proxy. The temporary alert test monitor should be deleted when no longer needed. Prometheus/Grafana alerts remain a separate task.

# On monitor-01
cd /opt/monitoring
docker compose config -q
docker inspect uptime-kuma --format '{{.State.Status}}'
docker exec uptime-kuma printenv NOTIFICATION_PROXY
# Avoid printing notification configuration or bot tokens.

Update (2026-11-03): the operator reports NAS (LXC 105) node-exporter instrumentation and Prometheus scraping are complete. The specific target address/job was not included in the supplied verification output; inspect the live Prometheus target inventory before reproducing this configuration.

GitHub Actions monitoring scope decision (2026-11-03)

Final intended scope — no further GitHub Actions API exporter planned. The existing local Docker-log observers and Prometheus/Grafana metrics meet the operator's monitoring needs. Keep container status, exporter health, log-inferred current-job labels, CPU/RAM capacity-relative gauges, and Kuma availability/Telegram checks as the maintained observability surface.

Existing coverage includes gha-general-01 runner log observations and lenovo-lap backend/infrastructure runner monitoring. The operator confirms that this implementation is sufficient and considers issue #7 finished at this scope. The earlier issue proposed API-derived queue, success/failure, job duration, and originating-repository telemetry; those features are intentionally out of scope, not implemented or silently represented as available. No authenticated GitHub API polling, extra exporter, or additional dashboard expansion is planned. Revisit only for a separately approved requirement. Distinguish a running Docker container and log-derived job inference from authoritative GitHub online and job-result state.

GitHub Actions runner observability (2026-10-22)

The deployed gha-general-01 runner inventory is KC + DFAIR only. Runner operations, maintenance, systemd safety overrides, and nightly source-update checks are documented in Self-hosted GitHub Actions runners. PL was removed from Docker and its dedicated systemd unit was disabled and removed on 2026-10-22; its old pl-gha-runner.service alias is masked. Removing PL from GitHub's organization-level runner registration is still not independently verified. The retired runner must not be reintroduced by backup restore or service automation.

Runner Private source repo Container Service
KC KC-GHA-Runner kc-gha-runner-local gha-runner-kc.service
DFAIR dfair_runner dfair-gha-runner-local gha-runner-dfair.service

The repos are private. Their exact GitHub URLs are excluded from the unauthenticated lychee link checker in .lycheeignore. Other links remain checked as usual.

A Python HTTP exporter runs as gha-runner-job-exporter.service on gha-general-01 (192.168.10.189:9119) and reads local Docker runner logs. Its repository copy and systemd unit are under gha_runner_job_exporter.py and gha-runner-job-exporter.service. It exposes:

  • gha_runner_job_observer_up{runner="kc|dfair|kc_timing"}: 1 when Docker logs can be read; not proof GitHub has registered or marked the runner online.
  • gha_runner_job_observer_initialized: whether a recognizable log marker was seen.
  • gha_runner_current_job{runner="...",job="..."} 1: inferred unfinished job; absent when no active job is inferred. Grafana maps absence to None running when the observer is healthy.

This observer does not use GitHub APIs or expose the originating repository. It can miss very short jobs or infer stale status if log history rolls over. Update (2026-11-03): the operator reports validating an active job and its subsequent idle transition in Prometheus/Grafana for issue #104. Do not confuse None running with a failed exporter: first inspect the observer metrics and actual docker logs.

The Prometheus config on monitor-01 is bind-mounted from /opt/monitoring/prometheus/prometheus.yml and includes:

  - job_name: gha-runner-jobs
    scrape_interval: 15s
    static_configs:
      - targets:
          - '192.168.10.189:9119'

On 2026-10-22 promtool check config passed, Prometheus was HUP-reloaded, curl from monitor-01 reached the exporter, and the Prometheus API returned gha_runner_job_observer_up=1 for both kc and dfair. On 2026-10-27 the VM-local exporter additionally returned observer and initialized values of 1 for kc_timing; the user reports the Prometheus target was configured, but its new series in the Prometheus API have not yet been independently verified. This verifies the exporter-to-Prometheus path, not a running-job capture or alert notification. Limit port 9119 to the monitoring host; it has no authentication and Docker access held by the exporter service is privileged.

Diagnostic commands

On gha-general-01:

sudo systemctl status gha-runner-job-exporter.service --no-pager
curl -fsS http://192.168.10.189:9119/metrics
docker logs --tail 30 kc-gha-runner-local
docker logs --tail 30 dfair-gha-runner-local
docker logs --tail 30 kc-timing-runner-local

On monitor-01:

sudo docker exec prometheus promtool check config /etc/prometheus/prometheus.yml
curl -fsSG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=gha_runner_job_observer_up' | jq '.data.result'

The versioned Infrastructure Overview dashboard JSON is a Grafana 13 Scenes-based JSON, based on the supplied dashboard and updated with three runners, current-job stat panels, and corrected single-value container-health queries. CPU and RAM gauges use actual host capacity; network gauges show throughput without an assumed link maximum. Because KC uses host networking, its per-container network metric may be absent. The user reported importing/updating a Grafana dashboard with the job panels showing None running while KC and DFAIR latest observed jobs had completed. Import of this exact version-controlled revision into live Grafana has not been independently re-exported and compared. Treat live dashboard synchronization as a verification step.

Existing Caddy scrape (:9180) and other infrastructure panels are retained. The KC-Timing runner was observed Online / Idle in GitHub on 2026-10-27, and its exporter produces kc_timing series. The updated dashboard adds a KC-Timing container-status panel and a current-job stat; the operator now reports the live dashboard and running-job transitions verified (2026-11-03). Authoritative GitHub API runner/job lifecycle metrics remain outside #104 and are tracked under #7. Related tracking: #7 (job-level monitoring), #8 (router telemetry).

Alerting — Telegram deployed; Prometheus/Grafana follow-up

Telegram DOWN/recovery notifications in Uptime Kuma 2.5.7 were deployed and exercised 2026-10-25 via CT 107; the operator reports revalidating the new monitoring checks on 2026-11-03. The operator also reports certificate expiry and GPU outage/fallback alerts complete under #104. The live alert expressions and routing were not supplied and must not be reconstructed from these notes.

Additional alert ideas (configuration not inventoried here)

Alert Condition Severity
Host down Node exporter unreachable >2m Critical
High CPU >90% for >10m Warning
High RAM >85% Warning
Disk full >80% on any mount Warning
GHA runner container down cAdvisor shows container not running Warning

The thresholds in this table are examples, not proof of currently configured native rules. The operator reports GPU and certificate alerting complete; whether a separate Alertmanager or Grafana-native routing pipeline exists has not been independently inventoried. Kuma has its own tested DOWN/recovery notification path.

Prometheus outage and recovery — 2026-10-22 to 2026-10-25

Prometheus was found Exited (0) on 2026-10-25, while Grafana and Uptime Kuma remained healthy. The Prometheus log recorded a graceful SIGTERM at 2026-10-22 20:34:33 UTC. Host journal and package logs confirmed a user-initiated apt upgrade -y on 2026-10-22 at 16:33 EDT installed Docker packages and restarted containerd and Docker at 16:34 EDT. All other monitoring containers recovered, but Prometheus stayed stopped despite its unless-stopped restart policy; the exact non-restart mechanism remains unverified. Kuma reported ECONNREFUSED to 192.168.10.158:9090, and Grafana queries failed until recovery.

On monitor-01, sudo docker start prometheus restored service. /-/healthy and /-/ready endpoints returned healthy/ready, Prometheus up query reported 9/9 targets UP, and Grafana dashboards displayed data again. Docker and containerd are both enabled at boot; Prometheus retains restart policy unless-stopped. Historical samples from the stopped period cannot be regenerated. This incident was not caused by the later WireGuard renaming and relay changes.

For future Docker package upgrades, perform a post-upgrade sudo docker ps -a, check each critical container, query http://127.0.0.1:9090/-/ready and review Kuma. Do not indiscriminately recreate containers or discard volumes.

llm-01 NVIDIA GPU telemetry and endpoint monitoring (2026-10-29)

Live checks completed: Both Caddy HTTPS LLM endpoints https://llm.home.arpa/v1/models and https://llm-long.home.arpa/v1/models have green Uptime Kuma monitors. Monitoring uses unauthenticated HTTP with 401 as the accepted response, establishing reachability/auth enforcement, not authenticated inference or model correctness. To allow Kuma from monitor-01 (192.168.10.158), the @trusted remote_ip matcher in both Caddy virtual hosts additionally permits this single IP; the Trusted VLAN and HP 192.168.10.147 entries remain. Do not broadly enable VLAN 10. As of 2026-11-03, the operator confirms HTTPS CA validation and notification testing for the monitoring checks; exact per-monitor notification settings were not exported.

On llm-01 (192.168.10.162), the NVIDIA Quadro P4000 has an independent read-only nvidia-smi-exporter.service on 192.168.10.162:9835/metrics, running via a systemd dynamic user. The versioned exporter and clean service unit are at gpu-monitoring/. Runtime tests returned HTTP 200 with GPU utilization, VRAM used/total bytes, temperature, power, and aggregate compute process count. Two resident llama.cpp processes consumed 7,201 MiB of 8,192 MiB at measurement. This exporter does not indicate which process owns how much memory and is not an inference readiness check.

llm-01 /etc/nftables.conf retains the existing Caddy-only restrictions on TCP 8080/8081 and adds, after loopback accept:

meta nfproto ipv6 tcp dport 9835 drop
ip saddr != 192.168.10.158 tcp dport 9835 drop

This allows remote IPv4 scrapes from monitor-01 only; input policy remains ACCEPT for other ports. The firewall was syntax-checked and reloaded, and monitor-01 successfully scraped /metrics afterward. Tests from disallowed remote sources were not independently recorded.

Prometheus is the prometheus Docker container on monitor-01. The actual bind-mounted configuration is /opt/monitoring/prometheus/prometheus.yml (container path /etc/prometheus/prometheus.yml). An additional job was deployed:

  - job_name: "llm-01-gpu"
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets: ["192.168.10.162:9835"]
        labels:
          host: "llm-01"
          role: "gpu-inference"

curl -X POST http://127.0.0.1:9090/-/reload succeeded, target health became up with no scrape error (~29.5 ms), and PromQL returned 7.03 GiB used VRAM. docker exec prometheus promtool check config /etc/prometheus/prometheus.yml returned SUCCESS. This makes 10 active targets at verification. The dashboard's new GPU row is tracked in infra-overview.json; see dashboard import and validation before deploying it.

# On monitor-01
curl -fsSG --data-urlencode 'query=up{job="llm-01-gpu"}' \
  http://127.0.0.1:9090/api/v1/query
curl -fsS http://192.168.10.162:9835/metrics
docker exec prometheus promtool check config /etc/prometheus/prometheus.yml

Updated 2026-11-03: The operator reports GPU outage/fallback alerts and live Grafana UI rendering validated under #104; actual alert expressions and live dashboard export were not supplied with this documentation update. Qwen3 8B and Qwen3.5 2B were concurrently resident at the earlier measurement; monitoring cannot by itself demonstrate Codex-to-local fallback or adaptive model swapping.