The Challenge

3 hosts on a Tailscale mesh, 1 central monitoring stack

Goals

• Ship metrics (CPU, RAM, disk, network) from all hosts to central Prometheus
• Ship logs from all hosts to central Loki
• Monitor Tailscale itself (connection status, latency, DERP relay usage)
• All traffic over the encrypted Tailscale mesh — no public exposure

The Answer: Grafana Alloy

node-exporter   → Prometheus scrapes :9100
promtail        → pushes logs to Loki
                   (2 processes, 2 configs)

grafana-alloy   → pushes metrics via remote_write
                → pushes logs via loki.write
                   (1 process, 1 config)

Why Alloy Wins

• EOL Promtail dies March 2, 2026 — Alloy is the official replacement
• Built-in prometheus.exporter.unix = the node_exporter, inside Alloy
• Push-based — no firewall rules, no Prometheus scrape targets to maintain
• Native ARM64 binary — runs on the Pi 5
• One config file per host, uses constants.hostname for auto-labeling

Promtail is Dead

Migration Path

# Convert existing promtail config to Alloy format automatically
alloy convert --source-format=promtail --output=/etc/alloy/config.alloy promtail.yaml

Log Shipper Comparison

If the Pi ever gets RAM-constrained, Fluent Bit (10 MiB) for logs + bare node-exporter (20 MiB) is the ultra-lean fallback.

Alloy Resource Footprint

Official Estimates (from Grafana docs)

Comparison on Raspberry Pi 5 (8GB)

The tradeoff: ~3x more RAM for consolidation + future-proofing. On an 8GB Pi with 7.4 GiB free, 200 MiB is nothing.

Alloy Configuration

Complete remote host agent config

// /etc/alloy/config.alloy — drop on every remote host

// ═══ METRICS (replaces node-exporter) ═══
prometheus.exporter.unix "host" {
  set_collectors = ["cpu","meminfo","diskstats",
    "filesystem","netdev","loadavg","uname","processes"]
}

prometheus.scrape "host_metrics" {
  targets    = prometheus.exporter.unix.host.targets
  forward_to = [prometheus.relabel.add_host.receiver]
  scrape_interval = "30s"
}

prometheus.relabel "add_host" {
  forward_to = [prometheus.remote_write.central.receiver]
  rule {
    target_label = "host"
    replacement  = constants.hostname
  }
}

prometheus.remote_write "central" {
  endpoint {
    url = "http://100.x.x.4:9090/api/v1/write"
  }
}

// ═══ LOGS (replaces promtail) ═══
loki.source.journal "journal" {
  max_age    = "12h"
  forward_to = [loki.write.central.receiver]
  labels     = { host = constants.hostname, job = "journal" }
}

loki.write "central" {
  endpoint {
    url = "http://100.x.x.4:3100/loki/api/v1/push"
  }
  external_labels = { host = constants.hostname }
}

Tailscale Native Metrics

Built-in since Tailscale v1.78 — zero dependencies

Each Tailscale node exposes Prometheus metrics natively. Enable with:

tailscale set --webclient    # exposes :5252/metrics over the tailnet

Available Metrics

Path labels on throughput: direct_ipv4, direct_ipv6, derp — tells you if traffic is direct or relayed.

Tailscale Exporters

Fleet-level visibility via the Tailscale API

Pre-built Grafana Dashboards

• Dashboard 24177 — Tailscale Overview (fleet-wide)
• Dashboard 24178 — Tailscale Machine (per-device)
• Both from the tailscale-mixin in the adinhodovic repo

Pull vs Push Architecture

Why Hybrid?

Push for host metrics + logs (Alloy handles both, zero Prometheus config per host). Pull for Tailscale metrics (native :5252 endpoint, already exposed). Best of both worlds.

Security Model

Tailscale provides the security layer

• All traffic is WireGuard-encrypted end-to-end — no additional TLS needed between nodes
• Only enrolled devices on your tailnet can reach services
• For a 3-device personal homelab, no additional auth is required

Best Practices

Example Tailscale ACL

{
  "acls": [
    { "action": "accept",
      "src": ["100.x.x.4"],       // Mac Mini only
      "dst": ["*:9100", "*:5252"] }   // monitoring ports
  ]
}

Target Architecture

Prometheus Config Updates

Add to existing ~/monitoring-stack/prometheus/prometheus.yml

# Enable remote write receiver (add to docker-compose command)
# command: '--web.enable-remote-write-receiver'

scrape_configs:
  # ... existing jobs ...

  # Tailscale client metrics from all hosts
  - job_name: 'tailscale-clients'
    static_configs:
      - targets: ['host.docker.internal:5252']
        labels: { host: 'mac-mini' }
      - targets: ['100.x.x.45:5252']
        labels: { host: 'digitalocean' }
      - targets: ['100.x.x.8:5252']
        labels: { host: 'raspberry-pi' }
    scrape_interval: 30s

  # Tailscale fleet exporter (runs locally)
  - job_name: 'tailscale-fleet'
    static_configs:
      - targets: ['tailscale-exporter:9090']

Alloy pushes host metrics via remote_write — no scrape targets needed for those.

Grafana Dashboards

Import these dashboard IDs

Import via Grafana UI or API

# Via CLI (from Mac Mini)
curl -s -u dan:Sl33py!!! \
  http://localhost:3000/api/dashboards/import \
  -H 'Content-Type: application/json' \
  -d '{"dashboard":{"id":null},"overwrite":true,
       "inputs":[{"name":"DS_PROMETHEUS","type":"datasource",
                  "pluginId":"prometheus","value":"Prometheus"}],
       "pluginId":"","folderId":0,
       "gnetId": 24177}'   # or 24178, or 1860

Pi 5 Deployment Steps

Full install sequence for the Raspberry Pi

# 1. Install Grafana Alloy (ARM64)
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key \
  | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] \
  https://apt.grafana.com stable main" \
  | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install -y alloy

# 2. Deploy config (see Slide 6 for full config)
sudo tee /etc/alloy/config.alloy < alloy-config.alloy

# 3. Enable Tailscale client metrics
sudo tailscale set --webclient

# 4. Start Alloy
sudo systemctl enable --now alloy

# 5. Verify
curl http://localhost:12345       # Alloy debug UI
journalctl -u alloy -f           # Alloy logs

Note Run Alloy as systemd service on the Pi, not Docker, to minimize RAM overhead (~100-200 MiB vs ~500 MiB in Docker).

DigitalOcean Deployment

Run Alloy in Docker alongside existing containers

# Add to existing docker-compose.yml or run standalone

docker run -d \
  --name alloy \
  --restart always \
  --network host \
  -v /var/log:/var/log:ro \
  -v /proc:/host/proc:ro \
  -v /sys:/host/sys:ro \
  -v ./alloy-config.alloy:/etc/alloy/config.alloy \
  grafana/alloy:latest \
  run /etc/alloy/config.alloy

Enable Tailscale metrics

sudo tailscale set --webclient

Tailscale Exporter (fleet metrics, runs on Mac Mini)

# Add to ~/monitoring-stack/docker-compose.yml
  tailscale-exporter:
    image: ghcr.io/adinhodovic/tailscale-exporter:latest
    container_name: tailscale-exporter
    restart: always
    environment:
      - TS_CLIENT_ID=<your-oauth-client-id>
      - TS_CLIENT_SECRET=<your-oauth-secret>
      - [email protected]

Implementation Checklist