Skip to content

Self-host Grafana and Prometheus

Dashboards and alerts for every VM you run, signed in with exe.dev.

Deploy on exe.dev

Shelley handles the setup. About 15 minutes.

Prefer a terminal?
curl -fsSL https://exe-staging.dev/deploy/grafana/prompt.txt | ssh exe.dev new --prompt=/dev/stdin

Grafana draws the dashboards, Prometheus collects the numbers, and a small exporter on each VM reports CPU, memory, disk, network and systemd units. This recipe runs all of it on one exe.dev VM that watches itself and your other VMs, checks your websites, and emails you when something breaks.

An exe.dev VM suits it well. It's a persistent machine with a real disk, so months of metrics stay put. Grafana is private by default, and your exe.dev login is the Grafana login: no admin password to set or leak. Other VMs are scraped through exe.dev's VM-to-VM integration, with no exporter exposed to the internet and no keys on either VM. Alerts go out through exe.dev's email gateway, Slack and Discord integrations, and AI features use the LLM gateway, so no SMTP password, webhook URL or API key sits on the disk. And Shelley is on board to write the next dashboard or alert for you.

What you get#

  • Grafana at the VM's private URL, signed in with exe.dev, with the owner as admin.
  • Prometheus collecting this VM's metrics, with a fleet dashboard and Node Exporter Full.
  • Five starter alert rules that email the owner through exe.dev, no SMTP server.
  • Website checks ready to fill in, and AI title and description suggestions through exe.dev's LLM gateway.
  • Everything restarts after a reboot; nightly backups with one checked.
What's running, and where
Private https://my-grafana.exe.xyz/ Grafana, signed in as you; you're its admin
Shared https://my-grafana.exe.xyz/ Teammates you share the VM with, as Grafana viewers
my-grafana.exe.xyz exe.dev VM
Grafana 13.3 grafana/grafana in Docker Compose; exe.dev login is the Grafana login :8000~/monitoring/grafana/data
Prometheus 3.16 prom/prometheus; 90 days or 5 GB of metrics, whichever comes first :9090~/monitoring/prometheus-data
node_exporter 1.12 A systemd service on this VM and on every VM it watches :9100
Blackbox exporter 0.29 HTTPS and certificate checks for your websites
Nightly backup Grafana's database and all config; two weeks kept ~/monitoring/backups

Deploy with Shelley#

Click Deploy and Shelley will set up Grafana on a new VM, then check that it works. No config-file wrangling required.

Prefer a terminal?
curl -fsSL \
  https://exe-staging.dev/deploy/grafana/prompt.txt \
  | ssh exe.dev new --prompt=/dev/stdin
What Shelley is told
Set up Grafana on this VM by following the exe.dev recipe at https://exe-staging.dev/deploy/grafana.md from top to bottom.

The recipe's example VM name is my-grafana. This VM's name is the output of `hostname`: use it wherever the recipe writes my-grafana. Skip the step that creates the VM; you are on it.

You are done when all of these are true. Check each one yourself:
- Grafana at the VM's private URL, signed in with exe.dev, with the owner as admin.
- Prometheus collecting this VM's metrics, with a fleet dashboard and Node Exporter Full.
- Five starter alert rules that email the owner through exe.dev, no SMTP server.
- Website checks ready to fill in, and AI title and description suggestions through exe.dev's LLM gateway.
- Everything restarts after a reboot; nightly backups with one checked.

Read the whole recipe, including the closed details. For initial setup,
follow "Install by hand" from top to bottom, skipping VM creation and
SSH. The walkthrough at the top is for the owner afterwards: don't add
other VMs or websites, create integrations, or set up Loki or Tailscale.
Do not run Restore or Update.
Work as exedev in ~/monitoring. Never overwrite an existing grafana.env.
The owner's email comes from the reflection integration; if it prints
nothing, ask the owner for their exe.dev email and use that.
Don't send a test alert or email; the owner will. Leave the VM private
and do not reboot. Verify docker.service and node_exporter are enabled,
every container restarts unless-stopped, and the checks in each step pass.
Finish with https://<vm>.exe.xyz/ and the three steps from "Watch
another VM" with this VM's name filled in.

Finish with the URLs I should open.

Also at /deploy/grafana/prompt.txt. The recipe it follows is /deploy/grafana.md.

Open Grafana#

Open the URL Shelley gives you, like https://my-grafana.exe.xyz/. Sign in to exe.dev if asked, and you're in Grafana as its admin; there's no Grafana password.

Open Dashboards → VMs:

  • VM fleet shows every VM Prometheus watches: up or down, CPU, memory, disk, failed systemd units, and the websites you check.
  • Node Exporter Full goes deep on one VM. Pick it in the Nodename menu at the top.

Right now both show one VM, this one. Next, add another.

Watch another VM#

Each VM you want to watch runs node_exporter on port 9100. exe.dev's proxy keeps that port private, and a VM-to-VM integration lets this VM, and only this VM, read it. Say the other VM is called web1:

  1. On web1, install node_exporter:

    curl -fsSL https://exe-staging.dev/deploy/grafana/node-exporter.sh | sh
    

    It downloads node_exporter from its GitHub release, checks the checksum, and runs it as a systemd service. It ends by printing node_exporter 1.12.1 is running.

  2. On your computer, let this VM reach web1's port 9100:

    ssh exe.dev integrations add http-proxy --name node-web1 \
      --target https://web1.exe.xyz:9100/ --peer --attach vm:my-grafana
    
  3. On my-grafana, tell Prometheus about it:

    cd ~/monitoring
    vm=web1
    printf -- '- targets: [node-%s.int.exe.xyz]\n  labels: {vm: %s, __scheme__: https}\n' \
      "$vm" "$vm" > "prometheus/vms/$vm.yml"
    sleep 40; curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job="node"}' | jq -c '.data.result[] | [.metric.vm, .value[1]]'
    

    Each VM should print with "1". Prometheus picks up new files within seconds, and web1 appears on VM fleet. A "0" means the integration or node_exporter isn't answering; see Troubleshooting.

The integration injects a key that only works for web1, and it never lands on either VM. Requests reach web1 marked X-Exedev-Source-Vm: my-grafana. To stop watching web1, delete prometheus/vms/web1.yml and run ssh exe.dev integrations remove node-web1.

Already on Tailscale? Scrape over your tailnet instead

If your VMs are on your Tailscale network, Prometheus can scrape them directly, without an integration per VM. Join this VM and each watched VM (Tailscale is preinstalled; exe.dev's Tailscale integration joins without a login link):

sudo systemctl enable --now tailscaled
sudo tailscale up --hostname=$(hostname)

Then list the VM by its Tailscale name, over plain HTTP:

cd ~/monitoring
vm=web1
printf -- '- targets: ["%s:9100"]\n  labels: {vm: %s}\n' "$vm" "$vm" > "prometheus/vms/$vm.yml"

The Prometheus container resolves names through the VM, so MagicDNS names work. Every device on your tailnet can now read port 9100 unless your Tailscale access rules say otherwise.

Check your websites#

The Blackbox exporter fetches URLs like a visitor would and records whether they answered, how fast, and when their TLS certificate expires. List your sites, one URL per line, on my-grafana:

cd ~/monitoring
cat > prometheus/sites/sites.yml <<'EOF'
- targets:
    - https://example.com/
    - https://app.example.com/healthz
EOF

They appear under Websites on VM fleet within a minute. Two alert rules watch them: Site down after five minutes of failures, and Certificate expiring two weeks ahead.

A private exe.dev app answers outsiders with exe.dev's login page and a 401, so it shows as down. To check it, make a VM-to-VM integration for it as above (--target https://my-app.exe.xyz/) and list https://<integration>.int.exe.xyz/ instead. For a public status page and a friendlier editor, run Uptime Kuma instead or as well.

Get alerts#

Five starter rules live in Alerting → Alert rules → VM alerts: VM down, disk above 85%, a failed systemd unit, site down, and certificate expiring. They go to the Email me contact point, which mails you through exe.dev's email gateway.

Try it: open Alerting → Contact points, edit Email me, and press Test. The email arrives from exe.dev within a minute. Edit the rules' thresholds, or add your own, right in Grafana.

Slack, Discord or Telegram, with the secret kept off the VM

exe.dev holds the webhook URL or bot token and injects it, so Grafana only ever sees an int.exe.xyz address. Set up the integration once, attach it to my-grafana, then add a contact point in Alerting → Contact points → Add contact point, and pick it in Alerting → Notification policies (edit the default policy, or add a child policy).

Slack. On your computer, run ssh exe.dev integrations setup slack, pick a channel, then ssh exe.dev integrations attach <name> vm:my-grafana. In Grafana choose Slack, leave Token empty, and set Webhook URL to https://<name>.int.exe.xyz/. See Slack integration.

Discord. Run ssh exe.dev integrations setup discord, then attach it the same way. In Grafana choose Discord, set Webhook URL to https://<name>.int.exe.xyz/, and turn on Use Discord's Webhook Username: exe.dev rejects messages that set their own name or avatar. See Discord integration.

Telegram. Add the Telegram Bot API integration from the integration catalog with your bot's token, and attach it. Grafana's Telegram contact point wants the token itself, so use a Webhook contact point instead: URL https://<name>.int.exe.xyz/sendMessage, method POST, and under Optional Webhook settings → Custom Payload, this template with your chat ID in place of 123456789:

{{ coll.Dict "chat_id" "123456789" "text" (tmpl.Exec "default.message" .) | data.ToJSON }}

All three are rate limited by exe.dev (Slack and Discord to five messages a minute), which suits alerts. Keep Email me as a fallback.

Share it with your team#

On your computer:

ssh exe.dev share add my-grafana teammate@example.com

Once they've signed in to exe.dev, they open the same URL and Grafana makes them an account as a Viewer. Promote them in Administration → Users and access → Users. ssh exe.dev share add my-grafana team shares with your whole exe.dev team, and ssh exe.dev share remove my-grafana teammate@example.com locks someone out again. Grafana has no sign-out button here; signing out of exe.dev signs you out.

The proxy shares every port 3000-9999 with the same people. Prometheus listens only inside the VM, but node_exporter on 9100 is readable by anyone the VM is shared with.

Let AI help#

The LLM plugin is set up to use exe.dev's LLM gateway, with no API key. While editing a panel or dashboard, press ✨ Auto generate beside Title or Description, or let it summarize your changes when you save.

For bigger jobs, ask Shelley on the VM. It can read the config in ~/monitoring, query Prometheus at localhost:9090, write a dashboard JSON into grafana/dashboards/, or add an alert rule with Grafana's API. For example: "Add a dashboard for the Postgres on db1 and alert me if it has more than 80 connections."

Add logs with Loki (optional)

Loki stores logs the way Prometheus stores metrics, and Grafana Alloy ships them. This adds both to the stack and collects this VM's systemd journal and Docker container logs; Loki keeps 30 days. On my-grafana:

cd ~/monitoring
mkdir -p loki-data
cat > loki.yaml <<'EOF'
auth_enabled: false
server:
  http_listen_port: 3100
common:
  path_prefix: /loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
limits_config:
  retention_period: 30d
compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  delete_request_store: filesystem
EOF

cat > alloy.alloy <<'EOF'
loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

loki.relabel "journal" {
  forward_to = []
  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }
}

loki.source.journal "journal" {
  path          = "/var/log/journal"
  relabel_rules = loki.relabel.journal.rules
  labels        = {job = "journal", vm = sys.env("VM")}
  forward_to    = [loki.write.local.receiver]
}

discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "containers" {
  targets = discovery.docker.containers.targets
  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  labels     = {job = "docker", vm = sys.env("VM")}
  forward_to = [loki.write.local.receiver]
}
EOF

cat > compose.loki.yaml <<EOF
services:
  loki:
    image: grafana/loki:3.7.10
    restart: unless-stopped
    user: "1000:1000"
    command: ["-config.file=/etc/loki/loki.yaml"]
    volumes:
      - ./loki.yaml:/etc/loki/loki.yaml:ro
      - ./loki-data:/loki
  alloy:
    image: grafana/alloy:v1.22.0
    restart: unless-stopped
    command: ["run", "--storage.path=/var/lib/alloy/data", "/etc/alloy/config.alloy"]
    environment:
      VM: $(hostname)
    volumes:
      - ./alloy.alloy:/etc/alloy/config.alloy:ro
      - /var/log/journal:/var/log/journal:ro
      - /etc/machine-id:/etc/machine-id:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on: [loki]
EOF

cat > grafana/provisioning/datasources/loki.yaml <<'EOF'
apiVersion: 1
datasources:
  - name: Loki
    uid: loki
    type: loki
    access: proxy
    url: http://loki:3100
EOF

echo 'COMPOSE_FILE=compose.yaml:compose.loki.yaml' >> .env
docker compose up -d --wait
docker compose restart grafana

In Grafana, open Drilldown → Logs, or Explore with the Loki data source and the query {vm="my-grafana"}. Alloy reads the Docker socket and the journal, so treat it as root-equivalent; Loki is reachable only inside the Compose network. Loki's data in loki-data isn't in the nightly backup.

Keep your data#

Prometheus keeps 90 days of metrics or 5 GB, whichever fills first, in ~/monitoring/prometheus-data. Grafana's dashboards, users and alert settings live in ~/monitoring/grafana/data/grafana.db. Each night a backup of Grafana's database and all the config goes to ~/monitoring/backups. Metrics history isn't in it: it's big, and it rebuilds itself.

Keep a copy off the VM. On your computer:

scp -r my-grafana.exe.xyz:monitoring/backups .

Check disk use on the VM with du -sh ~/monitoring/*-data and df -h /. The Disk almost full alert fires at 85%.

The nuts and bolts#

Happy clicking Deploy? You can stop here. These are Shelley's setup steps and your maintenance reference.

Install by hand

Create the VM#

On your computer (pick another name if my-grafana is taken):

ssh exe.dev new --name=my-grafana
ssh my-grafana.exe.xyz

All remaining installation commands run on the VM, as exedev.

Install node_exporter and start Docker at boot#

This VM watches itself too, so it gets node_exporter like any other. The script installs the official release binary as a systemd service on port 9100:

sudo systemctl enable --now docker.service
curl -fsSL https://exe-staging.dev/deploy/grafana/node-exporter.sh | sh

It ends with node_exporter 1.12.1 is running on <vm>:9100.

Write the configuration#

Grafana makes the VM's owner its admin, so it needs your exe.dev email. The reflection integration knows it:

mkdir -p ~/monitoring && cd ~/monitoring
mkdir -p grafana/data grafana/dashboards grafana/provisioning/datasources \
  grafana/provisioning/dashboards grafana/provisioning/plugins \
  prometheus/vms prometheus/sites prometheus-data backups
owner=$(curl -fsS https://reflection.int.exe.xyz/email | jq -r '.email // empty')
echo "Grafana admin: ${owner:?not found; set owner=you@example.com and run this block again}"
test -e grafana.env || (umask 077 && cat > grafana.env <<EOF
GF_SERVER_ROOT_URL=https://$(hostname).exe.xyz/
GF_SECURITY_ADMIN_USER=$owner
GF_SECURITY_ADMIN_PASSWORD=$(openssl rand -hex 24)
EOF
)

grafana.env (mode 600) holds a random password for Grafana's built-in admin. You never type it, because the login form is off, but keep it with your backups.

The Compose file. Grafana listens on port 8000, the one exe.dev's proxy serves, and trusts the proxy's X-ExeDev-Email header as the user name. Prometheus listens only on the VM's loopback:

cat > compose.yaml <<'EOF'
name: monitoring
services:
  grafana:
    image: grafana/grafana:13.3.1
    restart: unless-stopped
    user: "1000:1000"
    ports: ["8000:3000"]
    env_file: grafana.env
    environment:
      GF_AUTH_PROXY_ENABLED: "true"
      GF_AUTH_PROXY_HEADER_NAME: X-ExeDev-Email
      GF_AUTH_PROXY_HEADER_PROPERTY: username
      GF_AUTH_PROXY_HEADERS: Email:X-ExeDev-Email
      GF_AUTH_PROXY_AUTO_SIGN_UP: "true"
      GF_AUTH_DISABLE_LOGIN_FORM: "true"
      GF_AUTH_DISABLE_SIGNOUT_MENU: "true"
      GF_AUTH_BASIC_ENABLED: "false"
      GF_USERS_AUTO_ASSIGN_ORG_ROLE: Viewer
      GF_PLUGINS_PREINSTALL_SYNC: grafana-llm-app@1.2.0
    volumes:
      - ./grafana/data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/etc/grafana/dashboards:ro
    depends_on: [prometheus]
  prometheus:
    image: prom/prometheus:v3.16.0
    restart: unless-stopped
    user: "1000:1000"
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=90d
      - --storage.tsdb.retention.size=5GB
      - --web.enable-lifecycle
    ports: ["127.0.0.1:9090:9090"]
    extra_hosts: ["host.docker.internal:host-gateway"]
    volumes:
      - ./prometheus:/etc/prometheus:ro
      - ./prometheus-data:/prometheus
  blackbox:
    image: prom/blackbox-exporter:v0.29.1
    restart: unless-stopped
EOF

Prometheus scrapes every VM listed in prometheus/vms/ (this one over the Docker host gateway, the others through their integrations) and probes every URL in prometheus/sites/. It rereads both directories on its own:

cat > prometheus/prometheus.yml <<'EOF'
global:
  scrape_interval: 30s
  evaluation_interval: 30s
scrape_configs:
  - job_name: node
    file_sd_configs:
      - files: [/etc/prometheus/vms/*.yml]
  - job_name: sites
    scrape_interval: 60s
    metrics_path: /probe
    params:
      module: [http_2xx]
    file_sd_configs:
      - files: [/etc/prometheus/sites/*.yml]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115
  - job_name: prometheus
    static_configs:
      - targets: [localhost:9090]
EOF
printf -- '- targets: [host.docker.internal:9100]\n  labels: {vm: %s}\n' \
  "$(hostname)" > "prometheus/vms/$(hostname).yml"

Grafana's data source, dashboards, and LLM plugin. The dashboards are the community's Node Exporter Full (Apache-2.0, pinned to a commit) and this recipe's VM fleet. The LLM plugin talks to exe.dev's gateway as an OpenAI-compatible provider, with no key:

cat > grafana/provisioning/datasources/prometheus.yaml <<'EOF'
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 30s
EOF

cat > grafana/provisioning/dashboards/files.yaml <<'EOF'
apiVersion: 1
providers:
  - name: files
    folder: VMs
    type: file
    options:
      path: /etc/grafana/dashboards
EOF

cat > grafana/provisioning/plugins/llm.yaml <<'EOF'
apiVersion: 1
apps:
  - type: grafana-llm-app
    jsonData:
      provider: custom
      openAI:
        url: https://llm.int.exe.xyz
        apiPath: /openai/v1
      models:
        default: base
        mapping:
          base: gpt-5.4-mini
          large: gpt-5.5
EOF

curl -fsSL -o grafana/dashboards/node-exporter-full.json \
  https://raw.githubusercontent.com/rfmoz/grafana-dashboards/b07d8a90d8fadd112e9ac841b777ee863cda850b/prometheus/node-exporter-full.json
curl -fsSL -o grafana/dashboards/fleet.json https://exe-staging.dev/deploy/grafana/fleet.json
jq -r .title grafana/dashboards/*.json

jq prints Node Exporter Full and VM fleet.

Start the stack#

docker compose up -d --wait
docker compose ps
curl -fsS localhost:8000/api/health | jq -r .database
curl -fsS -H "X-ExeDev-Email: $owner" localhost:8000/api/user | jq -r '.login, .isGrafanaAdmin'
sleep 35; curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job="node"}' | jq -c '.data.result[] | [.metric.vm, .value[1]]'

All three services should be Up. The checks print ok, then your email and true (you're Grafana's admin), then this VM with "1".

Add the starter alerts#

alerts.sh uses Grafana's API to create the Email me contact point (exe.dev's email gateway, to the owner), point the notification policy at it, and add the five rules. They stay editable in Grafana's UI:

curl -fsSL -o alerts.sh https://exe-staging.dev/deploy/grafana/alerts.sh
chmod +x alerts.sh && ./alerts.sh

It prints Alerts ready: 5 rules in 'VM alerts', sent to <your email> by email.

Set up nightly backups#

This copies Grafana's SQLite database safely while it runs, checks the copy, and archives it with the config. Two weeks are kept:

cat > backup.sh <<'EOF'
#!/bin/sh
set -eu
umask 077
cd "$(dirname "$0")"
mkdir -p backups
work=$(mktemp -d)
trap 'rm -rf "$work"' EXIT
sqlite3 grafana/data/grafana.db ".backup '$work/grafana.db'"
test "$(sqlite3 "$work/grafana.db" 'PRAGMA integrity_check;')" = ok
out="backups/monitoring-$(date +%Y%m%d-%H%M%S).tar.gz"
tar -czf "$out.tmp" compose.yaml grafana.env alerts.sh backup.sh \
  prometheus grafana/provisioning grafana/dashboards -C "$work" grafana.db
mv "$out.tmp" "$out"
find backups -name 'monitoring-*.tar.gz' -mtime +14 -delete
EOF
chmod +x backup.sh

sudo tee /etc/systemd/system/monitoring-backup.service >/dev/null <<'EOF'
[Unit]
Description=Back up Grafana and the monitoring config

[Service]
Type=oneshot
User=exedev
ExecStart=/home/exedev/monitoring/backup.sh
EOF
sudo tee /etc/systemd/system/monitoring-backup.timer >/dev/null <<'EOF'
[Unit]
Description=Back up Grafana nightly

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now monitoring-backup.timer
sudo systemctl start monitoring-backup.service
systemctl list-timers monitoring-backup.timer
tar -tzf backups/monitoring-*.tar.gz | grep -c .

The timer lists its next run, and tar counts the files in the first archive. Open https://<vm>.exe.xyz/ and try the walkthrough above.

Back up and restore

Take an extra backup any time, on the VM:

cd ~/monitoring && ./backup.sh && ls -lh backups | tail -3

Restore#

On a new VM, run Install by hand first, then copy your archive into ~/monitoring/backups. On the VM, substitute the archive's name; the current database and config are set aside, not deleted:

cd ~/monitoring
backup=backups/monitoring-YYYYMMDD-HHMMSS.tar.gz
old="before-restore-$(date +%s)"
tar -tzf "$backup" >/dev/null &&
docker compose stop grafana &&
mkdir "$old" && cp -a grafana.env compose.yaml prometheus grafana/provisioning grafana/dashboards "$old"/ &&
mv grafana/data/grafana.db "$old"/ && rm -f grafana/data/grafana.db-wal grafana/data/grafana.db-shm &&
tar -xzf "$backup" --exclude=grafana.db &&
tar -xzf "$backup" -O grafana.db > grafana/data/grafana.db &&
docker compose up -d --wait

On a VM with a different name, fix GF_SERVER_ROOT_URL in grafana.env and rename this VM's file in prometheus/vms/, then run docker compose up -d. The VM-to-VM integrations must be attached to the new VM (ssh exe.dev integrations attach node-web1 vm:<new-vm>).

Update

The images are pinned to the versions this recipe was tested with. To update, read the release notes for Grafana, Prometheus and the Blackbox exporter, change the tags in compose.yaml, then on the VM:

cd ~/monitoring
./backup.sh && docker compose pull && docker compose up -d --wait

For node_exporter, rerun the script with the version you want, on each VM: curl -fsSL https://exe-staging.dev/deploy/grafana/node-exporter.sh | NODE_EXPORTER_VERSION=1.12.1 sh.

Something not working?

Start with the logs and Prometheus's view of its targets. On the VM:

cd ~/monitoring
docker compose ps
docker compose logs --tail=50 grafana prometheus
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | [.labels.vm // .labels.instance, .health, .lastError] | @tsv'
  • A VM shows down with 401 or 307. The integration isn't attached to this VM, or its target is wrong. Check curl -s https://reflection.int.exe.xyz/integrations | jq -r '.integrations[].name' and the target's port (:9100).

  • A VM shows down with 502 or connection refused. node_exporter isn't running there: systemctl status node_exporter on that VM.

  • Down after a reboot? Run sudo systemctl enable --now docker.service.

  • Grafana shows a login form or "invalid username or password". You reached it without exe.dev's proxy, so there's no X-ExeDev-Email. Use https://<vm>.exe.xyz/, not an SSH tunnel.

  • You're a Viewer, not the admin. Grafana's admin is whoever GF_SECURITY_ADMIN_USER in grafana.env named at its first start. Act as that account from the VM to promote yourself:

    cd ~/monitoring
    me=you@example.com
    admin=$(sed -n 's/^GF_SECURITY_ADMIN_USER=//p' grafana.env)
    id=$(curl -s -H "X-ExeDev-Email: $admin" "localhost:8000/api/users/lookup?loginOrEmail=$me" | jq .id)
    curl -s -X PUT -H "X-ExeDev-Email: $admin" -H 'Content-Type: application/json' \
      -d '{"isGrafanaAdmin":true}' localhost:8000/api/admin/users/$id/permissions
    curl -s -X PATCH -H "X-ExeDev-Email: $admin" -H 'Content-Type: application/json' \
      -d '{"role":"Admin"}' localhost:8000/api/org/users/$id
    
  • No alert email? Check spam, then Alerting → Contact points for the error next to Email me, and docker compose logs grafana | grep -i webhook.

  • ✨ Auto generate is missing. docker compose logs grafana | grep -i llm; the plugin installs when Grafana starts, so it needs the internet then.

Frequently asked questions#

How much does it cost to self-host Grafana on exe.dev?

Grafana, Prometheus and the exporters are free and open source. You pay for your exe.dev plan; one VM can watch the rest of your VMs. AI suggestions use your exe.dev LLM allocation.

Can it monitor servers that aren't on exe.dev?

Yes. Put them on your Tailscale network with this VM, or give them an HTTPS address Prometheus can scrape, and list them in a file under prometheus/vms/ the same way. Websites anywhere can be checked with the Blackbox exporter.

Why not just use Grafana Cloud?

Grafana Cloud's free tier is generous, but your VMs' metrics then leave exe.dev, each VM needs an agent with a Cloud API key, and retention and series counts are capped. Here the data stays on a disk you own, sign-in is your exe.dev login, and VMs are scraped without any key on them.

How much disk does Prometheus need?

About 2,000 series per VM at a 30-second scrape, which is roughly 10 MB a day per VM after compression. The 5 GB cap holds 90 days for five to six VMs; raise --storage.tsdb.retention.size or resize the disk for more.

Is the auth proxy login safe?

Grafana believes the X-ExeDev-Email header on any request that reaches port 8000. Only exe.dev's proxy and programs on the VM itself can reach that port, and the proxy replaces whatever header a visitor sends, so outsiders can't forge it. Anyone who can run commands on the VM can act as any Grafana user, the same as they could read its disk.

Who watches the watcher?

Grafana can't alert you while its own VM is down. Watch https://my-grafana.exe.xyz/api/health from somewhere else, such as an Uptime Kuma VM or an outside service.

Updated Nov 16, 2026 · tested end to end Nov 16, 2026 · Markdown · manifest.json · Something wrong? Tell us