Monitoring Guide

Product: v0.19.0 · Contract: OpenAPI · Spec ops: Ingestion cancel & fairness

Monitoring Guide

Observability for EdgeQuake Deployments

This guide covers monitoring, logging, and alerting for EdgeQuake in production environments.

---

Observability Stack

┌─────────────────────────────────────────────────────────────────┐
│                   OBSERVABILITY OVERVIEW                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐          │
│  │  EdgeQuake  │───▶│   Logs      │───▶│ Log Aggr.   │          │
│  │   Server    │    │  (stdout)   │    │ (Loki/ELK)  │          │
│  └──────┬──────┘    └─────────────┘    └─────────────┘          │
│         │                                                       │
│         ├─────────▶ /health endpoints                           │
│         │                                                       │
│         ├─────────▶ GET /metrics (Prometheus, live)             │
│         │              edgequake_http_* / edgequake_query_*     │
│         │              OTLP traces: optional `--features otel` │
│         │                                                       │
│         └─────────▶ PostgreSQL metrics                          │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

---

Health Endpoints

EdgeQuake provides built-in health endpoints:

Endpoint	Purpose	Response
GET /health	Basic health	{ "status": "healthy", "version": "0.19.0", ... }
GET /ready	Readiness check	JSON blockers; 503 when not ready
GET /live	Kubernetes liveness	Process check
GET /api/v1/pipeline/queue-metrics	Ingest backpressure + fairness	Pending depth, park waiters, store contention

Basic Health Check

curl http://localhost:8080/health

{
  "status": "healthy",
  "version": "0.19.0",
  "storage_mode": "postgresql"
}

Readiness Check

curl http://localhost:8080/ready

200 — ready for traffic:

{
  "ready": true,
  "blockers": [],
  "operator_action": null
}

503 — not ready (ReadinessResponse with actionable blockers):

{
  "ready": false,
  "blockers": [
    "store_contention_critical(pool_util=Some(0.92),quarantine=6)"
  ],
  "operator_action": "Scale DB pool or reduce ingest; inspect compensation quarantine DLQ"
}

Common /ready blockers (v0.19.0):

Blocker prefix	Cause	Operator action
Migration / M038 / pgvector	Schema or index not ready	Run migrations; see PostgreSQL migration guide
storage_ping_failed	KV / vector / graph ping timeout	Check DATABASE_URL, pool saturation
task_queue_critical	Pending depth above critical threshold	Scale WORKER_THREADS or reduce ingest rate
store_contention_critical	Pool util or compensation quarantine SLO breached	Tune pool; inspect compensation_quarantine:{document_id}:* KV keys

Env thresholds for store contention: EDGEQUAKE_DB_POOL_UTIL_WARN=0.75, EDGEQUAKE_DB_POOL_UTIL_CRITICAL=0.90, EDGEQUAKE_COMPENSATION_QUARANTINE_WARN=1, EDGEQUAKE_COMPENSATION_QUARANTINE_CRITICAL=5.

Queue Metrics

curl http://localhost:8080/api/v1/pipeline/queue-metrics | jq .

Key fields (SPEC-057):

Field	Meaning
pressure	normal | elevated | critical — mirrors /ready queue gate
tenant_park_waiters	Tasks parked on fairness semaphore (expected under local LLM clamp)
cancel_intent_count / cancel_intent_total	Cooperative cancel in flight / lifetime
max_tasks_per_tenant	Effective cap (local providers clamp to 1 unless overridden)
store_contention.level	normal | elevated | critical
store_contention.db_pool_utilization	Active pool utilization
store_contention.compensation_quarantine_total	Merge cleanup failures (not a park issue)

Prometheus: edgequake_compensation_quarantine_total tracks quarantine events. High tenant_park_waiters with low quarantine = fairness working; rising quarantine = AGE/pgvector delete errors — see Ingestion cancel & fairness.

---

Logging

Log Format

EdgeQuake uses structured JSON logging via the tracing crate:

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "INFO",
  "target": "edgequake_api::handlers::documents",
  "message": "Document uploaded successfully",
  "fields": {
    "document_id": "doc_123",
    "workspace_id": "ws_456",
    "duration_ms": 1234
  }
}

Log Levels

Level	RUST_LOG Setting	Use Case
Error	error	Critical failures
Warn	warn	Degraded but working
Info	info	Production operations
Debug	debug	Development debugging
Trace	trace	Detailed tracing

Recommended Production Settings

# Production
RUST_LOG="edgequake=info,tower_http=info,sqlx=warn"

# Development
RUST_LOG="edgequake=debug,tower_http=debug"

# Troubleshooting
RUST_LOG="edgequake=trace,sqlx=debug"

Component-Specific Logging

# Pipeline debugging
RUST_LOG="edgequake_pipeline=debug"

# Query engine debugging
RUST_LOG="edgequake_query=debug"

# API request tracing
RUST_LOG="tower_http=debug"

# Database query logging
RUST_LOG="sqlx=debug"

---

Log Aggregation

Loki + Grafana

Docker Compose addition:

services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yaml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

  grafana:
    image: grafana/grafana:10.0.0
    ports:
      - "3001:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

ELK Stack

Filebeat configuration:

filebeat.inputs:
  - type: container
    paths:
      - "/var/lib/docker/containers/*/*.log"
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/lib/docker/containers/"

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

---

Key Metrics to Monitor

Application Metrics

Metric	Source	Alert Threshold
Request latency	Logs	p99 > 2s
Error rate	Logs	> 1%
Active connections	PostgreSQL	> 80% pool
Background task queue	Logs	> 100 pending

PostgreSQL Metrics

Metric	Query	Alert Threshold
Connection count	pg_stat_activity	> 80% max
Cache hit ratio	pg_stat_database	< 95%
Index usage	pg_stat_user_indexes	Unused indexes
Table bloat	pgstattuple	> 30%

LLM Provider Metrics

Metric	Source	Alert Threshold
Token usage	Provider API	Budget threshold
Error rate	Logs	> 5%
Latency	Logs	> 10s
Rate limits	Provider API	Near limit

---

Alerting Rules

Prometheus Alerting (Example)

groups:
  - name: edgequake
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: High error rate detected

      - alert: SlowQueries
        expr: histogram_quantile(0.99, query_duration_seconds_bucket) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: Query latency above 2s

      - alert: DatabaseConnectionsHigh
        expr: pg_stat_activity_count > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: PostgreSQL connections high

---

Dashboard Examples

Key Panels for Grafana

1. Request Overview

   • Requests per second
   • Error rate
   • Latency percentiles (p50, p95, p99)

2. Document Processing

   • Documents indexed per minute
   • Processing time distribution
   • Queue depth

3. Query Performance

   • Query latency by mode
   • Context retrieval time
   • LLM generation time

4. Resource Usage

   • CPU usage
   • Memory usage
   • PostgreSQL connections
   • Disk I/O

Sample Query Panel

# Loki query for request latency
{app="edgequake"} |= "request completed" | json | duration_ms > 1000

---

Distributed tracing (SPEC-018)

EdgeQuake ships with OpenTelemetry-compatible tracing via edgequake-observability:

Capability	How
HTTP spans	http_request with request_id, trace_id, semantic error fields
Pipeline spans	pipeline_chunk_extraction, sota_query_pipeline
OTLP export	Build with --features otel or Docker ENABLE_OTEL=true
Correlation	X-Request-ID + W3C traceparent (API + WebUI)
Error context	ErrorEvent levelled logs + API details.diagnostics

Docker + Jaeger (one command):

cd edgequake/docker
docker compose -f docker-compose.yml -f docker-compose.observability.yml \
  --profile observability up --build
# Jaeger UI: http://localhost:16686

Production env:

export EDGEQUAKE_LOG_FORMAT=json
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317
export RUST_LOG=edgequake_api=info,edgequake_storage=warn

Full operator guide: OBSERVABILITY.md

---

PostgreSQL Monitoring

Essential Views

-- Active connections
SELECT count(*) as connections,
       state,
       wait_event_type
FROM pg_stat_activity
WHERE datname = 'edgequake'
GROUP BY state, wait_event_type;

-- Long-running queries
SELECT pid,
       now() - pg_stat_activity.query_start AS duration,
       query
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes'
  AND state != 'idle';

-- Table sizes
SELECT schemaname,
       relname,
       pg_size_pretty(pg_total_relation_size(relid)) as total_size
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 10;

-- Index usage
SELECT schemaname,
       relname,
       indexrelname,
       idx_scan,
       idx_tup_read
FROM pg_stat_user_indexes
ORDER BY idx_scan ASC
LIMIT 10;

Vector Storage Metrics

-- Vector index stats (pgvector)
SELECT indexname,
       pg_size_pretty(pg_relation_size(indexname::regclass)) as size
FROM pg_indexes
WHERE indexdef LIKE '%vector%';

-- Chunk count per workspace
SELECT workspace_id,
       count(*) as chunk_count
FROM chunks
GROUP BY workspace_id
ORDER BY chunk_count DESC;

Graph Storage Metrics

-- Entity count
SELECT count(*) FROM ag_catalog.cypher('edgequake_graph', $$
  MATCH (n) RETURN count(n)
$$) AS (count agtype);

-- Relationship count
SELECT count(*) FROM ag_catalog.cypher('edgequake_graph', $$
  MATCH ()-[r]->() RETURN count(r)
$$) AS (count agtype);

---

Troubleshooting

High Memory Usage

1. Check background task queue
2. Review connection pool size
3. Analyze PostgreSQL memory settings

# Check process memory
ps aux | grep edgequake

# Check PostgreSQL memory
psql -c "SHOW shared_buffers; SHOW work_mem;"

Slow Queries

1. Enable query logging

RUST_LOG="edgequake_query=debug,sqlx=debug"

2. Check PostgreSQL slow query log

-- Enable slow query logging
ALTER SYSTEM SET log_min_duration_statement = 1000;  -- 1 second
SELECT pg_reload_conf();

LLM Errors

1. Check provider status
2. Review rate limits
3. Verify API keys

# Test OpenAI connectivity
curl https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY"

# Test Ollama connectivity
curl http://localhost:11434/api/tags

---

Backup Monitoring

PostgreSQL Backups

# Check last backup time
pg_dump --version-only edgequake

# Verify backup size
ls -lh /backups/edgequake-*.sql.gz

Backup Alert Rule

- alert: BackupTooOld
  expr: time() - backup_last_success_timestamp > 86400
  for: 1h
  labels:
    severity: critical
  annotations:
    summary: No successful backup in 24 hours

---

Summary

┌─────────────────────────────────────────────────────────────────┐
│                   MONITORING CHECKLIST                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                   │
│  ✅ Health endpoints configured for load balancer               │
│  ✅ Structured logging enabled                                   │
│  ✅ Log aggregation set up (Loki/ELK)                           │
│  ✅ Key metrics identified and dashboarded                      │
│  ✅ Alert rules defined for critical conditions                 │
│  ✅ PostgreSQL monitoring enabled                               │
│  ✅ LLM provider usage tracked                                  │
│  ✅ Backup verification automated                               │
│                                                                   │
└─────────────────────────────────────────────────────────────────┘

---

See Also

• Deployment Guide - Production deployment
• Configuration Reference - All settings
• Troubleshooting Guide - Problem solving