> ## Documentation Index
> Fetch the complete documentation index at: https://docs.grantex.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations

> Run Grantex in production — health checks, required configuration, graceful shutdown, connection management, and webhook delivery.

This guide covers the operational aspects of running the Grantex auth service in production. For initial setup and deployment options, see the [Self-Hosting guide](/guides/self-hosting).

## Health Check Endpoint

The auth service exposes a health check at `GET /health` that probes both PostgreSQL and Redis:

```bash theme={null}
curl https://your-auth-service/health
```

### Healthy response (200)

```json theme={null}
{
  "status": "ok"
}
```

### Degraded response (503)

When one or more dependencies are unreachable, the endpoint returns `503` with the failing components:

```json theme={null}
{
  "status": "degraded",
  "failing": ["db", "redis"]
}
```

The `failing` array can contain `"db"` (PostgreSQL unreachable), `"redis"` (Redis unreachable), or both.

<Tip>
  Wire this endpoint into your load balancer's health check. Use a 5-second interval and 3-consecutive-failure threshold for removing unhealthy instances.
</Tip>

The health endpoint is unauthenticated (`skipAuth: true`) but still consumes the 5,000/min default Fastify IP policy; only JWKS is exempt.

## Required Configuration

The auth service validates required environment variables at startup and **exits immediately** if any are missing. This fail-fast behavior prevents partial startup with a broken configuration.

| Variable          | Required | Validated at | Failure mode                                   |
| ----------------- | -------- | ------------ | ---------------------------------------------- |
| `DATABASE_URL`    | Yes      | Startup      | Process exits with error                       |
| `REDIS_URL`       | Yes      | Startup      | Process exits with error                       |
| `RSA_PRIVATE_KEY` | Yes\*    | Startup      | Process exits unless `AUTO_GENERATE_KEYS=true` |
| `JWT_ISSUER`      | No       | Startup      | Defaults to `https://grantex.dev`              |

\* In development, set `AUTO_GENERATE_KEYS=true` to auto-generate an ephemeral RSA keypair. Never use this in production.

### Full environment variable reference

See the [Self-Hosting guide](/guides/self-hosting#5-environment-variable-reference) for common variables. `apps/auth-service/src/config.ts` in the deployed release is the authoritative configuration schema, including feature-specific validation.

### Trusted Client IPs

`TRUST_PROXY` defaults to `false`, so the service ignores forwarded client-IP
headers and uses the direct socket address. Enable it only when the service is
reachable exclusively through proxies you control. Accepted values are a fixed
hop count from `1` to `16` or a comma-separated proxy IP/CIDR allowlist.

Hop counts are safe only for a fixed network path with direct backend access
blocked. If request paths have different lengths, use stable proxy IP/CIDR
entries instead. Never enable blanket trust; broad values are rejected at
startup.

### Startup sequence

The auth service boots in this order:

1. **OpenTelemetry tracing** -- initialized first to hook module loading (only if `OTEL_EXPORTER_OTLP_ENDPOINT` is set)
2. **RSA key initialization** -- loads or generates the RSA keypair for JWT signing
3. **Ed25519 key initialization** -- optional, for DID/VC support
4. **Database connection** -- connects to PostgreSQL
5. **Migrations** -- runs all `*.sql` migration files idempotently
6. **Redis connection** -- connects to Redis with lazy connect
7. **Seed data** -- creates dev accounts if `SEED_API_KEY` or `SEED_SANDBOX_KEY` are set
8. **HTTP server** -- starts listening on `PORT` (default 3001)
9. **Background workers** -- webhook delivery worker and anomaly detection worker start polling

If any step fails, the process exits with code 1 and logs the error to stdout.

## Graceful Shutdown

The auth service handles `SIGTERM` signals for graceful shutdown. When running in Kubernetes or Docker, the container runtime sends `SIGTERM` before force-killing the process.

When OpenTelemetry tracing is enabled, the `SIGTERM` handler flushes pending trace spans before the process exits:

```
process.on('SIGTERM', () => {
  sdk.shutdown().catch(console.error);
});
```

### Kubernetes configuration

Set a `terminationGracePeriodSeconds` that gives the service enough time to finish in-flight requests:

```yaml theme={null}
spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: auth-service
      livenessProbe:
        httpGet:
          path: /health
          port: 3001
        initialDelaySeconds: 10
        periodSeconds: 5
      readinessProbe:
        httpGet:
          path: /health
          port: 3001
        initialDelaySeconds: 5
        periodSeconds: 5
```

## Database Connection

The auth service uses [postgres.js](https://github.com/porsager/postgres) for PostgreSQL connections. The connection is lazily initialized on first use and reused for the lifetime of the process.

### Connection string format

```
postgresql://user:password@host:5432/grantex?sslmode=require
```

Always use `sslmode=require` (or `verify-full` for stricter validation) in production.

### Connection pool behavior

postgres.js manages an internal connection pool. Default settings:

| Setting         | Default        | Description                                   |
| --------------- | -------------- | --------------------------------------------- |
| Max connections | 10             | Maximum concurrent connections                |
| Idle timeout    | 0 (no timeout) | Connections stay open until the process exits |
| Connect timeout | 30s            | Time to wait for a new connection             |

For high-throughput deployments, tune the pool by passing options to the postgres constructor in `db/client.ts`.

### Monitoring connections

Check active connections via PostgreSQL:

```sql theme={null}
SELECT count(*) FROM pg_stat_activity
WHERE datname = 'grantex' AND state = 'active';
```

## Redis Connection

The auth service uses [ioredis](https://github.com/redis/ioredis) with lazy connect. Redis stores:

* **Rate limit counters** -- standard developer API-key plan buckets and developer-keyed route buckets; active Fastify default/route policy counters remain process-local unless configured with a shared store
* **Ephemeral token metadata** -- in-flight authorization request state

Commerce, the SCIM Bearer data-plane routes under `/scim/v2/*`, admin, and other custom-auth routes do not consume the standard plan buckets; their active Fastify policies still apply. The standard API-key-authenticated `/v1/scim/tokens` management routes do consume the plan bucket.

### Reconnection behavior

ioredis automatically reconnects when the Redis connection drops. Default behavior:

* Retries with exponential backoff (starting at 50ms, capped at 2 seconds)
* No maximum retry count -- reconnects indefinitely
* Queues commands during disconnection and replays them on reconnect

### Data durability

Redis is not the source of truth. If Redis data is lost:

* **Redis-backed limits reset** -- standard developer plan and developer-keyed route buckets start fresh, allowing a brief burst; process-local active Fastify IP policy counters are unaffected
* **In-flight auth requests fail** -- users must restart the consent flow
* **No permanent data is lost** -- PostgreSQL is the durable store

<Note>
  For high-availability deployments, use Redis Sentinel or Redis Cluster. ioredis supports both modes natively.
</Note>

## Webhook Delivery

The auth service delivers webhooks with automatic retry and exponential backoff. A background worker polls the `webhook_deliveries` table every 30 seconds for pending deliveries.

### Retry policy

| Attempt   | Delay       | Cumulative wait |
| --------- | ----------- | --------------- |
| 1st retry | 30 seconds  | 30s             |
| 2nd retry | 60 seconds  | 1.5 min         |
| 3rd retry | 120 seconds | 3.5 min         |
| 4th retry | 240 seconds | 7.5 min         |
| 5th retry | 480 seconds | 15.5 min        |

The backoff formula is `30 * 2^attempt` seconds. After 5 failed attempts (configurable via `max_attempts` in the deliveries table), the delivery is marked as `failed`.

### Delivery mechanics

* **Timeout**: Each delivery attempt has a 10-second timeout
* **Success**: Any 2xx response marks the delivery as `delivered`
* **Failure**: Non-2xx responses or network errors trigger a retry
* **Signature**: Every payload includes an `X-Grantex-Signature` header for HMAC verification
* **User-Agent**: Requests are sent with `Grantex-Webhooks/0.1`

### Monitoring deliveries

Query delivery status for a webhook endpoint:

```bash theme={null}
curl https://api.grantex.dev/v1/webhooks/<webhook-id>/deliveries \
  -H "Authorization: Bearer <api-key>"
```

This returns delivery history with status (`pending`, `delivered`, `failed`), attempt count, and error details.

## Background Workers

Two background workers run after the HTTP server starts:

| Worker                | Interval     | Purpose                                                                          |
| --------------------- | ------------ | -------------------------------------------------------------------------------- |
| **Webhook delivery**  | 30 seconds   | Processes pending webhook deliveries with exponential backoff                    |
| **Anomaly detection** | Configurable | Scans for unusual patterns (rate spikes, off-hours activity, high failure rates) |

Both workers are started in the main process. They run on `setInterval` timers and execute one initial run immediately on startup.

### Worker health

Workers log errors to stdout but do not crash the process. If a worker iteration fails, it retries on the next interval. Monitor worker health by checking for `[webhook-delivery]` and `[anomaly-detection]` prefixed log messages.

## Logging

All logs are emitted as structured JSON to stdout, compatible with:

* **Datadog** -- auto-parsed by the Datadog agent
* **Grafana Loki** -- ingestible via Promtail
* **AWS CloudWatch Logs** -- auto-parsed in JSON format
* **Google Cloud Logging** -- structured log entries

Each log entry includes:

```json theme={null}
{
  "level": "info",
  "time": 1712345678901,
  "reqId": "a1b2c3d4-e5f6-...",
  "msg": "request completed",
  "responseTime": 12
}
```

### Log levels

| Level   | When                                                       |
| ------- | ---------------------------------------------------------- |
| `info`  | Normal request lifecycle (start, complete)                 |
| `warn`  | Deprecated API usage, approaching rate limits              |
| `error` | Failed requests, worker errors, database connection issues |
| `fatal` | Startup failures (missing config, migration errors)        |

## Database Migrations

Migrations run **automatically on every startup**. The auth service reads all `*.sql` files from the `migrations/` directory and executes each one using idempotent DDL (`CREATE TABLE IF NOT EXISTS`, etc.).

To upgrade, restart the service. New migration files are applied automatically. There is no separate migration command or rollback mechanism -- migrations are designed to be forward-only and non-destructive.

## Operational Checklist

<Check>Health check is wired to your load balancer</Check>
<Check>`TRUST_PROXY` is disabled or matches the verified, network-restricted proxy chain</Check>
<Check>All required environment variables are set</Check>
<Check>Database connection uses SSL (`sslmode=require`)</Check>
<Check>Redis is on a private network with authentication</Check>
<Check>Log forwarding is configured (Datadog, Loki, CloudWatch, etc.)</Check>
<Check>Prometheus metrics are scraped from `/metrics`</Check>
<Check>CPU and memory limits are set in your container orchestrator</Check>
<Check>Automated database backups are configured</Check>
<Check>Webhook endpoints are monitored for delivery failures</Check>
<Check>Alerting rules are set for error rate spikes and health check failures</Check>