Backup & Restore

Scope

• Database: PostgreSQL (database/postgresql-0), DB/user sub2api
• Git source: manifests/sub2api-argocd.yaml, owned by argocd/ops-docs
• Current release: OCI chart 0.1.18, application 0.2.9
• Application image: ghcr.io/wei-shaw/sub2api@sha256:996a0ea43500550f233a2f653de58dcd77f42d3e244805894fcccdd32b612147
• Runtime data: application/sub2api-data, 10Gi, local-path, RWO
• Redis data: 8Gi, local-path, RWO; AOF is enabled
• Runtime Secrets: application/sub2api-auth, application/sub2api-external-postgresql, and application/sub2api-redis

Sub2API executes PostgreSQL migrations automatically on startup. Migrations are forward-only, so every chart or image upgrade requires a verified pg_dump before the Git version change.

Pre-Upgrade Backup

1.create a protected operation directory

BACKUP_ROOT=/home/aaron/Ops/backups/sub2api
BACKUP_DIR="${BACKUP_ROOT}/upgrade-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$BACKUP_DIR"
chmod 700 "$BACKUP_DIR"
printf '%s\n' "$BACKUP_DIR"

Use the generated UTC directory for the entire operation. Do not hard-code a previous operation timestamp into future commands.

2.dump PostgreSQL without printing its password

set +x
PG_PASSWORD="$(kubectl -n application get secret \
  sub2api-external-postgresql \
  -o jsonpath='{.data.postgres-password}' | base64 -d)"
test -n "$PG_PASSWORD"

printf '%s\n' "$PG_PASSWORD" | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec pg_dump -U sub2api -d sub2api -Fc' \
  > "$BACKUP_DIR/sub2api.dump"

unset PG_PASSWORD

The password is passed on stdin and is not written into the backup directory.

3.capture application data and non-secret metadata

kubectl -n application exec deployment/sub2api -- \
  tar -C /app/data -czf - . > "$BACKUP_DIR/sub2api-data.tgz"

git -C /home/aaron/Ops/docs fetch origin main
git -C /home/aaron/Ops/docs rev-parse origin/main \
  > "$BACKUP_DIR/git-revision.txt"
git -C /home/aaron/Ops/docs show origin/main:manifests/sub2api-argocd.yaml \
  > "$BACKUP_DIR/sub2api-argocd.yaml"

kubectl -n application get pvc \
  -l app.kubernetes.io/instance=sub2api -o yaml \
  > "$BACKUP_DIR/pvc-metadata.yaml"

Do not export Kubernetes Secret objects into this directory. Back up Secret values only through the approved secret-management process. Redis AOF supports restart recovery on its PVC, but it is not a substitute for the PostgreSQL dump. Do not copy live AOF files as if they were a consistent database backup.

4.verify artifacts before upgrading

test -s "$BACKUP_DIR/sub2api.dump"
test -s "$BACKUP_DIR/sub2api-data.tgz"

kubectl -n database exec -i postgresql-0 -- pg_restore --list \
  < "$BACKUP_DIR/sub2api.dump" \
  > "$BACKUP_DIR/sub2api.dump.list"
test -s "$BACKUP_DIR/sub2api.dump.list"

sha256sum \
  "$BACKUP_DIR/sub2api.dump" \
  "$BACKUP_DIR/sub2api-data.tgz" \
  "$BACKUP_DIR/sub2api-argocd.yaml" \
  "$BACKUP_DIR/pvc-metadata.yaml" \
  "$BACKUP_DIR/git-revision.txt" \
  > "$BACKUP_DIR/SHA256SUMS"
sha256sum -c "$BACKUP_DIR/SHA256SUMS"

Verified backup: 2026-08-14

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260814T063611Z (directory mode 700, files mode 600). Only the shared PostgreSQL sub2api database was dumped; n8n and other databases were not touched.
• The host had no pg_restore; PostgreSQL Pod database/postgresql-0 had pg_restore 18.3 and GNU tar 1.34. The dump was copied temporarily with kubectl cp, checked with Pod-local pg_restore --list, and the temporary Pod file was removed. The previous kubectl exec -i streaming validation was not reused.
• Non-empty artifacts: sub2api.dump (19,640,824 bytes), sub2api-data.tgz (10,616,251 bytes), and sub2api.dump.list (77,853 bytes). Git revision, manifest, and PVC metadata were also captured.
• sha256sum -c SHA256SUMS and pg_restore --list both succeeded.
• The captured origin/main revision was ce838b424f10316fc604a4a21517c90c7b6b97ae. No failure occurred; no upgrade, Argo CD sync, or GitOps/cluster change was performed.

Verified upgrade: 2026-09-05 (application 0.2.0)

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260905T000035Z (upgrade commit 5230546610c437a02ee45f5b80cd804779c9170c). The scoped sub2api.dump (47,520,105 bytes) and sub2api-data.tgz (10,192,905 bytes) were non-empty; pg_restore --list and SHA-256 checks passed.
• schema_migrations advanced from 268 to 277 records; the latest migration is 233_group_free_openai_fast.sql.
• ArgoCD reported Synced/Healthy; Deployment was 1/1 on the pinned ghcr.io/wei-shaw/sub2api@sha256:271bb3b34661803681cabf54e99811ab8e248b0dd4c88b09ea1226e22dea5751 image.
• Existing application and Redis PVCs remained Bound, Service endpoints were Ready, and public health/settings checks succeeded.
• No rollback was required.

Verified upgrade: 2026-09-05 (application 0.2.1)

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260905T115349Z-178860 (upgrade commit 21d02345de47d75454841e464756f6cf9349cc3c). The scoped sub2api.dump and sub2api-data.tgz were non-empty; pg_restore --list and SHA-256 checks passed.
• ArgoCD reported Synced/Healthy at 2026-09-05T13:03:41Z; Deployment was 1/1, Pod ready with 0 restarts, internal /health returned {"status":"ok"}, Service endpoint ready, and Redis StatefulSet 1/1.
• Existing application and Redis PVCs remained Bound; health and settings checks succeeded.
• No rollback was required.

Verified upgrade: 2026-09-10

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260910T061248Z; PostgreSQL dump 46,520,009 bytes, /app/data archive 10,214,969 bytes; backup artifacts and pg_restore --list, archive listing, SHA-256 all passed; GitOps commit 7bd44adb5ed1b61976b507e1f32bc990738782b2; sub2api ArgoCD target/observed 0.1.14 Synced/Healthy operation Succeeded; runtime verification: Deployment 1/1, Pod ready 0 restarts, chart sub2api-0.1.14/app 0.2.4, actual image digest matches, Service endpoint ready/serving, internal /health HTTP 200, Redis StatefulSet 1/1.
• No rollback required.

Verified upgrade: 2026-09-16

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260916T053309Z (directory mode 700, files mode 600). PostgreSQL dump 27,593,406 bytes, /app/data archive 4,579,406 bytes; backup artifacts and pg_restore --list, archive listing, SHA-256 all passed; GitOps commit e2c0e4576276f6a299bbd27e80f40af883f8e877 (chore(sub2api): upgrade to chart 0.1.15); sub2api ArgoCD target/observed 0.1.15 Synced/Healthy operation Succeeded; runtime verification: Deployment 1/1, Pod sub2api-7c9c4b95bf-r6cpg Running/Ready, restarts 0, chart sub2api-0.1.15/app 0.2.5, actual image digest matches sha256:4c5dffab6e5ba4d3bd5382f19aad9654847b4e23de1a3d48e190146a3e6eb977, Service endpoint ready at 10.42.0.198:8080, internal /health HTTP 200 body {"status":"ok"}, Redis StatefulSet ready at 10.42.0.12:6379 with 1 historical restart.
• No rollback required.

Verified upgrade: 2026-09-20 (application 0.2.7)

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260920T051850Z
• Backup artifacts: sub2api.dump (29,365,379 bytes, SHA-256 d9a0fb1870cde3405d47765291190dbc0bf17bd724708dc1fd7f448a8027eace), sub2api-data.tgz (4,598,099 bytes, SHA-256 f6b1691f6ba771f415e78caebedb0da225ef665a7af22c3b9d9854c6f4e097f7), pg_restore --list (1,210 entries), archive listing (10 entries), checksums passed.
• GitOps commit: 67d8fef5859ec579cee101da22cc4988ca789b84 (chore: upgrade sub2api)
• ArgoCD status: Synced/Healthy, operation Succeeded, observed chart revision 0.1.16
• Deployment: 1/1 ready; Pod running/ready with 0 restarts
• Service endpoint: 10.42.0.224:8080
• Internal /health HTTP 200 (body discarded)
• Dedicated Redis StatefulSet: 1/1 ready; Redis Pod has 1 historical restart
• No rollback required.

Verified upgrade: 2026-09-23 (chart 0.1.17 / application 0.2.8)

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260923T222508Z (directory mode 700, files mode 600). The scoped PostgreSQL dump was 32,663,617 bytes and the /app/data archive was 4,614,791 bytes. pg_restore --list returned 1,210 entries, the archive listing contained 10 entries, and all SHA-256 checks passed. The initial streamed validation timed out; validation using a controlled temporary file inside the PostgreSQL container passed.
• GitOps commit: 646d50010cf39090567098bc819e74b06976a5ef; only manifests/sub2api-argocd.yaml changed, from chart 0.1.16 to 0.1.17. Mirror chart PR #8 merged as 87ae24c. The image digest is sha256:11b2dc8d9ea297c676581daf1322d71200eeed58ebfb5fecf2a991fa2741fb86.
• ArgoCD parent and child reconciled successfully; Sub2API was Synced/Healthy on chart 0.1.17. Deployment was 1/1, the Pod was Ready with 0 restarts, and its image ID matched the target digest. The Service endpoint 10.42.0.226:8080 was ready; internal and public /health checks returned HTTP 200. Dedicated Redis was 1/1 with a ready endpoint; both PVCs remained Bound.
• Migrations completed without errors. Preflight found 27 non-numeric legacy reasoning-effort keys, which migration 239 removed without conversion; one existing affiliate-ledger row received a NULL operation ID under migration 240, with no unique-index conflict.
• Image pull took 10m37s and briefly caused ProgressDeadlineExceeded; the rollout recovered. Existing pricing-repository timeout warnings remained. No rollback was required. Deployment rollback is a new Git revert; migrations are forward-only, so database recovery requires an isolated restore and a deliberate configuration switch.

Verified upgrade: 2026-09-28 (chart 0.1.18 / application 0.2.9)

• Backup directory: /home/aaron/Ops/backups/sub2api/upgrade-20260928T135349Z (directory mode 700, files mode 600). The scoped PostgreSQL dump was 37,222,935 bytes with 1,214 pg_restore --list entries; the /app/data archive was 4,631,639 bytes with 10 entries. SHA-256 checks passed, and the captured origin/main revision was 622ea9171286d9e218bd15a9569ce8c1b565d48b.
• GitOps commit: b37fe3549cc8c65bba6d268c45ffdda97f86d9ca (chore(sub2api): upgrade to chart 0.1.18); only manifests/sub2api-argocd.yaml changed, from chart 0.1.17 to 0.1.18. Mirror chart PR #9 merged as a748960; the published chart 0.1.18 was anonymously pulled back with package SHA-256 196406aa97b89ff747cc0317ad71d2eaa3b22adbc0057c7d4aec2ec126ad35e4 matching Git. The image digest is sha256:996a0ea43500550f233a2f653de58dcd77f42d3e244805894fcccdd32b612147.
• ArgoCD synced Sub2API to chart 0.1.18 at 2026-09-28T13:58Z (operation Succeeded) and reported Synced/Healthy by 14:00Z. Deployment was 1/1, Pod sub2api-8c7fbd445-bhhnp Ready with 0 restarts, and its image ID matched the target digest exactly. The Service endpoint 10.42.0.239:8080 was ready; internal /health returned {"status":"ok"} and public https://token.72602.space/health returned HTTP 200. Dedicated Redis was 1/1 and both PVCs remained Bound.
• Application 0.2.9 introduced no new database migrations: the latest applied migration remains 240_affiliate_ledger_operation_id.sql (applied 2026-09-23) and schema_migrations holds 289 records. No migration or startup errors appeared in the logs; the rollout completed in about two minutes without ProgressDeadlineExceeded.
• No rollback was required. Deployment rollback is a new Git revert; migrations are forward-only, so database recovery requires an isolated restore and a deliberate configuration switch.

Restore PostgreSQL Safely

Restore into a separate database first. Do not overwrite the live sub2api database during an upgrade rollback.

1.create the restore database

BACKUP_DIR=/home/aaron/Ops/backups/sub2api/<approved-backup-directory>
test -s "$BACKUP_DIR/sub2api.dump"

set +x
read -rsp 'PostgreSQL admin password: ' POSTGRES_ADMIN_PASSWORD; printf '\n'
test -n "$POSTGRES_ADMIN_PASSWORD"

printf '%s\n' "$POSTGRES_ADMIN_PASSWORD" | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec createdb -U postgres -O sub2api sub2api_restore'

unset POSTGRES_ADMIN_PASSWORD

2.restore and verify with the application database user

set +x
PG_PASSWORD="$(kubectl -n application get secret \
  sub2api-external-postgresql \
  -o jsonpath='{.data.postgres-password}' | base64 -d)"
test -n "$PG_PASSWORD"

{ printf '%s\n' "$PG_PASSWORD"; cat "$BACKUP_DIR/sub2api.dump"; } | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec pg_restore -U sub2api -d sub2api_restore --exit-on-error --no-owner --no-privileges'

printf '%s\n' "$PG_PASSWORD" | \
  kubectl -n database exec -i postgresql-0 -- \
  sh -c 'IFS= read -r PGPASSWORD; export PGPASSWORD; exec psql -U sub2api -d sub2api_restore -c "\\dt"'

unset PG_PASSWORD

3.switch only through a reviewed Git recovery change

Change externalPostgresql.database to sub2api_restore in manifests/sub2api-argocd.yaml, commit and push the reviewed recovery change, then reconcile ops-docs and sub2api. Restore sub2api-data only during a planned maintenance window with the workload quiesced; never extract the archive over a running Pod.

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

curl -fsS https://token.72602.space/health
curl -fsS https://token.72602.space/api/v1/settings/public
kubectl -n application logs deployment/sub2api --since=10m

Do not use kubectl rollout undo, delete PVCs, or delete Secrets as restore or rollback steps.

App-Level S3 Backup Check

The live S3 endpoint is https://api.minio.72602.space, and Sub2API uses the sub2api bucket. The MinIO API Ingress source keeps nginx.ingress.kubernetes.io/proxy-body-size: "0" scoped to that API host so backup uploads are not rejected by the default 1 MiB ingress limit. Verify a scheduled or manual backup from logs without printing credentials, tokens, object names, or backup content:

kubectl -n application logs deployment/sub2api --since=30m
kubectl -n basic-components logs deployment/ingress-nginx-controller --since=30m
kubectl -n storage get ingress minio-api

Install (ArgoCD)

🚀Installation

kubectl get namespace application >/dev/null 2>&1 || \
  kubectl create namespace application

set +x
read -rsp 'Sub2API admin password: ' ADMIN_PASSWORD; printf '\n'
read -rsp 'Sub2API PostgreSQL password: ' POSTGRES_PASSWORD; printf '\n'
read -rsp 'Sub2API Redis password: ' REDIS_PASSWORD; printf '\n'
JWT_SECRET="$(openssl rand -hex 32)"
TOTP_KEY="$(openssl rand -hex 32)"

test -n "$ADMIN_PASSWORD"
test -n "$POSTGRES_PASSWORD"
test -n "$REDIS_PASSWORD"

kubectl -n application create secret generic sub2api-auth \
  --from-literal=admin-password="$ADMIN_PASSWORD" \
  --from-literal=jwt-secret="$JWT_SECRET" \
  --from-literal=totp-encryption-key="$TOTP_KEY" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n application create secret generic sub2api-external-postgresql \
  --from-literal=postgres-password="$POSTGRES_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n application create secret generic sub2api-redis \
  --from-literal=redis-password="$REDIS_PASSWORD" \
  --dry-run=client -o yaml | kubectl apply -f -

unset ADMIN_PASSWORD POSTGRES_PASSWORD REDIS_PASSWORD JWT_SECRET TOTP_KEY
kubectl -n application get secret \
  sub2api-auth sub2api-external-postgresql sub2api-redis

git -C /home/aaron/Ops/docs fetch origin main
git -C /home/aaron/Ops/docs \
  show origin/main:manifests/sub2api-argocd.yaml >/dev/null

kubectl -n argocd get application ops-docs \
  -o jsonpath='{.spec.source.repoURL}{"\n"}{.spec.source.path}{"\n"}'

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

argocd app get sub2api --hard-refresh
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

kubectl -n argocd get application sub2api \
  -o jsonpath='{.spec.source.repoURL}{"\n"}{.spec.source.chart}{" "}{.spec.source.targetRevision}{"\n"}'

kubectl -n application get deployment sub2api \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"="}{.image}{"\n"}{end}'

kubectl -n application rollout status deployment/sub2api --timeout=600s
kubectl -n application get pods,svc,ingress,pvc
kubectl -n application get certificate,certificaterequest,order,challenge
kubectl -n application get endpointslice \
  -l kubernetes.io/service-name=sub2api

curl -fsS https://token.72602.space/health
curl -fsS https://token.72602.space/api/v1/settings/public

set +x
read -rsp 'Sub2API API token: ' SUB2API_API_TOKEN; printf '\n'
curl -fsS \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  https://token.72602.space/v1/models

read -rp 'Model ID for smoke generation: ' MODEL_ID
curl -fsS https://token.72602.space/v1/chat/completions \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  --json "{\"model\":\"${MODEL_ID}\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with OK.\"}],\"max_tokens\":8}"
unset SUB2API_API_TOKEN MODEL_ID

Ops

Web Page

sub2api web page (https://token.72602.space)

Current Release

argocd app get ops-docs --hard-refresh
argocd app get sub2api --hard-refresh

kubectl -n argocd get application sub2api \
  -o jsonpath='{.spec.source.repoURL}{"\nchart="}{.spec.source.chart}{" "}{.spec.source.targetRevision}{"\nimage.tag="}{.spec.source.helm.parameters[?(@.name=="image.tag")].value}{"\n"}'

kubectl -n application get deployment sub2api \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"="}{.image}{"\n"}{end}'

kubectl -n application get pods,svc,ingress,pvc
kubectl -n application get certificate,certificaterequest,order,challenge

The expected values are OCI chart 0.1.18, application 0.2.9, image ghcr.io/wei-shaw/sub2api@sha256:996a0ea43500550f233a2f653de58dcd77f42d3e244805894fcccdd32b612147, namespace application, and host token.72602.space. The application PVC is 10Gi; the Redis PVC is 8Gi with AOF enabled. Both use local-path and RWO.

Sync From Git

argocd/ops-docs owns manifests/sub2api-argocd.yaml from https://github.com/AaronYang0628/docs.git. Reconcile the Git parent before the OCI Helm child:

git -C /home/aaron/Ops/docs fetch origin main
argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

argocd app get sub2api --hard-refresh
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

Do not apply manifests/sub2api-argocd.yaml directly as a second ownership path.

Rolling Upgrade

1.back up PostgreSQL before changing chart or image values

Use the Backup & Restore runbook. Sub2API executes PostgreSQL migrations automatically at startup, and migrations are forward-only. Verify the dump before continuing.

2.update the Git source

Edit only the reviewed targetRevision, image.tag, or required values in manifests/sub2api-argocd.yaml, then inspect and publish the change:

git -C /home/aaron/Ops/docs diff --check -- manifests/sub2api-argocd.yaml
git -C /home/aaron/Ops/docs diff -- manifests/sub2api-argocd.yaml
git -C /home/aaron/Ops/docs add manifests/sub2api-argocd.yaml
git -C /home/aaron/Ops/docs commit -m "chore: upgrade sub2api"
git -C /home/aaron/Ops/docs push origin HEAD:main

3.reconcile parent and child Applications

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300

argocd app get sub2api --hard-refresh
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600
kubectl -n application rollout status deployment/sub2api --timeout=600s
kubectl -n application get endpointslice \
  -l kubernetes.io/service-name=sub2api -o wide

The Deployment explicitly uses maxUnavailable: 0 and maxSurge: 1. Kubernetes adds the new Ready Pod to the EndpointSlice before terminating the old Pod, and the Service selects only Ready endpoints. This protects new requests during rollout, but it does not guarantee completion of requests that are already attached to the terminating Pod. Long generations still require the application to handle graceful termination and draining correctly.

4.verify the public and authenticated model path

curl -fsS https://token.72602.space/health
curl -fsS https://token.72602.space/api/v1/settings/public

set +x
read -rsp 'Sub2API API token: ' SUB2API_API_TOKEN; printf '\n'
curl -fsS \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  https://token.72602.space/v1/models

read -rp 'Model ID for smoke generation: ' MODEL_ID
curl -fsS https://token.72602.space/v1/chat/completions \
  -H "Authorization: Bearer ${SUB2API_API_TOKEN}" \
  --json "{\"model\":\"${MODEL_ID}\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply with OK.\"}],\"max_tokens\":8}"
unset SUB2API_API_TOKEN MODEL_ID

Admin MCP

The Ops Agent Pod runs a local MCP server sub2api-admin against the cluster-local endpoint http://sub2api.application.svc.cluster.local:8080. The Deployment injects the API key from the dedicated runtime Secret application/sub2api-mcp key admin-api-key. Provision the Secret through manifests/ops-agent/create-sub2api-mcp-secret.sh; the value never enters Git or opencode.json. The MCP only accepts the configured cluster-local host, every authenticated GET runs through a reviewed route allowlist, sensitive fields are recursively redacted, and no generic authenticated write tool is exposed.

Enter the key only into the helper’s silent interactive prompt. Never place the key in shell arguments, chat, Git, OpenCode config, logs, or command output.

Tools

Nine tools are registered:

The global OpenCode permission set denies every sub2api-admin_* tool. The 72602-k3s-maintainer agent allows the six read-only tools directly and asks before recharge, update_user_limits, or set_account_schedulable. Use the maintainer for any authenticated mutation.

Recharge protocol

Recharge is additive USD-style internal credit only; set and subtract operations are not exposed. Before every charge:

1. Collect an exact email and a positive amount.
2. Call sub2api-admin_preview_recharge. The server fuzzy-searches but accepts only one exact normalized email match.
3. Read back the matched email, the immutable user id, status, the current balance, the amount, and the expected balance; obtain explicit confirmation for that exact tuple.
4. Generate one stable Idempotency-Key of 16-128 URL-safe characters. Keep and reuse that key when the result is uncertain; never generate a fresh key for a retry.
5. Call sub2api-admin_recharge with operation: add, the exact email and id, the audit note, and the idempotency key.
6. Re-read the user and balance history; record the exact verified or accepted_unverified outcome. Never rewrite an unverified accepted response as success.

No account is recharged as part of installation. Do not recharge inactive users unless the operator explicitly confirms the inactive status.

Limits and model availability

Sub2API v0.1.176 has no configurable per-model RPM. Operator scope:

• User RPM and concurrency via sub2api-admin_update_user_limits.
• Group RPM is set through the admin UI; this MCP does not expose it.
• rate_multiplier changes billing, not request rate.

For temporary model unavailability, call sub2api-admin_model_control_options first and identify the model’s actual scheduling boundary:

• Dedicated account: sub2api-admin_set_account_schedulable toggles every model served by that account; roll back with the inverse value.
• Restricted channel: remove the exact model from the channel pricing/model list while restrict_models=true; preserve the previous payload for rollback. This MCP does not mutate channels, use the admin UI.
• Composite model: disable the exact composite route; preserve prior state. Use the admin UI.
• models_list_config only hides discovery. It does not block direct requests and must never be reported as a disable.

Scheduling changes affect new requests only. They do not cancel requests or streams already in progress. If the model spans multiple accounts, channels, or routes, enumerate and confirm every affected target before mutation.

Secret rotation

Sub2API supports only one Admin API Key. Generating or regenerating the key in the admin UI immediately invalidates the previous key, so the MCP will briefly fail to authenticate until the Ops Agent workload restarts with the new value. Plan an expected maintenance window of one to two restart cycles. Rotate in this exact order without intermediate pauses:

1. Generate the new key in the Sub2API admin UI and immediately enter it into the silent prompt of manifests/ops-agent/create-sub2api-mcp-secret.sh. Confirm the helper reports application/sub2api-mcp keys=admin-api-key.
2. Restart the managed workload so the new env var is injected:

   kubectl -n application rollout restart deployment/ops-agent
   kubectl -n application rollout status deployment/ops-agent --timeout=300s

3. Verify global/health reports healthy: true, /mcp reports exactly {"sub2api-admin":{"status":"connected"}}, and the merged OpenCode config carries the cluster-local SUB2API_BASE_URL without the new key.
4. Through 72602-k3s-maintainer, run sub2api-admin_read against /api/v1/admin/system/version; the live application must respond 0.1.176.

The upstream “generate key” action is itself the invalidation step; no follow-up disable of the previous key is needed or supported.

Tests: node --test .opencode/mcp/sub2api-admin/server.test.mjs passes three tests covering the allowlist, path-traversal rejection, and the recharge idempotency path.

GitOps Rollback

Restore the previous reviewed chart and image values, and any reviewed OpenCode configuration, with git revert or a new commit, then let ArgoCD converge:

git -C /home/aaron/Ops/docs log --oneline -- manifests/sub2api-argocd.yaml .opencode/opencode.json
git -C /home/aaron/Ops/docs revert <change-commit>
git -C /home/aaron/Ops/docs push origin HEAD:main
# Record rollback as a new Git revert commit (upgrade commit 67d8fef5859ec579cee101da22cc4988ca789b84); backup artifacts are in `/home/aaron/Ops/backups/sub2api/upgrade-20260920T051850Z` for isolated DB restore.

argocd app get ops-docs --hard-refresh
argocd app sync ops-docs --revision main
argocd app wait ops-docs --sync --health --timeout 300
argocd app sync sub2api
argocd app wait sub2api --sync --health --timeout 600

kubectl -n application rollout restart deployment/ops-agent
kubectl -n application rollout status deployment/ops-agent --timeout=300s

After the Ops Agent Pod rolls back and no Deployment references the helper Secret, remove it:

test -z "$(kubectl -n application get deployment -o json \
  | jq -r '.items[].spec.template.spec.containers[]
          | select(.env[]?.valueFrom.secretKeyRef.name=="sub2api-mcp")
          | .name')" && \
  kubectl -n application delete secret sub2api-mcp

An image/chart rollback does not reverse a forward-only PostgreSQL migration. Confirm compatibility with the migrated schema; when database recovery is required, restore the pre-upgrade dump into a separate database and switch via a reviewed Git change. Do not use kubectl rollout undo, and do not delete PVCs or Secrets as a rollback step until the Git revert and ops-agent restart are Healthy.

Troubleshooting

kubectl -n application logs deployment/sub2api --since=10m
kubectl -n application get events --sort-by=.lastTimestamp
kubectl -n application get endpointslice \
  -l kubernetes.io/service-name=sub2api -o yaml
argocd app get sub2api --hard-refresh

🛎️FAQ

kubectl -n argocd get application sub2api \
  -o jsonpath='{.spec.source.helm.values}'
kubectl -n application get secret sub2api-external-postgresql
kubectl -n application logs deployment/sub2api --since=10m

redis:
  auth:
    existingSecret: sub2api-redis
    existingSecretPasswordKey: redis-password

kubectl -n application get secret sub2api-redis
kubectl -n application get deployment sub2api \
  -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}{" <- "}{.valueFrom.secretKeyRef.name}{"/"}{.valueFrom.secretKeyRef.key}{"\n"}{end}'
kubectl -n application logs deployment/sub2api --since=10m