Provider outage and manual fallback
- Status: Deterministic local mocks exist; real-provider operations pending
Scope
This runbook covers future meal analysis, nutrition lookup, identity, subscription, notification, or health-provider degradation. No real provider is integrated in M0.
M0 behavior
.env.example enables deterministic synthetic mocks for local development. Mocks make tests repeatable and avoid vendor calls. They do not simulate every timeout, malformed response, quota limit, callback delay, or privacy failure unless a test explicitly does so.
Do not switch a future production environment to mocks to conceal an outage.
Detection signals
Future signals are PENDING:
- Timeout, error, throttle, and malformed-response rates by adapter
- Queue age, depth, retries, and dead-letter count
- Callback signature/freshness failures
- Entitlement reconciliation drift
- Analysis completion p90 against the under-15-second target and stale pending records
- Vendor status/incident notice corroborated with PlatePilot telemetry
Keep provider payloads and adult content out of metrics and routine logs.
Future response
- Confirm the affected adapter, operations, start time, and data/privacy impact.
- Open an incident at the appropriate severity.
- Stop unbounded retries and protect queue/database capacity.
- Use the adapter kill switch or circuit breaker when approved.
- Preserve unaffected functions and existing adult records.
- Present honest pending/unavailable states and a next action.
- For analysis or lookup, offer manual meal entry without requiring a subscription-provider response.
- For billing, avoid fabricating entitlement; preserve non-paid access and reconcile later.
- For identity, do not bypass authentication. Existing sessions follow approved risk rules.
- Contact the provider through the future approved support/security route; share minimum necessary identifiers.
All production controls and contact routes above are PENDING implementation.
Queue and analysis recovery
Before replaying future jobs:
- Confirm the provider is stable and capacity limits are known.
- Verify jobs are authorized, unexpired, idempotent, and still desired.
- Do not analyze a photo after its deletion or retention deadline.
- Ensure the API deletes each exact
quarantine/ version after committing its sanitized
processing/ source, and that the worker deletes processing sources after analysis; do not rely
on the 1-day lifecycle fallbacks during ordinary operation.
- Replay gradually with bounded concurrency.
- Quarantine malformed or repeatedly failing jobs.
- Reconcile duplicate/provisional results and quota accounting.
Never overwrite an adult-confirmed manual correction with a late automated result.
Communication guidance
Recommended:
Photo analysis is temporarily unavailable. You can enter this meal manually and try analysis again later.
Avoid naming a vendor unless approved, promising a recovery time without evidence, or implying saved data was lost when only analysis is delayed.
Exit criteria
- Error and latency signals remain healthy through a monitoring window.
- Backlog is draining safely without duplicate records or quota errors.
- Manual entries and adult-confirmed values are preserved.
- Provider privacy/security questions are resolved.
- Reconciliation and incident follow-ups have owners.