Operations Runtime

Support-facing runtime notes confirmed in current backend code.

• Main app and realtime server both use `express-session` with `connect-redis`.
• Relevant env vars:
• `SESSION_COOKIE_NAME` default: `sessionId`
• `SESSION_SECRET`
• `COOKIE_TTL`
• `COOKIE_DOMAIN`
• `REDIS_SESSION_HOST`
• `REDIS_SESSION_PORT`
• `REDIS_SESSION_PREFIX`
• Session cookie defaults:
• `sameSite: lax`
• `httpOnly: true`
• `secure: true` in production
• Important TTL nuance:
• cookie max-age defaults to `7 days` unless `COOKIE_TTL` overrides it
• Redis session-store TTL is hard-coded to `24 hours`
• Redis clients now log availability transitions for session and queue clients. Repeated error and reconnect logs are quieted for `60` seconds, while unavailable, ready, and disconnected transitions are logged when state changes.
• `/api/init/whoami` also self-heals org selection:
• if the user has no `currentOrganization`, it selects the first org
• if the user has no orgs at all, it creates `My Organization` with `2000` credits

• Main app health endpoint:
• `GET /health`
• returns `status` and `timestamp`
• in non-production, also returns `env`, `session`, and `uptime`
• Realtime server health endpoint:
• `GET /chat/realtime/health`
• returns `status`, `service: realtime`, and `timestamp`

• User WS tokens use `COOKIE_TTL`, defaulting to `7 days`.
• Webchat WS tokens use `COOKIE_TTL` when set, otherwise default to `24 hours`.
• Webchat JWT session tokens use `WEBCHAT_JWT_TTL`, defaulting to `7 days`.

• Generated widget snippets start from `API_DOMAIN`.
• If org webchat settings set `widgetDomain = proxy` and `API_PROXY_DOMAIN` exists, snippets use the proxy domain instead.
• Current snippets load `/widget/alloy-webchat-widget.js` and initialize with:
• `apiKey`
• `eventId`
• `eventId` is routing metadata for the widget entry point. The runtime execution graph still starts from the published workflow or employee workflow tied to that event.

• In development, the backend can proxy local widget assets when `LOCAL_WIDGET_DEV_PROXY_ENABLED` is enabled or when `NODE_ENV=development`.
• Proxy target comes from `LOCAL_WIDGET_DEV_ORIGIN`.
• Proxied paths include `/widget` and widget asset paths under it.
• `/widget/alloy-webchat-widget.js` and HTML responses get `Cache-Control: no-cache, no-store, must-revalidate`.
• Hashed widget assets under `/widget/assets/` get `Cache-Control: public, max-age=31536000, immutable`.
• If the local widget origin is missing or unavailable, the proxy returns an error instead of silently falling through.

• Pending realtime session configs are stored in Redis under:
• `${REDIS_PREFIX || 'alloy_dev'}realtime:{sessionId}`
• TTL is `20 minutes`.
• Configs are one-time use and are deleted on first successful retrieval.
• Ally realtime session creation is:
• `POST /chat/realtime/ally/create-session/:organization_id`
• Ally realtime runs seed runtime context with:
• `targetOrganizationId`
• `userId`

• Main app request-body limits:
• JSON: `300mb`
• URL-encoded: `300mb`
• HTTP server timeouts for both app and realtime:
• `requestTimeout = 600000`
• `headersTimeout = 600000`
• `keepAliveTimeout = 5000`
• Storage uploads still enforce `10 MB` per file.
• Internal-chat attachments are limited to:
• max `5` files
• max `10 MB` each
• allowed MIME families: images, PDF, SQLite, plain text, XML

• Mastra agent generation receives the shared Mastra logger.
• The logger uses Pino-backed redaction for sensitive provider diagnostic fields, including API keys, authorization values, prompts, messages, tool payloads, request bodies, response bodies, response headers, user IDs, run IDs, message IDs, and conversation IDs.
• AI provider errors logged during primary-model and fallback-model retries are summarized before logging, so support logs can show provider failure metadata without raw prompts, tool inputs, headers, or provider response bodies.

• Both servers use `http-graceful-shutdown`.
• Shutdown timeout is `450000 ms`.
• Cleanup closes:
• `redisClient`
• `redisQueueClient`
• Sequelize
• `uncaughtException` and `unhandledRejection` are reported to Sentry and `telegramError`.
• Telegram delivery requires:
• `TELEGRAM_BOT_API_KEY`
• `TELEGRAM_CHAT_ID`
• Optional Telegram thread routing:
• `TELEGRAM_EVENTS_THREAD_ID`
• `TELEGRAM_ERRORS_THREAD_ID`
• `TELEGRAM_FEEDBACK_THREAD_ID`
• `TELEGRAM_WARNINGS_THREAD_ID`

• Every organization has a runtime `status` field: `active` or `inactive`.
• Default for new organizations is `active`. Status changes are performed directly in the database.
• `Organization.safeData()` exposes `status` so the frontend can display the current org state.
• Inactive organizations keep their data and allow authenticated app/admin access, but block all external API usage and runtime automation:
• Public API (`/api/public/*`) returns `403` with `{ error: "Organization is inactive", code: "ORGANIZATION_INACTIVE" }` (except `OPTIONS` for CORS).
• Public trigger API and public webchat/trigger entry points are blocked.
• Private API mutating methods (`POST`, `PUT`, `PATCH`, `DELETE`) are blocked.
• Private `GET /api/private/stargate/?system=...` (long-polling for tasks) is blocked.
• Workflow and trigger runs started from the app UI, AI tools, manual scheduler runs, and realtime voice are blocked.
• Inbound webhooks (Slack, MS Teams, Telegram) return `200` with the `ORGANIZATION_INACTIVE` payload instead of `403`, so external platforms do not retry.
• Hosted MCP endpoints are blocked.
• Read-only private `GET` endpoints (storage listing, metadata, downloads) continue to work.

• Session issues:
• verify Redis availability
• compare cookie TTL vs Redis store TTL
• verify cookie domain and allowed origins
• Wrong widget snippet domain:
• check org webchat widget settings plus `API_DOMAIN` and `API_PROXY_DOMAIN`
• Realtime connect failures:
• check the pending Redis key has not expired or been consumed already
• confirm the workflow/voice models still exist
• Missing Telegram alerts:
• verify bot token, chat ID, optional thread IDs, and bot access to the target chat

• The mailer supports two providers, selected by env var:
• `MAILERSEND_API_TOKEN` — uses MailerSend (commercial email API).
• `SMTP_HOST` — uses SMTP (self-hosted or relay).
• When both are set, MailerSend takes priority.
• SMTP env vars: `SMTP_HOST`, `SMTP_PORT`, `SMTP_SECURE`, `SMTP_USER`, `SMTP_PASS`, `SMTP_FROM`.
• Both providers share EJS templates and the same `sendEmail` contract.
• If neither provider is configured, sending throws at runtime.

• Developer-only install and migration scripts include:
• the backend root installer, which orchestrates component setup, dependency installation, database startup, database creation, and migrations
• per-component environment installers for the backend, frontend, webchat widget, authentication stack, core Docker stack, and optional Google Workspace MCP stack
• `src/scripts/installer/install.ts` for application bootstrap after the first user exists
• `src/scripts/export/organization.ts` for creating an organization SQL dump
• `src/scripts/import/organization.ts` for importing a dump in a transaction
• `scripts/check-search-tools.js` for checking configured storage search tools during install validation
• Environment installers preserve existing values, back up existing environment files, derive or generate values where possible, and prompt for remaining values. They can be rerun to fill missing settings.
• The core Docker installer renders nginx configuration and local TLS certificates from tracked templates. Rendered configuration, generated certificates, and dynamic environment files are ignored by Git.
• The core Docker Compose stack includes a MinIO S3-compatible object store. The storage client supports the local self-signed TLS setup when the deployment explicitly enables insecure TLS for development; this must not be enabled in production.
• The application bootstrap installer lists existing users and asks which user ID should become system admin.
• It then ensures that user belongs to the system organization, resolves or creates the global Ally, stores the global Ally setting, and seeds missing default prompts, model, and fallback settings without overwriting existing values.
• The organization import script wraps inserts in `BEGIN` / `COMMIT`; insert conflicts abort the import and roll back previous inserts.
• These scripts are operational tooling, not product UI.

• `npm run cron:install` renders `cron/crontab` and replaces only the block between the Alloy managed-cron markers in the current user's crontab.
• The installer requires `BACKEND_PATH`.
• `CRON_LOG_DIR` sets the cron output directory. When omitted, it defaults to the current user's `logs/cron` directory; the installer creates the directory when needed.
• The rendered crontab prepends the Node executable directory to `PATH`, followed by `/usr/local/bin:/usr/bin:/bin`.
• The current managed job runs `npm run cron:refresh-ai-provider-tokens` hourly and appends stdout/stderr to `refresh-ai-provider-tokens.log` in the cron log directory.
• Manual changes inside the managed block are overwritten on the next install; entries outside the block are preserved.

• `PUT /api/admin/settings/:key` now validates that `alloy_employee_id` and `onboarding_employee_id` values reference actual global system AI teammates (employees with `is_system: true` and `owned_by_user_id: null`).
• Setting either key to a personal Ally employee returns `400` with `"Setting value must reference a global system AI teammate"`.
• A migration (`20260710120000-repair-global-system-employee-settings.js`) automatically repairs system settings that were pointing to personal Ally employees, reassigning them to the correct global system employees.