m8trxDeployer Architecture

Network Architecture

Each customer is fully isolated at the AWS network layer. There are no VPC peering connections, no shared subnets, and no AWS-internal paths between customer VPCs. Operator and Brain reach each customer only over Tailscale (an encrypted overlay) using customer-specific tags. Each Agent customer is also reachable from the public internet on its own subdomain over HTTPS, with TLS terminated by Caddy on the instance.

VPC Design (Per Customer)

Security Group Per Template

DNS & Subdomain Architecture

Every M8trx Agent instance gets its own HTTPS subdomain under the platform zone (m8trx.ai). Subdomains are created and torn down by the dashboard through the Cloudflare API (dashboard/services/domain.py) — DNS only (proxied=false). Cloudflare does not proxy traffic; TLS is terminated on the customer instance by Caddy using a Let's Encrypt cert obtained at first boot via HTTP-01. Endpoints: GET / POST / DELETE /api/platform/domains/{customer_name}.

When a customer has one agent the subdomain label defaults to the customer name (acme.m8trx.ai). When a customer has multiple agents, each instance is deployed with a distinct subdomain label so its DNS record points at its own EIP — the convention is {customer-name}-{role} (e.g. acme-inbox.m8trx.ai, acme-finance.m8trx.ai) or {customer-name}-{n}. The deploy form's Subdomain field is the operator's lever for this — it accepts any DNS-safe label, and the same label is used as the customer's display name on the welcome card.

Tailscale Overlay = Per-Customer "VLAN"

Operator and Brain traffic to customer instances does not traverse the public internet — every M8trx Agent instance joins a private Tailscale tailnet under a customer-specific tag (tag:m8trx-cust-<customer-uid>). All of one customer's agents share the same tag, so an ACL rule that says "src=tag, dst=tag:*" gives them a private network they can use to call each other. There is no equivalent rule across different customer tags, so cross-customer traffic is denied by default. Functionally, this is a per-customer VLAN built in software.

The dashboard mints a tag-bound, ephemeral, reusable, 90-day auth key per customer and writes it to SSM (the same key is reused across that customer's agents — that's why it's reusable). Each customer instance's user_data.sh joins the tailnet at first boot with that key, hostname set to the EC2 instance-id so the dashboard can find it for Tailnet Lock approval.

Compute Architecture

Two customer templates ship today. Both run on the same hardened Ubuntu 22.04 base AMI (Packer-baked) with identical OS controls. They differ in what runs above the OS and how the workload bootstraps. A planned third mode will pack multiple low-traffic customers onto a shared host without weakening per-customer isolation.

Two Customer Templates

M8trx Agent First-Boot Bootstrap (Path B)

The M8trx Agent template uses the same hardened AMI as the baseline — no agent-specific Packer build. The full stack is fetched and started at first boot via cloud-init. This means deploying a new Agent customer always picks up the latest M8trxAgent source (subject to a fixed --depth 1 clone of main); operator re-bakes are not in the critical path.

Cloud-init output is tee'd to /var/log/m8trx-agent/first-boot.log for SSM-side debugging. Total user-data is kept under the AWS 16 KB user-data hard limit — anything bigger goes into the cloned repo, not user-data.

Audit Rules (auditd) — Immutable, both templates

Planned: Multi-Tenant Compute (foundations in place, host-level orchestration pending)

IAM & Access Control

Each customer instance assumes a unique IAM role with a permission boundary that hard-caps maximum privileges.

Organization SCPs

Brain Telemetry — additional inline scope

Each M8trx Agent customer's instance role carries one extra inline policy, brain_telemetry_ssm_read, scoped to only that customer's SSM path. The permission boundary explicitly permits ssm:GetParameter / ssm:GetParameters so the inline policy isn't blocked at the boundary. Full detail with example policy JSON: Brain & Tailscale tab.

Brain Telemetry & Tailscale Mesh

A control plane that connects every M8trx Agent customer instance to a central Brain service over a Tailscale tailnet. The Brain stores per-customer metadata (events, agents, API keys) and is what the operator dashboard reads from when monitoring fleet posture. The mesh is the only network path the Brain ever uses to reach a customer; there is no public-internet path from Brain to customer.

The Customer UID — one identifier ties everything together

Every Agent customer has a single, stable, public identifier — the Customer UID — derived from the deployer-side customer name on first deploy and never reused. The UID drives every other per-customer artifact in the platform; getting it right matters because it is the only string we use to scope cross-system access.

A customer can run multiple agents (e.g. an inbox agent and a finance agent). All of that customer's agents share the same UID, the same Tailscale tag, the same brain customer entry, and the same SSM namespace — but get distinct subdomains and distinct EC2 instances. The shared tag is what gives them a private network to talk to each other on (the per-customer "VLAN" — see Network tab).

Status: the infra layer (Tailscale tagging, brain mint idempotency, SSM scoping, Cloudflare provisioning) is already ready for multiple agents under one UID — the auth key is reusable, the tag's accept rule has no device-count limit, and the SSM IAM scope wildcards over the namespace. The piece that is not yet wired is the deploy form: today it treats each new deploy as a new customer (1 deployer-name = 1 brain mint = 1 EC2). The next step is a "Tenant" dropdown that lets the operator add a new agent under an existing customer UID — re-using the brain customer, the Tailscale tag, and the SSM namespace, while creating a fresh subdomain and EC2.

SSM Parameter Layout (per customer)

All parameters live under /m8trx/<deployer-customer-name>/* (the deployer name, not the brain id — so user-data needs only the one name terraform already templates in). Five are SecureString (KMS-encrypted via alias/aws/ssm); only brain-customer-id and brain-url are stored in the clear.

Pre-Deploy Lifecycle (provision)

Order matters. Each step streams progress to the dashboard's job log via on_output. If any step fails, the deploy aborts before terraform apply touches AWS — so a half-provisioned customer never enters the brain. Source: dashboard/services/brain_telemetry.py.

Post-Apply: Tailnet Lock approval

After terraform apply succeeds, the dashboard polls Tailscale for a device with hostname == <instance-id>. When it appears (cloud-init has finished step 4 of bootstrap), the dashboard calls POST /api/v2/device/{device_id}/key with {"keyExpiryDisabled": false} to approve it under Tailnet Lock. This step is best-effort: if cloud-init is slow, the operator can still approve manually from the Tailscale admin console; the deploy job logs a warning rather than failing.

Destroy Lifecycle

Settings UI (one-time, fleet-wide)

All five fleet-wide brain-telemetry values are entered once via the dashboard's topbar settings cog (📡 modal). GET endpoints return masked values (last 4 chars + bullets); POST writes them to state.json.

IAM Scope (instance-side)

Each customer's EC2 instance role has a scoped brain_telemetry_ssm_read inline policy. The role can read only its own customer's path — a leaked instance role cannot pivot to another customer's brain key, Tailscale auth key, or PAT.

resource "aws_iam_role_policy" "brain_telemetry_ssm_read" {
  policy = jsonencode({
    Statement = [{
      Sid      = "ReadOwnBrainTelemetryParams"
      Effect   = "Allow"
      Action   = ["ssm:GetParameter", "ssm:GetParameters"]
      Resource = "arn:aws:ssm:*:*:parameter/m8trx/${local.ssm_customer_name}/*"
    }]
  })
}

The permission boundary on each instance role also explicitly permits ssm:GetParameter + ssm:GetParameters — without that, the policy above would still be denied at the boundary. local.ssm_customer_name strips any -agent resource-naming suffix back to the deployer name, so SSM paths and tfvars keys stay aligned.

Brain-Side Monitoring (what the team sees)

Once an Agent customer is online, Paperclip POSTs events (login, task created, task settled, finance/leads activity, errors) to the brain over the customer's bearer. The brain rolls these into per-customer dashboards (covered in the brain repo's own docs). Things the team monitors at the platform layer:

• Customer presence on the tailnet — every Agent customer should appear as a single device tagged tag:m8trx-cust-<id>. A missing or duplicate device is a deploy/health red flag.
• Last-event recency per customer — gap > 24h on a paid customer is worth investigating (instance down, brain unreachable, paperclip crash loop).
• Auth-key expiry pressure — keys are 90-day ephemeral; the team rotates ahead of expiry by re-deploying or rotating manually. Operator dashboard has a planned "key TTL" column on the customer list.
• Cross-customer ACL drift — if anyone manually edits the Tailscale ACL outside of tailscale_client.ensure_customer_acl_entries and adds a cross-tag accept rule, that breaks the per-customer trust boundary. Periodic ACL diff against the expected shape is the planned defense.

Failure Modes & Operator Actions

Data Protection

All data encrypted at rest with per-customer KMS keys and in transit with TLS.

AI Agent Sandboxing

AI agents handle sensitive customer data — email, financial documents, chats, business records. They run inside a Docker compose project with multiple independent restriction layers, so a single failure (a vulnerable Python dep, a leaked secret, a user-data exploit) does not give code-execution on the host or read access to another customer. The mechanisms below apply to both templates; the M8trx Agent template adds Caddy + Postgres + Ollama as additional compose services bound to the same protections.

Per-Service Notes

Management Access

Two distinct access paths for the operator team: (1) reaching the operations dashboard over Tailscale, and (2) reaching individual customer instances for shell or telemetry. Neither path uses a dedicated management proxy any more — the mgmt-proxy Terraform module was retired on 2026-04-21 because Tailscale-on-the-dashboard-host plus AWS SSM Session Manager cover both paths at zero extra cost.

Option Comparison

Setup — Tailscale on the Dashboard Host

# On the dashboard host (one-time):
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --hostname=m8trx-deployer-dashboard --advertise-tags=tag:mgmt
# follow the printed URL to authorise

# Then on your laptop, install Tailscale and sign in to the same tailnet.

export DASHBOARD_HOST=$(tailscale ip -4 | head -1)   # e.g. 100.98.195.91
export DASHBOARD_PORT=8443
export DASHBOARD_USER=admin
export DASHBOARD_PASS='<strong-password>'    # required, no default
cd dashboard && python main.py

Setup — SSM Port-Forward (break-glass for the dashboard)

# Install SSM plugin: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html

aws ssm start-session \
  --target i-DASHBOARD_INSTANCE_ID \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["8443"],"localPortNumber":["8443"]}'

# Open https://localhost:8443 in your browser.

Setup — SSM Session Manager (customer-instance shell)

Triggered from the dashboard's per-customer detail view. The dashboard runs ssm:SendCommand with an inline AWS-RunShellScript document, polling every 3s for output (max 300s). Sessions land as the ubuntu user (not root, not ssm-user) — so paths and ownership match what the customer's compose stack expects. Commands are validated against a blocklist (rm -rf /, mkfs, shutdown, pipes-to-shell, etc.) and capped at 2000 chars. Endpoint: POST /api/customers/{name}/exec.

# Or from the AWS CLI directly:
aws ssm start-session --target i-CUSTOMER_INSTANCE_ID
# lands as ubuntu, full audit in CloudTrail

Security Properties

Monitoring & Detection

Multi-layered monitoring with automated alerting across all customer environments.

Incident Response

One-command quarantine triggered from the dashboard UI or CLI.

Security Controls Matrix

Every threat has at least 3 independent control layers. If one fails, the others still protect.

Per-User Access Control

Each customer instance supports multiple users with role-based access, per-user data isolation, granular permissions, and full audit logging. This is the application layer running inside the Docker sandbox — distinct from the deployer-side dashboard auth (HTTP Basic Auth) and from the per-customer Caddy basic-auth that gates the agent UI. The model below applies to the baseline (custom-app) template; the M8trx Agent template uses Paperclip's own user/permission model — see Paperclip docs.

Roles & Permissions Matrix

Granular Permissions

API Endpoints

Security Controls

Repository Structure

Per-Customer Cost Breakdown (one customer, t3.medium, audited 2026-05-05)

Defaults from terraform/environments/prod/*.tfvars.json: instance_type=t3.medium, volume_size_gb=30. Retention from terraform/modules/vpc/main.tf (flow logs: 30d) and terraform/modules/ec2/user_data.sh (syslog 90d, auth 365d, audit 365d, agent 90d).

Shared platform cost: ~$3–6 / mo. SNS topic (free tier covers low-volume), Terraform state S3 bucket + DynamoDB lock (~$1/mo combined), CloudTrail (first trail free), AWS Config recorder ($0.003 per recorded item — adds up at scale, but trivial for the small fleet). The mgmt-proxy ($7/mo) was removed on 2026-04-21; Tailscale + SSM cover the same access surface at $0. Tailscale is on the free 100-device tier; the Brain server runs on existing infrastructure.

Cost Audit Findings (2026-05-05)

• Flow-log retention drift. 3 of 5 customer flow-log groups in production have retentionInDays = None (never expire) — m8trx-cleantest4-agent, m8trx-dustin-bot, m8trx-smoke-test-2. The VPC module sets retention_in_days = 30, so these are likely log groups created by VPC Flow Logs auto-provisioning before the Terraform-managed group existed. Action: set retention manually with aws logs put-retention-policy --log-group-name <name> --retention-in-days 30 for the three drifted groups, then audit the VPC module to ensure the log group is always created by Terraform (not auto-provisioned by AWS) so retention is enforced from day one.
• CW agent custom metrics are empty in practice. Baseline template's user_data installs amazon-cloudwatch-agent and configures cpu/mem/disk/net publishing under namespace m8trx/<customer>, but aws cloudwatch list-metrics returns empty for both m8trx/keithenterprises and m8trx/agent2. The user_data has || echo "Warning" fallbacks so failures don't abort boot — they just hide. Action: SSM into the baseline instance and run amazon-cloudwatch-agent-ctl -a status to see whether the agent is running and authorized to push metrics. If the install is failing silently, decide: fix it, or remove the metrics block from user_data and lean on the dashboard's SSM-fallback for memory/disk like the Agent template already does.
• Path B's missing CW agent is intentional. Not a finding — surfaced here for clarity. The Agent template's user_data is deliberately minimal (the 16 KB AWS user-data hard limit drove a slim-down in 94a1d04). The dashboard already handles this — Phase 3 → Monitoring notes that memory/disk are fetched on demand via SSM RunCommand when CW agent data is unavailable.

Deployment Guide

Complete step-by-step instructions for initial platform setup and deploying new customer instances. Follow these in order.

Phase 1 — One-Time Platform Setup

These steps are done once to set up the platform infrastructure. After this, deploying a new customer is a single dashboard click that takes ~5 minutes.

cd packer
./build-ami.sh    # runs `packer build` and prints the new AMI ID

{
  "aws_region": "us-east-2",
  "project_name": "m8trx",
  "admin_email": "ops@m8trx.ai",
  "base_ami_id": "ami-XXXXXXXXX",         // from step 1, used by both templates
  "m8trx_agent_ami_id": "ami-XXXXXXXXX",  // same as base_ami_id (Path B)
  "platform_domain": "m8trx.ai"
}

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --hostname=m8trx-deployer-dashboard --advertise-tags=tag:mgmt
# follow the printed URL to authorize

cd dashboard
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

export DASHBOARD_HOST=$(tailscale ip -4 | head -1)
export DASHBOARD_PORT=8443
export DASHBOARD_USER=admin
export DASHBOARD_PASS='<strong-password>'   # REQUIRED — no default
export AWS_REGION=us-east-2
export PROJECT_NAME=m8trx
python3 main.py

# In AWS Organizations console:
# Policies > Service control policies > Create policy
# Paste contents of policies/scp-guardrails.json
# Attach to the OU containing your account

Phase 2 — Deploying a New Customer (Repeatable)

After Phase 1, deploying a new customer is a single dashboard click that takes ~5 minutes. The form picks the template; the dashboard handles brain-telemetry provisioning, terraform apply, Cloudflare DNS, and Tailnet Lock approval as one orchestrated job.

Phase 3 — Ongoing Operations

Troubleshooting

API Token Usage Tracking

Per-customer Claude API token consumption analytics with cost estimation and configurable alert thresholds. Tracks input, output, cache read, and cache creation tokens across all Claude model tiers.

Architecture Overview

Data Model

Usage data is stored in a local SQLite database (usage.db) with the following schema.

Indexed on (customer), (day), and unique on (customer, day, model) for efficient querying and upsert operations.

Pricing & Cost Calculation

Costs are estimated using per-model pricing rates (per 1M tokens). The calc_cost() function computes costs across all four token types.

API Endpoints

All endpoints require HTTP Basic Auth. Mounted under /api/usage/ via FastAPI router.

Cost Alert Thresholds

Key Files