ContentEngine — Executive System Overview

Author: Alton Wells Date: March 2026 Status: Final Architecture Specification Consolidates: Technical Specification v3 + Addendums #1, #2, #3

Table of Contents

1. Executive Summary
2. Technology Stack
3. Data Architecture
4. Master Workflow
5. Agent System
6. Strategy Layer
7. Content Layer
8. Production Layer
9. Content Type Registry
10. Refresh Queue System
11. Observability & Audit Trail
12. Cost Model
13. Risk Matrix
14. Future: Filesystem-as-Context

1. Executive Summary

ContentEngine is an autonomous AI content production system that replaces manual content marketing workflows with a three-layer agentic pipeline. The system is built on Mastra (TypeScript agent framework), LangExtract (structured document extraction), and Firecrawl (web crawling and sitemap intelligence). It manages the full content lifecycle from competitive intelligence through publication and performance monitoring.

The system is designed for Consul, a B2B SaaS AI executive assistant targeting CEOs and founders at a $200/month price point. Content must build trust with sophisticated, high-ticket decision-makers, making E-E-A-T signals, programmatic SEO validation, and editorial quality non-negotiable.

Core Architectural Principles

• Humans set strategy and approve output. AI executes everything in between. Three human gates govern strategic decisions, editorial review, and final publication.
• No vectors. No embeddings. Structured extractions, hierarchical summaries, and an explicit content relationship graph replace RAG. Stanford's 2025 research shows embedding precision collapses 87% beyond 50K documents. This approach scales without dimensional decay.
• Programmatic SEO validation is non-negotiable. Every published piece must pass a 10/10 deterministic SEO check across 10 blocking checks, 6 warning checks, and 6 informational metrics.
• Context is navigated, not stuffed. Agents traverse a four-level hierarchy (Domain → Cluster → Page → Entity) loading only what they need. Typical planning context: ~14,000 tokens instead of millions.
• Everything is traceable. Every extraction maps to its source location. Every graph edge has provenance. Every agent decision is recorded in a structured audit trail with token usage, tool call sequences, and cost tracking.

2. Technology Stack

2.1 Core Framework

2.2 Extraction & Intelligence

2.3 LLM Providers

Claude Sonnet 4 (anthropic/claude-sonnet-4-20250514) powers all nine Mastra agents across strategy, content, and production layers. Temperature is configured per agent: 0.8 for the Writer (creative generation), 0.4 for the Editor (precision), 0.1–0.3 for production agents (mechanical execution). Gemini 2.5 Flash handles all LangExtract extraction pipelines and hierarchical summary generation.

2.4 Data & Hosting

PostgreSQL (no pgvector) serves as the primary database for all structured data, extraction entities, graph adjacency tables, summaries, and content plans using JSONB for flexible extraction attributes. Drizzle ORM provides type-safe database access. The application layer uses Next.js 15+ for the UI (calendar, editor, dashboards) hosted on Vercel, with agent workers and the LangExtract sidecar running on Railway. Trigger.dev handles durable job scheduling for crawls, extractions, summary regeneration, and post-publish monitoring.

2.5 System Architecture

2.6 External APIs

3. Data Architecture

ContentEngine replaces conventional RAG/vector embeddings with a three-layer memory model. All data is stored in PostgreSQL as structured, queryable records.

3.1 Three-Layer Memory Model

3.2 Layer 1 — Structured Extraction (LangExtract)

Every document entering the system — competitor pages, our content, SERPs, AI Overviews, brand voice samples — is processed through LangExtract extraction pipelines. Raw text becomes structured, source-grounded entities stored in typed Postgres tables. Agents query structured data, not fuzzy similarity scores.

Six extraction classes are defined, each with a fixed schema, dedicated prompt description, and a minimum of three few-shot examples:

Integration decision: LangExtract runs as a Python FastAPI sidecar (not the unofficial Node SDK) because critical features — multi-pass extraction, cross-chunk coreference resolution, and controlled generation via Gemini schema constraints — are Python-only. A circuit breaker pattern (3 consecutive failures opens the circuit) mitigates the Python–TypeScript bridge as a single point of failure.

3.3 Layer 2 — Hierarchical Document Summaries

A four-level summary tree enables agents to navigate from broad domain context to specific entity-level detail, loading only relevant branches:

Agents start at Level 0 and drill down only into relevant branches. Summaries are regenerated from fresh extractions and timestamped for versioning.

3.4 Layer 3 — Content Relationship Graph

An adjacency table in Postgres with typed edges connects content entities. This replaces vector similarity for all "find related content" operations. Each edge carries a confidence score, provenance (which agent or system created it), and a last-validated timestamp.

4. Master Workflow

ContentEngine operates across three workflow layers with human control at each transition. Workflows own the sequence ("what happens next"); agents own the execution ("how do I do this step"). This is deterministic orchestration with agentic execution — no supervisor/sub-agent patterns.

4.1 End-to-End Flow

4.2 Human Gates

Each human touchpoint is a Mastra workflow suspension. The workflow suspends and provides a structured payload describing the review task. The Next.js app renders the appropriate UI and calls the resume endpoint when the human completes their action.

Design principle: Image placement is manual at Gate 3. The Writer Agent outputs [IMAGE: description] markers as placement suggestions. Image selection requires brand aesthetic judgment, rights verification, and contextual sensitivity that current AI image generation does not handle reliably at production quality.

5. Agent System

5.1 Agent Configuration

Every agent's model, temperature, and step budget are configurable at runtime via a database settings table. No model IDs are hardcoded. This allows model swaps without redeployment. Changes take effect on the next agent invocation.

Temperature rationale: Strategy agents are moderate (creative planning grounded in data). The Writer is highest (creative generation). The Editor is low (precision). Production agents are near-zero (mechanical execution).

5.2 Tool Inventory (26 Tools)

Tools are grouped into four categories. Each agent receives only the tools it needs — a smaller tool surface means fewer irrelevant calls, lower token usage, and easier debugging.

5.3 Tool-to-Agent Assignment

5.4 Context Navigation Pattern

The Content Strategy Agent — the most complex decision-maker with 10 tools — demonstrates how agents navigate the hierarchy to build precisely relevant context:

1. Read active strategy directives (~500 tokens)
2. Read combined domain summary for the big picture (~500 tokens)
3. Ingest Competitive Intelligence report from workflow state (~2,000 tokens)
4. Ingest Search Landscape report from workflow state (~2,000 tokens)
5. Read per-pillar cluster summaries (~3,000 tokens)
6. Check current calendar to avoid duplication (~1,000 tokens)
7. Traverse graph for gaps, cannibalization, and outperformance (~1,500 tokens)
8. Drill into specific page summaries for top candidates (~2,000 tokens)

Total: ~14,000 tokens of precisely relevant context, versus the impossibility of stuffing 847+ full pages into a context window.

6. Strategy Layer

6.1 Competitive Intelligence Agent

Continuously analyzes the competitor database — structured LangExtract data, not raw HTML — and produces actionable competitive insights. Runs weekly for full analysis and daily for a lightweight change digest.

Outputs: Competitor moves (with relevance scoring and source extraction IDs), content gaps (with estimated impact), positioning insights. Every finding includes provenance for traceability.

6.2 Search Landscape Agent

Monitors keyword performance, SERP composition, AI Overview appearances, and search trends using structured SERP data. Runs daily for ranking changes and AI Overview monitoring, weekly for full landscape analysis.

Outputs: Ranking changes with trend classification, AI Overview citation alerts, emerging keyword opportunities, declining content flags, SERP feature opportunities.

6.3 Content Strategy Agent

The brain of the system. Synthesizes both prior agent outputs with content inventory, strategy directives, and graph relationships to produce a prioritized, scheduled content plan.

Scoring model: strategic_alignment × search_opportunity × competitive_urgency × gap_severity. The agent checks the graph for cannibalization before recommending new content, respects human-added calendar items as fixed constraints, and suggests scheduling based on capacity.

Outputs: Prioritized plan items with title, target keyword, content type, rationale, competitive context, schedule date, priority (1–3), estimated impact, internal link targets, and graph evidence. Plan items are written to the database with source: "ai_generated" and the workflow suspends for human calendar review.

7. Content Layer

7.1 Content Brief Agent

For each approved plan item, generates a detailed brief including: complete H2/H3 outline with keyword mapping per section, competitor differentiation strategy (using structured extraction data), internal linking targets from the graph, external resource recommendations, and brand voice requirements. The brief is saved and the workflow suspends for human approval (Gate 2).

7.2 Writer Agent

Receives the approved brief, brand voice extractions, and strategy context as dynamic instructions. Produces a complete first draft that follows the outline exactly, hits word count targets (±10%), integrates keywords naturally, includes all specified links, and marks image placement opportunities as [IMAGE: description] for human insertion.

Deliberate constraint: The Writer has only 4 tools (webSearch, queryExtractions, traverseContentGraph, and the brief context). Most of its context comes from the pre-assembled brief, not from live queries.

7.3 Editor Agent

Reviews the draft against seven dimensions: language correctness, verbal consistency (terminology, voice), brand voice adherence (compared against extracted patterns), factual grounding (claims verified against brief sources and web), structural quality, link integrity (all internal links verified against published pages), and keyword optimization.

Outputs: Overall pass/needs_revision assessment, specific edits with location, type, severity (critical/suggested), and fix suggestions. Voice consistency score (0–100), readability score, and optionally a revised draft.

7.4 Revision Loop

If the Editor returns "needs_revision" with critical edits, the draft returns to the Writer with the edit list as additional context. Maximum 2 revision cycles. After 2 cycles, the draft proceeds to human review regardless — humans catch what agents miss. This prevents infinite loops while maintaining quality.

8. Production Layer

8.1 Human Draft Review (Gate 3)

The most important step in the entire system. The workflow suspends and presents the draft alongside the brief in a side-by-side editor interface. The human reviewer:

• Reads and assesses the full draft against the brief
• Adds personal experience and original insights (the irreplaceable 20%)
• Places and curates images at [IMAGE:] markers
• Edits for voice and brand consistency
• Fact-checks statistics and claims
• Approves or rejects (rejection sends back to Content Layer with notes)

8.2 Final Cleanup Agent

A lightweight technical-only pass after human edits. Checks markdown formatting validity, image alt text and dimensions, internal link resolution, heading hierarchy integrity, and consistent list formatting. Does not change tone, wording, or content.

8.3 SEO Validation Engine

Deterministic code, not an LLM. Operates as a CI/CD deployment gate with three severity tiers:

8.4 Publishing & Post-Publish Pipeline

The post-publish pipeline closes the data loop: extraction → summary regeneration (L2 → L1 → L0) → graph edge construction → bidirectional linking → monitoring. Content is not rolled back on pipeline failure — the content is live, and internal data syncs on the next scheduled job.

9. Content Type Registry

ContentEngine produces eight content types, each with a distinct structural template, schema mapping, internal linking profile, image density rule, and refresh cadence.

Each type carries a structural scaffold that the Writer Agent must follow exactly and the SEO Validation Engine checks at the deployment gate. Every page carries a @graph array of JSON-LD entities including Organization, WebSite, WebPage, the content-type-specific schema, author Person entity, and BreadcrumbList.

9.1 Schema Architecture

9.2 Author Entities & E-E-A-T Strategy

Three author entities are defined, each with a dedicated profile page at /authors/[slug]/:

• Alton Wells — Primary human author with full E-E-A-T credentials
• Stan — Secondary human author
• Auctor — AI editorial assistant, presented transparently. Profile describes the human-AI editorial process: articles are researched and drafted using AI, then reviewed, enriched with original insights, and approved by a human author.

Author profile pages include headshot/avatar, bio, expertise tags linked to pillar pages, social links, recent article feed, and ProfilePage + Person schema markup.

10. Refresh Queue System

Every published page is re-evaluated on a maximum 90-day cycle. Re-evaluation does not mean automatic refresh — it means the system scores refresh urgency and only pages crossing a threshold enter the active queue.

10.1 Urgency Score (0–100)

10.2 Refresh Tiers

Refreshes are mixed into the content calendar at a 70/30 ratio (new content / refreshes). Light refreshes consume 0.25 capacity units, moderate 0.5, and heavy 1.0. As the content library grows, the ratio naturally shifts. If the refresh queue is empty, 100% of capacity goes to new content.

11. Observability & Audit Trail

Every agent invocation produces exactly one audit log row capturing workflow context, timing, token usage, cost estimates, the ordered sequence of tool invocations, and output summaries.

11.1 What Gets Logged

• Workflow ID, run ID, step ID, and agent ID for full traceability
• Model ID used (resolved from config at invocation time)
• Start time, completion time, and duration in milliseconds
• Steps used vs. max steps configured
• Input tokens, output tokens, total tokens, and estimated cost (USD)
• Ordered tool call sequence with input summaries, output summaries, per-call duration, and token usage
• Status: success, failed, failed_max_steps, or timeout
• Error message and type if applicable

11.2 Dashboard Views

• Cost by period: daily/weekly/monthly spend breakdown by agent
• Agent efficiency: average steps used, max-steps failures, step utilization ratio
• Workflow run detail: full step-by-step timeline for any specific run
• Slowest agents: average and max duration for performance optimization

11.3 Error Handling

• maxSteps policy: When an agent exhausts its step budget, the workflow captures the partial output, marks the step as failed_max_steps, logs the full tool call history, sends a Slack notification, and halts the workflow run. This is not a silent failure.
• Malformed output: Zod validation catches invalid structured output. One retry with a corrective note. If retry fails, step fails.
• Tool failures: Agents see error messages and can retry or use alternative approaches. The LangExtract sidecar uses a circuit breaker (3 consecutive failures opens the circuit, 60-second recovery window).
• Post-publish pipeline: Each job retries 3× with exponential backoff (10s, 30s, 90s). Failed jobs do not roll back the publication — the content is live, but internal data is temporarily out of sync until the next scheduled job catches up.

11.4 SEO Validation Tracking

Every validation run (pre-publish, refresh evaluation, scheduled weekly audit, manual) is stored with full check results. A weekly Trigger.dev job runs the complete validation suite against all published pages to catch degradation from external changes. Validation score decay feeds into the refresh urgency score.

12. Cost Model

Monthly estimates based on 50 published pieces, 10 competitors, and 100 tracked keywords:

13. Risk Matrix

14. Future: Filesystem-as-Context

The current hierarchical summary approach works well but has a ceiling: summaries are pre-generated snapshots. As the content library scales to thousands of pages, keeping summaries fresh becomes continuous compute cost, and pre-computing what context agents need is inherently wasteful.

The filesystem-as-context pattern (inspired by Andrej Karpathy's context engineering framework and Anthropic's Skills system) offers a superior approach at scale: instead of pre-loading context, structure all system knowledge as a navigable filesystem. Agents use ls, grep, glob, and file reading to pull exactly the context they need for the current task.

Context efficiency gain: An agent navigating the filesystem builds ~3,500 tokens of precisely relevant context for a planning task, compared to ~14,000 tokens with the hierarchical summary approach — because the agent decides what to load based on the actual task.

Implementation effort: 5–7 weeks on top of the base system: filesystem generation pipeline (2–3 weeks), sandbox environment per agent session (1–2 weeks), filesystem-aware agent prompts (1 week), and a hybrid SQL + filesystem approach for real-time data.

Recommended path: Build the base system using hierarchical summaries + graph first. Validate at current scale. Implement the filesystem layer when agent context quality becomes a bottleneck (likely at 500+ pages, 10+ competitors, 50+ pieces/month).

This is a living specification. Addendums #1–3 provide full implementation detail for each layer.