ContentEngine — Technical Specification v3

Author: Alton Wells
Date: March 2026
Status: Final Architecture Specification

Executive Summary

ContentEngine is an autonomous AI content production system built on Mastra (TypeScript agent framework), LangExtract (structured document extraction), and Firecrawl (web crawling/sitemap intelligence). It replaces the conventional RAG/vector embedding approach with a three-layer memory architecture: structured extraction via LangExtract, hierarchical document summaries, and an explicit content relationship graph — all stored in PostgreSQL.

The system operates across three workflow layers — Strategy, Content, and Production — with human control points at strategic decision-making (content calendar approval), editorial review (draft quality gate), and final publication (image placement, last-mile polish). AI handles everything between those gates: competitive intelligence gathering, search landscape analysis, content planning, brief generation, writing, editing, SEO optimization, and publishing.

Core architectural principles:

• Humans set strategy and approve output. AI executes everything in between.
• No vectors. No embeddings. Structured extractions + hierarchical summaries + graph relationships replace RAG. Stanford's 2025 research shows embedding precision collapses 87% beyond 50K documents. Our approach scales without dimensional decay.
• Programmatic SEO validation is non-negotiable. Every published piece must pass a 10/10 deterministic SEO check. No exceptions.
• Context is navigated, not stuffed. Agents traverse a hierarchy (Domain → Cluster → Page → Entity) to load only what they need. Total context per planning session: ~13,000 tokens instead of millions.
• Everything is traceable. Every extraction maps to its source location. Every graph edge has provenance. Every agent decision can be audited.

Table of Contents

1. Frameworks, Libraries & Integrations
2. Data Architecture
3. Master Workflow
4. Strategy Layer — Detailed Specification
5. Content Layer — Detailed Specification
6. Production Layer — Detailed Specification
7. Content Calendar & Application Interface
8. Programmatic SEO Validation Engine
9. Cost Model
10. Risk Matrix
11. Future: Filesystem-as-Context Architecture

1. Frameworks, Libraries & Integrations

Core Framework

Extraction & Intelligence

LLM Providers

Data & Storage

Application & Deployment

External APIs

Integration Architecture

2. Data Architecture

2.1 Memory Model: Three Layers Replacing RAG

Layer 1 — LangExtract Structured Extraction: Every document entering the system (competitor pages, our content, SERPs, brand voice samples) is processed through LangExtract extraction pipelines. Raw text becomes structured, source-grounded entities in typed Postgres tables. Agents query structured data, not fuzzy similarity scores.

Layer 2 — Hierarchical Document Summaries: A four-level summary tree where agents navigate from broad (domain-level) to specific (entity-level) context. Each level is an LLM-generated summary of the level below it. Agents start at Level 0 and drill down only into relevant branches.

Layer 3 — Content Relationship Graph: An adjacency table in Postgres with typed edges connecting content entities. Replaces vector similarity for all "find related content" operations.

Relationship types:

2.2 Database Schema

Competitor Intelligence:

Search & Keywords:

Our Content:

Content Calendar & Strategy:

Hierarchical Summaries:

3. Master Workflow

Human touchpoints (red nodes):

4. Strategy Layer

4.1 Competitive Intelligence Agent

Purpose: Continuously analyzes the competitor database (structured LangExtract data, not raw HTML) and produces actionable competitive insights.

Agent Definition:

Tools:

Output Schema:

Schedule: Weekly full analysis. Daily change digest (lightweight — only queryCompetitorChanges + readDomainSummary).

4.2 Search Landscape Agent

Purpose: Monitors keyword performance, SERP composition, AI Overview appearances, and search trends using structured SERP data (LangExtract-processed, not raw HTML parsing).

Agent Definition:

Tools:

Output Schema:

Schedule: Daily for ranking changes and AI Overview monitoring. Weekly for full landscape analysis.

4.3 Content Strategy Agent (The Planner)

Purpose: The brain. Synthesizes Competitive Intelligence output, Search Landscape output, our content inventory (via hierarchical summaries), our strategy directives (human-set), and graph relationships to produce a prioritized, scheduled content plan.

Agent Definition:

How the agent navigates context (the hierarchy in action):

Tools:

Output Schema:

→ Workflow SUSPENDS here. Plan items are written to content_plan_items with source: "ai_generated". Human reviews in the Calendar UI, approves/rejects/edits items, adds their own items. On approval, workflow resumes and approved items are queued for brief generation.

4.4 Content Brief Agent

Purpose: For each approved content_plan_item, generates a detailed content brief that the Writer Agent executes against.

Agent Definition:

Tools:

Output Schema:

→ Workflow SUSPENDS here. Brief written to content_briefs with approved: false. Human reviews in the app, approves or requests changes. On approval, workflow resumes and brief is passed to the Content Layer.

5. Content Layer

5.1 Writer Agent

Purpose: Receives an approved content brief and produces a complete first draft that follows the outline, hits word count targets, incorporates keywords naturally, includes internal and external links, and matches brand voice.

Agent Definition:

Tools:

Output: Full markdown content body with frontmatter, internal links, external links, and [IMAGE: ...] placement markers for human image insertion.

5.2 Editor Agent

Purpose: Reviews the Writer's draft for language correctness, verbal consistency, brand voice adherence, factual grounding, structural quality, link integrity, and keyword optimization. Produces specific edits and a revised draft.

Agent Definition:

Output Schema:

5.3 Content Layer Workflow

Revision loop: If the Editor returns needs_revision with critical edits, the draft goes back to the Writer with the edit list as additional context. Maximum 2 revision cycles. After 2 cycles, the draft proceeds to human review regardless (humans catch what agents miss).

6. Production Layer

6.1 Human Review & Editing

This is the most important step in the entire system. The workflow suspends via .waitForEvent("human-review-complete") and the human reviewer performs all of the following in the app's editor interface:

What the human does:

Why image placement is manual: Image selection requires brand aesthetic judgment, rights verification, and contextual sensitivity that current AI image generation doesn't handle reliably at production quality. The [IMAGE: ...] markers from the Writer Agent serve as placement suggestions — the human decides what actually goes there.

6.2 Final Cleanup Agent

Purpose: A lightweight pass after human edits to ensure formatting consistency, link integrity, and proper markdown structure. Not a creative agent — strictly a technical cleanup.

6.3 SEO Validation → Publishing Flow

After cleanup, the content enters the programmatic SEO validation engine (defined in detail in Section 8). If it scores 10/10, it proceeds to the Publishing Agent. If it fails any check, the Final Cleanup Agent attempts auto-fixes for the specific failures, then revalidates. Maximum 3 fix-revalidate cycles. If still failing, escalate to human with a specific failure report.

Publishing Agent (tool-driven, minimal LLM reasoning):

Post-publish pipeline (critical — this closes the data loop):

7. Content Calendar & Application Interface

7.1 Content Calendar View

The calendar is the primary human control surface. It shows planned, in-progress, and published content on a timeline.

Calendar item data model:

Calendar capabilities:

7.2 Content Editor / Review Interface

7.3 Additional Views

8. Programmatic SEO Validation Engine

This is deterministic code, not an LLM. Every check has binary pass/fail logic. All 10 must pass for publication.

Check 1: Meta Title

Check 2: Meta Description

• Length: 150–160 characters
• Contains primary keyword
• Includes call-to-action or value proposition
• Unique across site

Check 3: Heading Hierarchy

• Exactly one H1 containing primary keyword
• H2s use secondary keywords
• No skipped levels (H1 → H3 without H2)
• Logical nesting throughout

Check 4: Keyword Optimization

• Primary keyword in: title, H1, first 100 words, at least one H2, meta description
• Keyword density: 0.5%–2.5%
• Secondary keywords present naturally
• No keyword stuffing patterns (3+ identical phrases in sequence)

Check 5: Internal Linking

• Minimum 3 internal links
• All resolve to real published pages (verified against our_pages table)
• Anchor text is descriptive (no "click here", "read more")
• Anchor text is diversified (not all exact-match keyword)
• Links are contextually placed (not dumped in a footer list)

Check 6: External Linking

• At least 1 external link to authoritative source
• No links to competitor domains (checked against competitors table blocklist)
• External links are contextually relevant
• Proper rel attributes on new-tab links

Check 7: Content Quality Metrics

• Word count within ±10% of brief target
• Readability score within configured range
• No duplicate content (checked against our_page_extractions for same primary keyword via cannibalizes graph edge — not vector similarity)
• No paragraph exceeds 300 words
• Sentence length variety present

Check 8: Technical SEO

• Valid JSON-LD schema markup present and parseable
• Canonical URL set correctly
• Open Graph tags: og:title, og:description, og:image
• Twitter Card tags present
• All images have alt text
• All images have width/height dimensions

Check 9: URL & Slug

• URL-friendly (lowercase, hyphens, no special characters)
• Contains primary keyword or close variant
• Under 60 characters
• No duplicate slug in our_pages table

Check 10: Mobile & Performance

• All images have explicit width/height (prevents Cumulative Layout Shift)
• Images use loading="lazy" (except above-the-fold hero)
• No inline styles that break mobile viewport
• Tables have responsive handling
• No excessively large embedded content

Validation output:

9. Cost Model

10. Risk Matrix

11. Future: Filesystem-as-Context Architecture

The Problem This Solves

Even with the hierarchical summary approach, there's an architectural ceiling: summaries are pre-generated snapshots. As the content library grows to thousands of pages and the competitive landscape shifts daily, keeping summaries fresh and relevant becomes a continuous compute cost. More fundamentally, pre-computing what context an agent might need is inherently wasteful — you're guessing ahead of time which summaries will matter for which tasks.

The filesystem-as-context pattern, articulated by Andrej Karpathy's "context engineering" framework and demonstrated by Anthropic's Skills system, offers a potentially superior approach: don't pre-load context. Let agents navigate to it on demand.

The Core Idea

Instead of generating hierarchical summaries that agents read passively, you structure all system knowledge as a navigable filesystem. Agents use ls, grep, glob, and file reading to pull exactly the context they need for the current task, building their context window incrementally with only signal, never noise.

How an Agent Would Navigate

When the Content Strategy Agent needs to plan next month's content:

Compare this to the hierarchical summary approach (~14,000 tokens, some of which may be irrelevant to this specific planning task). The filesystem approach lets the agent decide what to load based on the actual task, not pre-generated summaries that try to anticipate what might be needed.

What It Would Take to Implement

This is a significant architectural change but builds cleanly on top of the LangExtract + Graph foundation already specified in this document. The core work:

1. Filesystem generation pipeline. A scheduled job that reads from PostgreSQL (extractions, summaries, graph edges, performance data) and writes structured markdown files to a mounted filesystem. Each file follows a consistent schema: frontmatter with metadata, then structured content. This is the bridge between the database and the agent's navigable context. Estimated effort: 2–3 weeks for the generation logic, templates, and scheduling.

2. Sandbox environment per agent session. Each agent invocation gets a read-only mounted view of the filesystem. Mastra's tool system exposes ls, cat, grep, and glob as tools the agent can call. The agent navigates the filesystem using bash-like commands it already knows from training data. This is simpler than building custom SQL-backed query tools — the filesystem IS the query interface. Estimated effort: 1–2 weeks for the sandbox tooling and Mastra integration.

3. Filesystem-aware agent prompts. Agent instructions are updated to describe the filesystem structure and navigation patterns. Instead of "use the readClusterSummaries tool," the prompt says "the competitor data is in /competitors/. Start by reading the INDEX.md, then drill into specific competitor directories as needed." This leverages the model's existing training on filesystem navigation. Estimated effort: 1 week of prompt engineering and testing.

4. Hybrid SQL + filesystem approach. Not everything moves to the filesystem. High-frequency queries (keyword rankings, performance metrics, real-time GSC data) stay in PostgreSQL with dedicated tools. The filesystem handles the slower-changing strategic context: content inventories, competitive analysis, brand voice, editorial plans. The agent has both filesystem tools and database tools available and chooses the right one for the task. Estimated effort: included in items 1–3 above.

5. Write-back pattern. When agents need to create outputs (content plan items, briefs), they write to specific directories (e.g., /strategy/calendar/drafts/) which a sync job picks up and persists to PostgreSQL. This keeps the database as the authoritative source while giving agents a natural write interface. Estimated effort: 1 week.

Total estimated implementation effort: 5–7 weeks on top of the base system specified in this document.

The tradeoff is clear: The filesystem approach produces tighter, more relevant context windows (3,500 tokens vs. 14,000) because agents load only what they actually need for the current task. The cost is an additional generation pipeline and the operational complexity of keeping the filesystem in sync with the database. For a system operating at scale (500+ pages, 10+ competitors, 50+ pieces per month), the context efficiency gains likely justify the investment. For smaller operations, the hierarchical summary approach specified in the main architecture is sufficient.

The recommended path: build the base system using the hierarchical summary + graph approach first (Sections 2–8 of this spec), validate it works at your current scale, then implement the filesystem layer as a context optimization when agent context quality becomes a bottleneck.

This is a living specification. Update as architectural decisions are made during implementation.