Campfire Core

Campfire Core is a Nuxt 4 application and server API for a personal reading dashboard. It stores ingested items in PostgreSQL with pgvector, ranks them into a three-part dashboard, and uses local or hosted language models for embeddings and news-event processing.

Project outline

• pages/ contains the dashboard, people, setup, recently-read, and collection views. Collections include owned and promoted evergreen collections, and model-suggested clusters.
• components/ contains the dashboard item, connection, modal, navigation, login, message, HTML-preview, and loading UI.
• server/api/ contains the authenticated browser API, setup endpoints, and internal maintenance endpoints for embedding, item expiration, events, and news.
• server/domain/ contains ranking, story matching, affinity, link, and news-evidence logic that is mostly covered by tests/.
• server/utils/ contains database access, content extraction, embeddings, news jobs, collection ordering, identity handling, and reinforcement helpers.
• server/db/ contains the Drizzle schema and checked-in PostgreSQL migrations.

The production container builds the Nuxt server, runs pending migrations at startup, and listens on port 3000.

Development

Testing commands, database setup, browser tests, and the functionality coverage map are documented in tests/README.md.

npm ci
npm run dev

The application expects PostgreSQL with the vector extension (for example, the pgvector/pgvector image). The default connection is postgres://n8n:changeme@postgres:5432/n8n; set NUXT_PG_CONNECTION_STRING for another database.

Progressive web app

Production builds include an installable PWA with standalone display, app icons, and a service worker. Serve Campfire over HTTPS (or localhost for testing), then use the browser's install option; on iOS, use Safari's Add to Home Screen. The service worker caches static assets and shows a reconnect page when offline. App pages and API responses are fetched from the network without being cached.

PWA registration is disabled during npm run dev. To verify it locally, run npm run build followed by npm start.

Runtime configuration

All settings are read from Nuxt runtime configuration. The important variables are:

Variable	Purpose	Default
NUXT_PG_CONNECTION_STRING	PostgreSQL connection string	postgres://n8n:changeme@postgres:5432/n8n
NUXT_OPENWEBUI_URL	Ollama-compatible Open WebUI endpoint	http://openwebui:8080
NUXT_OPENWEBUI_API_KEY	Optional Open WebUI bearer token	empty
NUXT_OPENROUTER_URL	OpenRouter API endpoint	https://openrouter.ai/api/v1
NUXT_OPENROUTER_API_KEY	Optional hosted news-model token	empty
NUXT_EMBED_MODEL	Embedding model	qwen3-embedding:0.6b
NUXT_EMBED_DIMENSIONS	pgvector embedding dimensions	768
NUXT_EMBED_BATCH_SIZE	Embeddings per request	32
NUXT_EMBED_MAX_CHARACTERS	Maximum embedding chunk size	8000
NUXT_EMBED_NUM_CTX	Ollama embedding context size	32768
NUXT_NEWS_MODEL	News classification model	typesafe/jev-1.13
NUXT_NEWS_SUMMARY_MODEL	News summary model	gemma4:e4b
NUXT_NEWS_BATCH_SIZE	Items processed per news batch	20
NUXT_NEWS_MAX_CANDIDATES	Candidate events per item	12
NUXT_NEWS_MAX_CHARACTERS	News evidence text limit	6000
NUXT_NEWS_CONVERSATION_CONFIDENCE	Conversation-event confidence threshold	0.9
NUXT_NEWS_MIN_MEMBERS	Minimum event members	2
NUXT_SETUP_TOKEN	Optional token protecting first-user setup	empty
NUXT_N8N_ENGAGEMENT_WEBHOOK_URL	Optional engagement/collection webhook	empty
NUXT_READING_TIME_WPM	Words per minute for article estimates	200
NUXT_METUBE_URL	MeTube base URL for durable metadata jobs	empty
NUXT_METUBE_HOSTS	Hosts handled by MeTube	youtube.com,youtu.be
NUXT_METUBE_YTDL_OPTIONS_PRESET	Optional MeTube preset enabling info JSON	empty
NUXT_METUBE_POLL_INTERVAL_MS	MeTube history polling interval	10000

Embedding operations use Open WebUI. News classification can use OpenRouter when its API key is configured and otherwise uses the Open WebUI Ollama endpoint.

Items with no explicit consumeSeconds get an article estimate when they first appear in the dashboard. If NUXT_METUBE_URL is set, matching video links are instead handled by a durable job: Campfire queues a metadata-only yt-dlp job (skip_download plus writeinfojson), polls /history, downloads the generated .info.json, stores its duration, and clears the completed MeTube job. The job resumes after either service restarts and re-queues a URL if MeTube lost its history. MeTube must allow ytdl_options_overrides, or have a preset named by NUXT_METUBE_YTDL_OPTIONS_PRESET that enables skip_download, writeinfojson, extract_flat=false, and no YouTube player_skip values. Set MeTube's DELETE_FILE_ON_TRASHCAN=true so its /delete call removes the downloaded media after the metadata has been read.

Run the checked-in migrations with:

npm run db:migrate

API and maintenance jobs

Agent access via MCP

Campfire serves a read-only MCP server at https://<your-production-host>/api/mcp using Streamable HTTP. Deploy the updated container to make the endpoint available on the production instance. It uses the same PostgreSQL database and API-key authentication as the application; no separate process or database credentials are needed for the agent.

Configure an MCP client with that URL and either an X-API-Key header or Authorization: Bearer <your-campfire-api-key>. Use the API key issued during Campfire account setup. Every request authenticates the key and scopes queries to its user; revoked keys are rejected.

For example, in a client that supports remote MCP URLs and headers:

{
  "mcpServers": {
    "campfire": {
      "url": "https://<your-production-host>/api/mcp",
      "headers": {
        "Authorization": "Bearer <your-campfire-api-key>"
      }
    }
  }
}

The exact configuration wrapper depends on the client. Supply the key through the client's secret/environment mechanism where available.

Available tools:

Tool	Data
search_items	Text search and filters for source, author, conversation, collection, event, medium, disposition, read state and ingestion dates
get_item	Item metadata and stored description, including reply/conversation references
list_sources	Ingestion source IDs, names and source keys
list_collections	Owned evergreen collections, optionally searched by name/description
list_events	News events and summaries, optionally filtered by status/text
search_people	People and source-scoped author identities

Lists accept limit (default 20, maximum 100) and offset (default 0, maximum 10,000), and return results and nextOffset (null at the end). Search is literal, case-insensitive substring matching. Item search includes historical read/expired items and previews descriptions up to 2,000 characters. get_item returns up to 50,000 characters with a descriptionTruncated flag. Descriptions are stored text/HTML, not remotely fetched article bodies. Collection filters include pending recommendations and added items. Date filters use ISO 8601 timestamps, including a timezone.

Tools query stored data without marking items read, fetching external pages, running maintenance jobs, or modifying collections. API-key usage timestamps are updated by the existing authentication middleware. The transport is stateless with JSON responses, so it works across replicas and restarts; clients should use POST requests, with no persistent SSE connection or session ID. The reverse proxy must forward /api/mcp and the authentication headers. Browser requests with an Origin header must have the same origin as the Campfire endpoint.

Check the deployed endpoint with:

curl 'https://<your-production-host>/api/mcp' \
  -H "Authorization: Bearer $CAMPFIRE_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  --data '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"check","version":"1"}}}'

Browser API

Browser-facing routes are under /api/v1: the dashboard feed (fire), collections and recommendations, item content and connections, people, unread state, recently-read items, evergreen saves, and engagement. /api/auth holds the setup and authentication checks. /api/v1/internal contains authenticated automation and maintenance endpoints and should not be exposed publicly.

The news endpoints are designed for scheduled n8n calls:

• POST /api/v1/internal/news/process starts classification and returns a jobId; poll its status endpoint.
• POST /api/v1/internal/news/reconcile merges or splits event records; poll its status endpoint.
• POST /api/v1/internal/news/expire removes expired events.
• GET /api/v1/internal/news/visibility diagnoses event visibility gates.

Embedding backfills, item expiration, event insertion/regeneration, Semble confirmations, and unsorted-content backfills are also available under the internal namespace. Engagement synchronization reconciles self-authored items and updates author affinity models. Daily item maintenance expires old dashboard items. Collection recommendation maintenance refreshes suggested clusters and cached collection links. These jobs keep maintenance out of client-facing requests. Inspect server/api/v1/internal/ for the exact request shapes.

Discourse ingestion

Discourse ingestion runs in n8n's Poll Discourse workflow. Polls run every 30 minutes, with a 25-minute execution timeout and requests spaced 30 seconds apart to avoid overlapping runs and forum rate limits.

The poller compares exact stored Discourse post IDs against each topic's post stream. For large topics, fetched replies are kept in a separate new_posts field so merging topic metadata cannot replace them with the first page of posts. Already-stored posts are filtered before calling Ingest campfire item, which runs once per post to preserve URL and conversation associations.

When diagnosing stale Discourse items, check for HTTP 429 errors and verify that the ingestion output contains newly inserted items, rather than only updates to old posts. A successful execution alone does not establish freshness.

Zulip ingestion

Zulip ingestion runs in n8n's Poll Zulip workflow. Keep the Zulip message id separate from the Campfire source UUID: Get Source ID must return id AS source_id, and Call 'ingest item' must map its source_id input from that field. Deduplication keys use the original message ID in the form zulip:<realm>:<message-id>. Returning the source UUID as id overwrites the message ID during Merge Source ID, causing different messages to collide with an existing item instead of appearing as new unread chats.

Run Ingest campfire item once per message (mode: each) to preserve each message's URL and conversation associations. When diagnosing missing chats, inspect the merged message ID and ingestion deduplication key, then verify that newly imported unread items exist; successful polling alone is insufficient.

Verification

npm run lint
npm test
npm run typecheck
npm run build
npx drizzle-kit check

Container

The Dockerfile builds a production Node server, runs database migrations on startup, and exposes port 3000. Use a persistent local checkout as the Compose build context so updates fetch Git changes incrementally rather than resolving a remote Git context on every build.

From the directory containing your docker-compose.yml (for example, ~/docker/openwebui), clone the repository once:

git clone --depth 1 https://code.incremental.social/thepaperpilot/campfire-core.git campfire-core

Set the existing Campfire service's build to ./campfire-core, keeping your ports, environment, networks, and other service settings. For example:

services:
  campfire:
    build: ./campfire-core
    ports:
      - '3000:3000'
    environment:
      NUXT_PG_CONNECTION_STRING: postgres://n8n:changeme@postgres:5432/n8n
      NUXT_OPENWEBUI_URL: http://openwebui:8080
      NUXT_OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:-}
      NUXT_EMBED_MODEL: qwen3-embedding:0.6b
      NUXT_EMBED_DIMENSIONS: '768'
      NUXT_EMBED_BATCH_SIZE: '32'
      NUXT_EMBED_MAX_CHARACTERS: '8000'
      NUXT_EMBED_NUM_CTX: '32768'
      NUXT_NEWS_MODEL: typesafe/jev-1.13
      NUXT_NEWS_SUMMARY_MODEL: gemma4:e4b
      NUXT_NEWS_CONVERSATION_CONFIDENCE: '0.9'
      NUXT_NEWS_MIN_MEMBERS: '2'
      NUXT_SETUP_TOKEN: ${CAMPFIRE_SETUP_TOKEN:-}

Provide the runtime variables through Compose. The container waits for the database connection and runs pending migrations before serving traffic.

Build and start the service:

docker compose up -d --build campfire

For subsequent updates, run from the same Compose directory:

git -C campfire-core pull --ff-only
docker compose up -d --build campfire

The local checkout must be updated before building; Compose does not pull Git changes for a local context. Dependency layers are cached separately from application source, and npm downloads use a BuildKit cache. Source-only updates therefore rebuild the application without reinstalling production dependencies.

If a build spends most of its time at load git source, that delay occurs before the Dockerfile runs; changing Dockerfile layers or .dockerignore cannot fix that remote fetch. Switching the service to the local context above removes that step. The obsolete Compose version warning is unrelated; remove the top-level version key from your Compose file to clear it.

Evergreen ingestion

When inserting items from an ingestion workflow, set items.disposition to 'evergreen' to force durable material to stay evergreen during news processing. Set items.evergreen_at to the save/ingestion timestamp, or leave it null to have news processing fill it in. Existing evergreen timestamps are preserved.

For example, include these fields in the ingestion SQL insert:

INSERT INTO items (user_id, source_id, dedupe_key, title, medium, disposition, evergreen_at)
VALUES ($1, $2, $3, $4, 'article', 'evergreen', now());

Jev still recommends collections for these items, but its classification cannot turn them into news or attach them to an event. Items without a matching collection appear in recent evergreen items on the dashboard. The existing restriction on alert and conversation mediums still applies: those mediums are always time-sensitive.

Automated News Events

The n8n event workflow should periodically call the authenticated POST /api/v1/internal/news/process endpoint. Campfire scans recent unassigned items, asks the configured Ollama model whether each belongs to an active event, and creates or updates the event automatically. Processing runs as a background job, so the POST responds immediately with a jobId:

POST /api/v1/internal/news/process        (empty body)
GET  /api/v1/internal/news/process/status?jobId=...

Poll the status endpoint until status is completed (the full result is in result) or failed (details in error). Pass {"lookbackMinutes": 4320} in the body only when recovering a backlog; the default 30-minute window is right for steady state. The lower-level POST /api/v1/internal/events endpoint is also available when n8n has already performed classification.

{
  "title": "Example event",
  "summary": "What happened and why it matters.",
  "members": [
    {
      "itemId": "00000000-0000-0000-0000-000000000000",
      "confidence": 0.94,
      "relationship": "report"
    }
  ],
  "model": "llama3.2:3b",
  "modelVersion": "news-v1"
}

Events appear as priority items in Hearth. Their members are read individually, including messages selected from otherwise unrelated source conversations.

Run reconciliation less frequently, such as every 3-4 hours:

POST /api/v1/internal/news/reconcile        (empty body)
GET  /api/v1/internal/news/reconcile/status?jobId=...

Poll the status endpoint for the result. It uses event-vector similarity followed by Ollama verification before merging duplicate event records, splitting events that contain distinct topics, and regenerating the resulting summaries.

Run expiration cleanup daily:

POST /api/v1/internal/news/expire

If the dashboard shows no events despite related articles, score every active event against each visibility gate with:

GET /api/v1/internal/news/visibility

It reports how many events pass recency, expiry, member-count, confidence, and unread filters, plus per-event details for the largest events.

It marks past-due events expired and removes expired records after the default 30-day retention period.

Backfill title and content for unsorted and recommended URL items with:

POST /api/v1/internal/unsorted/backfill

The endpoint is safe to rerun; items with existing content and items with medium conversation or status are skipped. It processes pending and failed entries in both unsorted and collection recommendations. Migration 0042_reset_webpage_content clears existing content for eligible URL items once so the next scheduled backfill re-extracts them as Markdown.

Re-evaluate existing unsorted items against all writable collections with:

POST /api/v1/internal/collections/recommend

This runs as a background job and can create recommendations for multiple collections from one item. Poll the returned job ID with the news process status endpoint. Items are only re-evaluated when they have not been recommended yet or when a writable collection has changed, so this endpoint can safely run from a regular cron job. The same run refreshes suggested clusters and cached Semble collection links.

Reconcile self-authored activity and synchronize engagement-derived models with:

POST /api/v1/internal/engagement/sync

Run this endpoint in the 15-minute maintenance sequence so new engagement affects ranking promptly.

Expire old dashboard items daily with:

POST /api/v1/internal/items/expire

Client-facing routes only record the immediate user action and do not perform this maintenance.

Interest-profile learning is retired. Existing profiles are retained as archived data, but no longer derived, reinforced, decayed or exposed through the API/MCP. The old sync-interests and embed-interests maintenance URLs are compatibility handlers for previously published workflows. Discovery now comes from suggested collections; see collection discovery.