Discovery

Find articles worth writing about - source-first crawl plus optional search-first URL discovery.

Vertical: Digital · Status: ✅ Live Back to Digital · workflows index

---

Trigger

Daily morning cron sweep, plus manual fetch per tenant.

Flow

source crawl (configured sources)
   + search-first discovery (LLM-built queries -> Tavily)
        |
isArticleShapedUrl gate     <- rejects landing/hub pages
        |
extraction cascade: Readability -> Tavily -> Crawl4AI/Apify
        |
record scraperSource per row
        |
land as enrichmentStatus: pending

The URL gate

isArticleShapedUrl rejects landing and product pages (/savings-account, /blogs) before they ever reach the feed - only date-stamped, id-bearing or 4+-word-slug URLs pass.

Every row also records scraperSource (local | tavily | crawl4ai | apify | rss | html-fallback), so the extractor mix is queryable rather than guessed at.

Agents involved

Query generation is a small OpenAI call, capped at 700 output tokens and pinned to OpenAI - it must not fall through the script provider chain.

Gaps

Query generation does not learn from what got queued. Falls back to deterministic queries at the configured count when the LLM fails.

Roadmap source

Page 3 - Platform Discovery, Competition Research; page 1 Outside world DATA — see roadmap_ref_extracted.md.