Finlytic News Service

Finlytic News is a scalable C# microservice designed to periodically scrape, deduplicate, pre-filter, and process financial articles before distributing them to downstream consumers via MQTT. It leverages a dynamic adapter-based link discovery system, headless browser automation (Playwright/HtmlAgilityPack) to bypass dynamic site scrapers, integrates with n8n workflows for AI-driven classification, and maintains a resilient state machine database cache.

Architecture & Data Flow

graph TD
    Cron[Cron Trigger: 15m + Jitter] -->|Start Discovery| ADS[ArticleDiscoveryService]
    ADS -->|Fetch Content| HTML[RSS Feed / HTML Page]

    ADS -->|Parse XML| RSS[RssDiscoveryAdapter]
    ADS -->|Parse HTML| Agility[Custom Host Adapters e.g. HtmlAgilityPack]

    RSS -->|Extract Links| Links[Discovered URLs]
    Agility -->|Extract Links| Links

    Links -->|Redirect Resolve & Playwright| Body[Article Content]
    Body -->|Idempotency Check| DB{URL/Hash in PostgreSQL?}
    DB -->|Exists: Skip| Skip[End Process]
    DB -->|New: Load Index| IndexFilter[Asset Index Pre-Filter]

    AssetIndex[assets/index/index.json] -->|ISIN & Name List| IndexFilter
    IndexFilter -->|Matched Assets| Webhook[n8n AI Webhook Pipeline]

    Webhook -->|AI Extraction & Score| DBWrite[Save Article & Assets]
    DBWrite -->|State = Completed| Broadcast[MQTT Live Broadcast]

    Downstream[Downstream Services] <-->|MQTT RPC / Recovery| MqttRpc[MQTT RPC Server]

Core Features & Capabilities

1. Dynamic Article Discovery & Scraping Architecture

Adapter Separation: Discovery Adapters (RssDiscoveryAdapter) & Detail Scrapers (FinanznachrichtenScraperAdapter).
Stealth Timing: 15-minute cron with randomized jitter offsets.
Headless Scraping: Playwright & HtmlAgilityPack node parsing.
Redirect Resolution: Automatic URL expansion for canonical links.

2. Data Deduplication & Asset Pre-Filtering

Database Idempotency: Source URL uniqueness index.
Asset Index Pre-Filtering: Matches raw text against assets/index/index.json.
n8n AI Webhook Integration: Enrichment & classification pipeline.

3. Database State Management & Performance Indexes

Lifecycle States: Pending -> Processing -> Completed / Failed.
PostgreSQL Indizes: Optimized compound index on (PublishedAt, ScrapedAt), Status, SourceUrl, Isin.

4. Event-Driven Messaging (MQTT)

Zero-Allocation Broadcast: Publishes completed articles to finlytic/news/new and finlytic/news/classification.
MQTT RPC Handler: Serves historical article queries for gateway endpoints.

Feature Status

Implemented Features

RSS & HTML Adapter discovery architecture (ArticleSourceEntity).
Playwright headless browser detail scraping.
n8n AI webhook integration.
Database compound indexes on (PublishedAt, ScrapedAt), Status, SourceUrl.
MQTT broadcast & RPC response handling.
Zero-Allocation serialization via FinlyticJsonSerializerContext.

Planned / Future Features

Multi-language translation pipeline (auto-translate foreign language news articles to DE/EN).
Direct RSS Feed WebSub (PubSubHubbub) real-time push ingestion for instant zero-latency discovery.

3.2 KiB Raw Blame History