# Finlytic News Service

Finlytic News is a scalable C# microservice designed to periodically scrape, deduplicate, pre-filter, and process financial articles before distributing them to downstream consumers via MQTT. It leverages a dynamic adapter-based link discovery system, headless browser automation (Playwright/HtmlAgilityPack) to bypass dynamic site scrapers, integrates with n8n workflows for AI-driven classification, and maintains a resilient state machine database cache.

---

## Architecture & Data Flow

```mermaid
graph TD
    Cron[Cron Trigger: 15m + Jitter] -->|Start Discovery| ADS[ArticleDiscoveryService]
    ADS -->|Fetch Content| HTML[RSS Feed / HTML Page]

    ADS -->|Parse XML| RSS[RssDiscoveryAdapter]
    ADS -->|Parse HTML| Agility[Custom Host Adapters e.g. HtmlAgilityPack]

    RSS -->|Extract Links| Links[Discovered URLs]
    Agility -->|Extract Links| Links

    Links -->|Redirect Resolve & Playwright| Body[Article Content]
    Body -->|Idempotency Check| DB{URL/Hash in PostgreSQL?}
    DB -->|Exists: Skip| Skip[End Process]
    DB -->|New: Load Index| IndexFilter[Asset Index Pre-Filter]

    AssetIndex[assets/index/index.json] -->|ISIN & Name List| IndexFilter
    IndexFilter -->|Matched Assets| Webhook[n8n AI Webhook Pipeline]

    Webhook -->|AI Extraction & Score| DBWrite[Save Article & Assets]
    DBWrite -->|State = Completed| Broadcast[MQTT Live Broadcast]

    Downstream[Downstream Services] <-->|MQTT RPC / Recovery| MqttRpc[MQTT RPC Server]
```

---

## Core Features & Capabilities

### 1. Dynamic Article Discovery & Scraping Architecture
- **Adapter Separation**: Discovery Adapters (`RssDiscoveryAdapter`) & Detail Scrapers (`FinanznachrichtenScraperAdapter`).
- **Stealth Timing**: 15-minute cron with randomized jitter offsets.
- **Headless Scraping**: Playwright & HtmlAgilityPack node parsing.
- **Redirect Resolution**: Automatic URL expansion for canonical links.

### 2. Data Deduplication & Asset Pre-Filtering
- **Database Idempotency**: Source URL uniqueness index.
- **Asset Index Pre-Filtering**: Matches raw text against `assets/index/index.json`.
- **n8n AI Webhook Integration**: Enrichment & classification pipeline.

### 3. Database State Management & Performance Indexes
- **Lifecycle States**: `Pending` -> `Processing` -> `Completed` / `Failed`.
- **PostgreSQL Indizes**: Optimized compound index on `(PublishedAt, ScrapedAt)`, `Status`, `SourceUrl`, `Isin`.

### 4. Event-Driven Messaging (MQTT)
- **Zero-Allocation Broadcast**: Publishes completed articles to `finlytic/news/new` and `finlytic/news/classification`.
- **MQTT RPC Handler**: Serves historical article queries for gateway endpoints.

---

## Feature Status

### Implemented Features
- [x] RSS & HTML Adapter discovery architecture (`ArticleSourceEntity`).
- [x] Playwright headless browser detail scraping.
- [x] n8n AI webhook integration.
- [x] Database compound indexes on `(PublishedAt, ScrapedAt)`, `Status`, `SourceUrl`.
- [x] MQTT broadcast & RPC response handling.
- [x] Zero-Allocation serialization via `FinlyticJsonSerializerContext`.

### Planned / Future Features
- [ ] Multi-language translation pipeline (auto-translate foreign language news articles to DE/EN).
- [ ] Direct RSS Feed WebSub (PubSubHubbub) real-time push ingestion for instant zero-latency discovery.