feat(News): refactor news service and add sentiment tracking

2026-08-09 21:01:41 +02:00
parent 605e41cfeb
commit aff831cf58
390 changed files with 115094 additions and 9 deletions
@@ -0,0 +1,71 @@
+# Finlytic News Service
+
+Finlytic News is a scalable C# microservice designed to periodically scrape, deduplicate, pre-filter, and process financial articles before distributing them to downstream consumers via MQTT. It leverages a dynamic adapter-based link discovery system, headless browser automation (Playwright/HtmlAgilityPack) to bypass dynamic site scrapers, integrates with n8n workflows for AI-driven classification, and maintains a resilient state machine database cache.
+
+---
+
+## Architecture & Data Flow
+
+```mermaid
+graph TD
+    Cron[Cron Trigger: 15m + Jitter] -->|Start Discovery| ADS[ArticleDiscoveryService]
+    ADS -->|Fetch Content| HTML[RSS Feed / HTML Page]
+
+    ADS -->|Parse XML| RSS[RssDiscoveryAdapter]
+    ADS -->|Parse HTML| Agility[Custom Host Adapters e.g. HtmlAgilityPack]
+
+    RSS -->|Extract Links| Links[Discovered URLs]
+    Agility -->|Extract Links| Links
+
+    Links -->|Redirect Resolve & Playwright| Body[Article Content]
+    Body -->|Idempotency Check| DB{URL/Hash in PostgreSQL?}
+    DB -->|Exists: Skip| Skip[End Process]
+    DB -->|New: Load Index| IndexFilter[Asset Index Pre-Filter]
+
+    AssetIndex[assets/index/index.json] -->|ISIN & Name List| IndexFilter
+    IndexFilter -->|Matched Assets| Webhook[n8n AI Webhook Pipeline]
+
+    Webhook -->|AI Extraction & Score| DBWrite[Save Article & Assets]
+    DBWrite -->|State = Completed| Broadcast[MQTT Live Broadcast]
+
+    Downstream[Downstream Services] <-->|MQTT RPC / Recovery| MqttRpc[MQTT RPC Server]
+```
+
+---
+
+## Core Features & Capabilities
+
+### 1. Dynamic Article Discovery & Scraping Architecture
+- **Adapter Separation**: Discovery Adapters (`RssDiscoveryAdapter`) & Detail Scrapers (`FinanznachrichtenScraperAdapter`).
+- **Stealth Timing**: 15-minute cron with randomized jitter offsets.
+- **Headless Scraping**: Playwright & HtmlAgilityPack node parsing.
+- **Redirect Resolution**: Automatic URL expansion for canonical links.
+
+### 2. Data Deduplication & Asset Pre-Filtering
+- **Database Idempotency**: Source URL uniqueness index.
+- **Asset Index Pre-Filtering**: Matches raw text against `assets/index/index.json`.
+- **n8n AI Webhook Integration**: Enrichment & classification pipeline.
+
+### 3. Database State Management & Performance Indexes
+- **Lifecycle States**: `Pending` -> `Processing` -> `Completed` / `Failed`.
+- **PostgreSQL Indizes**: Optimized compound index on `(PublishedAt, ScrapedAt)`, `Status`, `SourceUrl`, `Isin`.
+
+### 4. Event-Driven Messaging (MQTT)
+- **Zero-Allocation Broadcast**: Publishes completed articles to `finlytic/news/new` and `finlytic/news/classification`.
+- **MQTT RPC Handler**: Serves historical article queries for gateway endpoints.
+
+---
+
+## Feature Status
+
+### Implemented Features
+- [x] RSS & HTML Adapter discovery architecture (`ArticleSourceEntity`).
+- [x] Playwright headless browser detail scraping.
+- [x] n8n AI webhook integration.
+- [x] Database compound indexes on `(PublishedAt, ScrapedAt)`, `Status`, `SourceUrl`.
+- [x] MQTT broadcast & RPC response handling.
+- [x] Zero-Allocation serialization via `FinlyticJsonSerializerContext`.
+
+### Planned / Future Features
+- [ ] Multi-language translation pipeline (auto-translate foreign language news articles to DE/EN).
+- [ ] Direct RSS Feed WebSub (PubSubHubbub) real-time push ingestion for instant zero-latency discovery.