72 lines
3.2 KiB
Markdown
72 lines
3.2 KiB
Markdown
# Finlytic News Service
|
|
|
|
Finlytic News is a scalable C# microservice designed to periodically scrape, deduplicate, pre-filter, and process financial articles before distributing them to downstream consumers via MQTT. It leverages a dynamic adapter-based link discovery system, headless browser automation (Playwright/HtmlAgilityPack) to bypass dynamic site scrapers, integrates with n8n workflows for AI-driven classification, and maintains a resilient state machine database cache.
|
|
|
|
---
|
|
|
|
## Architecture & Data Flow
|
|
|
|
```mermaid
|
|
graph TD
|
|
Cron[Cron Trigger: 15m + Jitter] -->|Start Discovery| ADS[ArticleDiscoveryService]
|
|
ADS -->|Fetch Content| HTML[RSS Feed / HTML Page]
|
|
|
|
ADS -->|Parse XML| RSS[RssDiscoveryAdapter]
|
|
ADS -->|Parse HTML| Agility[Custom Host Adapters e.g. HtmlAgilityPack]
|
|
|
|
RSS -->|Extract Links| Links[Discovered URLs]
|
|
Agility -->|Extract Links| Links
|
|
|
|
Links -->|Redirect Resolve & Playwright| Body[Article Content]
|
|
Body -->|Idempotency Check| DB{URL/Hash in PostgreSQL?}
|
|
DB -->|Exists: Skip| Skip[End Process]
|
|
DB -->|New: Load Index| IndexFilter[Asset Index Pre-Filter]
|
|
|
|
AssetIndex[assets/index/index.json] -->|ISIN & Name List| IndexFilter
|
|
IndexFilter -->|Matched Assets| Webhook[n8n AI Webhook Pipeline]
|
|
|
|
Webhook -->|AI Extraction & Score| DBWrite[Save Article & Assets]
|
|
DBWrite -->|State = Completed| Broadcast[MQTT Live Broadcast]
|
|
|
|
Downstream[Downstream Services] <-->|MQTT RPC / Recovery| MqttRpc[MQTT RPC Server]
|
|
```
|
|
|
|
---
|
|
|
|
## Core Features & Capabilities
|
|
|
|
### 1. Dynamic Article Discovery & Scraping Architecture
|
|
- **Adapter Separation**: Discovery Adapters (`RssDiscoveryAdapter`) & Detail Scrapers (`FinanznachrichtenScraperAdapter`).
|
|
- **Stealth Timing**: 15-minute cron with randomized jitter offsets.
|
|
- **Headless Scraping**: Playwright & HtmlAgilityPack node parsing.
|
|
- **Redirect Resolution**: Automatic URL expansion for canonical links.
|
|
|
|
### 2. Data Deduplication & Asset Pre-Filtering
|
|
- **Database Idempotency**: Source URL uniqueness index.
|
|
- **Asset Index Pre-Filtering**: Matches raw text against `assets/index/index.json`.
|
|
- **n8n AI Webhook Integration**: Enrichment & classification pipeline.
|
|
|
|
### 3. Database State Management & Performance Indexes
|
|
- **Lifecycle States**: `Pending` -> `Processing` -> `Completed` / `Failed`.
|
|
- **PostgreSQL Indizes**: Optimized compound index on `(PublishedAt, ScrapedAt)`, `Status`, `SourceUrl`, `Isin`.
|
|
|
|
### 4. Event-Driven Messaging (MQTT)
|
|
- **Zero-Allocation Broadcast**: Publishes completed articles to `finlytic/news/new` and `finlytic/news/classification`.
|
|
- **MQTT RPC Handler**: Serves historical article queries for gateway endpoints.
|
|
|
|
---
|
|
|
|
## Feature Status
|
|
|
|
### Implemented Features
|
|
- [x] RSS & HTML Adapter discovery architecture (`ArticleSourceEntity`).
|
|
- [x] Playwright headless browser detail scraping.
|
|
- [x] n8n AI webhook integration.
|
|
- [x] Database compound indexes on `(PublishedAt, ScrapedAt)`, `Status`, `SourceUrl`.
|
|
- [x] MQTT broadcast & RPC response handling.
|
|
- [x] Zero-Allocation serialization via `FinlyticJsonSerializerContext`.
|
|
|
|
### Planned / Future Features
|
|
- [ ] Multi-language translation pipeline (auto-translate foreign language news articles to DE/EN).
|
|
- [ ] Direct RSS Feed WebSub (PubSubHubbub) real-time push ingestion for instant zero-latency discovery.
|