feat(Assets): update asset services and background workers

2026-08-09 21:01:39 +02:00
parent b57cc9894c
commit e0778b88ea
32 changed files with 1020 additions and 2062 deletions
@@ -0,0 +1,67 @@
+# Finlytic Assets Service
+
+Finlytic Assets is a scalable C# microservice designed for asset discovery, metadata ingestion, and quick querying. It interfaces directly with the Trade Republic API via WebSockets and exposes a high-performance RPC interface over MQTT to other services in the Finlytic ecosystem.
+
+---
+
+## Architecture Overview
+
+```mermaid
+graph TD
+    TR[Trade Republic WebSocket API] <-->|WS Protocol| TRS[TradeRepublicService]
+    TRS <-->|Ingest| ADS[AssetsFullScanService]
+    ADS <-->|Save / Update| DB[PostgreSQL Database]
+    DB -->|Trigger Index Update| AIS[AssetsIndexService]
+    AIS -->|Write Cache| Index[assets/index/index.json]
+
+    MS[Other Finlytic Services] <-->|MQTT Request-Reply| MqttClient[AssetsMqttClient]
+    MqttClient <-->|Query Cache| DB
+```
+
+---
+
+## Core Features & Workflows
+
+### 1. Automated WebSocket scraping
+- **Continuous Scan**: The background worker (`AssetsFullScanService`) iterates through all supported asset types:
+  - **Stocks** (`StockEntity`)
+  - **Crypto** (`CryptoEntity`)
+  - **Derivatives** (`DerivativeEntity`)
+  - **Bonds** (`BondEntity`)
+  - **Funds/ETFs** (`EtfEntity`)
+- **Resilient Reconnection**: Uses a robust custom socket wrapper (`TradeRepublicClient`) that closes automatically after 5 minutes of inactivity to mimic human interaction profiles and reconnects dynamically when a request is made.
+
+### 2. Scanner State Persistence & Crash Recovery
+- **Resilient State**: The scan loop continuously persists progress to the database (`Settings` table), recording `CurrentScanningType` and `CurrentScanningPage`.
+- **Exit Recovery**: Resumes exactly from the last saved page and asset type upon restart.
+
+### 3. Stealth Scheduling with Randomized Jitter
+- **Dynamic Delays**: Scanner wait times are read dynamically from database configuration (`Settings`).
+- **Jitter Offset**: Randomized offsets between batches and asset types to prevent rate limiting.
+
+### 4. Active vs. Dead Asset Tracking
+- **Recency Filter**: Assets updated within 14 days are active. Inactive assets are flagged.
+
+### 5. MQTT RPC Interface
+- **Get Asset by ISIN**: Subscribes to `services/request/assets_Get/#`.
+- **Omnibox Search**: Subscribes to `services/request/assets_Search/#`.
+- **JIT Fallback**: Performs JIT-lookup directly against the Trade Republic API for unknown ISINs.
+
+### 6. Local Indexing File (`index.json`)
+- **Automatic Regeneration**: Serializes active assets list (`ISIN`, `Name`) to shared volume `assets/index/index.json`.
+
+---
+
+## Feature Status
+
+### Implemented Features
+- [x] Trade Republic WebSocket scanner for Stocks, Crypto, Derivatives, Bonds, ETFs.
+- [x] State persistence & crash recovery in PostgreSQL.
+- [x] JIT Trade Republic lookup fallback for new ISINs.
+- [x] Shared local index file generation (`index.json`).
+- [x] Zero-Allocation MQTT RPC handlers.
+- [x] Pure Worker Service architecture (No Kestrel HTTP server).
+
+### Planned Features
+- [ ] Real-time WebSocket price tick streaming over MQTT topic `finlytic/assets/ticks/{isin}`.
+- [ ] Derivative option-chain Greeks calculator integration.