Init

2026-07-19 12:13:14 +02:00
commit 6337e63a77
46 changed files with 4411 additions and 0 deletions
@@ -0,0 +1,66 @@
+# Finlytic Assets Service
+
+Finlytic Assets is a scalable C# microservice designed for asset discovery, metadata ingestion, and quick querying. It interfaces directly with the Trade Republic API via WebSockets and exposes a high-performance RPC interface over MQTT to other services in the Finlytic ecosystem.
+
+---
+
+## Architecture Overview
+
+```mermaid
+graph TD
+    TR[Trade Republic WebSocket API] <-->|WS Protocol| TRS[TradeRepublicService]
+    TRS <-->|Ingest| ADS[AssetsFullScanService]
+    ADS <-->|Save / Update| DB[PostgreSQL Database]
+    DB -->|Trigger Index Update| AIS[AssetsIndexService]
+    AIS -->|Write Cache| Index[assets/index/index.json]
+    
+    MS[Other Finlytic Services] <-->|MQTT Request-Reply| MqttClient[AssetsMqttClient]
+    MqttClient <-->|Query Cache| DB
+```
+
+---
+
+## Core Features & Workflows
+
+### 1. Automated WebSocket scraping
+- **Continuous Scan**: The background worker (`AssetsFullScanService`) iterates through all support asset types:
+  - **Stocks** (`StockEntity`)
+  - **Crypto** (`CryptoEntity`)
+  - **Derivatives** (`DerivativeEntity`)
+  - **Bonds** (`BondEntity`)
+  - **Funds/ETFs** (`EtfEntity`)
+- **Resilient Reconnection**: Uses a robust custom socket wrapper (`TradeRepublicClient`) that closes automatically after 5 minutes of inactivity to mimic human interaction profiles and reconnects dynamically when a request is made.
+
+### 2. Stealth Scheduling with Randomized Jitter
+- **Dynamic Delays**: Scanner wait times are read dynamically from database configuration (`Settings`).
+- **Jitter Offset**: High/low jitter values (randomized offsets between batches and asset types) are injected to avoid predictable traffic patterns and prevent Trade Republic rate limiting.
+
+### 3. Active vs. Dead Asset Tracking
+- **Recency Filter**: Assets are continuously updated with their latest metadata. If an asset has not been updated within **14 days**, it is considered inactive, de-listed, or "dead".
+- **Implicit Filtration**: Queries by other services only return active/valid records unless explicitly requested otherwise.
+
+### 4. MQTT RPC Interface
+- **Get Asset by ISIN**:
+  - Subscribes to: `services/request/assets_Get/#`
+  - Returns a list of all active assets matching the specified ISIN across different instrument types (e.g., matching both a stock and its derivative tracker).
+- **Omnibox Search**:
+  - Subscribes to: `services/request/assets_Search/#`
+  - Evaluates search keywords against local records (matching `ISIN`, `Name`, or tags like `Region`, `Country`, `Sector`, etc.).
+  - **JIT Fallback**: If a search query is formatted as an unknown valid ISIN, it performs a JIT-lookup directly against the Trade Republic API and registers the discovered asset before replying.
+
+### 5. Local Indexing File (`index.json`)
+- **Automatic Regeneration**: On database modifications, `AssetsIndexService` is triggered to regenerate `assets/index/index.json`.
+- **Pre-Filtering**: This lean index (containing only `ISIN` and `Name` properties) is written to a shared volume to allow other services to perform instant pre-filtering without querying the database or sending network requests.
+
+---
+
+## Technical Components
+
+| Component | Class / Interface | Responsibility |
+| :--- | :--- | :--- |
+| **Ingestion Worker** | `AssetsFullScanService` | Runs the main cron loop, scraping and paginating all Trade Republic asset types. |
+| **Database Store** | [IAssetsDbService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/AssetsDbService.cs) | Handles CRUD operations, stateful merges, JIT lookups, and active record filtering. |
+| **WebSocket Handler** | [ITradeRepublicService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/TradeRepublicService.cs) | Wraps the connection and request dispatching for the Trade Republic API. |
+| **Local Indexer** | [IAssetsIndexService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/AssetsIndexService.cs) | Serializes the active assets list to `index.json` for external service indexing. |
+| **Messaging Broker** | `AssetsMqttClient` | Listens for incoming MQTT requests, queries the database, and responds to RPC channels. |
+| **Config Service** | [ISettingsDbService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/SettingsDbService.cs) | Manages scanner configurations (scan intervals, active pointers, offsets). |