67 lines
4.2 KiB
Markdown
67 lines
4.2 KiB
Markdown
# Finlytic Assets Service
|
|
|
|
Finlytic Assets is a scalable C# microservice designed for asset discovery, metadata ingestion, and quick querying. It interfaces directly with the Trade Republic API via WebSockets and exposes a high-performance RPC interface over MQTT to other services in the Finlytic ecosystem.
|
|
|
|
---
|
|
|
|
## Architecture Overview
|
|
|
|
```mermaid
|
|
graph TD
|
|
TR[Trade Republic WebSocket API] <-->|WS Protocol| TRS[TradeRepublicService]
|
|
TRS <-->|Ingest| ADS[AssetsFullScanService]
|
|
ADS <-->|Save / Update| DB[PostgreSQL Database]
|
|
DB -->|Trigger Index Update| AIS[AssetsIndexService]
|
|
AIS -->|Write Cache| Index[assets/index/index.json]
|
|
|
|
MS[Other Finlytic Services] <-->|MQTT Request-Reply| MqttClient[AssetsMqttClient]
|
|
MqttClient <-->|Query Cache| DB
|
|
```
|
|
|
|
---
|
|
|
|
## Core Features & Workflows
|
|
|
|
### 1. Automated WebSocket scraping
|
|
- **Continuous Scan**: The background worker (`AssetsFullScanService`) iterates through all support asset types:
|
|
- **Stocks** (`StockEntity`)
|
|
- **Crypto** (`CryptoEntity`)
|
|
- **Derivatives** (`DerivativeEntity`)
|
|
- **Bonds** (`BondEntity`)
|
|
- **Funds/ETFs** (`EtfEntity`)
|
|
- **Resilient Reconnection**: Uses a robust custom socket wrapper (`TradeRepublicClient`) that closes automatically after 5 minutes of inactivity to mimic human interaction profiles and reconnects dynamically when a request is made.
|
|
|
|
### 2. Stealth Scheduling with Randomized Jitter
|
|
- **Dynamic Delays**: Scanner wait times are read dynamically from database configuration (`Settings`).
|
|
- **Jitter Offset**: High/low jitter values (randomized offsets between batches and asset types) are injected to avoid predictable traffic patterns and prevent Trade Republic rate limiting.
|
|
|
|
### 3. Active vs. Dead Asset Tracking
|
|
- **Recency Filter**: Assets are continuously updated with their latest metadata. If an asset has not been updated within **14 days**, it is considered inactive, de-listed, or "dead".
|
|
- **Implicit Filtration**: Queries by other services only return active/valid records unless explicitly requested otherwise.
|
|
|
|
### 4. MQTT RPC Interface
|
|
- **Get Asset by ISIN**:
|
|
- Subscribes to: `services/request/assets_Get/#`
|
|
- Returns a list of all active assets matching the specified ISIN across different instrument types (e.g., matching both a stock and its derivative tracker).
|
|
- **Omnibox Search**:
|
|
- Subscribes to: `services/request/assets_Search/#`
|
|
- Evaluates search keywords against local records (matching `ISIN`, `Name`, or tags like `Region`, `Country`, `Sector`, etc.).
|
|
- **JIT Fallback**: If a search query is formatted as an unknown valid ISIN, it performs a JIT-lookup directly against the Trade Republic API and registers the discovered asset before replying.
|
|
|
|
### 5. Local Indexing File (`index.json`)
|
|
- **Automatic Regeneration**: On database modifications, `AssetsIndexService` is triggered to regenerate `assets/index/index.json`.
|
|
- **Pre-Filtering**: This lean index (containing only `ISIN` and `Name` properties) is written to a shared volume to allow other services to perform instant pre-filtering without querying the database or sending network requests.
|
|
|
|
---
|
|
|
|
## Technical Components
|
|
|
|
| Component | Class / Interface | Responsibility |
|
|
| :--- | :--- | :--- |
|
|
| **Ingestion Worker** | `AssetsFullScanService` | Runs the main cron loop, scraping and paginating all Trade Republic asset types. |
|
|
| **Database Store** | [IAssetsDbService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/AssetsDbService.cs) | Handles CRUD operations, stateful merges, JIT lookups, and active record filtering. |
|
|
| **WebSocket Handler** | [ITradeRepublicService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/TradeRepublicService.cs) | Wraps the connection and request dispatching for the Trade Republic API. |
|
|
| **Local Indexer** | [IAssetsIndexService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/AssetsIndexService.cs) | Serializes the active assets list to `index.json` for external service indexing. |
|
|
| **Messaging Broker** | `AssetsMqttClient` | Listens for incoming MQTT requests, queries the database, and responds to RPC channels. |
|
|
| **Config Service** | [ISettingsDbService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/SettingsDbService.cs) | Manages scanner configurations (scan intervals, active pointers, offsets). |
|