chore: update root config, scripts, and remaining files
This commit is contained in:
+59
-47
@@ -1,66 +1,78 @@
|
||||
# Finlytic Assets Service
|
||||
# Finlytic Enterprise System Architecture
|
||||
|
||||
Finlytic Assets is a scalable C# microservice designed for asset discovery, metadata ingestion, and quick querying. It interfaces directly with the Trade Republic API via WebSockets and exposes a high-performance RPC interface over MQTT to other services in the Finlytic ecosystem.
|
||||
Finlytic is an enterprise financial intelligence platform composed of high-performance C# .NET 8 microservices, a web gateway (`FinlyticBackend`), a Flutter application (`FinlyticApp`), a React web interface (`FinlyticWeb`), and a real-time MQTT event mesh.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
## Ecosystem Architecture Overview
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
TR[Trade Republic WebSocket API] <-->|WS Protocol| TRS[TradeRepublicService]
|
||||
TRS <-->|Ingest| ADS[AssetsFullScanService]
|
||||
ADS <-->|Save / Update| DB[PostgreSQL Database]
|
||||
DB -->|Trigger Index Update| AIS[AssetsIndexService]
|
||||
AIS -->|Write Cache| Index[assets/index/index.json]
|
||||
|
||||
MS[Other Finlytic Services] <-->|MQTT Request-Reply| MqttClient[AssetsMqttClient]
|
||||
MqttClient <-->|Query Cache| DB
|
||||
App[FinlyticApp (Flutter)] -->|HTTP REST & SignalR| Backend[FinlyticBackend]
|
||||
Web[FinlyticWeb (React)] -->|HTTP REST & SignalR| Backend
|
||||
|
||||
Backend <-->|MQTT Pub/Sub & RPC| Broker[MQTT Broker (EMQX / Mosquitto)]
|
||||
|
||||
News[FinlyticNews Service] <-->|MQTT| Broker
|
||||
Sentiment[FinlyticSentiment Service] <-->|MQTT| Broker
|
||||
Assets[FinlyticAssets Service] <-->|MQTT| Broker
|
||||
Fundamentals[FinlyticFundamentals Service] <-->|MQTT| Broker
|
||||
TA[FinlyticTechnicalAnalysis Service] <-->|MQTT| Broker
|
||||
Trades[FinlyticTrades Service] <-->|MQTT| Broker
|
||||
Analyzer[FinlyticAnalyzer Service] <-->|MQTT| Broker
|
||||
|
||||
TR[Trade Republic WS API] <--> Assets
|
||||
N8N[n8n Webhook / FinBERT] <--> News
|
||||
N8N <--> Sentiment
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Core Features & Workflows
|
||||
## Core System Principles & Rules
|
||||
|
||||
### 1. Automated WebSocket scraping
|
||||
- **Continuous Scan**: The background worker (`AssetsFullScanService`) iterates through all support asset types:
|
||||
- **Stocks** (`StockEntity`)
|
||||
- **Crypto** (`CryptoEntity`)
|
||||
- **Derivatives** (`DerivativeEntity`)
|
||||
- **Bonds** (`BondEntity`)
|
||||
- **Funds/ETFs** (`EtfEntity`)
|
||||
- **Resilient Reconnection**: Uses a robust custom socket wrapper (`TradeRepublicClient`) that closes automatically after 5 minutes of inactivity to mimic human interaction profiles and reconnects dynamically when a request is made.
|
||||
1. **Single Web Gateway (`FinlyticBackend`)**:
|
||||
- `FinlyticBackend` is the **only** microservice hosting HTTP REST and SignalR WebSocket endpoints for external clients (`FinlyticApp`, `FinlyticWeb`).
|
||||
- All background worker microservices (`FinlyticNews`, `FinlyticSentiment`, `FinlyticAssets`, `FinlyticFundamentals`, `FinlyticTechnicalAnalysis`, `FinlyticTrades`, `FinlyticAnalyzer`) operate strictly as `IHostedService` worker engines with zero Kestrel HTTP webservers.
|
||||
|
||||
### 2. Stealth Scheduling with Randomized Jitter
|
||||
- **Dynamic Delays**: Scanner wait times are read dynamically from database configuration (`Settings`).
|
||||
- **Jitter Offset**: High/low jitter values (randomized offsets between batches and asset types) are injected to avoid predictable traffic patterns and prevent Trade Republic rate limiting.
|
||||
2. **Exclusive Inter-Service Communication via MQTT**:
|
||||
- All background microservices communicate strictly over MQTT topics (Pub/Sub & RPC).
|
||||
- High-performance, zero-allocation serialization is enforced using `.NET 8 JSON Source Generators` (`FinlyticJsonSerializerContext`).
|
||||
|
||||
### 3. Active vs. Dead Asset Tracking
|
||||
- **Recency Filter**: Assets are continuously updated with their latest metadata. If an asset has not been updated within **14 days**, it is considered inactive, de-listed, or "dead".
|
||||
- **Implicit Filtration**: Queries by other services only return active/valid records unless explicitly requested otherwise.
|
||||
|
||||
### 4. MQTT RPC Interface
|
||||
- **Get Asset by ISIN**:
|
||||
- Subscribes to: `services/request/assets_Get/#`
|
||||
- Returns a list of all active assets matching the specified ISIN across different instrument types (e.g., matching both a stock and its derivative tracker).
|
||||
- **Omnibox Search**:
|
||||
- Subscribes to: `services/request/assets_Search/#`
|
||||
- Evaluates search keywords against local records (matching `ISIN`, `Name`, or tags like `Region`, `Country`, `Sector`, etc.).
|
||||
- **JIT Fallback**: If a search query is formatted as an unknown valid ISIN, it performs a JIT-lookup directly against the Trade Republic API and registers the discovered asset before replying.
|
||||
|
||||
### 5. Local Indexing File (`index.json`)
|
||||
- **Automatic Regeneration**: On database modifications, `AssetsIndexService` is triggered to regenerate `assets/index/index.json`.
|
||||
- **Pre-Filtering**: This lean index (containing only `ISIN` and `Name` properties) is written to a shared volume to allow other services to perform instant pre-filtering without querying the database or sending network requests.
|
||||
3. **Absolute Prohibition of Mock/Demo Data**:
|
||||
- No mock data, hardcoded fallback arrays, or fake dummy responses are permitted in any microservice or frontend client.
|
||||
- Either real data is queried from database contexts (PostgreSQL) / external APIs, or empty result sets / explicit exceptions are returned.
|
||||
|
||||
---
|
||||
|
||||
## Technical Components
|
||||
## Microservices Breakdown
|
||||
|
||||
| Component | Class / Interface | Responsibility |
|
||||
| Project | Type | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| **Ingestion Worker** | `AssetsFullScanService` | Runs the main cron loop, scraping and paginating all Trade Republic asset types. |
|
||||
| **Database Store** | [IAssetsDbService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/AssetsDbService.cs) | Handles CRUD operations, stateful merges, JIT lookups, and active record filtering. |
|
||||
| **WebSocket Handler** | [ITradeRepublicService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/TradeRepublicService.cs) | Wraps the connection and request dispatching for the Trade Republic API. |
|
||||
| **Local Indexer** | [IAssetsIndexService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/AssetsIndexService.cs) | Serializes the active assets list to `index.json` for external service indexing. |
|
||||
| **Messaging Broker** | `AssetsMqttClient` | Listens for incoming MQTT requests, queries the database, and responds to RPC channels. |
|
||||
| **Config Service** | [ISettingsDbService](file:///e:/Projects/FinlyticAssets/FinlyticAssets/Services/SettingsDbService.cs) | Manages scanner configurations (scan intervals, active pointers, offsets). |
|
||||
| **`FinlyticCore`** | Class Library | Shared DTOs, domain models, MQTT infrastructure (`ManagedMqttClient`), and JSON Source Generator context. |
|
||||
| **`FinlyticNews`** | Worker Service | Scraping (Playwright/RSS), deduplication, n8n AI enrichment, and news state persistence. |
|
||||
| **`FinlyticSentiment`** | Worker Service | FinBERT AI sentiment evaluation, ISIN/Sector sentiment aggregation over MQTT. |
|
||||
| **`FinlyticAssets`** | Worker Service | Trade Republic WebSocket full-scan ingestion, asset metadata indexing (`index.json`), and ISIN JIT lookup. |
|
||||
| **`FinlyticFundamentals`** | Worker Service | Financial fundamentals scraping, SEC/Financial Modeling Prep integration, and corporate calendar events. |
|
||||
| **`FinlyticTechnicalAnalysis`** | Worker Service | Real-time technical indicators (RSI, MACD, EMA, Supertrend) and chart pattern detection. |
|
||||
| **`FinlyticTrades`** | Worker Service | Trade lifecycle management (Active, Closed, TTL worker, Feedback exporter). |
|
||||
| **`FinlyticAnalyzer`** | Worker Service | 3-layer filter engine, VIX regime tracking, win-rate calculator, and trade signal generation. |
|
||||
| **`FinlyticBackend`** | Web API / Gateway | ASP.NET Core REST API, JWT authentication, SignalR Hubs (`NewsHub`, `TradeHub`), and MQTT bridge. |
|
||||
| **`FinlyticWeb`** | Web Application | React/Next.js dashboard web application. |
|
||||
| **`FinlyticApp`** | Mobile/Cross-Platform App | Flutter application built with Clean Architecture (`models/`, `repositories/`, `bloc/`). |
|
||||
|
||||
---
|
||||
|
||||
## Status of Implemented & Planned Features
|
||||
|
||||
### Implemented Features
|
||||
- [x] Zero-Allocation MQTT RPC & Event Mesh across all .NET 8 microservices.
|
||||
- [x] Removal of Kestrel HTTP servers from all background worker services (`FinlyticAnalyzer`, `FinlyticTrades`, etc.).
|
||||
- [x] Complete removal of all mock/demo fallbacks in backend and frontend.
|
||||
- [x] PostgreSQL database indexes on `PublishedAt`, `Status`, `SourceUrl`, `Isin`, `CreatedAt`.
|
||||
- [x] Clean Architecture migration across all 7 modules in `FinlyticApp` (Trades, Assets, Favorites, Auth, Admin, Calendar, Search).
|
||||
- [x] SignalR Real-Time Hubs (`NewsHub`, `TradeHub`) with MQTT-to-SignalR broadcasting.
|
||||
|
||||
### Planned Features
|
||||
- [ ] Automated backtesting engine for multi-year strategy evaluation in `FinlyticAnalyzer`.
|
||||
- [ ] Order Execution Integration (automated broker API order routing).
|
||||
- [ ] Push Notifications for iOS/Android via Firebase Cloud Messaging in production deployment.
|
||||
|
||||
Reference in New Issue
Block a user