System Ecosystem Overview
통합 콘텐츠 서비스를 위한 5개 컴포넌트: UI 및 API 서빙(Sb), 데이터 수집 및 요약 오케스트레이션(DP), AI 에이전트 로직(AG), AI 모델 인퍼런스(AM), RAG 벡터 검색(RL)
Last Updated: 2026-01-15
graph TD
%% 스타일 정의
classDef spring fill:#6db33f,stroke:#fff,stroke-width:2px,color:white;
classDef python fill:#3776ab,stroke:#fff,stroke-width:2px,color:white;
classDef ai fill:#9c27b0,stroke:#fff,stroke-width:2px,color:white;
classDef rag fill:#00796b,stroke:#fff,stroke-width:2px,color:white;
classDef db fill:#e1e1e1,stroke:#333,stroke-width:2px;
classDef external fill:#f9f9f9,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5;
classDef user fill:#ff9900,stroke:#fff,stroke-width:2px,color:white;
%% 외부 요소
User((User
Browser)):::user ExtData[External Data Sources
Naver News / Yahoo Finance /
Hacker News / YouTube / Google Search]:::external %% 데이터 저장소 subgraph DataStorage [Shared Data Storage] MySQL[(MySQL
Account DB / Contents DB)]:::db Redis[(Redis
Task Queue / State)]:::db Qdrant[(Qdrant
Vector Store)]:::db R2[Cloudflare R2
Snapshot Backup]:::db end %% Sb - 메인 서버 subgraph SbServer [Sb - sbServer - Render] direction TB SB[Spring Boot 3.2.4
Java 21]:::spring SB_UI[Web Dashboard
Thymeleaf] SB_API[REST API
11 Content Modules] SB_AUTH[Auth Layer
JWT + Google OAuth] end %% DP - 데이터 수집 파이프라인 subgraph DPServer [DP - data-collect-pipeline] DP[Prefect Pipeline
Python 3.12]:::python DP_COLLECT[Collectors
News/Market/Tech/YouTube/Search] DP_SEARCH[Search Flow
1min polling] DP_MONITOR[Summary Monitor
5min polling] DP_REPORT[Report Generator
Daily Markdown] DP_RAG[RAG Ingest
Vector Sync] end %% AG - AI 에이전트 subgraph AGServer [AG - AI-aGent-AG - Port 8200] AG[FastAPI + LangGraph
Worker Process]:::ai AG_CLASS[Classification Node
LLM-based Routing] AG_HANDLERS[4 Summary Nodes
News/Tech/YouTube/Search] AG_MAPREDUCE[Map-Reduce Pipeline
Cluster + Summarize] AG_RAG[RAG Ingest Node
Vector DB Sync] AG_ENHANCE[Enhancement Pipeline
Visual Prompt + Image] end %% AM - AI 모델 러너 subgraph AMServer [AM - AI-Model-Runner-AM - Port 8100] AM[FastAPI Server
+ Queue Worker]:::ai AM_LLM[MLX LLM
Qwen3-8B-MLX-4bit] AM_IMG[FLUX.1-schnell
8-bit quantized] AM_MGR[Model Manager
Auto Switch] end %% RL - RAG 서버 subgraph RLServer [RL - Rag-LlamaIndex-RL - Port 8080] RL[FastAPI + LlamaIndex
Knowledge Engine]:::rag RL_EMBED[Embedding Model
BAAI/bge-m3] RL_QUERY[Vector Search
Hybrid Query] RL_SNAP[Snapshot Manager
Cloud Sync] end %% 연결 관계: 사용자 → Sb User -->|HTTP/HTTPS| SB SB --> SB_UI SB --> SB_API SB --> SB_AUTH SB_API -->|JPA + MyBatis| MySQL %% DP 데이터 수집 흐름 DP --> DP_COLLECT DP --> DP_SEARCH DP --> DP_MONITOR DP --> DP_REPORT DP --> DP_RAG DP_COLLECT -->|1. Crawl & API| ExtData DP_COLLECT -->|2. Store Raw Data| MySQL DP_SEARCH -->|Google Custom Search| ExtData %% DP → AG 요약 요청 흐름 DP_MONITOR -->|3. tasks:pending Queue| AG DP_MONITOR -->|4. Poll task status| Redis %% AG 내부 처리 흐름 AG --> AG_CLASS AG_CLASS --> AG_HANDLERS AG_HANDLERS --> AG_MAPREDUCE AG_MAPREDUCE --> AG_RAG AG_RAG --> AG_ENHANCE %% AG → RL RAG 연동 AG_RAG -->|HTTP POST /ingest/graph| RL RL --> RL_EMBED RL --> RL_QUERY RL --> RL_SNAP RL_EMBED --> Qdrant RL_SNAP --> R2 %% AG → AM 추론 요청 (Redis Queue) AG_MAPREDUCE -->|5. am:requests queue
Redis LPUSH| AM AM --> AM_MGR AM_MGR --> AM_LLM AM_MGR --> AM_IMG %% AM → Redis 결과 AM -->|am:result:id
Redis| Redis %% 결과 저장 DP_MONITOR -->|6. Store Summary| MySQL DP_REPORT -->|Generate Daily Report| MySQL DP_RAG -->|7. Register RAG Ingest| Redis
Browser)):::user ExtData[External Data Sources
Naver News / Yahoo Finance /
Hacker News / YouTube / Google Search]:::external %% 데이터 저장소 subgraph DataStorage [Shared Data Storage] MySQL[(MySQL
Account DB / Contents DB)]:::db Redis[(Redis
Task Queue / State)]:::db Qdrant[(Qdrant
Vector Store)]:::db R2[Cloudflare R2
Snapshot Backup]:::db end %% Sb - 메인 서버 subgraph SbServer [Sb - sbServer - Render] direction TB SB[Spring Boot 3.2.4
Java 21]:::spring SB_UI[Web Dashboard
Thymeleaf] SB_API[REST API
11 Content Modules] SB_AUTH[Auth Layer
JWT + Google OAuth] end %% DP - 데이터 수집 파이프라인 subgraph DPServer [DP - data-collect-pipeline] DP[Prefect Pipeline
Python 3.12]:::python DP_COLLECT[Collectors
News/Market/Tech/YouTube/Search] DP_SEARCH[Search Flow
1min polling] DP_MONITOR[Summary Monitor
5min polling] DP_REPORT[Report Generator
Daily Markdown] DP_RAG[RAG Ingest
Vector Sync] end %% AG - AI 에이전트 subgraph AGServer [AG - AI-aGent-AG - Port 8200] AG[FastAPI + LangGraph
Worker Process]:::ai AG_CLASS[Classification Node
LLM-based Routing] AG_HANDLERS[4 Summary Nodes
News/Tech/YouTube/Search] AG_MAPREDUCE[Map-Reduce Pipeline
Cluster + Summarize] AG_RAG[RAG Ingest Node
Vector DB Sync] AG_ENHANCE[Enhancement Pipeline
Visual Prompt + Image] end %% AM - AI 모델 러너 subgraph AMServer [AM - AI-Model-Runner-AM - Port 8100] AM[FastAPI Server
+ Queue Worker]:::ai AM_LLM[MLX LLM
Qwen3-8B-MLX-4bit] AM_IMG[FLUX.1-schnell
8-bit quantized] AM_MGR[Model Manager
Auto Switch] end %% RL - RAG 서버 subgraph RLServer [RL - Rag-LlamaIndex-RL - Port 8080] RL[FastAPI + LlamaIndex
Knowledge Engine]:::rag RL_EMBED[Embedding Model
BAAI/bge-m3] RL_QUERY[Vector Search
Hybrid Query] RL_SNAP[Snapshot Manager
Cloud Sync] end %% 연결 관계: 사용자 → Sb User -->|HTTP/HTTPS| SB SB --> SB_UI SB --> SB_API SB --> SB_AUTH SB_API -->|JPA + MyBatis| MySQL %% DP 데이터 수집 흐름 DP --> DP_COLLECT DP --> DP_SEARCH DP --> DP_MONITOR DP --> DP_REPORT DP --> DP_RAG DP_COLLECT -->|1. Crawl & API| ExtData DP_COLLECT -->|2. Store Raw Data| MySQL DP_SEARCH -->|Google Custom Search| ExtData %% DP → AG 요약 요청 흐름 DP_MONITOR -->|3. tasks:pending Queue| AG DP_MONITOR -->|4. Poll task status| Redis %% AG 내부 처리 흐름 AG --> AG_CLASS AG_CLASS --> AG_HANDLERS AG_HANDLERS --> AG_MAPREDUCE AG_MAPREDUCE --> AG_RAG AG_RAG --> AG_ENHANCE %% AG → RL RAG 연동 AG_RAG -->|HTTP POST /ingest/graph| RL RL --> RL_EMBED RL --> RL_QUERY RL --> RL_SNAP RL_EMBED --> Qdrant RL_SNAP --> R2 %% AG → AM 추론 요청 (Redis Queue) AG_MAPREDUCE -->|5. am:requests queue
Redis LPUSH| AM AM --> AM_MGR AM_MGR --> AM_LLM AM_MGR --> AM_IMG %% AM → Redis 결과 AM -->|am:result:id
Redis| Redis %% 결과 저장 DP_MONITOR -->|6. Store Summary| MySQL DP_REPORT -->|Generate Daily Report| MySQL DP_RAG -->|7. Register RAG Ingest| Redis
Component Roles & Tech Stack
| Component | Project | Role & Responsibilities | Tech Stack |
|---|---|---|---|
| Sb | sbServer |
통합 백엔드 & UI 서버 - 11개 콘텐츠 모듈: Prompt/Notebook, News, Search, Weather, Market, Map, Chat, Document, Restaurant, SearchSummary - Restaurant 모듈: Haversine 거리 계산, 위치 기반 추천 - 이중 데이터소스: Account DB + Contents DB - 인증: JWT Access/Refresh Token + Google OAuth - WebSocket 실시간 채팅 지원 - 레이어드 아키텍처: Driver → Application → Infra |
Java 21, Spring Boot 3.2.4 JPA + MyBatis, MySQL Thymeleaf, OpenFeign Redis, WebSocket Swagger/OpenAPI |
| DP | data-collect-pipeline |
데이터 수집 & 요약 오케스트레이션 - 7개 Prefect Flow: News, Market, Tech, YouTube, Search(1min), Summary Request, Summary Monitor - 스케줄: 매시(뉴스/마켓/유튜브), 4x/일(테크), 22시(요약 요청), 5분(모니터링), 1분(검색) - Collectors: Naver News API, Yahoo Finance, Hacker News, YouTube Transcript, Google Custom Search - Report Generator: 마켓 + 뉴스요약 + 테크요약 → 일일 마크다운 리포트 - Async 처리: Redis Queue 기반 비동기 요약 요청/모니터링 - RAG Ingestion: 완료된 요약을 RL 서버로 동기화 |
Python 3.12, Prefect MySQL, Redis (Queue + Task State) yfinance, trafilatura, scrapetube youtube-transcript-api, google-api |
| AG | AI-aGent-AG |
AI 에이전트 서버 (Port 8200) - LangGraph 기반 그래프 처리: StateGraph + Conditional Edges - Worker Process: Redis Queue(tasks:pending) 소비 - Classification Node: LLM 기반 입력 텍스트 분류 → 적절한 핸들러 라우팅 - 4개 Summary Nodes: NEWS_TOTAL, TECH_TOTAL, YOUTUBE, SEARCH + DEFAULT - Map-Reduce Pipeline: Cluster(SBERT + K-Means) → Map → Reduce (반복 병합) - RAG Ingest Node: RL 서버로 Vector DB 동기화 - Enhancement Pipeline: Judge → Visual Prompt → Resource Gate → AM Dispatch - Redis Checkpointer: 상태 저장 및 재개(Resume/Interrupt) 지원 |
Python, FastAPI, Uvicorn LangGraph (StateGraph) SentenceTransformers (SBERT) scikit-learn, markdown-it-py Redis (tasks:pending queue) httpx (async HTTP to RL) |
| AM | AI-Model-Runner-AM |
AI 모델 인퍼런스 서버 (Port 8100) - Queue Worker 아키텍처: main.py (API) + worker.py (추론) - LLM: MLX-LM 기반 Qwen3-8B-MLX-4bit (max_tokens: 131,072) - Image: FLUX.1-schnell (8-bit quantized via mflux) - Model Manager: LLM/Image 중 하나만 메모리 유지, 자동 전환 - 가드레일: 프롬프트 길이 검증 (400K자 제한) - <think> 태그 필터링 |
Python, FastAPI, Uvicorn mlx-lm (Apple Silicon) mflux (FLUX.1), Pillow Redis (am:requests queue) |
| RL | Rag-LlamaIndex-RL |
RAG 벡터 검색 서버 (Port 8080) - LlamaIndex 기반 Knowledge Engine - Vector Store: Qdrant (로컬 파일 모드, 1024-dim) - Embedding: BAAI/bge-m3 (다국어 지원) - Graph Ingest API: AG에서 요약 텍스트 수신 → 벡터화 - Query API: 하이브리드 검색 (Vector + Entity) - Snapshot Manager: R2 클라우드 자동 동기화 (1시간 주기) - RDB Sync Script: MySQL → Qdrant 일괄 동기화 |
Python, FastAPI, Uvicorn LlamaIndex 0.10+ Qdrant (Vector Store) sentence-transformers boto3 (S3/R2 Sync) PyMySQL, Redis |
Data Flow & Processing Pipeline
sequenceDiagram
autonumber
participant User as User (Browser)
participant Sb as Sb (sbServer)
participant DB as MySQL
participant Redis as Redis
participant DP as DP (Pipeline)
participant AG as AG (Worker)
participant AM as AM (Worker)
participant RL as RL (RAG)
participant Ext as External Sources
Note over DP,Ext: 1. Data Collection (Hourly: News/Market/YouTube, 4x Daily: Tech, 1min: Search)
DP->>Ext: Crawl via Collectors
Note right of Ext: Naver News API
Yahoo Finance
Hacker News
YouTube Transcript
Google Custom Search Ext-->>DP: Raw Data (JSON) DP->>DB: Store (news_contents, market_indices, tech_contents, youtube_summaries) Note over DP,AM: 2. Summary Request (Daily 22:00 UTC) DP->>DP: Fetch recent 24h data from DB DP->>DP: Deduplicate by link/URL DP->>Redis: LPUSH tasks:pending (task_id) Redis-->>DP: task_id (UUID) DP->>Redis: Store in waiting:summary:* sets Note over AG,RL: 3. AI Processing (LangGraph Worker) AG->>Redis: BRPOP tasks:pending AG->>AG: classification_node() Note right of AG: LLM classifies input type:
NEWS_TOTAL, TECH_TOTAL,
YOUTUBE, SEARCH, DEFAULT AG->>AG: Route to summary node AG->>AG: cluster_node() + map_node() AG->>Redis: LPUSH am:requests AM->>Redis: BRPOP am:requests AM->>AM: switch_to_llm() AM->>AM: MLX Qwen3-8B Inference AM->>Redis: SET am:result:{request_id} AG->>Redis: Poll am:result:{request_id} AG->>AG: reduce_node() (iterate) AG->>RL: POST /api/v1/ingest/graph (summary text) RL->>RL: Embed with bge-m3 RL-->>AG: {entities_added, success} AG->>Redis: Update task status to COMPLETED Note over DP,DB: 4. Result Polling & Storage (Every 5 min) DP->>Redis: Get all pending tasks DP->>AG: Check status for each task_id AG-->>DP: {status: COMPLETED, result: ...} DP->>DB: save_daily_news_summary() / save_daily_tech_summary() DP->>DP: Check if both summaries ready DP->>DP: ReportGenerator.generate() DP->>DB: save_daily_report(markdown) DP->>Redis: Register RAG ingest request Note over User,Sb: 5. User Dashboard Access User->>Sb: Request /news/summary/daily-news Sb->>Sb: @RequireOAuth JWT validation Sb->>DB: Query via JPA Repository DB-->>Sb: DailyNewsSummaryEntity Sb-->>User: Render Dashboard (Thymeleaf)
Yahoo Finance
Hacker News
YouTube Transcript
Google Custom Search Ext-->>DP: Raw Data (JSON) DP->>DB: Store (news_contents, market_indices, tech_contents, youtube_summaries) Note over DP,AM: 2. Summary Request (Daily 22:00 UTC) DP->>DP: Fetch recent 24h data from DB DP->>DP: Deduplicate by link/URL DP->>Redis: LPUSH tasks:pending (task_id) Redis-->>DP: task_id (UUID) DP->>Redis: Store in waiting:summary:* sets Note over AG,RL: 3. AI Processing (LangGraph Worker) AG->>Redis: BRPOP tasks:pending AG->>AG: classification_node() Note right of AG: LLM classifies input type:
NEWS_TOTAL, TECH_TOTAL,
YOUTUBE, SEARCH, DEFAULT AG->>AG: Route to summary node AG->>AG: cluster_node() + map_node() AG->>Redis: LPUSH am:requests AM->>Redis: BRPOP am:requests AM->>AM: switch_to_llm() AM->>AM: MLX Qwen3-8B Inference AM->>Redis: SET am:result:{request_id} AG->>Redis: Poll am:result:{request_id} AG->>AG: reduce_node() (iterate) AG->>RL: POST /api/v1/ingest/graph (summary text) RL->>RL: Embed with bge-m3 RL-->>AG: {entities_added, success} AG->>Redis: Update task status to COMPLETED Note over DP,DB: 4. Result Polling & Storage (Every 5 min) DP->>Redis: Get all pending tasks DP->>AG: Check status for each task_id AG-->>DP: {status: COMPLETED, result: ...} DP->>DB: save_daily_news_summary() / save_daily_tech_summary() DP->>DP: Check if both summaries ready DP->>DP: ReportGenerator.generate() DP->>DB: save_daily_report(markdown) DP->>Redis: Register RAG ingest request Note over User,Sb: 5. User Dashboard Access User->>Sb: Request /news/summary/daily-news Sb->>Sb: @RequireOAuth JWT validation Sb->>DB: Query via JPA Repository DB-->>Sb: DailyNewsSummaryEntity Sb-->>User: Render Dashboard (Thymeleaf)
DP (data-collect-pipeline) Internal Architecture
graph TB
classDef task fill:#ff9900,stroke:#333,stroke-width:1px,color:white;
classDef adapter fill:#3776ab,stroke:#333,stroke-width:1px,color:white;
classDef storage fill:#e1e1e1,stroke:#333,stroke-width:2px;
subgraph Orchestration [Prefect Orchestration - main_prefect.py]
MAIN[7 Concurrent Flows]
end
subgraph Tasks [Task Modules - 7 Flows]
T_NEWS[news_tasks.py
Hourly]:::task T_MARKET[market_tasks.py
Hourly]:::task T_TECH[tech_trend_tasks.py
4x Daily]:::task T_YT[syukaworld_tasks.py
Hourly]:::task T_SEARCH[search_tasks.py
Every 1 min]:::task T_SUMMARY[summary_tasks.py
Daily 22:00]:::task T_MONITOR[summary_monitor_tasks.py
Every 5 min]:::task end subgraph Adapters [Adapter Layer - Singleton Pattern] subgraph Collectors [adapter/collector/] C_NEWS[NewsCollector
Naver API + Trafilatura]:::adapter C_MARKET[MarketCollector
yfinance]:::adapter C_TECH[TechCollector
Hacker News]:::adapter C_YT[YouTubeCollector
scrapetube + transcript]:::adapter C_SEARCH[GoogleSearchCollector
Custom Search API]:::adapter end subgraph Processing [adapter/llm/ & queue/] LLM[LLMProcessor
sync & async]:::adapter QUEUE[TaskQueue
Redis Queue Client]:::adapter end subgraph Storage [adapter/db/ & redis/] DB_REPO[DBRepository
MySQL CRUD]:::adapter REDIS_REPO[RedisRepository
Task State + RAG Queue]:::adapter end REPORT[ReportGenerator
Markdown]:::adapter end subgraph External [External Storage] MYSQL[(MySQL)]:::storage REDIS[(Redis)]:::storage R2[Cloudflare R2]:::storage end MAIN --> T_NEWS MAIN --> T_MARKET MAIN --> T_TECH MAIN --> T_YT MAIN --> T_SEARCH MAIN --> T_SUMMARY MAIN --> T_MONITOR T_NEWS --> C_NEWS T_MARKET --> C_MARKET T_TECH --> C_TECH T_YT --> C_YT T_SEARCH --> C_SEARCH T_SUMMARY --> QUEUE T_MONITOR --> QUEUE QUEUE --> REDIS C_NEWS --> DB_REPO C_MARKET --> DB_REPO C_TECH --> DB_REPO C_YT --> DB_REPO T_MONITOR --> DB_REPO T_MONITOR --> REDIS_REPO T_MONITOR --> REPORT DB_REPO --> MYSQL REDIS_REPO --> REDIS REPORT --> MYSQL
Hourly]:::task T_MARKET[market_tasks.py
Hourly]:::task T_TECH[tech_trend_tasks.py
4x Daily]:::task T_YT[syukaworld_tasks.py
Hourly]:::task T_SEARCH[search_tasks.py
Every 1 min]:::task T_SUMMARY[summary_tasks.py
Daily 22:00]:::task T_MONITOR[summary_monitor_tasks.py
Every 5 min]:::task end subgraph Adapters [Adapter Layer - Singleton Pattern] subgraph Collectors [adapter/collector/] C_NEWS[NewsCollector
Naver API + Trafilatura]:::adapter C_MARKET[MarketCollector
yfinance]:::adapter C_TECH[TechCollector
Hacker News]:::adapter C_YT[YouTubeCollector
scrapetube + transcript]:::adapter C_SEARCH[GoogleSearchCollector
Custom Search API]:::adapter end subgraph Processing [adapter/llm/ & queue/] LLM[LLMProcessor
sync & async]:::adapter QUEUE[TaskQueue
Redis Queue Client]:::adapter end subgraph Storage [adapter/db/ & redis/] DB_REPO[DBRepository
MySQL CRUD]:::adapter REDIS_REPO[RedisRepository
Task State + RAG Queue]:::adapter end REPORT[ReportGenerator
Markdown]:::adapter end subgraph External [External Storage] MYSQL[(MySQL)]:::storage REDIS[(Redis)]:::storage R2[Cloudflare R2]:::storage end MAIN --> T_NEWS MAIN --> T_MARKET MAIN --> T_TECH MAIN --> T_YT MAIN --> T_SEARCH MAIN --> T_SUMMARY MAIN --> T_MONITOR T_NEWS --> C_NEWS T_MARKET --> C_MARKET T_TECH --> C_TECH T_YT --> C_YT T_SEARCH --> C_SEARCH T_SUMMARY --> QUEUE T_MONITOR --> QUEUE QUEUE --> REDIS C_NEWS --> DB_REPO C_MARKET --> DB_REPO C_TECH --> DB_REPO C_YT --> DB_REPO T_MONITOR --> DB_REPO T_MONITOR --> REDIS_REPO T_MONITOR --> REPORT DB_REPO --> MYSQL REDIS_REPO --> REDIS REPORT --> MYSQL
AG (AI-aGent-AG) Internal Architecture - LangGraph Worker
graph TB
classDef worker fill:#ff5722,stroke:#333,stroke-width:1px,color:white;
classDef node fill:#9c27b0,stroke:#333,stroke-width:1px,color:white;
classDef client fill:#3776ab,stroke:#333,stroke-width:1px,color:white;
classDef rag fill:#00796b,stroke:#333,stroke-width:1px,color:white;
subgraph Worker [worker.py - AGWorker]
direction LR
W_MAIN[Main Loop
BRPOP tasks:pending]:::worker W_PROCESS[process_batch_task_with_graph
LangGraph Invoke]:::worker W_DISPATCH[check_and_resume_dispatched
10s Periodic]:::worker end subgraph LangGraph [LangGraph StateGraph - main_graph.py] START_N((START)) CLASSIFY[classification_node
LLM Classification]:::node subgraph SummaryNodes [Summary Nodes] CLUSTER[cluster_node
SBERT + K-Means]:::node MAP[map_node
Chunk Summarize]:::node REDUCE[reduce_node
Merge 5 at a time]:::node YT_SUM[youtube_summary_node]:::node SEARCH_SUM[search_summary_node]:::node DEFAULT[default_node
Q&A]:::node end subgraph RAGNode [RAG Integration] RAG_INGEST[rag_ingest_node
HTTP to RL Server]:::rag end subgraph EnhanceNodes [Enhancement Nodes] JUDGE[enhancement_judge_node
Score 1-10]:::node VISUAL[visual_prompt_node
Generate Prompt]:::node GATE[resource_gate_node
Check AM Queue/Memory]:::node AM_DISP[am_dispatch_node
Send to AM + Interrupt]:::node AM_RES[am_result_node
Resume + Fetch Result]:::node end END_N((END)) end subgraph Clients [External Clients] AM_CLI[AMClient
Redis LPUSH am:requests]:::client RL_CLI[RLClient
HTTP POST /ingest/graph]:::client end subgraph Checkpointer [Redis Checkpointer] CKPT[RedisCheckpointSaver
State Persistence] end W_MAIN --> W_PROCESS W_PROCESS --> START_N W_DISPATCH --> CKPT START_N --> CLASSIFY CLASSIFY -->|NEWS_TOTAL| CLUSTER CLASSIFY -->|TECH_TOTAL| CLUSTER CLASSIFY -->|YOUTUBE| YT_SUM CLASSIFY -->|SEARCH| SEARCH_SUM CLASSIFY -->|DEFAULT| DEFAULT CLUSTER --> MAP MAP --> REDUCE REDUCE -->|iterate| MAP REDUCE --> RAG_INGEST YT_SUM --> RAG_INGEST SEARCH_SUM --> RAG_INGEST DEFAULT --> RAG_INGEST RAG_INGEST --> JUDGE RAG_INGEST --> RL_CLI JUDGE -->|enhance score >= 8| VISUAL JUDGE -->|end| END_N VISUAL --> GATE GATE -->|dispatch| AM_DISP GATE -->|wait| END_N AM_DISP --> AM_RES AM_RES --> END_N MAP --> AM_CLI AM_CLI --> REDUCE
BRPOP tasks:pending]:::worker W_PROCESS[process_batch_task_with_graph
LangGraph Invoke]:::worker W_DISPATCH[check_and_resume_dispatched
10s Periodic]:::worker end subgraph LangGraph [LangGraph StateGraph - main_graph.py] START_N((START)) CLASSIFY[classification_node
LLM Classification]:::node subgraph SummaryNodes [Summary Nodes] CLUSTER[cluster_node
SBERT + K-Means]:::node MAP[map_node
Chunk Summarize]:::node REDUCE[reduce_node
Merge 5 at a time]:::node YT_SUM[youtube_summary_node]:::node SEARCH_SUM[search_summary_node]:::node DEFAULT[default_node
Q&A]:::node end subgraph RAGNode [RAG Integration] RAG_INGEST[rag_ingest_node
HTTP to RL Server]:::rag end subgraph EnhanceNodes [Enhancement Nodes] JUDGE[enhancement_judge_node
Score 1-10]:::node VISUAL[visual_prompt_node
Generate Prompt]:::node GATE[resource_gate_node
Check AM Queue/Memory]:::node AM_DISP[am_dispatch_node
Send to AM + Interrupt]:::node AM_RES[am_result_node
Resume + Fetch Result]:::node end END_N((END)) end subgraph Clients [External Clients] AM_CLI[AMClient
Redis LPUSH am:requests]:::client RL_CLI[RLClient
HTTP POST /ingest/graph]:::client end subgraph Checkpointer [Redis Checkpointer] CKPT[RedisCheckpointSaver
State Persistence] end W_MAIN --> W_PROCESS W_PROCESS --> START_N W_DISPATCH --> CKPT START_N --> CLASSIFY CLASSIFY -->|NEWS_TOTAL| CLUSTER CLASSIFY -->|TECH_TOTAL| CLUSTER CLASSIFY -->|YOUTUBE| YT_SUM CLASSIFY -->|SEARCH| SEARCH_SUM CLASSIFY -->|DEFAULT| DEFAULT CLUSTER --> MAP MAP --> REDUCE REDUCE -->|iterate| MAP REDUCE --> RAG_INGEST YT_SUM --> RAG_INGEST SEARCH_SUM --> RAG_INGEST DEFAULT --> RAG_INGEST RAG_INGEST --> JUDGE RAG_INGEST --> RL_CLI JUDGE -->|enhance score >= 8| VISUAL JUDGE -->|end| END_N VISUAL --> GATE GATE -->|dispatch| AM_DISP GATE -->|wait| END_N AM_DISP --> AM_RES AM_RES --> END_N MAP --> AM_CLI AM_CLI --> REDUCE
AM (AI-Model-Runner-AM) Internal Architecture
graph TB
classDef api fill:#ff9900,stroke:#333,stroke-width:1px,color:white;
classDef model fill:#e91e63,stroke:#333,stroke-width:1px,color:white;
classDef mgr fill:#4caf50,stroke:#333,stroke-width:1px,color:white;
classDef worker fill:#2196f3,stroke:#333,stroke-width:1px,color:white;
subgraph APIServer [main.py - FastAPI Server Port 8100]
direction LR
LLM_GEN[POST /llm/generate]:::api
LLM_STREAM[POST /llm/stream
SSE]:::api LLM_STATUS[GET /llm/status]:::api IMG_GEN[POST /image/generate]:::api IMG_STATUS[GET /image/status]:::api HEALTH[GET /health]:::api QUEUE_ST[GET /queue/status]:::api end subgraph Worker [worker.py - AMWorker] WORKER[AMWorker.run
Main Loop]:::worker PROC_LLM[_process_llm
LLM Inference]:::worker PROC_IMG[_process_image
Image Generation]:::worker VALIDATE[_validate_prompt_length
400K chars limit]:::worker CLEAN[_clean_llm_response
Remove think tags]:::worker end subgraph Manager [Model Manager - model_manager.py] MGR[switch_to_llm
switch_to_image]:::mgr STATUS[get_model_status]:::mgr end subgraph LLMAdapter [LanguageModelAdapter - Singleton] LLM_LOAD[_load_model
Lazy Loading]:::model LLM_INFER[generate_response
Blocking]:::model LLM_STREAM_GEN[generate_response_stream
Generator]:::model LLM_UNLOAD[unload_model
gc.collect]:::model MLX[mlx-lm
Qwen3-8B-MLX-4bit] end subgraph ImageAdapter [ImageModelAdapter - Singleton] IMG_LOAD[_load_model
mflux FLUX.1]:::model FLUX[FLUX.1-schnell
8-bit quantized] IMG_TO_B64[_image_to_base64
PIL to Base64 PNG]:::model IMG_UNLOAD[unload_model
Auto after gen]:::model end subgraph RedisAdapter [RedisQueueAdapter] POP[pop_request
BRPOP am:requests] PUSH[push_result
SET am:result:id] REDIS[(Redis
am:requests queue)] end HEALTH --> STATUS QUEUE_ST --> RedisAdapter WORKER -->|BRPOP| POP POP --> REDIS WORKER --> VALIDATE VALIDATE --> PROC_LLM VALIDATE --> PROC_IMG PROC_LLM --> MGR PROC_IMG --> MGR MGR -->|Unload Image| IMG_UNLOAD MGR -->|Load LLM| LLM_LOAD MGR -->|Unload LLM| LLM_UNLOAD MGR -->|Load Image| IMG_LOAD LLM_LOAD --> MLX LLM_INFER --> MLX PROC_LLM --> CLEAN CLEAN --> PUSH IMG_LOAD --> FLUX FLUX --> IMG_TO_B64 IMG_TO_B64 --> PUSH PUSH --> REDIS
SSE]:::api LLM_STATUS[GET /llm/status]:::api IMG_GEN[POST /image/generate]:::api IMG_STATUS[GET /image/status]:::api HEALTH[GET /health]:::api QUEUE_ST[GET /queue/status]:::api end subgraph Worker [worker.py - AMWorker] WORKER[AMWorker.run
Main Loop]:::worker PROC_LLM[_process_llm
LLM Inference]:::worker PROC_IMG[_process_image
Image Generation]:::worker VALIDATE[_validate_prompt_length
400K chars limit]:::worker CLEAN[_clean_llm_response
Remove think tags]:::worker end subgraph Manager [Model Manager - model_manager.py] MGR[switch_to_llm
switch_to_image]:::mgr STATUS[get_model_status]:::mgr end subgraph LLMAdapter [LanguageModelAdapter - Singleton] LLM_LOAD[_load_model
Lazy Loading]:::model LLM_INFER[generate_response
Blocking]:::model LLM_STREAM_GEN[generate_response_stream
Generator]:::model LLM_UNLOAD[unload_model
gc.collect]:::model MLX[mlx-lm
Qwen3-8B-MLX-4bit] end subgraph ImageAdapter [ImageModelAdapter - Singleton] IMG_LOAD[_load_model
mflux FLUX.1]:::model FLUX[FLUX.1-schnell
8-bit quantized] IMG_TO_B64[_image_to_base64
PIL to Base64 PNG]:::model IMG_UNLOAD[unload_model
Auto after gen]:::model end subgraph RedisAdapter [RedisQueueAdapter] POP[pop_request
BRPOP am:requests] PUSH[push_result
SET am:result:id] REDIS[(Redis
am:requests queue)] end HEALTH --> STATUS QUEUE_ST --> RedisAdapter WORKER -->|BRPOP| POP POP --> REDIS WORKER --> VALIDATE VALIDATE --> PROC_LLM VALIDATE --> PROC_IMG PROC_LLM --> MGR PROC_IMG --> MGR MGR -->|Unload Image| IMG_UNLOAD MGR -->|Load LLM| LLM_LOAD MGR -->|Unload LLM| LLM_UNLOAD MGR -->|Load Image| IMG_LOAD LLM_LOAD --> MLX LLM_INFER --> MLX PROC_LLM --> CLEAN CLEAN --> PUSH IMG_LOAD --> FLUX FLUX --> IMG_TO_B64 IMG_TO_B64 --> PUSH PUSH --> REDIS
RL (Rag-LlamaIndex-RL) Internal Architecture
graph TB
classDef api fill:#ff9900,stroke:#333,stroke-width:1px,color:white;
classDef engine fill:#00796b,stroke:#333,stroke-width:1px,color:white;
classDef storage fill:#e1e1e1,stroke:#333,stroke-width:2px;
classDef cloud fill:#2196f3,stroke:#333,stroke-width:1px,color:white;
subgraph APIServer [main.py - FastAPI Server Port 8080]
direction LR
INGEST[POST /api/v1/ingest/graph
Graph Ingest]:::api QUERY[POST /api/v1/query
Vector Search]:::api HEALTH[GET /api/v1/health]:::api STATS[GET /api/v1/stats]:::api end subgraph Engine [KnowledgeEngine - Singleton] INIT[initialize
Load Models + Restore]:::engine INGEST_FN[ingest_graph
Entity → TextNode]:::engine QUERY_FN[query
Vector Search]:::engine SHUTDOWN[shutdown
Save + Upload]:::engine end subgraph Processing [Document Processing] SPLITTER[SentenceSplitter
512 tokens, 50 overlap]:::engine EMBED[BAAI/bge-m3
1024-dim Embedding]:::engine UUID[UUID5 Generator
Deterministic IDs]:::engine end subgraph VectorStore [Qdrant Vector Store] QDRANT_LOCAL[(Qdrant Local
./data/qdrant)]:::storage COLLECTION[knowledge_graph
Collection]:::storage end subgraph SnapshotManager [Snapshot Manager] DOWNLOAD[download_snapshot
Startup Restore]:::cloud UPLOAD[periodic_upload
1 Hour Interval]:::cloud FINAL[final_upload
Shutdown Save]:::cloud end subgraph CloudStorage [Cloudflare R2] R2[(R2 Bucket
rag-snapshot.tar.gz)]:::storage end subgraph RDBSync [scripts/sync_rdb_to_qdrant.py] SYNC[MySQL → Qdrant
Batch Sync]:::engine end INGEST --> INGEST_FN QUERY --> QUERY_FN HEALTH --> Engine STATS --> Engine INIT --> DOWNLOAD DOWNLOAD --> R2 SHUTDOWN --> FINAL FINAL --> R2 UPLOAD --> R2 INGEST_FN --> SPLITTER SPLITTER --> EMBED EMBED --> UUID UUID --> QDRANT_LOCAL QDRANT_LOCAL --> COLLECTION QUERY_FN --> EMBED EMBED --> QDRANT_LOCAL SYNC --> QDRANT_LOCAL
Graph Ingest]:::api QUERY[POST /api/v1/query
Vector Search]:::api HEALTH[GET /api/v1/health]:::api STATS[GET /api/v1/stats]:::api end subgraph Engine [KnowledgeEngine - Singleton] INIT[initialize
Load Models + Restore]:::engine INGEST_FN[ingest_graph
Entity → TextNode]:::engine QUERY_FN[query
Vector Search]:::engine SHUTDOWN[shutdown
Save + Upload]:::engine end subgraph Processing [Document Processing] SPLITTER[SentenceSplitter
512 tokens, 50 overlap]:::engine EMBED[BAAI/bge-m3
1024-dim Embedding]:::engine UUID[UUID5 Generator
Deterministic IDs]:::engine end subgraph VectorStore [Qdrant Vector Store] QDRANT_LOCAL[(Qdrant Local
./data/qdrant)]:::storage COLLECTION[knowledge_graph
Collection]:::storage end subgraph SnapshotManager [Snapshot Manager] DOWNLOAD[download_snapshot
Startup Restore]:::cloud UPLOAD[periodic_upload
1 Hour Interval]:::cloud FINAL[final_upload
Shutdown Save]:::cloud end subgraph CloudStorage [Cloudflare R2] R2[(R2 Bucket
rag-snapshot.tar.gz)]:::storage end subgraph RDBSync [scripts/sync_rdb_to_qdrant.py] SYNC[MySQL → Qdrant
Batch Sync]:::engine end INGEST --> INGEST_FN QUERY --> QUERY_FN HEALTH --> Engine STATS --> Engine INIT --> DOWNLOAD DOWNLOAD --> R2 SHUTDOWN --> FINAL FINAL --> R2 UPLOAD --> R2 INGEST_FN --> SPLITTER SPLITTER --> EMBED EMBED --> UUID UUID --> QDRANT_LOCAL QDRANT_LOCAL --> COLLECTION QUERY_FN --> EMBED EMBED --> QDRANT_LOCAL SYNC --> QDRANT_LOCAL
Sb (sbServer) Internal Layered Architecture
3-tier 레이어 구조: Driver(진입점) → Application(비즈니스 로직) → Infra(외부 연동)
graph TB
classDef driver fill:#ff9900,stroke:#333,stroke-width:1px;
classDef app fill:#6db33f,stroke:#333,stroke-width:1px,color:white;
classDef infra fill:#3776ab,stroke:#333,stroke-width:1px,color:white;
classDef common fill:#607d8b,stroke:#333,stroke-width:1px,color:white;
classDef newmod fill:#e91e63,stroke:#333,stroke-width:1px,color:white;
subgraph Driver ["Driver Layer - Entry Points"]
WEB["Web Controllers
Thymeleaf Dashboard"]:::driver API["REST Controllers
ApiV1Controller"]:::driver end subgraph Auth ["Auth Module"] AUTH_CTRL["LocalOAuthController"]:::app AUTH_SVC["UserService
GoogleAuthService"]:::app JWT["JwtUtil
HS256"]:::app end subgraph Contents ["Contents Modules - 10 Domains"] PROMPT["Prompt/Notebook
Markdown Docs"]:::app NEWS["News
Summaries"]:::app SEARCH["Search
Full-text"]:::app SEARCH_SUM["SearchSummary
Queue + History"]:::app WEATHER["Weather"]:::app MARKET["Market"]:::app MAP_SVC["Map
Google Places"]:::app DOC["Document"]:::app RESTAURANT["Restaurant
Location-based"]:::newmod end subgraph DAO ["Data Access Objects"] MD_DAO["MarkdownDocumentDAO"]:::infra NB_DAO["NotebookDAO"]:::infra NEWS_DAO["NewsDAO / DailyReportDAO"]:::infra REST_DAO["RestaurantDAO / MenuDAO"]:::infra SEARCH_DAO["SearchSummaryDAO"]:::infra end subgraph Repo ["JPA Repositories"] MD_REPO["MarkdownDocumentRepository
Cursor Pagination"]:::infra NB_REPO["NotebookRepository"]:::infra USER_REPO["UserRepository"]:::infra REST_REPO["RestaurantRepository
Haversine Query"]:::infra SEARCH_REPO["SearchSummaryRepository
Pagination"]:::infra end subgraph ExtAPI ["External API Clients"] NAVER["NaverNewsApiCaller
WebClient"]:::infra GOOGLE_API["GoogleOAuthApiCaller"]:::infra WEATHER_API["WeatherApiClient
Feign"]:::infra MAP_API["NearByApiClient
Google Places"]:::infra end subgraph Queue ["Redis Queue"] SEARCH_QUEUE["SearchSummaryQueuePublisher
queue:search:request"]:::infra end subgraph Common ["Common - Cross-cutting"] CONFIG["DataSourceConfig
Account + Contents"]:::common ASPECT["OAuthCheckAspect
RequireOAuth"]:::common ADVICE["ResponseWrapperAdvice
ExceptionHandler"]:::common XSS["XssSanitizer"]:::common end subgraph Storage ["Databases"] ACCOUNT_DB[("Account DB
Users, LocalAccounts")] CONTENTS_DB[("Contents DB
Documents, News, Restaurants")] REDIS_DB[("Redis
Search Queue")] end WEB --> PROMPT WEB --> NEWS WEB --> SEARCH WEB --> WEATHER WEB --> MARKET WEB --> RESTAURANT API --> AUTH_CTRL API --> PROMPT API --> NEWS API --> SEARCH API --> SEARCH_SUM API --> WEATHER API --> MARKET API --> MAP_SVC API --> DOC API --> RESTAURANT AUTH_CTRL --> AUTH_SVC AUTH_SVC --> JWT AUTH_SVC --> USER_REPO PROMPT --> MD_DAO PROMPT --> NB_DAO NEWS --> NEWS_DAO SEARCH --> MD_DAO SEARCH --> NEWS_DAO SEARCH_SUM --> SEARCH_DAO SEARCH_SUM --> SEARCH_QUEUE RESTAURANT --> REST_DAO MD_DAO --> MD_REPO NB_DAO --> NB_REPO NEWS_DAO --> MD_REPO REST_DAO --> REST_REPO SEARCH_DAO --> SEARCH_REPO MD_REPO --> ACCOUNT_DB MD_REPO --> CONTENTS_DB NB_REPO --> CONTENTS_DB USER_REPO --> ACCOUNT_DB REST_REPO --> CONTENTS_DB SEARCH_REPO --> CONTENTS_DB SEARCH_QUEUE --> REDIS_DB NEWS --> NAVER AUTH_SVC --> GOOGLE_API WEATHER --> WEATHER_API MAP_SVC --> MAP_API ASPECT -.->|Intercept| API
Thymeleaf Dashboard"]:::driver API["REST Controllers
ApiV1Controller"]:::driver end subgraph Auth ["Auth Module"] AUTH_CTRL["LocalOAuthController"]:::app AUTH_SVC["UserService
GoogleAuthService"]:::app JWT["JwtUtil
HS256"]:::app end subgraph Contents ["Contents Modules - 10 Domains"] PROMPT["Prompt/Notebook
Markdown Docs"]:::app NEWS["News
Summaries"]:::app SEARCH["Search
Full-text"]:::app SEARCH_SUM["SearchSummary
Queue + History"]:::app WEATHER["Weather"]:::app MARKET["Market"]:::app MAP_SVC["Map
Google Places"]:::app DOC["Document"]:::app RESTAURANT["Restaurant
Location-based"]:::newmod end subgraph DAO ["Data Access Objects"] MD_DAO["MarkdownDocumentDAO"]:::infra NB_DAO["NotebookDAO"]:::infra NEWS_DAO["NewsDAO / DailyReportDAO"]:::infra REST_DAO["RestaurantDAO / MenuDAO"]:::infra SEARCH_DAO["SearchSummaryDAO"]:::infra end subgraph Repo ["JPA Repositories"] MD_REPO["MarkdownDocumentRepository
Cursor Pagination"]:::infra NB_REPO["NotebookRepository"]:::infra USER_REPO["UserRepository"]:::infra REST_REPO["RestaurantRepository
Haversine Query"]:::infra SEARCH_REPO["SearchSummaryRepository
Pagination"]:::infra end subgraph ExtAPI ["External API Clients"] NAVER["NaverNewsApiCaller
WebClient"]:::infra GOOGLE_API["GoogleOAuthApiCaller"]:::infra WEATHER_API["WeatherApiClient
Feign"]:::infra MAP_API["NearByApiClient
Google Places"]:::infra end subgraph Queue ["Redis Queue"] SEARCH_QUEUE["SearchSummaryQueuePublisher
queue:search:request"]:::infra end subgraph Common ["Common - Cross-cutting"] CONFIG["DataSourceConfig
Account + Contents"]:::common ASPECT["OAuthCheckAspect
RequireOAuth"]:::common ADVICE["ResponseWrapperAdvice
ExceptionHandler"]:::common XSS["XssSanitizer"]:::common end subgraph Storage ["Databases"] ACCOUNT_DB[("Account DB
Users, LocalAccounts")] CONTENTS_DB[("Contents DB
Documents, News, Restaurants")] REDIS_DB[("Redis
Search Queue")] end WEB --> PROMPT WEB --> NEWS WEB --> SEARCH WEB --> WEATHER WEB --> MARKET WEB --> RESTAURANT API --> AUTH_CTRL API --> PROMPT API --> NEWS API --> SEARCH API --> SEARCH_SUM API --> WEATHER API --> MARKET API --> MAP_SVC API --> DOC API --> RESTAURANT AUTH_CTRL --> AUTH_SVC AUTH_SVC --> JWT AUTH_SVC --> USER_REPO PROMPT --> MD_DAO PROMPT --> NB_DAO NEWS --> NEWS_DAO SEARCH --> MD_DAO SEARCH --> NEWS_DAO SEARCH_SUM --> SEARCH_DAO SEARCH_SUM --> SEARCH_QUEUE RESTAURANT --> REST_DAO MD_DAO --> MD_REPO NB_DAO --> NB_REPO NEWS_DAO --> MD_REPO REST_DAO --> REST_REPO SEARCH_DAO --> SEARCH_REPO MD_REPO --> ACCOUNT_DB MD_REPO --> CONTENTS_DB NB_REPO --> CONTENTS_DB USER_REPO --> ACCOUNT_DB REST_REPO --> CONTENTS_DB SEARCH_REPO --> CONTENTS_DB SEARCH_QUEUE --> REDIS_DB NEWS --> NAVER AUTH_SVC --> GOOGLE_API WEATHER --> WEATHER_API MAP_SVC --> MAP_API ASPECT -.->|Intercept| API
AI Service Integration Guidelines
DP → AG 비동기 처리
TaskQueue Protocol:
1. TaskQueue.submit_task(type, input) → Redis LPUSH tasks:pending
2. Returns task_id (UUID)
3. Store in waiting:summary:* sets
4. Poll get_task_status() every 5 min
5. Timeout: 3 hours (TTL = timeout * 2)
AM 메모리 관리
Model Manager Strategy:
- LLM (~8GB) / Image (~4GB) 중 하나만 메모리 유지
- switch_to_llm(): Image 언로드 → LLM 로드
- switch_to_image(): LLM 언로드 → Image 로드
- Image 생성 후 자동 언로드 (finally block)
- gc.collect() + MLX Metal cache clear
AG LangGraph 파이프라인
Processing Pipeline:
1. Classification: LLM이 입력 타입 분류
2. Clustering: SBERT + K-Means (Silhouette 최적 K)
3. Map: 각 청크 → AM LLM 요약
4. Reduce: 5개씩 병합, 반복 → 최종 요약
5. RAG Ingest: RL 서버로 요약 전송
6. Enhancement: 선택적 이미지 생성
RL RAG 검색
Knowledge Engine:
- Embedding: BAAI/bge-m3 (1024-dim)
- Vector Store: Qdrant (로컬 파일)
- Ingest: Entity → TextNode → Embed
- Query: Cosine Similarity Search
- Snapshot: R2로 1시간 주기 자동 백업
AG API Endpoints
POST /api/v1/generate- 동기 텍스트 처리POST /api/v1/generate/async- 비동기 (task_id 반환)GET /api/v1/task/{id}- 작업 상태 조회POST /api/v1/generate/image- AM 이미지 프록시GET /health- 서버 + AM 상태
AM API Endpoints
POST /api/v1/llm/generate- LLM 텍스트 생성POST /api/v1/llm/stream- LLM SSE 스트리밍GET /api/v1/llm/status- LLM 로드 상태POST /api/v1/image/generate- FLUX.1 이미지 생성GET /health- 서버 + 모델 + Redis 상태GET /queue/status- Queue 대기 요청 수
RL API Endpoints
POST /api/v1/ingest/graph- 그래프 데이터 수집POST /api/v1/query- 벡터 검색 쿼리GET /api/v1/health- 서버 상태GET /api/v1/stats- 엔진 통계
Redis Queue Protocol
tasks:pending- DP→AG 작업 큐 (LPUSH)task:{id}- 작업 상태 Hasham:requests- AG→AM 요청 큐 (LPUSH)am:result:{id}- AM 결과 저장waiting:summary:*- DP 대기 작업queue:search:request- Sb→DP 검색 큐graph:dispatched- AG DISPATCHED Set
DP Execution Schedule (Prefect)
| Flow | Cron Schedule | Frequency | Description |
|---|---|---|---|
| Hourly News Collection | 0 * * * * |
Every hour | Naver News API → news_contents |
| Hourly Market Collection | 0 * * * * |
Every hour | Yahoo Finance (KRW, KOSPI, S&P500, BTC, OIL, GOLD) → market_indices |
| Tech Trend Workflow | 0 8,12,18,22 * * * |
4x daily (KST) | Hacker News Top 10 → tech_contents |
| Hourly YouTube Syuka | 0 * * * * |
Every hour | @syukaworld channel → youtube_summaries |
| Search Summary Flow | */1 * * * * |
Every minute | Process queue:search:request → search_summaries |
| Daily Summary Request | 0 22 * * * |
22:00 UTC daily | Submit news/tech summaries to AG (Async - returns immediately) |
| Summary Monitor Flow | */5 * * * * |
Every 5 minutes | Poll AG, store results, generate daily report, register RAG ingest |
Service Ports & Connections
Service Ports
| Service | Port | Protocol |
|---|---|---|
| Sb sbServer | 443 (HTTPS) | REST + WebSocket |
| AG AI-aGent-AG | 8200 | REST + Redis Queue |
| AM AI-Model-Runner-AM | 8100 | REST + Redis Queue |
| RL Rag-LlamaIndex-RL | 8080 | REST (HTTP) |
| DP Prefect Server UI | 4200 | HTTP |
Data Connections
| Service | Host | Port |
|---|---|---|
| MySQL (Synology) | mylifeis123.synology.me | 3306 |
| Redis (Synology) | mylifeis123.synology.me | 6379 |
| Qdrant (Local) | ./data/qdrant | - |
| Cloudflare R2 | *.r2.cloudflarestorage.com | 443 |