System Ecosystem Overview
통합 AI 서비스를 위한 4개 컴포넌트: UI/API 서빙(Sb), 통합 AI 에이전트(TB = AG+AM+RL 단일 프로세스), 데이터 수집 파이프라인(DP), Discord 봇(DB)
TB는 AG(오케스트레이션), AM(모델 인퍼런스), RL(RAG 벡터 검색)을 하나의 프로세스로 통합하여 직접 함수 호출로 통신
Discord Bot은 TB와 HTTP API로 직접 통신하며, GitHub Webhook 및 Jira 연동 제공
Last Updated: 2026-03-17
graph TD
%% 스타일 정의
classDef spring fill:#6db33f,stroke:#fff,stroke-width:2px,color:white;
classDef python fill:#3776ab,stroke:#fff,stroke-width:2px,color:white;
classDef tb fill:#e91e63,stroke:#fff,stroke-width:2px,color:white;
classDef ag fill:#9c27b0,stroke:#fff,stroke-width:2px,color:white;
classDef am fill:#ff5722,stroke:#fff,stroke-width:2px,color:white;
classDef rag fill:#00796b,stroke:#fff,stroke-width:2px,color:white;
classDef db fill:#e1e1e1,stroke:#333,stroke-width:2px;
classDef external fill:#f9f9f9,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5;
classDef user fill:#ff9900,stroke:#fff,stroke-width:2px,color:white;
classDef notify fill:#5865F2,stroke:#fff,stroke-width:2px,color:white;
classDef discord fill:#5865F2,stroke:#fff,stroke-width:2px,color:white;
%% 외부 요소
User((User
Browser)):::user DiscordUser((Discord
User)):::user ExtData[External Data Sources
Naver News / Yahoo Finance /
Hacker News / YouTube / Google Search]:::external NotifyTargets[Notification Channels
Discord / Notion]:::notify GitHub[GitHub
Webhook Events]:::external Jira[Jira / Atlassian
Issue Tracker]:::external %% 데이터 저장소 subgraph DataStorage [Shared Data Storage] MySQL[(MySQL
Account DB / Contents DB
Traces / Heartbeat)]:::db Redis[(Redis
Task Queue / State / Pub-Sub)]:::db Qdrant[(Qdrant
Vector Store - Local File)]:::db R2[Cloudflare R2
Snapshot Backup]:::db end %% Sb - 메인 서버 subgraph SbServer [Sb - sbServer - Render] direction TB SB[Spring Boot 3.2.4
Java 21]:::spring SB_UI[Web Dashboard
Thymeleaf] SB_API[REST API
Chat / Search / News / Music] SB_AUTH[Auth Layer
JWT + Google OAuth] end %% Discord Bot subgraph DiscordBot [DB - discord-bot] direction TB DB_BOT[Discord Bot
discord.py 2.x]:::discord DB_WEBHOOK[FastAPI Webhook
Port 8001]:::discord DB_ATTEND[Attendance
SQLite] DB_JIRA[Jira Client
Issue Search] end %% TB - 통합 AI 서버 subgraph TBServer [TB - Total-Bot-TB - Single Process] direction TB TB_API[FastAPI Server
Python 3.9+]:::tb subgraph AGModule [AG Module - Orchestration] AG_WORKER[AG Worker
Redis Queue Consumer]:::ag AG_GRAPH[LangGraph Native
Agent Loop + Tools]:::ag AG_HEARTBEAT[Heartbeat Service
APScheduler Cron]:::ag AG_MODULE[Module Loader
MD Prompt Templates]:::ag AG_NOTION[Notion Client
Calendar / Pages]:::ag end subgraph AMModule [AM Module - Model Inference] AM_LLM[MLX LLM
Qwen2.5-14B-4bit]:::am AM_VLM[MLX VLM
Qwen2.5-VL-7B]:::am AM_TTS[mlx-audio
Qwen3-TTS-1.7B]:::am AM_CODER[MLX Coder
Qwen2.5-Coder-14B]:::am AM_MGR[ModelLifecycleManager
Exclusive Load Policy]:::am end subgraph RLModule [RL Module - Knowledge Engine] RL_ENGINE[KnowledgeEngine
LlamaIndex + Qdrant]:::rag RL_EMBED[Embedding Model
BAAI/bge-m3 1024-dim]:::rag RL_SNAP[Snapshot Manager
Cloud Sync 1hr]:::rag end end %% DP - 데이터 수집 파이프라인 subgraph DPServer [DP - data-collect-pipeline] DP[Prefect Pipeline
Python 3.12]:::python DP_COLLECT[Collectors
News/Market/YouTube] DP_SEARCH[Search Flow
1min polling] DP_MONITOR[Summary Monitor
5min polling] end %% 연결 관계: 사용자 → Sb User -->|HTTP/HTTPS| SB SB --> SB_UI SB --> SB_API SB --> SB_AUTH SB_API -->|JPA + MyBatis| MySQL %% Discord 사용자 → Discord Bot DiscordUser -->|Slash Commands| DB_BOT GitHub -->|Webhook POST| DB_WEBHOOK DB_JIRA -->|REST API| Jira %% Discord Bot → TB (HTTP) DB_BOT -->|HTTP API
POST /api/v1/generate| TB_API DB_BOT -->|HTTP API
POST /api/v1/tts/synthesize| TB_API %% Sb → TB 검색 큐 SB_API -->|Redis LPUSH
tasks:pending| Redis %% TB API 서버 TB_API --> AGModule TB_API --> AMModule TB_API --> RLModule %% AG 내부 AG_WORKER -->|RPOP tasks:pending| Redis AG_WORKER --> AG_GRAPH %% AG → AM 직접 함수 호출 AG_GRAPH -->|Direct Function Call| AM_LLM AG_GRAPH -->|Direct Function Call| AM_VLM AG_GRAPH -->|Direct Function Call| AM_CODER %% AM 모델 관리 AM_MGR --> AM_LLM AM_MGR --> AM_VLM AM_MGR --> AM_TTS AM_MGR --> AM_CODER %% AG → RL 직접 함수 호출 AG_GRAPH -->|Direct Function Call| RL_ENGINE RL_ENGINE --> RL_EMBED RL_SNAP --> R2 RL_EMBED --> Qdrant %% AG → Notion AG_NOTION -->|Notion API| NotifyTargets %% Heartbeat AG_HEARTBEAT -->|Scheduled Tasks| AG_GRAPH AG_MODULE -->|Prompt Templates| AG_GRAPH %% TB → 알림 채널 TB_API -->|Webhook / Bot API| NotifyTargets %% DP 데이터 수집 흐름 DP --> DP_COLLECT DP --> DP_SEARCH DP --> DP_MONITOR DP_COLLECT -->|Crawl and API| ExtData DP_COLLECT -->|Store Raw Data| MySQL %% DP ↔ TB 비동기 통신 DP_MONITOR -->|LPUSH tasks:pending| Redis DP_MONITOR -->|Poll task status| Redis %% 결과 저장 AG_WORKER -->|Store Result| Redis AG_WORKER -->|Pub/Sub channel:task_completed| Redis DP_MONITOR -->|Store Summary| MySQL %% TB → MySQL TB_API -->|Traces / Results / Heartbeat| MySQL
Browser)):::user DiscordUser((Discord
User)):::user ExtData[External Data Sources
Naver News / Yahoo Finance /
Hacker News / YouTube / Google Search]:::external NotifyTargets[Notification Channels
Discord / Notion]:::notify GitHub[GitHub
Webhook Events]:::external Jira[Jira / Atlassian
Issue Tracker]:::external %% 데이터 저장소 subgraph DataStorage [Shared Data Storage] MySQL[(MySQL
Account DB / Contents DB
Traces / Heartbeat)]:::db Redis[(Redis
Task Queue / State / Pub-Sub)]:::db Qdrant[(Qdrant
Vector Store - Local File)]:::db R2[Cloudflare R2
Snapshot Backup]:::db end %% Sb - 메인 서버 subgraph SbServer [Sb - sbServer - Render] direction TB SB[Spring Boot 3.2.4
Java 21]:::spring SB_UI[Web Dashboard
Thymeleaf] SB_API[REST API
Chat / Search / News / Music] SB_AUTH[Auth Layer
JWT + Google OAuth] end %% Discord Bot subgraph DiscordBot [DB - discord-bot] direction TB DB_BOT[Discord Bot
discord.py 2.x]:::discord DB_WEBHOOK[FastAPI Webhook
Port 8001]:::discord DB_ATTEND[Attendance
SQLite] DB_JIRA[Jira Client
Issue Search] end %% TB - 통합 AI 서버 subgraph TBServer [TB - Total-Bot-TB - Single Process] direction TB TB_API[FastAPI Server
Python 3.9+]:::tb subgraph AGModule [AG Module - Orchestration] AG_WORKER[AG Worker
Redis Queue Consumer]:::ag AG_GRAPH[LangGraph Native
Agent Loop + Tools]:::ag AG_HEARTBEAT[Heartbeat Service
APScheduler Cron]:::ag AG_MODULE[Module Loader
MD Prompt Templates]:::ag AG_NOTION[Notion Client
Calendar / Pages]:::ag end subgraph AMModule [AM Module - Model Inference] AM_LLM[MLX LLM
Qwen2.5-14B-4bit]:::am AM_VLM[MLX VLM
Qwen2.5-VL-7B]:::am AM_TTS[mlx-audio
Qwen3-TTS-1.7B]:::am AM_CODER[MLX Coder
Qwen2.5-Coder-14B]:::am AM_MGR[ModelLifecycleManager
Exclusive Load Policy]:::am end subgraph RLModule [RL Module - Knowledge Engine] RL_ENGINE[KnowledgeEngine
LlamaIndex + Qdrant]:::rag RL_EMBED[Embedding Model
BAAI/bge-m3 1024-dim]:::rag RL_SNAP[Snapshot Manager
Cloud Sync 1hr]:::rag end end %% DP - 데이터 수집 파이프라인 subgraph DPServer [DP - data-collect-pipeline] DP[Prefect Pipeline
Python 3.12]:::python DP_COLLECT[Collectors
News/Market/YouTube] DP_SEARCH[Search Flow
1min polling] DP_MONITOR[Summary Monitor
5min polling] end %% 연결 관계: 사용자 → Sb User -->|HTTP/HTTPS| SB SB --> SB_UI SB --> SB_API SB --> SB_AUTH SB_API -->|JPA + MyBatis| MySQL %% Discord 사용자 → Discord Bot DiscordUser -->|Slash Commands| DB_BOT GitHub -->|Webhook POST| DB_WEBHOOK DB_JIRA -->|REST API| Jira %% Discord Bot → TB (HTTP) DB_BOT -->|HTTP API
POST /api/v1/generate| TB_API DB_BOT -->|HTTP API
POST /api/v1/tts/synthesize| TB_API %% Sb → TB 검색 큐 SB_API -->|Redis LPUSH
tasks:pending| Redis %% TB API 서버 TB_API --> AGModule TB_API --> AMModule TB_API --> RLModule %% AG 내부 AG_WORKER -->|RPOP tasks:pending| Redis AG_WORKER --> AG_GRAPH %% AG → AM 직접 함수 호출 AG_GRAPH -->|Direct Function Call| AM_LLM AG_GRAPH -->|Direct Function Call| AM_VLM AG_GRAPH -->|Direct Function Call| AM_CODER %% AM 모델 관리 AM_MGR --> AM_LLM AM_MGR --> AM_VLM AM_MGR --> AM_TTS AM_MGR --> AM_CODER %% AG → RL 직접 함수 호출 AG_GRAPH -->|Direct Function Call| RL_ENGINE RL_ENGINE --> RL_EMBED RL_SNAP --> R2 RL_EMBED --> Qdrant %% AG → Notion AG_NOTION -->|Notion API| NotifyTargets %% Heartbeat AG_HEARTBEAT -->|Scheduled Tasks| AG_GRAPH AG_MODULE -->|Prompt Templates| AG_GRAPH %% TB → 알림 채널 TB_API -->|Webhook / Bot API| NotifyTargets %% DP 데이터 수집 흐름 DP --> DP_COLLECT DP --> DP_SEARCH DP --> DP_MONITOR DP_COLLECT -->|Crawl and API| ExtData DP_COLLECT -->|Store Raw Data| MySQL %% DP ↔ TB 비동기 통신 DP_MONITOR -->|LPUSH tasks:pending| Redis DP_MONITOR -->|Poll task status| Redis %% 결과 저장 AG_WORKER -->|Store Result| Redis AG_WORKER -->|Pub/Sub channel:task_completed| Redis DP_MONITOR -->|Store Summary| MySQL %% TB → MySQL TB_API -->|Traces / Results / Heartbeat| MySQL
Component Roles & Tech Stack
| Component | Project | Role & Responsibilities | Tech Stack |
|---|---|---|---|
| Sb | sbServer |
통합 백엔드 & UI 서버 - Chat 중심 허브: TB와 Redis 큐로 통신하는 메시징 중앙 허브 - 콘텐츠 모듈: Chat, Search, News, Prompt/Notebook, Market, Music, Voice, Heartbeat, Viewer - Restaurant 모듈: Haversine 거리 계산, 위치 기반 추천 - 이중 데이터소스: Account DB + Contents DB - 인증: JWT Access/Refresh Token + Google OAuth - WebSocket: 음악 플레이어 실시간 제어 - 레이어드 아키텍처: Driver → Application → Infra |
Java 21, Spring Boot 3.2.4 JPA + MyBatis, MySQL Thymeleaf, OpenFeign Redis (6 Queue/Channel 패턴) WebSocket, JDA 5.6.1 Swagger/OpenAPI |
| DB | discord-bot |
Discord 알림 봇 & Webhook 서버 - GitHub Webhook: Push/PR/Issue/Release 이벤트 → Discord 채널 브로드캐스트 - 출퇴근 관리: 버튼 기반 체크인/아웃, A/B/C 등급 분류, 반차/반반차 지원 - TB AI 연동: /tb ask (AI 질의), /tb tts (음성 합성), /tb status (상태 확인) - Jira 연동: 이슈 검색, 상세 조회, 담당 이슈 필터링 - PR 리뷰어 멘션: GitHub → Discord 사용자 매핑 - 멀티 서버 지원: 동적 채널 등록 시스템 |
Python 3.12+, discord.py 2.x FastAPI + Uvicorn (Webhook) aiohttp (HTTP Client) SQLite (출퇴근 기록) YAML (채널/사용자 설정) Docker Compose (2 인스턴스) |
| TB | Total-Bot-TB- |
통합 AI 에이전트 서버 (Single Process) - AG+AM+RL 통합: 3개 모듈을 단일 프로세스로 병합, 직접 함수 호출로 통신 - AG 모듈: LangGraph Native Agent Loop (agent_node ↔ tool_node), 20개 내장 도구 - AM 모듈: MLX LLM (Qwen2.5-14B), VLM (Qwen2.5-VL-7B), Coder (Qwen2.5-Coder-14B), TTS (Qwen3-TTS) - RL 모듈: LlamaIndex + Qdrant 벡터 검색, BAAI/bge-m3 임베딩 - ModelLifecycleManager: Exclusive Load (한 번에 하나의 모델만 로드) - 51개 API 엔드포인트: 7개 라우터 (system, tasks, heartbeat, modules, tts, coder, preferences) - 알림/연동: Discord Webhook/Bot, Notion (Calendar/Pages/Search) - 뉴스/테크 수집 기능 내장 (DP에서 이관) |
Python 3.9+, FastAPI, Uvicorn LangGraph (Native Agent Loop) mlx-lm (Qwen2.5-14B-4bit) Qwen2.5-VL-7B, Qwen2.5-Coder-14B mlx-audio (Qwen3-TTS-1.7B) LlamaIndex, Qdrant (Local File) sentence-transformers (BAAI/bge-m3) APScheduler, Playwright, Scrapling Redis, MySQL, boto3 (R2/S3) |
| DP | data-collect-pipeline |
데이터 수집 파이프라인 - Prefect Flow: News, Market, Tech, YouTube, Search(1min), Summary Monitor - 스케줄: 매시(뉴스/마켓/유튜브), 4x/일(테크), 1분(검색), 5분(모니터링) - Collectors: Naver News API, Yahoo Finance, Hacker News, YouTube Transcript, Google Custom Search - 일부 요약 기능 TB로 이관: 뉴스/테크 요약 생성은 TB의 도구로 처리 |
Python 3.12, Prefect MySQL, Redis (Queue + Task State) yfinance, trafilatura, scrapetube youtube-transcript-api, google-api |
Data Flow & Processing Pipeline
sequenceDiagram
autonumber
participant User as User (Browser)
participant DUser as Discord User
participant Sb as Sb (sbServer)
participant DB as MySQL
participant Redis as Redis
participant DP as DP (Pipeline)
participant DBot as Discord Bot
participant TB as TB (Total-Bot)
participant AG as AG Module
participant AM as AM Module
participant RL as RL Module
participant Ext as External Sources
participant Notify as Notifications
Note over DP,Ext: 1. Data Collection (Hourly: News/Market/YouTube, 4x Daily: Tech, 1min: Search)
DP->>Ext: Crawl via Collectors
Ext-->>DP: Raw Data (JSON)
DP->>DB: Store (news_contents, market_indices, tech_contents)
Note over Sb,TB: 2. User Chat Request (via sbServer)
User->>Sb: POST /api/v1/chat/send
Sb->>DB: Save TaskResultEntity
Sb->>Redis: LPUSH tasks:pending (task_id)
Sb-->>User: {task_id, status: pending}
Note over DBot,TB: 3. Discord Bot Request (via HTTP)
DUser->>DBot: /tb ask "question"
DBot->>TB: POST /api/v1/generate
TB-->>DBot: {task_id}
DBot->>TB: GET /api/v1/task/{id} (polling 2s)
Note over TB,RL: 4. TB Unified Processing (Single Process)
AG->>Redis: RPOP tasks:pending
AG->>AG: route_entry - Agent or Simple
AG->>AG: agent_node - LLM ReAct Reasoning
AG->>AM: Direct Call - ensure_loaded + generate
Note right of AM: Exclusive Load Policy
Only ONE model at a time AM-->>AG: LLM Response + Tool Selection AG->>AG: tool_node - Execute Selected Tool AG->>RL: Direct Call - query_knowledge RL-->>AG: Vector Search Results AG->>AG: report_node - Final Synthesis AG->>RL: rag_ingest_node - Store to Qdrant AG->>Redis: Update task status to COMPLETED AG->>Redis: PUBLISH channel:task_completed Note over DBot,Notify: 5. Discord Bot Result Delivery TB-->>DBot: {status: completed, result: "..."} DBot-->>DUser: Discord Reply + TTS Audio Note over TB,Notify: 6. Heartbeat Proactive Tasks (APScheduler) AG->>AG: Heartbeat slot triggers (cron) AG->>AG: Execute via LangGraph AG->>AM: Direct Call - LLM Inference AG->>Notify: Deliver to Discord / Notion Note over User,Sb: 7. User Dashboard Access User->>Sb: GET /api/v1/chat/result/{taskId} Sb->>Redis: HGETALL task:{taskId} Redis-->>Sb: Task Result Sb-->>User: ChatResponse (with result)
Only ONE model at a time AM-->>AG: LLM Response + Tool Selection AG->>AG: tool_node - Execute Selected Tool AG->>RL: Direct Call - query_knowledge RL-->>AG: Vector Search Results AG->>AG: report_node - Final Synthesis AG->>RL: rag_ingest_node - Store to Qdrant AG->>Redis: Update task status to COMPLETED AG->>Redis: PUBLISH channel:task_completed Note over DBot,Notify: 5. Discord Bot Result Delivery TB-->>DBot: {status: completed, result: "..."} DBot-->>DUser: Discord Reply + TTS Audio Note over TB,Notify: 6. Heartbeat Proactive Tasks (APScheduler) AG->>AG: Heartbeat slot triggers (cron) AG->>AG: Execute via LangGraph AG->>AM: Direct Call - LLM Inference AG->>Notify: Deliver to Discord / Notion Note over User,Sb: 7. User Dashboard Access User->>Sb: GET /api/v1/chat/result/{taskId} Sb->>Redis: HGETALL task:{taskId} Redis-->>Sb: Task Result Sb-->>User: ChatResponse (with result)
TB (Total-Bot-TB-) Unified Architecture - Single Process
AG + AM + RL을 단일 프로세스로 통합. 모듈 간 직접 함수 호출로 HTTP 오버헤드 제거. LangGraph Native Agent Loop.
graph TB
classDef api fill:#ff9900,stroke:#333,stroke-width:1px,color:white;
classDef ag fill:#9c27b0,stroke:#333,stroke-width:1px,color:white;
classDef am fill:#ff5722,stroke:#333,stroke-width:1px,color:white;
classDef rl fill:#00796b,stroke:#333,stroke-width:1px,color:white;
classDef common fill:#607d8b,stroke:#333,stroke-width:1px,color:white;
classDef storage fill:#e1e1e1,stroke:#333,stroke-width:2px;
classDef notify fill:#5865F2,stroke:#333,stroke-width:1px,color:white;
subgraph MainProcess [main.py - FastAPI Lifespan]
direction TB
API[FastAPI Server
Uvicorn]:::api subgraph Routers [API Routers - 7 Routers / 51 Endpoints] R_TASKS[Tasks Router
generate / ingest / history]:::api R_SYSTEM[System Router
health / status / models / settings]:::api R_HEARTBEAT[Heartbeat Router
slots / execute / audio]:::api R_TTS[TTS Router
synthesize / voices / clone]:::api R_CODER[Coder Router
review / analyze]:::api R_MODULES[Modules Router
CRUD / run / tools]:::api R_PREFS[Preferences Router
user settings]:::api end subgraph AGMod [AG Module - Orchestration Layer] AG_WORKER[AG Worker
Async Loop: RPOP tasks:pending]:::ag AG_GRAPH[LangGraph Native Agent Loop
agent_node / tool_node / report / ingest]:::ag AG_TOOLS[20 Built-in Tools
Search / Analysis / Knowledge / Vision / Code / Notion]:::ag AG_HEARTBEAT[HeartbeatService
APScheduler Cron Jobs]:::ag AG_MODULE[ModuleLoader
MD Prompt Templates]:::ag AG_NOTIFY[NotificationRouter
Multi-Channel Delivery]:::ag end subgraph AMMod [AM Module - Model Inference Layer] AM_MLM[ModelLifecycleManager
Exclusive Load Policy]:::am AM_LLM[LanguageModelAdapter
MLX Qwen2.5-14B-4bit / 8GB]:::am AM_VLM[VisionModelAdapter
Qwen2.5-VL-7B / 5GB]:::am AM_CODER[CoderModelAdapter
Qwen2.5-Coder-14B / 9GB]:::am AM_TTS[TTSAdapter
Qwen3-TTS-1.7B / 3.4GB]:::am end subgraph RLMod [RL Module - Knowledge Engine Layer] RL_ENGINE[KnowledgeEngine
Singleton / Async Init]:::rl RL_EMBED[HuggingFace Embedding
BAAI/bge-m3 1024-dim]:::rl RL_INDEX[LlamaIndex
SentenceSplitter 512/50]:::rl RL_SNAP[SnapshotManager
1hr Periodic Upload]:::rl end subgraph CommonMod [Common - Shared Infrastructure] MLX_LOCK[MLX Lock
Thread-Safe Access]:::common MODEL_REG[ModelRegistry
Adapter Protocol]:::common SCHEMAS[Pydantic Schemas
AM / RL / Heartbeat]:::common end end subgraph External [External Storage] MYSQL[(MySQL
Traces / Results / Heartbeat / Modules)]:::storage REDIS[(Redis
tasks:pending / task state / pub-sub)]:::storage QDRANT[(Qdrant Local
./rl/data/qdrant)]:::storage R2[Cloudflare R2
Snapshots]:::storage end subgraph NotifyChannels [Notification Channels] DISCORD_WH[Discord
Webhook]:::notify DISCORD_BOT[Discord
Bot API]:::notify NOTION[Notion
Calendar / Pages]:::notify end %% API → Routers API --> Routers %% Routers → Modules R_TASKS --> AG_WORKER R_SYSTEM --> AM_MLM R_HEARTBEAT --> AG_HEARTBEAT R_TTS --> AM_TTS R_CODER --> AM_CODER R_MODULES --> AG_MODULE %% AG 내부 AG_WORKER --> AG_GRAPH AG_GRAPH --> AG_TOOLS AG_HEARTBEAT --> AG_GRAPH AG_MODULE --> AG_GRAPH AG_GRAPH --> AG_NOTIFY %% AG → AM 직접 호출 AG_GRAPH -->|Direct Call| AM_LLM AG_GRAPH -->|Direct Call| AM_VLM AG_GRAPH -->|Direct Call| AM_CODER %% AM 모델 관리 AM_MLM --> AM_LLM AM_MLM --> AM_VLM AM_MLM --> AM_CODER AM_MLM --> AM_TTS AM_MLM --> MLX_LOCK %% AG → RL 직접 호출 AG_GRAPH -->|Direct Call| RL_ENGINE RL_ENGINE --> RL_EMBED RL_ENGINE --> RL_INDEX RL_INDEX --> QDRANT RL_SNAP --> R2 %% 외부 저장소 AG_WORKER --> REDIS AG_GRAPH --> MYSQL AG_NOTIFY --> DISCORD_WH AG_NOTIFY --> DISCORD_BOT AG_NOTIFY --> NOTION
Uvicorn]:::api subgraph Routers [API Routers - 7 Routers / 51 Endpoints] R_TASKS[Tasks Router
generate / ingest / history]:::api R_SYSTEM[System Router
health / status / models / settings]:::api R_HEARTBEAT[Heartbeat Router
slots / execute / audio]:::api R_TTS[TTS Router
synthesize / voices / clone]:::api R_CODER[Coder Router
review / analyze]:::api R_MODULES[Modules Router
CRUD / run / tools]:::api R_PREFS[Preferences Router
user settings]:::api end subgraph AGMod [AG Module - Orchestration Layer] AG_WORKER[AG Worker
Async Loop: RPOP tasks:pending]:::ag AG_GRAPH[LangGraph Native Agent Loop
agent_node / tool_node / report / ingest]:::ag AG_TOOLS[20 Built-in Tools
Search / Analysis / Knowledge / Vision / Code / Notion]:::ag AG_HEARTBEAT[HeartbeatService
APScheduler Cron Jobs]:::ag AG_MODULE[ModuleLoader
MD Prompt Templates]:::ag AG_NOTIFY[NotificationRouter
Multi-Channel Delivery]:::ag end subgraph AMMod [AM Module - Model Inference Layer] AM_MLM[ModelLifecycleManager
Exclusive Load Policy]:::am AM_LLM[LanguageModelAdapter
MLX Qwen2.5-14B-4bit / 8GB]:::am AM_VLM[VisionModelAdapter
Qwen2.5-VL-7B / 5GB]:::am AM_CODER[CoderModelAdapter
Qwen2.5-Coder-14B / 9GB]:::am AM_TTS[TTSAdapter
Qwen3-TTS-1.7B / 3.4GB]:::am end subgraph RLMod [RL Module - Knowledge Engine Layer] RL_ENGINE[KnowledgeEngine
Singleton / Async Init]:::rl RL_EMBED[HuggingFace Embedding
BAAI/bge-m3 1024-dim]:::rl RL_INDEX[LlamaIndex
SentenceSplitter 512/50]:::rl RL_SNAP[SnapshotManager
1hr Periodic Upload]:::rl end subgraph CommonMod [Common - Shared Infrastructure] MLX_LOCK[MLX Lock
Thread-Safe Access]:::common MODEL_REG[ModelRegistry
Adapter Protocol]:::common SCHEMAS[Pydantic Schemas
AM / RL / Heartbeat]:::common end end subgraph External [External Storage] MYSQL[(MySQL
Traces / Results / Heartbeat / Modules)]:::storage REDIS[(Redis
tasks:pending / task state / pub-sub)]:::storage QDRANT[(Qdrant Local
./rl/data/qdrant)]:::storage R2[Cloudflare R2
Snapshots]:::storage end subgraph NotifyChannels [Notification Channels] DISCORD_WH[Discord
Webhook]:::notify DISCORD_BOT[Discord
Bot API]:::notify NOTION[Notion
Calendar / Pages]:::notify end %% API → Routers API --> Routers %% Routers → Modules R_TASKS --> AG_WORKER R_SYSTEM --> AM_MLM R_HEARTBEAT --> AG_HEARTBEAT R_TTS --> AM_TTS R_CODER --> AM_CODER R_MODULES --> AG_MODULE %% AG 내부 AG_WORKER --> AG_GRAPH AG_GRAPH --> AG_TOOLS AG_HEARTBEAT --> AG_GRAPH AG_MODULE --> AG_GRAPH AG_GRAPH --> AG_NOTIFY %% AG → AM 직접 호출 AG_GRAPH -->|Direct Call| AM_LLM AG_GRAPH -->|Direct Call| AM_VLM AG_GRAPH -->|Direct Call| AM_CODER %% AM 모델 관리 AM_MLM --> AM_LLM AM_MLM --> AM_VLM AM_MLM --> AM_CODER AM_MLM --> AM_TTS AM_MLM --> MLX_LOCK %% AG → RL 직접 호출 AG_GRAPH -->|Direct Call| RL_ENGINE RL_ENGINE --> RL_EMBED RL_ENGINE --> RL_INDEX RL_INDEX --> QDRANT RL_SNAP --> R2 %% 외부 저장소 AG_WORKER --> REDIS AG_GRAPH --> MYSQL AG_NOTIFY --> DISCORD_WH AG_NOTIFY --> DISCORD_BOT AG_NOTIFY --> NOTION
TB LangGraph Pipeline - Native Agent Loop
LangGraph Native Agent Loop: agent_node(LLM 추론) ↔ tool_node(도구 실행) 반복 → report_node(최종 합성) → rag_ingest_node(벡터 저장)
graph TB
classDef entry fill:#ff9900,stroke:#333,stroke-width:1px,color:white;
classDef node fill:#9c27b0,stroke:#333,stroke-width:1px,color:white;
classDef tool fill:#4caf50,stroke:#333,stroke-width:1px,color:white;
classDef rag fill:#00796b,stroke:#333,stroke-width:1px,color:white;
START((START)):::entry
ROUTE[route_entry
Agent or Simple Path]:::node subgraph AgentLoop [Native Agent Loop - Orchestrator Path] AGENT[agent_node
LLM ReAct Reasoning
Tool Selection via JSON]:::node TOOL[tool_node
Execute Selected Tool
from 20 Built-in Tools]:::node CONTINUE{should_continue
tools or finish}:::node end subgraph ToolRegistry [20 Built-in Tools - 8 Categories] T_SEARCH[Search
query_google
search_naver_news
web_fetch / deep_web_fetch]:::tool T_ANALYSIS[Analysis
summarize
fact_check
query_weather]:::tool T_KNOWLEDGE[Knowledge
query_knowledge
store_knowledge]:::tool T_VISION[Vision
ocr]:::tool T_YOUTUBE[YouTube
youtube_info]:::tool T_CODER[Code
code_review
log_analysis]:::tool T_SHOPPING[Shopping
search_shopping]:::tool T_NOTION[Notion
query_notion_calendar
create_notion_page
search_notion
read_notion_page]:::tool T_COLLECT[Collect
collect_daily_news
collect_top_stories]:::tool end REPORT[report_node
Final Report Synthesis]:::node RAG[rag_ingest_node
Store to Qdrant Vector DB]:::rag SIMPLE[simple_node
Direct LLM Response]:::node END_N((END)):::entry %% Entry routing START --> ROUTE ROUTE -->|Orchestrator / Complex| AGENT ROUTE -->|Simple / Direct| SIMPLE %% Agent loop AGENT --> CONTINUE CONTINUE -->|tools - action selected| TOOL TOOL --> AGENT CONTINUE -->|finish - no more tools| REPORT %% Tool connections TOOL --> ToolRegistry ToolRegistry --> TOOL %% Post-processing REPORT --> RAG RAG --> END_N SIMPLE --> END_N
Agent or Simple Path]:::node subgraph AgentLoop [Native Agent Loop - Orchestrator Path] AGENT[agent_node
LLM ReAct Reasoning
Tool Selection via JSON]:::node TOOL[tool_node
Execute Selected Tool
from 20 Built-in Tools]:::node CONTINUE{should_continue
tools or finish}:::node end subgraph ToolRegistry [20 Built-in Tools - 8 Categories] T_SEARCH[Search
query_google
search_naver_news
web_fetch / deep_web_fetch]:::tool T_ANALYSIS[Analysis
summarize
fact_check
query_weather]:::tool T_KNOWLEDGE[Knowledge
query_knowledge
store_knowledge]:::tool T_VISION[Vision
ocr]:::tool T_YOUTUBE[YouTube
youtube_info]:::tool T_CODER[Code
code_review
log_analysis]:::tool T_SHOPPING[Shopping
search_shopping]:::tool T_NOTION[Notion
query_notion_calendar
create_notion_page
search_notion
read_notion_page]:::tool T_COLLECT[Collect
collect_daily_news
collect_top_stories]:::tool end REPORT[report_node
Final Report Synthesis]:::node RAG[rag_ingest_node
Store to Qdrant Vector DB]:::rag SIMPLE[simple_node
Direct LLM Response]:::node END_N((END)):::entry %% Entry routing START --> ROUTE ROUTE -->|Orchestrator / Complex| AGENT ROUTE -->|Simple / Direct| SIMPLE %% Agent loop AGENT --> CONTINUE CONTINUE -->|tools - action selected| TOOL TOOL --> AGENT CONTINUE -->|finish - no more tools| REPORT %% Tool connections TOOL --> ToolRegistry ToolRegistry --> TOOL %% Post-processing REPORT --> RAG RAG --> END_N SIMPLE --> END_N
AM - ModelLifecycleManager & Model Adapters
Exclusive Load 정책: 한 번에 하나의 모델만 로드. 4개 모델 중 요청 시 자동 전환 (LRU 기반 해제).
graph TB
classDef mgr fill:#ff5722,stroke:#333,stroke-width:1px,color:white;
classDef model fill:#9c27b0,stroke:#333,stroke-width:1px,color:white;
classDef lock fill:#607d8b,stroke:#333,stroke-width:1px,color:white;
classDef state fill:#4caf50,stroke:#333,stroke-width:1px,color:white;
subgraph MLM [ModelLifecycleManager - Exclusive Load]
MGR[Manager Core
Exclusive Policy: 1 Model at a time]:::mgr ENSURE[ensure_loaded
Unload Current → Load Requested]:::mgr RELEASE[release_model
Unload + gc.collect]:::mgr STATUS[get_status
All Model States]:::mgr SWITCH[switch_to_llm / tts / vlm / coder
Convenience Methods]:::mgr end subgraph Registry [ModelRegistry] REG[Adapter Registration
name to ModelAdapter]:::lock PROTOCOL[ModelAdapter Protocol
load / unload / is_loaded / memory_mb]:::lock end subgraph Models [Registered Model Adapters - 4 Models] LLM[LanguageModelAdapter
Qwen2.5-14B-Instruct-4bit
~8GB / Priority: HIGH]:::model VLM[VisionModelAdapter
Qwen2.5-VL-7B-Instruct
~5GB / Priority: NORMAL]:::model CODER[CoderModelAdapter
Qwen2.5-Coder-14B
~9GB / Priority: NORMAL]:::model TTS[TTSAdapter
Qwen3-TTS-12Hz-1.7B-bf16
~3.4GB / Priority: LOW]:::model end subgraph Lifecycle [Model States] UNLOADED[UNLOADED
No Memory]:::state LOADING[LOADING
Initializing]:::state LOADED[LOADED
Ready for Inference]:::state end subgraph Safety [Thread Safety] MLX_LOCK[MLX Lock
Global Thread Lock]:::lock GC[gc.collect
Metal Cache Clear]:::lock end MGR --> REG REG --> PROTOCOL PROTOCOL --> LLM PROTOCOL --> VLM PROTOCOL --> CODER PROTOCOL --> TTS ENSURE -->|1. Unload current model| RELEASE ENSURE -->|2. Load requested model| LOADING SWITCH --> ENSURE UNLOADED -->|ensure_loaded| LOADING LOADING --> LOADED LOADED -->|release_model| UNLOADED RELEASE --> GC LOADING --> MLX_LOCK
Exclusive Policy: 1 Model at a time]:::mgr ENSURE[ensure_loaded
Unload Current → Load Requested]:::mgr RELEASE[release_model
Unload + gc.collect]:::mgr STATUS[get_status
All Model States]:::mgr SWITCH[switch_to_llm / tts / vlm / coder
Convenience Methods]:::mgr end subgraph Registry [ModelRegistry] REG[Adapter Registration
name to ModelAdapter]:::lock PROTOCOL[ModelAdapter Protocol
load / unload / is_loaded / memory_mb]:::lock end subgraph Models [Registered Model Adapters - 4 Models] LLM[LanguageModelAdapter
Qwen2.5-14B-Instruct-4bit
~8GB / Priority: HIGH]:::model VLM[VisionModelAdapter
Qwen2.5-VL-7B-Instruct
~5GB / Priority: NORMAL]:::model CODER[CoderModelAdapter
Qwen2.5-Coder-14B
~9GB / Priority: NORMAL]:::model TTS[TTSAdapter
Qwen3-TTS-12Hz-1.7B-bf16
~3.4GB / Priority: LOW]:::model end subgraph Lifecycle [Model States] UNLOADED[UNLOADED
No Memory]:::state LOADING[LOADING
Initializing]:::state LOADED[LOADED
Ready for Inference]:::state end subgraph Safety [Thread Safety] MLX_LOCK[MLX Lock
Global Thread Lock]:::lock GC[gc.collect
Metal Cache Clear]:::lock end MGR --> REG REG --> PROTOCOL PROTOCOL --> LLM PROTOCOL --> VLM PROTOCOL --> CODER PROTOCOL --> TTS ENSURE -->|1. Unload current model| RELEASE ENSURE -->|2. Load requested model| LOADING SWITCH --> ENSURE UNLOADED -->|ensure_loaded| LOADING LOADING --> LOADED LOADED -->|release_model| UNLOADED RELEASE --> GC LOADING --> MLX_LOCK
RL Module - Knowledge Engine Internal Architecture
graph TB
classDef engine fill:#00796b,stroke:#333,stroke-width:1px,color:white;
classDef storage fill:#e1e1e1,stroke:#333,stroke-width:2px;
classDef cloud fill:#2196f3,stroke:#333,stroke-width:1px,color:white;
classDef caller fill:#9c27b0,stroke:#333,stroke-width:1px,color:white;
subgraph Callers [Direct Function Callers - AG Module]
AG_INGEST[rag_ingest_node
Graph Ingestion]:::caller AG_QUERY[Tool: query_knowledge
Similarity Search]:::caller AG_STORE[Tool: store_knowledge
Manual Ingestion]:::caller end subgraph Engine [KnowledgeEngine - Singleton / Async] INIT[initialize
Download Snapshot + Load Models]:::engine INGEST_FN[ingest_graph
Entity to TextNode]:::engine QUERY_FN[query
Vector Similarity Search]:::engine CACHE[Entity Cache
In-Memory Dictionary]:::engine SHUTDOWN[shutdown
Final Upload + Cleanup]:::engine end subgraph Processing [Document Processing] SPLITTER[SentenceSplitter
512 tokens / 50 overlap]:::engine EMBED[BAAI/bge-m3
1024-dim Embedding]:::engine UUID[UUID5 Generator
Deterministic IDs]:::engine end subgraph VectorStore [Qdrant Vector Store - Local File] QDRANT_LOCAL[(Qdrant Local
./rl/data/qdrant)]:::storage COLLECTION[knowledge_vectors
Collection]:::storage end subgraph SnapshotMgr [Snapshot Manager] DOWNLOAD[download_snapshot
Startup Restore from R2]:::cloud UPLOAD[periodic_upload
1 Hour Interval]:::cloud FINAL[final_upload
Shutdown Save to R2]:::cloud end subgraph CloudStorage [Cloudflare R2 - S3 Compatible] R2[(R2 Bucket
rag-snapshot.tar.gz)]:::storage end AG_INGEST -->|Direct Call| INGEST_FN AG_QUERY -->|Direct Call| QUERY_FN AG_STORE -->|Direct Call| INGEST_FN INIT --> DOWNLOAD DOWNLOAD --> R2 SHUTDOWN --> FINAL FINAL --> R2 UPLOAD --> R2 INGEST_FN --> SPLITTER SPLITTER --> EMBED EMBED --> UUID UUID --> QDRANT_LOCAL QDRANT_LOCAL --> COLLECTION QUERY_FN --> EMBED EMBED --> QDRANT_LOCAL INGEST_FN --> CACHE QUERY_FN --> CACHE
Graph Ingestion]:::caller AG_QUERY[Tool: query_knowledge
Similarity Search]:::caller AG_STORE[Tool: store_knowledge
Manual Ingestion]:::caller end subgraph Engine [KnowledgeEngine - Singleton / Async] INIT[initialize
Download Snapshot + Load Models]:::engine INGEST_FN[ingest_graph
Entity to TextNode]:::engine QUERY_FN[query
Vector Similarity Search]:::engine CACHE[Entity Cache
In-Memory Dictionary]:::engine SHUTDOWN[shutdown
Final Upload + Cleanup]:::engine end subgraph Processing [Document Processing] SPLITTER[SentenceSplitter
512 tokens / 50 overlap]:::engine EMBED[BAAI/bge-m3
1024-dim Embedding]:::engine UUID[UUID5 Generator
Deterministic IDs]:::engine end subgraph VectorStore [Qdrant Vector Store - Local File] QDRANT_LOCAL[(Qdrant Local
./rl/data/qdrant)]:::storage COLLECTION[knowledge_vectors
Collection]:::storage end subgraph SnapshotMgr [Snapshot Manager] DOWNLOAD[download_snapshot
Startup Restore from R2]:::cloud UPLOAD[periodic_upload
1 Hour Interval]:::cloud FINAL[final_upload
Shutdown Save to R2]:::cloud end subgraph CloudStorage [Cloudflare R2 - S3 Compatible] R2[(R2 Bucket
rag-snapshot.tar.gz)]:::storage end AG_INGEST -->|Direct Call| INGEST_FN AG_QUERY -->|Direct Call| QUERY_FN AG_STORE -->|Direct Call| INGEST_FN INIT --> DOWNLOAD DOWNLOAD --> R2 SHUTDOWN --> FINAL FINAL --> R2 UPLOAD --> R2 INGEST_FN --> SPLITTER SPLITTER --> EMBED EMBED --> UUID UUID --> QDRANT_LOCAL QDRANT_LOCAL --> COLLECTION QUERY_FN --> EMBED EMBED --> QDRANT_LOCAL INGEST_FN --> CACHE QUERY_FN --> CACHE
DB (discord-bot) Internal Architecture
Discord Bot + FastAPI Webhook 서버가 동시 실행. TB와 HTTP API로 직접 통신. GitHub Webhook 이벤트 멀티서버 브로드캐스트.
graph TB
classDef bot fill:#5865F2,stroke:#333,stroke-width:1px,color:white;
classDef webhook fill:#ff9900,stroke:#333,stroke-width:1px,color:white;
classDef service fill:#4caf50,stroke:#333,stroke-width:1px,color:white;
classDef storage fill:#e1e1e1,stroke:#333,stroke-width:2px;
classDef external fill:#f9f9f9,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5;
subgraph DiscordBotApp [main.py - Concurrent Execution]
direction TB
subgraph BotCore [Discord Bot - discord.py 2.x]
BOT[DiscordBot
Slash Commands]:::bot CMD_TB["tb Commands
ask, tts, status"]:::bot CMD_ATTEND["attendance
Check-in, Check-out"]:::bot CMD_CHANNEL["channel set
Dynamic Registration"]:::bot CMD_GITHUB["github link
User Mapping"]:::bot CMD_JIRA["jira Commands
search, issue, my"]:::bot end subgraph WebhookServer [FastAPI Webhook Server - Port 8001] WH_API[FastAPI Server]:::webhook WH_GITHUB[POST /webhook/github
GitHub Event Receiver]:::webhook WH_SEND[POST /webhook/send
Generic Message API]:::webhook WH_HEALTH[GET /health]:::webhook end subgraph Services [Core Services] MESSENGER[Messenger
Discord Embed Sender]:::service CHANNEL_REG[ChannelRegistry
Multi-Server Channels]:::service GH_USER_REG[GitHubUserRegistry
GitHub to Discord Mapping]:::service ATTEND_SVC[AttendanceService
Grade Classification]:::service TB_CLIENT[TB API Client
Task Submit + Poll]:::service JIRA_CLIENT[Jira Client
REST API + JQL]:::service end subgraph Parsers [Event Processing] GH_PARSER[GitHubParser
9+ Event Types]:::service GH_FORMATTER[GitHubFormatter
Discord Embed Builder]:::service JIRA_FORMATTER[JiraFormatter
Issue Embed Builder]:::service end end subgraph LocalStorage [Local Storage] SQLITE[(SQLite
attendance.db)]:::storage YAML_CH[channels.yaml
Server Config]:::storage YAML_USERS[users.yaml
User Mapping]:::storage end subgraph ExternalSvc [External Services] TB_SVC[TB Server
HTTP :8000]:::external GITHUB_SVC[GitHub
Webhook Events]:::external JIRA_SVC[Jira/Atlassian
REST API]:::external DISCORD_SVC[Discord API
Bot Gateway]:::external end BOT --> CMD_TB BOT --> CMD_ATTEND BOT --> CMD_CHANNEL BOT --> CMD_GITHUB BOT --> CMD_JIRA CMD_TB --> TB_CLIENT CMD_ATTEND --> ATTEND_SVC CMD_CHANNEL --> CHANNEL_REG CMD_GITHUB --> GH_USER_REG CMD_JIRA --> JIRA_CLIENT WH_API --> WH_GITHUB WH_API --> WH_SEND WH_GITHUB --> GH_PARSER GH_PARSER --> GH_FORMATTER GH_FORMATTER --> MESSENGER MESSENGER --> CHANNEL_REG TB_CLIENT -->|POST /api/v1/generate| TB_SVC TB_CLIENT -->|GET /api/v1/task/id| TB_SVC TB_CLIENT -->|POST /api/v1/tts/synthesize| TB_SVC JIRA_CLIENT --> JIRA_SVC GITHUB_SVC --> WH_GITHUB BOT --> DISCORD_SVC ATTEND_SVC --> SQLITE CHANNEL_REG --> YAML_CH GH_USER_REG --> YAML_USERS
Slash Commands]:::bot CMD_TB["tb Commands
ask, tts, status"]:::bot CMD_ATTEND["attendance
Check-in, Check-out"]:::bot CMD_CHANNEL["channel set
Dynamic Registration"]:::bot CMD_GITHUB["github link
User Mapping"]:::bot CMD_JIRA["jira Commands
search, issue, my"]:::bot end subgraph WebhookServer [FastAPI Webhook Server - Port 8001] WH_API[FastAPI Server]:::webhook WH_GITHUB[POST /webhook/github
GitHub Event Receiver]:::webhook WH_SEND[POST /webhook/send
Generic Message API]:::webhook WH_HEALTH[GET /health]:::webhook end subgraph Services [Core Services] MESSENGER[Messenger
Discord Embed Sender]:::service CHANNEL_REG[ChannelRegistry
Multi-Server Channels]:::service GH_USER_REG[GitHubUserRegistry
GitHub to Discord Mapping]:::service ATTEND_SVC[AttendanceService
Grade Classification]:::service TB_CLIENT[TB API Client
Task Submit + Poll]:::service JIRA_CLIENT[Jira Client
REST API + JQL]:::service end subgraph Parsers [Event Processing] GH_PARSER[GitHubParser
9+ Event Types]:::service GH_FORMATTER[GitHubFormatter
Discord Embed Builder]:::service JIRA_FORMATTER[JiraFormatter
Issue Embed Builder]:::service end end subgraph LocalStorage [Local Storage] SQLITE[(SQLite
attendance.db)]:::storage YAML_CH[channels.yaml
Server Config]:::storage YAML_USERS[users.yaml
User Mapping]:::storage end subgraph ExternalSvc [External Services] TB_SVC[TB Server
HTTP :8000]:::external GITHUB_SVC[GitHub
Webhook Events]:::external JIRA_SVC[Jira/Atlassian
REST API]:::external DISCORD_SVC[Discord API
Bot Gateway]:::external end BOT --> CMD_TB BOT --> CMD_ATTEND BOT --> CMD_CHANNEL BOT --> CMD_GITHUB BOT --> CMD_JIRA CMD_TB --> TB_CLIENT CMD_ATTEND --> ATTEND_SVC CMD_CHANNEL --> CHANNEL_REG CMD_GITHUB --> GH_USER_REG CMD_JIRA --> JIRA_CLIENT WH_API --> WH_GITHUB WH_API --> WH_SEND WH_GITHUB --> GH_PARSER GH_PARSER --> GH_FORMATTER GH_FORMATTER --> MESSENGER MESSENGER --> CHANNEL_REG TB_CLIENT -->|POST /api/v1/generate| TB_SVC TB_CLIENT -->|GET /api/v1/task/id| TB_SVC TB_CLIENT -->|POST /api/v1/tts/synthesize| TB_SVC JIRA_CLIENT --> JIRA_SVC GITHUB_SVC --> WH_GITHUB BOT --> DISCORD_SVC ATTEND_SVC --> SQLITE CHANNEL_REG --> YAML_CH GH_USER_REG --> YAML_USERS
DP (data-collect-pipeline) Internal Architecture
데이터 수집 전담. 일부 요약 생성 기능은 TB의 내장 도구(collect_daily_news, collect_top_stories)로 이관됨.
graph TB
classDef task fill:#ff9900,stroke:#333,stroke-width:1px,color:white;
classDef adapter fill:#3776ab,stroke:#333,stroke-width:1px,color:white;
classDef storage fill:#e1e1e1,stroke:#333,stroke-width:2px;
subgraph Orchestration [Prefect Orchestration - main_prefect.py]
MAIN[Concurrent Flows]
end
subgraph Tasks [Task Modules]
T_NEWS[news_tasks.py
Hourly]:::task T_MARKET[market_tasks.py
Hourly]:::task T_TECH[tech_trend_tasks.py
4x Daily]:::task T_YT[syukaworld_tasks.py
Hourly]:::task T_SEARCH[search_tasks.py
Every 1 min]:::task T_MONITOR[summary_monitor_tasks.py
Every 5 min]:::task end subgraph Adapters [Adapter Layer - Singleton Pattern] subgraph Collectors [adapter/collector/] C_NEWS[NewsCollector
Naver API + Trafilatura]:::adapter C_MARKET[MarketCollector
yfinance]:::adapter C_TECH[TechCollector
Hacker News]:::adapter C_YT[YouTubeCollector
scrapetube + transcript]:::adapter C_SEARCH[GoogleSearchCollector
Custom Search API]:::adapter end subgraph Processing [adapter/queue/] QUEUE[TaskQueue
Redis Queue Client]:::adapter end subgraph Storage [adapter/db/ and redis/] DB_REPO[DBRepository
MySQL CRUD]:::adapter REDIS_REPO[RedisRepository
Task State]:::adapter end end subgraph ExternalStore [External Storage] MYSQL[(MySQL)]:::storage REDIS[(Redis)]:::storage end MAIN --> T_NEWS MAIN --> T_MARKET MAIN --> T_TECH MAIN --> T_YT MAIN --> T_SEARCH MAIN --> T_MONITOR T_NEWS --> C_NEWS T_MARKET --> C_MARKET T_TECH --> C_TECH T_YT --> C_YT T_SEARCH --> C_SEARCH T_MONITOR --> QUEUE QUEUE --> REDIS C_NEWS --> DB_REPO C_MARKET --> DB_REPO C_TECH --> DB_REPO C_YT --> DB_REPO T_MONITOR --> DB_REPO T_MONITOR --> REDIS_REPO DB_REPO --> MYSQL REDIS_REPO --> REDIS
Hourly]:::task T_MARKET[market_tasks.py
Hourly]:::task T_TECH[tech_trend_tasks.py
4x Daily]:::task T_YT[syukaworld_tasks.py
Hourly]:::task T_SEARCH[search_tasks.py
Every 1 min]:::task T_MONITOR[summary_monitor_tasks.py
Every 5 min]:::task end subgraph Adapters [Adapter Layer - Singleton Pattern] subgraph Collectors [adapter/collector/] C_NEWS[NewsCollector
Naver API + Trafilatura]:::adapter C_MARKET[MarketCollector
yfinance]:::adapter C_TECH[TechCollector
Hacker News]:::adapter C_YT[YouTubeCollector
scrapetube + transcript]:::adapter C_SEARCH[GoogleSearchCollector
Custom Search API]:::adapter end subgraph Processing [adapter/queue/] QUEUE[TaskQueue
Redis Queue Client]:::adapter end subgraph Storage [adapter/db/ and redis/] DB_REPO[DBRepository
MySQL CRUD]:::adapter REDIS_REPO[RedisRepository
Task State]:::adapter end end subgraph ExternalStore [External Storage] MYSQL[(MySQL)]:::storage REDIS[(Redis)]:::storage end MAIN --> T_NEWS MAIN --> T_MARKET MAIN --> T_TECH MAIN --> T_YT MAIN --> T_SEARCH MAIN --> T_MONITOR T_NEWS --> C_NEWS T_MARKET --> C_MARKET T_TECH --> C_TECH T_YT --> C_YT T_SEARCH --> C_SEARCH T_MONITOR --> QUEUE QUEUE --> REDIS C_NEWS --> DB_REPO C_MARKET --> DB_REPO C_TECH --> DB_REPO C_YT --> DB_REPO T_MONITOR --> DB_REPO T_MONITOR --> REDIS_REPO DB_REPO --> MYSQL REDIS_REPO --> REDIS
Sb (sbServer) Internal Layered Architecture
3-tier 레이어 구조: Driver(진입점) → Application(비즈니스 로직) → Infra(외부 연동). Chat이 TB 통신 중앙 허브 역할.
graph TB
classDef driver fill:#ff9900,stroke:#333,stroke-width:1px;
classDef app fill:#6db33f,stroke:#333,stroke-width:1px,color:white;
classDef infra fill:#3776ab,stroke:#333,stroke-width:1px,color:white;
classDef common fill:#607d8b,stroke:#333,stroke-width:1px,color:white;
classDef newmod fill:#e91e63,stroke:#333,stroke-width:1px,color:white;
subgraph Driver ["Driver Layer - Entry Points"]
WEB["Web Controllers
Thymeleaf Dashboard"]:::driver API["REST Controllers
ApiV1Controller"]:::driver end subgraph Auth ["Auth Module"] AUTH_CTRL["LocalOAuthController"]:::app AUTH_SVC["UserService
GoogleAuthService"]:::app JWT["JwtUtil
HS256"]:::app end subgraph Contents ["Contents Modules - Active Domains"] CHAT["Chat
TB 통신 중앙 허브"]:::newmod PROMPT["Prompt/Notebook
Markdown Docs"]:::app NEWS["News
Summaries"]:::app SEARCH["Search
Full-text"]:::app SEARCH_SUM["SearchSummary
Queue + History"]:::app MARKET["Market"]:::app MUSIC["Music Player
YouTube + WebSocket"]:::newmod VOICE["Voice
STT + Audio"]:::newmod HEARTBEAT["Heartbeat
System Monitor"]:::newmod VIEWER["Content Viewer"]:::app RESTAURANT["Restaurant
Location-based"]:::app end subgraph RedisQueues ["Redis Queue Publishers - 6 Patterns"] Q_CHAT["ChatQueuePublisher
tasks:pending"]:::infra Q_SEARCH["SearchSummaryQueuePublisher
queue:search:request"]:::infra Q_AUDIO["AudioQueuePublisher
audio_queue"]:::infra Q_STT["SttCommandPublisher
stt_command"]:::infra Q_HEARTBEAT["HeartbeatEventPublisher
stream:heartbeat:event"]:::infra Q_MUSIC["MusicEventPublisher
music:control"]:::infra end subgraph Repo ["JPA Repositories"] MD_REPO["MarkdownDocumentRepository
Cursor Pagination"]:::infra NB_REPO["NotebookRepository"]:::infra USER_REPO["UserRepository"]:::infra REST_REPO["RestaurantRepository
Haversine Query"]:::infra TASK_REPO["TaskResultRepository"]:::infra end subgraph ExtAPI ["External API Clients"] NAVER["NaverNewsApiCaller
WebClient"]:::infra GOOGLE_API["GoogleOAuthApiCaller"]:::infra WEATHER_API["WeatherApiClient
Feign"]:::infra MAP_API["NearByApiClient
Google Places"]:::infra end subgraph Common ["Common - Cross-cutting"] CONFIG["DataSourceConfig
Account + Contents"]:::common ASPECT["OAuthCheckAspect
RequireOAuth"]:::common ADVICE["ResponseWrapperAdvice
ExceptionHandler"]:::common end subgraph Storage ["Databases"] ACCOUNT_DB[("Account DB
Users, LocalAccounts")] CONTENTS_DB[("Contents DB
Documents, News, Tasks")] REDIS_DB[("Redis
Task Queues / Pub-Sub")] end WEB --> CHAT WEB --> PROMPT WEB --> NEWS WEB --> MUSIC API --> AUTH_CTRL API --> CHAT API --> PROMPT API --> NEWS API --> SEARCH API --> SEARCH_SUM API --> MARKET API --> VOICE API --> HEARTBEAT API --> RESTAURANT AUTH_CTRL --> AUTH_SVC AUTH_SVC --> JWT AUTH_SVC --> USER_REPO CHAT --> Q_CHAT CHAT --> TASK_REPO SEARCH_SUM --> Q_SEARCH VOICE --> Q_AUDIO VOICE --> Q_STT HEARTBEAT --> Q_HEARTBEAT MUSIC --> Q_MUSIC PROMPT --> MD_REPO PROMPT --> NB_REPO RESTAURANT --> REST_REPO MD_REPO --> CONTENTS_DB NB_REPO --> CONTENTS_DB USER_REPO --> ACCOUNT_DB REST_REPO --> CONTENTS_DB TASK_REPO --> CONTENTS_DB Q_CHAT --> REDIS_DB Q_SEARCH --> REDIS_DB NEWS --> NAVER AUTH_SVC --> GOOGLE_API ASPECT -.->|Intercept| API
Thymeleaf Dashboard"]:::driver API["REST Controllers
ApiV1Controller"]:::driver end subgraph Auth ["Auth Module"] AUTH_CTRL["LocalOAuthController"]:::app AUTH_SVC["UserService
GoogleAuthService"]:::app JWT["JwtUtil
HS256"]:::app end subgraph Contents ["Contents Modules - Active Domains"] CHAT["Chat
TB 통신 중앙 허브"]:::newmod PROMPT["Prompt/Notebook
Markdown Docs"]:::app NEWS["News
Summaries"]:::app SEARCH["Search
Full-text"]:::app SEARCH_SUM["SearchSummary
Queue + History"]:::app MARKET["Market"]:::app MUSIC["Music Player
YouTube + WebSocket"]:::newmod VOICE["Voice
STT + Audio"]:::newmod HEARTBEAT["Heartbeat
System Monitor"]:::newmod VIEWER["Content Viewer"]:::app RESTAURANT["Restaurant
Location-based"]:::app end subgraph RedisQueues ["Redis Queue Publishers - 6 Patterns"] Q_CHAT["ChatQueuePublisher
tasks:pending"]:::infra Q_SEARCH["SearchSummaryQueuePublisher
queue:search:request"]:::infra Q_AUDIO["AudioQueuePublisher
audio_queue"]:::infra Q_STT["SttCommandPublisher
stt_command"]:::infra Q_HEARTBEAT["HeartbeatEventPublisher
stream:heartbeat:event"]:::infra Q_MUSIC["MusicEventPublisher
music:control"]:::infra end subgraph Repo ["JPA Repositories"] MD_REPO["MarkdownDocumentRepository
Cursor Pagination"]:::infra NB_REPO["NotebookRepository"]:::infra USER_REPO["UserRepository"]:::infra REST_REPO["RestaurantRepository
Haversine Query"]:::infra TASK_REPO["TaskResultRepository"]:::infra end subgraph ExtAPI ["External API Clients"] NAVER["NaverNewsApiCaller
WebClient"]:::infra GOOGLE_API["GoogleOAuthApiCaller"]:::infra WEATHER_API["WeatherApiClient
Feign"]:::infra MAP_API["NearByApiClient
Google Places"]:::infra end subgraph Common ["Common - Cross-cutting"] CONFIG["DataSourceConfig
Account + Contents"]:::common ASPECT["OAuthCheckAspect
RequireOAuth"]:::common ADVICE["ResponseWrapperAdvice
ExceptionHandler"]:::common end subgraph Storage ["Databases"] ACCOUNT_DB[("Account DB
Users, LocalAccounts")] CONTENTS_DB[("Contents DB
Documents, News, Tasks")] REDIS_DB[("Redis
Task Queues / Pub-Sub")] end WEB --> CHAT WEB --> PROMPT WEB --> NEWS WEB --> MUSIC API --> AUTH_CTRL API --> CHAT API --> PROMPT API --> NEWS API --> SEARCH API --> SEARCH_SUM API --> MARKET API --> VOICE API --> HEARTBEAT API --> RESTAURANT AUTH_CTRL --> AUTH_SVC AUTH_SVC --> JWT AUTH_SVC --> USER_REPO CHAT --> Q_CHAT CHAT --> TASK_REPO SEARCH_SUM --> Q_SEARCH VOICE --> Q_AUDIO VOICE --> Q_STT HEARTBEAT --> Q_HEARTBEAT MUSIC --> Q_MUSIC PROMPT --> MD_REPO PROMPT --> NB_REPO RESTAURANT --> REST_REPO MD_REPO --> CONTENTS_DB NB_REPO --> CONTENTS_DB USER_REPO --> ACCOUNT_DB REST_REPO --> CONTENTS_DB TASK_REPO --> CONTENTS_DB Q_CHAT --> REDIS_DB Q_SEARCH --> REDIS_DB NEWS --> NAVER AUTH_SVC --> GOOGLE_API ASPECT -.->|Intercept| API
Integration Guidelines & Key Protocols
TB 통합 아키텍처
Direct Function Call:
- AG → AM: switch_to_llm/vlm/coder()
- AG → RL: get_knowledge_engine()
- HTTP/Redis 오버헤드 제거
- 단일 프로세스 메모리 공유
- LangGraph Native Agent Loop
ModelLifecycleManager
Exclusive Load Strategy:
- LLM (~8GB) / VLM (~5GB) / Coder (~9GB) / TTS (~3.4GB)
- 한 번에 하나만 로드 (Exclusive Policy)
- ensure_loaded(): 현재 모델 해제 → 요청 모델 로드
- gc.collect() + MLX Metal cache clear
LangGraph Native Loop
Agent Loop Pipeline:
1. agent_node: LLM이 도구 선택 (JSON)
2. tool_node: 선택된 도구 실행
3. 반복 (should_continue 판단)
4. report_node: 최종 리포트 합성
5. rag_ingest_node: 벡터 DB 저장
- 20개 내장 도구 (8 카테고리)
Discord Bot 연동
HTTP API 직접 통신:
- /tb ask → POST /api/v1/generate
- Polling: GET /api/v1/task/{id} (2초)
- /tb tts → POST /api/v1/tts/synthesize
- GitHub Webhook → Discord 브로드캐스트
- Jira JQL 검색 + 이슈 조회
- 출퇴근 관리 (SQLite 기반)
TB API Endpoints (51)
POST /api/v1/generate- 작업 생성GET /api/v1/task/{id}- 상태/결과POST /api/v1/ingest- 벡터 수집DELETE /api/v1/qdrant/reset- 벡터 초기화GET /api/v1/tasks/history- 히스토리POST /api/v1/tts/synthesize- 음성 합성POST /api/v1/coder/review- 코드 리뷰POST /api/v1/coder/analyze- 로그 분석GET /api/v1/system/status- 시스템 상태POST /api/v1/models/{name}/load- 모델 로드
TB Heartbeat & Module API
GET /api/v1/heartbeat/slots- 슬롯 목록POST /api/v1/heartbeat/slots- 슬롯 생성PUT /api/v1/heartbeat/slots/{id}- 수정POST /api/v1/heartbeat/slots/{id}/toggle- 활성/비활성POST /api/v1/heartbeat/slots/{id}/test- 테스트 실행GET /api/v1/heartbeat/history- 실행 이력GET /api/v1/modules- 모듈 목록POST /api/v1/modules/{name}/run- 모듈 실행GET /api/v1/modules/tools- 도구 메타데이터PUT /api/v1/preferences- 사용자 설정
Redis Queue Protocol
tasks:pending- Sb/DP → TB 작업 큐task:{id}- 작업 상태 Hash (TTL 600s)channel:task_completed- 완료 Pub/Subqueue:search:request- Sb → DP 검색audio_queue:{session}- 음성 청크stt_command- STT 명령stream:heartbeat:event- HB 이벤트music:control- 음악 Pub/Sub
MySQL Tables (TB)
task_results- 작업 결과 영속 저장execution_traces- 노드별 실행 추적runtime_settings- 런타임 설정heartbeat_slots- 스케줄 정의heartbeat_deliveries- 실행 이력modules- MD 모듈 정의chat_history- 대화 이력
TB Startup Lifecycle - Lifespan Init
| Stage | Component | Description | Shutdown (Reverse Order) |
|---|---|---|---|
| 1 | RL KnowledgeEngine | R2에서 스냅샷 다운로드 → 임베딩 모델 로드 (bge-m3) → Qdrant 연결 | 최종 스냅샷 업로드 → 엔진 종료 |
| 2 | ModelLifecycleManager | LLM/VLM/Coder/TTS 어댑터 등록, Exclusive Load 정책 설정 | 모든 모델 언로드 + gc.collect |
| 3 | AG Worker | Async Task 시작: RPOP tasks:pending → LangGraph 처리 루프 | Worker Task 취소 |
| 4 | Module Loader | modules/*.md 파일 파싱 → MySQL modules 테이블 동기화 | - |
| 5 | Heartbeat Service | MySQL에서 슬롯 로드 → APScheduler 크론 잡 등록 | Heartbeat 서비스 중지 |
| 6 | Snapshot Upload | 1시간 주기 백그라운드 태스크: Qdrant → R2 자동 백업 | - |
DP Execution Schedule (Prefect)
| Flow | Cron Schedule | Frequency | Description |
|---|---|---|---|
| Hourly News Collection | 0 * * * * |
Every hour | Naver News API → news_contents |
| Hourly Market Collection | 0 * * * * |
Every hour | Yahoo Finance (KRW, KOSPI, S&P500, BTC, OIL, GOLD) → market_indices |
| Tech Trend Workflow | 0 8,12,18,22 * * * |
4x daily (KST) | Hacker News Top 10 → tech_contents |
| Hourly YouTube Syuka | 0 * * * * |
Every hour | @syukaworld channel → youtube_summaries |
| Search Summary Flow | */1 * * * * |
Every minute | Process queue:search:request → search_summaries |
| Summary Monitor Flow | */5 * * * * |
Every 5 minutes | Poll TB for completed tasks, store results |
Service Ports & Connections
Service Ports
| Service | Port | Protocol |
|---|---|---|
| Sb sbServer | 443 (HTTPS) | REST + WebSocket |
| TB Total-Bot-TB- | 8000 | REST (FastAPI) |
| DB discord-bot | 8001 | REST (FastAPI Webhook) |
| DP Prefect Server UI | 4200 | HTTP |
Inter-Service Communication
| From → To | Method | Channel |
|---|---|---|
| Sb → TB | Redis Queue | tasks:pending |
| DP → TB | Redis Queue | tasks:pending |
| DB → TB | HTTP API | POST /api/v1/generate |
| DB → TB | HTTP API | POST /api/v1/tts/synthesize |
| GitHub → DB | Webhook | POST /webhook/github |
| TB 내부 | Direct Call | AG ↔ AM ↔ RL 함수 호출 |
Data Connections
| Service | Host | Port |
|---|---|---|
| MySQL (Synology) | mylifeis123.synology.me | 3306 |
| Redis (Synology) | mylifeis123.synology.me | 6379 |
| Qdrant (TB Local) | ./rl/data/qdrant | - |
| Cloudflare R2 | *.r2.cloudflarestorage.com | 443 |
| SQLite (DB Local) | ./data/attendance.db | - |
Notification Channels
| Channel | Method | Usage |
|---|---|---|
| Discord Webhook | HTTPS POST | TB 알림 전달 (Embed + File) |
| Discord Bot API | Bot Token | TB 채널별 메시지 + 오디오 |
| Notion API | HTTPS | 캘린더 동기화 / 페이지 생성 |
| discord-bot | Slash Commands | 사용자 → TB AI 질의 / TTS |