refactor monitor structure and polish control console
This commit is contained in:
parent
c8b2d7639c
commit
47ba1c8a76
12
.env.example
12
.env.example
@ -75,6 +75,18 @@ MONGO_URI=
|
||||
MONGO_DB_NAME=tarab_all
|
||||
MONGO_INSTANCE_ID=ins-o9mmob66
|
||||
MONGO_CACHE_TTL_SECONDS=20
|
||||
MYSQL_HOST=
|
||||
MYSQL_PORT=3306
|
||||
MYSQL_USER=
|
||||
MYSQL_PASSWORD=
|
||||
MYSQL_DEFAULT_DATABASE=
|
||||
MYSQL_CACHE_TTL_SECONDS=10
|
||||
MYSQL_CONNECT_TIMEOUT_SECONDS=5
|
||||
MYSQL_READ_TIMEOUT_SECONDS=15
|
||||
MYSQL_WRITE_TIMEOUT_SECONDS=15
|
||||
MYSQL_DEFAULT_PAGE_SIZE=100
|
||||
MYSQL_MAX_PAGE_SIZE=200
|
||||
MYSQL_QUERY_MAX_ROWS=500
|
||||
CPU_WARN_PERCENT=60
|
||||
CPU_BAD_PERCENT=85
|
||||
MEMORY_WARN_PERCENT=70
|
||||
|
||||
@ -52,3 +52,25 @@
|
||||
- `配置` 页必须拆成页内 tabs:`Nacos / Go Env / 历史版本`;编辑区使用 sticky 工具条、metadata strip、`源码 / Diff` 切换。
|
||||
- `基础设施` 页必须有二级导航:`状态总览 / Nacos 服务 / Mongo 指标 / TAT`;Nacos 服务默认折叠实例,Mongo 次要指标默认折叠。
|
||||
- 长文本默认缩写,完整值只通过悬停、复制或展开明细承接,不在主列表直接铺满一整行。
|
||||
- `发布` 页主动作必须集中在一条 action bar;区块内部不重复摆放刷新、预检查、执行类按钮。
|
||||
- 高权重动作必须明显少于次动作;`保存 / 执行 / 立即刷新` 才能用高权重按钮,其余默认降为 ghost 或轻量控件。
|
||||
- `执行日志` 和 `服务日志` 默认不直接铺在主页面里;页面只保留摘要,完整日志通过统一弹层查看和导出。
|
||||
- `配置` 页左侧必须保持 explorer 式树状浏览,不回退到一张张大卡片;组折叠、当前项高亮、上下项切换都要直接可用。
|
||||
- `配置` 页左侧 explorer 不重复展示已经在 workbench head 里出现的 baseUrl、组数、环境信息。
|
||||
- `配置` 页不要重复出现“页面标题 / 分区标题 / 当前文件卡片”三层同义信息;当前文件、group、type 合并进工具条和 metadata strip。
|
||||
- `配置` 页左侧文件树优先高密度列表,压缩单项高度、内边距和顶部无效空白。
|
||||
- `任务型页面` 优先使用紧凑任务头,不再额外堆一张信息价值低的说明卡。
|
||||
- 主机视图里的百分比指标优先使用小圆环 + 数值标签,不回退到纯文本堆叠。
|
||||
- 主机行中部优先展示“在线服务 / 异常服务摘要”,避免重复出现“全部正常”这类低信息密度文案。
|
||||
- 主机分组标题必须带主机数、服务数、异常主机数,保证高密度时仍能快速扫读分区状态。
|
||||
- 总览页在全正常状态下不继续渲染空异常区;异常摘要、异常主机、异常基础设施为空时直接隐藏,正常项只保留紧凑摘要。
|
||||
- 首页异常摘要优先使用分段状态带;正常健康态保持紧凑卡组,不回退到多张孤立大卡。
|
||||
- 首页服务矩阵的每个业务分组应作为独立分组卡展示,分组头直接包含服务数、节点数、异常数。
|
||||
- 首页的网关 / 应用 / 支付分组允许使用轻微色温差异和导向线区分,但必须克制,不把分组色做成高饱和装饰。
|
||||
- 首页正常主机默认使用紧凑卡栅格;没有异常主机时不渲染大空态占位。
|
||||
- 主机指标拿不到时显示采集状态芯片,不用空圆环或占位短横伪装成正常指标。
|
||||
- `发布` 页更新目标和滚动重启优先使用多列紧凑服务卡,完整宽表只留给执行结果和明细。
|
||||
- 可选择条目优先使用胶囊式勾选控件,已选态要明显强于未选态,不能只靠原生 checkbox 本体提示选择结果。
|
||||
- 首页侧栏卡、基础设施小卡、任务页工作区尽量复用同一套顶部导向线和轻表面语言,避免不同页面像不同产品拼接。
|
||||
- `数据库` 视图使用统一的工作台入口,`MySQL / MongoDB` 通过同一层引擎切换承接,不再把数据库操作散落在不同页面角落。
|
||||
- `MongoDB / MySQL` 操作页都采用 `左侧 explorer + 右侧 workbench` 结构;概览指标保持紧凑,危险操作集中到单独分区,不把删除类动作和普通查询动作混在一起。
|
||||
|
||||
21
AGENTS.md
21
AGENTS.md
@ -52,10 +52,31 @@
|
||||
- 首页首屏默认收敛为顶部导航、工具条、紧凑摘要条、服务矩阵,不额外插入大块最近异常横幅。
|
||||
- 最近异常进入可折叠详情侧栏顶部,不独立占据首页主宽度。
|
||||
- 服务矩阵按业务分组切成子矩阵,异常排序前置,表头和第一列保持 sticky。
|
||||
- 首页异常摘要优先使用分段状态带,不回退到零散大卡。
|
||||
- 服务矩阵的业务分组要做成独立分组卡,分组头直接给服务数、节点数、异常数。
|
||||
- 首页网关 / 应用 / 支付分组可以用轻微色温差异和导向线区分,但保持低饱和,不做装饰性大色块。
|
||||
- 首页主机区只展示业务主机;基础设施主机放到基础设施或主机视图承接。
|
||||
- 主机指标使用单行短文本,正常态降饱和,红橙色保留给异常和告警。
|
||||
- 首页没有异常主机时不渲染大空态,正常主机直接用紧凑卡栅格展示。
|
||||
- 主机指标缺失时直接显示采集状态芯片,不渲染空圆环和占位短横。
|
||||
- 监控筛选条只属于 `总览 / 服务 / 主机`;`发布 / 配置 / 基础设施` 必须使用各自专用工具条和页内导航。
|
||||
- `发布` 页按“构建参数 -> 更新目标 -> 执行日志 -> 滚动重启”组织,更新与重启目标都优先用紧凑表格,不再回退到大卡片流。
|
||||
- `配置` 页按 `Nacos / Go Env / 历史版本` tabs 组织,编辑器必须提供 sticky 工具条、metadata strip 和 `源码 / Diff` 切换。
|
||||
- `基础设施` 页按 `状态总览 / Nacos 服务 / Mongo 指标 / TAT` 二级导航组织,Nacos 服务默认折叠实例,Mongo 次要指标默认折叠。
|
||||
- 长文本默认缩写并提供复制或展开,不直接在主列表铺完整 digest、路径、长 ID 或长连接串。
|
||||
- `发布` 页主动作集中到单独 action bar,区块内部避免重复放刷新、预检查、执行类按钮。
|
||||
- 高权重动作必须少于次动作;只有 `保存 / 执行 / 立即刷新` 这类核心操作保留高权重按钮,其余动作降为 ghost 或轻量控件。
|
||||
- `执行日志` 和 `服务日志` 默认通过统一弹层查看;主页面只保留摘要和上下文,不铺大块日志窗口。
|
||||
- `配置` 页左侧使用 explorer 式树状导航,避免大卡片式配置清单;当前项高亮、组折叠、上下项跳转必须可用。
|
||||
- `配置` 页左侧 explorer 不重复展示已经在 workbench head 里出现的 baseUrl、组数、环境信息。
|
||||
- `配置` 页不要重复堆叠大标题和当前文件卡片;当前文件信息应并入工具条和 metadata strip,左侧文件树保持高密度。
|
||||
- `任务型页面` 优先使用紧凑任务头,不再额外堆一张信息价值低的说明卡。
|
||||
- 主机视图中的百分比指标优先使用小圆环 + 数值标签,不回退到纯文本百分比堆叠。
|
||||
- 主机行中部优先展示服务摘要,避免重复出现“全部正常”这类低信息密度文案。
|
||||
- 主机分组标题需要同时给出主机数、服务数、异常主机数,确保高密度下仍然易扫读。
|
||||
- 总览页全正常时不继续渲染空异常区;正常主机和 `Nacos / Mongo / TAT` 统一收成紧凑摘要。
|
||||
- `发布` 页更新目标和滚动重启默认使用多列紧凑卡片,避免一项一整行的浪费布局。
|
||||
- 可选择条目优先使用胶囊式勾选控件,已选态必须明显强于未选态,不能只靠原生 checkbox 提示。
|
||||
- 首页侧栏卡、基础设施小卡、任务工作区尽量复用同一套顶部导向线和轻表面语言,避免跨页设计语言漂移。
|
||||
- `数据库` 视图使用统一的工作台入口,`MySQL / MongoDB` 通过同一层引擎切换承接,不再把数据库操作散落在不同页面角落。
|
||||
- `MongoDB / MySQL` 操作页统一采用 `左侧 explorer + 右侧 workbench` 结构;概览指标保持紧凑,危险操作集中在单独分区,不把删除类动作和普通查询动作混在一起。
|
||||
|
||||
30
README.md
30
README.md
@ -11,7 +11,8 @@
|
||||
- 服务健康:直接探测各业务服务健康接口
|
||||
- 主机资源:通过 `SSH` 采集 `CPU / 内存 / 磁盘 / 进程数`
|
||||
- Nacos:直连 Nacos OpenAPI,列出命名空间、服务和注册实例
|
||||
- MongoDB:直连 Mongo,采集 `serverStatus / listDatabases / replSetGetStatus`
|
||||
- MongoDB:直连 Mongo,采集 `serverStatus / listDatabases / replSetGetStatus`,并提供库表浏览、分页查询、CRUD、聚合、Distinct、索引管理
|
||||
- MySQL:直连云 MySQL,提供运行状态、库表浏览、100 条分页查询、CRUD、列/索引变更和 SQL 工作台
|
||||
- 宿主机运维:日志、滚动重启、compose 同步、服务更新统一走 `SSH`
|
||||
- 带外引导:只在打通 `deploy -> target:22` 时使用 `TAT`
|
||||
|
||||
@ -20,9 +21,10 @@
|
||||
- `server.py`:gateway 入口
|
||||
- `yumi_server.py`:Yumi 内部实例入口
|
||||
- `haiyi_server.py`:HaiYi 内部实例入口
|
||||
- `gateway/`:统一登录、入口选择页和反代
|
||||
- `monitor/`:Yumi 监控与运维实现
|
||||
- `haiyi_monitor/`:HaiYi 占位实例
|
||||
- `entry/`:统一登录、入口选择页和反代
|
||||
- `core/`:共享配置、鉴权、缓存和路径工具
|
||||
- `monitors/yumi/`:Yumi 监控与运维实现
|
||||
- `monitors/haiyi/`:HaiYi 占位实例
|
||||
- `config/targets.json`:监控目标和实例 ID
|
||||
- `static/`:前端页面
|
||||
- `scripts/`:启停、自启安装、前台运行脚本
|
||||
@ -51,6 +53,11 @@ REFRESH_INTERVAL_OPTIONS=5,10,15,30,60
|
||||
NACOS_BASE_URL=http://10.2.1.17:8848
|
||||
NACOS_NAMESPACE_ID=...
|
||||
MONGO_URI=mongodb://...
|
||||
MYSQL_HOST=10.x.x.x
|
||||
MYSQL_PORT=3306
|
||||
MYSQL_USER=readonly_or_admin
|
||||
MYSQL_PASSWORD=***
|
||||
MYSQL_DEFAULT_DATABASE=
|
||||
DEPLOY_REGION=me-saudi-arabia
|
||||
DEPLOY_INSTANCE_ID=
|
||||
TENCENT_SECRET_ID=***
|
||||
@ -115,6 +122,18 @@ cp .env.example .env
|
||||
AUTH_COOKIE_SECURE=1
|
||||
```
|
||||
|
||||
MySQL 工作台可选参数:
|
||||
|
||||
```bash
|
||||
MYSQL_CACHE_TTL_SECONDS=10
|
||||
MYSQL_CONNECT_TIMEOUT_SECONDS=5
|
||||
MYSQL_READ_TIMEOUT_SECONDS=15
|
||||
MYSQL_WRITE_TIMEOUT_SECONDS=15
|
||||
MYSQL_DEFAULT_PAGE_SIZE=100
|
||||
MYSQL_MAX_PAGE_SIZE=200
|
||||
MYSQL_QUERY_MAX_ROWS=500
|
||||
```
|
||||
|
||||
## 本地运行
|
||||
|
||||
```bash
|
||||
@ -263,4 +282,7 @@ python3 ops/sync_runtime_compose.py --sync-remote
|
||||
- 服务更新:本地构建选中服务、推送 Harbor、更新完整 compose,并只重建选中的服务
|
||||
- 滚动重启:可按服务选择 Java / Golang 节点,逐台重启并等待健康恢复
|
||||
- MongoDB:连接、内存、网络、操作计数、复制集、数据库统计
|
||||
- Mongo 工作页:列出所有数据库和集合,文档默认每页 `100` 条,支持 `find / insertOne / insertMany / replaceOne / updateOne / updateMany / deleteOne / deleteMany / aggregate / distinct / createIndex / dropIndex / dropIndexes / createCollection / renameCollection / dropCollection / dropDatabase`
|
||||
- MySQL:连接状态、版本、当前/最大连接数、QPS/TPS、数据库大小、主从/只读状态
|
||||
- MySQL 工作页:列出所有数据库和表,默认每页 `100` 条,支持筛选排序、行级增删改、增加/删除列、增加/删除索引、常用 SQL 快捷操作和单语句 SQL 执行
|
||||
- 自动刷新:前端可切换刷新间隔,并保存在浏览器本地
|
||||
|
||||
1
core/__init__.py
Normal file
1
core/__init__.py
Normal file
@ -0,0 +1 @@
|
||||
"""Shared runtime layer for gateway and monitors."""
|
||||
@ -280,6 +280,31 @@ MONGO_SERVER_SELECTION_TIMEOUT_MS = env_int("MONGO_SERVER_SELECTION_TIMEOUT_MS",
|
||||
# Mongo socket 超时。
|
||||
MONGO_SOCKET_TIMEOUT_MS = env_int("MONGO_SOCKET_TIMEOUT_MS", 5000)
|
||||
|
||||
# MySQL 主机地址。
|
||||
MYSQL_HOST = env_str("MYSQL_HOST", "")
|
||||
# MySQL 端口。
|
||||
MYSQL_PORT = env_int("MYSQL_PORT", 3306)
|
||||
# MySQL 登录账号。
|
||||
MYSQL_USER = env_str("MYSQL_USER", "")
|
||||
# MySQL 登录密码;允许空串,兼容特殊场景。
|
||||
MYSQL_PASSWORD = os.environ.get("MYSQL_PASSWORD", "")
|
||||
# MySQL 默认库;为空时由页面显式选择。
|
||||
MYSQL_DEFAULT_DATABASE = env_str("MYSQL_DEFAULT_DATABASE", "")
|
||||
# MySQL 指标与元数据缓存秒数。
|
||||
MYSQL_CACHE_TTL_SECONDS = env_float("MYSQL_CACHE_TTL_SECONDS", 8.0)
|
||||
# MySQL 建连超时秒数。
|
||||
MYSQL_CONNECT_TIMEOUT_SECONDS = env_int("MYSQL_CONNECT_TIMEOUT_SECONDS", 5)
|
||||
# MySQL 读超时秒数。
|
||||
MYSQL_READ_TIMEOUT_SECONDS = env_int("MYSQL_READ_TIMEOUT_SECONDS", 8)
|
||||
# MySQL 写超时秒数。
|
||||
MYSQL_WRITE_TIMEOUT_SECONDS = env_int("MYSQL_WRITE_TIMEOUT_SECONDS", 15)
|
||||
# MySQL 表浏览默认分页大小。
|
||||
MYSQL_DEFAULT_PAGE_SIZE = env_int("MYSQL_DEFAULT_PAGE_SIZE", 100)
|
||||
# MySQL 表浏览最大分页大小。
|
||||
MYSQL_MAX_PAGE_SIZE = env_int("MYSQL_MAX_PAGE_SIZE", 200)
|
||||
# SQL 工作台单次最多返回多少行。
|
||||
MYSQL_QUERY_MAX_ROWS = env_int("MYSQL_QUERY_MAX_ROWS", 500)
|
||||
|
||||
# CPU 告警阈值。
|
||||
CPU_WARN_PERCENT = env_float("CPU_WARN_PERCENT", 60.0)
|
||||
# CPU 严重阈值。
|
||||
1
entry/__init__.py
Normal file
1
entry/__init__.py
Normal file
@ -0,0 +1 @@
|
||||
"""Unified gateway entry package."""
|
||||
@ -10,7 +10,7 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
from monitor.auth import (
|
||||
from core.auth import (
|
||||
build_clear_session_cookie,
|
||||
build_session_cookie,
|
||||
create_session,
|
||||
@ -20,7 +20,7 @@ from monitor.auth import (
|
||||
parse_session_token,
|
||||
authenticate_user,
|
||||
)
|
||||
from monitor.config import (
|
||||
from core.config import (
|
||||
APP_BIND,
|
||||
APP_PORT,
|
||||
GATEWAY_PROXY_TIMEOUT_SECONDS,
|
||||
@ -32,7 +32,7 @@ from monitor.config import (
|
||||
YUMI_INTERNAL_PORT,
|
||||
utc_now,
|
||||
)
|
||||
from monitor.http_paths import login_redirect_target
|
||||
from core.http_paths import login_redirect_target
|
||||
|
||||
|
||||
def response_json(payload: dict[str, Any] | list[Any]) -> bytes:
|
||||
@ -1 +0,0 @@
|
||||
from __future__ import annotations
|
||||
@ -1 +0,0 @@
|
||||
from __future__ import annotations
|
||||
@ -10,7 +10,7 @@ os.environ.setdefault("APP_BIND", os.environ.get("HAIYI_INTERNAL_HOST", "127.0.0
|
||||
os.environ.setdefault("APP_PORT", os.environ.get("HAIYI_INTERNAL_PORT", "2028"))
|
||||
os.environ["APP_BASE_PATH"] = "/haiyi"
|
||||
|
||||
from haiyi_monitor.http_app import main
|
||||
from monitors.haiyi.http_app import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@ -1,2 +0,0 @@
|
||||
"""hy-app-monitor package."""
|
||||
|
||||
1
monitors/__init__.py
Normal file
1
monitors/__init__.py
Normal file
@ -0,0 +1 @@
|
||||
"""Monitor application packages."""
|
||||
1
monitors/haiyi/__init__.py
Normal file
1
monitors/haiyi/__init__.py
Normal file
@ -0,0 +1 @@
|
||||
"""HaiYi monitor package."""
|
||||
@ -7,8 +7,8 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from monitor.config import APP_BASE_PATH, APP_BIND, APP_PORT, INTERNAL_PROXY_SHARED_SECRET, STATIC_DIR, utc_now
|
||||
from monitor.http_paths import join_base_path, strip_base_path
|
||||
from core.config import APP_BASE_PATH, APP_BIND, APP_PORT, INTERNAL_PROXY_SHARED_SECRET, STATIC_DIR, utc_now
|
||||
from core.http_paths import join_base_path, strip_base_path
|
||||
|
||||
|
||||
def response_json(payload: dict[str, Any] | list[Any]) -> bytes:
|
||||
1
monitors/yumi/__init__.py
Normal file
1
monitors/yumi/__init__.py
Normal file
@ -0,0 +1 @@
|
||||
"""Yumi monitor package."""
|
||||
@ -10,7 +10,7 @@ import shlex
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from .config import ROOT, RUNTIME_BASE_DIR, load_hosts, utc_now
|
||||
from core.config import ROOT, RUNTIME_BASE_DIR, load_hosts, utc_now
|
||||
from .runtime_config import read_remote_text, write_remote_text
|
||||
from .ssh_remote import copy_remote_file, run_host_script_checked
|
||||
|
||||
@ -2,10 +2,11 @@ from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from .config import THRESHOLDS, load_hosts, settings_payload, utc_now
|
||||
from core.config import THRESHOLDS, load_hosts, settings_payload, utc_now
|
||||
from .host_metrics import get_host_metrics
|
||||
from .http_probe import get_service_payload
|
||||
from .mongo_metrics import get_mongo_payload
|
||||
from .mysql_metrics import get_mysql_overview_payload
|
||||
from .nacos import get_nacos_payload
|
||||
|
||||
|
||||
@ -116,6 +117,8 @@ def build_monitor_payload() -> dict[str, Any]:
|
||||
nacos_payload = get_nacos_payload()
|
||||
# 获取 Mongo 指标。
|
||||
mongo_payload = get_mongo_payload()
|
||||
# 获取 MySQL 指标。
|
||||
mysql_payload = get_mysql_overview_payload()
|
||||
# 组装主机维度视图。
|
||||
grouped_hosts = build_host_payload(hosts, service_payload, metrics_by_host)
|
||||
# 统计主机指标成功数。
|
||||
@ -137,9 +140,13 @@ def build_monitor_payload() -> dict[str, Any]:
|
||||
"mongoDatabaseTotal": int(mongo_payload.get("summary", {}).get("databaseTotal", 0)),
|
||||
"mongoCollectionTotal": int(mongo_payload.get("summary", {}).get("collectionTotal", 0)),
|
||||
"mongoOk": 1 if mongo_payload.get("ok") else 0,
|
||||
"mysqlDatabaseTotal": int(mysql_payload.get("summary", {}).get("databaseTotal", 0)),
|
||||
"mysqlTableTotal": int(mysql_payload.get("summary", {}).get("tableTotal", 0)),
|
||||
"mysqlOk": 1 if mysql_payload.get("ok") else 0,
|
||||
},
|
||||
"hosts": grouped_hosts,
|
||||
"items": service_payload["items"],
|
||||
"nacos": nacos_payload,
|
||||
"mongo": mongo_payload,
|
||||
"mysql": mysql_payload,
|
||||
}
|
||||
@ -5,8 +5,8 @@ import concurrent.futures
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from .cache import TTLCache
|
||||
from .config import HOST_METRIC_CACHE_TTL_SECONDS, HOST_METRIC_CPU_SAMPLE_SECONDS, HOST_METRIC_TIMEOUT_SECONDS, utc_now
|
||||
from core.cache import TTLCache
|
||||
from core.config import HOST_METRIC_CACHE_TTL_SECONDS, HOST_METRIC_CPU_SAMPLE_SECONDS, HOST_METRIC_TIMEOUT_SECONDS, utc_now
|
||||
from .ssh_remote import run_host_script
|
||||
|
||||
|
||||
@ -7,9 +7,42 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
from .config import APP_BASE_PATH, APP_BIND, APP_PORT, INTERNAL_PROXY_SHARED_SECRET, STATIC_DIR, utc_now
|
||||
from core.config import APP_BASE_PATH, APP_BIND, APP_PORT, INTERNAL_PROXY_SHARED_SECRET, STATIC_DIR, utc_now
|
||||
from .dashboard import build_monitor_payload
|
||||
from .http_paths import join_base_path, strip_base_path
|
||||
from core.http_paths import join_base_path, strip_base_path
|
||||
from .mongo_admin import (
|
||||
aggregate_mongo_documents_payload,
|
||||
create_mongo_collection_payload,
|
||||
create_mongo_index_payload,
|
||||
delete_mongo_documents_payload,
|
||||
distinct_mongo_values_payload,
|
||||
drop_mongo_collection_payload,
|
||||
drop_mongo_database_payload,
|
||||
drop_mongo_index_payload,
|
||||
drop_mongo_secondary_indexes_payload,
|
||||
get_mongo_admin_overview_payload,
|
||||
get_mongo_collection_detail_payload,
|
||||
insert_mongo_document_payload,
|
||||
query_mongo_documents_payload,
|
||||
rename_mongo_collection_payload,
|
||||
replace_mongo_document_payload,
|
||||
update_mongo_documents_payload,
|
||||
)
|
||||
from .mysql_admin import (
|
||||
add_mysql_column_payload,
|
||||
add_mysql_index_payload,
|
||||
delete_mysql_row_payload,
|
||||
drop_mysql_column_payload,
|
||||
drop_mysql_index_payload,
|
||||
execute_mysql_sql_payload,
|
||||
get_mysql_databases_payload,
|
||||
get_mysql_table_schema_payload,
|
||||
get_mysql_tables_payload,
|
||||
insert_mysql_row_payload,
|
||||
query_mysql_table_rows_payload,
|
||||
update_mysql_row_payload,
|
||||
)
|
||||
from .mysql_metrics import get_mysql_overview_payload
|
||||
from .nacos import (
|
||||
get_nacos_config_list_payload,
|
||||
get_nacos_config_payload,
|
||||
@ -178,6 +211,120 @@ class MonitorHandler(BaseHTTPRequestHandler):
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
# 返回 MySQL 概览指标。
|
||||
if local_path == "/api/mysql/overview":
|
||||
try:
|
||||
payload = get_mysql_overview_payload()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
|
||||
|
||||
return self.send_payload(
|
||||
HTTPStatus.OK,
|
||||
response_json(payload),
|
||||
"application/json; charset=utf-8",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
# 返回 MySQL 库列表。
|
||||
if local_path == "/api/mysql/databases":
|
||||
try:
|
||||
payload = get_mysql_databases_payload()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
|
||||
|
||||
return self.send_payload(
|
||||
HTTPStatus.OK,
|
||||
response_json(payload),
|
||||
"application/json; charset=utf-8",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
# 返回某个库下的表列表。
|
||||
if local_path == "/api/mysql/tables":
|
||||
database_name = self.query_value(query, "database")
|
||||
if not database_name:
|
||||
return self.send_error_payload(
|
||||
HTTPStatus.BAD_REQUEST,
|
||||
"database is required",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
try:
|
||||
payload = get_mysql_tables_payload(database_name)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
|
||||
|
||||
return self.send_payload(
|
||||
HTTPStatus.OK,
|
||||
response_json(payload),
|
||||
"application/json; charset=utf-8",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
# 返回 MySQL 表结构、索引和建表语句。
|
||||
if local_path == "/api/mysql/table-schema":
|
||||
database_name = self.query_value(query, "database")
|
||||
table_name = self.query_value(query, "table")
|
||||
if not database_name or not table_name:
|
||||
return self.send_error_payload(
|
||||
HTTPStatus.BAD_REQUEST,
|
||||
"database and table are required",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
try:
|
||||
payload = get_mysql_table_schema_payload(database_name, table_name)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
|
||||
|
||||
return self.send_payload(
|
||||
HTTPStatus.OK,
|
||||
response_json(payload),
|
||||
"application/json; charset=utf-8",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
# 返回 Mongo 管理页的数据库与集合树。
|
||||
if local_path == "/api/mongo-admin/overview":
|
||||
try:
|
||||
payload = get_mongo_admin_overview_payload()
|
||||
except ValueError as exc:
|
||||
return self.send_error_payload(HTTPStatus.BAD_REQUEST, str(exc), send_body=send_body)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
|
||||
|
||||
return self.send_payload(
|
||||
HTTPStatus.OK,
|
||||
response_json(payload),
|
||||
"application/json; charset=utf-8",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
# 返回当前集合的 stats 和索引列表。
|
||||
if local_path == "/api/mongo-admin/collection-detail":
|
||||
database_name = self.query_value(query, "database")
|
||||
collection_name = self.query_value(query, "collection")
|
||||
if not database_name or not collection_name:
|
||||
return self.send_error_payload(
|
||||
HTTPStatus.BAD_REQUEST,
|
||||
"database and collection are required",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
try:
|
||||
payload = get_mongo_collection_detail_payload(database_name, collection_name)
|
||||
except ValueError as exc:
|
||||
return self.send_error_payload(HTTPStatus.BAD_REQUEST, str(exc), send_body=send_body)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
|
||||
|
||||
return self.send_payload(
|
||||
HTTPStatus.OK,
|
||||
response_json(payload),
|
||||
"application/json; charset=utf-8",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
# 返回可滚动重启的业务服务。
|
||||
if local_path == "/api/ops/restart-targets":
|
||||
try:
|
||||
@ -341,12 +488,81 @@ class MonitorHandler(BaseHTTPRequestHandler):
|
||||
if local_path == "/api/runtime/golang-config":
|
||||
return self.handle_go_save(payload)
|
||||
|
||||
if local_path == "/api/mysql/rows/query":
|
||||
return self.handle_mysql_action(query_mysql_table_rows_payload, payload)
|
||||
|
||||
if local_path == "/api/mysql/rows/insert":
|
||||
return self.handle_mysql_action(insert_mysql_row_payload, payload, status=HTTPStatus.CREATED)
|
||||
|
||||
if local_path == "/api/mysql/rows/update":
|
||||
return self.handle_mysql_action(update_mysql_row_payload, payload)
|
||||
|
||||
if local_path == "/api/mysql/rows/delete":
|
||||
return self.handle_mysql_action(delete_mysql_row_payload, payload)
|
||||
|
||||
if local_path == "/api/mysql/columns/add":
|
||||
return self.handle_mysql_action(add_mysql_column_payload, payload)
|
||||
|
||||
if local_path == "/api/mysql/columns/drop":
|
||||
return self.handle_mysql_action(drop_mysql_column_payload, payload)
|
||||
|
||||
if local_path == "/api/mysql/indexes/add":
|
||||
return self.handle_mysql_action(add_mysql_index_payload, payload, status=HTTPStatus.CREATED)
|
||||
|
||||
if local_path == "/api/mysql/indexes/drop":
|
||||
return self.handle_mysql_action(drop_mysql_index_payload, payload)
|
||||
|
||||
if local_path == "/api/mysql/sql/execute":
|
||||
return self.handle_mysql_action(execute_mysql_sql_payload, payload)
|
||||
|
||||
if local_path == "/api/ops/rolling-restart":
|
||||
return self.handle_rolling_restart(payload)
|
||||
|
||||
if local_path == "/api/ops/update-services":
|
||||
return self.handle_update_services(payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/query":
|
||||
return self.handle_mongo_admin_action(query_mongo_documents_payload, payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/documents/insert":
|
||||
return self.handle_mongo_admin_action(insert_mongo_document_payload, payload, status=HTTPStatus.CREATED)
|
||||
|
||||
if local_path == "/api/mongo-admin/documents/replace":
|
||||
return self.handle_mongo_admin_action(replace_mongo_document_payload, payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/documents/update":
|
||||
return self.handle_mongo_admin_action(update_mongo_documents_payload, payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/documents/delete":
|
||||
return self.handle_mongo_admin_action(delete_mongo_documents_payload, payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/aggregate":
|
||||
return self.handle_mongo_admin_action(aggregate_mongo_documents_payload, payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/distinct":
|
||||
return self.handle_mongo_admin_action(distinct_mongo_values_payload, payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/indexes/create":
|
||||
return self.handle_mongo_admin_action(create_mongo_index_payload, payload, status=HTTPStatus.CREATED)
|
||||
|
||||
if local_path == "/api/mongo-admin/indexes/drop":
|
||||
return self.handle_mongo_admin_action(drop_mongo_index_payload, payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/indexes/drop-secondary":
|
||||
return self.handle_mongo_admin_action(drop_mongo_secondary_indexes_payload, payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/collections/create":
|
||||
return self.handle_mongo_admin_action(create_mongo_collection_payload, payload, status=HTTPStatus.CREATED)
|
||||
|
||||
if local_path == "/api/mongo-admin/collections/rename":
|
||||
return self.handle_mongo_admin_action(rename_mongo_collection_payload, payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/collections/drop":
|
||||
return self.handle_mongo_admin_action(drop_mongo_collection_payload, payload)
|
||||
|
||||
if local_path == "/api/mongo-admin/databases/drop":
|
||||
return self.handle_mongo_admin_action(drop_mongo_database_payload, payload)
|
||||
|
||||
return self.send_error_payload(HTTPStatus.NOT_FOUND, "not found", send_body=True)
|
||||
|
||||
def redirect(self, location: str, *, send_body: bool) -> None:
|
||||
@ -444,6 +660,27 @@ class MonitorHandler(BaseHTTPRequestHandler):
|
||||
send_body=True,
|
||||
)
|
||||
|
||||
def handle_mysql_action(
|
||||
self,
|
||||
action: Any,
|
||||
payload: dict[str, Any],
|
||||
*,
|
||||
status: HTTPStatus = HTTPStatus.OK,
|
||||
) -> None:
|
||||
try:
|
||||
result = action(payload)
|
||||
except ValueError as exc:
|
||||
return self.send_error_payload(HTTPStatus.BAD_REQUEST, str(exc), send_body=True)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=True)
|
||||
|
||||
return self.send_payload(
|
||||
status,
|
||||
response_json(result),
|
||||
"application/json; charset=utf-8",
|
||||
send_body=True,
|
||||
)
|
||||
|
||||
def handle_rolling_restart(self, payload: dict[str, Any]) -> None:
|
||||
services = payload.get("services")
|
||||
if not isinstance(services, list):
|
||||
@ -491,6 +728,28 @@ class MonitorHandler(BaseHTTPRequestHandler):
|
||||
send_body=True,
|
||||
)
|
||||
|
||||
def handle_mongo_admin_action(
|
||||
self,
|
||||
builder: Any,
|
||||
payload: dict[str, Any],
|
||||
*,
|
||||
status: HTTPStatus = HTTPStatus.OK,
|
||||
) -> None:
|
||||
try:
|
||||
# Mongo 管理页所有写操作和查询操作都统一走这里收敛异常。
|
||||
result = builder(payload)
|
||||
except ValueError as exc:
|
||||
return self.send_error_payload(HTTPStatus.BAD_REQUEST, str(exc), send_body=True)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=True)
|
||||
|
||||
return self.send_payload(
|
||||
status,
|
||||
response_json(result),
|
||||
"application/json; charset=utf-8",
|
||||
send_body=True,
|
||||
)
|
||||
|
||||
def query_value(self, query: dict[str, list[str]], key: str) -> str:
|
||||
# 统一读取单值 query 参数。
|
||||
return str((query.get(key) or [""])[0]).strip()
|
||||
@ -6,8 +6,8 @@ import urllib.error
|
||||
import urllib.request
|
||||
import time
|
||||
|
||||
from .cache import TTLCache
|
||||
from .config import PROBE_CACHE_TTL_SECONDS, PROBE_MAX_WORKERS, PROBE_TIMEOUT_SECONDS, flatten_services, utc_now
|
||||
from core.cache import TTLCache
|
||||
from core.config import PROBE_CACHE_TTL_SECONDS, PROBE_MAX_WORKERS, PROBE_TIMEOUT_SECONDS, flatten_services, utc_now
|
||||
|
||||
|
||||
# 服务探测使用独立缓存,避免页面轮询直接把内网接口打爆。
|
||||
879
monitors/yumi/mongo_admin.py
Normal file
879
monitors/yumi/mongo_admin.py
Normal file
@ -0,0 +1,879 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from core.config import MONGO_DB_NAME, utc_now
|
||||
from .mongo_metrics import build_mongo_client
|
||||
|
||||
|
||||
# 文档浏览默认每页 100 条,和页面默认值保持一致。
|
||||
DEFAULT_PAGE_SIZE = 100
|
||||
# 单次最多放开到 200 条,避免把浏览器一次撑爆。
|
||||
MAX_PAGE_SIZE = 200
|
||||
# 浏览页默认按 _id 倒序,和大多数运维排查习惯一致。
|
||||
DEFAULT_SORT_SPEC = {"_id": -1}
|
||||
# 列表里的 JSON 预览文本做一个上限,避免整行被超长文档拉坏。
|
||||
PREVIEW_TEXT_LIMIT = 280
|
||||
# Mongo 常见索引类型统一在这里收口,页面允许直接传这些值。
|
||||
ALLOWED_INDEX_DIRECTIONS = {"text", "hashed", "2dsphere", "2d"}
|
||||
|
||||
|
||||
def require_bson_json_util() -> Any:
|
||||
try:
|
||||
# PyMongo 安装后会自带 bson 包,这里延迟导入避免缺依赖时整个服务起不来。
|
||||
from bson import json_util
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 管理页不可用时直接把依赖错误返回给上层。
|
||||
raise RuntimeError(f"bson unavailable: {exc}") from exc
|
||||
|
||||
# 返回 bson 的扩展 JSON 工具。
|
||||
return json_util
|
||||
|
||||
|
||||
def get_mongo_client() -> Any:
|
||||
# 统一复用现有 Mongo 建连逻辑,避免管理页和监控页配置口径分叉。
|
||||
client, error = build_mongo_client()
|
||||
|
||||
# 建连失败时直接抛统一错误。
|
||||
if client is None:
|
||||
raise RuntimeError(error)
|
||||
|
||||
# 先做一次 ping,尽早把连通性错误暴露出来。
|
||||
client.admin.command("ping")
|
||||
return client
|
||||
|
||||
|
||||
def clamp_page_size(value: Any, default: int = DEFAULT_PAGE_SIZE) -> int:
|
||||
try:
|
||||
# 页面传来的值可能是字符串,这里统一转整数。
|
||||
numeric = int(value)
|
||||
except Exception: # noqa: BLE001
|
||||
numeric = default
|
||||
|
||||
# 最终把页大小限制在一个安全区间内。
|
||||
return max(1, min(numeric, MAX_PAGE_SIZE))
|
||||
|
||||
|
||||
def clamp_page(value: Any, default: int = 1) -> int:
|
||||
try:
|
||||
# 页码同样允许前端传字符串。
|
||||
numeric = int(value)
|
||||
except Exception: # noqa: BLE001
|
||||
numeric = default
|
||||
|
||||
# 页码至少从 1 开始。
|
||||
return max(1, numeric)
|
||||
|
||||
|
||||
def require_name(value: Any, field_name: str) -> str:
|
||||
# 数据库名、集合名、索引名都按最基础的字符串校验处理。
|
||||
name = str(value or "").strip()
|
||||
|
||||
# 名称缺失时直接报错。
|
||||
if not name:
|
||||
raise ValueError(f"{field_name} is required")
|
||||
|
||||
# Mongo 名称里不允许带空字节。
|
||||
if "\x00" in name:
|
||||
raise ValueError(f"{field_name} contains invalid character")
|
||||
|
||||
return name
|
||||
|
||||
|
||||
def ensure_mutable_collection_name(name: str) -> str:
|
||||
collection_name = require_name(name, "collection")
|
||||
|
||||
# 系统集合默认不开放结构变更,避免误删系统元数据。
|
||||
if collection_name.startswith("system."):
|
||||
raise ValueError("system collections are read only in this page")
|
||||
|
||||
return collection_name
|
||||
|
||||
|
||||
def parse_extended_json(value: Any, field_name: str, *, default: Any = None) -> Any:
|
||||
# 空值时直接回退默认值,方便页面把空输入视作 {} 或 []。
|
||||
if value is None:
|
||||
return default
|
||||
|
||||
# 字符串输入按原文解析,避免前端先 stringify 再导致转义双重嵌套。
|
||||
if isinstance(value, str):
|
||||
text = value.strip()
|
||||
if not text:
|
||||
return default
|
||||
else:
|
||||
# 对象输入统一先转成标准 JSON 文本,再交给 bson 扩展解析。
|
||||
text = json.dumps(value, ensure_ascii=False)
|
||||
|
||||
try:
|
||||
return require_bson_json_util().loads(text)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise ValueError(f"invalid {field_name}: {exc}") from exc
|
||||
|
||||
|
||||
def dump_extended_json(value: Any, *, pretty: bool = False) -> str:
|
||||
helper = require_bson_json_util()
|
||||
options = getattr(helper, "RELAXED_JSON_OPTIONS", None)
|
||||
|
||||
# 列表页和编辑器都统一输出 relaxed extended JSON,前后端回写更稳定。
|
||||
return helper.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
indent=2 if pretty else None,
|
||||
json_options=options,
|
||||
)
|
||||
|
||||
|
||||
def to_builtin_json(value: Any) -> Any:
|
||||
# 先转扩展 JSON,再回到纯 Python 结构,这样 http_app 可以继续用标准 json.dumps。
|
||||
return json.loads(dump_extended_json(value, pretty=False))
|
||||
|
||||
|
||||
def compact_json_preview(value: Any, limit: int = PREVIEW_TEXT_LIMIT) -> str:
|
||||
# 预览统一压成单行,避免表格高度失控。
|
||||
compact = dump_extended_json(value, pretty=False).replace("\n", " ").strip()
|
||||
if len(compact) <= limit:
|
||||
return compact
|
||||
return f"{compact[:limit]}..."
|
||||
|
||||
|
||||
def mongo_document_row(document: dict[str, Any], *, page: int, position: int) -> dict[str, Any]:
|
||||
# 单条文档既返回结构化内容,也返回编辑器直接可用的 JSON 文本。
|
||||
document_json = dump_extended_json(document, pretty=True)
|
||||
document_id = document.get("_id")
|
||||
|
||||
return {
|
||||
"key": f"{page}-{position}-{compact_json_preview(document_id, limit=64)}",
|
||||
"idText": compact_json_preview(document_id, limit=72),
|
||||
"idFilterText": dump_extended_json({"_id": document_id}, pretty=True),
|
||||
"previewText": compact_json_preview(document),
|
||||
"jsonText": document_json,
|
||||
"document": to_builtin_json(document),
|
||||
}
|
||||
|
||||
|
||||
def normalize_sort_spec(sort_value: Any) -> list[tuple[str, int]]:
|
||||
parsed = parse_extended_json(sort_value, "sort", default=DEFAULT_SORT_SPEC)
|
||||
|
||||
# 浏览页主路径只支持对象排序,界面也按这个格式产出。
|
||||
if not isinstance(parsed, dict):
|
||||
raise ValueError("sort must be a json object")
|
||||
|
||||
items: list[tuple[str, int]] = []
|
||||
for field_name, direction in parsed.items():
|
||||
name = str(field_name or "").strip()
|
||||
if not name:
|
||||
continue
|
||||
|
||||
try:
|
||||
numeric_direction = int(direction)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise ValueError(f"invalid sort direction for {name}") from exc
|
||||
|
||||
if numeric_direction not in {-1, 1}:
|
||||
raise ValueError(f"sort direction for {name} must be 1 or -1")
|
||||
|
||||
items.append((name, numeric_direction))
|
||||
|
||||
# 排序对象全空时回退默认值。
|
||||
return items or list(DEFAULT_SORT_SPEC.items())
|
||||
|
||||
|
||||
def normalize_index_keys(keys_value: Any) -> list[tuple[str, Any]]:
|
||||
parsed = parse_extended_json(keys_value, "keys", default={})
|
||||
|
||||
# 索引创建页同样只接受对象格式。
|
||||
if not isinstance(parsed, dict):
|
||||
raise ValueError("keys must be a json object")
|
||||
|
||||
keys: list[tuple[str, Any]] = []
|
||||
for field_name, direction in parsed.items():
|
||||
name = str(field_name or "").strip()
|
||||
if not name:
|
||||
continue
|
||||
|
||||
if isinstance(direction, str):
|
||||
normalized_direction = direction.strip().lower()
|
||||
if normalized_direction in ALLOWED_INDEX_DIRECTIONS:
|
||||
keys.append((name, normalized_direction))
|
||||
continue
|
||||
|
||||
try:
|
||||
numeric_direction = int(normalized_direction)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise ValueError(
|
||||
f"index direction for {name} must be 1, -1, text, hashed, 2dsphere or 2d",
|
||||
) from exc
|
||||
else:
|
||||
try:
|
||||
numeric_direction = int(direction)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise ValueError(
|
||||
f"index direction for {name} must be 1, -1, text, hashed, 2dsphere or 2d",
|
||||
) from exc
|
||||
|
||||
if numeric_direction not in {-1, 1}:
|
||||
raise ValueError(f"index direction for {name} must be 1, -1, text, hashed, 2dsphere or 2d")
|
||||
|
||||
keys.append((name, numeric_direction))
|
||||
|
||||
if not keys:
|
||||
raise ValueError("keys must not be empty")
|
||||
|
||||
return keys
|
||||
|
||||
|
||||
def require_json_object(value: Any, field_name: str, *, default: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
parsed = parse_extended_json(value, field_name, default=default)
|
||||
if parsed is None:
|
||||
return {}
|
||||
|
||||
# filter / projection / options / replacement 都要求对象。
|
||||
if not isinstance(parsed, dict):
|
||||
raise ValueError(f"{field_name} must be a json object")
|
||||
|
||||
return parsed
|
||||
|
||||
|
||||
def require_json_array(value: Any, field_name: str, *, default: list[Any] | None = None) -> list[Any]:
|
||||
parsed = parse_extended_json(value, field_name, default=default)
|
||||
if parsed is None:
|
||||
return []
|
||||
|
||||
# aggregation pipeline 明确要求数组。
|
||||
if not isinstance(parsed, list):
|
||||
raise ValueError(f"{field_name} must be a json array")
|
||||
|
||||
return parsed
|
||||
|
||||
|
||||
def require_insert_documents(value: Any) -> tuple[list[dict[str, Any]], bool]:
|
||||
# Mongo 写入既支持单文档,也支持批量数组。
|
||||
parsed = parse_extended_json(value, "document", default={})
|
||||
|
||||
if isinstance(parsed, dict):
|
||||
return [parsed], False
|
||||
|
||||
if not isinstance(parsed, list):
|
||||
raise ValueError("document must be a json object or json array")
|
||||
|
||||
if not parsed:
|
||||
raise ValueError("document array must not be empty")
|
||||
|
||||
documents: list[dict[str, Any]] = []
|
||||
for index, item in enumerate(parsed, start=1):
|
||||
if not isinstance(item, dict):
|
||||
raise ValueError(f"document[{index}] must be a json object")
|
||||
documents.append(item)
|
||||
|
||||
return documents, True
|
||||
|
||||
|
||||
def ensure_safe_aggregate_pipeline(pipeline: list[Any]) -> list[Any]:
|
||||
for stage in pipeline:
|
||||
# 每个 stage 都必须是对象,否则驱动会直接报错,这里提前统一提示。
|
||||
if not isinstance(stage, dict):
|
||||
raise ValueError("aggregate pipeline stage must be a json object")
|
||||
|
||||
# 当前页面不开放会落地写数据的聚合阶段。
|
||||
stage_names = {str(key or "") for key in stage.keys()}
|
||||
if "$out" in stage_names or "$merge" in stage_names:
|
||||
raise ValueError("aggregate pipeline does not allow $out or $merge")
|
||||
|
||||
return pipeline
|
||||
|
||||
|
||||
def collection_handle(client: Any, database_name: str, collection_name: str) -> Any:
|
||||
# 集合句柄统一从这里拿,保证数据库名和集合名都先校验。
|
||||
return client[require_name(database_name, "database")][require_name(collection_name, "collection")]
|
||||
|
||||
|
||||
def collection_stats_payload(client: Any, database_name: str, collection_name: str) -> dict[str, Any]:
|
||||
database = client[require_name(database_name, "database")]
|
||||
collection = require_name(collection_name, "collection")
|
||||
|
||||
# collStats 能拿到集合大小、文档数、索引数等管理页最常用信息。
|
||||
stats = database.command("collStats", collection)
|
||||
index_list = list(database[collection].list_indexes())
|
||||
|
||||
indexes = []
|
||||
for item in index_list:
|
||||
key_spec = dict(item.get("key") or {})
|
||||
indexes.append(
|
||||
{
|
||||
"name": str(item.get("name") or "").strip(),
|
||||
"keys": to_builtin_json(key_spec),
|
||||
"keysText": dump_extended_json(key_spec, pretty=True),
|
||||
"unique": bool(item.get("unique")),
|
||||
"sparse": bool(item.get("sparse")),
|
||||
"hidden": bool(item.get("hidden")),
|
||||
"expireAfterSeconds": item.get("expireAfterSeconds"),
|
||||
"partialFilterExpression": to_builtin_json(item.get("partialFilterExpression"))
|
||||
if item.get("partialFilterExpression") is not None
|
||||
else None,
|
||||
"partialFilterExpressionText": dump_extended_json(item.get("partialFilterExpression"), pretty=True)
|
||||
if item.get("partialFilterExpression") is not None
|
||||
else "",
|
||||
}
|
||||
)
|
||||
|
||||
# 把 _id_ 放在第一行,其他索引按名称排序。
|
||||
indexes.sort(key=lambda item: (0 if item["name"] == "_id_" else 1, item["name"]))
|
||||
|
||||
return {
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"updatedAt": utc_now(),
|
||||
"stats": {
|
||||
"count": int(stats.get("count") or 0),
|
||||
"sizeMb": round(float(stats.get("size") or 0) / 1024 / 1024, 2),
|
||||
"storageSizeMb": round(float(stats.get("storageSize") or 0) / 1024 / 1024, 2),
|
||||
"avgObjSizeBytes": round(float(stats.get("avgObjSize") or 0), 2),
|
||||
"indexCount": int(stats.get("nindexes") or 0),
|
||||
"totalIndexSizeMb": round(float(stats.get("totalIndexSize") or 0) / 1024 / 1024, 2),
|
||||
"capped": bool(stats.get("capped")),
|
||||
"wiredTigerUri": str((stats.get("wiredTiger") or {}).get("uri") or "").strip(),
|
||||
},
|
||||
"indexes": indexes,
|
||||
}
|
||||
|
||||
|
||||
def get_mongo_admin_overview_payload() -> dict[str, Any]:
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
# 概览页既要列数据库,也要列集合,所以这里一次性把树形结构拼完。
|
||||
databases_raw = list((client.admin.command("listDatabases", 1).get("databases") or []))
|
||||
overview_items: list[dict[str, Any]] = []
|
||||
|
||||
for database_item in databases_raw:
|
||||
database_name = str(database_item.get("name") or "").strip()
|
||||
if not database_name:
|
||||
continue
|
||||
|
||||
database = client[database_name]
|
||||
collection_items: list[dict[str, Any]] = []
|
||||
|
||||
# listCollections 先拿到集合名和类型,再按需补充 stats。
|
||||
for collection_info in database.list_collections():
|
||||
collection_name = str(collection_info.get("name") or "").strip()
|
||||
if not collection_name:
|
||||
continue
|
||||
|
||||
try:
|
||||
collection_stats = database.command("collStats", collection_name)
|
||||
collection_items.append(
|
||||
{
|
||||
"name": collection_name,
|
||||
"type": str(collection_info.get("type") or "collection").strip() or "collection",
|
||||
"documentCount": int(collection_stats.get("count") or 0),
|
||||
"sizeMb": round(float(collection_stats.get("size") or 0) / 1024 / 1024, 2),
|
||||
"storageSizeMb": round(
|
||||
float(collection_stats.get("storageSize") or 0) / 1024 / 1024,
|
||||
2,
|
||||
),
|
||||
"indexCount": int(collection_stats.get("nindexes") or 0),
|
||||
"capped": bool(collection_stats.get("capped")),
|
||||
"error": "",
|
||||
}
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 某个集合取 stats 失败时仍保留在树里,方便继续点进去排查。
|
||||
collection_items.append(
|
||||
{
|
||||
"name": collection_name,
|
||||
"type": str(collection_info.get("type") or "collection").strip() or "collection",
|
||||
"documentCount": 0,
|
||||
"sizeMb": 0.0,
|
||||
"storageSizeMb": 0.0,
|
||||
"indexCount": 0,
|
||||
"capped": False,
|
||||
"error": str(exc),
|
||||
}
|
||||
)
|
||||
|
||||
# 集合列表默认按名字排序,侧边 explorer 更稳定。
|
||||
collection_items.sort(key=lambda item: item["name"].lower())
|
||||
|
||||
overview_items.append(
|
||||
{
|
||||
"name": database_name,
|
||||
"empty": bool(database_item.get("empty")),
|
||||
"sizeOnDiskMb": round(float(database_item.get("sizeOnDisk") or 0) / 1024 / 1024, 2),
|
||||
"collectionCount": len(collection_items),
|
||||
"collections": collection_items,
|
||||
}
|
||||
)
|
||||
|
||||
# 聚焦库优先,其余按磁盘占用倒序,和现有 Mongo 指标页一致。
|
||||
overview_items.sort(
|
||||
key=lambda item: (
|
||||
0 if item["name"] == MONGO_DB_NAME else 1,
|
||||
-float(item.get("sizeOnDiskMb") or 0.0),
|
||||
item["name"].lower(),
|
||||
)
|
||||
)
|
||||
|
||||
default_database = next((item["name"] for item in overview_items if item["name"] == MONGO_DB_NAME), "")
|
||||
if not default_database and overview_items:
|
||||
default_database = overview_items[0]["name"]
|
||||
|
||||
default_collection = ""
|
||||
if default_database:
|
||||
default_database_item = next((item for item in overview_items if item["name"] == default_database), None)
|
||||
if default_database_item and default_database_item["collections"]:
|
||||
default_collection = default_database_item["collections"][0]["name"]
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"focusedDatabase": MONGO_DB_NAME,
|
||||
"selectedDatabase": default_database,
|
||||
"selectedCollection": default_collection,
|
||||
"items": overview_items,
|
||||
}
|
||||
finally:
|
||||
# 管理页操作完成后立即关连接,避免长期挂着无用游标。
|
||||
client.close()
|
||||
|
||||
|
||||
def get_mongo_collection_detail_payload(database_name: str, collection_name: str) -> dict[str, Any]:
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
return collection_stats_payload(client, database_name, collection_name)
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def query_mongo_documents_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = require_name(payload.get("collection"), "collection")
|
||||
page = clamp_page(payload.get("page"), default=1)
|
||||
page_size = clamp_page_size(payload.get("pageSize"), default=DEFAULT_PAGE_SIZE)
|
||||
filter_spec = require_json_object(payload.get("filter"), "filter", default={})
|
||||
projection_spec = require_json_object(payload.get("projection"), "projection", default={})
|
||||
sort_spec = normalize_sort_spec(payload.get("sort"))
|
||||
|
||||
# 页面上的编辑和删除都依赖 _id,所以这里强制保留。
|
||||
if projection_spec.get("_id") == 0:
|
||||
projection_spec = dict(projection_spec)
|
||||
projection_spec.pop("_id", None)
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
collection = collection_handle(client, database_name, collection_name)
|
||||
total = collection.count_documents(filter_spec)
|
||||
skip = max(page - 1, 0) * page_size
|
||||
|
||||
cursor = collection.find(filter_spec, projection_spec or None).sort(sort_spec).skip(skip).limit(page_size)
|
||||
documents = list(cursor)
|
||||
rows = [
|
||||
mongo_document_row(document, page=page, position=skip + index)
|
||||
for index, document in enumerate(documents, start=1)
|
||||
]
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"page": page,
|
||||
"pageSize": page_size,
|
||||
"total": total,
|
||||
"pageCount": max((total + page_size - 1) // page_size, 1),
|
||||
"filterText": dump_extended_json(filter_spec, pretty=True),
|
||||
"projectionText": dump_extended_json(projection_spec, pretty=True),
|
||||
"sortText": dump_extended_json(dict(sort_spec), pretty=True),
|
||||
"items": rows,
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def insert_mongo_document_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = ensure_mutable_collection_name(payload.get("collection"))
|
||||
documents, many = require_insert_documents(payload.get("document"))
|
||||
ordered = bool(payload.get("ordered", True))
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
collection = collection_handle(client, database_name, collection_name)
|
||||
if many:
|
||||
result = collection.insert_many(documents, ordered=ordered)
|
||||
inserted_ids = [to_builtin_json(inserted_id) for inserted_id in result.inserted_ids]
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"insertedCount": len(inserted_ids),
|
||||
"insertedIds": inserted_ids,
|
||||
"message": "insertMany completed",
|
||||
}
|
||||
|
||||
result = collection.insert_one(documents[0])
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"insertedCount": 1,
|
||||
"insertedId": to_builtin_json(result.inserted_id),
|
||||
"insertedIdText": dump_extended_json(result.inserted_id, pretty=False),
|
||||
"message": "insertOne completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def replace_mongo_document_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = ensure_mutable_collection_name(payload.get("collection"))
|
||||
filter_spec = require_json_object(payload.get("filter"), "filter", default={})
|
||||
replacement = require_json_object(payload.get("document"), "document")
|
||||
upsert = bool(payload.get("upsert"))
|
||||
|
||||
# replaceOne 只接受完整文档,不能混进 $set 这类更新操作符。
|
||||
if any(str(key).startswith("$") for key in replacement.keys()):
|
||||
raise ValueError("replace document must not contain update operators")
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
result = collection_handle(client, database_name, collection_name).replace_one(
|
||||
filter_spec,
|
||||
replacement,
|
||||
upsert=upsert,
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"matchedCount": int(result.matched_count),
|
||||
"modifiedCount": int(result.modified_count),
|
||||
"upsertedId": to_builtin_json(result.upserted_id) if result.upserted_id is not None else None,
|
||||
"message": "replaceOne completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def update_mongo_documents_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = ensure_mutable_collection_name(payload.get("collection"))
|
||||
filter_spec = require_json_object(payload.get("filter"), "filter", default={})
|
||||
update_spec = parse_extended_json(payload.get("update"), "update", default={})
|
||||
many = bool(payload.get("many"))
|
||||
upsert = bool(payload.get("upsert"))
|
||||
allow_empty_filter = bool(payload.get("allowEmptyFilter"))
|
||||
|
||||
# 空 filter + 多文档更新风险过高,前端必须显式确认。
|
||||
if many and not filter_spec and not allow_empty_filter:
|
||||
raise ValueError("empty filter requires explicit confirmation")
|
||||
|
||||
# update 支持操作符对象或更新管道数组。
|
||||
if isinstance(update_spec, dict):
|
||||
if not update_spec:
|
||||
raise ValueError("update must not be empty")
|
||||
if not any(str(key).startswith("$") for key in update_spec.keys()):
|
||||
raise ValueError("update must contain update operators or use replace")
|
||||
elif isinstance(update_spec, list):
|
||||
ensure_safe_aggregate_pipeline(update_spec)
|
||||
else:
|
||||
raise ValueError("update must be a json object or json array")
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
collection = collection_handle(client, database_name, collection_name)
|
||||
result = (
|
||||
collection.update_many(filter_spec, update_spec, upsert=upsert)
|
||||
if many
|
||||
else collection.update_one(filter_spec, update_spec, upsert=upsert)
|
||||
)
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"matchedCount": int(result.matched_count),
|
||||
"modifiedCount": int(result.modified_count),
|
||||
"upsertedId": to_builtin_json(result.upserted_id) if result.upserted_id is not None else None,
|
||||
"message": "updateMany completed" if many else "updateOne completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def delete_mongo_documents_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = ensure_mutable_collection_name(payload.get("collection"))
|
||||
filter_spec = require_json_object(payload.get("filter"), "filter", default={})
|
||||
many = bool(payload.get("many"))
|
||||
allow_empty_filter = bool(payload.get("allowEmptyFilter"))
|
||||
|
||||
# deleteMany 空 filter 同样要求显式确认。
|
||||
if many and not filter_spec and not allow_empty_filter:
|
||||
raise ValueError("empty filter requires explicit confirmation")
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
collection = collection_handle(client, database_name, collection_name)
|
||||
result = collection.delete_many(filter_spec) if many else collection.delete_one(filter_spec)
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"deletedCount": int(result.deleted_count),
|
||||
"message": "deleteMany completed" if many else "deleteOne completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def aggregate_mongo_documents_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = require_name(payload.get("collection"), "collection")
|
||||
page = clamp_page(payload.get("page"), default=1)
|
||||
page_size = clamp_page_size(payload.get("pageSize"), default=DEFAULT_PAGE_SIZE)
|
||||
pipeline = ensure_safe_aggregate_pipeline(require_json_array(payload.get("pipeline"), "pipeline", default=[]))
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
collection = collection_handle(client, database_name, collection_name)
|
||||
skip = max(page - 1, 0) * page_size
|
||||
|
||||
# 先跑计数管道,再跑当前页数据。
|
||||
total_result = list(collection.aggregate([*pipeline, {"$count": "total"}]))
|
||||
total = int((total_result[0] or {}).get("total") or 0) if total_result else 0
|
||||
documents = list(collection.aggregate([*pipeline, {"$skip": skip}, {"$limit": page_size}]))
|
||||
rows = [
|
||||
{
|
||||
"key": f"agg-{page}-{skip + index}",
|
||||
"idText": compact_json_preview(document.get("_id"), limit=72) if "_id" in document else "-",
|
||||
"idFilterText": dump_extended_json({"_id": document.get("_id")}, pretty=True)
|
||||
if "_id" in document
|
||||
else "{}",
|
||||
"previewText": compact_json_preview(document),
|
||||
"jsonText": dump_extended_json(document, pretty=True),
|
||||
"document": to_builtin_json(document),
|
||||
}
|
||||
for index, document in enumerate(documents, start=1)
|
||||
]
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"page": page,
|
||||
"pageSize": page_size,
|
||||
"total": total,
|
||||
"pageCount": max((total + page_size - 1) // page_size, 1),
|
||||
"pipelineText": dump_extended_json(pipeline, pretty=True),
|
||||
"items": rows,
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def distinct_mongo_values_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = require_name(payload.get("collection"), "collection")
|
||||
field_name = require_name(payload.get("field"), "field")
|
||||
filter_spec = require_json_object(payload.get("filter"), "filter", default={})
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
values = collection_handle(client, database_name, collection_name).distinct(field_name, filter_spec)
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"field": field_name,
|
||||
"filterText": dump_extended_json(filter_spec, pretty=True),
|
||||
"count": len(values),
|
||||
"values": to_builtin_json(values),
|
||||
"valuesText": dump_extended_json(values, pretty=True),
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def create_mongo_index_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = ensure_mutable_collection_name(payload.get("collection"))
|
||||
index_keys = normalize_index_keys(payload.get("keys"))
|
||||
index_options = require_json_object(payload.get("options"), "options", default={})
|
||||
|
||||
# name 允许前端留空,驱动会自动生成。
|
||||
if not str(index_options.get("name") or "").strip():
|
||||
index_options.pop("name", None)
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
index_name = collection_handle(client, database_name, collection_name).create_index(index_keys, **index_options)
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"indexName": str(index_name),
|
||||
"message": "createIndex completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def drop_mongo_index_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = ensure_mutable_collection_name(payload.get("collection"))
|
||||
index_name = require_name(payload.get("indexName"), "indexName")
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
collection_handle(client, database_name, collection_name).drop_index(index_name)
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"indexName": index_name,
|
||||
"message": "dropIndex completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def drop_mongo_secondary_indexes_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = ensure_mutable_collection_name(payload.get("collection"))
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
collection = collection_handle(client, database_name, collection_name)
|
||||
index_names = [
|
||||
str(item.get("name") or "").strip()
|
||||
for item in collection.list_indexes()
|
||||
if str(item.get("name") or "").strip() and str(item.get("name") or "").strip() != "_id_"
|
||||
]
|
||||
|
||||
# Mongo 的 drop_indexes 会保留 _id_ 索引,这里先把删除名单回给前端方便确认结果。
|
||||
if not index_names:
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"droppedCount": 0,
|
||||
"droppedNames": [],
|
||||
"message": "no secondary indexes",
|
||||
}
|
||||
|
||||
collection.drop_indexes()
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"droppedCount": len(index_names),
|
||||
"droppedNames": index_names,
|
||||
"message": "dropIndexes completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def create_mongo_collection_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = ensure_mutable_collection_name(payload.get("collection"))
|
||||
options = require_json_object(payload.get("options"), "options", default={})
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
client[database_name].create_collection(collection_name, **options)
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"message": "createCollection completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def rename_mongo_collection_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
source_name = ensure_mutable_collection_name(payload.get("collection"))
|
||||
target_name = ensure_mutable_collection_name(payload.get("targetCollection"))
|
||||
drop_target = bool(payload.get("dropTarget"))
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
collection_handle(client, database_name, source_name).rename(target_name, dropTarget=drop_target)
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": source_name,
|
||||
"targetCollection": target_name,
|
||||
"message": "renameCollection completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def drop_mongo_collection_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
collection_name = ensure_mutable_collection_name(payload.get("collection"))
|
||||
confirm_name = require_name(payload.get("confirmName"), "confirmName")
|
||||
|
||||
# 删除集合前要求前端把目标名再传一遍,减少误触。
|
||||
if confirm_name != collection_name:
|
||||
raise ValueError("confirmName does not match collection")
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
collection_handle(client, database_name, collection_name).drop()
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"collection": collection_name,
|
||||
"message": "dropCollection completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def drop_mongo_database_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
database_name = require_name(payload.get("database"), "database")
|
||||
confirm_name = require_name(payload.get("confirmName"), "confirmName")
|
||||
|
||||
# 删除数据库同样要求二次校验目标名。
|
||||
if confirm_name != database_name:
|
||||
raise ValueError("confirmName does not match database")
|
||||
|
||||
client = get_mongo_client()
|
||||
|
||||
try:
|
||||
client.drop_database(database_name)
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": database_name,
|
||||
"message": "dropDatabase completed",
|
||||
}
|
||||
finally:
|
||||
client.close()
|
||||
@ -3,8 +3,8 @@ from __future__ import annotations
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from .cache import TTLCache
|
||||
from .config import (
|
||||
from core.cache import TTLCache
|
||||
from core.config import (
|
||||
MONGO_CACHE_TTL_SECONDS,
|
||||
MONGO_CONNECT_TIMEOUT_MS,
|
||||
MONGO_DB_NAME,
|
||||
719
monitors/yumi/mysql_admin.py
Normal file
719
monitors/yumi/mysql_admin.py
Normal file
@ -0,0 +1,719 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from core.config import MYSQL_DEFAULT_DATABASE, MYSQL_DEFAULT_PAGE_SIZE, MYSQL_MAX_PAGE_SIZE, MYSQL_QUERY_MAX_ROWS, utc_now
|
||||
from .mysql_common import (
|
||||
build_mysql_connection,
|
||||
build_row_identity,
|
||||
database_exists,
|
||||
get_preferred_identity_columns,
|
||||
list_database_summaries,
|
||||
list_table_columns,
|
||||
list_table_indexes,
|
||||
list_table_summaries,
|
||||
normalize_column_name,
|
||||
normalize_database_name,
|
||||
normalize_mysql_row,
|
||||
normalize_sql_text,
|
||||
normalize_table_name,
|
||||
quote_identifier,
|
||||
sql_statement_kind,
|
||||
table_exists,
|
||||
)
|
||||
from .mysql_metrics import clear_mysql_metrics_cache, get_mysql_overview_payload
|
||||
|
||||
|
||||
def browser_error_payload(error: str) -> dict[str, Any]:
|
||||
# 浏览器和 SQL 工作台都统一返回这套最小错误结构。
|
||||
return {"ok": False, "updatedAt": utc_now(), "error": error}
|
||||
|
||||
|
||||
def clamp_page_size(value: Any) -> int:
|
||||
try:
|
||||
# 前端没传时回到默认 100;过大时强制截断,避免一次拖太多数据。
|
||||
return max(1, min(int(value or MYSQL_DEFAULT_PAGE_SIZE), MYSQL_MAX_PAGE_SIZE))
|
||||
except Exception: # noqa: BLE001
|
||||
return MYSQL_DEFAULT_PAGE_SIZE
|
||||
|
||||
|
||||
def clamp_page_number(value: Any) -> int:
|
||||
try:
|
||||
# 页码最小固定为 1。
|
||||
return max(1, int(value or 1))
|
||||
except Exception: # noqa: BLE001
|
||||
return 1
|
||||
|
||||
|
||||
def normalize_sql_fragment(value: Any, *, field_name: str) -> str:
|
||||
# DDL 片段允许空格和括号,但不允许多语句。
|
||||
actual = str(value or "").strip()
|
||||
if not actual:
|
||||
raise ValueError(f"{field_name} is required")
|
||||
if ";" in actual:
|
||||
raise ValueError(f"{field_name} must be a single fragment")
|
||||
return actual
|
||||
|
||||
|
||||
def get_mysql_databases_payload() -> dict[str, Any]:
|
||||
# 概览里已经做了缓存,这里直接复用,避免重复查 information_schema。
|
||||
overview = get_mysql_overview_payload()
|
||||
if not overview.get("ok"):
|
||||
return {
|
||||
"ok": False,
|
||||
"updatedAt": overview.get("updatedAt"),
|
||||
"selectedDatabase": "",
|
||||
"items": [],
|
||||
"error": overview.get("error") or "mysql unavailable",
|
||||
}
|
||||
|
||||
items = list(overview.get("databases") or [])
|
||||
business_items = [item for item in items if not item.get("system")]
|
||||
preferred = str(MYSQL_DEFAULT_DATABASE or "").strip()
|
||||
|
||||
# 默认优先选 .env 指定库;没有时选第一个业务库,再退到第一个可见库。
|
||||
selected_database = (
|
||||
preferred
|
||||
if preferred and any(item.get("name") == preferred for item in items)
|
||||
else (business_items[0]["name"] if business_items else (items[0]["name"] if items else ""))
|
||||
)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": overview.get("updatedAt"),
|
||||
"selectedDatabase": selected_database,
|
||||
"items": items,
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def get_mysql_tables_payload(database: str) -> dict[str, Any]:
|
||||
target_database = normalize_database_name(database)
|
||||
|
||||
try:
|
||||
with build_mysql_connection(autocommit=True) as connection:
|
||||
if not database_exists(connection, target_database):
|
||||
raise ValueError(f"database not found: {target_database}")
|
||||
items = list_table_summaries(connection, target_database)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {
|
||||
"ok": False,
|
||||
"updatedAt": utc_now(),
|
||||
"database": target_database,
|
||||
"items": [],
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": target_database,
|
||||
"selectedTable": items[0]["name"] if items else "",
|
||||
"items": items,
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def get_mysql_table_schema_payload(database: str, table: str) -> dict[str, Any]:
|
||||
target_database = normalize_database_name(database)
|
||||
target_table = normalize_table_name(table)
|
||||
|
||||
try:
|
||||
with build_mysql_connection(target_database, autocommit=True) as connection:
|
||||
if not table_exists(connection, target_database, target_table):
|
||||
raise ValueError(f"table not found: {target_database}.{target_table}")
|
||||
|
||||
tables = list_table_summaries(connection, target_database)
|
||||
table_summary = next((item for item in tables if item.get("name") == target_table), None)
|
||||
columns = list_table_columns(connection, target_database, target_table)
|
||||
indexes = list_table_indexes(connection, target_database, target_table)
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# SHOW CREATE TABLE 能拿到最完整的 DDL,方便页面直接展示和复制。
|
||||
cursor.execute(f"SHOW CREATE TABLE {quote_identifier(target_table)}")
|
||||
create_row = cursor.fetchone() or {}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {
|
||||
"ok": False,
|
||||
"updatedAt": utc_now(),
|
||||
"database": target_database,
|
||||
"table": target_table,
|
||||
"columns": [],
|
||||
"indexes": [],
|
||||
"createTable": "",
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": target_database,
|
||||
"table": target_table,
|
||||
"tableSummary": table_summary or {},
|
||||
"columns": columns,
|
||||
"indexes": indexes,
|
||||
"identity": get_preferred_identity_columns(columns, indexes),
|
||||
"createTable": str(create_row.get("Create Table") or "").strip(),
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def build_filter_clause(
|
||||
columns: list[dict[str, Any]],
|
||||
keyword: str,
|
||||
filters: list[dict[str, Any]],
|
||||
) -> tuple[str, list[Any]]:
|
||||
column_names = {column["name"] for column in columns}
|
||||
clauses: list[str] = []
|
||||
params: list[Any] = []
|
||||
normalized_keyword = str(keyword or "").strip()
|
||||
|
||||
# 关键字搜索默认对当前表全部列做模糊匹配,适合值班快速扫数据。
|
||||
if normalized_keyword:
|
||||
search_clauses: list[str] = []
|
||||
for column in columns:
|
||||
search_clauses.append(f"CAST({quote_identifier(column['name'])} AS CHAR) LIKE %s")
|
||||
params.append(f"%{normalized_keyword}%")
|
||||
if search_clauses:
|
||||
clauses.append(f"({' OR '.join(search_clauses)})")
|
||||
|
||||
# 附加过滤器支持前端后续扩展成多条件。
|
||||
for item in filters:
|
||||
column_name = normalize_column_name(item.get("column"))
|
||||
if column_name not in column_names:
|
||||
raise ValueError(f"unknown filter column: {column_name}")
|
||||
|
||||
operator = str(item.get("operator") or "eq").strip().lower()
|
||||
sql_column = quote_identifier(column_name)
|
||||
value = item.get("value")
|
||||
|
||||
if operator == "eq":
|
||||
if value is None:
|
||||
clauses.append(f"{sql_column} IS NULL")
|
||||
else:
|
||||
clauses.append(f"{sql_column} = %s")
|
||||
params.append(value)
|
||||
continue
|
||||
if operator == "ne":
|
||||
if value is None:
|
||||
clauses.append(f"{sql_column} IS NOT NULL")
|
||||
else:
|
||||
clauses.append(f"{sql_column} <> %s")
|
||||
params.append(value)
|
||||
continue
|
||||
if operator == "gt":
|
||||
clauses.append(f"{sql_column} > %s")
|
||||
params.append(value)
|
||||
continue
|
||||
if operator == "ge":
|
||||
clauses.append(f"{sql_column} >= %s")
|
||||
params.append(value)
|
||||
continue
|
||||
if operator == "lt":
|
||||
clauses.append(f"{sql_column} < %s")
|
||||
params.append(value)
|
||||
continue
|
||||
if operator == "le":
|
||||
clauses.append(f"{sql_column} <= %s")
|
||||
params.append(value)
|
||||
continue
|
||||
if operator == "in":
|
||||
if not isinstance(value, list) or not value:
|
||||
raise ValueError("in filter requires a non-empty array value")
|
||||
placeholders = ", ".join(["%s"] * len(value))
|
||||
clauses.append(f"{sql_column} IN ({placeholders})")
|
||||
params.extend(value)
|
||||
continue
|
||||
if operator == "not_in":
|
||||
if not isinstance(value, list) or not value:
|
||||
raise ValueError("not_in filter requires a non-empty array value")
|
||||
placeholders = ", ".join(["%s"] * len(value))
|
||||
clauses.append(f"{sql_column} NOT IN ({placeholders})")
|
||||
params.extend(value)
|
||||
continue
|
||||
if operator == "between":
|
||||
if not isinstance(value, list) or len(value) != 2:
|
||||
raise ValueError("between filter requires a two-item array value")
|
||||
clauses.append(f"{sql_column} BETWEEN %s AND %s")
|
||||
params.extend(value)
|
||||
continue
|
||||
if operator == "not_between":
|
||||
if not isinstance(value, list) or len(value) != 2:
|
||||
raise ValueError("not_between filter requires a two-item array value")
|
||||
clauses.append(f"{sql_column} NOT BETWEEN %s AND %s")
|
||||
params.extend(value)
|
||||
continue
|
||||
if operator == "like":
|
||||
clauses.append(f"CAST({sql_column} AS CHAR) LIKE %s")
|
||||
params.append(f"%{value}%")
|
||||
continue
|
||||
if operator == "not_like":
|
||||
clauses.append(f"CAST({sql_column} AS CHAR) NOT LIKE %s")
|
||||
params.append(f"%{value}%")
|
||||
continue
|
||||
if operator == "is_null":
|
||||
clauses.append(f"{sql_column} IS NULL")
|
||||
continue
|
||||
if operator == "is_not_null":
|
||||
clauses.append(f"{sql_column} IS NOT NULL")
|
||||
continue
|
||||
|
||||
raise ValueError(f"unsupported filter operator: {operator}")
|
||||
|
||||
if not clauses:
|
||||
return "", params
|
||||
return f"WHERE {' AND '.join(clauses)}", params
|
||||
|
||||
|
||||
def query_mysql_table_rows_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
target_database = normalize_database_name(payload.get("database"))
|
||||
target_table = normalize_table_name(payload.get("table"))
|
||||
page = clamp_page_number(payload.get("page"))
|
||||
page_size = clamp_page_size(payload.get("pageSize"))
|
||||
keyword = str(payload.get("keyword") or "").strip()
|
||||
filters = payload.get("filters") if isinstance(payload.get("filters"), list) else []
|
||||
|
||||
try:
|
||||
with build_mysql_connection(target_database, autocommit=True) as connection:
|
||||
if not table_exists(connection, target_database, target_table):
|
||||
raise ValueError(f"table not found: {target_database}.{target_table}")
|
||||
|
||||
columns = list_table_columns(connection, target_database, target_table)
|
||||
indexes = list_table_indexes(connection, target_database, target_table)
|
||||
identity_spec = get_preferred_identity_columns(columns, indexes)
|
||||
column_names = {column["name"] for column in columns}
|
||||
|
||||
requested_sort = str(payload.get("orderBy") or "").strip()
|
||||
order_by = requested_sort if requested_sort in column_names else (identity_spec["columns"][0] if identity_spec["columns"] else columns[0]["name"])
|
||||
order_direction = "DESC" if str(payload.get("orderDirection") or "").strip().lower() == "desc" else "ASC"
|
||||
|
||||
where_clause, params = build_filter_clause(columns, keyword, filters)
|
||||
offset = (page - 1) * page_size
|
||||
quoted_table = quote_identifier(target_table)
|
||||
select_columns = ", ".join(quote_identifier(column["name"]) for column in columns)
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# 先统计总量,分页器和页数都依赖这个值。
|
||||
cursor.execute(
|
||||
f"SELECT COUNT(*) AS totalRows FROM {quoted_table} {where_clause}",
|
||||
tuple(params),
|
||||
)
|
||||
total_rows = int((cursor.fetchone() or {}).get("totalRows") or 0)
|
||||
|
||||
# 行浏览默认按指定列排序,避免翻页时顺序漂移。
|
||||
cursor.execute(
|
||||
f"""
|
||||
SELECT {select_columns}
|
||||
FROM {quoted_table}
|
||||
{where_clause}
|
||||
ORDER BY {quote_identifier(order_by)} {order_direction}
|
||||
LIMIT %s OFFSET %s
|
||||
""",
|
||||
tuple([*params, page_size, offset]),
|
||||
)
|
||||
rows = cursor.fetchall() or []
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {
|
||||
"ok": False,
|
||||
"updatedAt": utc_now(),
|
||||
"database": target_database,
|
||||
"table": target_table,
|
||||
"rows": [],
|
||||
"totalRows": 0,
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
# 每一行都带 identity,后续 update / delete 不需要前端自己猜 where 条件。
|
||||
normalized_rows = []
|
||||
for row in rows:
|
||||
normalized_rows.append(
|
||||
{
|
||||
"values": normalize_mysql_row(row),
|
||||
"identity": build_row_identity(row, identity_spec),
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": target_database,
|
||||
"table": target_table,
|
||||
"page": page,
|
||||
"pageSize": page_size,
|
||||
"totalRows": total_rows,
|
||||
"totalPages": max(1, (total_rows + page_size - 1) // page_size) if total_rows > 0 else 1,
|
||||
"keyword": keyword,
|
||||
"orderBy": order_by,
|
||||
"orderDirection": order_direction.lower(),
|
||||
"columns": columns,
|
||||
"indexes": indexes,
|
||||
"identity": identity_spec,
|
||||
"rows": normalized_rows,
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def normalize_write_values(values: Any, columns: list[dict[str, Any]]) -> list[tuple[str, Any]]:
|
||||
if not isinstance(values, dict):
|
||||
raise ValueError("values must be an object")
|
||||
|
||||
writable_columns = {
|
||||
column["name"]: column
|
||||
for column in columns
|
||||
if "generated" not in str(column.get("extra") or "").strip().lower()
|
||||
}
|
||||
normalized: list[tuple[str, Any]] = []
|
||||
|
||||
# 只允许写入真实存在的列,避免页面或调用方传进来脏字段。
|
||||
for raw_name, raw_value in values.items():
|
||||
column_name = normalize_column_name(raw_name)
|
||||
if column_name not in writable_columns:
|
||||
raise ValueError(f"unknown writable column: {column_name}")
|
||||
|
||||
# JSON 对象和数组转成字符串后再交给 MySQL,自定义 SQL 控制台仍可手写更复杂格式。
|
||||
if isinstance(raw_value, (dict, list)):
|
||||
normalized.append((column_name, json.dumps(raw_value, ensure_ascii=False)))
|
||||
else:
|
||||
normalized.append((column_name, raw_value))
|
||||
|
||||
return normalized
|
||||
|
||||
|
||||
def build_identity_clause(identity: dict[str, Any], columns: list[dict[str, Any]]) -> tuple[str, list[Any]]:
|
||||
if not isinstance(identity, dict):
|
||||
raise ValueError("identity is required")
|
||||
|
||||
allowed_columns = {column["name"] for column in columns}
|
||||
identity_columns = [normalize_column_name(item) for item in identity.get("columns") or []]
|
||||
identity_values = identity.get("values") if isinstance(identity.get("values"), dict) else {}
|
||||
|
||||
if not identity_columns:
|
||||
raise ValueError("identity columns are required")
|
||||
|
||||
clauses: list[str] = []
|
||||
params: list[Any] = []
|
||||
|
||||
# where 条件只按 identity 指定列构造,避免把整行都塞进 where。
|
||||
for column_name in identity_columns:
|
||||
if column_name not in allowed_columns:
|
||||
raise ValueError(f"unknown identity column: {column_name}")
|
||||
if column_name not in identity_values:
|
||||
raise ValueError(f"missing identity value: {column_name}")
|
||||
|
||||
value = identity_values.get(column_name)
|
||||
quoted = quote_identifier(column_name)
|
||||
if value is None:
|
||||
clauses.append(f"{quoted} IS NULL")
|
||||
else:
|
||||
clauses.append(f"{quoted} = %s")
|
||||
params.append(value)
|
||||
|
||||
return " AND ".join(clauses), params
|
||||
|
||||
|
||||
def insert_mysql_row_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
target_database = normalize_database_name(payload.get("database"))
|
||||
target_table = normalize_table_name(payload.get("table"))
|
||||
|
||||
try:
|
||||
with build_mysql_connection(target_database, autocommit=False) as connection:
|
||||
columns = list_table_columns(connection, target_database, target_table)
|
||||
pairs = normalize_write_values(payload.get("values"), columns)
|
||||
if not pairs:
|
||||
raise ValueError("at least one column value is required")
|
||||
|
||||
quoted_table = quote_identifier(target_table)
|
||||
column_sql = ", ".join(quote_identifier(name) for name, _ in pairs)
|
||||
placeholders = ", ".join(["%s"] * len(pairs))
|
||||
params = [value for _, value in pairs]
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# 插入时只拼接列名,值全部走参数化。
|
||||
cursor.execute(
|
||||
f"INSERT INTO {quoted_table} ({column_sql}) VALUES ({placeholders})",
|
||||
tuple(params),
|
||||
)
|
||||
affected_rows = int(cursor.rowcount or 0)
|
||||
last_insert_id = cursor.lastrowid
|
||||
connection.commit()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return browser_error_payload(str(exc))
|
||||
|
||||
clear_mysql_metrics_cache()
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"message": f"已插入 {target_database}.{target_table}",
|
||||
"affectedRows": affected_rows,
|
||||
"lastInsertId": last_insert_id,
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def update_mysql_row_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
target_database = normalize_database_name(payload.get("database"))
|
||||
target_table = normalize_table_name(payload.get("table"))
|
||||
|
||||
try:
|
||||
with build_mysql_connection(target_database, autocommit=False) as connection:
|
||||
columns = list_table_columns(connection, target_database, target_table)
|
||||
pairs = normalize_write_values(payload.get("values"), columns)
|
||||
if not pairs:
|
||||
raise ValueError("at least one column value is required")
|
||||
|
||||
where_clause, where_params = build_identity_clause(payload.get("identity"), columns)
|
||||
set_clause = ", ".join(f"{quote_identifier(name)} = %s" for name, _ in pairs)
|
||||
params = [value for _, value in pairs] + where_params
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# 更新固定只影响一行;identity 使用主键或唯一索引时语义最稳定。
|
||||
cursor.execute(
|
||||
f"UPDATE {quote_identifier(target_table)} SET {set_clause} WHERE {where_clause} LIMIT 1",
|
||||
tuple(params),
|
||||
)
|
||||
affected_rows = int(cursor.rowcount or 0)
|
||||
connection.commit()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return browser_error_payload(str(exc))
|
||||
|
||||
clear_mysql_metrics_cache()
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"message": f"已更新 {target_database}.{target_table}",
|
||||
"affectedRows": affected_rows,
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def delete_mysql_row_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
target_database = normalize_database_name(payload.get("database"))
|
||||
target_table = normalize_table_name(payload.get("table"))
|
||||
|
||||
try:
|
||||
with build_mysql_connection(target_database, autocommit=False) as connection:
|
||||
columns = list_table_columns(connection, target_database, target_table)
|
||||
where_clause, where_params = build_identity_clause(payload.get("identity"), columns)
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# 删除同样只允许命中一行,避免前端误带了过宽的 where 条件。
|
||||
cursor.execute(
|
||||
f"DELETE FROM {quote_identifier(target_table)} WHERE {where_clause} LIMIT 1",
|
||||
tuple(where_params),
|
||||
)
|
||||
affected_rows = int(cursor.rowcount or 0)
|
||||
connection.commit()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return browser_error_payload(str(exc))
|
||||
|
||||
clear_mysql_metrics_cache()
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"message": f"已删除 {target_database}.{target_table} 中的 1 行",
|
||||
"affectedRows": affected_rows,
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def add_mysql_column_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
target_database = normalize_database_name(payload.get("database"))
|
||||
target_table = normalize_table_name(payload.get("table"))
|
||||
column_name = normalize_column_name(payload.get("columnName"))
|
||||
column_type = normalize_sql_fragment(payload.get("columnType"), field_name="columnType")
|
||||
nullable = bool(payload.get("nullable", True))
|
||||
comment = str(payload.get("comment") or "").strip()
|
||||
position = str(payload.get("position") or "last").strip().lower()
|
||||
default_mode = str(payload.get("defaultMode") or "none").strip().lower()
|
||||
default_value = payload.get("defaultValue")
|
||||
|
||||
clauses = [
|
||||
f"ALTER TABLE {quote_identifier(target_table)} ADD COLUMN {quote_identifier(column_name)} {column_type}",
|
||||
"NULL" if nullable else "NOT NULL",
|
||||
]
|
||||
params: list[Any] = []
|
||||
|
||||
# 默认值支持最常用的三种模式;更复杂表达式留给 SQL 工作台。
|
||||
if default_mode == "literal":
|
||||
clauses.append("DEFAULT %s")
|
||||
params.append(default_value)
|
||||
elif default_mode == "null":
|
||||
clauses.append("DEFAULT NULL")
|
||||
elif default_mode != "none":
|
||||
raise ValueError(f"unsupported default mode: {default_mode}")
|
||||
|
||||
if comment:
|
||||
clauses.append("COMMENT %s")
|
||||
params.append(comment)
|
||||
|
||||
if position == "first":
|
||||
clauses.append("FIRST")
|
||||
elif position == "after":
|
||||
after_column = normalize_column_name(payload.get("afterColumn"))
|
||||
clauses.append(f"AFTER {quote_identifier(after_column)}")
|
||||
elif position != "last":
|
||||
raise ValueError(f"unsupported position: {position}")
|
||||
|
||||
try:
|
||||
with build_mysql_connection(target_database, autocommit=False) as connection:
|
||||
with connection.cursor() as cursor:
|
||||
# 列类型本身是 SQL 语法片段,只做单语句校验,不做过度限制。
|
||||
cursor.execute(" ".join(clauses), tuple(params))
|
||||
connection.commit()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return browser_error_payload(str(exc))
|
||||
|
||||
clear_mysql_metrics_cache()
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"message": f"已新增列 {target_table}.{column_name}",
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def drop_mysql_column_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
target_database = normalize_database_name(payload.get("database"))
|
||||
target_table = normalize_table_name(payload.get("table"))
|
||||
column_name = normalize_column_name(payload.get("columnName"))
|
||||
|
||||
try:
|
||||
with build_mysql_connection(target_database, autocommit=False) as connection:
|
||||
with connection.cursor() as cursor:
|
||||
# 删除列是高影响操作,但用户明确要求支持,这里保持单列单次执行。
|
||||
cursor.execute(
|
||||
f"ALTER TABLE {quote_identifier(target_table)} DROP COLUMN {quote_identifier(column_name)}",
|
||||
)
|
||||
connection.commit()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return browser_error_payload(str(exc))
|
||||
|
||||
clear_mysql_metrics_cache()
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"message": f"已删除列 {target_table}.{column_name}",
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def add_mysql_index_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
target_database = normalize_database_name(payload.get("database"))
|
||||
target_table = normalize_table_name(payload.get("table"))
|
||||
index_name = normalize_column_name(payload.get("indexName"))
|
||||
index_mode = str(payload.get("indexMode") or "normal").strip().lower()
|
||||
columns = payload.get("columns") if isinstance(payload.get("columns"), list) else []
|
||||
normalized_columns = [normalize_column_name(item) for item in columns]
|
||||
|
||||
if not normalized_columns:
|
||||
raise ValueError("at least one index column is required")
|
||||
|
||||
if index_mode == "unique":
|
||||
prefix = f"ADD UNIQUE INDEX {quote_identifier(index_name)}"
|
||||
elif index_mode == "fulltext":
|
||||
prefix = f"ADD FULLTEXT INDEX {quote_identifier(index_name)}"
|
||||
elif index_mode == "normal":
|
||||
prefix = f"ADD INDEX {quote_identifier(index_name)}"
|
||||
else:
|
||||
raise ValueError(f"unsupported index mode: {index_mode}")
|
||||
|
||||
column_sql = ", ".join(quote_identifier(item) for item in normalized_columns)
|
||||
|
||||
try:
|
||||
with build_mysql_connection(target_database, autocommit=False) as connection:
|
||||
with connection.cursor() as cursor:
|
||||
# 索引列名都走标识符转义,避免前端选择值被直接拼进 SQL。
|
||||
cursor.execute(
|
||||
f"ALTER TABLE {quote_identifier(target_table)} {prefix} ({column_sql})",
|
||||
)
|
||||
connection.commit()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return browser_error_payload(str(exc))
|
||||
|
||||
clear_mysql_metrics_cache()
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"message": f"已新增索引 {target_table}.{index_name}",
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def drop_mysql_index_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
target_database = normalize_database_name(payload.get("database"))
|
||||
target_table = normalize_table_name(payload.get("table"))
|
||||
index_name = normalize_column_name(payload.get("indexName"))
|
||||
|
||||
try:
|
||||
with build_mysql_connection(target_database, autocommit=False) as connection:
|
||||
with connection.cursor() as cursor:
|
||||
# PRIMARY KEY 的删除语法和普通索引不同,这里单独分支。
|
||||
if index_name == "PRIMARY":
|
||||
cursor.execute(
|
||||
f"ALTER TABLE {quote_identifier(target_table)} DROP PRIMARY KEY",
|
||||
)
|
||||
else:
|
||||
cursor.execute(
|
||||
f"ALTER TABLE {quote_identifier(target_table)} DROP INDEX {quote_identifier(index_name)}",
|
||||
)
|
||||
connection.commit()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return browser_error_payload(str(exc))
|
||||
|
||||
clear_mysql_metrics_cache()
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"message": f"已删除索引 {target_table}.{index_name}",
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def execute_mysql_sql_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
target_database = str(payload.get("database") or MYSQL_DEFAULT_DATABASE or "").strip() or None
|
||||
sql = normalize_sql_text(payload.get("sql"))
|
||||
kind = sql_statement_kind(sql)
|
||||
mutating_kinds = {"insert", "update", "delete", "replace", "alter", "create", "drop", "truncate", "rename"}
|
||||
|
||||
try:
|
||||
with build_mysql_connection(target_database, autocommit=False) as connection:
|
||||
with connection.cursor() as cursor:
|
||||
# SQL 工作台当前只执行单条语句,结果集最多保留固定行数。
|
||||
cursor.execute(sql)
|
||||
if cursor.description:
|
||||
columns = [str(item[0] or "").strip() for item in cursor.description]
|
||||
fetched_rows = cursor.fetchmany(MYSQL_QUERY_MAX_ROWS + 1)
|
||||
truncated = len(fetched_rows) > MYSQL_QUERY_MAX_ROWS
|
||||
rows = [normalize_mysql_row(item) for item in fetched_rows[:MYSQL_QUERY_MAX_ROWS]]
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": target_database or "",
|
||||
"kind": kind,
|
||||
"columns": columns,
|
||||
"rows": rows,
|
||||
"rowCount": len(rows),
|
||||
"truncated": truncated,
|
||||
"error": "",
|
||||
}
|
||||
|
||||
connection.commit()
|
||||
affected_rows = int(cursor.rowcount or 0)
|
||||
last_insert_id = cursor.lastrowid
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return browser_error_payload(str(exc))
|
||||
|
||||
if kind in mutating_kinds:
|
||||
clear_mysql_metrics_cache()
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"database": target_database or "",
|
||||
"kind": kind,
|
||||
"affectedRows": affected_rows,
|
||||
"lastInsertId": last_insert_id,
|
||||
"message": f"{kind.upper()} 执行完成",
|
||||
"error": "",
|
||||
}
|
||||
421
monitors/yumi/mysql_common.py
Normal file
421
monitors/yumi/mysql_common.py
Normal file
@ -0,0 +1,421 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date, datetime, time, timedelta
|
||||
from decimal import Decimal
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from core.config import (
|
||||
MYSQL_CONNECT_TIMEOUT_SECONDS,
|
||||
MYSQL_DEFAULT_DATABASE,
|
||||
MYSQL_HOST,
|
||||
MYSQL_PASSWORD,
|
||||
MYSQL_PORT,
|
||||
MYSQL_READ_TIMEOUT_SECONDS,
|
||||
MYSQL_USER,
|
||||
MYSQL_WRITE_TIMEOUT_SECONDS,
|
||||
utc_now,
|
||||
)
|
||||
|
||||
|
||||
# 系统库默认排在业务库后面,避免主列表被系统库占满。
|
||||
SYSTEM_DATABASES = {"information_schema", "performance_schema", "mysql", "sys"}
|
||||
|
||||
|
||||
def mysql_error_payload(error: str) -> dict[str, Any]:
|
||||
# MySQL 基础设施页统一返回同一套失败结构,前端不需要额外分支。
|
||||
return {
|
||||
"ok": False,
|
||||
"updatedAt": utc_now(),
|
||||
"info": {},
|
||||
"connections": {},
|
||||
"throughput": {},
|
||||
"replication": {"configured": False, "role": "", "readOnly": False, "members": []},
|
||||
"summary": {
|
||||
"databaseTotal": 0,
|
||||
"tableTotal": 0,
|
||||
"dataSizeMb": 0.0,
|
||||
"indexSizeMb": 0.0,
|
||||
"totalSizeMb": 0.0,
|
||||
},
|
||||
"databases": [],
|
||||
"error": error,
|
||||
}
|
||||
|
||||
|
||||
def require_pymysql() -> Any:
|
||||
try:
|
||||
# 运行时再导入驱动,避免缺少依赖时整个服务无法启动。
|
||||
import pymysql
|
||||
except Exception as exc: # noqa: BLE001
|
||||
raise RuntimeError(f"pymysql unavailable: {exc}") from exc
|
||||
|
||||
# 导入成功后直接返回模块对象,调用方继续拿 DictCursor。
|
||||
return pymysql
|
||||
|
||||
|
||||
def build_mysql_connection(database: str | None = None, *, autocommit: bool = True) -> Any:
|
||||
# 主机和账号缺失时直接给出明确错误,避免后面连库报一串底层异常。
|
||||
missing = [name for name, value in {"MYSQL_HOST": MYSQL_HOST, "MYSQL_USER": MYSQL_USER}.items() if not str(value).strip()]
|
||||
if missing:
|
||||
raise RuntimeError(f"missing mysql config in .env: {', '.join(missing)}")
|
||||
|
||||
# 默认库允许为空;表浏览场景会显式选择数据库。
|
||||
target_database = str(database or MYSQL_DEFAULT_DATABASE or "").strip() or None
|
||||
pymysql = require_pymysql()
|
||||
|
||||
# 统一走 utf8mb4,避免表结构和数据浏览时乱码。
|
||||
return pymysql.connect(
|
||||
host=MYSQL_HOST,
|
||||
port=MYSQL_PORT,
|
||||
user=MYSQL_USER,
|
||||
password=MYSQL_PASSWORD,
|
||||
database=target_database,
|
||||
charset="utf8mb4",
|
||||
cursorclass=pymysql.cursors.DictCursor,
|
||||
autocommit=autocommit,
|
||||
connect_timeout=MYSQL_CONNECT_TIMEOUT_SECONDS,
|
||||
read_timeout=MYSQL_READ_TIMEOUT_SECONDS,
|
||||
write_timeout=MYSQL_WRITE_TIMEOUT_SECONDS,
|
||||
)
|
||||
|
||||
|
||||
def quote_identifier(name: str) -> str:
|
||||
# 库、表、列名都统一走反引号包裹,并对内层反引号做转义。
|
||||
actual = str(name or "").strip()
|
||||
if not actual:
|
||||
raise ValueError("identifier is required")
|
||||
return f"`{actual.replace('`', '``')}`"
|
||||
|
||||
|
||||
def normalize_database_name(name: str) -> str:
|
||||
# 数据库名作为路由参数时必须非空,统一在这里兜底校验。
|
||||
actual = str(name or "").strip()
|
||||
if not actual:
|
||||
raise ValueError("database is required")
|
||||
return actual
|
||||
|
||||
|
||||
def normalize_table_name(name: str) -> str:
|
||||
# 表名同样必须非空。
|
||||
actual = str(name or "").strip()
|
||||
if not actual:
|
||||
raise ValueError("table is required")
|
||||
return actual
|
||||
|
||||
|
||||
def normalize_column_name(name: str) -> str:
|
||||
# 列名校验单独抽出来,便于 schema 变更和排序共用。
|
||||
actual = str(name or "").strip()
|
||||
if not actual:
|
||||
raise ValueError("column is required")
|
||||
return actual
|
||||
|
||||
|
||||
def is_system_database(name: str) -> bool:
|
||||
# 系统库统一小写比较,避免大小写混淆。
|
||||
return str(name or "").strip().lower() in SYSTEM_DATABASES
|
||||
|
||||
|
||||
def normalize_mysql_value(value: Any) -> Any:
|
||||
# Decimal 保留成字符串,避免金额精度被前端浮点化。
|
||||
if isinstance(value, Decimal):
|
||||
return str(value)
|
||||
|
||||
# 日期和时间都压成易读字符串。
|
||||
if isinstance(value, datetime):
|
||||
return value.strftime("%Y-%m-%d %H:%M:%S")
|
||||
if isinstance(value, date):
|
||||
return value.strftime("%Y-%m-%d")
|
||||
if isinstance(value, time):
|
||||
return value.strftime("%H:%M:%S")
|
||||
if isinstance(value, timedelta):
|
||||
return str(value)
|
||||
|
||||
# 字节内容尽量按 UTF-8 展示;失败时回退成十六进制。
|
||||
if isinstance(value, (bytes, bytearray, memoryview)):
|
||||
raw = bytes(value)
|
||||
try:
|
||||
return raw.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return f"0x{raw.hex()}"
|
||||
|
||||
# dict / list 理论上常见于 JSON 列,这里直接转成 JSON 文本方便展示和编辑。
|
||||
if isinstance(value, (dict, list)):
|
||||
return json.dumps(value, ensure_ascii=False)
|
||||
|
||||
# 其余标量直接原样返回。
|
||||
return value
|
||||
|
||||
|
||||
def normalize_mysql_row(row: dict[str, Any]) -> dict[str, Any]:
|
||||
# 一行记录中的每个字段都做 JSON 安全转换。
|
||||
return {key: normalize_mysql_value(value) for key, value in row.items()}
|
||||
|
||||
|
||||
def list_database_summaries(connection: Any) -> list[dict[str, Any]]:
|
||||
# 数据库级别汇总统一从 information_schema.tables 读取,能同时拿到表数和容量。
|
||||
query = """
|
||||
SELECT
|
||||
s.SCHEMA_NAME AS schemaName,
|
||||
COALESCE(t.tableTotal, 0) AS tableTotal,
|
||||
COALESCE(t.dataBytes, 0) AS dataBytes,
|
||||
COALESCE(t.indexBytes, 0) AS indexBytes,
|
||||
COALESCE(t.rowEstimate, 0) AS rowEstimate
|
||||
FROM information_schema.SCHEMATA AS s
|
||||
LEFT JOIN (
|
||||
SELECT
|
||||
table_schema AS schemaName,
|
||||
COUNT(*) AS tableTotal,
|
||||
SUM(COALESCE(data_length, 0)) AS dataBytes,
|
||||
SUM(COALESCE(index_length, 0)) AS indexBytes,
|
||||
SUM(COALESCE(table_rows, 0)) AS rowEstimate
|
||||
FROM information_schema.TABLES
|
||||
GROUP BY table_schema
|
||||
) AS t
|
||||
ON t.schemaName = s.SCHEMA_NAME
|
||||
ORDER BY
|
||||
CASE
|
||||
WHEN s.SCHEMA_NAME IN ('information_schema', 'performance_schema', 'mysql', 'sys') THEN 1
|
||||
ELSE 0
|
||||
END,
|
||||
COALESCE(t.dataBytes, 0) + COALESCE(t.indexBytes, 0) DESC,
|
||||
s.SCHEMA_NAME ASC
|
||||
"""
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# 直接读取全部库级摘要,前端 sidebar 需要完整列表。
|
||||
cursor.execute(query)
|
||||
rows = cursor.fetchall() or []
|
||||
|
||||
# 把字节数转换成 MiB,并附带是否系统库的标记。
|
||||
return [
|
||||
{
|
||||
"name": str(item.get("schemaName") or "").strip(),
|
||||
"system": is_system_database(item.get("schemaName")),
|
||||
"tableTotal": int(item.get("tableTotal") or 0),
|
||||
"rowEstimate": int(item.get("rowEstimate") or 0),
|
||||
"dataSizeMb": round(float(item.get("dataBytes") or 0) / 1024 / 1024, 2),
|
||||
"indexSizeMb": round(float(item.get("indexBytes") or 0) / 1024 / 1024, 2),
|
||||
"totalSizeMb": round(
|
||||
(float(item.get("dataBytes") or 0) + float(item.get("indexBytes") or 0)) / 1024 / 1024,
|
||||
2,
|
||||
),
|
||||
}
|
||||
for item in rows
|
||||
]
|
||||
|
||||
|
||||
def list_table_summaries(connection: Any, database: str) -> list[dict[str, Any]]:
|
||||
target_database = normalize_database_name(database)
|
||||
|
||||
# 表级摘要统一从 information_schema.tables 读取,便于在浏览器里快速切表。
|
||||
query = """
|
||||
SELECT
|
||||
TABLE_NAME AS tableName,
|
||||
ENGINE AS engine,
|
||||
TABLE_ROWS AS rowEstimate,
|
||||
DATA_LENGTH AS dataBytes,
|
||||
INDEX_LENGTH AS indexBytes,
|
||||
AUTO_INCREMENT AS autoIncrementValue,
|
||||
CREATE_TIME AS createTime,
|
||||
UPDATE_TIME AS updateTime,
|
||||
TABLE_COLLATION AS tableCollation
|
||||
FROM information_schema.TABLES
|
||||
WHERE TABLE_SCHEMA = %s
|
||||
ORDER BY COALESCE(DATA_LENGTH, 0) + COALESCE(INDEX_LENGTH, 0) DESC, TABLE_NAME ASC
|
||||
"""
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# 同一库内表数量通常有限,这里直接全量读取。
|
||||
cursor.execute(query, (target_database,))
|
||||
rows = cursor.fetchall() or []
|
||||
|
||||
return [
|
||||
{
|
||||
"name": str(item.get("tableName") or "").strip(),
|
||||
"engine": str(item.get("engine") or "").strip(),
|
||||
"rowEstimate": int(item.get("rowEstimate") or 0),
|
||||
"dataSizeMb": round(float(item.get("dataBytes") or 0) / 1024 / 1024, 2),
|
||||
"indexSizeMb": round(float(item.get("indexBytes") or 0) / 1024 / 1024, 2),
|
||||
"totalSizeMb": round(
|
||||
(float(item.get("dataBytes") or 0) + float(item.get("indexBytes") or 0)) / 1024 / 1024,
|
||||
2,
|
||||
),
|
||||
"autoIncrement": item.get("autoIncrementValue"),
|
||||
"createTime": normalize_mysql_value(item.get("createTime")),
|
||||
"updateTime": normalize_mysql_value(item.get("updateTime")),
|
||||
"collation": str(item.get("tableCollation") or "").strip(),
|
||||
}
|
||||
for item in rows
|
||||
]
|
||||
|
||||
|
||||
def list_table_columns(connection: Any, database: str, table: str) -> list[dict[str, Any]]:
|
||||
target_database = normalize_database_name(database)
|
||||
target_table = normalize_table_name(table)
|
||||
|
||||
# 列元数据以 ordinal_position 排序,保证浏览器和真实表结构一致。
|
||||
query = """
|
||||
SELECT
|
||||
COLUMN_NAME AS columnName,
|
||||
DATA_TYPE AS dataType,
|
||||
COLUMN_TYPE AS columnType,
|
||||
IS_NULLABLE AS isNullable,
|
||||
COLUMN_DEFAULT AS columnDefault,
|
||||
COLUMN_KEY AS columnKey,
|
||||
EXTRA AS extraValue,
|
||||
COLUMN_COMMENT AS columnComment,
|
||||
ORDINAL_POSITION AS ordinalPosition
|
||||
FROM information_schema.COLUMNS
|
||||
WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s
|
||||
ORDER BY ORDINAL_POSITION ASC
|
||||
"""
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# 表结构操作都依赖列元数据,这里统一一次性拿齐。
|
||||
cursor.execute(query, (target_database, target_table))
|
||||
rows = cursor.fetchall() or []
|
||||
|
||||
return [
|
||||
{
|
||||
"name": str(item.get("columnName") or "").strip(),
|
||||
"dataType": str(item.get("dataType") or "").strip(),
|
||||
"columnType": str(item.get("columnType") or "").strip(),
|
||||
"nullable": str(item.get("isNullable") or "").strip().upper() == "YES",
|
||||
"default": normalize_mysql_value(item.get("columnDefault")),
|
||||
"key": str(item.get("columnKey") or "").strip(),
|
||||
"extra": str(item.get("extraValue") or "").strip(),
|
||||
"comment": str(item.get("columnComment") or "").strip(),
|
||||
"ordinalPosition": int(item.get("ordinalPosition") or 0),
|
||||
}
|
||||
for item in rows
|
||||
]
|
||||
|
||||
|
||||
def list_table_indexes(connection: Any, database: str, table: str) -> list[dict[str, Any]]:
|
||||
target_database = normalize_database_name(database)
|
||||
target_table = normalize_table_name(table)
|
||||
|
||||
# 索引明细先按序号取出,再在 Python 侧聚合成前端更容易消费的结构。
|
||||
query = """
|
||||
SELECT
|
||||
INDEX_NAME AS indexName,
|
||||
COLUMN_NAME AS columnName,
|
||||
NON_UNIQUE AS nonUnique,
|
||||
INDEX_TYPE AS indexType,
|
||||
SEQ_IN_INDEX AS seqInIndex,
|
||||
SUB_PART AS subPart
|
||||
FROM information_schema.STATISTICS
|
||||
WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s
|
||||
ORDER BY INDEX_NAME ASC, SEQ_IN_INDEX ASC
|
||||
"""
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
# information_schema.STATISTICS 能完整覆盖 primary / unique / 普通索引。
|
||||
cursor.execute(query, (target_database, target_table))
|
||||
rows = cursor.fetchall() or []
|
||||
|
||||
grouped: dict[str, dict[str, Any]] = {}
|
||||
for item in rows:
|
||||
index_name = str(item.get("indexName") or "").strip()
|
||||
target = grouped.setdefault(
|
||||
index_name,
|
||||
{
|
||||
"name": index_name,
|
||||
"primary": index_name == "PRIMARY",
|
||||
"unique": int(item.get("nonUnique") or 0) == 0,
|
||||
"type": str(item.get("indexType") or "").strip(),
|
||||
"columns": [],
|
||||
},
|
||||
)
|
||||
target["columns"].append(
|
||||
{
|
||||
"name": str(item.get("columnName") or "").strip(),
|
||||
"subPart": int(item.get("subPart") or 0) or None,
|
||||
"position": int(item.get("seqInIndex") or 0),
|
||||
}
|
||||
)
|
||||
|
||||
# 输出时保证 primary 优先,再按名字排序。
|
||||
return sorted(
|
||||
grouped.values(),
|
||||
key=lambda item: (0 if item["primary"] else 1, item["name"].lower()),
|
||||
)
|
||||
|
||||
|
||||
def get_preferred_identity_columns(columns: list[dict[str, Any]], indexes: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
# 行级更新优先使用主键;没有主键时退到第一个唯一索引;再不行才退到全列比对。
|
||||
primary = next((item for item in indexes if item.get("primary")), None)
|
||||
if primary:
|
||||
return {"mode": "primary", "columns": [column["name"] for column in primary["columns"]]}
|
||||
|
||||
unique = next((item for item in indexes if item.get("unique")), None)
|
||||
if unique:
|
||||
return {"mode": "unique", "columns": [column["name"] for column in unique["columns"]]}
|
||||
|
||||
return {"mode": "all", "columns": [column["name"] for column in columns]}
|
||||
|
||||
|
||||
def build_row_identity(row: dict[str, Any], identity_spec: dict[str, Any]) -> dict[str, Any]:
|
||||
# identity 只保留用于匹配的字段,避免前端把整行作为 where 条件带回。
|
||||
columns = [str(item or "").strip() for item in identity_spec.get("columns") or [] if str(item or "").strip()]
|
||||
return {
|
||||
"mode": str(identity_spec.get("mode") or "").strip(),
|
||||
"columns": columns,
|
||||
"values": {column: normalize_mysql_value(row.get(column)) for column in columns},
|
||||
}
|
||||
|
||||
|
||||
def table_exists(connection: Any, database: str, table: str) -> bool:
|
||||
target_database = normalize_database_name(database)
|
||||
target_table = normalize_table_name(table)
|
||||
|
||||
# DDL 和浏览接口都需要一个轻量存在性校验,避免后面拼 SQL 才报错。
|
||||
with connection.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT 1
|
||||
FROM information_schema.TABLES
|
||||
WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s
|
||||
LIMIT 1
|
||||
""",
|
||||
(target_database, target_table),
|
||||
)
|
||||
return cursor.fetchone() is not None
|
||||
|
||||
|
||||
def database_exists(connection: Any, database: str) -> bool:
|
||||
target_database = normalize_database_name(database)
|
||||
|
||||
# 库级存在性校验单独抽出来,浏览器切库时可以复用。
|
||||
with connection.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT 1
|
||||
FROM information_schema.SCHEMATA
|
||||
WHERE SCHEMA_NAME = %s
|
||||
LIMIT 1
|
||||
""",
|
||||
(target_database,),
|
||||
)
|
||||
return cursor.fetchone() is not None
|
||||
|
||||
|
||||
def normalize_sql_text(sql: str) -> str:
|
||||
# SQL 工作台当前只允许单条语句,避免一键提交多段脚本。
|
||||
text = str(sql or "").strip()
|
||||
if not text:
|
||||
raise ValueError("sql is required")
|
||||
|
||||
# 只允许尾部分号,正文里再出现分号就拒绝执行。
|
||||
stripped = text.rstrip().rstrip(";").rstrip()
|
||||
if ";" in stripped:
|
||||
raise ValueError("only single sql statement is allowed")
|
||||
return stripped
|
||||
|
||||
|
||||
def sql_statement_kind(sql: str) -> str:
|
||||
# 读取首个关键字即可分辨查询类还是变更类语句。
|
||||
parts = normalize_sql_text(sql).split(None, 1)
|
||||
return parts[0].strip().lower() if parts else ""
|
||||
249
monitors/yumi/mysql_metrics.py
Normal file
249
monitors/yumi/mysql_metrics.py
Normal file
@ -0,0 +1,249 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from core.cache import TTLCache
|
||||
from core.config import MYSQL_CACHE_TTL_SECONDS, MYSQL_HOST, MYSQL_PORT, utc_now
|
||||
from .mysql_common import build_mysql_connection, list_database_summaries, mysql_error_payload, normalize_mysql_value
|
||||
|
||||
|
||||
# MySQL 概览卡片和基础设施页都复用同一份汇总数据。
|
||||
MYSQL_OVERVIEW_CACHE = TTLCache[dict[str, Any]](MYSQL_CACHE_TTL_SECONDS)
|
||||
# QPS / TPS 需要用两次采样求差值,这里单独保存最近一次原始计数器。
|
||||
MYSQL_RATE_LOCK = threading.Lock()
|
||||
MYSQL_RATE_SNAPSHOT: dict[str, Any] | None = None
|
||||
|
||||
|
||||
def clear_mysql_metrics_cache() -> None:
|
||||
# 数据写入、DDL 之后清理缓存,避免概览继续显示旧指标。
|
||||
MYSQL_OVERVIEW_CACHE.clear()
|
||||
|
||||
|
||||
def coerce_int(mapping: dict[str, Any], key: str) -> int:
|
||||
try:
|
||||
# MySQL status / variables 大多返回字符串,这里统一转整数。
|
||||
return int(mapping.get(key) or 0)
|
||||
except Exception: # noqa: BLE001
|
||||
return 0
|
||||
|
||||
|
||||
def coerce_bool(mapping: dict[str, Any], key: str) -> bool:
|
||||
value = str(mapping.get(key) or "").strip().lower()
|
||||
return value in {"1", "on", "yes", "true"}
|
||||
|
||||
|
||||
def collect_global_status(connection: Any) -> dict[str, Any]:
|
||||
names = (
|
||||
"Threads_connected",
|
||||
"Threads_running",
|
||||
"Questions",
|
||||
"Com_commit",
|
||||
"Com_rollback",
|
||||
"Uptime",
|
||||
)
|
||||
|
||||
# 这里选最小必要状态量,避免每次都拉全量 GLOBAL STATUS。
|
||||
with connection.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"""
|
||||
SHOW GLOBAL STATUS
|
||||
WHERE Variable_name IN (%s, %s, %s, %s, %s, %s)
|
||||
""",
|
||||
names,
|
||||
)
|
||||
rows = cursor.fetchall() or []
|
||||
|
||||
return {str(item.get("Variable_name") or ""): item.get("Value") for item in rows}
|
||||
|
||||
|
||||
def collect_global_variables(connection: Any) -> dict[str, Any]:
|
||||
names = (
|
||||
"version",
|
||||
"version_comment",
|
||||
"server_uuid",
|
||||
"server_id",
|
||||
"max_connections",
|
||||
"read_only",
|
||||
"super_read_only",
|
||||
)
|
||||
|
||||
# 版本、只读状态和最大连接数都在 GLOBAL VARIABLES 里。
|
||||
with connection.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"""
|
||||
SHOW GLOBAL VARIABLES
|
||||
WHERE Variable_name IN (%s, %s, %s, %s, %s, %s, %s)
|
||||
""",
|
||||
names,
|
||||
)
|
||||
rows = cursor.fetchall() or []
|
||||
|
||||
return {str(item.get("Variable_name") or ""): item.get("Value") for item in rows}
|
||||
|
||||
|
||||
def collect_replication_payload(connection: Any, variables: dict[str, Any]) -> dict[str, Any]:
|
||||
# 先把只读状态打平,这一层即使复制信息拿不到也能稳定展示。
|
||||
payload = {
|
||||
"configured": False,
|
||||
"role": "unknown",
|
||||
"readOnly": coerce_bool(variables, "read_only"),
|
||||
"superReadOnly": coerce_bool(variables, "super_read_only"),
|
||||
"sourceHost": "",
|
||||
"secondsBehindSource": None,
|
||||
"ioRunning": False,
|
||||
"sqlRunning": False,
|
||||
"state": "",
|
||||
"error": "",
|
||||
}
|
||||
|
||||
# 云 MySQL 版本不同,复制状态命令可能叫 SHOW SLAVE STATUS 或 SHOW REPLICA STATUS。
|
||||
commands = ["SHOW REPLICA STATUS", "SHOW SLAVE STATUS"]
|
||||
for command in commands:
|
||||
try:
|
||||
with connection.cursor() as cursor:
|
||||
cursor.execute(command)
|
||||
row = cursor.fetchone()
|
||||
except Exception: # noqa: BLE001
|
||||
# 云库版本和权限口径不一致时,这两条命令经常会报语法或权限错误。
|
||||
# 这里不把它直接当成实例异常,避免主节点或单节点 MySQL 被误标成故障。
|
||||
continue
|
||||
|
||||
# 没有行时说明当前节点没有暴露复制状态,继续尝试下一条命令。
|
||||
if not row:
|
||||
continue
|
||||
|
||||
payload["configured"] = True
|
||||
payload["sourceHost"] = str(
|
||||
row.get("Source_Host")
|
||||
or row.get("Master_Host")
|
||||
or ""
|
||||
).strip()
|
||||
payload["secondsBehindSource"] = (
|
||||
int(row.get("Seconds_Behind_Source") or row.get("Seconds_Behind_Master") or 0)
|
||||
if row.get("Seconds_Behind_Source") is not None or row.get("Seconds_Behind_Master") is not None
|
||||
else None
|
||||
)
|
||||
payload["ioRunning"] = str(
|
||||
row.get("Replica_IO_Running")
|
||||
or row.get("Slave_IO_Running")
|
||||
or ""
|
||||
).strip().lower() == "yes"
|
||||
payload["sqlRunning"] = str(
|
||||
row.get("Replica_SQL_Running")
|
||||
or row.get("Slave_SQL_Running")
|
||||
or ""
|
||||
).strip().lower() == "yes"
|
||||
payload["state"] = str(
|
||||
row.get("Replica_SQL_Running_State")
|
||||
or row.get("Slave_SQL_Running_State")
|
||||
or ""
|
||||
).strip()
|
||||
payload["error"] = str(
|
||||
row.get("Last_SQL_Error")
|
||||
or row.get("Last_IO_Error")
|
||||
or ""
|
||||
).strip()
|
||||
payload["role"] = "replica"
|
||||
return payload
|
||||
|
||||
# 没拿到复制详情时,退化成基于只读状态的粗略判断。
|
||||
payload["role"] = "replica" if payload["readOnly"] or payload["superReadOnly"] else "primary"
|
||||
return payload
|
||||
|
||||
|
||||
def collect_throughput_payload(status: dict[str, Any]) -> dict[str, Any]:
|
||||
current = {
|
||||
"capturedAt": time.time(),
|
||||
"questions": coerce_int(status, "Questions"),
|
||||
"commits": coerce_int(status, "Com_commit"),
|
||||
"rollbacks": coerce_int(status, "Com_rollback"),
|
||||
}
|
||||
|
||||
# 首次采样时没有可对比的上一帧,这里返回 0 但仍记住快照。
|
||||
with MYSQL_RATE_LOCK:
|
||||
global MYSQL_RATE_SNAPSHOT
|
||||
previous = MYSQL_RATE_SNAPSHOT
|
||||
MYSQL_RATE_SNAPSHOT = current
|
||||
|
||||
if previous is None:
|
||||
return {
|
||||
"qps": 0.0,
|
||||
"tps": 0.0,
|
||||
"sampleSeconds": 0.0,
|
||||
"questionsDelta": 0,
|
||||
"transactionsDelta": 0,
|
||||
}
|
||||
|
||||
elapsed = max(current["capturedAt"] - float(previous.get("capturedAt") or 0.0), 0.001)
|
||||
questions_delta = max(current["questions"] - int(previous.get("questions") or 0), 0)
|
||||
transactions_delta = max(
|
||||
(current["commits"] + current["rollbacks"]) - (int(previous.get("commits") or 0) + int(previous.get("rollbacks") or 0)),
|
||||
0,
|
||||
)
|
||||
|
||||
return {
|
||||
"qps": round(questions_delta / elapsed, 2),
|
||||
"tps": round(transactions_delta / elapsed, 2),
|
||||
"sampleSeconds": round(elapsed, 2),
|
||||
"questionsDelta": questions_delta,
|
||||
"transactionsDelta": transactions_delta,
|
||||
}
|
||||
|
||||
|
||||
def build_mysql_overview_payload() -> dict[str, Any]:
|
||||
try:
|
||||
# 指标和元数据统一走一条连接,减少直连云 MySQL 的并发压力。
|
||||
with build_mysql_connection(autocommit=True) as connection:
|
||||
with connection.cursor() as cursor:
|
||||
# ping 能尽早暴露账号、网络和权限问题。
|
||||
cursor.execute("SELECT 1 AS ok")
|
||||
cursor.fetchone()
|
||||
|
||||
variables = collect_global_variables(connection)
|
||||
status = collect_global_status(connection)
|
||||
replication = collect_replication_payload(connection, variables)
|
||||
throughput = collect_throughput_payload(status)
|
||||
databases = list_database_summaries(connection)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return mysql_error_payload(str(exc))
|
||||
|
||||
# 库级汇总同时作为基础设施概览卡片的数据源。
|
||||
business_databases = [item for item in databases if not item.get("system")]
|
||||
summary = {
|
||||
"databaseTotal": len(business_databases),
|
||||
"tableTotal": sum(int(item.get("tableTotal") or 0) for item in business_databases),
|
||||
"dataSizeMb": round(sum(float(item.get("dataSizeMb") or 0.0) for item in business_databases), 2),
|
||||
"indexSizeMb": round(sum(float(item.get("indexSizeMb") or 0.0) for item in business_databases), 2),
|
||||
"totalSizeMb": round(sum(float(item.get("totalSizeMb") or 0.0) for item in business_databases), 2),
|
||||
}
|
||||
|
||||
# 当前连接数、最大连接数和版本信息都单独拉平,前端展示更直接。
|
||||
return {
|
||||
"ok": True,
|
||||
"updatedAt": utc_now(),
|
||||
"info": {
|
||||
"host": f"{MYSQL_HOST}:{MYSQL_PORT}",
|
||||
"version": str(variables.get("version") or "").strip(),
|
||||
"versionComment": str(variables.get("version_comment") or "").strip(),
|
||||
"serverUuid": str(variables.get("server_uuid") or "").strip(),
|
||||
"serverId": str(variables.get("server_id") or "").strip(),
|
||||
"uptimeSeconds": coerce_int(status, "Uptime"),
|
||||
},
|
||||
"connections": {
|
||||
"current": coerce_int(status, "Threads_connected"),
|
||||
"running": coerce_int(status, "Threads_running"),
|
||||
"max": coerce_int(variables, "max_connections"),
|
||||
},
|
||||
"throughput": throughput,
|
||||
"replication": replication,
|
||||
"summary": summary,
|
||||
"databases": databases,
|
||||
"error": "",
|
||||
}
|
||||
|
||||
|
||||
def get_mysql_overview_payload() -> dict[str, Any]:
|
||||
# 概览卡片和表浏览 sidebar 共用缓存,避免每次点 tab 都重查 information_schema。
|
||||
return MYSQL_OVERVIEW_CACHE.get_or_build(build_mysql_overview_payload)
|
||||
@ -8,9 +8,9 @@ import urllib.parse
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
from .cache import TTLCache
|
||||
from core.cache import TTLCache
|
||||
from .config_validation import validate_config_content
|
||||
from .config import (
|
||||
from core.config import (
|
||||
NACOS_BASE_URL,
|
||||
NACOS_CACHE_TTL_SECONDS,
|
||||
NACOS_DISCOVERY_CLUSTER_NAME,
|
||||
@ -6,7 +6,7 @@ import hashlib
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from .config import RUNTIME_BASE_DIR, load_hosts, utc_now
|
||||
from core.config import RUNTIME_BASE_DIR, load_hosts, utc_now
|
||||
from .ssh_remote import read_remote_text as ssh_read_remote_text
|
||||
from .ssh_remote import write_remote_text as ssh_write_remote_text
|
||||
|
||||
@ -5,7 +5,7 @@ import shlex
|
||||
from typing import Any
|
||||
|
||||
from .compose_templates import container_name
|
||||
from .config import SERVICE_LOG_EXPORT_MAX_LINES, SERVICE_LOG_PREVIEW_LINES, flatten_services, load_hosts, utc_now
|
||||
from core.config import SERVICE_LOG_EXPORT_MAX_LINES, SERVICE_LOG_PREVIEW_LINES, flatten_services, load_hosts, utc_now
|
||||
from .ssh_remote import run_host_script
|
||||
|
||||
|
||||
@ -7,7 +7,7 @@ import time
|
||||
from typing import Any
|
||||
|
||||
from .compose_templates import sync_remote_host_compose_template
|
||||
from .config import (
|
||||
from core.config import (
|
||||
ROLLING_RESTART_POLL_SECONDS,
|
||||
ROLLING_RESTART_STABILIZE_SECONDS,
|
||||
ROLLING_RESTART_TIMEOUT_SECONDS,
|
||||
@ -21,7 +21,7 @@ from .compose_templates import (
|
||||
sync_remote_host_compose_template,
|
||||
update_local_service_image,
|
||||
)
|
||||
from .config import (
|
||||
from core.config import (
|
||||
ADMIN_FRONTEND_BUILD_SCRIPT,
|
||||
ADMIN_FRONTEND_COMPOSE_FILE,
|
||||
ADMIN_FRONTEND_COMPOSE_SERVICE,
|
||||
@ -13,7 +13,7 @@ import re
|
||||
import uuid
|
||||
from typing import Any
|
||||
|
||||
from .config import ROOT, utc_now
|
||||
from core.config import ROOT, utc_now
|
||||
from .service_release import deploy_updated_services_payload
|
||||
|
||||
|
||||
@ -9,7 +9,7 @@ from pathlib import Path
|
||||
import subprocess
|
||||
from typing import Any
|
||||
|
||||
from .config import (
|
||||
from core.config import (
|
||||
SSH_CONNECT_TIMEOUT_SECONDS,
|
||||
SSH_KNOWN_HOSTS_PATH,
|
||||
SSH_PORT,
|
||||
@ -5,7 +5,7 @@ import json
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from .config import DEPLOY_REGION, TAT_METRIC_TIMEOUT_SECONDS, TENCENT_SECRET_ID, TENCENT_SECRET_KEY, utc_now
|
||||
from core.config import DEPLOY_REGION, TAT_METRIC_TIMEOUT_SECONDS, TENCENT_SECRET_ID, TENCENT_SECRET_KEY, utc_now
|
||||
|
||||
|
||||
def decode_task_output(output: str) -> str:
|
||||
@ -12,7 +12,7 @@ ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from monitor.compose_templates import managed_compose_host_names, save_local_host_compose_template, sync_remote_compose_templates
|
||||
from monitors.yumi.compose_templates import managed_compose_host_names, save_local_host_compose_template, sync_remote_compose_templates
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
|
||||
@ -1,4 +1,5 @@
|
||||
tencentcloud-sdk-python>=3.0.0
|
||||
pymongo>=4.8.0
|
||||
PyMySQL>=1.1.1
|
||||
PyYAML>=6.0.2
|
||||
javaproperties>=0.8.2
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from gateway.http_app import main
|
||||
from entry.http_app import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
2184
static/app.css
2184
static/app.css
File diff suppressed because it is too large
Load Diff
2827
static/app.js
2827
static/app.js
File diff suppressed because it is too large
Load Diff
2136
static/index.html
2136
static/index.html
File diff suppressed because it is too large
Load Diff
@ -10,7 +10,7 @@ os.environ.setdefault("APP_BIND", os.environ.get("YUMI_INTERNAL_HOST", "127.0.0.
|
||||
os.environ.setdefault("APP_PORT", os.environ.get("YUMI_INTERNAL_PORT", "2027"))
|
||||
os.environ["APP_BASE_PATH"] = "/yumi"
|
||||
|
||||
from monitor.http_app import main
|
||||
from monitors.yumi.http_app import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user