refactor monitor structure and polish control console

This commit is contained in:
ZuoZuo 2026-04-23 04:19:55 +08:00
parent c8b2d7639c
commit 47ba1c8a76
46 changed files with 9399 additions and 458 deletions

View File

@ -75,6 +75,18 @@ MONGO_URI=
MONGO_DB_NAME=tarab_all
MONGO_INSTANCE_ID=ins-o9mmob66
MONGO_CACHE_TTL_SECONDS=20
MYSQL_HOST=
MYSQL_PORT=3306
MYSQL_USER=
MYSQL_PASSWORD=
MYSQL_DEFAULT_DATABASE=
MYSQL_CACHE_TTL_SECONDS=10
MYSQL_CONNECT_TIMEOUT_SECONDS=5
MYSQL_READ_TIMEOUT_SECONDS=15
MYSQL_WRITE_TIMEOUT_SECONDS=15
MYSQL_DEFAULT_PAGE_SIZE=100
MYSQL_MAX_PAGE_SIZE=200
MYSQL_QUERY_MAX_ROWS=500
CPU_WARN_PERCENT=60
CPU_BAD_PERCENT=85
MEMORY_WARN_PERCENT=70

View File

@ -52,3 +52,25 @@
- `配置` 页必须拆成页内 tabs`Nacos / Go Env / 历史版本`;编辑区使用 sticky 工具条、metadata strip、`源码 / Diff` 切换。
- `基础设施` 页必须有二级导航:`状态总览 / Nacos 服务 / Mongo 指标 / TAT`Nacos 服务默认折叠实例Mongo 次要指标默认折叠。
- 长文本默认缩写,完整值只通过悬停、复制或展开明细承接,不在主列表直接铺满一整行。
- `发布` 页主动作必须集中在一条 action bar区块内部不重复摆放刷新、预检查、执行类按钮。
- 高权重动作必须明显少于次动作;`保存 / 执行 / 立即刷新` 才能用高权重按钮,其余默认降为 ghost 或轻量控件。
- `执行日志``服务日志` 默认不直接铺在主页面里;页面只保留摘要,完整日志通过统一弹层查看和导出。
- `配置` 页左侧必须保持 explorer 式树状浏览,不回退到一张张大卡片;组折叠、当前项高亮、上下项切换都要直接可用。
- `配置` 页左侧 explorer 不重复展示已经在 workbench head 里出现的 baseUrl、组数、环境信息。
- `配置` 页不要重复出现“页面标题 / 分区标题 / 当前文件卡片”三层同义信息当前文件、group、type 合并进工具条和 metadata strip。
- `配置` 页左侧文件树优先高密度列表,压缩单项高度、内边距和顶部无效空白。
- `任务型页面` 优先使用紧凑任务头,不再额外堆一张信息价值低的说明卡。
- 主机视图里的百分比指标优先使用小圆环 + 数值标签,不回退到纯文本堆叠。
- 主机行中部优先展示“在线服务 / 异常服务摘要”,避免重复出现“全部正常”这类低信息密度文案。
- 主机分组标题必须带主机数、服务数、异常主机数,保证高密度时仍能快速扫读分区状态。
- 总览页在全正常状态下不继续渲染空异常区;异常摘要、异常主机、异常基础设施为空时直接隐藏,正常项只保留紧凑摘要。
- 首页异常摘要优先使用分段状态带;正常健康态保持紧凑卡组,不回退到多张孤立大卡。
- 首页服务矩阵的每个业务分组应作为独立分组卡展示,分组头直接包含服务数、节点数、异常数。
- 首页的网关 / 应用 / 支付分组允许使用轻微色温差异和导向线区分,但必须克制,不把分组色做成高饱和装饰。
- 首页正常主机默认使用紧凑卡栅格;没有异常主机时不渲染大空态占位。
- 主机指标拿不到时显示采集状态芯片,不用空圆环或占位短横伪装成正常指标。
- `发布` 页更新目标和滚动重启优先使用多列紧凑服务卡,完整宽表只留给执行结果和明细。
- 可选择条目优先使用胶囊式勾选控件,已选态要明显强于未选态,不能只靠原生 checkbox 本体提示选择结果。
- 首页侧栏卡、基础设施小卡、任务页工作区尽量复用同一套顶部导向线和轻表面语言,避免不同页面像不同产品拼接。
- `数据库` 视图使用统一的工作台入口,`MySQL / MongoDB` 通过同一层引擎切换承接,不再把数据库操作散落在不同页面角落。
- `MongoDB / MySQL` 操作页都采用 `左侧 explorer + 右侧 workbench` 结构;概览指标保持紧凑,危险操作集中到单独分区,不把删除类动作和普通查询动作混在一起。

View File

@ -52,10 +52,31 @@
- 首页首屏默认收敛为顶部导航、工具条、紧凑摘要条、服务矩阵,不额外插入大块最近异常横幅。
- 最近异常进入可折叠详情侧栏顶部,不独立占据首页主宽度。
- 服务矩阵按业务分组切成子矩阵,异常排序前置,表头和第一列保持 sticky。
- 首页异常摘要优先使用分段状态带,不回退到零散大卡。
- 服务矩阵的业务分组要做成独立分组卡,分组头直接给服务数、节点数、异常数。
- 首页网关 / 应用 / 支付分组可以用轻微色温差异和导向线区分,但保持低饱和,不做装饰性大色块。
- 首页主机区只展示业务主机;基础设施主机放到基础设施或主机视图承接。
- 主机指标使用单行短文本,正常态降饱和,红橙色保留给异常和告警。
- 首页没有异常主机时不渲染大空态,正常主机直接用紧凑卡栅格展示。
- 主机指标缺失时直接显示采集状态芯片,不渲染空圆环和占位短横。
- 监控筛选条只属于 `总览 / 服务 / 主机``发布 / 配置 / 基础设施` 必须使用各自专用工具条和页内导航。
- `发布` 页按“构建参数 -> 更新目标 -> 执行日志 -> 滚动重启”组织,更新与重启目标都优先用紧凑表格,不再回退到大卡片流。
- `配置` 页按 `Nacos / Go Env / 历史版本` tabs 组织,编辑器必须提供 sticky 工具条、metadata strip 和 `源码 / Diff` 切换。
- `基础设施` 页按 `状态总览 / Nacos 服务 / Mongo 指标 / TAT` 二级导航组织Nacos 服务默认折叠实例Mongo 次要指标默认折叠。
- 长文本默认缩写并提供复制或展开,不直接在主列表铺完整 digest、路径、长 ID 或长连接串。
- `发布` 页主动作集中到单独 action bar区块内部避免重复放刷新、预检查、执行类按钮。
- 高权重动作必须少于次动作;只有 `保存 / 执行 / 立即刷新` 这类核心操作保留高权重按钮,其余动作降为 ghost 或轻量控件。
- `执行日志``服务日志` 默认通过统一弹层查看;主页面只保留摘要和上下文,不铺大块日志窗口。
- `配置` 页左侧使用 explorer 式树状导航,避免大卡片式配置清单;当前项高亮、组折叠、上下项跳转必须可用。
- `配置` 页左侧 explorer 不重复展示已经在 workbench head 里出现的 baseUrl、组数、环境信息。
- `配置` 页不要重复堆叠大标题和当前文件卡片;当前文件信息应并入工具条和 metadata strip左侧文件树保持高密度。
- `任务型页面` 优先使用紧凑任务头,不再额外堆一张信息价值低的说明卡。
- 主机视图中的百分比指标优先使用小圆环 + 数值标签,不回退到纯文本百分比堆叠。
- 主机行中部优先展示服务摘要,避免重复出现“全部正常”这类低信息密度文案。
- 主机分组标题需要同时给出主机数、服务数、异常主机数,确保高密度下仍然易扫读。
- 总览页全正常时不继续渲染空异常区;正常主机和 `Nacos / Mongo / TAT` 统一收成紧凑摘要。
- `发布` 页更新目标和滚动重启默认使用多列紧凑卡片,避免一项一整行的浪费布局。
- 可选择条目优先使用胶囊式勾选控件,已选态必须明显强于未选态,不能只靠原生 checkbox 提示。
- 首页侧栏卡、基础设施小卡、任务工作区尽量复用同一套顶部导向线和轻表面语言,避免跨页设计语言漂移。
- `数据库` 视图使用统一的工作台入口,`MySQL / MongoDB` 通过同一层引擎切换承接,不再把数据库操作散落在不同页面角落。
- `MongoDB / MySQL` 操作页统一采用 `左侧 explorer + 右侧 workbench` 结构;概览指标保持紧凑,危险操作集中在单独分区,不把删除类动作和普通查询动作混在一起。

View File

@ -11,7 +11,8 @@
- 服务健康:直接探测各业务服务健康接口
- 主机资源:通过 `SSH` 采集 `CPU / 内存 / 磁盘 / 进程数`
- Nacos直连 Nacos OpenAPI列出命名空间、服务和注册实例
- MongoDB直连 Mongo采集 `serverStatus / listDatabases / replSetGetStatus`
- MongoDB直连 Mongo采集 `serverStatus / listDatabases / replSetGetStatus`并提供库表浏览、分页查询、CRUD、聚合、Distinct、索引管理
- MySQL直连云 MySQL提供运行状态、库表浏览、100 条分页查询、CRUD、列/索引变更和 SQL 工作台
- 宿主机运维日志、滚动重启、compose 同步、服务更新统一走 `SSH`
- 带外引导:只在打通 `deploy -> target:22` 时使用 `TAT`
@ -20,9 +21,10 @@
- `server.py`gateway 入口
- `yumi_server.py`Yumi 内部实例入口
- `haiyi_server.py`HaiYi 内部实例入口
- `gateway/`:统一登录、入口选择页和反代
- `monitor/`Yumi 监控与运维实现
- `haiyi_monitor/`HaiYi 占位实例
- `entry/`:统一登录、入口选择页和反代
- `core/`:共享配置、鉴权、缓存和路径工具
- `monitors/yumi/`Yumi 监控与运维实现
- `monitors/haiyi/`HaiYi 占位实例
- `config/targets.json`:监控目标和实例 ID
- `static/`:前端页面
- `scripts/`:启停、自启安装、前台运行脚本
@ -51,6 +53,11 @@ REFRESH_INTERVAL_OPTIONS=5,10,15,30,60
NACOS_BASE_URL=http://10.2.1.17:8848
NACOS_NAMESPACE_ID=...
MONGO_URI=mongodb://...
MYSQL_HOST=10.x.x.x
MYSQL_PORT=3306
MYSQL_USER=readonly_or_admin
MYSQL_PASSWORD=***
MYSQL_DEFAULT_DATABASE=
DEPLOY_REGION=me-saudi-arabia
DEPLOY_INSTANCE_ID=
TENCENT_SECRET_ID=***
@ -115,6 +122,18 @@ cp .env.example .env
AUTH_COOKIE_SECURE=1
```
MySQL 工作台可选参数:
```bash
MYSQL_CACHE_TTL_SECONDS=10
MYSQL_CONNECT_TIMEOUT_SECONDS=5
MYSQL_READ_TIMEOUT_SECONDS=15
MYSQL_WRITE_TIMEOUT_SECONDS=15
MYSQL_DEFAULT_PAGE_SIZE=100
MYSQL_MAX_PAGE_SIZE=200
MYSQL_QUERY_MAX_ROWS=500
```
## 本地运行
```bash
@ -263,4 +282,7 @@ python3 ops/sync_runtime_compose.py --sync-remote
- 服务更新:本地构建选中服务、推送 Harbor、更新完整 compose并只重建选中的服务
- 滚动重启:可按服务选择 Java / Golang 节点,逐台重启并等待健康恢复
- MongoDB连接、内存、网络、操作计数、复制集、数据库统计
- Mongo 工作页:列出所有数据库和集合,文档默认每页 `100` 条,支持 `find / insertOne / insertMany / replaceOne / updateOne / updateMany / deleteOne / deleteMany / aggregate / distinct / createIndex / dropIndex / dropIndexes / createCollection / renameCollection / dropCollection / dropDatabase`
- MySQL连接状态、版本、当前/最大连接数、QPS/TPS、数据库大小、主从/只读状态
- MySQL 工作页:列出所有数据库和表,默认每页 `100` 条,支持筛选排序、行级增删改、增加/删除列、增加/删除索引、常用 SQL 快捷操作和单语句 SQL 执行
- 自动刷新:前端可切换刷新间隔,并保存在浏览器本地

1
core/__init__.py Normal file
View File

@ -0,0 +1 @@
"""Shared runtime layer for gateway and monitors."""

View File

@ -280,6 +280,31 @@ MONGO_SERVER_SELECTION_TIMEOUT_MS = env_int("MONGO_SERVER_SELECTION_TIMEOUT_MS",
# Mongo socket 超时。
MONGO_SOCKET_TIMEOUT_MS = env_int("MONGO_SOCKET_TIMEOUT_MS", 5000)
# MySQL 主机地址。
MYSQL_HOST = env_str("MYSQL_HOST", "")
# MySQL 端口。
MYSQL_PORT = env_int("MYSQL_PORT", 3306)
# MySQL 登录账号。
MYSQL_USER = env_str("MYSQL_USER", "")
# MySQL 登录密码;允许空串,兼容特殊场景。
MYSQL_PASSWORD = os.environ.get("MYSQL_PASSWORD", "")
# MySQL 默认库;为空时由页面显式选择。
MYSQL_DEFAULT_DATABASE = env_str("MYSQL_DEFAULT_DATABASE", "")
# MySQL 指标与元数据缓存秒数。
MYSQL_CACHE_TTL_SECONDS = env_float("MYSQL_CACHE_TTL_SECONDS", 8.0)
# MySQL 建连超时秒数。
MYSQL_CONNECT_TIMEOUT_SECONDS = env_int("MYSQL_CONNECT_TIMEOUT_SECONDS", 5)
# MySQL 读超时秒数。
MYSQL_READ_TIMEOUT_SECONDS = env_int("MYSQL_READ_TIMEOUT_SECONDS", 8)
# MySQL 写超时秒数。
MYSQL_WRITE_TIMEOUT_SECONDS = env_int("MYSQL_WRITE_TIMEOUT_SECONDS", 15)
# MySQL 表浏览默认分页大小。
MYSQL_DEFAULT_PAGE_SIZE = env_int("MYSQL_DEFAULT_PAGE_SIZE", 100)
# MySQL 表浏览最大分页大小。
MYSQL_MAX_PAGE_SIZE = env_int("MYSQL_MAX_PAGE_SIZE", 200)
# SQL 工作台单次最多返回多少行。
MYSQL_QUERY_MAX_ROWS = env_int("MYSQL_QUERY_MAX_ROWS", 500)
# CPU 告警阈值。
CPU_WARN_PERCENT = env_float("CPU_WARN_PERCENT", 60.0)
# CPU 严重阈值。

1
entry/__init__.py Normal file
View File

@ -0,0 +1 @@
"""Unified gateway entry package."""

View File

@ -10,7 +10,7 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from typing import Any
from urllib.parse import parse_qs, urlparse
from monitor.auth import (
from core.auth import (
build_clear_session_cookie,
build_session_cookie,
create_session,
@ -20,7 +20,7 @@ from monitor.auth import (
parse_session_token,
authenticate_user,
)
from monitor.config import (
from core.config import (
APP_BIND,
APP_PORT,
GATEWAY_PROXY_TIMEOUT_SECONDS,
@ -32,7 +32,7 @@ from monitor.config import (
YUMI_INTERNAL_PORT,
utc_now,
)
from monitor.http_paths import login_redirect_target
from core.http_paths import login_redirect_target
def response_json(payload: dict[str, Any] | list[Any]) -> bytes:

View File

@ -1 +0,0 @@
from __future__ import annotations

View File

@ -1 +0,0 @@
from __future__ import annotations

View File

@ -10,7 +10,7 @@ os.environ.setdefault("APP_BIND", os.environ.get("HAIYI_INTERNAL_HOST", "127.0.0
os.environ.setdefault("APP_PORT", os.environ.get("HAIYI_INTERNAL_PORT", "2028"))
os.environ["APP_BASE_PATH"] = "/haiyi"
from haiyi_monitor.http_app import main
from monitors.haiyi.http_app import main
if __name__ == "__main__":

View File

@ -1,2 +0,0 @@
"""hy-app-monitor package."""

1
monitors/__init__.py Normal file
View File

@ -0,0 +1 @@
"""Monitor application packages."""

View File

@ -0,0 +1 @@
"""HaiYi monitor package."""

View File

@ -7,8 +7,8 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from typing import Any
from urllib.parse import urlparse
from monitor.config import APP_BASE_PATH, APP_BIND, APP_PORT, INTERNAL_PROXY_SHARED_SECRET, STATIC_DIR, utc_now
from monitor.http_paths import join_base_path, strip_base_path
from core.config import APP_BASE_PATH, APP_BIND, APP_PORT, INTERNAL_PROXY_SHARED_SECRET, STATIC_DIR, utc_now
from core.http_paths import join_base_path, strip_base_path
def response_json(payload: dict[str, Any] | list[Any]) -> bytes:

View File

@ -0,0 +1 @@
"""Yumi monitor package."""

View File

@ -10,7 +10,7 @@ import shlex
import re
from typing import Any
from .config import ROOT, RUNTIME_BASE_DIR, load_hosts, utc_now
from core.config import ROOT, RUNTIME_BASE_DIR, load_hosts, utc_now
from .runtime_config import read_remote_text, write_remote_text
from .ssh_remote import copy_remote_file, run_host_script_checked

View File

@ -2,10 +2,11 @@ from __future__ import annotations
from typing import Any
from .config import THRESHOLDS, load_hosts, settings_payload, utc_now
from core.config import THRESHOLDS, load_hosts, settings_payload, utc_now
from .host_metrics import get_host_metrics
from .http_probe import get_service_payload
from .mongo_metrics import get_mongo_payload
from .mysql_metrics import get_mysql_overview_payload
from .nacos import get_nacos_payload
@ -116,6 +117,8 @@ def build_monitor_payload() -> dict[str, Any]:
nacos_payload = get_nacos_payload()
# 获取 Mongo 指标。
mongo_payload = get_mongo_payload()
# 获取 MySQL 指标。
mysql_payload = get_mysql_overview_payload()
# 组装主机维度视图。
grouped_hosts = build_host_payload(hosts, service_payload, metrics_by_host)
# 统计主机指标成功数。
@ -137,9 +140,13 @@ def build_monitor_payload() -> dict[str, Any]:
"mongoDatabaseTotal": int(mongo_payload.get("summary", {}).get("databaseTotal", 0)),
"mongoCollectionTotal": int(mongo_payload.get("summary", {}).get("collectionTotal", 0)),
"mongoOk": 1 if mongo_payload.get("ok") else 0,
"mysqlDatabaseTotal": int(mysql_payload.get("summary", {}).get("databaseTotal", 0)),
"mysqlTableTotal": int(mysql_payload.get("summary", {}).get("tableTotal", 0)),
"mysqlOk": 1 if mysql_payload.get("ok") else 0,
},
"hosts": grouped_hosts,
"items": service_payload["items"],
"nacos": nacos_payload,
"mongo": mongo_payload,
"mysql": mysql_payload,
}

View File

@ -5,8 +5,8 @@ import concurrent.futures
import json
from typing import Any
from .cache import TTLCache
from .config import HOST_METRIC_CACHE_TTL_SECONDS, HOST_METRIC_CPU_SAMPLE_SECONDS, HOST_METRIC_TIMEOUT_SECONDS, utc_now
from core.cache import TTLCache
from core.config import HOST_METRIC_CACHE_TTL_SECONDS, HOST_METRIC_CPU_SAMPLE_SECONDS, HOST_METRIC_TIMEOUT_SECONDS, utc_now
from .ssh_remote import run_host_script

View File

@ -7,9 +7,42 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from typing import Any
from urllib.parse import parse_qs, urlparse
from .config import APP_BASE_PATH, APP_BIND, APP_PORT, INTERNAL_PROXY_SHARED_SECRET, STATIC_DIR, utc_now
from core.config import APP_BASE_PATH, APP_BIND, APP_PORT, INTERNAL_PROXY_SHARED_SECRET, STATIC_DIR, utc_now
from .dashboard import build_monitor_payload
from .http_paths import join_base_path, strip_base_path
from core.http_paths import join_base_path, strip_base_path
from .mongo_admin import (
aggregate_mongo_documents_payload,
create_mongo_collection_payload,
create_mongo_index_payload,
delete_mongo_documents_payload,
distinct_mongo_values_payload,
drop_mongo_collection_payload,
drop_mongo_database_payload,
drop_mongo_index_payload,
drop_mongo_secondary_indexes_payload,
get_mongo_admin_overview_payload,
get_mongo_collection_detail_payload,
insert_mongo_document_payload,
query_mongo_documents_payload,
rename_mongo_collection_payload,
replace_mongo_document_payload,
update_mongo_documents_payload,
)
from .mysql_admin import (
add_mysql_column_payload,
add_mysql_index_payload,
delete_mysql_row_payload,
drop_mysql_column_payload,
drop_mysql_index_payload,
execute_mysql_sql_payload,
get_mysql_databases_payload,
get_mysql_table_schema_payload,
get_mysql_tables_payload,
insert_mysql_row_payload,
query_mysql_table_rows_payload,
update_mysql_row_payload,
)
from .mysql_metrics import get_mysql_overview_payload
from .nacos import (
get_nacos_config_list_payload,
get_nacos_config_payload,
@ -178,6 +211,120 @@ class MonitorHandler(BaseHTTPRequestHandler):
send_body=send_body,
)
# 返回 MySQL 概览指标。
if local_path == "/api/mysql/overview":
try:
payload = get_mysql_overview_payload()
except Exception as exc: # noqa: BLE001
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
return self.send_payload(
HTTPStatus.OK,
response_json(payload),
"application/json; charset=utf-8",
send_body=send_body,
)
# 返回 MySQL 库列表。
if local_path == "/api/mysql/databases":
try:
payload = get_mysql_databases_payload()
except Exception as exc: # noqa: BLE001
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
return self.send_payload(
HTTPStatus.OK,
response_json(payload),
"application/json; charset=utf-8",
send_body=send_body,
)
# 返回某个库下的表列表。
if local_path == "/api/mysql/tables":
database_name = self.query_value(query, "database")
if not database_name:
return self.send_error_payload(
HTTPStatus.BAD_REQUEST,
"database is required",
send_body=send_body,
)
try:
payload = get_mysql_tables_payload(database_name)
except Exception as exc: # noqa: BLE001
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
return self.send_payload(
HTTPStatus.OK,
response_json(payload),
"application/json; charset=utf-8",
send_body=send_body,
)
# 返回 MySQL 表结构、索引和建表语句。
if local_path == "/api/mysql/table-schema":
database_name = self.query_value(query, "database")
table_name = self.query_value(query, "table")
if not database_name or not table_name:
return self.send_error_payload(
HTTPStatus.BAD_REQUEST,
"database and table are required",
send_body=send_body,
)
try:
payload = get_mysql_table_schema_payload(database_name, table_name)
except Exception as exc: # noqa: BLE001
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
return self.send_payload(
HTTPStatus.OK,
response_json(payload),
"application/json; charset=utf-8",
send_body=send_body,
)
# 返回 Mongo 管理页的数据库与集合树。
if local_path == "/api/mongo-admin/overview":
try:
payload = get_mongo_admin_overview_payload()
except ValueError as exc:
return self.send_error_payload(HTTPStatus.BAD_REQUEST, str(exc), send_body=send_body)
except Exception as exc: # noqa: BLE001
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
return self.send_payload(
HTTPStatus.OK,
response_json(payload),
"application/json; charset=utf-8",
send_body=send_body,
)
# 返回当前集合的 stats 和索引列表。
if local_path == "/api/mongo-admin/collection-detail":
database_name = self.query_value(query, "database")
collection_name = self.query_value(query, "collection")
if not database_name or not collection_name:
return self.send_error_payload(
HTTPStatus.BAD_REQUEST,
"database and collection are required",
send_body=send_body,
)
try:
payload = get_mongo_collection_detail_payload(database_name, collection_name)
except ValueError as exc:
return self.send_error_payload(HTTPStatus.BAD_REQUEST, str(exc), send_body=send_body)
except Exception as exc: # noqa: BLE001
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=send_body)
return self.send_payload(
HTTPStatus.OK,
response_json(payload),
"application/json; charset=utf-8",
send_body=send_body,
)
# 返回可滚动重启的业务服务。
if local_path == "/api/ops/restart-targets":
try:
@ -341,12 +488,81 @@ class MonitorHandler(BaseHTTPRequestHandler):
if local_path == "/api/runtime/golang-config":
return self.handle_go_save(payload)
if local_path == "/api/mysql/rows/query":
return self.handle_mysql_action(query_mysql_table_rows_payload, payload)
if local_path == "/api/mysql/rows/insert":
return self.handle_mysql_action(insert_mysql_row_payload, payload, status=HTTPStatus.CREATED)
if local_path == "/api/mysql/rows/update":
return self.handle_mysql_action(update_mysql_row_payload, payload)
if local_path == "/api/mysql/rows/delete":
return self.handle_mysql_action(delete_mysql_row_payload, payload)
if local_path == "/api/mysql/columns/add":
return self.handle_mysql_action(add_mysql_column_payload, payload)
if local_path == "/api/mysql/columns/drop":
return self.handle_mysql_action(drop_mysql_column_payload, payload)
if local_path == "/api/mysql/indexes/add":
return self.handle_mysql_action(add_mysql_index_payload, payload, status=HTTPStatus.CREATED)
if local_path == "/api/mysql/indexes/drop":
return self.handle_mysql_action(drop_mysql_index_payload, payload)
if local_path == "/api/mysql/sql/execute":
return self.handle_mysql_action(execute_mysql_sql_payload, payload)
if local_path == "/api/ops/rolling-restart":
return self.handle_rolling_restart(payload)
if local_path == "/api/ops/update-services":
return self.handle_update_services(payload)
if local_path == "/api/mongo-admin/query":
return self.handle_mongo_admin_action(query_mongo_documents_payload, payload)
if local_path == "/api/mongo-admin/documents/insert":
return self.handle_mongo_admin_action(insert_mongo_document_payload, payload, status=HTTPStatus.CREATED)
if local_path == "/api/mongo-admin/documents/replace":
return self.handle_mongo_admin_action(replace_mongo_document_payload, payload)
if local_path == "/api/mongo-admin/documents/update":
return self.handle_mongo_admin_action(update_mongo_documents_payload, payload)
if local_path == "/api/mongo-admin/documents/delete":
return self.handle_mongo_admin_action(delete_mongo_documents_payload, payload)
if local_path == "/api/mongo-admin/aggregate":
return self.handle_mongo_admin_action(aggregate_mongo_documents_payload, payload)
if local_path == "/api/mongo-admin/distinct":
return self.handle_mongo_admin_action(distinct_mongo_values_payload, payload)
if local_path == "/api/mongo-admin/indexes/create":
return self.handle_mongo_admin_action(create_mongo_index_payload, payload, status=HTTPStatus.CREATED)
if local_path == "/api/mongo-admin/indexes/drop":
return self.handle_mongo_admin_action(drop_mongo_index_payload, payload)
if local_path == "/api/mongo-admin/indexes/drop-secondary":
return self.handle_mongo_admin_action(drop_mongo_secondary_indexes_payload, payload)
if local_path == "/api/mongo-admin/collections/create":
return self.handle_mongo_admin_action(create_mongo_collection_payload, payload, status=HTTPStatus.CREATED)
if local_path == "/api/mongo-admin/collections/rename":
return self.handle_mongo_admin_action(rename_mongo_collection_payload, payload)
if local_path == "/api/mongo-admin/collections/drop":
return self.handle_mongo_admin_action(drop_mongo_collection_payload, payload)
if local_path == "/api/mongo-admin/databases/drop":
return self.handle_mongo_admin_action(drop_mongo_database_payload, payload)
return self.send_error_payload(HTTPStatus.NOT_FOUND, "not found", send_body=True)
def redirect(self, location: str, *, send_body: bool) -> None:
@ -444,6 +660,27 @@ class MonitorHandler(BaseHTTPRequestHandler):
send_body=True,
)
def handle_mysql_action(
self,
action: Any,
payload: dict[str, Any],
*,
status: HTTPStatus = HTTPStatus.OK,
) -> None:
try:
result = action(payload)
except ValueError as exc:
return self.send_error_payload(HTTPStatus.BAD_REQUEST, str(exc), send_body=True)
except Exception as exc: # noqa: BLE001
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=True)
return self.send_payload(
status,
response_json(result),
"application/json; charset=utf-8",
send_body=True,
)
def handle_rolling_restart(self, payload: dict[str, Any]) -> None:
services = payload.get("services")
if not isinstance(services, list):
@ -491,6 +728,28 @@ class MonitorHandler(BaseHTTPRequestHandler):
send_body=True,
)
def handle_mongo_admin_action(
self,
builder: Any,
payload: dict[str, Any],
*,
status: HTTPStatus = HTTPStatus.OK,
) -> None:
try:
# Mongo 管理页所有写操作和查询操作都统一走这里收敛异常。
result = builder(payload)
except ValueError as exc:
return self.send_error_payload(HTTPStatus.BAD_REQUEST, str(exc), send_body=True)
except Exception as exc: # noqa: BLE001
return self.send_error_payload(HTTPStatus.BAD_GATEWAY, str(exc), send_body=True)
return self.send_payload(
status,
response_json(result),
"application/json; charset=utf-8",
send_body=True,
)
def query_value(self, query: dict[str, list[str]], key: str) -> str:
# 统一读取单值 query 参数。
return str((query.get(key) or [""])[0]).strip()

View File

@ -6,8 +6,8 @@ import urllib.error
import urllib.request
import time
from .cache import TTLCache
from .config import PROBE_CACHE_TTL_SECONDS, PROBE_MAX_WORKERS, PROBE_TIMEOUT_SECONDS, flatten_services, utc_now
from core.cache import TTLCache
from core.config import PROBE_CACHE_TTL_SECONDS, PROBE_MAX_WORKERS, PROBE_TIMEOUT_SECONDS, flatten_services, utc_now
# 服务探测使用独立缓存,避免页面轮询直接把内网接口打爆。

View File

@ -0,0 +1,879 @@
from __future__ import annotations
import json
from typing import Any
from core.config import MONGO_DB_NAME, utc_now
from .mongo_metrics import build_mongo_client
# 文档浏览默认每页 100 条,和页面默认值保持一致。
DEFAULT_PAGE_SIZE = 100
# 单次最多放开到 200 条,避免把浏览器一次撑爆。
MAX_PAGE_SIZE = 200
# 浏览页默认按 _id 倒序,和大多数运维排查习惯一致。
DEFAULT_SORT_SPEC = {"_id": -1}
# 列表里的 JSON 预览文本做一个上限,避免整行被超长文档拉坏。
PREVIEW_TEXT_LIMIT = 280
# Mongo 常见索引类型统一在这里收口,页面允许直接传这些值。
ALLOWED_INDEX_DIRECTIONS = {"text", "hashed", "2dsphere", "2d"}
def require_bson_json_util() -> Any:
try:
# PyMongo 安装后会自带 bson 包,这里延迟导入避免缺依赖时整个服务起不来。
from bson import json_util
except Exception as exc: # noqa: BLE001
# 管理页不可用时直接把依赖错误返回给上层。
raise RuntimeError(f"bson unavailable: {exc}") from exc
# 返回 bson 的扩展 JSON 工具。
return json_util
def get_mongo_client() -> Any:
# 统一复用现有 Mongo 建连逻辑,避免管理页和监控页配置口径分叉。
client, error = build_mongo_client()
# 建连失败时直接抛统一错误。
if client is None:
raise RuntimeError(error)
# 先做一次 ping尽早把连通性错误暴露出来。
client.admin.command("ping")
return client
def clamp_page_size(value: Any, default: int = DEFAULT_PAGE_SIZE) -> int:
try:
# 页面传来的值可能是字符串,这里统一转整数。
numeric = int(value)
except Exception: # noqa: BLE001
numeric = default
# 最终把页大小限制在一个安全区间内。
return max(1, min(numeric, MAX_PAGE_SIZE))
def clamp_page(value: Any, default: int = 1) -> int:
try:
# 页码同样允许前端传字符串。
numeric = int(value)
except Exception: # noqa: BLE001
numeric = default
# 页码至少从 1 开始。
return max(1, numeric)
def require_name(value: Any, field_name: str) -> str:
# 数据库名、集合名、索引名都按最基础的字符串校验处理。
name = str(value or "").strip()
# 名称缺失时直接报错。
if not name:
raise ValueError(f"{field_name} is required")
# Mongo 名称里不允许带空字节。
if "\x00" in name:
raise ValueError(f"{field_name} contains invalid character")
return name
def ensure_mutable_collection_name(name: str) -> str:
collection_name = require_name(name, "collection")
# 系统集合默认不开放结构变更,避免误删系统元数据。
if collection_name.startswith("system."):
raise ValueError("system collections are read only in this page")
return collection_name
def parse_extended_json(value: Any, field_name: str, *, default: Any = None) -> Any:
# 空值时直接回退默认值,方便页面把空输入视作 {} 或 []。
if value is None:
return default
# 字符串输入按原文解析,避免前端先 stringify 再导致转义双重嵌套。
if isinstance(value, str):
text = value.strip()
if not text:
return default
else:
# 对象输入统一先转成标准 JSON 文本,再交给 bson 扩展解析。
text = json.dumps(value, ensure_ascii=False)
try:
return require_bson_json_util().loads(text)
except Exception as exc: # noqa: BLE001
raise ValueError(f"invalid {field_name}: {exc}") from exc
def dump_extended_json(value: Any, *, pretty: bool = False) -> str:
helper = require_bson_json_util()
options = getattr(helper, "RELAXED_JSON_OPTIONS", None)
# 列表页和编辑器都统一输出 relaxed extended JSON前后端回写更稳定。
return helper.dumps(
value,
ensure_ascii=False,
indent=2 if pretty else None,
json_options=options,
)
def to_builtin_json(value: Any) -> Any:
# 先转扩展 JSON再回到纯 Python 结构,这样 http_app 可以继续用标准 json.dumps。
return json.loads(dump_extended_json(value, pretty=False))
def compact_json_preview(value: Any, limit: int = PREVIEW_TEXT_LIMIT) -> str:
# 预览统一压成单行,避免表格高度失控。
compact = dump_extended_json(value, pretty=False).replace("\n", " ").strip()
if len(compact) <= limit:
return compact
return f"{compact[:limit]}..."
def mongo_document_row(document: dict[str, Any], *, page: int, position: int) -> dict[str, Any]:
# 单条文档既返回结构化内容,也返回编辑器直接可用的 JSON 文本。
document_json = dump_extended_json(document, pretty=True)
document_id = document.get("_id")
return {
"key": f"{page}-{position}-{compact_json_preview(document_id, limit=64)}",
"idText": compact_json_preview(document_id, limit=72),
"idFilterText": dump_extended_json({"_id": document_id}, pretty=True),
"previewText": compact_json_preview(document),
"jsonText": document_json,
"document": to_builtin_json(document),
}
def normalize_sort_spec(sort_value: Any) -> list[tuple[str, int]]:
parsed = parse_extended_json(sort_value, "sort", default=DEFAULT_SORT_SPEC)
# 浏览页主路径只支持对象排序,界面也按这个格式产出。
if not isinstance(parsed, dict):
raise ValueError("sort must be a json object")
items: list[tuple[str, int]] = []
for field_name, direction in parsed.items():
name = str(field_name or "").strip()
if not name:
continue
try:
numeric_direction = int(direction)
except Exception as exc: # noqa: BLE001
raise ValueError(f"invalid sort direction for {name}") from exc
if numeric_direction not in {-1, 1}:
raise ValueError(f"sort direction for {name} must be 1 or -1")
items.append((name, numeric_direction))
# 排序对象全空时回退默认值。
return items or list(DEFAULT_SORT_SPEC.items())
def normalize_index_keys(keys_value: Any) -> list[tuple[str, Any]]:
parsed = parse_extended_json(keys_value, "keys", default={})
# 索引创建页同样只接受对象格式。
if not isinstance(parsed, dict):
raise ValueError("keys must be a json object")
keys: list[tuple[str, Any]] = []
for field_name, direction in parsed.items():
name = str(field_name or "").strip()
if not name:
continue
if isinstance(direction, str):
normalized_direction = direction.strip().lower()
if normalized_direction in ALLOWED_INDEX_DIRECTIONS:
keys.append((name, normalized_direction))
continue
try:
numeric_direction = int(normalized_direction)
except Exception as exc: # noqa: BLE001
raise ValueError(
f"index direction for {name} must be 1, -1, text, hashed, 2dsphere or 2d",
) from exc
else:
try:
numeric_direction = int(direction)
except Exception as exc: # noqa: BLE001
raise ValueError(
f"index direction for {name} must be 1, -1, text, hashed, 2dsphere or 2d",
) from exc
if numeric_direction not in {-1, 1}:
raise ValueError(f"index direction for {name} must be 1, -1, text, hashed, 2dsphere or 2d")
keys.append((name, numeric_direction))
if not keys:
raise ValueError("keys must not be empty")
return keys
def require_json_object(value: Any, field_name: str, *, default: dict[str, Any] | None = None) -> dict[str, Any]:
parsed = parse_extended_json(value, field_name, default=default)
if parsed is None:
return {}
# filter / projection / options / replacement 都要求对象。
if not isinstance(parsed, dict):
raise ValueError(f"{field_name} must be a json object")
return parsed
def require_json_array(value: Any, field_name: str, *, default: list[Any] | None = None) -> list[Any]:
parsed = parse_extended_json(value, field_name, default=default)
if parsed is None:
return []
# aggregation pipeline 明确要求数组。
if not isinstance(parsed, list):
raise ValueError(f"{field_name} must be a json array")
return parsed
def require_insert_documents(value: Any) -> tuple[list[dict[str, Any]], bool]:
# Mongo 写入既支持单文档,也支持批量数组。
parsed = parse_extended_json(value, "document", default={})
if isinstance(parsed, dict):
return [parsed], False
if not isinstance(parsed, list):
raise ValueError("document must be a json object or json array")
if not parsed:
raise ValueError("document array must not be empty")
documents: list[dict[str, Any]] = []
for index, item in enumerate(parsed, start=1):
if not isinstance(item, dict):
raise ValueError(f"document[{index}] must be a json object")
documents.append(item)
return documents, True
def ensure_safe_aggregate_pipeline(pipeline: list[Any]) -> list[Any]:
for stage in pipeline:
# 每个 stage 都必须是对象,否则驱动会直接报错,这里提前统一提示。
if not isinstance(stage, dict):
raise ValueError("aggregate pipeline stage must be a json object")
# 当前页面不开放会落地写数据的聚合阶段。
stage_names = {str(key or "") for key in stage.keys()}
if "$out" in stage_names or "$merge" in stage_names:
raise ValueError("aggregate pipeline does not allow $out or $merge")
return pipeline
def collection_handle(client: Any, database_name: str, collection_name: str) -> Any:
# 集合句柄统一从这里拿,保证数据库名和集合名都先校验。
return client[require_name(database_name, "database")][require_name(collection_name, "collection")]
def collection_stats_payload(client: Any, database_name: str, collection_name: str) -> dict[str, Any]:
database = client[require_name(database_name, "database")]
collection = require_name(collection_name, "collection")
# collStats 能拿到集合大小、文档数、索引数等管理页最常用信息。
stats = database.command("collStats", collection)
index_list = list(database[collection].list_indexes())
indexes = []
for item in index_list:
key_spec = dict(item.get("key") or {})
indexes.append(
{
"name": str(item.get("name") or "").strip(),
"keys": to_builtin_json(key_spec),
"keysText": dump_extended_json(key_spec, pretty=True),
"unique": bool(item.get("unique")),
"sparse": bool(item.get("sparse")),
"hidden": bool(item.get("hidden")),
"expireAfterSeconds": item.get("expireAfterSeconds"),
"partialFilterExpression": to_builtin_json(item.get("partialFilterExpression"))
if item.get("partialFilterExpression") is not None
else None,
"partialFilterExpressionText": dump_extended_json(item.get("partialFilterExpression"), pretty=True)
if item.get("partialFilterExpression") is not None
else "",
}
)
# 把 _id_ 放在第一行,其他索引按名称排序。
indexes.sort(key=lambda item: (0 if item["name"] == "_id_" else 1, item["name"]))
return {
"database": database_name,
"collection": collection_name,
"updatedAt": utc_now(),
"stats": {
"count": int(stats.get("count") or 0),
"sizeMb": round(float(stats.get("size") or 0) / 1024 / 1024, 2),
"storageSizeMb": round(float(stats.get("storageSize") or 0) / 1024 / 1024, 2),
"avgObjSizeBytes": round(float(stats.get("avgObjSize") or 0), 2),
"indexCount": int(stats.get("nindexes") or 0),
"totalIndexSizeMb": round(float(stats.get("totalIndexSize") or 0) / 1024 / 1024, 2),
"capped": bool(stats.get("capped")),
"wiredTigerUri": str((stats.get("wiredTiger") or {}).get("uri") or "").strip(),
},
"indexes": indexes,
}
def get_mongo_admin_overview_payload() -> dict[str, Any]:
client = get_mongo_client()
try:
# 概览页既要列数据库,也要列集合,所以这里一次性把树形结构拼完。
databases_raw = list((client.admin.command("listDatabases", 1).get("databases") or []))
overview_items: list[dict[str, Any]] = []
for database_item in databases_raw:
database_name = str(database_item.get("name") or "").strip()
if not database_name:
continue
database = client[database_name]
collection_items: list[dict[str, Any]] = []
# listCollections 先拿到集合名和类型,再按需补充 stats。
for collection_info in database.list_collections():
collection_name = str(collection_info.get("name") or "").strip()
if not collection_name:
continue
try:
collection_stats = database.command("collStats", collection_name)
collection_items.append(
{
"name": collection_name,
"type": str(collection_info.get("type") or "collection").strip() or "collection",
"documentCount": int(collection_stats.get("count") or 0),
"sizeMb": round(float(collection_stats.get("size") or 0) / 1024 / 1024, 2),
"storageSizeMb": round(
float(collection_stats.get("storageSize") or 0) / 1024 / 1024,
2,
),
"indexCount": int(collection_stats.get("nindexes") or 0),
"capped": bool(collection_stats.get("capped")),
"error": "",
}
)
except Exception as exc: # noqa: BLE001
# 某个集合取 stats 失败时仍保留在树里,方便继续点进去排查。
collection_items.append(
{
"name": collection_name,
"type": str(collection_info.get("type") or "collection").strip() or "collection",
"documentCount": 0,
"sizeMb": 0.0,
"storageSizeMb": 0.0,
"indexCount": 0,
"capped": False,
"error": str(exc),
}
)
# 集合列表默认按名字排序,侧边 explorer 更稳定。
collection_items.sort(key=lambda item: item["name"].lower())
overview_items.append(
{
"name": database_name,
"empty": bool(database_item.get("empty")),
"sizeOnDiskMb": round(float(database_item.get("sizeOnDisk") or 0) / 1024 / 1024, 2),
"collectionCount": len(collection_items),
"collections": collection_items,
}
)
# 聚焦库优先,其余按磁盘占用倒序,和现有 Mongo 指标页一致。
overview_items.sort(
key=lambda item: (
0 if item["name"] == MONGO_DB_NAME else 1,
-float(item.get("sizeOnDiskMb") or 0.0),
item["name"].lower(),
)
)
default_database = next((item["name"] for item in overview_items if item["name"] == MONGO_DB_NAME), "")
if not default_database and overview_items:
default_database = overview_items[0]["name"]
default_collection = ""
if default_database:
default_database_item = next((item for item in overview_items if item["name"] == default_database), None)
if default_database_item and default_database_item["collections"]:
default_collection = default_database_item["collections"][0]["name"]
return {
"ok": True,
"updatedAt": utc_now(),
"focusedDatabase": MONGO_DB_NAME,
"selectedDatabase": default_database,
"selectedCollection": default_collection,
"items": overview_items,
}
finally:
# 管理页操作完成后立即关连接,避免长期挂着无用游标。
client.close()
def get_mongo_collection_detail_payload(database_name: str, collection_name: str) -> dict[str, Any]:
client = get_mongo_client()
try:
return collection_stats_payload(client, database_name, collection_name)
finally:
client.close()
def query_mongo_documents_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = require_name(payload.get("collection"), "collection")
page = clamp_page(payload.get("page"), default=1)
page_size = clamp_page_size(payload.get("pageSize"), default=DEFAULT_PAGE_SIZE)
filter_spec = require_json_object(payload.get("filter"), "filter", default={})
projection_spec = require_json_object(payload.get("projection"), "projection", default={})
sort_spec = normalize_sort_spec(payload.get("sort"))
# 页面上的编辑和删除都依赖 _id所以这里强制保留。
if projection_spec.get("_id") == 0:
projection_spec = dict(projection_spec)
projection_spec.pop("_id", None)
client = get_mongo_client()
try:
collection = collection_handle(client, database_name, collection_name)
total = collection.count_documents(filter_spec)
skip = max(page - 1, 0) * page_size
cursor = collection.find(filter_spec, projection_spec or None).sort(sort_spec).skip(skip).limit(page_size)
documents = list(cursor)
rows = [
mongo_document_row(document, page=page, position=skip + index)
for index, document in enumerate(documents, start=1)
]
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"page": page,
"pageSize": page_size,
"total": total,
"pageCount": max((total + page_size - 1) // page_size, 1),
"filterText": dump_extended_json(filter_spec, pretty=True),
"projectionText": dump_extended_json(projection_spec, pretty=True),
"sortText": dump_extended_json(dict(sort_spec), pretty=True),
"items": rows,
}
finally:
client.close()
def insert_mongo_document_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = ensure_mutable_collection_name(payload.get("collection"))
documents, many = require_insert_documents(payload.get("document"))
ordered = bool(payload.get("ordered", True))
client = get_mongo_client()
try:
collection = collection_handle(client, database_name, collection_name)
if many:
result = collection.insert_many(documents, ordered=ordered)
inserted_ids = [to_builtin_json(inserted_id) for inserted_id in result.inserted_ids]
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"insertedCount": len(inserted_ids),
"insertedIds": inserted_ids,
"message": "insertMany completed",
}
result = collection.insert_one(documents[0])
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"insertedCount": 1,
"insertedId": to_builtin_json(result.inserted_id),
"insertedIdText": dump_extended_json(result.inserted_id, pretty=False),
"message": "insertOne completed",
}
finally:
client.close()
def replace_mongo_document_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = ensure_mutable_collection_name(payload.get("collection"))
filter_spec = require_json_object(payload.get("filter"), "filter", default={})
replacement = require_json_object(payload.get("document"), "document")
upsert = bool(payload.get("upsert"))
# replaceOne 只接受完整文档,不能混进 $set 这类更新操作符。
if any(str(key).startswith("$") for key in replacement.keys()):
raise ValueError("replace document must not contain update operators")
client = get_mongo_client()
try:
result = collection_handle(client, database_name, collection_name).replace_one(
filter_spec,
replacement,
upsert=upsert,
)
return {
"ok": True,
"updatedAt": utc_now(),
"matchedCount": int(result.matched_count),
"modifiedCount": int(result.modified_count),
"upsertedId": to_builtin_json(result.upserted_id) if result.upserted_id is not None else None,
"message": "replaceOne completed",
}
finally:
client.close()
def update_mongo_documents_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = ensure_mutable_collection_name(payload.get("collection"))
filter_spec = require_json_object(payload.get("filter"), "filter", default={})
update_spec = parse_extended_json(payload.get("update"), "update", default={})
many = bool(payload.get("many"))
upsert = bool(payload.get("upsert"))
allow_empty_filter = bool(payload.get("allowEmptyFilter"))
# 空 filter + 多文档更新风险过高,前端必须显式确认。
if many and not filter_spec and not allow_empty_filter:
raise ValueError("empty filter requires explicit confirmation")
# update 支持操作符对象或更新管道数组。
if isinstance(update_spec, dict):
if not update_spec:
raise ValueError("update must not be empty")
if not any(str(key).startswith("$") for key in update_spec.keys()):
raise ValueError("update must contain update operators or use replace")
elif isinstance(update_spec, list):
ensure_safe_aggregate_pipeline(update_spec)
else:
raise ValueError("update must be a json object or json array")
client = get_mongo_client()
try:
collection = collection_handle(client, database_name, collection_name)
result = (
collection.update_many(filter_spec, update_spec, upsert=upsert)
if many
else collection.update_one(filter_spec, update_spec, upsert=upsert)
)
return {
"ok": True,
"updatedAt": utc_now(),
"matchedCount": int(result.matched_count),
"modifiedCount": int(result.modified_count),
"upsertedId": to_builtin_json(result.upserted_id) if result.upserted_id is not None else None,
"message": "updateMany completed" if many else "updateOne completed",
}
finally:
client.close()
def delete_mongo_documents_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = ensure_mutable_collection_name(payload.get("collection"))
filter_spec = require_json_object(payload.get("filter"), "filter", default={})
many = bool(payload.get("many"))
allow_empty_filter = bool(payload.get("allowEmptyFilter"))
# deleteMany 空 filter 同样要求显式确认。
if many and not filter_spec and not allow_empty_filter:
raise ValueError("empty filter requires explicit confirmation")
client = get_mongo_client()
try:
collection = collection_handle(client, database_name, collection_name)
result = collection.delete_many(filter_spec) if many else collection.delete_one(filter_spec)
return {
"ok": True,
"updatedAt": utc_now(),
"deletedCount": int(result.deleted_count),
"message": "deleteMany completed" if many else "deleteOne completed",
}
finally:
client.close()
def aggregate_mongo_documents_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = require_name(payload.get("collection"), "collection")
page = clamp_page(payload.get("page"), default=1)
page_size = clamp_page_size(payload.get("pageSize"), default=DEFAULT_PAGE_SIZE)
pipeline = ensure_safe_aggregate_pipeline(require_json_array(payload.get("pipeline"), "pipeline", default=[]))
client = get_mongo_client()
try:
collection = collection_handle(client, database_name, collection_name)
skip = max(page - 1, 0) * page_size
# 先跑计数管道,再跑当前页数据。
total_result = list(collection.aggregate([*pipeline, {"$count": "total"}]))
total = int((total_result[0] or {}).get("total") or 0) if total_result else 0
documents = list(collection.aggregate([*pipeline, {"$skip": skip}, {"$limit": page_size}]))
rows = [
{
"key": f"agg-{page}-{skip + index}",
"idText": compact_json_preview(document.get("_id"), limit=72) if "_id" in document else "-",
"idFilterText": dump_extended_json({"_id": document.get("_id")}, pretty=True)
if "_id" in document
else "{}",
"previewText": compact_json_preview(document),
"jsonText": dump_extended_json(document, pretty=True),
"document": to_builtin_json(document),
}
for index, document in enumerate(documents, start=1)
]
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"page": page,
"pageSize": page_size,
"total": total,
"pageCount": max((total + page_size - 1) // page_size, 1),
"pipelineText": dump_extended_json(pipeline, pretty=True),
"items": rows,
}
finally:
client.close()
def distinct_mongo_values_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = require_name(payload.get("collection"), "collection")
field_name = require_name(payload.get("field"), "field")
filter_spec = require_json_object(payload.get("filter"), "filter", default={})
client = get_mongo_client()
try:
values = collection_handle(client, database_name, collection_name).distinct(field_name, filter_spec)
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"field": field_name,
"filterText": dump_extended_json(filter_spec, pretty=True),
"count": len(values),
"values": to_builtin_json(values),
"valuesText": dump_extended_json(values, pretty=True),
}
finally:
client.close()
def create_mongo_index_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = ensure_mutable_collection_name(payload.get("collection"))
index_keys = normalize_index_keys(payload.get("keys"))
index_options = require_json_object(payload.get("options"), "options", default={})
# name 允许前端留空,驱动会自动生成。
if not str(index_options.get("name") or "").strip():
index_options.pop("name", None)
client = get_mongo_client()
try:
index_name = collection_handle(client, database_name, collection_name).create_index(index_keys, **index_options)
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"indexName": str(index_name),
"message": "createIndex completed",
}
finally:
client.close()
def drop_mongo_index_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = ensure_mutable_collection_name(payload.get("collection"))
index_name = require_name(payload.get("indexName"), "indexName")
client = get_mongo_client()
try:
collection_handle(client, database_name, collection_name).drop_index(index_name)
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"indexName": index_name,
"message": "dropIndex completed",
}
finally:
client.close()
def drop_mongo_secondary_indexes_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = ensure_mutable_collection_name(payload.get("collection"))
client = get_mongo_client()
try:
collection = collection_handle(client, database_name, collection_name)
index_names = [
str(item.get("name") or "").strip()
for item in collection.list_indexes()
if str(item.get("name") or "").strip() and str(item.get("name") or "").strip() != "_id_"
]
# Mongo 的 drop_indexes 会保留 _id_ 索引,这里先把删除名单回给前端方便确认结果。
if not index_names:
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"droppedCount": 0,
"droppedNames": [],
"message": "no secondary indexes",
}
collection.drop_indexes()
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"droppedCount": len(index_names),
"droppedNames": index_names,
"message": "dropIndexes completed",
}
finally:
client.close()
def create_mongo_collection_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = ensure_mutable_collection_name(payload.get("collection"))
options = require_json_object(payload.get("options"), "options", default={})
client = get_mongo_client()
try:
client[database_name].create_collection(collection_name, **options)
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"message": "createCollection completed",
}
finally:
client.close()
def rename_mongo_collection_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
source_name = ensure_mutable_collection_name(payload.get("collection"))
target_name = ensure_mutable_collection_name(payload.get("targetCollection"))
drop_target = bool(payload.get("dropTarget"))
client = get_mongo_client()
try:
collection_handle(client, database_name, source_name).rename(target_name, dropTarget=drop_target)
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": source_name,
"targetCollection": target_name,
"message": "renameCollection completed",
}
finally:
client.close()
def drop_mongo_collection_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
collection_name = ensure_mutable_collection_name(payload.get("collection"))
confirm_name = require_name(payload.get("confirmName"), "confirmName")
# 删除集合前要求前端把目标名再传一遍,减少误触。
if confirm_name != collection_name:
raise ValueError("confirmName does not match collection")
client = get_mongo_client()
try:
collection_handle(client, database_name, collection_name).drop()
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"collection": collection_name,
"message": "dropCollection completed",
}
finally:
client.close()
def drop_mongo_database_payload(payload: dict[str, Any]) -> dict[str, Any]:
database_name = require_name(payload.get("database"), "database")
confirm_name = require_name(payload.get("confirmName"), "confirmName")
# 删除数据库同样要求二次校验目标名。
if confirm_name != database_name:
raise ValueError("confirmName does not match database")
client = get_mongo_client()
try:
client.drop_database(database_name)
return {
"ok": True,
"updatedAt": utc_now(),
"database": database_name,
"message": "dropDatabase completed",
}
finally:
client.close()

View File

@ -3,8 +3,8 @@ from __future__ import annotations
from datetime import datetime, timezone
from typing import Any
from .cache import TTLCache
from .config import (
from core.cache import TTLCache
from core.config import (
MONGO_CACHE_TTL_SECONDS,
MONGO_CONNECT_TIMEOUT_MS,
MONGO_DB_NAME,

View File

@ -0,0 +1,719 @@
from __future__ import annotations
import json
from typing import Any
from core.config import MYSQL_DEFAULT_DATABASE, MYSQL_DEFAULT_PAGE_SIZE, MYSQL_MAX_PAGE_SIZE, MYSQL_QUERY_MAX_ROWS, utc_now
from .mysql_common import (
build_mysql_connection,
build_row_identity,
database_exists,
get_preferred_identity_columns,
list_database_summaries,
list_table_columns,
list_table_indexes,
list_table_summaries,
normalize_column_name,
normalize_database_name,
normalize_mysql_row,
normalize_sql_text,
normalize_table_name,
quote_identifier,
sql_statement_kind,
table_exists,
)
from .mysql_metrics import clear_mysql_metrics_cache, get_mysql_overview_payload
def browser_error_payload(error: str) -> dict[str, Any]:
# 浏览器和 SQL 工作台都统一返回这套最小错误结构。
return {"ok": False, "updatedAt": utc_now(), "error": error}
def clamp_page_size(value: Any) -> int:
try:
# 前端没传时回到默认 100过大时强制截断避免一次拖太多数据。
return max(1, min(int(value or MYSQL_DEFAULT_PAGE_SIZE), MYSQL_MAX_PAGE_SIZE))
except Exception: # noqa: BLE001
return MYSQL_DEFAULT_PAGE_SIZE
def clamp_page_number(value: Any) -> int:
try:
# 页码最小固定为 1。
return max(1, int(value or 1))
except Exception: # noqa: BLE001
return 1
def normalize_sql_fragment(value: Any, *, field_name: str) -> str:
# DDL 片段允许空格和括号,但不允许多语句。
actual = str(value or "").strip()
if not actual:
raise ValueError(f"{field_name} is required")
if ";" in actual:
raise ValueError(f"{field_name} must be a single fragment")
return actual
def get_mysql_databases_payload() -> dict[str, Any]:
# 概览里已经做了缓存,这里直接复用,避免重复查 information_schema。
overview = get_mysql_overview_payload()
if not overview.get("ok"):
return {
"ok": False,
"updatedAt": overview.get("updatedAt"),
"selectedDatabase": "",
"items": [],
"error": overview.get("error") or "mysql unavailable",
}
items = list(overview.get("databases") or [])
business_items = [item for item in items if not item.get("system")]
preferred = str(MYSQL_DEFAULT_DATABASE or "").strip()
# 默认优先选 .env 指定库;没有时选第一个业务库,再退到第一个可见库。
selected_database = (
preferred
if preferred and any(item.get("name") == preferred for item in items)
else (business_items[0]["name"] if business_items else (items[0]["name"] if items else ""))
)
return {
"ok": True,
"updatedAt": overview.get("updatedAt"),
"selectedDatabase": selected_database,
"items": items,
"error": "",
}
def get_mysql_tables_payload(database: str) -> dict[str, Any]:
target_database = normalize_database_name(database)
try:
with build_mysql_connection(autocommit=True) as connection:
if not database_exists(connection, target_database):
raise ValueError(f"database not found: {target_database}")
items = list_table_summaries(connection, target_database)
except Exception as exc: # noqa: BLE001
return {
"ok": False,
"updatedAt": utc_now(),
"database": target_database,
"items": [],
"error": str(exc),
}
return {
"ok": True,
"updatedAt": utc_now(),
"database": target_database,
"selectedTable": items[0]["name"] if items else "",
"items": items,
"error": "",
}
def get_mysql_table_schema_payload(database: str, table: str) -> dict[str, Any]:
target_database = normalize_database_name(database)
target_table = normalize_table_name(table)
try:
with build_mysql_connection(target_database, autocommit=True) as connection:
if not table_exists(connection, target_database, target_table):
raise ValueError(f"table not found: {target_database}.{target_table}")
tables = list_table_summaries(connection, target_database)
table_summary = next((item for item in tables if item.get("name") == target_table), None)
columns = list_table_columns(connection, target_database, target_table)
indexes = list_table_indexes(connection, target_database, target_table)
with connection.cursor() as cursor:
# SHOW CREATE TABLE 能拿到最完整的 DDL方便页面直接展示和复制。
cursor.execute(f"SHOW CREATE TABLE {quote_identifier(target_table)}")
create_row = cursor.fetchone() or {}
except Exception as exc: # noqa: BLE001
return {
"ok": False,
"updatedAt": utc_now(),
"database": target_database,
"table": target_table,
"columns": [],
"indexes": [],
"createTable": "",
"error": str(exc),
}
return {
"ok": True,
"updatedAt": utc_now(),
"database": target_database,
"table": target_table,
"tableSummary": table_summary or {},
"columns": columns,
"indexes": indexes,
"identity": get_preferred_identity_columns(columns, indexes),
"createTable": str(create_row.get("Create Table") or "").strip(),
"error": "",
}
def build_filter_clause(
columns: list[dict[str, Any]],
keyword: str,
filters: list[dict[str, Any]],
) -> tuple[str, list[Any]]:
column_names = {column["name"] for column in columns}
clauses: list[str] = []
params: list[Any] = []
normalized_keyword = str(keyword or "").strip()
# 关键字搜索默认对当前表全部列做模糊匹配,适合值班快速扫数据。
if normalized_keyword:
search_clauses: list[str] = []
for column in columns:
search_clauses.append(f"CAST({quote_identifier(column['name'])} AS CHAR) LIKE %s")
params.append(f"%{normalized_keyword}%")
if search_clauses:
clauses.append(f"({' OR '.join(search_clauses)})")
# 附加过滤器支持前端后续扩展成多条件。
for item in filters:
column_name = normalize_column_name(item.get("column"))
if column_name not in column_names:
raise ValueError(f"unknown filter column: {column_name}")
operator = str(item.get("operator") or "eq").strip().lower()
sql_column = quote_identifier(column_name)
value = item.get("value")
if operator == "eq":
if value is None:
clauses.append(f"{sql_column} IS NULL")
else:
clauses.append(f"{sql_column} = %s")
params.append(value)
continue
if operator == "ne":
if value is None:
clauses.append(f"{sql_column} IS NOT NULL")
else:
clauses.append(f"{sql_column} <> %s")
params.append(value)
continue
if operator == "gt":
clauses.append(f"{sql_column} > %s")
params.append(value)
continue
if operator == "ge":
clauses.append(f"{sql_column} >= %s")
params.append(value)
continue
if operator == "lt":
clauses.append(f"{sql_column} < %s")
params.append(value)
continue
if operator == "le":
clauses.append(f"{sql_column} <= %s")
params.append(value)
continue
if operator == "in":
if not isinstance(value, list) or not value:
raise ValueError("in filter requires a non-empty array value")
placeholders = ", ".join(["%s"] * len(value))
clauses.append(f"{sql_column} IN ({placeholders})")
params.extend(value)
continue
if operator == "not_in":
if not isinstance(value, list) or not value:
raise ValueError("not_in filter requires a non-empty array value")
placeholders = ", ".join(["%s"] * len(value))
clauses.append(f"{sql_column} NOT IN ({placeholders})")
params.extend(value)
continue
if operator == "between":
if not isinstance(value, list) or len(value) != 2:
raise ValueError("between filter requires a two-item array value")
clauses.append(f"{sql_column} BETWEEN %s AND %s")
params.extend(value)
continue
if operator == "not_between":
if not isinstance(value, list) or len(value) != 2:
raise ValueError("not_between filter requires a two-item array value")
clauses.append(f"{sql_column} NOT BETWEEN %s AND %s")
params.extend(value)
continue
if operator == "like":
clauses.append(f"CAST({sql_column} AS CHAR) LIKE %s")
params.append(f"%{value}%")
continue
if operator == "not_like":
clauses.append(f"CAST({sql_column} AS CHAR) NOT LIKE %s")
params.append(f"%{value}%")
continue
if operator == "is_null":
clauses.append(f"{sql_column} IS NULL")
continue
if operator == "is_not_null":
clauses.append(f"{sql_column} IS NOT NULL")
continue
raise ValueError(f"unsupported filter operator: {operator}")
if not clauses:
return "", params
return f"WHERE {' AND '.join(clauses)}", params
def query_mysql_table_rows_payload(payload: dict[str, Any]) -> dict[str, Any]:
target_database = normalize_database_name(payload.get("database"))
target_table = normalize_table_name(payload.get("table"))
page = clamp_page_number(payload.get("page"))
page_size = clamp_page_size(payload.get("pageSize"))
keyword = str(payload.get("keyword") or "").strip()
filters = payload.get("filters") if isinstance(payload.get("filters"), list) else []
try:
with build_mysql_connection(target_database, autocommit=True) as connection:
if not table_exists(connection, target_database, target_table):
raise ValueError(f"table not found: {target_database}.{target_table}")
columns = list_table_columns(connection, target_database, target_table)
indexes = list_table_indexes(connection, target_database, target_table)
identity_spec = get_preferred_identity_columns(columns, indexes)
column_names = {column["name"] for column in columns}
requested_sort = str(payload.get("orderBy") or "").strip()
order_by = requested_sort if requested_sort in column_names else (identity_spec["columns"][0] if identity_spec["columns"] else columns[0]["name"])
order_direction = "DESC" if str(payload.get("orderDirection") or "").strip().lower() == "desc" else "ASC"
where_clause, params = build_filter_clause(columns, keyword, filters)
offset = (page - 1) * page_size
quoted_table = quote_identifier(target_table)
select_columns = ", ".join(quote_identifier(column["name"]) for column in columns)
with connection.cursor() as cursor:
# 先统计总量,分页器和页数都依赖这个值。
cursor.execute(
f"SELECT COUNT(*) AS totalRows FROM {quoted_table} {where_clause}",
tuple(params),
)
total_rows = int((cursor.fetchone() or {}).get("totalRows") or 0)
# 行浏览默认按指定列排序,避免翻页时顺序漂移。
cursor.execute(
f"""
SELECT {select_columns}
FROM {quoted_table}
{where_clause}
ORDER BY {quote_identifier(order_by)} {order_direction}
LIMIT %s OFFSET %s
""",
tuple([*params, page_size, offset]),
)
rows = cursor.fetchall() or []
except Exception as exc: # noqa: BLE001
return {
"ok": False,
"updatedAt": utc_now(),
"database": target_database,
"table": target_table,
"rows": [],
"totalRows": 0,
"error": str(exc),
}
# 每一行都带 identity后续 update / delete 不需要前端自己猜 where 条件。
normalized_rows = []
for row in rows:
normalized_rows.append(
{
"values": normalize_mysql_row(row),
"identity": build_row_identity(row, identity_spec),
}
)
return {
"ok": True,
"updatedAt": utc_now(),
"database": target_database,
"table": target_table,
"page": page,
"pageSize": page_size,
"totalRows": total_rows,
"totalPages": max(1, (total_rows + page_size - 1) // page_size) if total_rows > 0 else 1,
"keyword": keyword,
"orderBy": order_by,
"orderDirection": order_direction.lower(),
"columns": columns,
"indexes": indexes,
"identity": identity_spec,
"rows": normalized_rows,
"error": "",
}
def normalize_write_values(values: Any, columns: list[dict[str, Any]]) -> list[tuple[str, Any]]:
if not isinstance(values, dict):
raise ValueError("values must be an object")
writable_columns = {
column["name"]: column
for column in columns
if "generated" not in str(column.get("extra") or "").strip().lower()
}
normalized: list[tuple[str, Any]] = []
# 只允许写入真实存在的列,避免页面或调用方传进来脏字段。
for raw_name, raw_value in values.items():
column_name = normalize_column_name(raw_name)
if column_name not in writable_columns:
raise ValueError(f"unknown writable column: {column_name}")
# JSON 对象和数组转成字符串后再交给 MySQL自定义 SQL 控制台仍可手写更复杂格式。
if isinstance(raw_value, (dict, list)):
normalized.append((column_name, json.dumps(raw_value, ensure_ascii=False)))
else:
normalized.append((column_name, raw_value))
return normalized
def build_identity_clause(identity: dict[str, Any], columns: list[dict[str, Any]]) -> tuple[str, list[Any]]:
if not isinstance(identity, dict):
raise ValueError("identity is required")
allowed_columns = {column["name"] for column in columns}
identity_columns = [normalize_column_name(item) for item in identity.get("columns") or []]
identity_values = identity.get("values") if isinstance(identity.get("values"), dict) else {}
if not identity_columns:
raise ValueError("identity columns are required")
clauses: list[str] = []
params: list[Any] = []
# where 条件只按 identity 指定列构造,避免把整行都塞进 where。
for column_name in identity_columns:
if column_name not in allowed_columns:
raise ValueError(f"unknown identity column: {column_name}")
if column_name not in identity_values:
raise ValueError(f"missing identity value: {column_name}")
value = identity_values.get(column_name)
quoted = quote_identifier(column_name)
if value is None:
clauses.append(f"{quoted} IS NULL")
else:
clauses.append(f"{quoted} = %s")
params.append(value)
return " AND ".join(clauses), params
def insert_mysql_row_payload(payload: dict[str, Any]) -> dict[str, Any]:
target_database = normalize_database_name(payload.get("database"))
target_table = normalize_table_name(payload.get("table"))
try:
with build_mysql_connection(target_database, autocommit=False) as connection:
columns = list_table_columns(connection, target_database, target_table)
pairs = normalize_write_values(payload.get("values"), columns)
if not pairs:
raise ValueError("at least one column value is required")
quoted_table = quote_identifier(target_table)
column_sql = ", ".join(quote_identifier(name) for name, _ in pairs)
placeholders = ", ".join(["%s"] * len(pairs))
params = [value for _, value in pairs]
with connection.cursor() as cursor:
# 插入时只拼接列名,值全部走参数化。
cursor.execute(
f"INSERT INTO {quoted_table} ({column_sql}) VALUES ({placeholders})",
tuple(params),
)
affected_rows = int(cursor.rowcount or 0)
last_insert_id = cursor.lastrowid
connection.commit()
except Exception as exc: # noqa: BLE001
return browser_error_payload(str(exc))
clear_mysql_metrics_cache()
return {
"ok": True,
"updatedAt": utc_now(),
"message": f"已插入 {target_database}.{target_table}",
"affectedRows": affected_rows,
"lastInsertId": last_insert_id,
"error": "",
}
def update_mysql_row_payload(payload: dict[str, Any]) -> dict[str, Any]:
target_database = normalize_database_name(payload.get("database"))
target_table = normalize_table_name(payload.get("table"))
try:
with build_mysql_connection(target_database, autocommit=False) as connection:
columns = list_table_columns(connection, target_database, target_table)
pairs = normalize_write_values(payload.get("values"), columns)
if not pairs:
raise ValueError("at least one column value is required")
where_clause, where_params = build_identity_clause(payload.get("identity"), columns)
set_clause = ", ".join(f"{quote_identifier(name)} = %s" for name, _ in pairs)
params = [value for _, value in pairs] + where_params
with connection.cursor() as cursor:
# 更新固定只影响一行identity 使用主键或唯一索引时语义最稳定。
cursor.execute(
f"UPDATE {quote_identifier(target_table)} SET {set_clause} WHERE {where_clause} LIMIT 1",
tuple(params),
)
affected_rows = int(cursor.rowcount or 0)
connection.commit()
except Exception as exc: # noqa: BLE001
return browser_error_payload(str(exc))
clear_mysql_metrics_cache()
return {
"ok": True,
"updatedAt": utc_now(),
"message": f"已更新 {target_database}.{target_table}",
"affectedRows": affected_rows,
"error": "",
}
def delete_mysql_row_payload(payload: dict[str, Any]) -> dict[str, Any]:
target_database = normalize_database_name(payload.get("database"))
target_table = normalize_table_name(payload.get("table"))
try:
with build_mysql_connection(target_database, autocommit=False) as connection:
columns = list_table_columns(connection, target_database, target_table)
where_clause, where_params = build_identity_clause(payload.get("identity"), columns)
with connection.cursor() as cursor:
# 删除同样只允许命中一行,避免前端误带了过宽的 where 条件。
cursor.execute(
f"DELETE FROM {quote_identifier(target_table)} WHERE {where_clause} LIMIT 1",
tuple(where_params),
)
affected_rows = int(cursor.rowcount or 0)
connection.commit()
except Exception as exc: # noqa: BLE001
return browser_error_payload(str(exc))
clear_mysql_metrics_cache()
return {
"ok": True,
"updatedAt": utc_now(),
"message": f"已删除 {target_database}.{target_table} 中的 1 行",
"affectedRows": affected_rows,
"error": "",
}
def add_mysql_column_payload(payload: dict[str, Any]) -> dict[str, Any]:
target_database = normalize_database_name(payload.get("database"))
target_table = normalize_table_name(payload.get("table"))
column_name = normalize_column_name(payload.get("columnName"))
column_type = normalize_sql_fragment(payload.get("columnType"), field_name="columnType")
nullable = bool(payload.get("nullable", True))
comment = str(payload.get("comment") or "").strip()
position = str(payload.get("position") or "last").strip().lower()
default_mode = str(payload.get("defaultMode") or "none").strip().lower()
default_value = payload.get("defaultValue")
clauses = [
f"ALTER TABLE {quote_identifier(target_table)} ADD COLUMN {quote_identifier(column_name)} {column_type}",
"NULL" if nullable else "NOT NULL",
]
params: list[Any] = []
# 默认值支持最常用的三种模式;更复杂表达式留给 SQL 工作台。
if default_mode == "literal":
clauses.append("DEFAULT %s")
params.append(default_value)
elif default_mode == "null":
clauses.append("DEFAULT NULL")
elif default_mode != "none":
raise ValueError(f"unsupported default mode: {default_mode}")
if comment:
clauses.append("COMMENT %s")
params.append(comment)
if position == "first":
clauses.append("FIRST")
elif position == "after":
after_column = normalize_column_name(payload.get("afterColumn"))
clauses.append(f"AFTER {quote_identifier(after_column)}")
elif position != "last":
raise ValueError(f"unsupported position: {position}")
try:
with build_mysql_connection(target_database, autocommit=False) as connection:
with connection.cursor() as cursor:
# 列类型本身是 SQL 语法片段,只做单语句校验,不做过度限制。
cursor.execute(" ".join(clauses), tuple(params))
connection.commit()
except Exception as exc: # noqa: BLE001
return browser_error_payload(str(exc))
clear_mysql_metrics_cache()
return {
"ok": True,
"updatedAt": utc_now(),
"message": f"已新增列 {target_table}.{column_name}",
"error": "",
}
def drop_mysql_column_payload(payload: dict[str, Any]) -> dict[str, Any]:
target_database = normalize_database_name(payload.get("database"))
target_table = normalize_table_name(payload.get("table"))
column_name = normalize_column_name(payload.get("columnName"))
try:
with build_mysql_connection(target_database, autocommit=False) as connection:
with connection.cursor() as cursor:
# 删除列是高影响操作,但用户明确要求支持,这里保持单列单次执行。
cursor.execute(
f"ALTER TABLE {quote_identifier(target_table)} DROP COLUMN {quote_identifier(column_name)}",
)
connection.commit()
except Exception as exc: # noqa: BLE001
return browser_error_payload(str(exc))
clear_mysql_metrics_cache()
return {
"ok": True,
"updatedAt": utc_now(),
"message": f"已删除列 {target_table}.{column_name}",
"error": "",
}
def add_mysql_index_payload(payload: dict[str, Any]) -> dict[str, Any]:
target_database = normalize_database_name(payload.get("database"))
target_table = normalize_table_name(payload.get("table"))
index_name = normalize_column_name(payload.get("indexName"))
index_mode = str(payload.get("indexMode") or "normal").strip().lower()
columns = payload.get("columns") if isinstance(payload.get("columns"), list) else []
normalized_columns = [normalize_column_name(item) for item in columns]
if not normalized_columns:
raise ValueError("at least one index column is required")
if index_mode == "unique":
prefix = f"ADD UNIQUE INDEX {quote_identifier(index_name)}"
elif index_mode == "fulltext":
prefix = f"ADD FULLTEXT INDEX {quote_identifier(index_name)}"
elif index_mode == "normal":
prefix = f"ADD INDEX {quote_identifier(index_name)}"
else:
raise ValueError(f"unsupported index mode: {index_mode}")
column_sql = ", ".join(quote_identifier(item) for item in normalized_columns)
try:
with build_mysql_connection(target_database, autocommit=False) as connection:
with connection.cursor() as cursor:
# 索引列名都走标识符转义,避免前端选择值被直接拼进 SQL。
cursor.execute(
f"ALTER TABLE {quote_identifier(target_table)} {prefix} ({column_sql})",
)
connection.commit()
except Exception as exc: # noqa: BLE001
return browser_error_payload(str(exc))
clear_mysql_metrics_cache()
return {
"ok": True,
"updatedAt": utc_now(),
"message": f"已新增索引 {target_table}.{index_name}",
"error": "",
}
def drop_mysql_index_payload(payload: dict[str, Any]) -> dict[str, Any]:
target_database = normalize_database_name(payload.get("database"))
target_table = normalize_table_name(payload.get("table"))
index_name = normalize_column_name(payload.get("indexName"))
try:
with build_mysql_connection(target_database, autocommit=False) as connection:
with connection.cursor() as cursor:
# PRIMARY KEY 的删除语法和普通索引不同,这里单独分支。
if index_name == "PRIMARY":
cursor.execute(
f"ALTER TABLE {quote_identifier(target_table)} DROP PRIMARY KEY",
)
else:
cursor.execute(
f"ALTER TABLE {quote_identifier(target_table)} DROP INDEX {quote_identifier(index_name)}",
)
connection.commit()
except Exception as exc: # noqa: BLE001
return browser_error_payload(str(exc))
clear_mysql_metrics_cache()
return {
"ok": True,
"updatedAt": utc_now(),
"message": f"已删除索引 {target_table}.{index_name}",
"error": "",
}
def execute_mysql_sql_payload(payload: dict[str, Any]) -> dict[str, Any]:
target_database = str(payload.get("database") or MYSQL_DEFAULT_DATABASE or "").strip() or None
sql = normalize_sql_text(payload.get("sql"))
kind = sql_statement_kind(sql)
mutating_kinds = {"insert", "update", "delete", "replace", "alter", "create", "drop", "truncate", "rename"}
try:
with build_mysql_connection(target_database, autocommit=False) as connection:
with connection.cursor() as cursor:
# SQL 工作台当前只执行单条语句,结果集最多保留固定行数。
cursor.execute(sql)
if cursor.description:
columns = [str(item[0] or "").strip() for item in cursor.description]
fetched_rows = cursor.fetchmany(MYSQL_QUERY_MAX_ROWS + 1)
truncated = len(fetched_rows) > MYSQL_QUERY_MAX_ROWS
rows = [normalize_mysql_row(item) for item in fetched_rows[:MYSQL_QUERY_MAX_ROWS]]
return {
"ok": True,
"updatedAt": utc_now(),
"database": target_database or "",
"kind": kind,
"columns": columns,
"rows": rows,
"rowCount": len(rows),
"truncated": truncated,
"error": "",
}
connection.commit()
affected_rows = int(cursor.rowcount or 0)
last_insert_id = cursor.lastrowid
except Exception as exc: # noqa: BLE001
return browser_error_payload(str(exc))
if kind in mutating_kinds:
clear_mysql_metrics_cache()
return {
"ok": True,
"updatedAt": utc_now(),
"database": target_database or "",
"kind": kind,
"affectedRows": affected_rows,
"lastInsertId": last_insert_id,
"message": f"{kind.upper()} 执行完成",
"error": "",
}

View File

@ -0,0 +1,421 @@
from __future__ import annotations
from datetime import date, datetime, time, timedelta
from decimal import Decimal
import json
from typing import Any
from core.config import (
MYSQL_CONNECT_TIMEOUT_SECONDS,
MYSQL_DEFAULT_DATABASE,
MYSQL_HOST,
MYSQL_PASSWORD,
MYSQL_PORT,
MYSQL_READ_TIMEOUT_SECONDS,
MYSQL_USER,
MYSQL_WRITE_TIMEOUT_SECONDS,
utc_now,
)
# 系统库默认排在业务库后面,避免主列表被系统库占满。
SYSTEM_DATABASES = {"information_schema", "performance_schema", "mysql", "sys"}
def mysql_error_payload(error: str) -> dict[str, Any]:
# MySQL 基础设施页统一返回同一套失败结构,前端不需要额外分支。
return {
"ok": False,
"updatedAt": utc_now(),
"info": {},
"connections": {},
"throughput": {},
"replication": {"configured": False, "role": "", "readOnly": False, "members": []},
"summary": {
"databaseTotal": 0,
"tableTotal": 0,
"dataSizeMb": 0.0,
"indexSizeMb": 0.0,
"totalSizeMb": 0.0,
},
"databases": [],
"error": error,
}
def require_pymysql() -> Any:
try:
# 运行时再导入驱动,避免缺少依赖时整个服务无法启动。
import pymysql
except Exception as exc: # noqa: BLE001
raise RuntimeError(f"pymysql unavailable: {exc}") from exc
# 导入成功后直接返回模块对象,调用方继续拿 DictCursor。
return pymysql
def build_mysql_connection(database: str | None = None, *, autocommit: bool = True) -> Any:
# 主机和账号缺失时直接给出明确错误,避免后面连库报一串底层异常。
missing = [name for name, value in {"MYSQL_HOST": MYSQL_HOST, "MYSQL_USER": MYSQL_USER}.items() if not str(value).strip()]
if missing:
raise RuntimeError(f"missing mysql config in .env: {', '.join(missing)}")
# 默认库允许为空;表浏览场景会显式选择数据库。
target_database = str(database or MYSQL_DEFAULT_DATABASE or "").strip() or None
pymysql = require_pymysql()
# 统一走 utf8mb4避免表结构和数据浏览时乱码。
return pymysql.connect(
host=MYSQL_HOST,
port=MYSQL_PORT,
user=MYSQL_USER,
password=MYSQL_PASSWORD,
database=target_database,
charset="utf8mb4",
cursorclass=pymysql.cursors.DictCursor,
autocommit=autocommit,
connect_timeout=MYSQL_CONNECT_TIMEOUT_SECONDS,
read_timeout=MYSQL_READ_TIMEOUT_SECONDS,
write_timeout=MYSQL_WRITE_TIMEOUT_SECONDS,
)
def quote_identifier(name: str) -> str:
# 库、表、列名都统一走反引号包裹,并对内层反引号做转义。
actual = str(name or "").strip()
if not actual:
raise ValueError("identifier is required")
return f"`{actual.replace('`', '``')}`"
def normalize_database_name(name: str) -> str:
# 数据库名作为路由参数时必须非空,统一在这里兜底校验。
actual = str(name or "").strip()
if not actual:
raise ValueError("database is required")
return actual
def normalize_table_name(name: str) -> str:
# 表名同样必须非空。
actual = str(name or "").strip()
if not actual:
raise ValueError("table is required")
return actual
def normalize_column_name(name: str) -> str:
# 列名校验单独抽出来,便于 schema 变更和排序共用。
actual = str(name or "").strip()
if not actual:
raise ValueError("column is required")
return actual
def is_system_database(name: str) -> bool:
# 系统库统一小写比较,避免大小写混淆。
return str(name or "").strip().lower() in SYSTEM_DATABASES
def normalize_mysql_value(value: Any) -> Any:
# Decimal 保留成字符串,避免金额精度被前端浮点化。
if isinstance(value, Decimal):
return str(value)
# 日期和时间都压成易读字符串。
if isinstance(value, datetime):
return value.strftime("%Y-%m-%d %H:%M:%S")
if isinstance(value, date):
return value.strftime("%Y-%m-%d")
if isinstance(value, time):
return value.strftime("%H:%M:%S")
if isinstance(value, timedelta):
return str(value)
# 字节内容尽量按 UTF-8 展示;失败时回退成十六进制。
if isinstance(value, (bytes, bytearray, memoryview)):
raw = bytes(value)
try:
return raw.decode("utf-8")
except UnicodeDecodeError:
return f"0x{raw.hex()}"
# dict / list 理论上常见于 JSON 列,这里直接转成 JSON 文本方便展示和编辑。
if isinstance(value, (dict, list)):
return json.dumps(value, ensure_ascii=False)
# 其余标量直接原样返回。
return value
def normalize_mysql_row(row: dict[str, Any]) -> dict[str, Any]:
# 一行记录中的每个字段都做 JSON 安全转换。
return {key: normalize_mysql_value(value) for key, value in row.items()}
def list_database_summaries(connection: Any) -> list[dict[str, Any]]:
# 数据库级别汇总统一从 information_schema.tables 读取,能同时拿到表数和容量。
query = """
SELECT
s.SCHEMA_NAME AS schemaName,
COALESCE(t.tableTotal, 0) AS tableTotal,
COALESCE(t.dataBytes, 0) AS dataBytes,
COALESCE(t.indexBytes, 0) AS indexBytes,
COALESCE(t.rowEstimate, 0) AS rowEstimate
FROM information_schema.SCHEMATA AS s
LEFT JOIN (
SELECT
table_schema AS schemaName,
COUNT(*) AS tableTotal,
SUM(COALESCE(data_length, 0)) AS dataBytes,
SUM(COALESCE(index_length, 0)) AS indexBytes,
SUM(COALESCE(table_rows, 0)) AS rowEstimate
FROM information_schema.TABLES
GROUP BY table_schema
) AS t
ON t.schemaName = s.SCHEMA_NAME
ORDER BY
CASE
WHEN s.SCHEMA_NAME IN ('information_schema', 'performance_schema', 'mysql', 'sys') THEN 1
ELSE 0
END,
COALESCE(t.dataBytes, 0) + COALESCE(t.indexBytes, 0) DESC,
s.SCHEMA_NAME ASC
"""
with connection.cursor() as cursor:
# 直接读取全部库级摘要,前端 sidebar 需要完整列表。
cursor.execute(query)
rows = cursor.fetchall() or []
# 把字节数转换成 MiB并附带是否系统库的标记。
return [
{
"name": str(item.get("schemaName") or "").strip(),
"system": is_system_database(item.get("schemaName")),
"tableTotal": int(item.get("tableTotal") or 0),
"rowEstimate": int(item.get("rowEstimate") or 0),
"dataSizeMb": round(float(item.get("dataBytes") or 0) / 1024 / 1024, 2),
"indexSizeMb": round(float(item.get("indexBytes") or 0) / 1024 / 1024, 2),
"totalSizeMb": round(
(float(item.get("dataBytes") or 0) + float(item.get("indexBytes") or 0)) / 1024 / 1024,
2,
),
}
for item in rows
]
def list_table_summaries(connection: Any, database: str) -> list[dict[str, Any]]:
target_database = normalize_database_name(database)
# 表级摘要统一从 information_schema.tables 读取,便于在浏览器里快速切表。
query = """
SELECT
TABLE_NAME AS tableName,
ENGINE AS engine,
TABLE_ROWS AS rowEstimate,
DATA_LENGTH AS dataBytes,
INDEX_LENGTH AS indexBytes,
AUTO_INCREMENT AS autoIncrementValue,
CREATE_TIME AS createTime,
UPDATE_TIME AS updateTime,
TABLE_COLLATION AS tableCollation
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = %s
ORDER BY COALESCE(DATA_LENGTH, 0) + COALESCE(INDEX_LENGTH, 0) DESC, TABLE_NAME ASC
"""
with connection.cursor() as cursor:
# 同一库内表数量通常有限,这里直接全量读取。
cursor.execute(query, (target_database,))
rows = cursor.fetchall() or []
return [
{
"name": str(item.get("tableName") or "").strip(),
"engine": str(item.get("engine") or "").strip(),
"rowEstimate": int(item.get("rowEstimate") or 0),
"dataSizeMb": round(float(item.get("dataBytes") or 0) / 1024 / 1024, 2),
"indexSizeMb": round(float(item.get("indexBytes") or 0) / 1024 / 1024, 2),
"totalSizeMb": round(
(float(item.get("dataBytes") or 0) + float(item.get("indexBytes") or 0)) / 1024 / 1024,
2,
),
"autoIncrement": item.get("autoIncrementValue"),
"createTime": normalize_mysql_value(item.get("createTime")),
"updateTime": normalize_mysql_value(item.get("updateTime")),
"collation": str(item.get("tableCollation") or "").strip(),
}
for item in rows
]
def list_table_columns(connection: Any, database: str, table: str) -> list[dict[str, Any]]:
target_database = normalize_database_name(database)
target_table = normalize_table_name(table)
# 列元数据以 ordinal_position 排序,保证浏览器和真实表结构一致。
query = """
SELECT
COLUMN_NAME AS columnName,
DATA_TYPE AS dataType,
COLUMN_TYPE AS columnType,
IS_NULLABLE AS isNullable,
COLUMN_DEFAULT AS columnDefault,
COLUMN_KEY AS columnKey,
EXTRA AS extraValue,
COLUMN_COMMENT AS columnComment,
ORDINAL_POSITION AS ordinalPosition
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s
ORDER BY ORDINAL_POSITION ASC
"""
with connection.cursor() as cursor:
# 表结构操作都依赖列元数据,这里统一一次性拿齐。
cursor.execute(query, (target_database, target_table))
rows = cursor.fetchall() or []
return [
{
"name": str(item.get("columnName") or "").strip(),
"dataType": str(item.get("dataType") or "").strip(),
"columnType": str(item.get("columnType") or "").strip(),
"nullable": str(item.get("isNullable") or "").strip().upper() == "YES",
"default": normalize_mysql_value(item.get("columnDefault")),
"key": str(item.get("columnKey") or "").strip(),
"extra": str(item.get("extraValue") or "").strip(),
"comment": str(item.get("columnComment") or "").strip(),
"ordinalPosition": int(item.get("ordinalPosition") or 0),
}
for item in rows
]
def list_table_indexes(connection: Any, database: str, table: str) -> list[dict[str, Any]]:
target_database = normalize_database_name(database)
target_table = normalize_table_name(table)
# 索引明细先按序号取出,再在 Python 侧聚合成前端更容易消费的结构。
query = """
SELECT
INDEX_NAME AS indexName,
COLUMN_NAME AS columnName,
NON_UNIQUE AS nonUnique,
INDEX_TYPE AS indexType,
SEQ_IN_INDEX AS seqInIndex,
SUB_PART AS subPart
FROM information_schema.STATISTICS
WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s
ORDER BY INDEX_NAME ASC, SEQ_IN_INDEX ASC
"""
with connection.cursor() as cursor:
# information_schema.STATISTICS 能完整覆盖 primary / unique / 普通索引。
cursor.execute(query, (target_database, target_table))
rows = cursor.fetchall() or []
grouped: dict[str, dict[str, Any]] = {}
for item in rows:
index_name = str(item.get("indexName") or "").strip()
target = grouped.setdefault(
index_name,
{
"name": index_name,
"primary": index_name == "PRIMARY",
"unique": int(item.get("nonUnique") or 0) == 0,
"type": str(item.get("indexType") or "").strip(),
"columns": [],
},
)
target["columns"].append(
{
"name": str(item.get("columnName") or "").strip(),
"subPart": int(item.get("subPart") or 0) or None,
"position": int(item.get("seqInIndex") or 0),
}
)
# 输出时保证 primary 优先,再按名字排序。
return sorted(
grouped.values(),
key=lambda item: (0 if item["primary"] else 1, item["name"].lower()),
)
def get_preferred_identity_columns(columns: list[dict[str, Any]], indexes: list[dict[str, Any]]) -> dict[str, Any]:
# 行级更新优先使用主键;没有主键时退到第一个唯一索引;再不行才退到全列比对。
primary = next((item for item in indexes if item.get("primary")), None)
if primary:
return {"mode": "primary", "columns": [column["name"] for column in primary["columns"]]}
unique = next((item for item in indexes if item.get("unique")), None)
if unique:
return {"mode": "unique", "columns": [column["name"] for column in unique["columns"]]}
return {"mode": "all", "columns": [column["name"] for column in columns]}
def build_row_identity(row: dict[str, Any], identity_spec: dict[str, Any]) -> dict[str, Any]:
# identity 只保留用于匹配的字段,避免前端把整行作为 where 条件带回。
columns = [str(item or "").strip() for item in identity_spec.get("columns") or [] if str(item or "").strip()]
return {
"mode": str(identity_spec.get("mode") or "").strip(),
"columns": columns,
"values": {column: normalize_mysql_value(row.get(column)) for column in columns},
}
def table_exists(connection: Any, database: str, table: str) -> bool:
target_database = normalize_database_name(database)
target_table = normalize_table_name(table)
# DDL 和浏览接口都需要一个轻量存在性校验,避免后面拼 SQL 才报错。
with connection.cursor() as cursor:
cursor.execute(
"""
SELECT 1
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s
LIMIT 1
""",
(target_database, target_table),
)
return cursor.fetchone() is not None
def database_exists(connection: Any, database: str) -> bool:
target_database = normalize_database_name(database)
# 库级存在性校验单独抽出来,浏览器切库时可以复用。
with connection.cursor() as cursor:
cursor.execute(
"""
SELECT 1
FROM information_schema.SCHEMATA
WHERE SCHEMA_NAME = %s
LIMIT 1
""",
(target_database,),
)
return cursor.fetchone() is not None
def normalize_sql_text(sql: str) -> str:
# SQL 工作台当前只允许单条语句,避免一键提交多段脚本。
text = str(sql or "").strip()
if not text:
raise ValueError("sql is required")
# 只允许尾部分号,正文里再出现分号就拒绝执行。
stripped = text.rstrip().rstrip(";").rstrip()
if ";" in stripped:
raise ValueError("only single sql statement is allowed")
return stripped
def sql_statement_kind(sql: str) -> str:
# 读取首个关键字即可分辨查询类还是变更类语句。
parts = normalize_sql_text(sql).split(None, 1)
return parts[0].strip().lower() if parts else ""

View File

@ -0,0 +1,249 @@
from __future__ import annotations
import threading
import time
from typing import Any
from core.cache import TTLCache
from core.config import MYSQL_CACHE_TTL_SECONDS, MYSQL_HOST, MYSQL_PORT, utc_now
from .mysql_common import build_mysql_connection, list_database_summaries, mysql_error_payload, normalize_mysql_value
# MySQL 概览卡片和基础设施页都复用同一份汇总数据。
MYSQL_OVERVIEW_CACHE = TTLCache[dict[str, Any]](MYSQL_CACHE_TTL_SECONDS)
# QPS / TPS 需要用两次采样求差值,这里单独保存最近一次原始计数器。
MYSQL_RATE_LOCK = threading.Lock()
MYSQL_RATE_SNAPSHOT: dict[str, Any] | None = None
def clear_mysql_metrics_cache() -> None:
# 数据写入、DDL 之后清理缓存,避免概览继续显示旧指标。
MYSQL_OVERVIEW_CACHE.clear()
def coerce_int(mapping: dict[str, Any], key: str) -> int:
try:
# MySQL status / variables 大多返回字符串,这里统一转整数。
return int(mapping.get(key) or 0)
except Exception: # noqa: BLE001
return 0
def coerce_bool(mapping: dict[str, Any], key: str) -> bool:
value = str(mapping.get(key) or "").strip().lower()
return value in {"1", "on", "yes", "true"}
def collect_global_status(connection: Any) -> dict[str, Any]:
names = (
"Threads_connected",
"Threads_running",
"Questions",
"Com_commit",
"Com_rollback",
"Uptime",
)
# 这里选最小必要状态量,避免每次都拉全量 GLOBAL STATUS。
with connection.cursor() as cursor:
cursor.execute(
"""
SHOW GLOBAL STATUS
WHERE Variable_name IN (%s, %s, %s, %s, %s, %s)
""",
names,
)
rows = cursor.fetchall() or []
return {str(item.get("Variable_name") or ""): item.get("Value") for item in rows}
def collect_global_variables(connection: Any) -> dict[str, Any]:
names = (
"version",
"version_comment",
"server_uuid",
"server_id",
"max_connections",
"read_only",
"super_read_only",
)
# 版本、只读状态和最大连接数都在 GLOBAL VARIABLES 里。
with connection.cursor() as cursor:
cursor.execute(
"""
SHOW GLOBAL VARIABLES
WHERE Variable_name IN (%s, %s, %s, %s, %s, %s, %s)
""",
names,
)
rows = cursor.fetchall() or []
return {str(item.get("Variable_name") or ""): item.get("Value") for item in rows}
def collect_replication_payload(connection: Any, variables: dict[str, Any]) -> dict[str, Any]:
# 先把只读状态打平,这一层即使复制信息拿不到也能稳定展示。
payload = {
"configured": False,
"role": "unknown",
"readOnly": coerce_bool(variables, "read_only"),
"superReadOnly": coerce_bool(variables, "super_read_only"),
"sourceHost": "",
"secondsBehindSource": None,
"ioRunning": False,
"sqlRunning": False,
"state": "",
"error": "",
}
# 云 MySQL 版本不同,复制状态命令可能叫 SHOW SLAVE STATUS 或 SHOW REPLICA STATUS。
commands = ["SHOW REPLICA STATUS", "SHOW SLAVE STATUS"]
for command in commands:
try:
with connection.cursor() as cursor:
cursor.execute(command)
row = cursor.fetchone()
except Exception: # noqa: BLE001
# 云库版本和权限口径不一致时,这两条命令经常会报语法或权限错误。
# 这里不把它直接当成实例异常,避免主节点或单节点 MySQL 被误标成故障。
continue
# 没有行时说明当前节点没有暴露复制状态,继续尝试下一条命令。
if not row:
continue
payload["configured"] = True
payload["sourceHost"] = str(
row.get("Source_Host")
or row.get("Master_Host")
or ""
).strip()
payload["secondsBehindSource"] = (
int(row.get("Seconds_Behind_Source") or row.get("Seconds_Behind_Master") or 0)
if row.get("Seconds_Behind_Source") is not None or row.get("Seconds_Behind_Master") is not None
else None
)
payload["ioRunning"] = str(
row.get("Replica_IO_Running")
or row.get("Slave_IO_Running")
or ""
).strip().lower() == "yes"
payload["sqlRunning"] = str(
row.get("Replica_SQL_Running")
or row.get("Slave_SQL_Running")
or ""
).strip().lower() == "yes"
payload["state"] = str(
row.get("Replica_SQL_Running_State")
or row.get("Slave_SQL_Running_State")
or ""
).strip()
payload["error"] = str(
row.get("Last_SQL_Error")
or row.get("Last_IO_Error")
or ""
).strip()
payload["role"] = "replica"
return payload
# 没拿到复制详情时,退化成基于只读状态的粗略判断。
payload["role"] = "replica" if payload["readOnly"] or payload["superReadOnly"] else "primary"
return payload
def collect_throughput_payload(status: dict[str, Any]) -> dict[str, Any]:
current = {
"capturedAt": time.time(),
"questions": coerce_int(status, "Questions"),
"commits": coerce_int(status, "Com_commit"),
"rollbacks": coerce_int(status, "Com_rollback"),
}
# 首次采样时没有可对比的上一帧,这里返回 0 但仍记住快照。
with MYSQL_RATE_LOCK:
global MYSQL_RATE_SNAPSHOT
previous = MYSQL_RATE_SNAPSHOT
MYSQL_RATE_SNAPSHOT = current
if previous is None:
return {
"qps": 0.0,
"tps": 0.0,
"sampleSeconds": 0.0,
"questionsDelta": 0,
"transactionsDelta": 0,
}
elapsed = max(current["capturedAt"] - float(previous.get("capturedAt") or 0.0), 0.001)
questions_delta = max(current["questions"] - int(previous.get("questions") or 0), 0)
transactions_delta = max(
(current["commits"] + current["rollbacks"]) - (int(previous.get("commits") or 0) + int(previous.get("rollbacks") or 0)),
0,
)
return {
"qps": round(questions_delta / elapsed, 2),
"tps": round(transactions_delta / elapsed, 2),
"sampleSeconds": round(elapsed, 2),
"questionsDelta": questions_delta,
"transactionsDelta": transactions_delta,
}
def build_mysql_overview_payload() -> dict[str, Any]:
try:
# 指标和元数据统一走一条连接,减少直连云 MySQL 的并发压力。
with build_mysql_connection(autocommit=True) as connection:
with connection.cursor() as cursor:
# ping 能尽早暴露账号、网络和权限问题。
cursor.execute("SELECT 1 AS ok")
cursor.fetchone()
variables = collect_global_variables(connection)
status = collect_global_status(connection)
replication = collect_replication_payload(connection, variables)
throughput = collect_throughput_payload(status)
databases = list_database_summaries(connection)
except Exception as exc: # noqa: BLE001
return mysql_error_payload(str(exc))
# 库级汇总同时作为基础设施概览卡片的数据源。
business_databases = [item for item in databases if not item.get("system")]
summary = {
"databaseTotal": len(business_databases),
"tableTotal": sum(int(item.get("tableTotal") or 0) for item in business_databases),
"dataSizeMb": round(sum(float(item.get("dataSizeMb") or 0.0) for item in business_databases), 2),
"indexSizeMb": round(sum(float(item.get("indexSizeMb") or 0.0) for item in business_databases), 2),
"totalSizeMb": round(sum(float(item.get("totalSizeMb") or 0.0) for item in business_databases), 2),
}
# 当前连接数、最大连接数和版本信息都单独拉平,前端展示更直接。
return {
"ok": True,
"updatedAt": utc_now(),
"info": {
"host": f"{MYSQL_HOST}:{MYSQL_PORT}",
"version": str(variables.get("version") or "").strip(),
"versionComment": str(variables.get("version_comment") or "").strip(),
"serverUuid": str(variables.get("server_uuid") or "").strip(),
"serverId": str(variables.get("server_id") or "").strip(),
"uptimeSeconds": coerce_int(status, "Uptime"),
},
"connections": {
"current": coerce_int(status, "Threads_connected"),
"running": coerce_int(status, "Threads_running"),
"max": coerce_int(variables, "max_connections"),
},
"throughput": throughput,
"replication": replication,
"summary": summary,
"databases": databases,
"error": "",
}
def get_mysql_overview_payload() -> dict[str, Any]:
# 概览卡片和表浏览 sidebar 共用缓存,避免每次点 tab 都重查 information_schema。
return MYSQL_OVERVIEW_CACHE.get_or_build(build_mysql_overview_payload)

View File

@ -8,9 +8,9 @@ import urllib.parse
import urllib.request
from typing import Any
from .cache import TTLCache
from core.cache import TTLCache
from .config_validation import validate_config_content
from .config import (
from core.config import (
NACOS_BASE_URL,
NACOS_CACHE_TTL_SECONDS,
NACOS_DISCOVERY_CLUSTER_NAME,

View File

@ -6,7 +6,7 @@ import hashlib
import json
from typing import Any
from .config import RUNTIME_BASE_DIR, load_hosts, utc_now
from core.config import RUNTIME_BASE_DIR, load_hosts, utc_now
from .ssh_remote import read_remote_text as ssh_read_remote_text
from .ssh_remote import write_remote_text as ssh_write_remote_text

View File

@ -5,7 +5,7 @@ import shlex
from typing import Any
from .compose_templates import container_name
from .config import SERVICE_LOG_EXPORT_MAX_LINES, SERVICE_LOG_PREVIEW_LINES, flatten_services, load_hosts, utc_now
from core.config import SERVICE_LOG_EXPORT_MAX_LINES, SERVICE_LOG_PREVIEW_LINES, flatten_services, load_hosts, utc_now
from .ssh_remote import run_host_script

View File

@ -7,7 +7,7 @@ import time
from typing import Any
from .compose_templates import sync_remote_host_compose_template
from .config import (
from core.config import (
ROLLING_RESTART_POLL_SECONDS,
ROLLING_RESTART_STABILIZE_SECONDS,
ROLLING_RESTART_TIMEOUT_SECONDS,

View File

@ -21,7 +21,7 @@ from .compose_templates import (
sync_remote_host_compose_template,
update_local_service_image,
)
from .config import (
from core.config import (
ADMIN_FRONTEND_BUILD_SCRIPT,
ADMIN_FRONTEND_COMPOSE_FILE,
ADMIN_FRONTEND_COMPOSE_SERVICE,

View File

@ -13,7 +13,7 @@ import re
import uuid
from typing import Any
from .config import ROOT, utc_now
from core.config import ROOT, utc_now
from .service_release import deploy_updated_services_payload

View File

@ -9,7 +9,7 @@ from pathlib import Path
import subprocess
from typing import Any
from .config import (
from core.config import (
SSH_CONNECT_TIMEOUT_SECONDS,
SSH_KNOWN_HOSTS_PATH,
SSH_PORT,

View File

@ -5,7 +5,7 @@ import json
import time
from typing import Any
from .config import DEPLOY_REGION, TAT_METRIC_TIMEOUT_SECONDS, TENCENT_SECRET_ID, TENCENT_SECRET_KEY, utc_now
from core.config import DEPLOY_REGION, TAT_METRIC_TIMEOUT_SECONDS, TENCENT_SECRET_ID, TENCENT_SECRET_KEY, utc_now
def decode_task_output(output: str) -> str:

View File

@ -12,7 +12,7 @@ ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from monitor.compose_templates import managed_compose_host_names, save_local_host_compose_template, sync_remote_compose_templates
from monitors.yumi.compose_templates import managed_compose_host_names, save_local_host_compose_template, sync_remote_compose_templates
def parse_args() -> argparse.Namespace:

View File

@ -1,4 +1,5 @@
tencentcloud-sdk-python>=3.0.0
pymongo>=4.8.0
PyMySQL>=1.1.1
PyYAML>=6.0.2
javaproperties>=0.8.2

View File

@ -2,7 +2,7 @@
from __future__ import annotations
from gateway.http_app import main
from entry.http_app import main
if __name__ == "__main__":

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -10,7 +10,7 @@ os.environ.setdefault("APP_BIND", os.environ.get("YUMI_INTERNAL_HOST", "127.0.0.
os.environ.setdefault("APP_PORT", os.environ.get("YUMI_INTERNAL_PORT", "2027"))
os.environ["APP_BASE_PATH"] = "/yumi"
from monitor.http_app import main
from monitors.yumi.http_app import main
if __name__ == "__main__":