feat: add tat host metrics and systemd runtime
This commit is contained in:
ZuoZuo
parent
1d99f565eb
commit
53e0cf8a38
11
.env.example
Normal file
11
.env.example
Normal file
@ -0,0 +1,11 @@
|
||||
APP_BIND=0.0.0.0
|
||||
APP_PORT=2026
|
||||
PROBE_TIMEOUT_SECONDS=3
|
||||
PROBE_CACHE_TTL_SECONDS=5
|
||||
PROBE_MAX_WORKERS=12
|
||||
HOST_METRIC_CACHE_TTL_SECONDS=25
|
||||
TAT_METRIC_TIMEOUT_SECONDS=30
|
||||
TAT_CPU_SAMPLE_SECONDS=1
|
||||
DEPLOY_REGION=me-saudi-arabia
|
||||
TENCENT_SECRET_ID=
|
||||
TENCENT_SECRET_KEY=
|
||||
2
.gitignore
vendored
2
.gitignore
vendored
@ -2,3 +2,5 @@ run/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.DS_Store
|
||||
.env
|
||||
.venv/
|
||||
|
||||
38
AGENTS.md
Normal file
38
AGENTS.md
Normal file
@ -0,0 +1,38 @@
|
||||
# hy-app-monitor Agent Guide
|
||||
|
||||
## 项目目标
|
||||
|
||||
- 这是部署在 `deploy` 机器上的轻量监控面板。
|
||||
- 页面端口固定走 `2026`，不要再改回 `6666` 一类浏览器 unsafe port。
|
||||
- 服务健康检查直接走各业务机器的 HTTP 接口。
|
||||
- 主机 `CPU / 内存 / 磁盘 / 进程数` 统一走 `TAT` 远程采集，不额外安装 agent。
|
||||
|
||||
## 配置约定
|
||||
|
||||
- 项目根目录 `.env` 是运行时唯一配置源。
|
||||
- `.env` 必须保留在本地和远端项目目录里，但不能提交到 Git。
|
||||
- `.env.example` 只作为模板，不能写真实密钥。
|
||||
- `TENCENT_SECRET_ID` / `TENCENT_SECRET_KEY` / `DEPLOY_REGION` 从 `.env` 读取。
|
||||
- `config/targets.json` 里的 `instanceId` 不能删，`TAT` 主机采集依赖它。
|
||||
|
||||
## 运行约定
|
||||
|
||||
- 正式运行优先使用 `systemd`。
|
||||
- `scripts/install_systemd.sh` 负责安装 unit 并启用开机自启。
|
||||
- `scripts/start.sh` / `stop.sh` / `restart.sh` / `status.sh` 在检测到 unit 后应直接转调 `systemctl`。
|
||||
- `scripts/run_foreground.sh` 只给 `systemd` 用，不要把它改成后台模式。
|
||||
|
||||
## 部署约定
|
||||
|
||||
- 远端路径固定为 `/opt/hy-app-monitor`。
|
||||
- 部署入口是 `ops/deploy_via_tat.py`。
|
||||
- 该脚本需要把本地项目根目录 `.env` 同步到远端项目根目录 `.env`。
|
||||
- 远端 Python 运行时优先使用项目内 `.venv`。
|
||||
- `requirements-ops.txt` 里的依赖既服务于部署脚本，也服务于运行时 `TAT` 指标采集。
|
||||
|
||||
## 修改原则
|
||||
|
||||
- 不要把 `TAT` 采集替换成 agent 方案，除非用户明确要求。
|
||||
- 不要把密钥写进版本库。
|
||||
- 不要删除页面里的主机指标区块。
|
||||
- Python 文件保持高注释密度，优先写清楚每个步骤在做什么。
|
||||
111
README.md
111
README.md
@ -1,93 +1,94 @@
|
||||
# hy-app-monitor
|
||||
|
||||
一个给 `deploy` 机器使用的轻量监控页。
|
||||
部署在 `deploy` 机器上的轻量监控面板。
|
||||
|
||||
- 前端：静态 HTML + Vue 3 CDN
|
||||
- 后端：Python 3 标准库 HTTP 服务
|
||||
- 默认端口：`2026`
|
||||
- 默认监听：`0.0.0.0`
|
||||
|
||||
## 功能
|
||||
|
||||
- 从 `deploy` 机器内部探测内网 Java / Golang 服务健康状态
|
||||
- 对外提供监控页和 JSON API
|
||||
- 提供启动 / 停止 / 重启 / 状态脚本
|
||||
- 提供通过 TAT 放开 2026 端口和远端拉取部署的脚本
|
||||
- 页面端口：`2026`
|
||||
- 页面入口：`http://43.164.75.199:2026/`
|
||||
- 服务健康：直接探测各业务服务健康接口
|
||||
- 主机资源：通过 `TAT` 批量采集 `CPU / 内存 / 磁盘 / 进程数`
|
||||
|
||||
## 目录
|
||||
|
||||
- `server.py`：后端入口
|
||||
- `config/targets.json`：监控目标配置
|
||||
- `config/targets.json`：监控目标和实例 ID
|
||||
- `static/`：前端页面
|
||||
- `scripts/`：启停脚本
|
||||
- `ops/`：TAT 相关脚本
|
||||
- `scripts/`：启停、自启安装、前台运行脚本
|
||||
- `systemd/`：systemd unit
|
||||
- `ops/`：TAT 放行和远端部署脚本
|
||||
- `.env.example`：运行配置模板
|
||||
|
||||
## 本地启动
|
||||
## 运行配置
|
||||
|
||||
项目根目录 `.env` 是唯一运行配置源，不入库。
|
||||
|
||||
最少需要：
|
||||
|
||||
```bash
|
||||
cd /opt/hy-app-monitor
|
||||
bash scripts/start.sh
|
||||
APP_BIND=0.0.0.0
|
||||
APP_PORT=2026
|
||||
DEPLOY_REGION=me-saudi-arabia
|
||||
TENCENT_SECRET_ID=***
|
||||
TENCENT_SECRET_KEY=***
|
||||
```
|
||||
|
||||
默认访问：
|
||||
可以直接复制模板：
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
## 本地运行
|
||||
|
||||
```bash
|
||||
bash scripts/start.sh
|
||||
bash scripts/status.sh
|
||||
```
|
||||
|
||||
访问：
|
||||
|
||||
```bash
|
||||
http://127.0.0.1:2026/
|
||||
```
|
||||
|
||||
## 脚本
|
||||
## systemd 开机自启
|
||||
|
||||
在远端项目目录执行：
|
||||
|
||||
```bash
|
||||
bash scripts/install_systemd.sh
|
||||
```
|
||||
|
||||
常用命令：
|
||||
|
||||
```bash
|
||||
bash scripts/start.sh
|
||||
bash scripts/stop.sh
|
||||
bash scripts/restart.sh
|
||||
bash scripts/status.sh
|
||||
bash scripts/uninstall_systemd.sh
|
||||
```
|
||||
|
||||
## 环境变量
|
||||
|
||||
- `APP_BIND`：默认 `0.0.0.0`
|
||||
- `APP_PORT`：默认 `2026`
|
||||
- `TARGETS_FILE`：默认 `config/targets.json`
|
||||
- `PROBE_TIMEOUT_SECONDS`：默认 `3`
|
||||
- `PROBE_CACHE_TTL_SECONDS`：默认 `5`
|
||||
- `PROBE_MAX_WORKERS`：默认 `12`
|
||||
|
||||
## 用 TAT 放开 2026
|
||||
|
||||
先确保本机有腾讯云密钥环境变量：
|
||||
|
||||
```bash
|
||||
export TENCENT_SECRET_ID=...
|
||||
export TENCENT_SECRET_KEY=...
|
||||
```
|
||||
|
||||
执行：
|
||||
## 通过 TAT 放行端口
|
||||
|
||||
```bash
|
||||
python3 ops/open_port_via_tat.py
|
||||
```
|
||||
|
||||
默认会针对 deploy 机器 `ins-ntmpcd3a` 在 `me-saudi-arabia` 区域：
|
||||
默认会：
|
||||
|
||||
- 通过 TAT 在机器内尝试放开系统防火墙
|
||||
- 通过 TAT 调腾讯云 API 放开安全组 TCP `2026`
|
||||
1. 解析 `deploy` 机器安全组
|
||||
2. 通过 `TAT` 放开机器内防火墙
|
||||
3. 通过腾讯云 API 放开安全组 TCP `2026`
|
||||
|
||||
## 用 TAT 拉取并启动
|
||||
## 通过 TAT 远端部署
|
||||
|
||||
```bash
|
||||
python3 ops/deploy_via_tat.py
|
||||
```
|
||||
|
||||
默认远端目录：
|
||||
|
||||
```bash
|
||||
/opt/hy-app-monitor
|
||||
```
|
||||
|
||||
默认会：
|
||||
|
||||
1. clone / pull 仓库
|
||||
2. 给脚本加执行权限
|
||||
3. 重启服务
|
||||
4. 检查 2026 端口监听
|
||||
1. 拉取或更新 `/opt/hy-app-monitor`
|
||||
2. 把本地项目 `.env` 同步到远端项目 `.env`
|
||||
3. 创建远端 `.venv`
|
||||
4. 安装 `requirements-ops.txt`
|
||||
5. 安装并启用 `systemd`
|
||||
6. 重启服务并检查 `2026` 监听
|
||||
|
||||
@ -1,154 +1,160 @@
|
||||
{
|
||||
"hosts": [
|
||||
{
|
||||
"name": "gateway-1",
|
||||
"ip": "10.2.1.8",
|
||||
"group": "gateway",
|
||||
"services": [
|
||||
"hosts": [
|
||||
{
|
||||
"name": "gateway",
|
||||
"kind": "java",
|
||||
"port": 8080,
|
||||
"path": "/health"
|
||||
"name": "gateway-1",
|
||||
"ip": "10.2.1.8",
|
||||
"group": "gateway",
|
||||
"instanceId": "ins-aeyaoj40",
|
||||
"services": [
|
||||
{
|
||||
"name": "gateway",
|
||||
"kind": "java",
|
||||
"port": 8080,
|
||||
"path": "/health"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "gateway-2",
|
||||
"ip": "10.2.2.17",
|
||||
"group": "gateway",
|
||||
"instanceId": "ins-irvi5b7u",
|
||||
"services": [
|
||||
{
|
||||
"name": "gateway",
|
||||
"kind": "java",
|
||||
"port": 8080,
|
||||
"path": "/health"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "app-1",
|
||||
"ip": "10.2.11.3",
|
||||
"group": "app",
|
||||
"instanceId": "ins-n566mmeg",
|
||||
"services": [
|
||||
{
|
||||
"name": "auth",
|
||||
"kind": "java",
|
||||
"port": 1000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "other",
|
||||
"kind": "java",
|
||||
"port": 2400,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "live",
|
||||
"kind": "java",
|
||||
"port": 2500,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "console",
|
||||
"kind": "java",
|
||||
"port": 2700,
|
||||
"path": "/console/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "external",
|
||||
"kind": "java",
|
||||
"port": 3000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "golang",
|
||||
"kind": "golang",
|
||||
"port": 2900,
|
||||
"path": "/health"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "app-2",
|
||||
"ip": "10.2.12.6",
|
||||
"group": "app",
|
||||
"instanceId": "ins-kauk17la",
|
||||
"services": [
|
||||
{
|
||||
"name": "auth",
|
||||
"kind": "java",
|
||||
"port": 1000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "other",
|
||||
"kind": "java",
|
||||
"port": 2400,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "live",
|
||||
"kind": "java",
|
||||
"port": 2500,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "console",
|
||||
"kind": "java",
|
||||
"port": 2700,
|
||||
"path": "/console/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "external",
|
||||
"kind": "java",
|
||||
"port": 3000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "golang",
|
||||
"kind": "golang",
|
||||
"port": 2900,
|
||||
"path": "/health"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "pay-1",
|
||||
"ip": "10.2.11.14",
|
||||
"group": "pay",
|
||||
"instanceId": "ins-ceqfcxd2",
|
||||
"services": [
|
||||
{
|
||||
"name": "wallet",
|
||||
"kind": "java",
|
||||
"port": 2000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "order",
|
||||
"kind": "java",
|
||||
"port": 2600,
|
||||
"path": "/actuator/health"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "pay-2",
|
||||
"ip": "10.2.12.5",
|
||||
"group": "pay",
|
||||
"instanceId": "ins-awhb74q6",
|
||||
"services": [
|
||||
{
|
||||
"name": "wallet",
|
||||
"kind": "java",
|
||||
"port": 2000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "order",
|
||||
"kind": "java",
|
||||
"port": 2600,
|
||||
"path": "/actuator/health"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "gateway-2",
|
||||
"ip": "10.2.2.17",
|
||||
"group": "gateway",
|
||||
"services": [
|
||||
{
|
||||
"name": "gateway",
|
||||
"kind": "java",
|
||||
"port": 8080,
|
||||
"path": "/health"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "app-1",
|
||||
"ip": "10.2.11.3",
|
||||
"group": "app",
|
||||
"services": [
|
||||
{
|
||||
"name": "auth",
|
||||
"kind": "java",
|
||||
"port": 1000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "other",
|
||||
"kind": "java",
|
||||
"port": 2400,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "live",
|
||||
"kind": "java",
|
||||
"port": 2500,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "console",
|
||||
"kind": "java",
|
||||
"port": 2700,
|
||||
"path": "/console/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "external",
|
||||
"kind": "java",
|
||||
"port": 3000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "golang",
|
||||
"kind": "golang",
|
||||
"port": 2900,
|
||||
"path": "/health"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "app-2",
|
||||
"ip": "10.2.12.6",
|
||||
"group": "app",
|
||||
"services": [
|
||||
{
|
||||
"name": "auth",
|
||||
"kind": "java",
|
||||
"port": 1000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "other",
|
||||
"kind": "java",
|
||||
"port": 2400,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "live",
|
||||
"kind": "java",
|
||||
"port": 2500,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "console",
|
||||
"kind": "java",
|
||||
"port": 2700,
|
||||
"path": "/console/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "external",
|
||||
"kind": "java",
|
||||
"port": 3000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "golang",
|
||||
"kind": "golang",
|
||||
"port": 2900,
|
||||
"path": "/health"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "pay-1",
|
||||
"ip": "10.2.11.14",
|
||||
"group": "pay",
|
||||
"services": [
|
||||
{
|
||||
"name": "wallet",
|
||||
"kind": "java",
|
||||
"port": 2000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "order",
|
||||
"kind": "java",
|
||||
"port": 2600,
|
||||
"path": "/actuator/health"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "pay-2",
|
||||
"ip": "10.2.12.5",
|
||||
"group": "pay",
|
||||
"services": [
|
||||
{
|
||||
"name": "wallet",
|
||||
"kind": "java",
|
||||
"port": 2000,
|
||||
"path": "/actuator/health"
|
||||
},
|
||||
{
|
||||
"name": "order",
|
||||
"kind": "java",
|
||||
"port": 2600,
|
||||
"path": "/actuator/health"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
]
|
||||
}
|
||||
|
||||
@ -2,37 +2,107 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# base64 用来安全传输远端脚本和 .env 内容。
|
||||
import base64
|
||||
# json 用来构造腾讯云 SDK 请求。
|
||||
import json
|
||||
# 读取本地 .env 和运行环境变量。
|
||||
import os
|
||||
# TAT 轮询需要 sleep 和超时控制。
|
||||
import time
|
||||
# Path 用于稳定定位项目根目录。
|
||||
from pathlib import Path
|
||||
# 类型提示只用于提高可读性。
|
||||
from typing import Callable, TypeVar
|
||||
|
||||
# 腾讯云凭据对象。
|
||||
from tencentcloud.common import credential
|
||||
# 腾讯云 SDK 统一异常类型。
|
||||
from tencentcloud.common.exception.tencent_cloud_sdk_exception import TencentCloudSDKException
|
||||
# 客户端公共配置。
|
||||
from tencentcloud.common.profile.client_profile import ClientProfile
|
||||
# HTTP endpoint 配置。
|
||||
from tencentcloud.common.profile.http_profile import HttpProfile
|
||||
# TAT 客户端和模型。
|
||||
from tencentcloud.tat.v20201028 import models as tat_models, tat_client
|
||||
|
||||
|
||||
# 项目根目录，后面要从这里读取 .env。
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
# 默认 .env 路径。
|
||||
ENV_FILE = Path(os.environ.get("APP_ENV_FILE", ROOT / ".env")).expanduser()
|
||||
# 默认 deploy 机器实例 ID。
|
||||
DEFAULT_INSTANCE_ID = os.environ.get("DEPLOY_INSTANCE_ID", "ins-ntmpcd3a")
|
||||
# 默认区域。
|
||||
DEFAULT_REGION = os.environ.get("DEPLOY_REGION", "me-saudi-arabia")
|
||||
# 默认仓库地址。
|
||||
DEFAULT_REPO_URL = os.environ.get("MONITOR_REPO_URL", "git@gitea.haiyihy.com:hy/hy-app-monitor.git")
|
||||
# 默认分支。
|
||||
DEFAULT_REPO_REF = os.environ.get("MONITOR_REPO_REF", "main")
|
||||
# 默认远端部署目录。
|
||||
DEFAULT_REMOTE_DIR = os.environ.get("MONITOR_REMOTE_DIR", "/opt/hy-app-monitor")
|
||||
# 默认应用端口。
|
||||
DEFAULT_APP_PORT = os.environ.get("APP_PORT", "2026")
|
||||
# 泛型只用于 invoke 的返回类型提示。
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
def load_env_file(path: Path) -> None:
|
||||
# .env 不存在时允许继续使用外部环境变量。
|
||||
if not path.exists():
|
||||
return
|
||||
|
||||
# 逐行读取 .env。
|
||||
for raw_line in path.read_text(encoding="utf-8").splitlines():
|
||||
# 去掉两端空白。
|
||||
line = raw_line.strip()
|
||||
# 空行直接跳过。
|
||||
if not line:
|
||||
continue
|
||||
# 注释直接跳过。
|
||||
if line.startswith("#"):
|
||||
continue
|
||||
# 没有等号的无效行直接跳过。
|
||||
if "=" not in line:
|
||||
continue
|
||||
|
||||
# 只按第一个等号拆分。
|
||||
key, value = line.split("=", 1)
|
||||
# 清理 key 空白。
|
||||
key = key.strip()
|
||||
# 清理 value 空白。
|
||||
value = value.strip()
|
||||
|
||||
# 去掉成对双引号。
|
||||
if value.startswith('"') and value.endswith('"'):
|
||||
value = value[1:-1]
|
||||
|
||||
# 去掉成对单引号。
|
||||
if value.startswith("'") and value.endswith("'"):
|
||||
value = value[1:-1]
|
||||
|
||||
# 保持外部显式传入变量优先。
|
||||
os.environ.setdefault(key, value)
|
||||
|
||||
|
||||
# 先加载项目根目录的 .env。
|
||||
load_env_file(ENV_FILE)
|
||||
|
||||
|
||||
def env_required(name: str) -> str:
|
||||
# 读取环境变量并去掉两端空白。
|
||||
value = os.environ.get(name, "").strip()
|
||||
# 缺失时直接退出。
|
||||
if not value:
|
||||
raise SystemExit(f"missing env: {name}")
|
||||
# 返回有效值。
|
||||
return value
|
||||
|
||||
|
||||
def build_tat(secret_id: str, secret_key: str, region: str) -> tat_client.TatClient:
|
||||
# 构造腾讯云凭据对象。
|
||||
cred = credential.Credential(secret_id, secret_key)
|
||||
# 返回 TAT 客户端。
|
||||
return tat_client.TatClient(
|
||||
cred,
|
||||
region,
|
||||
@ -41,27 +111,67 @@ def build_tat(secret_id: str, secret_key: str, region: str) -> tat_client.TatCli
|
||||
|
||||
|
||||
def invoke(action: Callable[[], T], label: str, retries: int = 4, delay_seconds: int = 2) -> T:
|
||||
# 保存最后一次异常，便于最终抛出。
|
||||
last_error: Exception | None = None
|
||||
|
||||
# 按给定次数重试。
|
||||
for attempt in range(1, retries + 1):
|
||||
try:
|
||||
# 成功时直接返回。
|
||||
return action()
|
||||
except TencentCloudSDKException as exc:
|
||||
# 保存最后一次异常。
|
||||
last_error = exc
|
||||
# 最后一次不再等待。
|
||||
if attempt == retries:
|
||||
break
|
||||
# 做一个简单线性退避。
|
||||
time.sleep(delay_seconds * attempt)
|
||||
|
||||
# 有异常时直接抛出最后一次异常。
|
||||
if last_error is not None:
|
||||
raise last_error
|
||||
|
||||
# 理论上不会执行到这里，只做兜底。
|
||||
raise RuntimeError(f"{label} failed without error")
|
||||
|
||||
|
||||
def remote_script(repo_url: str, repo_ref: str, remote_dir: str, app_port: str) -> str:
|
||||
def load_runtime_env() -> str:
|
||||
# 如果项目根目录已经有 .env，就直接把整个文件同步到远端。
|
||||
if ENV_FILE.exists():
|
||||
return ENV_FILE.read_text(encoding="utf-8")
|
||||
|
||||
# 没有 .env 时，就从当前环境里兜底构造一个最小版本。
|
||||
lines = [
|
||||
f"APP_BIND={os.environ.get('APP_BIND', '0.0.0.0')}",
|
||||
f"APP_PORT={os.environ.get('APP_PORT', DEFAULT_APP_PORT)}",
|
||||
f"PROBE_TIMEOUT_SECONDS={os.environ.get('PROBE_TIMEOUT_SECONDS', '3')}",
|
||||
f"PROBE_CACHE_TTL_SECONDS={os.environ.get('PROBE_CACHE_TTL_SECONDS', '5')}",
|
||||
f"PROBE_MAX_WORKERS={os.environ.get('PROBE_MAX_WORKERS', '12')}",
|
||||
f"HOST_METRIC_CACHE_TTL_SECONDS={os.environ.get('HOST_METRIC_CACHE_TTL_SECONDS', '25')}",
|
||||
f"TAT_METRIC_TIMEOUT_SECONDS={os.environ.get('TAT_METRIC_TIMEOUT_SECONDS', '30')}",
|
||||
f"TAT_CPU_SAMPLE_SECONDS={os.environ.get('TAT_CPU_SAMPLE_SECONDS', '1')}",
|
||||
f"DEPLOY_REGION={os.environ.get('DEPLOY_REGION', DEFAULT_REGION)}",
|
||||
f"TENCENT_SECRET_ID={env_required('TENCENT_SECRET_ID')}",
|
||||
f"TENCENT_SECRET_KEY={env_required('TENCENT_SECRET_KEY')}",
|
||||
]
|
||||
|
||||
# 返回最终 .env 文本内容。
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def remote_script(repo_url: str, repo_ref: str, remote_dir: str, app_port: str, runtime_env_text: str) -> str:
|
||||
# 把 .env 文本做 base64 编码，远端可以无损写入。
|
||||
env_blob = base64.b64encode(runtime_env_text.encode("utf-8")).decode("ascii")
|
||||
|
||||
# 返回完整远端部署脚本。
|
||||
return f"""set -euo pipefail
|
||||
|
||||
REPO_URL={json.dumps(repo_url)}
|
||||
REPO_REF={json.dumps(repo_ref)}
|
||||
REMOTE_DIR={json.dumps(remote_dir)}
|
||||
APP_PORT={json.dumps(app_port)}
|
||||
ENV_BLOB={json.dumps(env_blob)}
|
||||
|
||||
export GIT_SSH_COMMAND='ssh -o StrictHostKeyChecking=accept-new'
|
||||
|
||||
@ -77,15 +187,32 @@ else
|
||||
fi
|
||||
|
||||
cd "$REMOTE_DIR"
|
||||
|
||||
python3 - <<'PY'
|
||||
import base64
|
||||
from pathlib import Path
|
||||
|
||||
# 把本地同步过来的 .env 内容写入远端项目根目录。
|
||||
content = base64.b64decode({json.dumps(env_blob)}).decode('utf-8')
|
||||
Path('.env').write_text(content, encoding='utf-8')
|
||||
PY
|
||||
|
||||
chmod +x scripts/*.sh
|
||||
APP_PORT="$APP_PORT" bash scripts/restart.sh
|
||||
sleep 1
|
||||
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install --quiet --disable-pip-version-check -r requirements-ops.txt
|
||||
|
||||
bash scripts/install_systemd.sh
|
||||
systemctl restart hy-app-monitor.service
|
||||
systemctl is-active hy-app-monitor.service
|
||||
ss -lntp | grep ":$APP_PORT" || true
|
||||
"""
|
||||
|
||||
|
||||
def run_tat_script(client: tat_client.TatClient, instance_id: str, script: str, timeout_seconds: int = 600) -> str:
|
||||
# 创建执行命令请求。
|
||||
req = tat_models.RunCommandRequest()
|
||||
# 把远端 shell 脚本下发到 deploy 机器执行。
|
||||
req.from_json_string(
|
||||
json.dumps(
|
||||
{
|
||||
@ -98,13 +225,21 @@ def run_tat_script(client: tat_client.TatClient, instance_id: str, script: str,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
# 发起命令执行。
|
||||
resp = json.loads(invoke(lambda: client.RunCommand(req), "RunCommand").to_json_string())
|
||||
# 获取 invocation id。
|
||||
invocation_id = resp["InvocationId"]
|
||||
# 轮询截止时间。
|
||||
deadline = time.time() + timeout_seconds + 120
|
||||
|
||||
# 开始轮询远端执行结果。
|
||||
while time.time() < deadline:
|
||||
# 两次轮询之间等待 3 秒。
|
||||
time.sleep(3)
|
||||
# 构造查询请求。
|
||||
query = tat_models.DescribeInvocationTasksRequest()
|
||||
# 按 invocation id 过滤本次任务。
|
||||
query.from_json_string(
|
||||
json.dumps(
|
||||
{
|
||||
@ -114,34 +249,63 @@ def run_tat_script(client: tat_client.TatClient, instance_id: str, script: str,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
# 查询执行结果。
|
||||
data = json.loads(invoke(lambda: client.DescribeInvocationTasks(query), "DescribeInvocationTasks").to_json_string())
|
||||
# 拿到任务列表。
|
||||
tasks = data.get("InvocationTaskSet") or []
|
||||
|
||||
# 任务还没生成时继续等。
|
||||
if not tasks:
|
||||
continue
|
||||
|
||||
# 这里只打单实例，所以取第一条即可。
|
||||
task = tasks[0]
|
||||
# 读取当前任务状态。
|
||||
status = task["TaskStatus"]
|
||||
|
||||
# 进入终态时退出轮询。
|
||||
if status in {"SUCCESS", "FAILED", "TIMEOUT", "CANCELLED", "START_FAILED"}:
|
||||
# 解码标准输出。
|
||||
output = base64.b64decode(task.get("TaskResult", {}).get("Output", "")).decode("utf-8", errors="replace")
|
||||
# 非成功状态直接退出。
|
||||
if status != "SUCCESS":
|
||||
raise SystemExit(output or f"TAT task failed: {status}")
|
||||
# 成功时返回远端输出。
|
||||
return output
|
||||
|
||||
# 轮询超时则退出。
|
||||
raise SystemExit("TAT task timed out")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
# 读取腾讯云密钥。
|
||||
secret_id = env_required("TENCENT_SECRET_ID")
|
||||
# 读取腾讯云密钥。
|
||||
secret_key = env_required("TENCENT_SECRET_KEY")
|
||||
# 读取 deploy 实例 ID。
|
||||
instance_id = os.environ.get("DEPLOY_INSTANCE_ID", DEFAULT_INSTANCE_ID)
|
||||
# 读取区域。
|
||||
region = os.environ.get("DEPLOY_REGION", DEFAULT_REGION)
|
||||
# 读取仓库地址。
|
||||
repo_url = os.environ.get("MONITOR_REPO_URL", DEFAULT_REPO_URL)
|
||||
# 读取仓库分支。
|
||||
repo_ref = os.environ.get("MONITOR_REPO_REF", DEFAULT_REPO_REF)
|
||||
# 读取远端部署目录。
|
||||
remote_dir = os.environ.get("MONITOR_REMOTE_DIR", DEFAULT_REMOTE_DIR)
|
||||
# 读取服务端口。
|
||||
app_port = os.environ.get("APP_PORT", DEFAULT_APP_PORT)
|
||||
# 读取要同步到远端的 .env 内容。
|
||||
runtime_env_text = load_runtime_env()
|
||||
|
||||
# 初始化 TAT 客户端。
|
||||
tat = build_tat(secret_id, secret_key, region)
|
||||
output = run_tat_script(tat, instance_id, remote_script(repo_url, repo_ref, remote_dir, app_port))
|
||||
# 远端执行部署脚本。
|
||||
output = run_tat_script(tat, instance_id, remote_script(repo_url, repo_ref, remote_dir, app_port, runtime_env_text))
|
||||
# 打印远端输出，便于观察部署结果。
|
||||
print(output.strip())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 作为脚本执行时进入主流程。
|
||||
main()
|
||||
|
||||
@ -2,38 +2,105 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# base64 用来把远端脚本和 JSON 安全塞进 TAT 请求。
|
||||
import base64
|
||||
# json 用来构造腾讯云 SDK 请求和解析响应。
|
||||
import json
|
||||
# 环境变量读取来自项目根目录的 .env。
|
||||
import os
|
||||
# 轮询 TAT 执行结果时需要 sleep 和超时控制。
|
||||
import time
|
||||
# Path 用于稳定定位项目根目录和 .env 文件。
|
||||
from pathlib import Path
|
||||
# 类型提示只用于提升可读性。
|
||||
from typing import Callable, TypeVar
|
||||
|
||||
# 腾讯云凭据对象。
|
||||
from tencentcloud.common import credential
|
||||
# 腾讯云 SDK 统一异常类型。
|
||||
from tencentcloud.common.exception.tencent_cloud_sdk_exception import TencentCloudSDKException
|
||||
# 客户端公共配置。
|
||||
from tencentcloud.common.profile.client_profile import ClientProfile
|
||||
# HTTP endpoint 配置。
|
||||
from tencentcloud.common.profile.http_profile import HttpProfile
|
||||
# CVM 用来查实例绑定的安全组。
|
||||
from tencentcloud.cvm.v20170312 import cvm_client, models as cvm_models
|
||||
# TAT 用来远程执行放行脚本。
|
||||
from tencentcloud.tat.v20201028 import models as tat_models, tat_client
|
||||
|
||||
|
||||
# 项目根目录，后面要从这里读 .env。
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
# 默认 .env 放在项目根目录，并且不入库。
|
||||
ENV_FILE = Path(os.environ.get("APP_ENV_FILE", ROOT / ".env")).expanduser()
|
||||
# 默认 deploy 机器实例 ID。
|
||||
DEFAULT_INSTANCE_ID = os.environ.get("DEPLOY_INSTANCE_ID", "ins-ntmpcd3a")
|
||||
# 默认区域。
|
||||
DEFAULT_REGION = os.environ.get("DEPLOY_REGION", "me-saudi-arabia")
|
||||
# 默认开放的端口。
|
||||
DEFAULT_PORT = int(os.environ.get("OPEN_PORT", "2026"))
|
||||
# 默认允许所有来源访问该端口。
|
||||
DEFAULT_CIDR = os.environ.get("OPEN_CIDR", "0.0.0.0/0")
|
||||
# 泛型只用于 invoke 的返回类型提示。
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
def load_env_file(path: Path) -> None:
|
||||
# .env 不存在时允许继续使用外部环境变量。
|
||||
if not path.exists():
|
||||
return
|
||||
|
||||
# 逐行解析 shell 风格的 KEY=VALUE。
|
||||
for raw_line in path.read_text(encoding="utf-8").splitlines():
|
||||
# 去掉空白方便判断。
|
||||
line = raw_line.strip()
|
||||
# 空行直接跳过。
|
||||
if not line:
|
||||
continue
|
||||
# 注释行直接跳过。
|
||||
if line.startswith("#"):
|
||||
continue
|
||||
# 无等号行直接跳过。
|
||||
if "=" not in line:
|
||||
continue
|
||||
|
||||
# 只按第一个等号拆分。
|
||||
key, value = line.split("=", 1)
|
||||
# 清理 key 空白。
|
||||
key = key.strip()
|
||||
# 清理 value 空白。
|
||||
value = value.strip()
|
||||
|
||||
# 去掉成对双引号。
|
||||
if value.startswith('"') and value.endswith('"'):
|
||||
value = value[1:-1]
|
||||
|
||||
# 去掉成对单引号。
|
||||
if value.startswith("'") and value.endswith("'"):
|
||||
value = value[1:-1]
|
||||
|
||||
# 只有环境里还没有这个 key 时才写入。
|
||||
os.environ.setdefault(key, value)
|
||||
|
||||
|
||||
# 先从项目根目录加载 .env。
|
||||
load_env_file(ENV_FILE)
|
||||
|
||||
|
||||
def env_required(name: str) -> str:
|
||||
# 读取环境变量并去掉两端空白。
|
||||
value = os.environ.get(name, "").strip()
|
||||
# 缺失时直接退出，避免后面报一串 SDK 错误。
|
||||
if not value:
|
||||
raise SystemExit(f"missing env: {name}")
|
||||
# 返回有效值。
|
||||
return value
|
||||
|
||||
|
||||
def build_cvm(secret_id: str, secret_key: str, region: str) -> cvm_client.CvmClient:
|
||||
# 构造腾讯云凭据对象。
|
||||
cred = credential.Credential(secret_id, secret_key)
|
||||
# 返回 CVM 客户端。
|
||||
return cvm_client.CvmClient(
|
||||
cred,
|
||||
region,
|
||||
@ -42,7 +109,9 @@ def build_cvm(secret_id: str, secret_key: str, region: str) -> cvm_client.CvmCli
|
||||
|
||||
|
||||
def build_tat(secret_id: str, secret_key: str, region: str) -> tat_client.TatClient:
|
||||
# 构造腾讯云凭据对象。
|
||||
cred = credential.Credential(secret_id, secret_key)
|
||||
# 返回 TAT 客户端。
|
||||
return tat_client.TatClient(
|
||||
cred,
|
||||
region,
|
||||
@ -51,34 +120,58 @@ def build_tat(secret_id: str, secret_key: str, region: str) -> tat_client.TatCli
|
||||
|
||||
|
||||
def invoke(action: Callable[[], T], label: str, retries: int = 4, delay_seconds: int = 2) -> T:
|
||||
# 保存最后一次异常，方便重试结束后抛出。
|
||||
last_error: Exception | None = None
|
||||
|
||||
# 按给定次数进行重试。
|
||||
for attempt in range(1, retries + 1):
|
||||
try:
|
||||
# 成功时直接返回结果。
|
||||
return action()
|
||||
except TencentCloudSDKException as exc:
|
||||
# 保存异常信息。
|
||||
last_error = exc
|
||||
# 最后一次重试不再 sleep，直接退出循环。
|
||||
if attempt == retries:
|
||||
break
|
||||
# 简单线性退避，降低偶发 EOF 的影响。
|
||||
time.sleep(delay_seconds * attempt)
|
||||
|
||||
# 有异常就把最后一次异常抛出去。
|
||||
if last_error is not None:
|
||||
raise last_error
|
||||
|
||||
# 理论上不会到这里，这里只是兜底。
|
||||
raise RuntimeError(f"{label} failed without error")
|
||||
|
||||
|
||||
def resolve_security_group_id(client: cvm_client.CvmClient, instance_id: str) -> str:
|
||||
# 创建查询实例详情的请求对象。
|
||||
req = cvm_models.DescribeInstancesRequest()
|
||||
# 只查询目标实例。
|
||||
req.from_json_string(json.dumps({"InstanceIds": [instance_id]}))
|
||||
# 发起实例查询请求。
|
||||
resp = json.loads(invoke(lambda: client.DescribeInstances(req), "DescribeInstances").to_json_string())
|
||||
# 拿到实例数组。
|
||||
instances = resp.get("InstanceSet") or []
|
||||
|
||||
# 实例不存在时直接退出。
|
||||
if not instances:
|
||||
raise SystemExit(f"instance not found: {instance_id}")
|
||||
|
||||
# 读取第一个绑定安全组。
|
||||
security_group_ids = instances[0].get("SecurityGroupIds") or []
|
||||
|
||||
# 没有绑定安全组时直接退出。
|
||||
if not security_group_ids:
|
||||
raise SystemExit(f"no security group attached: {instance_id}")
|
||||
|
||||
# 返回第一个安全组 ID。
|
||||
return str(security_group_ids[0])
|
||||
|
||||
|
||||
def remote_script(secret_id: str, secret_key: str, region: str, security_group_id: str, port: int, cidr: str) -> str:
|
||||
# 把必要参数序列化后再 base64，方便塞进远端 Python 脚本。
|
||||
payload = {
|
||||
"secret_id": secret_id,
|
||||
"secret_key": secret_key,
|
||||
@ -87,7 +180,11 @@ def remote_script(secret_id: str, secret_key: str, region: str, security_group_i
|
||||
"port": port,
|
||||
"cidr": cidr,
|
||||
}
|
||||
|
||||
# 远端脚本里直接解码这个 blob。
|
||||
blob = base64.b64encode(json.dumps(payload).encode("utf-8")).decode("ascii")
|
||||
|
||||
# 返回完整远端 shell 脚本。
|
||||
return f"""set -euo pipefail
|
||||
|
||||
if command -v firewall-cmd >/dev/null 2>&1; then
|
||||
@ -103,26 +200,30 @@ if command -v iptables >/dev/null 2>&1; then
|
||||
iptables -C INPUT -p tcp --dport {port} -j ACCEPT >/dev/null 2>&1 || iptables -I INPUT -p tcp --dport {port} -j ACCEPT || true
|
||||
fi
|
||||
|
||||
python3 -m pip install --user -q tencentcloud-sdk-python >/dev/null 2>&1 || true
|
||||
python3 -m pip install --quiet --disable-pip-version-check tencentcloud-sdk-python >/dev/null 2>&1 || true
|
||||
|
||||
python3 - <<'PY'
|
||||
import base64
|
||||
import json
|
||||
|
||||
from tencentcloud.common import credential
|
||||
from tencentcloud.common.exception.tencent_cloud_sdk_exception import TencentCloudSDKException
|
||||
from tencentcloud.common.profile.client_profile import ClientProfile
|
||||
from tencentcloud.common.profile.http_profile import HttpProfile
|
||||
from tencentcloud.vpc.v20170312 import vpc_client, models
|
||||
from tencentcloud.common.exception.tencent_cloud_sdk_exception import TencentCloudSDKException
|
||||
from tencentcloud.vpc.v20170312 import models, vpc_client
|
||||
|
||||
# 读取本次放行需要的参数。
|
||||
payload = json.loads(base64.b64decode({blob!r}).decode('utf-8'))
|
||||
# 构造凭据对象。
|
||||
cred = credential.Credential(payload['secret_id'], payload['secret_key'])
|
||||
# 初始化 VPC 客户端，用来写安全组规则。
|
||||
client = vpc_client.VpcClient(
|
||||
cred,
|
||||
payload['region'],
|
||||
ClientProfile(httpProfile=HttpProfile(endpoint='vpc.tencentcloudapi.com')),
|
||||
)
|
||||
|
||||
# 构造创建安全组入站规则的请求。
|
||||
req = models.CreateSecurityGroupPoliciesRequest()
|
||||
req.from_json_string(json.dumps({{
|
||||
'SecurityGroupId': payload['security_group_id'],
|
||||
@ -132,15 +233,17 @@ req.from_json_string(json.dumps({{
|
||||
'Port': str(payload['port']),
|
||||
'CidrBlock': payload['cidr'],
|
||||
'Action': 'ACCEPT',
|
||||
'PolicyDescription': f"hy-app-monitor {payload['port']}",
|
||||
'PolicyDescription': f"hy-app-monitor {{payload['port']}}",
|
||||
}}]
|
||||
}}
|
||||
}}))
|
||||
|
||||
try:
|
||||
# 创建安全组规则。
|
||||
client.CreateSecurityGroupPolicies(req)
|
||||
print('security_group_rule=created')
|
||||
except TencentCloudSDKException as exc:
|
||||
# 已存在时按成功处理，避免重复调用失败。
|
||||
if 'Duplicate' in str(exc) or 'already exists' in str(exc):
|
||||
print('security_group_rule=exists')
|
||||
else:
|
||||
@ -152,7 +255,9 @@ echo "port_opened={port}"
|
||||
|
||||
|
||||
def run_tat_script(client: tat_client.TatClient, instance_id: str, script: str, timeout_seconds: int = 300) -> str:
|
||||
# 创建执行命令请求对象。
|
||||
req = tat_models.RunCommandRequest()
|
||||
# 把 shell 脚本下发到目标实例执行。
|
||||
req.from_json_string(
|
||||
json.dumps(
|
||||
{
|
||||
@ -165,13 +270,21 @@ def run_tat_script(client: tat_client.TatClient, instance_id: str, script: str,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
# 发起执行命令请求。
|
||||
resp = json.loads(invoke(lambda: client.RunCommand(req), "RunCommand").to_json_string())
|
||||
# 取出这次执行的 invocation id。
|
||||
invocation_id = resp["InvocationId"]
|
||||
# 设置轮询截止时间。
|
||||
deadline = time.time() + timeout_seconds + 120
|
||||
|
||||
# 循环轮询任务结果。
|
||||
while time.time() < deadline:
|
||||
# 两次轮询之间稍微停一下。
|
||||
time.sleep(3)
|
||||
# 创建查询任务结果请求。
|
||||
query = tat_models.DescribeInvocationTasksRequest()
|
||||
# 按 invocation id 查询本次任务。
|
||||
query.from_json_string(
|
||||
json.dumps(
|
||||
{
|
||||
@ -181,37 +294,66 @@ def run_tat_script(client: tat_client.TatClient, instance_id: str, script: str,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
# 读取任务执行状态。
|
||||
data = json.loads(invoke(lambda: client.DescribeInvocationTasks(query), "DescribeInvocationTasks").to_json_string())
|
||||
# 拿到任务列表。
|
||||
tasks = data.get("InvocationTaskSet") or []
|
||||
|
||||
# 任务结果还没出来就继续等。
|
||||
if not tasks:
|
||||
continue
|
||||
|
||||
# 这里只打单实例，所以拿第一个任务即可。
|
||||
task = tasks[0]
|
||||
# 读取任务状态。
|
||||
status = task["TaskStatus"]
|
||||
|
||||
# 进入终态时退出轮询。
|
||||
if status in {"SUCCESS", "FAILED", "TIMEOUT", "CANCELLED", "START_FAILED"}:
|
||||
# 解码远端标准输出。
|
||||
output = base64.b64decode(task.get("TaskResult", {}).get("Output", "")).decode("utf-8", errors="replace")
|
||||
# 非 SUCCESS 直接退出并带上远端输出。
|
||||
if status != "SUCCESS":
|
||||
raise SystemExit(output or f"TAT task failed: {status}")
|
||||
# SUCCESS 时返回远端输出。
|
||||
return output
|
||||
|
||||
# 超过截止时间还没完成就直接报超时。
|
||||
raise SystemExit("TAT task timed out")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
# 从 .env 或外部环境中读取腾讯云密钥。
|
||||
secret_id = env_required("TENCENT_SECRET_ID")
|
||||
# 从 .env 或外部环境中读取腾讯云密钥。
|
||||
secret_key = env_required("TENCENT_SECRET_KEY")
|
||||
# 读取目标实例 ID。
|
||||
instance_id = os.environ.get("DEPLOY_INSTANCE_ID", DEFAULT_INSTANCE_ID)
|
||||
# 读取区域。
|
||||
region = os.environ.get("DEPLOY_REGION", DEFAULT_REGION)
|
||||
# 读取待放行端口。
|
||||
port = int(os.environ.get("OPEN_PORT", str(DEFAULT_PORT)))
|
||||
# 读取允许访问的网段。
|
||||
cidr = os.environ.get("OPEN_CIDR", DEFAULT_CIDR)
|
||||
|
||||
# 初始化 CVM 客户端，用来查安全组。
|
||||
cvm = build_cvm(secret_id, secret_key, region)
|
||||
# 初始化 TAT 客户端，用来远程执行脚本。
|
||||
tat = build_tat(secret_id, secret_key, region)
|
||||
# 优先用显式指定的安全组，否则自动从实例上解析。
|
||||
security_group_id = os.environ.get("SECURITY_GROUP_ID", "").strip() or resolve_security_group_id(cvm, instance_id)
|
||||
|
||||
# 远程执行端口放行脚本。
|
||||
output = run_tat_script(tat, instance_id, remote_script(secret_id, secret_key, region, security_group_id, port, cidr))
|
||||
# 打印实例 ID 方便确认目标机器。
|
||||
print(f"instance_id={instance_id}")
|
||||
# 打印实际使用的安全组 ID。
|
||||
print(f"security_group_id={security_group_id}")
|
||||
# 打印远端执行结果。
|
||||
print(output.strip())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 作为脚本执行时直接走主流程。
|
||||
main()
|
||||
|
||||
@ -5,11 +5,28 @@ ROOT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)"
|
||||
RUN_DIR="$ROOT_DIR/run"
|
||||
PID_FILE="$RUN_DIR/hy-app-monitor.pid"
|
||||
LOG_FILE="$RUN_DIR/hy-app-monitor.log"
|
||||
ENV_FILE="${APP_ENV_FILE:-$ROOT_DIR/.env}"
|
||||
SYSTEMD_SERVICE_NAME="hy-app-monitor.service"
|
||||
|
||||
if [ -f "$ENV_FILE" ]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
. "$ENV_FILE"
|
||||
set +a
|
||||
fi
|
||||
|
||||
APP_BIND="${APP_BIND:-0.0.0.0}"
|
||||
APP_PORT="${APP_PORT:-2026}"
|
||||
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
||||
TARGETS_FILE="${TARGETS_FILE:-$ROOT_DIR/config/targets.json}"
|
||||
|
||||
if [ -x "$ROOT_DIR/.venv/bin/python" ]; then
|
||||
PYTHON_BIN_DEFAULT="$ROOT_DIR/.venv/bin/python"
|
||||
else
|
||||
PYTHON_BIN_DEFAULT="python3"
|
||||
fi
|
||||
|
||||
PYTHON_BIN="${PYTHON_BIN:-$PYTHON_BIN_DEFAULT}"
|
||||
|
||||
mkdir -p "$RUN_DIR"
|
||||
|
||||
read_pid() {
|
||||
@ -26,3 +43,10 @@ is_running() {
|
||||
fi
|
||||
kill -0 "$pid" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
has_systemd_unit() {
|
||||
if ! command -v systemctl >/dev/null 2>&1; then
|
||||
return 1
|
||||
fi
|
||||
systemctl cat "$SYSTEMD_SERVICE_NAME" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
26
scripts/install_systemd.sh
Executable file
26
scripts/install_systemd.sh
Executable file
@ -0,0 +1,26 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
UNIT_SRC="$ROOT_DIR/systemd/hy-app-monitor.service"
|
||||
UNIT_DST="/etc/systemd/system/$SYSTEMD_SERVICE_NAME"
|
||||
|
||||
if [ ! -f "$UNIT_SRC" ]; then
|
||||
echo "systemd unit missing: $UNIT_SRC" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
echo "env file missing: $ENV_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
install -m 0644 "$UNIT_SRC" "$UNIT_DST"
|
||||
chmod +x "$ROOT_DIR"/scripts/*.sh
|
||||
systemctl daemon-reload
|
||||
systemctl enable "$SYSTEMD_SERVICE_NAME"
|
||||
systemctl restart "$SYSTEMD_SERVICE_NAME"
|
||||
systemctl is-active "$SYSTEMD_SERVICE_NAME"
|
||||
@ -2,6 +2,14 @@
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
if has_systemd_unit; then
|
||||
systemctl restart "$SYSTEMD_SERVICE_NAME"
|
||||
systemctl is-active "$SYSTEMD_SERVICE_NAME"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
"$SCRIPT_DIR/stop.sh" || true
|
||||
sleep 1
|
||||
|
||||
14
scripts/run_foreground.sh
Executable file
14
scripts/run_foreground.sh
Executable file
@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
if [ ! -f "$TARGETS_FILE" ]; then
|
||||
echo "targets file not found: $TARGETS_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
export APP_BIND APP_PORT TARGETS_FILE PYTHONUNBUFFERED=1
|
||||
exec "$PYTHON_BIN" "$ROOT_DIR/server.py"
|
||||
@ -5,6 +5,12 @@ SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
if has_systemd_unit; then
|
||||
systemctl start "$SYSTEMD_SERVICE_NAME"
|
||||
systemctl is-active "$SYSTEMD_SERVICE_NAME"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if is_running; then
|
||||
echo "hy-app-monitor already running pid=$(read_pid)"
|
||||
exit 0
|
||||
|
||||
@ -5,6 +5,15 @@ SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
if has_systemd_unit; then
|
||||
if systemctl is-active --quiet "$SYSTEMD_SERVICE_NAME"; then
|
||||
echo "hy-app-monitor running under systemd port=$APP_PORT"
|
||||
exit 0
|
||||
fi
|
||||
echo "hy-app-monitor stopped"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if is_running; then
|
||||
echo "hy-app-monitor running pid=$(read_pid) port=$APP_PORT"
|
||||
exit 0
|
||||
|
||||
@ -5,6 +5,12 @@ SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
if has_systemd_unit; then
|
||||
systemctl stop "$SYSTEMD_SERVICE_NAME"
|
||||
echo "hy-app-monitor stopped"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! is_running; then
|
||||
echo "hy-app-monitor not running"
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
16
scripts/uninstall_systemd.sh
Executable file
16
scripts/uninstall_systemd.sh
Executable file
@ -0,0 +1,16 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
UNIT_DST="/etc/systemd/system/$SYSTEMD_SERVICE_NAME"
|
||||
|
||||
if command -v systemctl >/dev/null 2>&1; then
|
||||
systemctl disable --now "$SYSTEMD_SERVICE_NAME" >/dev/null 2>&1 || true
|
||||
systemctl daemon-reload || true
|
||||
fi
|
||||
|
||||
rm -f "$UNIT_DST"
|
||||
echo "systemd unit removed: $UNIT_DST"
|
||||
723
server.py
723
server.py
@ -2,91 +2,232 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# 标准库并发工具，用来并行探测各个服务健康状态。
|
||||
import concurrent.futures
|
||||
# 标准 JSON 编解码，用来读取配置和输出接口响应。
|
||||
import json
|
||||
# 进程环境变量读取，用来接收 .env 和 systemd 注入的运行参数。
|
||||
import os
|
||||
# 线程锁用来保护缓存，避免并发请求重复打 TAT 和健康检查。
|
||||
import threading
|
||||
# 时间模块用于缓存过期判断和 CPU 采样间隔。
|
||||
import time
|
||||
# urllib 用于直接探测各台机器上的 HTTP 健康检查接口。
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
# 时间戳统一输出成 UTC ISO 字符串，前端展示更稳定。
|
||||
from datetime import datetime, timezone
|
||||
# HTTP 状态码常量，避免在代码里写魔法数字。
|
||||
from http import HTTPStatus
|
||||
# 使用标准库 HTTP 服务器承载监控页和 JSON API。
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
# 路径对象比字符串拼接更安全。
|
||||
from pathlib import Path
|
||||
# 类型提示用于提升代码可读性。
|
||||
from typing import Any
|
||||
# 解析请求路径时只关心 path 段。
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
# 项目根目录，后续所有相对路径都从这里展开。
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
# 静态资源目录，存放 HTML、CSS、JS、Vue 文件。
|
||||
STATIC_DIR = ROOT / "static"
|
||||
CONFIG_PATH = Path(os.environ.get("TARGETS_FILE", ROOT / "config" / "targets.json")).expanduser()
|
||||
APP_BIND = os.environ.get("APP_BIND", "0.0.0.0").strip() or "0.0.0.0"
|
||||
APP_PORT = int(os.environ.get("APP_PORT", "2026"))
|
||||
PROBE_TIMEOUT_SECONDS = float(os.environ.get("PROBE_TIMEOUT_SECONDS", "3"))
|
||||
PROBE_CACHE_TTL_SECONDS = float(os.environ.get("PROBE_CACHE_TTL_SECONDS", "5"))
|
||||
PROBE_MAX_WORKERS = int(os.environ.get("PROBE_MAX_WORKERS", "12"))
|
||||
# 默认 .env 文件放在项目根目录，并且不入库。
|
||||
DEFAULT_ENV_PATH = ROOT / ".env"
|
||||
|
||||
_CACHE_LOCK = threading.Lock()
|
||||
_CACHE_PAYLOAD: dict | None = None
|
||||
_CACHE_AT = 0.0
|
||||
|
||||
def load_env_file(path: Path) -> None:
|
||||
# 如果 .env 不存在就直接跳过，允许只靠外部环境变量运行。
|
||||
if not path.exists():
|
||||
return
|
||||
|
||||
# 逐行读取 .env，使用 shell 兼容的 KEY=VALUE 结构。
|
||||
for raw_line in path.read_text(encoding="utf-8").splitlines():
|
||||
# 去掉首尾空白，便于判断空行和注释。
|
||||
line = raw_line.strip()
|
||||
# 空行直接跳过。
|
||||
if not line:
|
||||
continue
|
||||
# 注释行直接跳过。
|
||||
if line.startswith("#"):
|
||||
continue
|
||||
# 没有等号的无效行直接跳过，避免把异常内容灌进环境变量。
|
||||
if "=" not in line:
|
||||
continue
|
||||
|
||||
# 只按第一个等号拆分，避免 value 里出现额外等号时被截断。
|
||||
key, value = line.split("=", 1)
|
||||
# 去掉 key 的两端空格。
|
||||
key = key.strip()
|
||||
# 去掉 value 的两端空格。
|
||||
value = value.strip()
|
||||
|
||||
# 去掉成对的双引号，兼容常见 .env 写法。
|
||||
if value.startswith('"') and value.endswith('"'):
|
||||
value = value[1:-1]
|
||||
|
||||
# 去掉成对的单引号，兼容常见 .env 写法。
|
||||
if value.startswith("'") and value.endswith("'"):
|
||||
value = value[1:-1]
|
||||
|
||||
# 只在环境变量不存在时写入，保证外部显式传入的值优先级更高。
|
||||
os.environ.setdefault(key, value)
|
||||
|
||||
|
||||
# 先加载项目根目录的 .env，后面常量都基于这个环境初始化。
|
||||
load_env_file(Path(os.environ.get("APP_ENV_FILE", DEFAULT_ENV_PATH)).expanduser())
|
||||
|
||||
# 配置文件路径默认指向 config/targets.json，也允许用环境变量覆盖。
|
||||
CONFIG_PATH = Path(os.environ.get("TARGETS_FILE", ROOT / "config" / "targets.json")).expanduser()
|
||||
# 默认监听所有网卡。
|
||||
APP_BIND = os.environ.get("APP_BIND", "0.0.0.0").strip() or "0.0.0.0"
|
||||
# 页面服务默认监听 2026，避开浏览器 unsafe port。
|
||||
APP_PORT = int(os.environ.get("APP_PORT", "2026"))
|
||||
# 单个服务健康探测超时时间。
|
||||
PROBE_TIMEOUT_SECONDS = float(os.environ.get("PROBE_TIMEOUT_SECONDS", "3"))
|
||||
# 服务健康探测缓存秒数。
|
||||
PROBE_CACHE_TTL_SECONDS = float(os.environ.get("PROBE_CACHE_TTL_SECONDS", "5"))
|
||||
# 服务探测并发数，避免串行等待太久。
|
||||
PROBE_MAX_WORKERS = int(os.environ.get("PROBE_MAX_WORKERS", "12"))
|
||||
# 主机指标缓存时间比服务探测更长，因为 TAT 成本更高。
|
||||
HOST_METRIC_CACHE_TTL_SECONDS = float(os.environ.get("HOST_METRIC_CACHE_TTL_SECONDS", "25"))
|
||||
# TAT 执行超时时间，主机采集会做一次 CPU 采样，所以时间略长一点。
|
||||
TAT_METRIC_TIMEOUT_SECONDS = int(os.environ.get("TAT_METRIC_TIMEOUT_SECONDS", "30"))
|
||||
# 采样 CPU 时的两次读数间隔。
|
||||
TAT_CPU_SAMPLE_SECONDS = float(os.environ.get("TAT_CPU_SAMPLE_SECONDS", "1"))
|
||||
|
||||
# 服务健康结果缓存锁。
|
||||
_SERVICE_CACHE_LOCK = threading.Lock()
|
||||
# 服务健康结果缓存内容。
|
||||
_SERVICE_CACHE_PAYLOAD: dict[str, Any] | None = None
|
||||
# 服务健康结果最后生成时间。
|
||||
_SERVICE_CACHE_AT = 0.0
|
||||
|
||||
# 主机指标缓存锁。
|
||||
_HOST_CACHE_LOCK = threading.Lock()
|
||||
# 主机指标缓存内容。
|
||||
_HOST_CACHE_PAYLOAD: dict[str, dict[str, Any]] | None = None
|
||||
# 主机指标最后生成时间。
|
||||
_HOST_CACHE_AT = 0.0
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
# 统一返回无微秒的 UTC ISO 时间，前后端更容易比较。
|
||||
return datetime.now(timezone.utc).replace(microsecond=0).isoformat()
|
||||
|
||||
|
||||
def load_targets() -> list[dict]:
|
||||
def load_hosts() -> list[dict[str, Any]]:
|
||||
# 读取监控配置文件。
|
||||
payload = json.loads(CONFIG_PATH.read_text(encoding="utf-8"))
|
||||
# hosts 节点不存在时回退成空列表。
|
||||
hosts = payload.get("hosts") or []
|
||||
normalized: list[dict] = []
|
||||
# 规范化后的主机列表。
|
||||
normalized: list[dict[str, Any]] = []
|
||||
|
||||
# 逐台主机提取可用字段。
|
||||
for host in hosts:
|
||||
services = host.get("services") or []
|
||||
for service in services:
|
||||
normalized.append(
|
||||
# 统一整理字段，避免前端和后端在不同地方重复判断默认值。
|
||||
normalized.append(
|
||||
{
|
||||
"host": str(host.get("name") or "").strip(),
|
||||
"group": str(host.get("group") or "").strip(),
|
||||
"ip": str(host.get("ip") or "").strip(),
|
||||
"instanceId": str(host.get("instanceId") or "").strip(),
|
||||
"services": host.get("services") or [],
|
||||
}
|
||||
)
|
||||
|
||||
# 返回主机列表给后续服务探测和 TAT 指标采集复用。
|
||||
return normalized
|
||||
|
||||
|
||||
def flatten_services(hosts: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
# 扁平化后的服务列表，便于并发探测。
|
||||
services: list[dict[str, Any]] = []
|
||||
|
||||
# 按主机展开它上面的所有服务。
|
||||
for host in hosts:
|
||||
# 逐个服务转换为统一结构。
|
||||
for service in host.get("services") or []:
|
||||
services.append(
|
||||
{
|
||||
"host": str(host.get("name") or "").strip(),
|
||||
"group": str(host.get("group") or "").strip(),
|
||||
"ip": str(host.get("ip") or "").strip(),
|
||||
"host": host["host"],
|
||||
"group": host["group"],
|
||||
"ip": host["ip"],
|
||||
"instanceId": host["instanceId"],
|
||||
"service": str(service.get("name") or "").strip(),
|
||||
"kind": str(service.get("kind") or "java").strip(),
|
||||
"port": int(service.get("port")),
|
||||
"path": str(service.get("path") or "/").strip(),
|
||||
}
|
||||
)
|
||||
return normalized
|
||||
|
||||
# 返回扁平服务列表供 HTTP 探测使用。
|
||||
return services
|
||||
|
||||
|
||||
def _healthy_from_body(service_kind: str, status_code: int, body_text: str) -> bool:
|
||||
def healthy_from_body(service_kind: str, status_code: int, body_text: str) -> bool:
|
||||
# 先把响应体统一转成小写，便于后续模糊判断。
|
||||
lowered = body_text.lower()
|
||||
|
||||
# Golang 服务当前按项目里的 /health 返回体做判断。
|
||||
if service_kind == "golang":
|
||||
return status_code == 200 and ('"code":"ok"' in lowered or '"code": "ok"' in lowered or '"status":"ok"' in lowered)
|
||||
return status_code == 200 and (
|
||||
'"code":"ok"' in lowered
|
||||
or '"code": "ok"' in lowered
|
||||
or '"status":"ok"' in lowered
|
||||
or '"status": "ok"' in lowered
|
||||
)
|
||||
|
||||
# 有些 Java 网关健康接口会返回 401/403，但服务本身是活的。
|
||||
if status_code in {401, 403}:
|
||||
return True
|
||||
|
||||
# 其余 Java 服务非 200 一律认为异常。
|
||||
if status_code != 200:
|
||||
return False
|
||||
|
||||
# Spring Boot actuator 常见格式是 status=UP。
|
||||
if '"status":"up"' in lowered or '"status": "up"' in lowered:
|
||||
return True
|
||||
|
||||
# 兼容只返回裸字符串 UP 的老接口。
|
||||
if body_text.strip() == "UP":
|
||||
return True
|
||||
|
||||
# 最后再做一次保守模糊判断。
|
||||
return "up" in lowered and "status" in lowered
|
||||
|
||||
|
||||
def probe_target(target: dict) -> dict:
|
||||
url = f"http://{target['ip']}:{target['port']}{target['path']}"
|
||||
def probe_service(service: dict[str, Any]) -> dict[str, Any]:
|
||||
# 拼出服务健康检查地址。
|
||||
url = f"http://{service['ip']}:{service['port']}{service['path']}"
|
||||
# 记录开始时间，用来计算耗时。
|
||||
started = time.perf_counter()
|
||||
# 构造请求对象，带一个简单的 User-Agent 便于区分来源。
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "hy-app-monitor/1.0"})
|
||||
|
||||
# 先准备公共返回结构。
|
||||
base = {
|
||||
**target,
|
||||
**service,
|
||||
"url": url,
|
||||
"statusCode": 0,
|
||||
"latencyMs": 0,
|
||||
"ok": False,
|
||||
"detail": "",
|
||||
}
|
||||
|
||||
try:
|
||||
# 发起 HTTP 请求探测服务。
|
||||
with urllib.request.urlopen(request, timeout=PROBE_TIMEOUT_SECONDS) as response:
|
||||
# 读取响应体，并按 UTF-8 兜底解码。
|
||||
body_text = response.read().decode("utf-8", errors="replace")
|
||||
# 计算请求耗时。
|
||||
latency_ms = int((time.perf_counter() - started) * 1000)
|
||||
ok = _healthy_from_body(target["kind"], response.status, body_text)
|
||||
# 基于状态码和响应体判断服务是否健康。
|
||||
ok = healthy_from_body(service["kind"], response.status, body_text)
|
||||
# 返回完整探测结果。
|
||||
return {
|
||||
**base,
|
||||
"statusCode": int(response.status),
|
||||
@ -95,9 +236,13 @@ def probe_target(target: dict) -> dict:
|
||||
"detail": body_text[:220].replace("\n", " ").strip(),
|
||||
}
|
||||
except urllib.error.HTTPError as exc:
|
||||
# 即使是 HTTP 错误，也尽量读取响应体做更准确的健康判断。
|
||||
body_text = exc.read().decode("utf-8", errors="replace")
|
||||
# 计算失败请求耗时。
|
||||
latency_ms = int((time.perf_counter() - started) * 1000)
|
||||
ok = _healthy_from_body(target["kind"], exc.code, body_text)
|
||||
# 用 HTTP 错误码继续判断是否属于“服务活着但被拒绝访问”。
|
||||
ok = healthy_from_body(service["kind"], exc.code, body_text)
|
||||
# 返回错误场景下的探测结果。
|
||||
return {
|
||||
**base,
|
||||
"statusCode": int(exc.code),
|
||||
@ -106,7 +251,9 @@ def probe_target(target: dict) -> dict:
|
||||
"detail": body_text[:220].replace("\n", " ").strip() or str(exc),
|
||||
}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 其他异常统一记录为失败，并把异常文本带给前端。
|
||||
latency_ms = int((time.perf_counter() - started) * 1000)
|
||||
# 返回失败结果，方便排查网络超时和连接拒绝等问题。
|
||||
return {
|
||||
**base,
|
||||
"latencyMs": latency_ms,
|
||||
@ -114,64 +261,522 @@ def probe_target(target: dict) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def build_monitor_payload() -> dict:
|
||||
targets = load_targets()
|
||||
workers = max(1, min(PROBE_MAX_WORKERS, len(targets) or 1))
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor:
|
||||
items = list(executor.map(probe_target, targets))
|
||||
def build_service_payload(hosts: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
# 把主机配置展开成服务配置。
|
||||
services = flatten_services(hosts)
|
||||
# 并发数至少 1，且不超过配置值和服务总数。
|
||||
workers = max(1, min(PROBE_MAX_WORKERS, len(services) or 1))
|
||||
|
||||
# 并发探测所有服务。
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor:
|
||||
items = list(executor.map(probe_service, services))
|
||||
|
||||
# 统计成功服务数量。
|
||||
ok_count = sum(1 for item in items if item["ok"])
|
||||
payload = {
|
||||
|
||||
# 返回服务层结果。
|
||||
return {
|
||||
"updatedAt": utc_now(),
|
||||
"summary": {
|
||||
"total": len(items),
|
||||
"ok": ok_count,
|
||||
"down": len(items) - ok_count,
|
||||
"hostTotal": len(hosts),
|
||||
},
|
||||
"items": items,
|
||||
}
|
||||
return payload
|
||||
|
||||
|
||||
def get_monitor_payload() -> dict:
|
||||
global _CACHE_PAYLOAD, _CACHE_AT
|
||||
def get_service_payload(hosts: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
# 使用独立缓存，减少频繁刷新带来的内网 HTTP 压力。
|
||||
global _SERVICE_CACHE_PAYLOAD, _SERVICE_CACHE_AT
|
||||
|
||||
# 记录当前时间，用来判断缓存是否过期。
|
||||
now = time.time()
|
||||
with _CACHE_LOCK:
|
||||
if _CACHE_PAYLOAD is not None and now - _CACHE_AT < PROBE_CACHE_TTL_SECONDS:
|
||||
return _CACHE_PAYLOAD
|
||||
payload = build_monitor_payload()
|
||||
with _CACHE_LOCK:
|
||||
_CACHE_PAYLOAD = payload
|
||||
_CACHE_AT = now
|
||||
|
||||
# 先在锁内检查服务缓存是否还能复用。
|
||||
with _SERVICE_CACHE_LOCK:
|
||||
if _SERVICE_CACHE_PAYLOAD is not None and now - _SERVICE_CACHE_AT < PROBE_CACHE_TTL_SECONDS:
|
||||
return _SERVICE_CACHE_PAYLOAD
|
||||
|
||||
# 缓存过期时重新构建服务探测结果。
|
||||
payload = build_service_payload(hosts)
|
||||
|
||||
# 在锁内写回最新缓存。
|
||||
with _SERVICE_CACHE_LOCK:
|
||||
_SERVICE_CACHE_PAYLOAD = payload
|
||||
_SERVICE_CACHE_AT = now
|
||||
|
||||
# 返回最新服务探测结果。
|
||||
return payload
|
||||
|
||||
|
||||
def response_json(payload: dict | list) -> bytes:
|
||||
def decode_task_output(output: str) -> str:
|
||||
# TAT 返回的输出是 base64 文本，这里统一做解码。
|
||||
if not output:
|
||||
return ""
|
||||
|
||||
# 解码成 UTF-8 文本，异常字符用 replace 兜底。
|
||||
import base64
|
||||
|
||||
return base64.b64decode(output).decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
def build_tat_client() -> tuple[Any | None, str]:
|
||||
# 如果没有腾讯云密钥，直接返回错误说明，前端就显示“无主机指标”。
|
||||
secret_id = os.environ.get("TENCENT_SECRET_ID", "").strip()
|
||||
secret_key = os.environ.get("TENCENT_SECRET_KEY", "").strip()
|
||||
region = os.environ.get("DEPLOY_REGION", "").strip()
|
||||
|
||||
# 密钥缺失时不再继续初始化 SDK。
|
||||
if not secret_id or not secret_key or not region:
|
||||
return None, "missing TAT credentials in .env"
|
||||
|
||||
try:
|
||||
# 运行时再动态导入 SDK，避免没装 SDK 时整个页面直接起不来。
|
||||
from tencentcloud.common import credential
|
||||
from tencentcloud.common.exception.tencent_cloud_sdk_exception import TencentCloudSDKException
|
||||
from tencentcloud.common.profile.client_profile import ClientProfile
|
||||
from tencentcloud.common.profile.http_profile import HttpProfile
|
||||
from tencentcloud.tat.v20201028 import tat_client
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# SDK 缺失时返回错误信息给前端。
|
||||
return None, f"tencentcloud sdk unavailable: {exc}"
|
||||
|
||||
# 创建凭据对象。
|
||||
cred = credential.Credential(secret_id, secret_key)
|
||||
# 创建 TAT 客户端，后续所有主机指标都通过它拉取。
|
||||
client = tat_client.TatClient(
|
||||
cred,
|
||||
region,
|
||||
ClientProfile(httpProfile=HttpProfile(endpoint="tat.tencentcloudapi.com")),
|
||||
)
|
||||
|
||||
# 成功时返回客户端和空错误。
|
||||
return client, ""
|
||||
|
||||
|
||||
def invoke_tat(action: Any, label: str, retries: int = 3, delay_seconds: int = 2) -> Any:
|
||||
# 这里延迟导入腾讯云异常类型，避免 SDK 缺失时模块导入直接失败。
|
||||
from tencentcloud.common.exception.tencent_cloud_sdk_exception import TencentCloudSDKException
|
||||
|
||||
# 保存最后一次异常，便于最终抛出。
|
||||
last_error: Exception | None = None
|
||||
|
||||
# 按给定次数做简单重试。
|
||||
for attempt in range(1, retries + 1):
|
||||
try:
|
||||
# 成功时直接返回调用结果。
|
||||
return action()
|
||||
except TencentCloudSDKException as exc:
|
||||
# 保存最后一次异常。
|
||||
last_error = exc
|
||||
# 最后一次重试不再 sleep。
|
||||
if attempt == retries:
|
||||
break
|
||||
# 做一个简单退避，避免瞬时网络抖动。
|
||||
time.sleep(delay_seconds * attempt)
|
||||
|
||||
# 有异常时直接抛出最后一次异常。
|
||||
if last_error is not None:
|
||||
raise last_error
|
||||
|
||||
# 理论上不会执行到这里，这里只是兜底。
|
||||
raise RuntimeError(f"{label} failed without error")
|
||||
|
||||
|
||||
def build_metrics_script() -> str:
|
||||
# 远端脚本本身只依赖 Python 标准库，避免每台机器再装额外 agent。
|
||||
return f"""set -euo pipefail
|
||||
python3 - <<'PY'
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import time
|
||||
|
||||
# 读取 /proc/stat 的第一行来采样 CPU。
|
||||
def read_cpu():
|
||||
with open('/proc/stat', 'r', encoding='utf-8') as handle:
|
||||
parts = handle.readline().split()[1:8]
|
||||
values = [int(item) for item in parts]
|
||||
idle = values[3] + values[4]
|
||||
total = sum(values)
|
||||
return idle, total
|
||||
|
||||
# 第一次 CPU 采样。
|
||||
idle_1, total_1 = read_cpu()
|
||||
# 等待一小段时间做第二次采样。
|
||||
time.sleep({TAT_CPU_SAMPLE_SECONDS})
|
||||
# 第二次 CPU 采样。
|
||||
idle_2, total_2 = read_cpu()
|
||||
|
||||
# 计算总差值，防止极端情况下出现 0。
|
||||
total_diff = max(total_2 - total_1, 1)
|
||||
# 计算 idle 差值。
|
||||
idle_diff = max(idle_2 - idle_1, 0)
|
||||
# 算出 CPU 使用率百分比。
|
||||
cpu_percent = round((1 - idle_diff / total_diff) * 100, 2)
|
||||
|
||||
# 解析内存信息。
|
||||
memory_info = {{}}
|
||||
with open('/proc/meminfo', 'r', encoding='utf-8') as handle:
|
||||
for line in handle:
|
||||
key, value = line.split(':', 1)
|
||||
memory_info[key] = int(value.strip().split()[0])
|
||||
|
||||
# 总内存，单位换成 GB。
|
||||
memory_total_gb = round(memory_info.get('MemTotal', 0) / 1024 / 1024, 2)
|
||||
# 可用内存，单位换成 GB。
|
||||
memory_available_gb = round(memory_info.get('MemAvailable', 0) / 1024 / 1024, 2)
|
||||
# 已用内存。
|
||||
memory_used_gb = round(max(memory_total_gb - memory_available_gb, 0), 2)
|
||||
# 内存使用率。
|
||||
memory_percent = round((memory_used_gb / memory_total_gb) * 100, 2) if memory_total_gb else 0.0
|
||||
|
||||
# 读取根分区磁盘占用。
|
||||
disk = shutil.disk_usage('/')
|
||||
# 总磁盘，单位换成 GB。
|
||||
disk_total_gb = round(disk.total / 1024 / 1024 / 1024, 2)
|
||||
# 已用磁盘，单位换成 GB。
|
||||
disk_used_gb = round(disk.used / 1024 / 1024 / 1024, 2)
|
||||
# 磁盘使用率。
|
||||
disk_percent = round((disk.used / disk.total) * 100, 2) if disk.total else 0.0
|
||||
|
||||
# 通过 /proc 目录估算当前进程数。
|
||||
process_count = len([name for name in os.listdir('/proc') if name.isdigit()])
|
||||
|
||||
# 输出统一 JSON，方便服务端直接解析。
|
||||
print(json.dumps({{
|
||||
'ok': True,
|
||||
'hostname': os.uname().nodename,
|
||||
'cpuPercent': cpu_percent,
|
||||
'memoryPercent': memory_percent,
|
||||
'memoryUsedGb': memory_used_gb,
|
||||
'memoryTotalGb': memory_total_gb,
|
||||
'diskPercent': disk_percent,
|
||||
'diskUsedGb': disk_used_gb,
|
||||
'diskTotalGb': disk_total_gb,
|
||||
'processCount': process_count,
|
||||
}}))
|
||||
PY
|
||||
"""
|
||||
|
||||
|
||||
def collect_host_metrics(hosts: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
|
||||
# 先把主机名和实例 ID 建立映射，后面 TAT 返回时要按实例回填。
|
||||
host_by_instance = {
|
||||
host["instanceId"]: host
|
||||
for host in hosts
|
||||
if host.get("instanceId")
|
||||
}
|
||||
|
||||
# 去重后的实例 ID 列表。
|
||||
instance_ids = list(host_by_instance.keys())
|
||||
|
||||
# 如果配置里没有 instanceId，直接返回错误占位。
|
||||
if not instance_ids:
|
||||
return {
|
||||
host["host"]: {
|
||||
"ok": False,
|
||||
"error": "missing instanceId in config/targets.json",
|
||||
"updatedAt": utc_now(),
|
||||
}
|
||||
for host in hosts
|
||||
}
|
||||
|
||||
# 初始化 TAT 客户端。
|
||||
client, error = build_tat_client()
|
||||
|
||||
# 没拿到客户端时，给所有主机写统一错误。
|
||||
if client is None:
|
||||
return {
|
||||
host["host"]: {
|
||||
"ok": False,
|
||||
"error": error,
|
||||
"updatedAt": utc_now(),
|
||||
}
|
||||
for host in hosts
|
||||
}
|
||||
|
||||
# 这里延迟导入 SDK model，避免上面客户端初始化失败时多余导入。
|
||||
from tencentcloud.tat.v20201028 import models as tat_models
|
||||
|
||||
# 准备执行远端指标采集脚本。
|
||||
req = tat_models.RunCommandRequest()
|
||||
# 批量下发到所有实例。
|
||||
req.from_json_string(
|
||||
json.dumps(
|
||||
{
|
||||
"CommandName": "hy-app-monitor-host-metrics",
|
||||
"CommandType": "SHELL",
|
||||
"Content": __import__("base64").b64encode(build_metrics_script().encode("utf-8")).decode("ascii"),
|
||||
"InstanceIds": instance_ids,
|
||||
"Timeout": TAT_METRIC_TIMEOUT_SECONDS,
|
||||
"WorkingDirectory": "/root",
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
# 发起 TAT 命令。
|
||||
response = json.loads(invoke_tat(lambda: client.RunCommand(req), "RunCommand").to_json_string())
|
||||
# 拿到本次调用 ID，后面轮询它的执行结果。
|
||||
invocation_id = response["InvocationId"]
|
||||
# 轮询截止时间比命令超时时间稍长一点。
|
||||
deadline = time.time() + TAT_METRIC_TIMEOUT_SECONDS + 60
|
||||
# 记录最终结果，key 是实例 ID。
|
||||
results: dict[str, dict[str, Any]] = {}
|
||||
|
||||
# 开始轮询所有实例的执行结果。
|
||||
while time.time() < deadline:
|
||||
# 轮询间隔不要太短，避免频繁打 API。
|
||||
time.sleep(2)
|
||||
|
||||
# 构造查询任务结果的请求。
|
||||
query = tat_models.DescribeInvocationTasksRequest()
|
||||
# 按 invocation id 过滤当前这次调用的所有实例结果。
|
||||
query.from_json_string(
|
||||
json.dumps(
|
||||
{
|
||||
"HideOutput": False,
|
||||
"Limit": 50,
|
||||
"Filters": [{"Name": "invocation-id", "Values": [invocation_id]}],
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
# 拉取本次调用的实例执行结果。
|
||||
data = json.loads(invoke_tat(lambda: client.DescribeInvocationTasks(query), "DescribeInvocationTasks").to_json_string())
|
||||
# 任务列表可能还没完全生成。
|
||||
tasks = data.get("InvocationTaskSet") or []
|
||||
|
||||
# 没有任务结果时继续等待。
|
||||
if not tasks:
|
||||
continue
|
||||
|
||||
# 遍历当前已返回的所有实例结果。
|
||||
for task in tasks:
|
||||
# 拿到实例 ID。
|
||||
instance_id = task.get("InstanceId", "")
|
||||
# 读取当前实例执行状态。
|
||||
status = task.get("TaskStatus", "")
|
||||
# 只处理终态，未完成的继续等下一轮。
|
||||
if status not in {"SUCCESS", "FAILED", "TIMEOUT", "CANCELLED", "START_FAILED"}:
|
||||
continue
|
||||
|
||||
# 成功时解码输出并尝试解析成 JSON。
|
||||
if status == "SUCCESS":
|
||||
try:
|
||||
# 读取远端标准输出。
|
||||
output_text = decode_task_output(task.get("TaskResult", {}).get("Output", ""))
|
||||
# 解析远端输出的 JSON。
|
||||
payload = json.loads(output_text or "{}")
|
||||
# 补齐统一时间字段。
|
||||
payload["updatedAt"] = utc_now()
|
||||
# 收下当前实例成功结果。
|
||||
results[instance_id] = payload
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 输出格式异常时仍然要保留错误信息。
|
||||
results[instance_id] = {
|
||||
"ok": False,
|
||||
"error": f"metrics parse failed: {exc}",
|
||||
"updatedAt": utc_now(),
|
||||
}
|
||||
continue
|
||||
|
||||
# 失败时解码输出并记录错误。
|
||||
results[instance_id] = {
|
||||
"ok": False,
|
||||
"error": decode_task_output(task.get("TaskResult", {}).get("Output", "")) or status,
|
||||
"updatedAt": utc_now(),
|
||||
}
|
||||
|
||||
# 如果已经收齐所有实例结果，就可以结束轮询。
|
||||
if len(results) >= len(instance_ids):
|
||||
break
|
||||
|
||||
# 按 host name 回填最终主机指标。
|
||||
payload_by_host: dict[str, dict[str, Any]] = {}
|
||||
|
||||
# 对每个主机构造最终输出。
|
||||
for host in hosts:
|
||||
# 读取主机对应实例 ID。
|
||||
instance_id = host.get("instanceId", "")
|
||||
# 读取该实例的结果，没有就补超时错误。
|
||||
metrics = results.get(
|
||||
instance_id,
|
||||
{
|
||||
"ok": False,
|
||||
"error": "metrics timeout or no task result",
|
||||
"updatedAt": utc_now(),
|
||||
},
|
||||
)
|
||||
# 补上实例 ID 便于前端排查。
|
||||
metrics["instanceId"] = instance_id
|
||||
# 以 host name 为 key 返回。
|
||||
payload_by_host[host["host"]] = metrics
|
||||
|
||||
# 返回按 host name 索引的主机指标结果。
|
||||
return payload_by_host
|
||||
|
||||
|
||||
def get_host_metrics(hosts: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
|
||||
# 使用独立缓存，减少页面刷新时反复打 TAT。
|
||||
global _HOST_CACHE_PAYLOAD, _HOST_CACHE_AT
|
||||
|
||||
# 记录当前时间。
|
||||
now = time.time()
|
||||
|
||||
# 在锁内优先读取缓存。
|
||||
with _HOST_CACHE_LOCK:
|
||||
if _HOST_CACHE_PAYLOAD is not None and now - _HOST_CACHE_AT < HOST_METRIC_CACHE_TTL_SECONDS:
|
||||
return _HOST_CACHE_PAYLOAD
|
||||
|
||||
# 缓存失效时重新采集。
|
||||
try:
|
||||
# 真正执行主机指标采集。
|
||||
payload = collect_host_metrics(hosts)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# 指标采集失败时不能拖垮整页，统一降级成错误占位。
|
||||
payload = {
|
||||
host["host"]: {
|
||||
"ok": False,
|
||||
"error": f"metrics collection failed: {exc}",
|
||||
"updatedAt": utc_now(),
|
||||
"instanceId": host.get("instanceId", ""),
|
||||
}
|
||||
for host in hosts
|
||||
}
|
||||
|
||||
# 把最新指标写回缓存。
|
||||
with _HOST_CACHE_LOCK:
|
||||
_HOST_CACHE_PAYLOAD = payload
|
||||
_HOST_CACHE_AT = now
|
||||
|
||||
# 返回最新指标。
|
||||
return payload
|
||||
|
||||
|
||||
def build_host_payload(
|
||||
hosts: list[dict[str, Any]],
|
||||
service_payload: dict[str, Any],
|
||||
metrics_by_host: dict[str, dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
# 把服务结果按 host 分组，便于构造主机视图。
|
||||
services_by_host: dict[str, list[dict[str, Any]]] = {}
|
||||
|
||||
# 遍历所有服务结果。
|
||||
for item in service_payload.get("items") or []:
|
||||
# 创建主机对应列表。
|
||||
services_by_host.setdefault(item["host"], []).append(item)
|
||||
|
||||
# 最终主机视图列表。
|
||||
grouped_hosts: list[dict[str, Any]] = []
|
||||
|
||||
# 按配置顺序输出主机，页面顺序更稳定。
|
||||
for host in hosts:
|
||||
# 读取该主机的服务列表。
|
||||
items = services_by_host.get(host["host"], [])
|
||||
# 统计该主机健康服务数量。
|
||||
ok_count = sum(1 for item in items if item["ok"])
|
||||
# 读取主机指标。
|
||||
metrics = metrics_by_host.get(
|
||||
host["host"],
|
||||
{
|
||||
"ok": False,
|
||||
"error": "metrics unavailable",
|
||||
"updatedAt": utc_now(),
|
||||
"instanceId": host.get("instanceId", ""),
|
||||
},
|
||||
)
|
||||
|
||||
# 构造主机对象。
|
||||
grouped_hosts.append(
|
||||
{
|
||||
"host": host["host"],
|
||||
"ip": host["ip"],
|
||||
"group": host["group"],
|
||||
"instanceId": host["instanceId"],
|
||||
"metrics": metrics,
|
||||
"serviceSummary": {
|
||||
"total": len(items),
|
||||
"ok": ok_count,
|
||||
"down": len(items) - ok_count,
|
||||
},
|
||||
"items": items,
|
||||
}
|
||||
)
|
||||
|
||||
# 返回按主机分组后的完整结构。
|
||||
return grouped_hosts
|
||||
|
||||
|
||||
def build_monitor_payload() -> dict[str, Any]:
|
||||
# 先读取监控配置。
|
||||
hosts = load_hosts()
|
||||
# 获取服务健康结果。
|
||||
service_payload = get_service_payload(hosts)
|
||||
# 获取主机资源指标。
|
||||
metrics_by_host = get_host_metrics(hosts)
|
||||
# 拼出主机维度视图。
|
||||
grouped_hosts = build_host_payload(hosts, service_payload, metrics_by_host)
|
||||
# 统计主机指标成功数量。
|
||||
host_metric_ok = sum(1 for host in grouped_hosts if host["metrics"].get("ok"))
|
||||
|
||||
# 返回给前端的统一结果。
|
||||
return {
|
||||
"updatedAt": utc_now(),
|
||||
"summary": {
|
||||
**service_payload["summary"],
|
||||
"hostMetricOk": host_metric_ok,
|
||||
"hostMetricDown": len(grouped_hosts) - host_metric_ok,
|
||||
},
|
||||
"hosts": grouped_hosts,
|
||||
"items": service_payload["items"],
|
||||
}
|
||||
|
||||
|
||||
def response_json(payload: dict[str, Any] | list[Any]) -> bytes:
|
||||
# 统一把 JSON 编码为 UTF-8 文本响应。
|
||||
return (json.dumps(payload, ensure_ascii=False, indent=2) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
class MonitorHandler(BaseHTTPRequestHandler):
|
||||
# 自定义服务名，便于 curl 时识别当前响应来自哪套服务。
|
||||
server_version = "HyAppMonitor/1.0"
|
||||
|
||||
def log_message(self, format: str, *args) -> None: # noqa: A003
|
||||
def log_message(self, format: str, *args: Any) -> None: # noqa: A003
|
||||
# 关闭默认访问日志，避免轮询接口把日志刷爆。
|
||||
return
|
||||
|
||||
def do_GET(self) -> None: # noqa: N802
|
||||
self._dispatch(send_body=True)
|
||||
# GET 请求需要返回响应头和响应体。
|
||||
self.dispatch(send_body=True)
|
||||
|
||||
def do_HEAD(self) -> None: # noqa: N802
|
||||
self._dispatch(send_body=False)
|
||||
# HEAD 请求只返回响应头，不返回响应体。
|
||||
self.dispatch(send_body=False)
|
||||
|
||||
def _dispatch(self, send_body: bool) -> None:
|
||||
def dispatch(self, send_body: bool) -> None:
|
||||
# 只解析 URL 的 path 部分。
|
||||
parsed = urlparse(self.path)
|
||||
|
||||
# 首页直接返回静态 HTML。
|
||||
if parsed.path in {"/", "/index.html"}:
|
||||
return self.serve_static("index.html", "text/html; charset=utf-8", send_body=send_body)
|
||||
|
||||
# CSS 静态资源。
|
||||
if parsed.path == "/app.css":
|
||||
return self.serve_static("app.css", "text/css; charset=utf-8", send_body=send_body)
|
||||
|
||||
# 前端业务 JS。
|
||||
if parsed.path == "/app.js":
|
||||
return self.serve_static("app.js", "application/javascript; charset=utf-8", send_body=send_body)
|
||||
|
||||
# 本地 Vue 运行时。
|
||||
if parsed.path == "/vue.js":
|
||||
return self.serve_static("vue.js", "application/javascript; charset=utf-8", send_body=send_body)
|
||||
|
||||
# 轻量健康检查接口。
|
||||
if parsed.path == "/api/health":
|
||||
return self.send_payload(
|
||||
HTTPStatus.OK,
|
||||
@ -179,25 +784,36 @@ class MonitorHandler(BaseHTTPRequestHandler):
|
||||
"application/json; charset=utf-8",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
# 主监控接口，返回服务和主机两类信息。
|
||||
if parsed.path == "/api/monitor/services":
|
||||
payload = get_monitor_payload()
|
||||
return self.send_payload(
|
||||
HTTPStatus.OK,
|
||||
response_json(payload),
|
||||
response_json(build_monitor_payload()),
|
||||
"application/json; charset=utf-8",
|
||||
send_body=send_body,
|
||||
)
|
||||
|
||||
# favicon 直接返回空内容即可。
|
||||
if parsed.path == "/favicon.ico":
|
||||
return self.send_payload(HTTPStatus.NO_CONTENT, b"", "image/x-icon", send_body=send_body)
|
||||
|
||||
# 其他路径统一返回 404 JSON。
|
||||
return self.send_error_payload(HTTPStatus.NOT_FOUND, "not found", send_body=send_body)
|
||||
|
||||
def serve_static(self, name: str, content_type: str, send_body: bool) -> None:
|
||||
# 计算静态文件实际路径。
|
||||
path = STATIC_DIR / name
|
||||
|
||||
# 文件不存在时直接返回 JSON 错误。
|
||||
if not path.exists():
|
||||
return self.send_error_payload(HTTPStatus.NOT_FOUND, "static file missing", send_body=send_body)
|
||||
|
||||
# 静态文件存在时直接返回二进制内容。
|
||||
self.send_payload(HTTPStatus.OK, path.read_bytes(), content_type, send_body=send_body)
|
||||
|
||||
def send_error_payload(self, status: HTTPStatus, message: str, send_body: bool) -> None:
|
||||
# 错误响应也走统一 JSON 结构，前端更容易处理。
|
||||
self.send_payload(
|
||||
status,
|
||||
response_json({"status": int(status), "error": message}),
|
||||
@ -206,20 +822,31 @@ class MonitorHandler(BaseHTTPRequestHandler):
|
||||
)
|
||||
|
||||
def send_payload(self, status: HTTPStatus, body: bytes, content_type: str, send_body: bool) -> None:
|
||||
# 写入 HTTP 状态码。
|
||||
self.send_response(status)
|
||||
# 写入内容类型。
|
||||
self.send_header("Content-Type", content_type)
|
||||
# 页面和接口都不缓存，保证看到的是当前状态。
|
||||
self.send_header("Cache-Control", "no-store")
|
||||
# 写入内容长度，便于浏览器正确处理 HEAD 请求。
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
# 结束响应头。
|
||||
self.end_headers()
|
||||
|
||||
# 只有 GET 这类需要响应体的请求才真正写 body。
|
||||
if send_body and body:
|
||||
self.wfile.write(body)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
# 创建线程型 HTTP 服务，允许多个请求并发处理。
|
||||
server = ThreadingHTTPServer((APP_BIND, APP_PORT), MonitorHandler)
|
||||
# 启动时打印监听地址，方便排查。
|
||||
print(f"hy-app-monitor listening on http://{APP_BIND}:{APP_PORT}", flush=True)
|
||||
# 进入永久监听循环。
|
||||
server.serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 作为脚本执行时启动主服务。
|
||||
main()
|
||||
|
||||
@ -41,6 +41,12 @@ body {
|
||||
padding: 28px 18px 48px;
|
||||
}
|
||||
|
||||
.boot-indicator {
|
||||
padding: 20px 24px 0;
|
||||
color: var(--muted);
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.hero {
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1.2fr) minmax(280px, 0.8fr);
|
||||
@ -79,13 +85,6 @@ body {
|
||||
letter-spacing: -0.05em;
|
||||
}
|
||||
|
||||
.hero-text {
|
||||
max-width: 60ch;
|
||||
margin: 14px 0 0;
|
||||
color: var(--muted);
|
||||
line-height: 1.7;
|
||||
}
|
||||
|
||||
.hero-side {
|
||||
display: grid;
|
||||
gap: 12px;
|
||||
@ -120,6 +119,10 @@ body {
|
||||
color: var(--bad);
|
||||
}
|
||||
|
||||
.stat-block.neutral strong {
|
||||
color: var(--accent);
|
||||
}
|
||||
|
||||
.toolbar {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
@ -217,6 +220,50 @@ body {
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.host-metrics {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, minmax(0, 1fr));
|
||||
gap: 12px;
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.metric-card {
|
||||
padding: 14px 16px;
|
||||
border-radius: 18px;
|
||||
border: 1px solid var(--line);
|
||||
background: var(--panel-strong);
|
||||
}
|
||||
|
||||
.metric-card small,
|
||||
.metric-card span {
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
.metric-card small {
|
||||
display: block;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.metric-card strong {
|
||||
display: block;
|
||||
margin-top: 6px;
|
||||
font-size: 1.1rem;
|
||||
letter-spacing: -0.02em;
|
||||
}
|
||||
|
||||
.metric-card span {
|
||||
display: block;
|
||||
margin-top: 6px;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.metric-error {
|
||||
margin: -4px 0 16px;
|
||||
color: var(--bad);
|
||||
font-size: 12px;
|
||||
word-break: break-word;
|
||||
}
|
||||
|
||||
.badge {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
@ -342,6 +389,7 @@ body {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.host-metrics,
|
||||
.service-grid {
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
}
|
||||
@ -360,6 +408,7 @@ body {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.host-metrics,
|
||||
.service-grid,
|
||||
.service-data {
|
||||
grid-template-columns: 1fr;
|
||||
|
||||
221
static/app.js
221
static/app.js
@ -1,113 +1,130 @@
|
||||
const { createApp } = Vue;
|
||||
|
||||
createApp({
|
||||
data() {
|
||||
return {
|
||||
loading: false,
|
||||
error: "",
|
||||
filter: "all",
|
||||
keyword: "",
|
||||
payload: {
|
||||
updatedAt: "",
|
||||
summary: { total: 0, ok: 0, down: 0 },
|
||||
items: [],
|
||||
},
|
||||
timer: null,
|
||||
};
|
||||
},
|
||||
computed: {
|
||||
summary() {
|
||||
return this.payload.summary || { total: 0, ok: 0, down: 0 };
|
||||
data() {
|
||||
return {
|
||||
loading: false,
|
||||
error: "",
|
||||
filter: "all",
|
||||
keyword: "",
|
||||
payload: {
|
||||
updatedAt: "",
|
||||
summary: {
|
||||
total: 0,
|
||||
ok: 0,
|
||||
down: 0,
|
||||
hostTotal: 0,
|
||||
hostMetricOk: 0,
|
||||
hostMetricDown: 0,
|
||||
},
|
||||
formattedUpdatedAt() {
|
||||
if (!this.payload.updatedAt) {
|
||||
return "-";
|
||||
hosts: [],
|
||||
},
|
||||
timer: null,
|
||||
};
|
||||
},
|
||||
computed: {
|
||||
summary() {
|
||||
return this.payload.summary || {
|
||||
total: 0,
|
||||
ok: 0,
|
||||
down: 0,
|
||||
hostTotal: 0,
|
||||
hostMetricOk: 0,
|
||||
hostMetricDown: 0,
|
||||
};
|
||||
},
|
||||
formattedUpdatedAt() {
|
||||
if (!this.payload.updatedAt) {
|
||||
return "-";
|
||||
}
|
||||
const date = new Date(this.payload.updatedAt);
|
||||
return Number.isNaN(date.getTime()) ? this.payload.updatedAt : date.toLocaleString();
|
||||
},
|
||||
groupedHosts() {
|
||||
const keyword = this.keyword.trim().toLowerCase();
|
||||
|
||||
return (this.payload.hosts || [])
|
||||
.map((host) => {
|
||||
const items = (host.items || []).filter((item) => {
|
||||
if (this.filter === "down" && item.ok) {
|
||||
return false;
|
||||
}
|
||||
const date = new Date(this.payload.updatedAt);
|
||||
return Number.isNaN(date.getTime()) ? this.payload.updatedAt : date.toLocaleString();
|
||||
},
|
||||
filteredItems() {
|
||||
const keyword = this.keyword.toLowerCase();
|
||||
return (this.payload.items || []).filter((item) => {
|
||||
if (this.filter === "down" && item.ok) {
|
||||
return false;
|
||||
}
|
||||
if (this.filter === "java" && item.kind !== "java") {
|
||||
return false;
|
||||
}
|
||||
if (this.filter === "golang" && item.kind !== "golang") {
|
||||
return false;
|
||||
}
|
||||
if (!keyword) {
|
||||
return true;
|
||||
}
|
||||
return [item.host, item.ip, item.service, item.kind, item.group]
|
||||
.join(" ")
|
||||
.toLowerCase()
|
||||
.includes(keyword);
|
||||
});
|
||||
},
|
||||
groupedHosts() {
|
||||
const map = new Map();
|
||||
for (const item of this.filteredItems) {
|
||||
if (!map.has(item.host)) {
|
||||
map.set(item.host, {
|
||||
host: item.host,
|
||||
ip: item.ip,
|
||||
group: item.group,
|
||||
groupLabel: this.groupLabel(item.group),
|
||||
downCount: 0,
|
||||
items: [],
|
||||
});
|
||||
}
|
||||
const group = map.get(item.host);
|
||||
group.items.push(item);
|
||||
if (!item.ok) {
|
||||
group.downCount += 1;
|
||||
}
|
||||
if (this.filter === "java" && item.kind !== "java") {
|
||||
return false;
|
||||
}
|
||||
return [...map.values()];
|
||||
},
|
||||
},
|
||||
methods: {
|
||||
groupLabel(group) {
|
||||
return {
|
||||
app: "应用节点",
|
||||
pay: "支付节点",
|
||||
gateway: "网关节点",
|
||||
}[group] || group;
|
||||
},
|
||||
async refresh() {
|
||||
this.loading = true;
|
||||
this.error = "";
|
||||
try {
|
||||
const response = await fetch("/api/monitor/services", { cache: "no-store" });
|
||||
if (!response.ok) {
|
||||
throw new Error(`HTTP ${response.status}`);
|
||||
}
|
||||
this.payload = await response.json();
|
||||
} catch (error) {
|
||||
this.error = error instanceof Error ? error.message : String(error);
|
||||
} finally {
|
||||
this.loading = false;
|
||||
if (this.filter === "golang" && item.kind !== "golang") {
|
||||
return false;
|
||||
}
|
||||
},
|
||||
if (!keyword) {
|
||||
return true;
|
||||
}
|
||||
return [item.host, item.ip, item.service, item.kind, item.group]
|
||||
.join(" ")
|
||||
.toLowerCase()
|
||||
.includes(keyword);
|
||||
});
|
||||
|
||||
return {
|
||||
...host,
|
||||
groupLabel: this.groupLabel(host.group),
|
||||
items,
|
||||
downCount: items.filter((item) => !item.ok).length,
|
||||
};
|
||||
})
|
||||
.filter((host) => host.items.length > 0 || !keyword);
|
||||
},
|
||||
mounted() {
|
||||
const root = document.getElementById("app");
|
||||
const bootIndicator = document.getElementById("boot-indicator");
|
||||
if (root) {
|
||||
root.removeAttribute("v-cloak");
|
||||
}
|
||||
if (bootIndicator) {
|
||||
bootIndicator.hidden = true;
|
||||
}
|
||||
this.refresh();
|
||||
this.timer = window.setInterval(() => this.refresh(), 10000);
|
||||
},
|
||||
methods: {
|
||||
groupLabel(group) {
|
||||
return {
|
||||
app: "应用节点",
|
||||
pay: "支付节点",
|
||||
gateway: "网关节点",
|
||||
}[group] || group;
|
||||
},
|
||||
beforeUnmount() {
|
||||
if (this.timer) {
|
||||
window.clearInterval(this.timer);
|
||||
}
|
||||
formatPercent(value) {
|
||||
if (value == null || Number.isNaN(Number(value))) {
|
||||
return "-";
|
||||
}
|
||||
return `${Number(value).toFixed(2)}%`;
|
||||
},
|
||||
formatGb(used, total) {
|
||||
if (used == null || total == null) {
|
||||
return "-";
|
||||
}
|
||||
return `${Number(used).toFixed(2)} / ${Number(total).toFixed(2)} GB`;
|
||||
},
|
||||
async refresh() {
|
||||
this.loading = true;
|
||||
this.error = "";
|
||||
try {
|
||||
const response = await fetch("/api/monitor/services", { cache: "no-store" });
|
||||
if (!response.ok) {
|
||||
throw new Error(`HTTP ${response.status}`);
|
||||
}
|
||||
this.payload = await response.json();
|
||||
} catch (error) {
|
||||
this.error = error instanceof Error ? error.message : String(error);
|
||||
} finally {
|
||||
this.loading = false;
|
||||
}
|
||||
},
|
||||
},
|
||||
mounted() {
|
||||
const root = document.getElementById("app");
|
||||
const bootIndicator = document.getElementById("boot-indicator");
|
||||
if (root) {
|
||||
root.removeAttribute("v-cloak");
|
||||
}
|
||||
if (bootIndicator) {
|
||||
bootIndicator.hidden = true;
|
||||
}
|
||||
this.refresh();
|
||||
this.timer = window.setInterval(() => this.refresh(), 10000);
|
||||
},
|
||||
beforeUnmount() {
|
||||
if (this.timer) {
|
||||
window.clearInterval(this.timer);
|
||||
}
|
||||
},
|
||||
}).mount("#app");
|
||||
|
||||
@ -11,50 +11,43 @@
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<div id="boot-indicator" class="boot-indicator">
|
||||
<strong>HY App Monitor</strong>
|
||||
<span>正在加载页面资源，请稍候。</span>
|
||||
</div>
|
||||
<div id="boot-indicator" class="boot-indicator">加载中...</div>
|
||||
<noscript>
|
||||
<div class="boot-indicator noscript-indicator">
|
||||
<strong>浏览器已禁用 JavaScript</strong>
|
||||
<span>当前页面依赖脚本渲染，请启用 JavaScript 后再访问。</span>
|
||||
</div>
|
||||
<div class="boot-indicator">请启用 JavaScript</div>
|
||||
</noscript>
|
||||
|
||||
<div id="app" v-cloak class="shell">
|
||||
<header class="hero">
|
||||
<div class="hero-copy">
|
||||
<p class="eyebrow">HY / Deploy Monitor</p>
|
||||
<h1>内网服务实时巡检面板</h1>
|
||||
<p class="hero-text">
|
||||
页面本身不直接请求内网服务。所有探测都由 deploy 机器代发，适合直接通过公网 2026 访问。
|
||||
</p>
|
||||
<p class="eyebrow">HY APP MONITOR</p>
|
||||
<h1>服务与主机监控</h1>
|
||||
</div>
|
||||
<div class="hero-side">
|
||||
<div class="stat-block">
|
||||
<span>总服务数</span>
|
||||
<span>服务总数</span>
|
||||
<strong>{{ summary.total }}</strong>
|
||||
</div>
|
||||
<div class="stat-block ok">
|
||||
<span>正常</span>
|
||||
<span>服务正常</span>
|
||||
<strong>{{ summary.ok }}</strong>
|
||||
</div>
|
||||
<div class="stat-block bad">
|
||||
<span>异常</span>
|
||||
<span>服务异常</span>
|
||||
<strong>{{ summary.down }}</strong>
|
||||
</div>
|
||||
<div class="stat-block neutral">
|
||||
<span>主机数</span>
|
||||
<strong>{{ summary.hostTotal }}</strong>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<section class="toolbar">
|
||||
<div class="toolbar-left">
|
||||
<button :class="['filter-chip', filter === 'all' ? 'active' : '']" @click="filter = 'all'">全部</button>
|
||||
<button :class="['filter-chip', filter === 'down' ? 'active' : '']"
|
||||
@click="filter = 'down'">只看异常</button>
|
||||
<button :class="['filter-chip', filter === 'java' ? 'active' : '']"
|
||||
@click="filter = 'java'">Java</button>
|
||||
<button :class="['filter-chip', filter === 'golang' ? 'active' : '']"
|
||||
@click="filter = 'golang'">Golang</button>
|
||||
<button :class="['filter-chip', filter === 'down' ? 'active' : '']" @click="filter = 'down'">只看异常</button>
|
||||
<button :class="['filter-chip', filter === 'java' ? 'active' : '']" @click="filter = 'java'">Java</button>
|
||||
<button :class="['filter-chip', filter === 'golang' ? 'active' : '']" @click="filter = 'golang'">Golang</button>
|
||||
</div>
|
||||
<div class="toolbar-right">
|
||||
<input v-model.trim="keyword" class="search" placeholder="搜索主机 / 服务 / IP" />
|
||||
@ -64,25 +57,52 @@
|
||||
|
||||
<section class="meta-strip">
|
||||
<span>最后刷新：{{ formattedUpdatedAt }}</span>
|
||||
<span>主机指标正常：{{ summary.hostMetricOk }} / {{ summary.hostTotal }}</span>
|
||||
<span v-if="error" class="bad-text">加载失败：{{ error }}</span>
|
||||
<span v-else>自动刷新：10 秒</span>
|
||||
</section>
|
||||
|
||||
<section class="hosts">
|
||||
<article v-for="group in groupedHosts" :key="group.host" class="host-panel">
|
||||
<header class="host-header">
|
||||
<div>
|
||||
<div class="host-identity">
|
||||
<h2>{{ group.host }}</h2>
|
||||
<p>{{ group.ip }} · {{ group.groupLabel }}</p>
|
||||
</div>
|
||||
<div class="host-badges">
|
||||
<span class="badge neutral">{{ group.items.length }} 项</span>
|
||||
<span class="badge neutral">{{ group.serviceSummary.total }} 项服务</span>
|
||||
<span :class="['badge', group.downCount > 0 ? 'bad' : 'ok']">
|
||||
{{ group.downCount > 0 ? `${group.downCount} 异常` : '全部正常' }}
|
||||
{{ group.downCount > 0 ? `${group.downCount} 异常` : '服务正常' }}
|
||||
</span>
|
||||
<span :class="['badge', group.metrics && group.metrics.ok ? 'ok' : 'bad']">
|
||||
{{ group.metrics && group.metrics.ok ? '主机指标正常' : '主机指标异常' }}
|
||||
</span>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<section class="host-metrics">
|
||||
<div class="metric-card">
|
||||
<small>CPU</small>
|
||||
<strong>{{ formatPercent(group.metrics.cpuPercent) }}</strong>
|
||||
</div>
|
||||
<div class="metric-card">
|
||||
<small>内存</small>
|
||||
<strong>{{ formatPercent(group.metrics.memoryPercent) }}</strong>
|
||||
<span>{{ formatGb(group.metrics.memoryUsedGb, group.metrics.memoryTotalGb) }}</span>
|
||||
</div>
|
||||
<div class="metric-card">
|
||||
<small>磁盘</small>
|
||||
<strong>{{ formatPercent(group.metrics.diskPercent) }}</strong>
|
||||
<span>{{ formatGb(group.metrics.diskUsedGb, group.metrics.diskTotalGb) }}</span>
|
||||
</div>
|
||||
<div class="metric-card">
|
||||
<small>进程数</small>
|
||||
<strong>{{ group.metrics.processCount ?? '-' }}</strong>
|
||||
<span>{{ group.metrics.hostname || group.instanceId }}</span>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<p v-if="group.metrics && group.metrics.error" class="metric-error">{{ group.metrics.error }}</p>
|
||||
|
||||
<div class="service-grid">
|
||||
<div v-for="item in group.items" :key="item.host + '-' + item.service" class="service-card">
|
||||
<div class="service-top">
|
||||
|
||||
16
systemd/hy-app-monitor.service
Normal file
16
systemd/hy-app-monitor.service
Normal file
@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=HY App Monitor
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/hy-app-monitor
|
||||
EnvironmentFile=-/opt/hy-app-monitor/.env
|
||||
ExecStart=/opt/hy-app-monitor/scripts/run_foreground.sh
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Loading…
x
Reference in New Issue
Block a user