199 lines
10 KiB
Markdown
199 lines
10 KiB
Markdown
# Room Outbox Worker
|
||
|
||
本文档定义当前 `room-service` outbox 的实际语义。房间写命令只负责提交 Room Cell 状态、command log、snapshot 和 `room_outbox`；腾讯云 IM、activity-service、notice-service、后续 push/inbox 等外部副作用都通过 outbox worker 或 RocketMQ 下游 consumer 异步完成。
|
||
|
||
## 核心结论
|
||
|
||
| 项目 | 当前规则 |
|
||
| --- | --- |
|
||
| 状态 owner | Room Cell 和 MySQL snapshot/command log 是房间状态权威来源 |
|
||
| 持久事实 | `room_outbox` 是房间外事件的持久事实源，和房间命令提交链路一起落 MySQL |
|
||
| 发布模式 | `outbox_worker.publish_mode` 支持 `direct`、`mq`、`dual` |
|
||
| MQ 职责 | RocketMQ 只做事件分发和宝箱延迟唤醒，不保存房间核心状态 |
|
||
| 下游语义 | 下游服务按 `event_id` 幂等消费，失败由自己的位点、重试和死信处理 |
|
||
| 主链路 | HTTP/gRPC 请求不等待 IM REST、activity、notice、push 或 inbox 投递完成 |
|
||
|
||
不做的事：
|
||
|
||
| 不做 | 原因 |
|
||
| --- | --- |
|
||
| 在 Room Cell 中保存投递状态 | 投递状态属于 outbox 或下游消费位点，不属于房间权威状态 |
|
||
| 用 Redis 替代 outbox | Redis 只保存热状态和短 TTL lease，不能作为恢复事实来源 |
|
||
| 用 RocketMQ 替代 `room_outbox` | 房间状态提交成功但 MQ 发布失败时，必须能从 MySQL 补偿发布 |
|
||
| 追求精确一次投递 | 外部系统和进程崩溃窗口只能做到 at-least-once，消费者必须按 `event_id` 去重 |
|
||
|
||
## 写命令链路
|
||
|
||
```text
|
||
gateway -> room-service command
|
||
-> command_id idempotency check
|
||
-> ensure Redis lease and Room Cell
|
||
-> mutate cloned RoomState
|
||
-> optional wallet debit before room gift state is committed
|
||
-> SaveMutation(command log + room_outbox + room meta)
|
||
-> snapshot when version hits snapshot_every_n
|
||
-> update list/presence projections best effort
|
||
-> return response
|
||
|
||
outbox worker
|
||
-> claim pending/retryable rows
|
||
-> if RoomTreasureCountdownStarted, schedule treasure open delayed wakeup when configured
|
||
-> publish by mode:
|
||
direct: Tencent IM publisher + activity gRPC publisher
|
||
mq: RocketMQ room_outbox publisher
|
||
dual: direct publishers + RocketMQ publisher
|
||
-> delivered / retryable / failed
|
||
```
|
||
|
||
主请求成功只表示房间状态已经提交；不表示腾讯云 IM、activity-service、notice-service 或任何后续通知已经投递完成。客户端实时表现应以请求响应、房间 snapshot 和腾讯云 IM 系统消息共同补偿。
|
||
|
||
## 发布模式
|
||
|
||
| `publish_mode` | 行为 | 适用场景 |
|
||
| --- | --- | --- |
|
||
| `direct` | worker 直接调用腾讯云 IM publisher 和 activity gRPC publisher；所有 direct publisher 成功后才标记 `delivered` | 本地开发、未接 MQ 的隔离测试 |
|
||
| `mq` | worker 只把 `room_outbox_event` 发布到 `hyapp_room_outbox`；RocketMQ 接收成功后标记 `delivered` | 线上推荐模式，activity、notice、IM bridge 各自 consumer group 解耦消费 |
|
||
| `dual` | 同时 direct 投递和 MQ 发布，全部成功后标记 `delivered` | 迁移或灰度期；要求所有消费端都能按 `event_id` 去重 |
|
||
|
||
在 `mq` 模式下，`room_outbox.status=delivered` 的含义是“事件已经可靠进入 RocketMQ”，不是“所有下游都已处理成功”。下游的成功、重试和死信由各自服务的消费位点负责，不能反写 `room_outbox.status`。
|
||
|
||
## 数据状态
|
||
|
||
`room_outbox.status` 使用：
|
||
|
||
| 状态 | 含义 |
|
||
| --- | --- |
|
||
| `pending` | 已落库，尚未被 worker 处理 |
|
||
| `delivering` | 已被 worker 抢占，`lock_until_ms` 前其他 worker 不抢 |
|
||
| `delivered` | 当前发布模式要求的 publisher 已处理成功；MQ 模式下表示 RocketMQ 已接收 |
|
||
| `retryable` | 上次发布失败，等待 `next_retry_at_ms` 后重试 |
|
||
| `failed` | 达到最大重试次数，进入死信，需要后台工具查看或重放 |
|
||
|
||
旧数据里可能存在历史 `published` 状态。当前 worker 不再写这个状态，新链路只写上表状态。
|
||
|
||
## RocketMQ Topics
|
||
|
||
| Topic | Producer | Consumer Group | 语义 |
|
||
| --- | --- | --- | --- |
|
||
| `hyapp_room_outbox` | `room-service` outbox worker | `hyapp-activity-room-outbox` | activity-service 消费房间事实，做活动、播报、系统消息等后续处理 |
|
||
| `hyapp_room_outbox` | `room-service` outbox worker | `hyapp-notice-room-outbox` | notice-service 消费私有通知相关事实，例如 `RoomUserKicked` |
|
||
| `hyapp_room_outbox` | `room-service` outbox worker | `hyapp-room-im-bridge` | room-service IM bridge 消费房间群系统消息和群成员控制事实 |
|
||
| `hyapp_room_treasure_open` | `room-service` treasure scheduler | `hyapp-room-treasure-open` | 宝箱倒计时到点唤醒 Room Cell 开箱命令 |
|
||
|
||
`hyapp_room_treasure_open` 不是业务状态源。消息只携带 `app_code、room_id、box_id、level、open_at_ms、command_id` 等唤醒参数；真正能否开箱仍由 room-service 加载 Room Cell 后校验当前宝箱状态、等级、`box_id`、`open_at_ms` 和 UTC 重置边界。
|
||
|
||
## 配置
|
||
|
||
`services/room-service/configs/*.yaml` 需要同时维护本地、Docker 和腾讯云示例配置：
|
||
|
||
```yaml
|
||
rocketmq:
|
||
enabled: false
|
||
name_servers: []
|
||
name_server_domain: ""
|
||
access_key: ""
|
||
secret_key: ""
|
||
namespace: ""
|
||
send_timeout: "3s"
|
||
retry: 2
|
||
room_outbox:
|
||
enabled: false
|
||
topic: "hyapp_room_outbox"
|
||
producer_group: "hyapp-room-outbox-producer"
|
||
tencent_im_consumer_enabled: false
|
||
tencent_im_consumer_group: "hyapp-room-im-bridge"
|
||
consumer_max_reconsume_times: 16
|
||
treasure_open:
|
||
enabled: false
|
||
topic: "hyapp_room_treasure_open"
|
||
producer_group: "hyapp-room-treasure-open-producer"
|
||
consumer_group: "hyapp-room-treasure-open"
|
||
consumer_max_reconsume_times: 16
|
||
|
||
outbox_worker:
|
||
enabled: true
|
||
publish_mode: "direct"
|
||
poll_interval: "1s"
|
||
batch_size: 100
|
||
publish_timeout: "3s"
|
||
retry_strategy: "exponential_backoff"
|
||
max_retry_count: 10
|
||
initial_backoff: "5s"
|
||
max_backoff: "5m"
|
||
```
|
||
|
||
配置规则：
|
||
|
||
| 字段 | 规则 |
|
||
| --- | --- |
|
||
| `rocketmq.enabled` | 控制 RocketMQ client 是否启用；本地默认关闭，线上示例开启 |
|
||
| `rocketmq.room_outbox.enabled` | 控制 room outbox 是否可发布到 MQ；`publish_mode=mq/dual` 时必须开启 |
|
||
| `rocketmq.room_outbox.tencent_im_consumer_enabled` | 控制 room-service 是否作为 IM bridge consumer 消费 `hyapp_room_outbox` |
|
||
| `rocketmq.treasure_open.enabled` | 控制宝箱倒计时是否发布和消费延迟唤醒消息 |
|
||
| `outbox_worker.publish_mode` | 本地默认 `direct`，线上建议 `mq`，迁移期可用 `dual` |
|
||
| `poll_interval` | worker 空轮询间隔，必须大于 0，避免 busy loop |
|
||
| `batch_size` | 每轮最大抢占条数，必须大于 0，避免全表扫描 |
|
||
| `publish_timeout` | 单条 outbox 发布 deadline，避免外部系统拖死 worker |
|
||
| `max_retry_count` | 达到次数后标记 `failed`，防止 poison event 永久刷日志 |
|
||
|
||
## 投递对象
|
||
|
||
| Publisher / Consumer | 行为 |
|
||
| --- | --- |
|
||
| RocketMQ room outbox publisher | 把 protobuf outbox 包装成统一 `room_outbox_event`，发布到 `hyapp_room_outbox` |
|
||
| Tencent IM direct publisher | `RoomCreated` 对已同步创建的房间群执行幂等 EnsureGroup；房间系统事件发送腾讯云 IM 群自定义消息 |
|
||
| Activity direct publisher | 本地或未接 MQ 时直接调用 activity-service room event 消费入口 |
|
||
| Treasure open scheduler | 在 `RoomTreasureCountdownStarted` 后发布到点延迟消息，唤醒 room-service 开箱 |
|
||
| Activity MQ consumer | 独立 consumer group 消费 `hyapp_room_outbox`，按 `event_id` 幂等推进活动/播报逻辑 |
|
||
| Notice MQ consumer | 独立 consumer group 消费 `hyapp_room_outbox`，写 `notice_delivery_events` 后投 C2C 私有通知 |
|
||
| Room IM bridge MQ consumer | 独立 consumer group 消费 `hyapp_room_outbox`，投递房间群系统消息和群成员控制 |
|
||
|
||
同一个事件允许被多个 consumer group 消费。任何 consumer 失败都不能阻塞 Room Cell 主链路，也不能影响其它 consumer group 的进度。
|
||
|
||
## 重试和死信
|
||
|
||
发布失败处理：
|
||
|
||
```text
|
||
publish failed
|
||
-> retry_count + 1
|
||
-> if retry_count >= max_retry_count: status=failed
|
||
-> else: status=retryable, next_retry_at_ms=now+backoff
|
||
```
|
||
|
||
`failed` 不再自动重试。后台应提供人工查看和重放工具，重放时必须保留原 `event_id` 或生成明确的新修复事件，避免消费端无法去重。
|
||
|
||
RocketMQ consumer 失败时不修改 `room_outbox`。consumer 应依赖 RocketMQ 重投、自己的幂等表和死信处理；对于已经执行业务副作用但 ACK 失败的窗口，下一次消费必须按 `event_id` 安全去重。
|
||
|
||
## 阶段耗时日志
|
||
|
||
每条房间写命令都会输出 `room_command_timing`：
|
||
|
||
| 字段 | 含义 |
|
||
| --- | --- |
|
||
| `idempotency_ms` | command log 幂等检查耗时 |
|
||
| `lease_ms` | Redis lease / Room Cell 装载耗时 |
|
||
| `wallet_debit_ms` | 送礼扣费耗时；非送礼为 0 |
|
||
| `save_mutation_ms` | command log + outbox + 房间 meta 提交耗时 |
|
||
| `snapshot_ms` | 快照持久化耗时；未触发为 0 |
|
||
| `projection_ms` | 列表和 presence 投影耗时 |
|
||
| `activity_ms` | 主链路 activity 同步耗时；当前固定为 0 |
|
||
| `tencent_im_ms` | 主链路腾讯 IM 同步耗时；CreateRoom 记录同步建群耗时，其余命令当前为 0 |
|
||
| `activity_async_outbox` | 本命令是否产生异步 activity/outbox 投递 |
|
||
| `tencent_im_async_outbox` | 本命令是否产生异步 IM/outbox 投递 |
|
||
|
||
排查接口卡顿时先看 `total_ms`，再看各阶段字段。送礼慢通常优先看 `wallet_debit_ms`，普通房间命令慢优先看 `save_mutation_ms`、`snapshot_ms` 和 `projection_ms`。
|
||
|
||
## 验收
|
||
|
||
| 场景 | 验收 |
|
||
| --- | --- |
|
||
| IM 不可用，`direct/dual` | 房间写命令仍成功，outbox 进入 `retryable` 或最终 `failed` |
|
||
| activity 不可用，`direct/dual` | 房间写命令仍成功，outbox 进入重试 |
|
||
| MQ 不可用，`mq/dual` | 房间写命令仍成功，outbox 进入重试；MQ 恢复后补发 |
|
||
| activity consumer 不可用，`mq` | `room_outbox` 不回退；RocketMQ 和 activity 消费位点负责积压与重试 |
|
||
| poison event | 达到 `max_retry_count` 后进入 `failed`，不再每秒刷日志 |
|
||
| 宝箱倒计时 | `RoomTreasureCountdownStarted` 发布延迟唤醒；到点后 room-service 重新校验状态再开箱 |
|
||
| 正常投递 | 同一 `event_id` 可被多个 consumer group 独立消费且重复投递不产生重复副作用 |
|
||
| 真实链路 | create/join/mic/sendGift/leave 均有 `room_command_timing` |
|