test: add test plan and test-features script

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
h z
2026-04-10 08:40:04 +01:00
parent 32dc9a4233
commit 41a49e10b3
2 changed files with 492 additions and 0 deletions

281
TEST-PLAN.md Normal file
View File

@@ -0,0 +1,281 @@
# Dirigent 插件测试计划
> 版本v0.3.x | 测试环境OpenClaw 2026.4.9(待升级)
---
## 测试架构说明
**参与者**
- `main``home-developer``test-ph0`:测试 agent
- `CT-Moderator`:主持人 bot发送/删除调度消息)
- `Proxy Bot`:模拟人类用户,所有测试中需要"人工发消息"的操作均通过 Proxy Bot 完成
**核心机制**
- `before_model_resolve`:轮次门控(非当前 speaker → NO_REPLY
- `agent_end`:推进下一轮次,含 tail-match 轮询确认消息已发送
- `message_received`:外部消息处理(唤醒休眠、中断 tail-match、已结束频道自动回复
---
## 一、前置检查
| # | 检查项 | 期望结果 |
|---|--------|---------|
| 1.1 | OpenClaw 升级到 2026.4.9gateway 正常启动 | 日志无报错 |
| 1.2 | Dirigent 插件 deploy`cp -r plugin/* ~/.openclaw/plugins/dirigent/` | 日志显示 `dirigent: plugin registered (v2)` |
| 1.3 | `dirigent-identity.json` 确认三个 agent 均已注册 | `main``home-developer``test-ph0` 均有 `agentId``discordUserId` 映射 |
| 1.4 | Proxy Bot 在目标 guild 内可见,有发消息权限 | 能在私有频道中发消息 |
---
## 二、Chat 模式报数测试2 个 agent
> **目的:** 验证 2-agent 轮次调度和休眠dormant机制
### 2.1 创建频道并发起报数
**步骤:**
1. 在 Discord 中创建一个新的私有文字频道,将 `main``home-developer` 的 bot 账号及 CT-Moderator 加入
2.`127.0.0.1:18789/dirigent` 控制页面中将该频道设置为 **chat** 模式
3. Proxy Bot 在频道中发送指令:
> 请报数,从 0 开始,每次回复你看到的最后一个数字 +1。超过 5 之后,所有人只能回复 `NO_REPLY`。
**期望(调度行为):**
- Proxy Bot 消息触发 `message_received` → speaker list 初始化
- CT-Moderator 发送 `<@discordUserId>➡️` 后立即删除
- `main` 先发言回复 `1`CT-Moderator 触发 `home-developer`
- `home-developer` 回复 `2`CT-Moderator 再次触发 `main`
- 如此交替,直到某 agent 回复 `6`(超过 5
**期望(休眠行为):**
- 两个 agent 均回复 `NO_REPLY` 后,日志显示 `entered dormant`
- Chat 模式:**不发送**空闲提醒
- 频道无任何后续消息
**期望聊天记录:**
```
Proxy Bot: 请报数,从 0 开始 ...
main: 1
home-developer: 2
main: 3
home-developer: 4
main: 5
home-developer: 6
main: NO_REPLY (静默,无 Discord 消息)
home-developer: NO_REPLY (静默,无 Discord 消息)
← 进入休眠,频道沉默
```
### 2.2 唤醒休眠并验证 Moderator 不触发再次唤醒
**前置:** 频道处于休眠
**步骤:**
1. Proxy Bot 在频道发任意消息
**期望:**
- 日志显示 `woke dormant channel`
- CT-Moderator 发调度消息触发第一个 speaker轮次恢复
- CT-Moderator 自身的调度消息**不触发**二次唤醒(日志无第二条 `woke dormant`
---
## 三、Chat 模式报数测试3 个 agent验证 Shuffle
> **目的:** 验证 3-agent shuffle 模式下轮次顺序在每个 cycle 结束后随机重排
### 3.1 创建频道并发起报数
**步骤:**
1. 在 Discord 中创建一个新的私有文字频道,将 `main``home-developer``test-ph0` 的 bot 账号及 CT-Moderator 加入
2.`127.0.0.1:18789/dirigent` 控制页面中将该频道设置为 **chat** 模式
3. Proxy Bot 在频道中发送指令:
> 请报数,从 0 开始,每次回复你看到的最后一个数字 +1。超过 7 之后,所有人只能回复 `NO_REPLY`。
**期望(调度行为):**
- 三个 agent 依次发言Cycle 1 顺序由初始化决定)
- 每轮 3 条消息为一个 cycle
- **Cycle 1 结束后**,下一个 cycle 的顺序应与上一个不同shuffle 生效)
- Shuffle 约束:上一个 cycle 的最后一个 speaker 不能成为下一个 cycle 的第一个 speaker
**验证 Shuffle**
- 观察 Discord 聊天记录,记录每 3 条消息的发言者顺序
- 至少经历 2 个完整 cycle确认顺序发生变化
- 日志中确认 3-agent 场景走 shuffle 分支
**期望(休眠行为):**
- 超过 7 后三个 agent 均 `NO_REPLY` → 日志显示 `entered dormant`
**期望聊天记录示例顺序仅供参考shuffle 后会不同):**
```
Proxy Bot: 请报数,从 0 开始 ...
[Cycle 1]
main: 1
home-developer: 2
test-ph0: 3
[Cycle 2 — shuffle 后顺序可能变化]
test-ph0: 4
main: 5
home-developer: 6
[Cycle 3]
home-developer: 7
test-ph0: 8 (超过 7
main: NO_REPLY
...
← 进入休眠
```
### 3.2 外部消息中断 tail-match
**步骤:**
1. 在某 agent 完成发言、tail-match 轮询期间(约 0-15s 内)
2. Proxy Bot 立即发一条消息
**期望:**
- 日志显示 `tail-match interrupted`
- 轮次正常推进至下一个 speaker不卡住
---
## 四、Discussion 模式测试(完整生命周期)
> **目的:** 验证 discussion 频道从创建到结束的全流程,包括 callback
### 4.1 创建 Channel AChat并通过 agent 发起讨论
**步骤:**
1. 在 Discord 中创建一个新的私有文字频道(即 **Channel A**),将 `main` 的 bot 账号及 CT-Moderator 加入
2.`127.0.0.1:18789/dirigent` 控制页面中将 Channel A 设置为 **chat** 模式
3. Proxy Bot 在 Channel A 发送指令:
> 请使用 `create-discussion-channel` 工具,邀请 `home-developer` 参与一个讨论,主题自定,讨论结束条件是达成至少 2 条共识。
**期望:**
- `main` 调用 `create-discussion-channel` 工具Discord 中出现新私有 **Discussion 频道**
- `dirigent-channels.json` 新增该频道记录mode=discussionconcluded=falseinitiatorAgentId=maincallbackChannelId=Channel A 的 ID
- CT-Moderator 在 Discussion 频道发送讨论指南discussionGuide
- CT-Moderator 发调度消息触发第一个 speaker
### 4.2 Discussion 轮次正常运转
**期望:**
- `main``home-developer` 在 Discussion 频道内交替发言
- 非当前 speaker 静默(`before_model_resolve` 返回 NO_REPLY
### 4.3 Discussion 休眠 → 空闲提醒
**触发条件:** 两个 agent 在同一 cycle 内均输出 NO_REPLY
**期望discussion 独有行为):**
- 日志显示 `entered dormant`
- CT-Moderator 在 Discussion 频道发送空闲提醒给 **initiator**main
```
<@main的discordUserId> Discussion is idle. Please summarize the results and call `discussion-complete`.
```
- 只发一次
### 4.4 `discussion-complete` 结束讨论 → Callback 验证
**步骤:** `main` 在 Discussion 频道中调用 `discussion-complete` 工具
**期望:**
- `dirigent-channels.json` 中该频道 `concluded` 变为 `true`
- CT-Moderator 在 **Channel A**callbackChannel发送
```
Discussion complete. Summary: /path/...
```
- Discussion 频道不再有任何 agent 发言
### 4.5 已结束 Discussion 频道:外部消息自动回复(单次,无循环)
**步骤:** Proxy Bot 在已结束的 Discussion 频道发一条消息
**期望:**
- CT-Moderator 回复**恰好一次**`This discussion is closed and no longer active.`
- CT-Moderator 自己的这条回复**不触发**新的 "closed" 回复(无限循环修复验证)
- 日志确认senderId 匹配 moderatorBotUserId → 跳过 concluded auto-reply
---
## 五、Report / Work 模式测试
### 5.1 创建 Report 频道
**操作:** 让任意 agent 调用 `create-report-channel`
**期望:**
- 频道创建成功
- Proxy Bot 在该频道发消息后agent 不响应mode=report → NO_REPLY
### 5.2 创建 Work 频道
**操作:** 让任意 agent 调用 `create-work-channel`
**期望:**
- 频道创建成功mode=worklocked
- 无轮次管理agent 自由响应
### 5.3 Locked Mode 不可更改
**操作:** 对 discussion/work 频道调用 `/set-channel-mode`
**期望:**
- 报错:`Channel is in locked mode`
---
## 六、边界条件 & 回归验证
| # | 场景 | 期望 |
|---|------|------|
| 6.1 | Gateway 重启后chat 频道收到 Proxy Bot 消息 | 重新初始化 speaker list轮次正常恢复 |
| 6.2 | Proxy Bot 连续快速发多条消息(压力测试) | blocked-pending 计数不超过 MAX=3不形成死循环 |
| 6.3 | 同一事件被多个 VM 上下文处理 | globalThis dedupBMR WeakSet / agent_end Set / concluded Set确保只执行一次 |
| 6.4 | `fetchVisibleChannelBotAccountIds` 返回空列表 | 不崩溃,日志警告,不发调度消息 |
---
## 七、日志关键词速查
正常流程应出现的日志:
```
dirigent: plugin registered (v2)
dirigent: initialized speaker list channel=... speakers=...
dirigent: before_model_resolve anchor set channel=...
dirigent: triggered next speaker agentId=...
dirigent: agent_end channel=... empty=false
dirigent: entered dormant
dirigent: woke dormant channel=...
dirigent: moderator message sent to channel=...
```
异常(需关注):
```
dirigent: tail-match timeout ← 15s 内消息未落地
dirigent: agent_end skipping stale ← 正常stale NO_REPLY 被过滤)
dirigent: before_model_resolve init in progress ← 并发初始化保护(正常)
```
---
## 八、测试顺序建议
```
前置检查 (§1)
2-agent 报数:轮次 + 休眠 + 唤醒 (§2)
3-agent 报数shuffle 验证 + tail-match 中断 (§3)
Discussion 完整生命周期:创建 → 轮次 → 空闲提醒 → 结束 → callback → 防循环 (§4)
Report/Work 频道 (§5)
边界条件 (§6)
```
---
*测试中如遇 agent 卡住超过 10 分钟,重启 gateway 后继续。偶发的 5-10 分钟响应延迟属正常Kimi 模型特性)。*

211
scripts/test-features.mjs Normal file
View File

@@ -0,0 +1,211 @@
#!/usr/bin/env node
/**
* Dirigent feature test script
* Tests: no-reply gate, end-symbol enforcement, turn management
*
* Usage:
* node scripts/test-features.mjs [channelId]
*
* Env:
* PROXY_TOKEN - path to PROXY_BOT_TOKEN file (default: ./PROXY_BOT_TOKEN)
* GUILD_ID - guild id (default: 1480860737902743686)
*
* Reads token from PROXY_BOT_TOKEN file.
*/
import fs from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const TOKEN_FILE = process.env.PROXY_TOKEN || path.resolve(__dirname, "../PROXY_BOT_TOKEN");
const GUILD_ID = process.env.GUILD_ID || "1480860737902743686";
const C = {
reset: "\x1b[0m", red: "\x1b[31m", green: "\x1b[32m",
yellow: "\x1b[33m", blue: "\x1b[34m", cyan: "\x1b[36m", bold: "\x1b[1m",
};
const c = (t, col) => `${C[col] || ""}${t}${C.reset}`;
const TOKEN = fs.readFileSync(TOKEN_FILE, "utf8").trim().split(/\s/)[0];
async function discord(method, path_, body) {
const r = await fetch(`https://discord.com/api/v10${path_}`, {
method,
headers: {
Authorization: `Bot ${TOKEN}`,
"Content-Type": "application/json",
},
body: body !== undefined ? JSON.stringify(body) : undefined,
});
const text = await r.text();
let json = null;
try { json = JSON.parse(text); } catch { json = { raw: text }; }
return { ok: r.ok, status: r.status, json };
}
async function sendMessage(channelId, content) {
const r = await discord("POST", `/channels/${channelId}/messages`, { content });
if (!r.ok) throw new Error(`send failed ${r.status}: ${JSON.stringify(r.json)}`);
return r.json;
}
async function getMessages(channelId, limit = 10) {
const r = await discord("GET", `/channels/${channelId}/messages?limit=${limit}`);
if (!r.ok) throw new Error(`fetch messages failed ${r.status}`);
return r.json; // newest first
}
async function sleep(ms) {
return new Promise(r => setTimeout(r, ms));
}
// Wait for agent responses: poll until we see `expectedCount` new messages from bots
async function waitForBotMessages(channelId, afterMsgId, expectedCount = 1, timeoutMs = 15000) {
const start = Date.now();
while (Date.now() - start < timeoutMs) {
await sleep(1500);
const msgs = await getMessages(channelId, 20);
const newBotMsgs = msgs.filter(m =>
BigInt(m.id) > BigInt(afterMsgId) &&
m.author?.bot === true &&
m.author?.id !== "1481189346097758298" // exclude our proxy bot
);
if (newBotMsgs.length >= expectedCount) return newBotMsgs;
}
return [];
}
function printMsg(m) {
const who = `${m.author?.username}(${m.author?.id})`;
const preview = (m.content || "").slice(0, 120).replace(/\n/g, "\\n");
console.log(` ${c(who, "cyan")}: ${preview}`);
}
// ─────────────────────────────────────────────────────────
// Test helpers
// ─────────────────────────────────────────────────────────
let passed = 0, failed = 0;
function check(label, cond, detail = "") {
if (cond) {
console.log(` ${c("✓", "green")} ${label}`);
passed++;
} else {
console.log(` ${c("✗", "red")} ${label}${detail ? `${detail}` : ""}`);
failed++;
}
}
// ─────────────────────────────────────────────────────────
// Main tests
// ─────────────────────────────────────────────────────────
async function main() {
// Resolve channel
let channelId = process.argv[2];
if (!channelId) {
// Create a fresh test channel
console.log(c("\n[setup] Creating private test channel...", "blue"));
const meR = await discord("GET", "/users/@me");
if (!meR.ok) { console.error("Cannot auth:", meR.json); process.exit(1); }
console.log(` proxy bot: ${meR.json.username} (${meR.json.id})`);
// Get guild roles to find @everyone
const guildR = await discord("GET", `/guilds/${GUILD_ID}`);
const guildEveryoneId = guildR.json?.id || GUILD_ID;
// Get guild members to find agent bots
const membersR = await discord("GET", `/guilds/${GUILD_ID}/members?limit=50`);
const bots = (membersR.json || [])
.filter(m => m.user?.bot && m.user?.id !== meR.json.id)
.map(m => m.user);
console.log(` agent bots in guild: ${bots.map(b => `${b.username}(${b.id})`).join(", ")}`);
const allowedUserIds = [meR.json.id, ...bots.map(b => b.id)];
const overwrites = [
{ id: guildEveryoneId, type: 0, allow: "0", deny: "1024" },
...allowedUserIds.map(id => ({ id, type: 1, allow: "1024", deny: "0" })),
];
const chR = await discord("POST", `/guilds/${GUILD_ID}/channels`, {
name: `dirigent-test-${Date.now().toString(36)}`,
type: 0,
permission_overwrites: overwrites,
});
if (!chR.ok) { console.error("Cannot create channel:", chR.json); process.exit(1); }
channelId = chR.json.id;
console.log(` created channel: #${chR.json.name} (${channelId})`);
} else {
console.log(c(`\n[setup] Using channel ${channelId}`, "blue"));
}
await sleep(1000);
// ─────────────────────────────────────────────────────────
// Test 1: Human sends message → agents should respond with end-symbol
// ─────────────────────────────────────────────────────────
console.log(c("\n[Test 1] Human message → agent response must end with 🔚", "bold"));
const msg1 = await sendMessage(channelId, "Hello from human proxy! Please introduce yourself briefly. 🔚");
console.log(` sent: "${msg1.content}"`);
console.log(" waiting up to 20s for bot responses...");
const botMsgs1 = await waitForBotMessages(channelId, msg1.id, 1, 20000);
if (botMsgs1.length === 0) {
check("Agent responded", false, "no bot messages received within 20s");
} else {
for (const m of botMsgs1) printMsg(m);
check(
"Agent response ends with 🔚",
botMsgs1.some(m => m.content?.trim().endsWith("🔚")),
`got: ${botMsgs1.map(m => m.content?.slice(-10)).join(" | ")}`
);
}
// ─────────────────────────────────────────────────────────
// Test 2: Turn order — only one agent per round
// After first agent replies, second agent should be next
// ─────────────────────────────────────────────────────────
console.log(c("\n[Test 2] Turn order — check /dirigent turn-status", "bold"));
console.log(" (Observational — check Discord channel for /dirigent turn-status output)");
console.log(c(" → Manually run /dirigent turn-status in the test channel to verify", "yellow"));
// ─────────────────────────────────────────────────────────
// Test 3: Bot message without end-symbol → no-reply gate
// We send a message that looks like a bot (not in humanList) — observe logs
// ─────────────────────────────────────────────────────────
console.log(c("\n[Test 3] Second round — agent should reply after human follow-up", "bold"));
await sleep(3000);
const msg3 = await sendMessage(channelId, "What is 2+2? Answer briefly. 🔚");
console.log(` sent: "${msg3.content}"`);
const botMsgs3 = await waitForBotMessages(channelId, msg3.id, 1, 20000);
if (botMsgs3.length === 0) {
check("Agent responded to follow-up", false, "no response within 20s");
} else {
for (const m of botMsgs3) printMsg(m);
check(
"Follow-up response ends with 🔚",
botMsgs3.some(m => m.content?.trim().endsWith("🔚")),
);
}
// ─────────────────────────────────────────────────────────
// Test 4: NO_REPLY behavior — ask something irrelevant to trigger NO_REPLY
// ─────────────────────────────────────────────────────────
console.log(c("\n[Test 4] NO_REPLY — agents with nothing to say should be silent", "bold"));
console.log(" (This is hard to assert automatically — check gateway logs for NO_REPLY routing)");
console.log(c(" → Watch `openclaw logs` for 'dirigent: before_model_resolve blocking out-of-turn'", "yellow"));
// ─────────────────────────────────────────────────────────
// Summary
// ─────────────────────────────────────────────────────────
console.log(c(`\n─────────────────────────────────────────────`, "blue"));
console.log(`Results: ${c(String(passed), "green")} passed, ${c(String(failed), "red")} failed`);
console.log(`Channel: https://discord.com/channels/${GUILD_ID}/${channelId}`);
console.log(c("─────────────────────────────────────────────\n", "blue"));
}
main().catch(e => { console.error(e); process.exit(1); });