人工智能AI AgentAgent 框架后端多智能体RAG工具调用Agent 记忆【免费下载链接】voltagentAI Agent Engineering Platform built on an Open Source TypeScript AI Agent Framework项目地址https://gitcode.com/gh_mirrors/vo/voltagent点击查看免费下载在 VoltAgent 的评测体系中,自定义评分器(custom scorer)是连接评估标准与Agent 实际输出的核心机制。本文基于仓库文档 building-custom-scorers 展开,系统讲解buildScorer提供的 Prepare → Analyze → Score → Reason 四步流水线、三类评分器形态(启发式、LLM 判别、混合)、在离线实验与 Agent 实时评估中的接入方式,以及参数化、weightedBlend加权融合等高级模式;并结合 评分器构建器源码 与 底层管道实现,还原每一步的上下文结构、错误处理与采样跳过逻辑,帮助你从会抄示例进阶到能读懂执行细节。一、何时需要自定义评分器VoltAgent 已提供一批预置评分器(详见 prebuilt-scorers),但以下场景必须自己实现:内置评分器与你的评估标准不匹配;需要领域特定的评估逻辑(如客服语气合规、代码答案可编译性);想把多种评估方法组合成一个复合指标;需要自定义阈值或打分刻度(例如 0–1 之外的 0–10 分制再归一化)。判断依据很简单:评分器的职责是给定 payload,输出 0~1 的分数及解释,只要这个映射无法用现成评分器表达,就适合走buildScorer流水线。二、四步评分器流水线:概念与源码级执行顺序buildScorer返回一个链式构建器(Builder),依次注册四个可选步骤,score为必填。文档给出的流水线如下:Input Payload ↓ ┌─────────────┐ │ Prepare │ → Transform validate input └─────────────┘ ↓ ┌─────────────┐ │ Analyze │ → Extract features insights └─────────────┘ ↓ ┌─────────────┐ │ Score │ → Calculate numeric score (0-1) └─────────────┘ ↓ ┌─────────────┐ │ Reason │ → Generate explanation └─────────────┘ ↓ Final Result从源码结构看,这条流水线由两层实现协作完成:构建层(builder.ts):ScorerBuilderImpl以私有状态保存已注册的prepare/analyze/score/reason步骤。build()时若缺少score步骤会直接抛出Scorer ... is missing a required score step.错误,并且每次重新注册步骤都会使缓存的定义失效,保证构建结果与最终注册集合一致;执行层(create-scorer.ts 中的createScorer):构建器把用户步骤适配为底层的preprocess → analyze → generateScore → generateReason顺序调用。每步输出会被写入results记录,score结果经过normalizeGenerateScore归一化(只接受有限数字,否则置为null),reason若返回字符串对象则同时合并其metadata;整条管道用try/catch包裹,任一步抛错时返回{ status: error, score, metadata, error }而不是让整个评估崩溃,错误对象上挂的metadata也会被合并进结果。这套设计解释了文档中各步骤可访问payload/params/results的说法:构建器传给每一步的上下文是 builder.ts 中定义的BuilderPrepareContext等类型,统一携带:上下文字段含义说明payload原始输入数据泛型Payload约束,可自定义接口获得类型提示params本次评估参数支持静态对象或由payload动态派生的函数,见第五节results前序步骤输出快照包含prepare、analyze、score、reason、raw(调试用原始结果)score(仅 reason 步)已算出的分数reason步骤额外获得当前score值每个步骤都支持同步或异步实现(返回unknown | Promiseunknown),这为 LLM 类异步评估留出了空间。2.1 Step 1:Prepare(可选)在评分前转换或校验输入 payload,典型工作是清洗文本、解析类型、设置默认值:.prepare(({ payload }) { // Clean and validate inputs const text String(payload.output || ).trim(); const minWords Number(payload.minWords || 5); return { text, minWords }; })该步返回值会被存为results.prepare,供后续步骤引用。2.2 Step 2:Analyze(可选)对已准备的数据做特征提取或更重的分析(包括调用外部 LLM):.analyze(({ prepared }) { // Extract features from prepared data const wordCount prepared.text.split(/\s/).length; const hasMinWords wordCount prepared.minWords; return { wordCount, hasMinWords }; })注意各步骤函数接收的是解构上下文;若想在analyze中引用 prepare 的输出,标准写法是从results.prepare读取(见下文完整示例)。2.3 Step 3:Score(必填)基于前面结果计算 0.0~1.0 的分数,并可返回metadata携带诊断信息:.score(({ payload, prepared, analysis }) { // Calculate score (0.0 to 1.0) const score analysis.hasMinWords ? 1.0 : 0.0; return { score, metadata: { wordCount: analysis.wordCount } }; })从 create-scorer.ts 的GenerateScoreResult类型看,score步既可以返回纯数字,也可以返回{ score, metadata? }对象——对象形式是附带元数据的推荐写法,metadata会合并进最终结果。2.4 Step 4:Reason(可选)为分数生成人类可读的解释,输出字符串即可:.reason(({ payload, score, metadata }) { // Provide explanation const passed score 0.5; return passed ? Output meets minimum word requirement (${metadata.wordCount} words) : Output too short (${metadata.wordCount} words, need ${payload.minWords}); })reason步骤的上下文额外带有score字段(见 builder.ts 的BuilderReasonContext),因此解释文案可以直接依据分数分支。三、完整示例:情感倾向评分器下面构建一个评估回复是否维持了目标情感倾向的评分器,完整覆盖四步:import { buildScorer } from voltagent/core; const sentimentScorer buildScorer({ id: sentiment-analyzer, label: Sentiment Analyzer, description: Evaluates response sentiment and positivity, }) .prepare(({ payload }) { // Step 1: Clean and prepare the text const text String(payload.output || ) .toLowerCase() .trim(); const targetSentiment String(payload.targetSentiment || positive); return { text, targetSentiment }; }) .analyze(({ results }) { // Step 2: Analyze sentiment indicators const prepared results.prepare as { text: string; targetSentiment: string }; const positiveWords [great, excellent, happy, wonderful, fantastic]; const negativeWords [bad, terrible, awful, horrible, poor]; const positiveCount positiveWords.filter((word) prepared.text.includes(word)).length; const negativeCount negativeWords.filter((word) prepared.text.includes(word)).length; const sentiment positiveCount negativeCount ? positive : negativeCount positiveCount ? negative : neutral; return { sentiment, positiveCount, negativeCount, matchesTarget: sentiment prepared.targetSentiment, }; }) .score(({ results }) { // Step 3: Calculate score based on sentiment match const analysis results.analyze as { sentiment: string; positiveCount: number; negativeCount: number; matchesTarget: boolean; }; const score analysis.matchesTarget ? 1.0 : 0.0; return { score, metadata: { detectedSentiment: analysis.sentiment, positiveWords: analysis.positiveCount, negativeWords: analysis.negativeCount, }, }; }) .reason(({ score, results }) { // Step 4: Explain the scoring decision const prepared results.prepare as { text: string; targetSentiment: string }; const metadata results.raw as any; if (score 1.0) { return ( Sentiment matches target (${prepared.targetSentiment}). Found ${metadata.positiveWords} positive and ${metadata.negativeWords} negative indicators. ); } return ( Sentiment mismatch. Expected ${prepared.targetSentiment} but detected ${metadata.detectedSentiment}. Found ${metadata.positiveWords} positive and ${metadata.negativeWords} negative indicators. ); }) .build();构建选项除必填的id外,还支持label、description、metadata、sampling与params(见 builder.ts 的BuildScorerCustomOptions),其中sampling用于控制该评分器的采样执行策略。运行结果示例输入 1:正面回复await sentimentScorer.run({ payload: { output: This is a fantastic solution! Great work on the implementation., targetSentiment: positive }, params: {} }); // Result: { score: 1.0, metadata: { detectedSentiment: positive, positiveWords: 2, negativeWords: 0 }, reason: Sentiment matches target (positive). Found 2 positive and 0 negative indicators. }输入 2:情感不匹配await sentimentScorer.run({ payload: { output: This approach seems problematic and could cause terrible issues., targetSentiment: positive }, params: {} }); // Result: { score: 0.0, metadata: { detectedSentiment: negative, positiveWords: 0, negativeWords: 1 }, reason: Sentiment mismatch. Expected positive but detected negative. Found 0 positive and 1 negative indicators. }从 builder.ts 的BuildScorerRunResult类型看,run()实际返回的结构比上面示例更丰富:除score、reason、metadata外,还有id、status(success | error | skipped三态)、durationMs(本次执行耗时)、sampling(采样元数据)与steps(各步骤输出快照)。steps快照让你可以在调试时直接查看 prepare/analyze 的中间产物,这正是文档中results.raw用于调试的底层来源。四、三类评分器形态4.1 启发式评分器(Heuristic)纯规则、零外部依赖,执行成本最低,适合长度、格式、关键词这类确定性检查:const lengthScorer buildScorer({ id: length-check, label: Length Validator, }) .score(({ payload }) { const length String(payload.output || ).length; const maxLength Number(payload.maxLength || 100); return { score: length maxLength ? 1.0 : 0.0, metadata: { length, maxLength }, }; }) .build();注意这里只注册了必填的score步——四步中任意一步都可省略,prepare/analyze/reason均为可选。4.2 LLM 判别评分器(LLM-Based)把重量级的语言模型调用放进analyze步(该步原生支持 async),用结构化输出约束打分尺度:import { Agent } from voltagent/core; import { openai } from ai-sdk/openai; import { z } from zod; const QUALITY_SCHEMA z.object({ score: z.number().min(0).max(10), reason: z.string(), }); const qualityScorer buildScorer({ id: quality-check, label: Response Quality, }) .analyze(async ({ payload }) { const agent new Agent({ name: quality-evaluator, model: openai(gpt-4o-mini), instructions: You evaluate response quality on a scale of 0-10, }); const prompt Rate the quality of this response: ${payload.output}; const result await agent.generateObject(prompt, QUALITY_SCHEMA); return result.object; }) .score(({ results }) { const analysis results.analyze as z.infertypeof QUALITY_SCHEMA; return { score: analysis.score / 10, metadata: { rating: analysis.score, reason: analysis.reason }, }; }) .build();这个示例展示了流水线的典型分工:analyze承担 I/O 密集的重活(调用模型),score只做轻量归一化(0–10 分制换算成 0–1)。这与后文性能优化建议keepscorelightweight完全一致。4.3 混合评分器(Hybrid)同一评分器内组合多种判据,用加权求和得到综合分:const hybridScorer buildScorer({ id: hybrid-validator, label: Comprehensive Validator, }) .analyze(({ payload }) { // Heuristic checks const hasProperLength String(payload.output || ).length 50; const hasNoErrors !String(payload.output || ).includes(error); // Could add LLM analysis here return { hasProperLength, hasNoErrors }; }) .score(({ results }) { // Combine multiple criteria const analysis results.analyze as { hasProperLength: boolean; hasNoErrors: boolean }; const lengthScore analysis.hasProperLength ? 0.5 : 0; const errorScore analysis.hasNoErrors ? 0.5 : 0; return { score: lengthScore errorScore, metadata: analysis, }; }) .build();与 4.2 的 LLM 版相比,混合评分器把规则判据 可选模型判据放在同一条流水线内,便于用单一score字段对外呈现。五、把评分器接进评估系统5.1 离线评估(Offline Evaluations)通过 voltagent/evals 的 createExperiment 将自定义评分器挂载到实验上,可传评分器实例本身,也可传实例 默认参数 阈值的配置对象:import { createExperiment } from voltagent/evals; export default createExperiment({ dataset: { name: customer-support }, experiment: { name: sentiment-test }, runner: async ({ item }) ({ output: await generateResponse(item.input), }), scorers: [ sentimentScorer, { scorer: lengthScorer, params: { maxLength: 200 }, threshold: 1.0, }, ], });threshold用于在实验汇总中计算通过率(pass rate)。更多实验机制见 offline-evaluations 文档。5.2 Agent 实时评估(Live Evaluations)把评分器注册进 Agent 的eval配置后,框架会在请求完成后调度评分器执行(实现见 agent/eval.ts,配置类型见 agent/types.ts 中的scorers: Recordstring, AgentEvalScorerConfig):import { Agent } from voltagent/core; const agent new Agent({ name: support-agent, model: openai(gpt-4o-mini), eval: { scorers: { sentiment: { scorer: sentimentScorer, params: { targetSentiment: positive }, }, }, sampling: { rate: 0.1 }, // Sample 10% of requests }, });sampling: { rate: 0.1 }表示只对 10% 的请求触发评分,控制在线成本;实时评估的整体机制见 live-evaluations。六、最佳实践6.1 类型安全为 payload 定义接口,并通过buildScorerPayload泛型注入,让 TypeScript 在payload.targetSentiment这类字段访问上给出完整提示:interface SentimentPayload { output: string; targetSentiment: positive | negative | neutral; } const typedScorer buildScorerSentimentPayload({ id: typed-sentiment, label: Typed Sentiment, }) .score(({ payload }) { // TypeScript knows payload structure const isPositive payload.targetSentiment positive; return { score: isPositive ? 1.0 : 0.0 }; }) .build();buildScorer的两个泛型参数分别是Payload与Params,默认均为Recordstring, unknown(见 builder.ts),同时约束两者可以进一步收紧params的类型。6.2 错误处理让评分器对意外输入保持鲁棒。有两点源码依据值得注意:其一,底层管道对步骤抛错有兜底(返回status: error),但显式处理能产出更有信息量的分数;其二,构建器在run()外层还叠加了采样判定,若被采样跳过则直接返回status: skipped、score: null(见 builder.ts)。.prepare(({ payload }) { try { const text String(payload.output || ); if (!text) throw new Error(Empty output); return { text }; } catch (error) { return { text: , error: error.message }; } })6.3 性能优化用prepare一次性完成校验与清洗,避免后续步骤重复处理;在analyze中缓存昂贵计算(如模型调用结果),供score/reason复用;保持score轻量,只做数值运算;reason只在确实需要解释时注册——它虽然开销小,但每多一步就多一次上下文快照。6.4 用测试验证评分器评分器本身是一个纯函数式的可测单元,用 vitest 直接断言分数与元数据即可:import { describe, it, expect } from vitest; describe(sentimentScorer, () { it(detects positive sentiment, async () { const result await sentimentScorer.run({ payload: { output: This is excellent!, targetSentiment: positive, }, params: {}, }); expect(result.score).toBe(1.0); expect(result.metadata.detectedSentiment).toBe(positive); }); it(handles empty input, async () { const result await sentimentScorer.run({ payload: { output: , targetSentiment: positive, }, params: {}, }); expect(result.score).toBeDefined(); expect(result.reason).toContain(neutral); }); });仓库中同样风格的构建器行为测试可参考 builder.spec.ts,底层管道测试见 create-scorer.spec.ts。七、高级模式7.1 静态与动态参数params既可以是构建时的静态默认值,也可以是一个由payload动态派生的函数。从 builder.ts 的#resolveParams实现看,参数解析遵循函数式/对象式基础参数 → 展开 → 用run()传入的params覆盖的合并顺序:interface KeywordParams { keyword: string; caseSensitive?: boolean; } const keywordScorer buildScorerRecordstring, unknown, KeywordParams({ id: keyword-match, params: { caseSensitive: false }, // default }) .score(({ payload, params }) { const output String(payload.output); const keyword params.keyword; const caseSensitive params.caseSensitive ?? false; const match caseSensitive ? output.includes(keyword) : output.toLowerCase().includes(keyword.toLowerCase()); return match ? 1 : 0; }) .build();动态参数:参数也可以完全由 payload 推导,让同一个评分器在不同数据行上使用不同判据:const dynamicScorer buildScorer({ id: dynamic-params, params: (payload) ({ expectedCategory: payload.category, threshold: payload.confidence ?? 0.8, }), }) .score(({ payload, params }) { const match payload.output params.expectedCategory; return match ? 1 : 0; }) .build();7.2 加权复合评分器(weightedBlend)weightedBlend把多个打分函数组合成一个score步骤,是官方提供的组合工具。结合 create-scorer.ts 的实现,有几个值得注意的语义:每个组件含id、weight与可选的step;若未提供step,则从context.results[id]读取既有结果,便于复用analyze阶段产出的分量;只有分数为有限数字的组件参与加权,最终分数按有效权重之和归一化计算,即某个组件缺失或失败不会让总分失真;全部缺失或总权重为 0 时返回score: 0;每个组件的得分都会写入context.results[component.id],并在metadata.blend下记录components(含normalizedWeight)与totalWeight,天然可审计。import { weightedBlend } from voltagent/core; const compositeScorer buildScorer({ id: composite, }) .score( weightedBlend([ { id: length, weight: 0.3, step: ({ payload }) { const length String(payload.output).length; return Math.min(length / 500, 1); }, }, { id: quality, weight: 0.7, step: async ({ payload }) { // Call LLM judge const result await evaluateQuality(payload.output); return result.score; }, }, ]) ) .build();八、小结与延伸自定义评分器的核心心智模型是四步各负其责,结果层层可见:prepare管输入卫生,analyze管特征与重计算,score管数值判定,reason管解释;底层由 createScorer 管道 保证顺序执行、错误隔离与元数据合并,构建器再提供类型上下文、参数解析与采样跳过。掌握这套机制后,你可以按以下路径继续深入仓库:常用评估需求先看 预置评分器;批量离线回归参见 离线评估;线上实时监控参见 Agent 实时评估;评分器聚合与汇总逻辑可进一步阅读 实验聚合器 与 实验评分器解析。赞分享人工智能AI AgentAgent 框架后端多智能体RAG工具调用Agent 记忆【免费下载链接】voltagentAI Agent Engineering Platform built on an Open Source TypeScript AI Agent Framework项目地址https://gitcode.com/gh_mirrors/vo/voltagent点击查看免费下载相关推荐VoltAgent 在线评估Live Evals实战在 Agent 上挂载启发式、LLM 评判与自定义评分器VoltAgent 在线评估Live Evals实战在 Agent 上挂载启发式、LLM 评判与自定义评分器 在 VoltAgent 中在线评估Liv人工智能AI AgentAgent 框架后端多智能体RAG工具调用Agent 记忆Agent 工作流AI 评测MCP 服务MCP Clients语音NeMo Evaluator 自定义 Benchmark 集成实战从 Framework Definition Files 到容器化评测流水线NeMo Evaluator 自定义 Benchmark 集成实战从 Framework Definition Files 到容器化评测流水线 NeMo EvAI 技能人工智能大模型深度学习MMDetection3D数据流水线详解与自定义实践MMDetection3D数据流水线详解与自定义实践 引言 在3D目标检测领域数据预处理和增强策略对模型性能至关重要。MMDetection3D作为OpenM人工智能计算机视觉深度学习自动驾驶上一篇掌握Node Redis版本管理从语义化版本到专业发布流程全攻略下一篇PanSou API接口使用手册GET/POST搜索参数全解析创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考