feat: add voice task success scorer - #232

AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-10-02.

Summary

Adds VoiceTaskSuccess, an LLM judge for whether a voice agent completed the caller's request, judged from the whole call (instructions, tool calls, tool results).

• templates/voice_task_success.yaml: A/B/C rubric over {{thread_with_system}}.
• JS and Python VoiceTaskSuccess: picks the conversation from a thread_with_system argument, else trace.getThread() / trace.get_thread(). With no conversation it returns a null score without calling the model. Otherwise it passes the messages to the template as a JSON string, so tool calls and results are included.
• js/manifest.ts entry, so it's available as a built-in scorer.
• SCORERS.md, README.md, AGENTS.md.

Testing

• Build, tsc --noEmit (one pre-existing error in js/render-messages.test.ts), vitest, pre-commit. Tests that need an OpenAI key fail the same way on main.
• Ran JS and Python (sync and async) against a live model for each source: thread_with_system, trace, no conversation (null), empty trace (null), and .partial.
• JS JSON.stringify and Python json.dumps produce identical text for the same conversation. The judge cites tool names and arguments (a transfer of $1000 to Bob when the caller asked for $10 to Alice scores 0).

🤖 Generated with Claude Code