[DRAFT] fix(service): stage session resources outside the agent sandbox tree - #661

Summary

Fixes b/571912956: Prevent AI coding agents under evaluation from discovering and reading the grading rubrics and test answer keys from their working directory.

The Situation: What Happened?

EvalBench tests and scores AI coding agents (such as Gemini CLI, Antigravity, and Claude Code) on complex multi-turn tasks. When a test run starts, the client sends over the test package:

1. The test scenario (the prompts and starter files for the agent)
2. The grading materials (the scoring rubric YAML, conversation plan, and custom Python validator scripts that grade whether the agent succeeded)

The Problem: Agents Were "Reading the Answer Key"

Previously, EvalBench placed all of these files inside the same folder:

/tmp_sessions/<session_id>/
├── fake_home/              <-- The AI agent worked here
└── google3/...             <-- Rubrics, test scenarios, & grading validators lived here

Because the agent's workspace (fake_home) and the grading files were right next to each other, an agent simply exploring its environment or running ls .. could see all the test files, rubrics, and Python grading scripts.

In actual benchmark runs, agents did exactly that: they read the grading rubric and validator code, saw the exact criteria the test was checking for, and tailored their responses to pass the evaluation.

This artificially inflated benchmark scores because the agent was effectively "cheating" by reading the answer key before finishing the exam.

The Fix: Separate the Answer Key from the Student's Desk

This PR completely separates the evaluation grading assets from the agent's working directory so an agent cannot reach the grading materials by looking around its workspace:

1. Move test assets out of the agent's path:

   • Grading materials / Answer keys: Now saved to /var/lib/evalbench/sessions/<session_id>/
   • Agent workspace: Stays in /tmp_sessions/<session_id>/fake_home/
     Because these are now in entirely different directory trees, an agent looking around its workspace or navigating up parent directories (..) will never find the rubrics or grading scripts.

2. Isolate starter files (work_dir):
   If a test scenario provides starter code for the agent to work on, EvalBench now makes a copy of only that specific starter directory inside the agent's sandbox. The agent works on the copy and has no link back to the rest of the test package.

3. Session cleanup:
   Updated the session manager to clean up both directories when a test run completes.

Changes Breakdown

• evalbench/util/sessionmgr.py: Points SESSION_RESOURCES_PATH to /var/lib/evalbench/sessions/ and sets up SESSION_SANDBOX_PATH for /tmp_sessions/. Ensures both paths are cleaned up when sessions are pruned.
• evalbench/evaluator/agentevaluator.py: Copies scenario starter code into the sandbox workspace so the agent never touches the parent resources tree.
• evalbench_service/Dockerfile: Pre-creates /var/lib/evalbench/sessions/ in the container.
• Generators: Swapped the hardcoded "/tmp_sessions" string for the shared constant.
• Unit Tests: Added tests in test/session_resources_test.py and test/agentevaluator_test.py verifying directory separation, sandbox copying, and cleanup.

Verification

• Added unit tests verifying:
  • Resource files and sandbox folders are created in disjoint directories.
  • Scenario starter files are safely copied into the sandbox without exposing neighboring test files.
  • Reaper correctly removes both resource and sandbox directories upon completion.
• Ran existing unit test suite to verify no regressions.