Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/npow/context-bench/llms.txt

Use this file to discover all available pages before exploring further.

CodeExecution evaluates code generation by executing the model’s output against the test cases provided in the dataset example, using a subprocess with a configurable timeout. A score of 1.0 (pass@1) means the first generated solution passes all tests. 
from context_bench.evaluators import CodeExecution

​
Constructor

CodeExecution(timeout: float = 10.0)
​
timeout
float
default:"10.0"
Maximum time in seconds to wait for the subprocess to complete. If execution exceeds this limit, the attempt is counted as a failure.

​
score()

def score(self, original: dict[str, Any], processed: dict[str, Any]) -> dict[str, float]
​
original
dict
required
The unmodified example dict. Expected keys vary by dataset:
  • HumanEval: "context" (function prompt), "test" (check function body), "entry_point" (function name to call).
  • MBPP: "test_list" (list of assertion strings), optionally "test_setup_code" (imports/helpers).
​
processed
dict
required
The output dict returned by the system under test. Must contain a "response" key with the generated code string.

​
Return values

​
pass_at_1
float
required
1.0 if the generated code passes all test cases (subprocess exit code 0), otherwise 0.0.
Any exception during execution — including syntax errors, runtime errors, assertion failures, and timeouts — results in pass_at_1: 0.0.

​
Example

from context_bench.evaluators import CodeExecution
ev = CodeExecution(timeout=10.0)
ev.score(
    {"context": "def add(a, b):\n", "test": "def check(c):\n    assert c(1,2)==3\n", "entry_point": "add"},
    {"response": "    return a + b\n"},
)
# {'pass_at_1': 1.0}

​
Auto-wired datasets

CodeExecution is automatically applied when any of the following datasets are selected:
CLI nameDataset
humanevalHumanEval
mbppMBPP

​
Implementation notes

The evaluator assembles the executable code differently depending on the dataset format: HumanEval style (test field present):
  1. Concatenate context + response + test into a single script.
  2. Append check(entry_point) to invoke the test harness.
MBPP style (test_list field present):
  1. Start with response.
  2. Append optional test_setup_code (imports, helpers).
  3. Append each assertion string from test_list.
The assembled script is passed to python3 -c via subprocess.run. Exit code 0 means all assertions passed. subprocess.TimeoutExpired and any other exceptions are caught and scored as 0.0.
Code execution runs untrusted model output in a subprocess on the host machine. Use a sandboxed environment for production benchmarking.

Build docs developers (and LLMs) love

Get started for freeTalk to us