Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/npow/context-bench/llms.txt

Use this file to discover all available pages before exploring further.

MathEquivalence compares mathematical answers by normalizing LaTeX notation and then attempting numeric comparison. It is implemented in pure Python with no external math libraries. 
from context_bench.evaluators import MathEquivalence

​
Constructor

MathEquivalence takes no constructor parameters. 
ev = MathEquivalence()

​
score()

def score(self, original: dict[str, Any], processed: dict[str, Any]) -> dict[str, float]
​
original
dict
required
The unmodified example dict. Must contain an "answer" key with the ground-truth mathematical expression or value.
​
processed
dict
required
The output dict returned by the system under test. Must contain a "response" key with the model’s answer string. If the response contains \boxed{...}, the content of the box is extracted and used as the answer.

​
Return values

​
math_equiv
float
required
1.0 if the reference and response are mathematically equivalent, otherwise 0.0.
If answer is empty, returns math_equiv: 1.0. If response is empty, returns math_equiv: 0.0.

​
Examples

from context_bench.evaluators import MathEquivalence
ev = MathEquivalence()
ev.score({"answer": r"\frac{1}{2}"}, {"response": "0.5"})
# {'math_equiv': 1.0}
ev.score({"answer": "42"}, {"response": r"The answer is $\boxed{42}$."})
# {'math_equiv': 1.0}

​
Auto-wired datasets

MathEquivalence is automatically applied when any of the following datasets are selected:
CLI nameDataset
mathMATH (competition mathematics)
gsm8kGSM8K (grade school math)
mgsmMGSM (multilingual math; configurable, e.g. mgsm:de)

​
Implementation notes

Equivalence is checked in a cascade — the first matching condition returns True:
  1. Boxed extraction — if the response contains \boxed{...}, the content is extracted using brace-depth matching before any other processing.
  2. Case-insensitive string match — whitespace-normalized strings are compared directly.
  3. LaTeX normalization — both strings are normalized and compared again. Normalization handles:
    • \frac{a}{b} → (a)/(b)
    • \text{}, \mathrm{}, \textbf{} → plain text
    • \left, \right → removed
    • \sqrt{x} → sqrt(x)
    • \cdot, \times → *
    • \pi → pi, \infty → inf
    • \pm → +-
    • LaTeX spacing commands removed
  4. Numeric comparison — if both normalized strings parse as numbers (including fractions a/b and percentages x%), they are compared with a relative tolerance of 1e-6.
  5. Numeric comparison on original strings — the same numeric comparison is attempted on the pre-normalization strings as a final fallback.

Build docs developers (and LLMs) love

Get started for freeTalk to us