Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/TrustifAI/trustifai/llms.txt

Use this file to discover all available pages before exploring further.

TrustifAI evaluates LLM and RAG responses through four trust signals — evidence coverage, epistemic consistency, semantic drift, and source diversity — and combines them into a single weighted Trust Score. This guide walks you through installing the SDK, configuring your environment, scoring a RAG response, and visualizing the reasoning behind the score.
1

Install TrustifAI

TrustifAI requires Python 3.10 or later. Install it from PyPI:
pip install trustifai
For full installation options including tracing and test extras, see the Installation guide.
2

Set up your environment

TrustifAI uses LiteLLM to call LLMs and embedding models, so you configure it by setting the appropriate API key for your provider. Create a .env file in your project root:
.env
OPENAI_API_KEY=<your-api-key>
# Add any other provider keys you need — see the Installation guide for the full list
Then point your config_file.yaml at the model you want to use:
config_file.yaml
llm:
  type: "openai"
  params:
    model_name: "gpt-4o"
    api_type: "chat_completion"
  kwargs:
    temperature: 0.01
    max_tokens: 2048
    logprobs: true

embeddings:
  type: "openai"
  params:
    model_name: "text-embedding-3-small"

metrics:
  - type: "evidence_coverage"
    enabled: true
    params:
      strategy: "llm"
      STRONG_GROUNDING: 0.85
      PARTIAL_GROUNDING: 0.60
  - type: "consistency"
    enabled: true
    params:
      STABLE_CONSISTENCY: 0.85
      FRAGILE_CONSISTENCY: 0.60
  - type: "source_diversity"
    enabled: true
    params:
      HIGH_DIVERSITY: 0.85
      MODERATE_DIVERSITY: 0.60
  - type: "semantic_drift"
    enabled: true
    params:
      STRONG_ALIGNMENT: 0.85
      PARTIAL_ALIGNMENT: 0.60
  - type: "trust_score"
    params:
      RELIABLE_TRUST: 0.80
      ACCEPTABLE_TRUST: 0.60

score_weights:
  - type: "evidence_coverage"
    params:
      weight: 0.40
  - type: "consistency"
    params:
      weight: 0.20
  - type: "source_diversity"
    params:
      weight: 0.10
  - type: "semantic_drift"
    params:
      weight: 0.30
3

Score a RAG response

Use MetricContext to bundle your query, answer, and retrieved documents, then call get_trust_score():
from trustifai import Trustifai, MetricContext
from langchain_core.documents import Document  # langchain is not required — used here for demo only

# 1. Define your RAG context
context = MetricContext(
    query="What is the capital of India?",
    answer="The capital is New Delhi.",
    documents=[
        Document(
            page_content="New Delhi is the capital of India.",
            metadata={"source": "wiki.txt"}
        )
    ]
)

# 2. Initialize the trust engine
trust_engine = Trustifai("config_file.yaml")

# 3. Calculate the trust score
result = trust_engine.get_trust_score(context)
print(f"Trust Score: {result['score']} | Decision: {result['label']}")
The result dictionary contains the aggregated score (0.0–1.0), a human-readable label (RELIABLE, ACCEPTABLE (WITH CAUTION), or UNRELIABLE), and a details dictionary with individual metric scores.
4

Generate with real-time confidence scoring

For online evaluation — where you want confidence alongside generation — use the generate() method. It calls your configured LLM with logprobs enabled and returns a confidence score derived from token log probabilities:
from trustifai import Trustifai

trust_engine = Trustifai(config_path="config_file.yaml")

result = trust_engine.generate(
    prompt="What is the capital of France?",
    system_prompt="You are a helpful assistant."
)

print(f"Response:    {result['response']}")
print(f"Confidence:  {result['metadata']['confidence_score']}")
print(f"Label:       {result['metadata']['confidence_label']}")
Confidence scoring relies on token log probabilities. Not all models expose logprobs, and poorly calibrated models may produce unreliable confidence estimates. Check your provider’s documentation before relying on this signal in production.
5

Visualize the reasoning graph

TrustifAI builds a directed acyclic graph (DAG) that shows how each metric score flows into the final trust decision. Pass your result to build_reasoning_graph(), then call visualize():
# Build the reasoning graph from your trust score result
graph = trust_engine.build_reasoning_graph(result)

# Save an interactive HTML file (opens in any browser)
trust_engine.visualize(graph, graph_type="pyvis")  # Saves to reasoning_graph.html

# Or generate Mermaid syntax for embedding in markdown docs
mermaid_output = trust_engine.visualize(graph, graph_type="mermaid")
print(mermaid_output)
The PyVis output is a physics-based interactive graph where nodes are color-coded by trust level: green for high trust, orange for partial, and red for low trust.

​
Next steps

Installation

Full installation options including trace and test extras, git clone setup, and all supported provider keys.

Configuration

Tune metric thresholds, adjust score weights, and configure LiteLLM providers.

Core concepts

Understand how each trust signal is computed and combined into the final score.

Custom metrics

Extend TrustifAI with your own evaluation logic using the BaseMetric interface.

Build docs developers (and LLMs) love

Get started for freeTalk to us