Skip to main content
Prism uses Qdrant as its vector database. All document chunks — regardless of file type — are stored as 768-dimensional vectors in a single collection named prism_documents. Every read and write operation is filtered by userId so that users only ever access their own data. The Qdrant client is initialized in lib/qdrant.ts using the @qdrant/js-client-rest package: 
import { QdrantClient } from '@qdrant/js-client-rest';

export const qdrantClient = new QdrantClient({
  url: process.env.QDRANT_CLUSTER_URL!,
  apiKey: process.env.QDRANT_API_KEY!,
});

​
Collection configuration

The prism_documents collection is created on first use by initializeCollection. If the collection already exists, the function ensures all payload indexes are present (index creation is idempotent — Qdrant returns an error if the index exists, which initializeCollection handles silently). 
await qdrantClient.createCollection('prism_documents', {
  vectors: {
    size: 768,
    distance: 'Cosine',
  },
  optimizers_config: {
    indexing_threshold: 10000,
  },
});
SettingValueNotes
Collection nameprism_documentsSingle collection for all users
Vector size768Matches Gemini text-embedding-004 output
DistanceCosineNormalized similarity for semantic text matching
Indexing threshold10,000Qdrant builds HNSW index after this many points

​
Payload indexes

Four keyword indexes are created at collection initialization. These make Qdrant’s filtered search efficient — without them, filtered queries require a full collection scan.
FieldSchemaUsed in
userIdkeywordAll search, scroll, and delete operations
documentIdkeywordChunk deletion, recommendations scroll
documentTypekeywordOptional filter in search and chat
categorykeywordOptional filter in search

​
Point structure

Each indexed chunk becomes a point in Qdrant with the following shape: 
{
  id: string,          // UUID (generated with crypto.randomUUID)
  vector: number[],    // float[768] from Gemini text-embedding-004
  payload: {
    documentId: string,
    chunkIndex: number,
    chunkText: string,
    documentName: string,
    documentType: string,   // 'PDF', 'DOCX', 'MD', 'TXT', 'TS', 'PY', 'IMAGE', etc.
    userId: string,
    uploadDate: string,     // ISO 8601 from Appwrite file.$createdAt
    category: string,       // auto-detected: 'Code', 'Report', 'Legal', 'Image', etc.
    pageNumber?: number,    // populated for PDF chunks (via indexDocumentChunk)
    section?: string,       // populated for structured documents (via indexDocumentChunk)
  }
}
batchIndexChunks (used by the /api/documents/index route) assigns a fresh crypto.randomUUID() to each point. indexDocumentChunk (the single-chunk variant) uses a deterministic ID of {documentId}_chunk_{chunkIndex}, which makes it safe to re-index individual chunks by upserting.

​
Indexing

​
batchIndexChunks

The primary indexing path. Called after batchGenerateEmbeddings produces all embeddings for a document: 
await batchIndexChunks(
  chunks.map((chunkText, index) => ({
    documentId,
    chunkIndex: index,
    chunkText,
    embedding: embeddings[index],   // float[768]
    metadata: {
      documentName,
      documentType,
      userId,
      uploadDate,
      category,
    },
  }))
);
Before calling qdrantClient.upsert, the function validates every embedding: 
const invalidChunks = chunks.filter(
  (chunk) => !Array.isArray(chunk.embedding) || chunk.embedding.length !== 768
);
if (invalidChunks.length > 0) {
  throw new Error(`Invalid embeddings: expected 768 dimensions, found ${...}`);
}
This validation prevents Qdrant from rejecting the upsert with a cryptic dimension mismatch error.

​
searchSimilarChunks

function searchSimilarChunks(
  queryEmbedding: number[],
  options?: {
    limit?: number;           // default: 5
    userId?: string;          // scopes results to one user
    documentType?: string[];  // filter to specific file types
    scoreThreshold?: number;  // default: 0.7
  }
)
The function builds a Qdrant filter from the provided options and issues two search requests:
  1. A diagnostic search for the top 3 results with no score threshold — logged for debugging.
  2. The actual search with score_threshold applied and the requested limit.
Returned results are normalized to a flat object: 
{
  id, score, documentId, chunkText, documentName,
  documentType, category, chunkIndex, pageNumber?, section?
}

​
Score thresholds by call site

Different parts of Prism use different thresholds to balance precision and recall:
Call sitescoreThresholdlimitReason
/api/chat0.45Lower threshold ensures enough context for RAG even when the question is loosely worded
/api/search/semantic0.5 (default)10 (default)Caller-configurable; default balances precision and coverage
searchSimilarChunks default0.75High precision for direct API use

​
Document recommendations

getRecommendations finds documents that are semantically similar to a given document. It does not require a text query — instead it uses the document’s own vector as the search probe.
1

Scroll for the source vector

Fetches the first chunk of the source document from Qdrant using the REST scroll API, filtered by both documentId and userId, with with_vector: true to retrieve the raw embedding.
2

Search for similar chunks

Runs qdrantClient.search using the retrieved vector as the query, excluding the source document with a must_not filter on documentId.
3

Aggregate by document

Groups matching chunks by documentId, tracking the highest chunk score per document and keeping the top 3 matching chunks for display.
4

Apply similarity threshold and rank

Filters out documents with a max score below 0.5 (SIMILARITY_THRESHOLD), then returns up to limit (default: 5) documents sorted by descending score.

​
Chunk deletion

When a document is deleted from Appwrite Storage, deleteDocumentChunks removes all associated Qdrant points: 
await qdrantClient.delete('prism_documents', {
  wait: true,
  filter: {
    must: [
      { key: 'documentId', match: { value: documentId } },
    ],
  },
});
The documentId payload index makes this a fast keyed delete rather than a scan. The deleteDocument function in lib/appwrite.ts calls /api/documents/delete after successfully removing the file from storage. If the Qdrant call fails, the storage file is already gone — the error is logged as a warning but does not surface to the user.
Because storage deletion and Qdrant deletion are not atomic, a crash between the two steps can leave orphaned vectors in prism_documents. These orphaned points do not affect other users (they are scoped by userId) but will consume storage in your Qdrant cluster. Re-indexing the document would restore consistency.

​
Collection stats

getCollectionStats returns a lightweight summary of the prism_documents collection: 
const stats = await getCollectionStats();
// { vectorCount: number, status: string, indexedVectorCount: number }
This is a utility function that can be used to check how many vectors are stored in the collection and whether the collection is in a healthy state.

Build docs developers (and LLMs) love

Get started for freeTalk to us