Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/neo4j-labs/neocarta/llms.txt

Use this file to discover all available pages before exploring further.

The Query Log connector parses a local query-log JSON file into the Neo4j semantic graph. Query log parsing reveals how users actually access your data — which tables and columns appear in queries, how they’re joined, and which common table expressions are used — without relying solely on declared schema metadata like foreign key constraints. This connector is distinct from BigQueryLogsConnector, which reads live query history directly from the BigQuery Cloud Logging API. The Query Log connector reads a local JSON file you have already exported. Use it when you want to replay a historical export, process logs from a non-BigQuery source in the BigQuery export format, or work offline.
Currently only the BigQuery JSON export format is supported as source. Logs from other databases stored in a compatible JSON structure can be ingested by passing source="bigquery".

​
What It Ingests

Node / RelationshipProperties
Querycontent (SQL text), query_id (hash)
CTEname, definition (SQL), query_id (parent query hash)
(:Query)-[:USES_TABLE]->(:Table)Derived from SQL parsing
(:Query)-[:USES_COLUMN]->(:Column)Derived from SQL parsing
(:Query)-[:DEFINES]->(:CTE)Inline CTEs parsed from each query
(:Column)-[:REFERENCES]->(:Column)Join criteria from SQL JOIN conditions, with criteria property
Schema nodes (Database, Schema, Table, Column) discovered during SQL parsing are also MERGEd into the graph with minimal properties (name only). If you have run a schema connector first, these MERGEs are no-ops against the existing enriched nodes.
Column nodes created solely from query log parsing do not have type, description, or key information — those come from a schema connector. nullable is set to True, is_primary_key and is_foreign_key to False for query-log-only columns.

​
Import

from neocarta.connectors.query_log import QueryLogConnector

​
Parameters

​
neo4j_driver
neo4j.Driver
required
Connected Neo4j driver instance.
​
database_name
str
default:"neo4j"
Target Neo4j database name.

​
ingest() Parameters

​
query_log_file
str
required
Filesystem path to the query log JSON file.
​
source
str
default:"bigquery"
Source format of the query log file. Currently "bigquery" is the only supported value.

​
Code Example

import os
from dotenv import load_dotenv
from neo4j import GraphDatabase
from neocarta.connectors.query_log import QueryLogConnector

load_dotenv()

neo4j_driver = GraphDatabase.driver(
    uri=os.getenv("NEO4J_URI"),
    auth=(os.getenv("NEO4J_USERNAME"), os.getenv("NEO4J_PASSWORD")),
)
neo4j_database = os.getenv("NEO4J_DATABASE", "neo4j")

connector = QueryLogConnector(
    neo4j_driver=neo4j_driver,
    database_name=neo4j_database,
)

connector.ingest(
    query_log_file="path/to/query_log.json",
    source="bigquery",   # currently the only supported format
)

neo4j_driver.close()
print("Query log connector completed successfully!")

​
CLI

pip install "neocarta[cli]"

neocarta query-log ingest \
  --query-log-file ./query_logs.json

​
Required Environment Variables

VariablePurpose
NEO4J_URINeo4j connection URI
NEO4J_USERNAMENeo4j username
NEO4J_PASSWORDNeo4j password
NEO4J_DATABASETarget Neo4j database (default: neo4j)

​
Obtaining BigQuery Query Logs

To export query logs from BigQuery in the expected JSON format, use the gcloud CLI: 
PROJECT_ID="your-project-id"

gcloud logging read \
  "resource.type=bigquery_resource AND protoPayload.methodName=jobservice.query" \
  --project=$PROJECT_ID \
  --format=json \
  --freshness=7d \
  --limit=1000 > query_log.json
You can also export logs via the Cloud Logging API or the GCP Console. Once the file is on disk, point --query-log-file at it.

​
Difference from neocarta bigquery logs

  • Reads a local JSON file you have already exported
  • Works offline, with archived logs, or with logs pre-filtered before import
  • CLI: neocarta query-log ingest --query-log-file ./logs.json
  • Class: QueryLogConnector

Build docs developers (and LLMs) love

Get started for freeTalk to us