Use this file to discover all available pages before exploring further.
Collectors are the ingestion layer of GitHub Threat Detector. Each collector module is responsible for one slice of the GitHub API — events, Actions runs, releases, contributors, workflow files, and commit metadata — and writes normalized rows into PostgreSQL using the upsert helpers in db/queries.py. All collectors are invoked through the collect CLI command and share a common HTTP client defined in collectors/github_client.py.
paginate requests up to MAX_PAGES pages (default: 10) with per_page=100 items per page. It stops early if any of the following conditions are met:
Condition
Action
Response status is 404, 403, or 429
Break immediately (no exception raised)
Response body is an empty list
Break — no more items
Link header has no rel="next"
Break — last page reached
MAX_PAGES pages have been fetched
Break — page cap reached
The 403/429 short-circuit means collection degrades gracefully under rate limiting: partial data is written to the database and the process exits cleanly rather than crashing.
The GitHub public Events API returns at most approximately 300 recent events per repository (10 pages × ~30 events), regardless of per_page. For high-traffic repositories this means only a short recent window is available. To monitor continuously, schedule collect on a cron and keep the collection interval shorter than the event window.
Fetches the event stream for a repository, an organization, or a user and stores each event in github_events.Functions:
Function
API endpoint
collect_for_repo(full_name)
GET /repos/{owner}/{repo}/events
collect_for_org(org)
GET /orgs/{org}/events
collect_repo_events(owner, repo)
GET /repos/{owner}/{repo}/events
collect_org_events(org)
GET /orgs/{org}/events
collect_user_events(username)
GET /users/{username}/events/public
Each event is stamped with a _source tag (e.g. "acme-corp_payments-service_events") before the batch is handed to upsert_github_events_batch(), which uses execute_values for a single multi-row INSERT … ON CONFLICT (id) DO NOTHING.Writes to:github_eventsCLI flag: enabled by default — runs whenever a --repos or --orgs target is given.
Fetches GitHub Actions workflow run records and stores them in workflow_runs.Functions:
Function
API endpoint
collect_workflow_runs(full_name, status=None)
GET /repos/{owner}/{repo}/actions/runs
The status parameter maps directly to the GitHub API’s status query parameter, allowing you to collect only completed or in_progress runs. The raw API response wraps runs inside a workflow_runs key; the collector unwraps that automatically.Workflow run records are upserted with ON CONFLICT (id) DO UPDATE, so re-running the collector refreshes status, conclusion, updated_at, and payload for runs that were still in progress on a previous collection.Writes to:workflow_runsCLI flag:--actions
Fetches published releases and stores them as synthetic ReleaseEvent rows in github_events, allowing release activity to flow through the same analyzer pipeline as native events.Functions:
Function
API endpoint
collect_releases(full_name)
GET /repos/{owner}/{repo}/releases
Because the GitHub Events API does not reliably surface ReleaseEvent for all repositories, this collector normalises each release into the standard event envelope:
{ "id": f"release-{release['id']}", # synthetic, avoids collision with real event IDs "type": "ReleaseEvent", "actor": {"id": author["id"], "login": author["login"]}, "repo": {"name": full_name}, "payload": {"action": "published", "release": release}, "created_at": release.get("published_at") or release.get("created_at"),}
The synthetic release- prefix on id guarantees these rows never collide with real GitHub event IDs, which are purely numeric strings.Writes to:github_events (with event_type = 'ReleaseEvent')CLI flag:--releases
Seeds the repo_contributors table with known contributors for a repository. This table is used as an allowlist by several analyzers to suppress findings for established maintainers.Functions:
Function
API endpoint
collect_contributors(full_name)
GET /repos/{owner}/{repo}/contributors
Anonymous contributors are excluded (anon=false). Each contributor row is upserted with ON CONFLICT (repo_name, actor_login) DO UPDATE so that contribution counts are refreshed on re-collection.Writes to:repo_contributorsCLI flag:--contributors
Fetches the raw YAML content of every .github/workflows/*.yml and .github/workflows/*.yaml file from a repository’s default branch. This is required for static-analysis and AI prompt-injection detection rules that operate on workflow source text.Functions:
Function
API endpoints used
collect_workflow_files(full_name)
GET /repos/{owner}/{repo} → GET /repos/{owner}/{repo}/git/trees/{branch}?recursive=1 → GET /repos/{owner}/{repo}/contents/{path}
The collection process has three steps:
Fetch repository metadata to identify the default branch.
Fetch the full recursive git tree for that branch and filter entries to .github/workflows/ paths ending in .yml or .yaml.
Fetch each workflow file via the Contents API. The API returns Base64-encoded content, which is decoded to UTF-8 before storage.
Files that the Contents API returns with an encoding other than base64 are skipped.Writes to:workflow_filesCLI flag:--workflow-files
Enriches default-branch push events with per-commit metadata from the Commits API. The resulting rows in push_commits enable tamper-detection rules that compare git author timestamps against committer timestamps or look for mismatched identities.Functions:
Function
API endpoints used
collect_push_commits(full_name)
GET /repos/{owner}/{repo} → GET /repos/{owner}/{repo}/commits/{sha}
The collector reads PushEvent rows from github_events for the target repository, filters to pushes on the default branch only (refs/heads/<default>), and fetches each head commit SHA individually from the Commits API. SHAs already present in push_commits are skipped, so the collector is safe to run incrementally.The following fields are extracted from the Commits API response and stored:
author.login / committer.login (GitHub user objects)