Buckets

Overview

Storage Buckets provide durable, shareable storage for jobs and workflows. A bucket can hold structured JSON data and any number of files. Jobs can fetch from and store to buckets at well-defined points in their lifecycle so outputs from one job are available as inputs to another, even when the jobs are not directly connected in a chain.

Key Features

Purpose: Persistent data/files exchange between jobs and workflows
Content types: JSON data object and file collection (zero or more files)
Access: Manage via UI and API with privilege-based controls
Job integration: Fetch at job start; store on job completion based on action conditions
Direct links: Files in buckets are downloadable by URL

When to Use Buckets

Cross-Job Handoff

Pass artifacts from build to deploy, or output from data prep to analysis:

// Build job stores artifacts
{
  "type": "store",
  "condition": "success",
  "bucket_id": "bme4wi6pg35",
  "bucket_sync": "files",
  "bucket_glob": "*.tar.gz"
}

// Deploy job fetches artifacts
{
  "type": "fetch",
  "condition": "start",
  "bucket_id": "bme4wi6pg35",
  "bucket_sync": "files",
  "bucket_glob": "*.tar.gz"
}

Workflow State Sharing

Checkpointing

Persist intermediate results for retries or manual inspection:

Store partial results during long-running jobs
Resume from checkpoint on failure
Inspect intermediate state for debugging

Shared State

Maintain small JSON documents that multiple jobs can read/update over time:

Configuration data shared across jobs
Counters and metrics aggregation
Lock files for coordination

Managing Buckets in the UI

Users with appropriate privileges can create, edit, and delete buckets:

Create Bucket

Provide a title and optional icon/notes. The ID is auto-generated:

// ID generation (from source)
if (!params.id) params.id = Tools.generateShortID('b');

Edit JSON Data

Buckets have a JSON “Data” pane for arbitrary user-defined data:

Click “Edit Data” to modify the JSON object
Data is shallow-merged when updated via API or actions
Namespace your keys to avoid collisions

Upload Files

Drag-and-drop or select multiple files:

Filenames are normalized (lowercased; non-alphanumerics become underscores)
Existing files with same normalized name are replaced
Uploads respect configured limits

Download & Delete

Click a file to download via direct URL
Remove individual files (deletions are permanent)
Links use files/bucket/<bucket_id>/<hash>/<filename> path

Upload Settings

Configured via client.bucket_upload_settings:

"bucket_upload_settings": {
  "max_files_per_bucket": 100,
  "max_file_size": 1073741824,
  "accepted_file_types": ""
}

File uploads are enforced server-side. Exceeding limits will result in an error.

Using Buckets in Jobs

Buckets integrate with jobs through two action types: Fetch Bucket and Store Bucket.

Fetch at Job Start

Use Fetch Bucket with the start condition to pull bucket content into the job’s input context before launch:

{
  "enabled": true,
  "condition": "start",
  "type": "fetch",
  "bucket_id": "bme4wi6pg35",
  "bucket_sync": "data_and_files",
  "bucket_glob": "*.csv"
}

Behavior:

Data: Shallow-merged into the job’s input.data
Files: Selected files added to job’s input file list and staged into temp directory

Avoid key collisions by namespacing your bucket data keys deliberately.

Store on Completion

Use Store Bucket with completion conditions (e.g., success, error, complete) to persist job outputs:

{
  "enabled": true,
  "condition": "success",
  "type": "store",
  "bucket_id": "bme4wi6pg35",
  "bucket_sync": "data_and_files",
  "bucket_glob": "*.mp4"
}

Behavior:

Data: Job output data written to bucket when bucket_sync includes data
Files: Job output files filtered by bucket_glob and stored when bucket_sync includes files

Action Parameters

bucket_id

Target Bucket ID (required)

// Validation from source code
if (!this.requireParams(params, {
  id: /^[a-z0-9_]+$/
}, callback)) return;

bucket_sync

Controls what is fetched/stored:

"data" - Only JSON data
"files" - Only files
"data_and_files" - Both (default)

bucket_glob

Optional glob pattern to filter files (default: *)Examples:

*.csv - CSV files only
report_*.{pdf,txt} - Reports in PDF or TXT format
**/*.log - All log files in any subdirectory

Workflows and Buckets

Buckets enable data sharing in complex workflow patterns:

Configure Upstream Nodes

Attach Store Bucket actions to upstream nodes:

{
  "enabled": true,
  "condition": "success",
  "type": "store",
  "bucket_id": "shared_artifacts",
  "bucket_sync": "data_and_files"
}

Configure Downstream Nodes

Attach Fetch Bucket actions to downstream nodes:

{
  "enabled": true,
  "condition": "start",
  "type": "fetch",
  "bucket_id": "shared_artifacts",
  "bucket_sync": "data_and_files"
}

This pattern is useful for fan-out/fan-in designs, optional branches, and long-lived shared state between periodic jobs.

Downloading Files by URL

Every bucket file includes a stable path for direct download:

GET https://your.xyops.example.com/files/bucket/<bucket_id>/<hash>/<filename>

Example:

curl -O https://your.xyops.example.com/files/bucket/bme4wi6pg35/bdY8zZ9nKynfFUb4xH6fA/report.csv

URLs are permalinks even if files are replaced, but not if deleted and re-added.

Programmatic Access

Access buckets programmatically during job execution:

Create Bucket & API Key

Create a storage bucket and save the Bucket ID
Create an API Key with edit_buckets privilege
Save the API key secret when prompted

Create Secret Vault

Add your API Key and Bucket ID as variables:

{
  "XYOPS_API_KEY": "your-api-key-secret",
  "XYOPS_BUCKET_ID": "bme4wi6pg35"
}

Assign the secret vault to your Event, Category, or Plugin.

Write Bucket Data

Use the write_bucket_data API during job execution:

#!/bin/sh
JSON_PAYLOAD='{"data": {"foo": "bar", "number": 1234}}'
API_URL="$JOB_BASE_URL/api/app/write_bucket_data/v1?id=$XYOPS_BUCKET_ID"

curl -sS "$API_URL" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $XYOPS_API_KEY" \
  -d "$JSON_PAYLOAD" >/dev/null

write_bucket_data performs a shallow merge write with locking to prevent corruption.

Read Bucket Data

Use the get_bucket API to retrieve data:

#!/bin/sh
API_URL="$JOB_BASE_URL/api/app/get_bucket/v1?id=$XYOPS_BUCKET_ID"
RESPONSE=$(curl -sS -H "X-API-Key: $XYOPS_API_KEY" "$API_URL")

echo "Response: $RESPONSE"

API Benefits Over Actions

Complete control over when data is read/written
Not limited to job start and completion
Can update bucket multiple times during job execution
Rugged design handles concurrent access from multiple jobs

Storage Architecture

Buckets use a multi-record storage structure:

// From source code: api/buckets.js
var bucket_path = 'buckets/' + params.id;
var records = {};
records[bucket_path + '/data'] = params.data || {};
records[bucket_path + '/files'] = params.files || [];

Metadata: Stored in global/buckets list
Data: Stored at buckets/<id>/data
Files: Stored at buckets/<id>/files

This separation allows efficient updates without loading entire bucket contents.

Best Practices

Namespacing

Use distinct keys in bucket JSON to avoid shallow-merge collisions:

{
  "build_job": {"version": "1.2.3", "commit": "abc123"},
  "test_job": {"passed": true, "coverage": 85}
}

Size Discipline

Prefer buckets for modest artifacts:

Keep JSON data small (< 1 MB)
Large datasets better handled via external storage
Reference external data by URL in bucket JSON

Cleanup

Consider lifecycle practices:

Replace/rotate files to keep buckets tidy
Delete unused buckets periodically
Stay within configured limits

Error Handling

Handle bucket operations gracefully:

Check API responses for errors
Validate bucket exists before fetching
Use try-catch around JSON parsing

API Reference

Key bucket API endpoints:

get_buckets - List all buckets
get_bucket - Get single bucket (includes data and file list)
create_bucket - Create new bucket
update_bucket - Update bucket metadata, data, or files
write_bucket_data - Shallow-merge write bucket data (with locking)
upload_bucket_files - Upload files to bucket
delete_bucket_file - Remove file from bucket
delete_bucket - Permanently delete bucket

Required Privileges

create_buckets: Create new buckets
edit_buckets: Edit bucket metadata, JSON data, and files
delete_buckets: Delete buckets and all contained data/files

Listing and fetching typically requires only a valid session or API Key.

Get Started

Core Concepts

Monitoring & Alerts

Organization

Operations

Plugins & Extensions

Advanced

Overview

Key Features

When to Use Buckets

Managing Buckets in the UI

Upload Settings

Using Buckets in Jobs

Fetch at Job Start

Store on Completion

Action Parameters

Workflows and Buckets

Downloading Files by URL

Programmatic Access

API Benefits Over Actions

Storage Architecture

Best Practices

API Reference

Required Privileges

Build docs developers (and LLMs) love

Get Started

Core Concepts

Monitoring & Alerts

Organization

Operations

Plugins & Extensions

Advanced

Documentation Index

​Overview

​Key Features

​When to Use Buckets

​Managing Buckets in the UI

​Upload Settings

​Using Buckets in Jobs

​Fetch at Job Start

​Store on Completion

​Action Parameters

​Workflows and Buckets

​Downloading Files by URL

​Programmatic Access

​API Benefits Over Actions

​Storage Architecture

​Best Practices

​API Reference

​Required Privileges

Build docs developers (and LLMs) love

Overview

Key Features

When to Use Buckets

Managing Buckets in the UI

Upload Settings

Using Buckets in Jobs

Fetch at Job Start

Store on Completion

Action Parameters

Workflows and Buckets

Downloading Files by URL

Programmatic Access

API Benefits Over Actions

Storage Architecture

Best Practices

API Reference

Required Privileges