HNU-TimeLetter Feishu Data Sync System Architecture

The sync system is the operational backbone of HNU-TimeLetter. It provides a reliable, observable pipeline for pulling content from Feishu Bitable, processing image attachments through Aliyun OSS, and materialising the JSON artifacts that the frontend consumes at runtime. The architecture is intentionally self-contained — no external message brokers, no distributed coordination — designed for a single long-lived Node.js process managed by PM2 on a self-hosted server.

The sync system requires a single-instance, always-on Node.js process. It is incompatible with Serverless / edge runtimes and containerised multi-replica environments. If the deployment model changes to multi-instance or blue-green, the file-based lock and job store must be migrated to an external distributed store.

Three-Layer Architecture

┌─────────────────────────────────────────────────────────────┐
│  TRIGGER LAYER                                              │
│  CLI · Admin API · Scheduler · Webhook                      │
└──────────────────────────┬──────────────────────────────────┘
                           │ createSyncJob / runSyncJob
┌──────────────────────────▼──────────────────────────────────┐
│  ORCHESTRATOR LAYER                                         │
│  SyncOrchestrator — job lifecycle, file lock, persistence   │
└──────────────────────────┬──────────────────────────────────┘
                           │ execute TableSyncModule[]
┌──────────────────────────▼──────────────────────────────────┐
│  MODULE LAYER                                               │
│  locations · characters · stories                           │
│  creation_headers · creation_board · contributors           │
└─────────────────────────────────────────────────────────────┘

Layer 1 — Trigger Layer

All four trigger paths call the same orchestrator entry point, ensuring consistent job lifecycle handling regardless of how a sync is initiated.

CLI — npm run sync

The simplest trigger. Run from a terminal or a CI/CD pipeline step.

npm run sync
# expands to: tsx src/scripts/sync-feishu.ts

The script calls runSyncJob directly, streams progress to stdout, and exits with code 0 on success or 1 on failure.

// src/scripts/sync-feishu.ts
await runSyncJob({
  kind: 'sync-data',
  tables: ['locations', 'stories', 'creation_board'],
  triggeredBy: 'cli',
});

Admin API — POST /api/admin/sync/jobs

Used by the /admin back-office UI for manual one-click syncs. The request body selects which tables to run and the dependency mode.

// POST /api/admin/sync/jobs
await createSyncJob({
  kind: 'sync-data',
  tables: ['locations', 'stories'],
  dependencyMode: 'read_local',
  triggeredBy: 'admin-ui',
});

Returns the new job record immediately with status: "queued". The job executes asynchronously; the UI polls GET /api/admin/sync/jobs/:jobId for updates.

Scheduler — node-schedule

Initialised at process startup inside src/lib/admin/. Reads the cron expression and defaultTables list from runtime/admin/sync-config.json.

// src/lib/admin/ (scheduler initialisation)
await createSyncJob({
  kind: 'sync-data',
  tables: config.defaultTables,
  triggeredBy: 'scheduler',
});

Scheduled jobs are subject to the same concurrency lock as manual jobs — if a previous job is still running when the cron fires, the new job is rejected with a 409 Conflict status.

Webhook — POST /api/admin/sync/webhook

Allows external systems (e.g. Feishu automation flows, GitHub Actions) to trigger a sync over HTTP. Requests must include the SYNC_WEBHOOK_SECRET in the Authorization header.

curl -X POST https://your-server/api/admin/sync/webhook \
  -H "Authorization: Bearer $SYNC_WEBHOOK_SECRET" \
  -H "Content-Type: application/json" \
  -d '{"tables": ["creation_board"]}'

Layer 2 — Orchestrator Layer

SyncOrchestrator in src/lib/sync/orchestrator.ts owns the entire job lifecycle: creation, locking, step execution, persistence, and cleanup.

Job Lifecycle

queued ──► running ──► success
                  ──► partial_success
                  ──► failed
                  ──► canceled

Status	Description
`queued`	Job record created; waiting to acquire the file lock
`running`	Lock acquired; table modules executing sequentially
`success`	All steps completed without errors
`partial_success`	At least one step succeeded; some steps failed but `continueOnTableError` was `true`
`failed`	Unrecoverable error, or all steps failed
`canceled`	Job was manually canceled before execution completed

File-Based Lock

A single lock file at runtime/admin/sync-lock.json prevents concurrent jobs. Its structure:

{
  "jobId": "sync-1746028800000-ab12",
  "pid": 12345,
  "createdAt": "2026-04-30T10:00:00.000Z",
  "expiresAt": "2026-04-30T10:30:00.000Z"
}

Behaviour	Detail
New job vs. running job	Returns `409 Conflict` with `currentJobId` in the response body
Lock timeout	`SYNC_LOCK_TIMEOUT_MS` — default 30 minutes (`1800000` ms)
Stale lock recovery	Expired lock: old job is marked `failed`, lock is released, new job proceeds

Job Persistence

Each job record is written to runtime/admin/sync-jobs/<jobId>.json. The store retains the most recent 100 jobs; older files are purged automatically. Job records are read by the admin UI to render the job history table.

Layer 3 — Module Layer

Each Feishu table has a dedicated TableSyncModule. Modules are registered in src/lib/sync/registry.ts and executed in dependency order by the orchestrator.

SyncTableKey Registry

// src/lib/sync/registry.ts
export const tableRegistry = {
  locations:        locationsModule,
  characters:       charactersModule,
  stories:          storiesModule,
  creation_headers: creationHeadersModule,
  creation_board:   creationBoardModule,
  contributors:     contributorsModule,
} satisfies Record<SyncTableKey, TableSyncModule>;

SyncTableKey is derived from SyncTableOutputMap — it is never maintained as a separate union type. To add a new table, add a key to SyncTableOutputMap in src/lib/sync/types.ts; SyncTableKey updates automatically.

Table Registry Reference

`SyncTableKey`	Feishu source	Output artifact	Dependencies
`locations`	地点表	`src/config/locations.json`	None
`characters`	角色表 (with avatar OSS back-fill)	`src/data/characters.json`	None
`stories`	展示表	`src/data/content.json` (stories nested in `LocationPoint[]`)	`locations`, `characters`
`creation_headers`	便签头表 (card metadata)	`src/data/creation-board-headers.json`	None
`creation_board`	揭示板 / 收集结果 view	`src/data/creation-board.json`	`creation_headers`
`contributors`	鸣谢名单	`src/data/contributors.json`	None

Step Status Values

Each module execution maps to one SyncJobStep. The step status reflects the outcome of that single table’s sync.

Status	Description
`pending`	Not yet started
`running`	Module actively fetching records and processing assets
`success`	All records processed without errors
`success_with_warnings`	Partial record failures; output artifact was written with available data
`skipped`	Skipped because a required dependency was missing, or the job was aborted before this step
`failed`	Fatal error; output artifact was not written

Dependency Mode

When a table depends on another (e.g. stories requires locations and characters), the orchestrator uses the dependencyMode to decide how to satisfy those dependencies.

Mode	Behaviour	When to use
`read_local`	(Default) Use the existing on-disk artifact for dependencies; only run the selected tables	Lightweight refresh of one table when dependencies are already up to date
`run_dependencies`	Automatically prepend dependency tables to the execution plan	Full update where the dependency chain may have changed
`strict`	Fail immediately if a dependency artifact is not present on disk	Validation environments where stale data is unacceptable

Core Type Definitions

// src/lib/sync/types.ts

export type SyncJobKind = 'sync-data';

export type SyncTableKey = keyof SyncTableOutputMap;

export interface SyncTableOutputMap {
  locations:        LocationCoords;
  characters:       Character[];
  stories:          LocationPoint[];
  creation_headers: CardHeaderInfo[];
  creation_board:   CreationIdea[];
  contributors:     Contributor[];
}

export type DependencyMode = 'read_local' | 'run_dependencies' | 'strict';

export interface SyncJobRecord {
  jobId: string;
  kind: SyncJobKind;
  status: 'queued' | 'running' | 'success' | 'partial_success' | 'failed' | 'canceled';
  tables: SyncTableKey[];
  effectiveTables: SyncTableKey[];   // tables after dependency resolution
  dependencyMode: DependencyMode;
  includeAssets: boolean;            // whether to run the OSS image pipeline
  continueOnTableError: boolean;     // whether to keep going after a step failure
  triggeredBy: 'admin-ui' | 'scheduler' | 'cli' | 'webhook';
  note?: string;
  createdAt: string;
  startedAt?: string;
  syncedAt?: string;
  finishedAt?: string;
  durationMs?: number;
  steps: SyncJobStep[];
  summary?: Record<string, unknown>;
  warnings: string[];
  errors: string[];
}

export interface SyncJobStep {
  step: string;     // SyncTableKey value
  status: 'pending' | 'running' | 'success' | 'success_with_warnings' | 'skipped' | 'failed';
  startedAt?: string;
  finishedAt?: string;
  summary?: {
    totalRecords?: number;
    successRecords?: number;
    skippedRecords?: number;
    failedRecords?: number;
    filesWritten?: string[];
    [key: string]: unknown;
  };
  warnings?: string[];
  errors?: string[];
}

OSS Image Pipeline

Image assets in Feishu are private attachments. The sync system downloads each attachment and re-hosts it on Aliyun OSS so the frontend can render them without Feishu credentials.

Detect attachment

The table module reads the attachment field (e.g. 角色头像, 大图, 请上传你的图片) from each Feishu record. If the corresponding OSS URL field is already populated, this record is skipped.

Download from Feishu

src/lib/sync/clients/feishu-drive-client.ts fetches the raw binary using the Feishu Drive API with the app’s OAuth token. The binary is held in memory (no temp files).

Compute MD5 for deduplication

src/lib/sync/shared/asset-processor.ts computes an MD5 hash of the binary content. The OSS object key is derived from this hash, so identical images are never uploaded twice.

Upload to Aliyun OSS

src/lib/sync/clients/oss-client.ts uploads the binary to the configured bucket. The resulting public URL follows the pattern https://<bucket>.<region>.aliyuncs.com/<md5>.<ext>.

Back-fill URL to Feishu

The orchestrator calls the Feishu Bitable update API to write the OSS URL back into the source record (e.g. 头像OSS_URL, 大图OSS_URL). On the next sync run, the OSS URL field is already populated and the download step is skipped.

The back-fill write requires the Feishu app to have Bitable record edit permission. If this permission is missing, the pipeline will still generate the OSS URL and write the JSON artifact, but the URL will not be persisted in Feishu — meaning the download and upload will repeat on every sync run.

File Write Strategy

All artifact files are written atomically to prevent corrupt reads if the process crashes mid-write.

Serialize data to JSON string
Write to <target>.tmp
Validate the .tmp file is parseable JSON
fs.rename(<target>.tmp, <target>)  ← atomic on POSIX

Empty-result guard: If Feishu returns zero records for a table, the existing artifact on disk is not overwritten. A warning is recorded on the job step and execution continues. This prevents a Feishu API outage from erasing production data.

Runtime Directory Layout

runtime/                          # SYNC_RUNTIME_DIR (not committed)
└── admin/
    ├── sync-config.json          # Scheduler cron + defaultTables config
    ├── sync-lock.json            # Single-instance concurrency lock
    └── sync-jobs/
        └── <jobId>.json          # One file per job (max 100, auto-purged)

logs/                             # SYNC_LOG_DIR (not committed)
└── sync/
    └── <jobId>.log               # Per-job log file

src/data/                         # Build-time artifacts (committed)
├── content.json
├── characters.json
├── creation-board.json
├── creation-board-headers.json
└── contributors.json

src/config/                       # Build-time config (committed)
└── locations.json

Environment Variables

Variable	Default	Description
`SYNC_RUNTIME_DIR`	`runtime/`	Root directory for job records and lock file
`SYNC_LOG_DIR`	`logs/sync/`	Directory for per-job log files
`SYNC_LOCK_TIMEOUT_MS`	`1800000`	Lock expiry in milliseconds (30 minutes)
`SYNC_WEBHOOK_SECRET`	(none)	Secret token for authenticating webhook-triggered syncs

runtime/ and logs/ are listed in .gitignore and should never be committed. The src/data/ and src/config/ artifacts are committed — they serve as the canonical build-time data and as fallback content for environments where no sync has run.

Source Module Directory

src/lib/sync/
├── types.ts                      # All type definitions (SyncJobRecord, SyncTableKey, …)
├── orchestrator.ts               # Job creation, execution, status management
├── job-store.ts                  # File-based job record persistence
├── lock.ts                       # File lock acquire / release / expire
├── config.ts                     # Read / write sync-config.json
├── paths.ts                      # Runtime path resolution from env vars
├── logger.ts                     # Per-job structured log writer
├── registry.ts                   # tableRegistry — module registration
├── clients/
│   ├── feishu-auth-client.ts     # OAuth token management
│   ├── feishu-bitable-client.ts  # Bitable read + record update
│   ├── feishu-drive-client.ts    # Attachment download
│   └── oss-client.ts             # Aliyun OSS upload
├── shared/
│   ├── asset-processor.ts        # MD5 computation + OSS dedup logic
│   ├── field-reader.ts           # Feishu field extraction utilities
│   ├── text.ts                   # String normalisation helpers
│   ├── dates.ts                  # Timestamp parsing helpers
│   └── files.ts                  # Atomic write helpers
├── tables/
│   ├── locations.module.ts
│   ├── characters.module.ts
│   ├── stories.module.ts
│   ├── creation-headers.module.ts
│   ├── creation-board.module.ts
│   └── contributors.module.ts
└── writers/
    ├── json-writer.ts            # Shared atomic JSON writer
    ├── locations.writer.ts
    ├── characters.writer.ts
    ├── content.writer.ts
    ├── creation-board.writer.ts
    └── creation-board-headers.writer.ts

Adding a New Feishu Table

To register a new Feishu table in the sync system, follow these steps in order:

Extend SyncTableOutputMap

In src/lib/sync/types.ts, add your new key and its output type. SyncTableKey updates automatically.

export interface SyncTableOutputMap {
  // ... existing entries ...
  my_new_table: MyNewType[];
}

Define the domain type (if new)

If MyNewType is a new entity, define it in src/lib/types.ts and import it into src/lib/sync/types.ts.

Implement the table module

Create src/lib/sync/tables/my-new-table.module.ts implementing TableSyncModule<'my_new_table'>.

Implement the writer (optional)

If the output format requires custom serialisation, create src/lib/sync/writers/my-new-table.writer.ts. Otherwise use the shared json-writer.ts.

Add the module to tableRegistry in src/lib/sync/registry.ts.

export const tableRegistry = {
  // ... existing entries ...
  my_new_table: myNewTableModule,
} satisfies Record<SyncTableKey, TableSyncModule>;

Update defaultTables (optional)

If the new table should sync automatically on the scheduler, add "my_new_table" to the defaultTables array in runtime/admin/sync-config.json.

Get Started

Architecture

Design

Guides

HNU-TimeLetter Feishu Data Sync System Architecture

Three-Layer Architecture

Layer 1 — Trigger Layer

Layer 2 — Orchestrator Layer

Job Lifecycle

File-Based Lock

Job Persistence

Layer 3 — Module Layer

SyncTableKey Registry

Table Registry Reference

Step Status Values

Dependency Mode

Core Type Definitions

OSS Image Pipeline

File Write Strategy

Runtime Directory Layout

Environment Variables

Source Module Directory

Adding a New Feishu Table

Build docs developers (and LLMs) love

Get Started

Architecture

Design

Guides

Documentation Index

​Three-Layer Architecture

​Layer 1 — Trigger Layer

​Layer 2 — Orchestrator Layer

​Job Lifecycle

​File-Based Lock

​Job Persistence

​Layer 3 — Module Layer

​SyncTableKey Registry

​Table Registry Reference

​Step Status Values

​Dependency Mode

​Core Type Definitions

​OSS Image Pipeline

​File Write Strategy

​Runtime Directory Layout

​Environment Variables

​Source Module Directory

​Adding a New Feishu Table

Build docs developers (and LLMs) love

Three-Layer Architecture

Layer 1 — Trigger Layer

Layer 2 — Orchestrator Layer

Job Lifecycle

File-Based Lock

Job Persistence

Layer 3 — Module Layer

SyncTableKey Registry

Table Registry Reference

Step Status Values

Dependency Mode

Core Type Definitions

OSS Image Pipeline

File Write Strategy

Runtime Directory Layout

Environment Variables

Source Module Directory

Adding a New Feishu Table