Skip to main content

​
Design Principles

1

Start simple

Begin with a basic Input → Output flow. Add transformation nodes one at a time and test after each addition.
2

Always test with real data

Use the test panel to load a real job from your feed and verify each node’s output before attaching to a live feed. The test input is exactly the document the pipeline receives during a sync.
3

Filter before you transform

Put Filter nodes right after Input. Every node downstream only runs for jobs that pass — with AI nodes, this directly cuts your OpenRouter bill.
4

Prefer Field Mapper and Liquid for deterministic transforms

Renames and reshapes don’t need an LLM. The Field Mapper covers most schema mapping; Liquid covers conditionals and formatting. Both are free and instant.
5

Preserve original data with JSON Merge

Don’t replace the whole document with raw AI output. Branch from Input to both the AI node and a JSON Merge node, so original fields are always preserved.
6

Use output schemas for AI nodes

Always define output schemas for AI nodes — they’re enforced via strict structured outputs, making downstream merging reliable.
7

Start from templates

The template picker offers pre-built pipelines for common use cases — including a zero-AI Filter + Field Mapper template that runs with no setup. Start there and customize.

​
Performance & Cost

AI nodes are the dominant source of both latency and cost: one LLM call per synced job, per AI node, billed to your OpenRouter key. The first sync after attaching runs your entire feed through the pipeline. Estimate before attaching: jobs in feed × AI nodes × cost per call.

​
AI Timeout & Retry Behavior

  • Each AI call has a 60-second timeout
  • Transient failures (timeouts, network errors, 5xx, rate limits) are retried up to 2 times with short backoff
  • If retries are exhausted, the job is skipped and counted as a transform failure — the sync continues
  • An invalid key or exhausted credits (401/402/403) fails the whole sync immediately with an actionable error
  • This behavior is automatic and not configurable

​
What happens to failed jobs?

Syncs are incremental: a job that failed transformation is not automatically retried on the next cycle — it’s only reprocessed when its content changes. After fixing a broken prompt or template, use Full resync on the destination card to re-run the whole feed.

​
AI Node Tips

  • Keep prompts specific — extract or transform specific fields rather than asking the model to process the entire job object
  • Include examples in the system prompt to improve output quality
  • Use the smallest/fastest model that produces acceptable quality
  • Always set an output schema — it’s enforced with strict structured outputs
  • Feed the AI a focused context via a Liquid or Field Mapper node first to cut token usage

​
JSON Merge Tips

  • Use deep merge when enriching nested objects (skills, requirements, metadata)
  • Use shallow merge when adding top-level fields only
  • Use patch wins when AI output should override original data
  • Use base wins when original data should take precedence

​
Common Patterns

​
Filter → Map

The cheapest useful pipeline — no AI at all. Sync only matching jobs, reshaped to your schema:

​
Branch and Merge

Send Input to both an AI node and a JSON Merge node. The AI extracts new fields, and JSON Merge combines them with the original data:

​
Filter → Prepare → Transform → Merge

Filter first so the AI only sees jobs you’ll actually sync, and prepare a focused context to reduce token usage:

​
FAQ

Yes. Each AI node receives the output of its upstream node. This is useful for multi-step extraction (e.g. extract → classify → summarize) — but remember each AI node multiplies per-job cost and latency.
Elasticsearch, Algolia, and Meilisearch. PostgreSQL has a fixed table schema and can’t store arbitrary transformed shapes, so pipelines can’t be attached to it.
Each call has a 60-second timeout and transient failures are retried twice. If retries are exhausted, that job is skipped and counted in the push log; the sync continues. Invalid keys or exhausted credits fail the whole sync immediately.
Filter, Field Mapper, Liquid, and JSON Merge nodes are effectively free. AI nodes add seconds per job; jobs are transformed concurrently to keep overall sync time reasonable.
Syncs are incremental and only process new or changed jobs. Use Full resync on the destination card to re-run the entire feed through the updated pipeline.
Pipelines run during feed syncs. You can build and test them standalone in the editor’s test panel, but production execution requires attaching them to a feed destination.
Pipelines are included with your subscription at no extra cost. AI nodes use your own OpenRouter API key — you pay OpenRouter directly for LLM usage.
No — detach it from the destination first. The editor shows which feeds a pipeline is attached to on the pipeline list page.