Skip to content

Pipelines & automation

Custom pipeline assembly

Compose your own pipeline from typed, validated steps on a canvas, and run it as one recorded job.

The named pipelines cover the common routes through an analysis, but a real project has its own order: pick, fit, transfer an assignment, then measure rates and take the result to a tensor. Chaining those by hand means re-uploading intermediate files and keeping track of which output went where, which is exactly the bookkeeping that gets lost.

Workflow Assembly gives you a canvas of blocks. Each block declares the types of what it consumes and produces, and the canvas refuses to connect ports of different types, so a chain that cannot work cannot be drawn. The platform applies the same rules again when you submit, rather than trusting the browser, then resolves the graph into a plan.

The plan runs as a single durable job — one run, one place to watch it, one set of results — instead of a string of separate submissions. Because it uses the platform's own steps, what comes out carries the same provenance as a result from any named pipeline.

What it needs

  • Datasets already stored on your account
  • A graph of typed blocks

What you get

  • One recorded run covering every step
  • Results filed and linked exactly as the named pipelines file theirs

What does the work

  • Temporal

    Open source, redistributed under its licenceMIT

    Durable execution of the assembled chain