Skip to content

Basic Concepts

Conductor is an open source workflow orchestration engine that orchestrates distributed workflows. You define workflows as code or as JSON, write workers in any language, and let Conductor handle state persistence, retries, timeouts, and flow control. Every step is durably recorded, so processes survive crashes, restarts, and network partitions without losing progress.

Workflow definitions are JSON-native — you can version them in source control, diff changes across releases, generate them programmatically, or let LLMs create and modify them at runtime. Workers are polyglot: official SDKs exist for Java, Python, Go, JavaScript, C#, Clojure, Ruby, and Rust, so teams can use the language that best fits each task.

Built-in system tasks handle common operations like HTTP calls, event publishing, inline transforms, and sub-workflow orchestration without writing custom code. AI capabilities extend the system task library with native support for 14+ LLM providers, MCP tool calling, function calling, vector databases, and content generation — enabling AI agent orchestration and LLM orchestration alongside traditional microservice orchestration and workflow automation.

What can Conductor do?

Start Task A Task B Task C End Start Task A SwitchCase Task B Task C End Start Task A SwitchCase Task B Do While Loop Task C End Start Task A SwitchCase Task B Task D Do While Loop Task C End Start Task A Worker AMicroservice Task B Worker BServerless Task C Worker CLegacy App End Start HTTP: Call API endpoint Event: Write to Kafka Inline: Execute JS End Start Search News Index Get Contextual Answer End Start Switch DefaultApproval HumanApproval End Start Task A ! Retry on failure Task B ⏱ Timeout afterx seconds Task C On failure End Start Task A Task B Task C FAILED Restart Rerun Retry ConductorWorkflow Engine Kafka NATS SQS AMQP Webhooks Start Task A Switch Task D View inputs, pull logs, restart from here End Load Balancer Instance 1API Server+ Sweeper Instance 2API Server+ Sweeper Shared Backends Database · Queue · Index · Lock

Core building blocks

  • Workflows — The blueprint of a process flow. A workflow is a JSON document that describes a directed graph of tasks, their dependencies, input/output mappings, and failure handling policies.
  • Tasks — The basic building blocks of a Conductor workflow. Tasks can be system tasks (executed by the engine) or worker tasks (executed by external workers polling for work).
  • Workers — The code that executes tasks in a Conductor workflow. Workers are language-agnostic processes that poll the Conductor server, execute business logic, and report results back.

Key differentiators

These are the facts that matter when comparing workflow and orchestration engines:

  • Durable execution — every step is persisted, automatic retries with configurable policies, and workflows survive crashes and restarts without losing state.
  • Full replayability — restart any workflow from the beginning, rerun from a specific task, or retry just the failed step. Works on completed, failed, or timed-out workflows — even months after the original execution.
  • Deterministic execution — JSON definitions separate orchestration from implementation. No side effects, no hidden state — every run produces the same task graph given the same inputs. Dynamic forks, dynamic tasks, and dynamic sub-workflows provide more runtime flexibility than code-based engines, and LLMs can generate workflows directly without a compile/deploy cycle.
  • 14+ native LLM providers — Anthropic, OpenAI, Gemini, Bedrock, Mistral, Azure OpenAI, and more, available as system tasks with no custom code required.
  • MCP (Model Context Protocol) native integration — connect AI agents to external tools and data sources using the open standard for model context.
  • 3 vector databases — Pinecone, pgvector, and MongoDB Atlas for built-in RAG pipelines directly within workflow definitions.
  • 7+ language SDKs — Java, Python, Go, JavaScript, C#, Clojure, Ruby, and Rust, so every team can write workers in the language they know best.
  • 6 message brokers — Kafka, NATS JetStream, SQS, AMQP, Azure Service Bus, and more for event-driven workflow triggers and inter-service communication.
  • 8+ persistence backends — PostgreSQL, MySQL, Redis, Cassandra, Elasticsearch, MongoDB, and others, letting you run Conductor on the infrastructure you already operate.
  • Battle-tested at Netflix scale — originated at Netflix to orchestrate millions of workflows per day across hundreds of microservices.

Deep dives