temporal: Workflow Orchestration Engine
Temporal Overview
- workflow engine that handles state management for workflows
- split into steps called “activities'
- Provides an API for
- registering workers that can run Temporal activities
- launch asynchronous workflows
- send signals to running workflows
- query state for a workflow.
Compared to a batch-oriented workflow engine such as Airflow, many of the core concepts have matching analogues for DAGs, sub-DAGs, and sensors.
The SDKs (available for different languages) allow expressing the concept of workflows (DAGs in Airflow), which execute some activities in a defined order. This is the entry point of any Temporal run. Workflows can receive signals and perform queries, which will be explained below. Worker nodes communicate with the Temporal server to determine which workflows should execute on an individual node. This is coordinated by assigning certain workflows on creation to named task queues which can be registered on individual nodes.
Activities (tasks in Airflow) are steps within a Temporal workflow that represent some operation or operations. These operations can contain any logic, but since they represent retryable units of work, they are often used to separate behavior such as calls to an external database or API. The output of an activity is serialized and stored by the backing data store of Temporal along with any relevant status information for the activity and workflow.
It is also possible to have child workflows (subDAGs in Airflow) that are workflows launched by other workflows. It is possible to configure the behavior of child workflows to fail or continue based on if the parent workflow fails or gets canceled.
There are two methods that allow interacting with workflows: signals and queries. Signals allow modifying the Temporal in memory state for a workflow. These methods are asynchronous and are not guaranteed to run immediately. If a Temporal workflow is depending on a condition modified by the signal, it will intelligently re-evaluate the condition to decide if a workflow should continue. Queries simply allow reading the state for a workflow. We usually use this feature to inspect the state of long-running workflows.
Temporal makes even long-running “wait” operations efficient by caching the state in a cold storage and only loading it in memory when needed.
The Java and Python SDKs provide a set of annotations used to decorate an interface that will let Temporal know how to run the different methods we are writing.
On my own behalf
The above overview is a shamesless but curated copy from various sources found using google.