To describe this pattern in short: You need to split up each process into multiple stages. Each stage has at least one step. While the stages have a clear sequence, the steps inside these stages can run in parallel or in random order, since they are not depending on each other, but on the stage.

This requires a decoupling of the execution and the organization of these steps. Usually, a runner-driven architecture is used in order to fulfill this requirement (SPMD). You’ll find such integrations in build- and deployment tools like Jenkins or GitLab.

In the Get Started section, you already touched the both major parts of ProcessPipelines, while configuring the heartbeat cron job (which is the organizational unit) and configuring the pipeline runners (which are the execution unit).

In the following image, you can see the responsibilities of both units.

The heartbeat is responsible for the initialization of new pipelines and publishing of messages for the runner if a step can be executed. And since the status of a pipeline is an aggregation of all status of its steps the heartbeat is also responsible for this aggregation.

The runner executes each step, which is in the queue and does not take care of its status or anything else. This gives us great scalability since these runners can run on different or multiple machines.

The pipelines are configured in XML files, which are merged between the modules. This happens in the same way as you already know from Magento’s configuration XMLs (e.g. config.xml, di.xml).

Every pipeline configuration has at least one condition, which will be checked periodically by the heartbeat. If a pipeline is ready to be spawned the heartbeat creates a new pipeline in the database.

Every step within a pipeline has at least one condition, which will be checked periodically by the heartbeat. If a step is ready to be executed the heartbeat pushes an execution message to the RabbitMQ.

Every message in the RabbitMQ will be executed by the next free runner. This means that the amount of runners is at the same time the limiting factor for parallel execution. The runner instantiates and executes executor class.