Process pipelines

One of the central components of the Pacemaker is the ProcessPipelines module.

The concept behind this module is the Pipeline Design Pattern.

The Pattern

To describe this pattern in short:

You need to split up each process into multiple stages. Each stage has at least one step.

While the stages have a precise sequence, the steps inside these stages can run in parallel or random order, since they do not depend on each other, but on the stage.

It requires a decoupling of the execution and the organization of these steps.

A runner-driven architecture is used to fulfill this requirement (SPMD).

You will find such integrations in build- and deployment tools like Jenkins or GitLab.


In the installation section, you already touched the both major parts of ProcessPipelines, while configuring the heartbeat cron job (which is the organizational unit) and configuring the pipeline runners (which are the execution unit).

Heartbeat and Runner

In the following image, you can see the responsibilities of both units.

pe pipeline architecture
Figure 1. Heartbeat and Runner

The heartbeat is responsible for the initialization of new pipelines and publishing messages for the runner if a step can be executed. And since the status of a pipeline is an aggregation of all status of its steps, the heartbeat is also responsible for this aggregation.

The runner performs every step that is in the queue but does not care about its status or anything else. It gives us high scalability since these runners can run on different or multiple machines.

Configuration and Process Execution

The pipelines are configured in XML files, which are merged between the modules. It happens in the same way you already know from Magento’s configuration XMLs (e.g. config.xml, di.xml).

Pipeline Initializer

Pipelines can also be defined by code (without XML files) if you need to create pipelines dynamically.

Read the Pipeline initializer documentation for details.

A pipeline configuration has at least one condition, which will be checked periodically by the heartbeat.

If a pipeline is ready to be spawned, the heartbeat creates a new pipeline in the database.

Every step within a pipeline has at least one condition, which will be checked periodically by the heartbeat.

If a step is ready to be executed, the heartbeat pushes an execution message to the RabbitMQ.

The next free runner will execute every message in the RabbitMQ.

The amount of runners is at the same time the limiting factor for parallel execution.

The runner instantiates and executes executor class.