Process pipelines
One of the central components of the Pacemaker is the ProcessPipelines module.
The concept behind this module is the Pipeline Design Pattern.
The Pattern
To describe this pattern in short:
You need to split up each process into multiple stages. Each stage has at least one step.
While the stages have a precise sequence, the steps inside these stages can run in parallel or random order, since they do not depend on each other, but on the stage.
It requires a decoupling of the execution and the organization of these steps.
A runner-driven architecture is used to fulfill this requirement (SPMD).
Realization
In the
installation
section, you already touched the both major parts of ProcessPipelines, while
configuring the heartbeat cron job
(which is the organizational unit) and
configuring the pipeline runners
(which are the execution unit).
Heartbeat and Runner
In the following image, you can see the responsibilities of both units.
The heartbeat is responsible for the initialization of new pipelines and publishing messages for the runner if a step can be executed. And since the status of a pipeline is an aggregation of all status of its steps, the heartbeat is also responsible for this aggregation.
The runner performs every step that is in the queue but does not care about its status or anything else. It gives us high scalability since these runners can run on different or multiple machines.
Configuration and Process Execution
The pipelines are configured in XML files, which are merged between the modules.
It happens in the same way you already know from Magento’s configuration XMLs (e.g. config.xml, di.xml).
|
Pipeline Initializer
Pipelines can also be defined by code (without XML files) if you need to create pipelines dynamically. Read the Pipeline initializer documentation for details. |
-
Please refer to the How to configure a processPipeline page.
A pipeline configuration has at least one condition, which will be checked periodically by the heartbeat.
If a pipeline is ready to be spawned, the heartbeat creates a new pipeline in the database.
-
Please refer to the How to create a pipeline condition page.
Every step within a pipeline has at least one condition, which will be checked periodically by the heartbeat.
If a step is ready to be executed, the heartbeat pushes an execution message to the RabbitMQ.
-
Please refer to the How to create a step condition page.
The next free runner will execute every message in the RabbitMQ.
|
The amount of runners is at the same time the limiting factor for parallel execution. The runner instantiates and executes executor class. |
-
Please refer to the How to create an executor page.