Queuing with BullMQ

BullMq is a robust queue system built on top of Redis BullMq allows to delay jobs, chain them, repeat and having failing strategies. Of course, voca use bullMq quiet intensively to build its Decidim deployments.

BullMq

Before going anyway further, please be sure to have read the BullMq documentation. There are some very important informations there:

• https://docs.bullmq.io/guide/queues
• https://docs.bullmq.io/guide/workers
• https://docs.bullmq.io/guide/flows/fail-parent

BullMq in Strapi

BullMq can be run within any node environment, and have not been specially designed for Strapi. We implement BullMQ queues worker in order to be able to keep Strapi Magics on, and enjoy bullMQ task processing.

UI to manage jobs

To illustrate the jobs you have, go and see your bullboard. This is a web interface, served by strapi. It will help us to have observability on jobs. To access the bullboard:

• url: https://<strapi_url>/_jobs
• username: BULLMQ_USERNAME
• password: BULLMQ_PASSWORD

In this interface, you will see in the UI different kind of jobs, in different states:

And each job other a small interface to access it’s information:

• data: The data we send to the job
• jobName: the uniq identifier representing the job implementation
• options: job options, like delay, number of retry, backoff strategies, etc.
• logs: a log screen for the job logs.
• progress: a number representing the progress of the job. This information is very useful to debug a particular job.

Enqueue tasks

In voca we use two BullMq interfaces: Queues and Flow. Queues are made to execute a single job, Flows are made to execute a sequence of jobs, with a relation parent-child.

To retrieve a queue or a flow, we use a configuration object (defined in backend/config/bullMq.ts):

• strapi.config.get("bullMq.mainQueue") for fast jobs
• strapi.config.get("bullMq.installQueue") for slow jobs
• strapi.config.get("bullMq.flow") for a flow
  • ℹ️ you will need to define the queue name later using strapi.config.get("bullMq.MAIN_QUEUE") or strapi.config.get("bullMq.INSTALL_QUEUE")

Once you get a queue, you will be able to add jobs:

strapi.config.get("bullMq.mainQueue").add({name: "my-awesome-job", data: {foo: "bar"})

As you can see in the exemple, BullMq will enqueue only the job name and options. The BullMq worker will then need a mapping between job name and implementation. We do define this mapping in the folder backend/src/jobs/index.ts

// Exemple of mapping
import successMail from './success-mail';


/**
 * Decidim related jobs
 */
export default {
	// you can then add: queue.add({name: "decidim::success-mail", ...})
  'decidim::success-mail': successMail,
};

To know which are the job names and related implementation faster, we have two configuration keys:

• strapi.config.get("bullMq.AVAILABLE_JOBS") an object with the jobName as key, the implementation as value
• strapi.config.get("bullMq.AVAILABLE_JOB_NAMES") a list of all the job names availables

The worker implementation is then trivial, we can take back the job name and run the related implementation. The only thing is to respect the same job signature:

const successMail: (job: Job, strapi: Strapi) => Promise<any>
// job.id => id of bullMq job
// job.data => the data passed

Chained Flows

We use in voca Flows that are actually just a linked list of dependancies. As Jelastic don’t handle well concurrent operation on the same environment, we want to be sure to execute jobs one after the other. Thus, we chain flow in a very simplistic manner (jobs have max. 1 child).

To make this flow more human readable, we have an utility function, named linkedFlow (backend/src/api/decidim/content-types/linked-flow.ts)

// Enqueue "infra::park::deploy", "infra::park::postDeploy", and finally "infra::park::mounted"
flow.add(
  linkedFlow(
    {name: 'infra::park::mounted', ...queueOption},
    [
      'infra::park::postDeploy',
      'infra::park::deploy',
    ],
    childQueueOptions
  )
);

Run BullMq worker

About long-running jobs

https://docs.bullmq.io/guide/jobs/stalled

https://docs.bullmq.io/bull/important-notes

🚧 This section is a work in progress