The initial problem
It isn't uncommon to use cron to run recurring tasks, it's not even a particularly bad idea. I've seen it before, albeit that one wasn't a very sophisticated solution - they had three different scripts, an hourly, a daily and a weekly script, and three corresponding entries in the crontab.
But for this application, I wanted something better. I wanted to prevent the scripts growing too big and I wanted it to be more configurable and more easily extendable. I also needed something that was not super foreign to our existing stack.
The kind of tasks or jobs that would have to run in the background were of all sorts of nature - there were email sending utilities, report and invoice generating utilities and other things that are common parts of a business management system. So how do we automate this?
Approach
This was a Yii2 application, so I immediately thought about the Yii console application. It is similar to some other command solutions in other framworks, it lets you execute custom scripts through the command line, while still providing access to framework utilities such as your DI container, a database connection and other things.
I also needed a message queue.
For Yii2, you can use the yii2-queue package to push messages, and it supports a variety of database drivers. We already had PosgreSQL for our relation data needs, and we used Redis for caching. I didn't want to introduce an additional service at this point because we did not expect that many messages just yet - but thanks to the library, we could have switched drivers later if we wanted to. For this reason, I decided to use Redis to serve as our message queue storage solution.
Solution
Our job system had to be configurable. We had different types of jobs, some of which had to run more frequently, some others less frequently, or sometimes not at all. Administrators had to be able to disable them on demand.
I created a table for these configurations, where the Admin could decide how often we would have to run certain tasks, or even configure particular days or times of the month for highly specific jobs - such as monthly billing, contracts and other things.
What I've done is run the console application which was checking the configuration, going over each of our job types and pushing the necessary messages to the queue.
Then the yii2-queue library came in to play, which uses yii queue/run (for cron) or yii queue/listen (for daemonized processes) to act as a consumer/worker, and it executed the jobs waiting in the queue.
This is what it looked like on a high level
Base on this diagram, it may not be obvious why we needed a message queue to begin with, we could have just executed the tasks through the console application! Right?
Well, the message queue was very helpful for us to push on-demand tasks from the main application to the queue as well, such as an ad-hoc email or report generation.
Results
The product owner was happy with the solution because we didn't need to introduce any new service we didn't already have, and we could swap the message queue driver in the future if we need a more scalable solution - though Redis can go a long way!
Thanks to the yii queue/info command, it was also easy to set up monitoring on the production server.
The Yii2 console application implementation could also be reused later. Background jobs was one of the many things I ended up using it for, it was also very useful to create parameterized commands to execute migrations on different - or all - databases.
Thank you for reading!