badass django with celery 🚬
In this article we will see how to schedule periodic task, like sending mail to managers every 2 hours or sending mail at 4 am daily
Before getting started let us understand some of Jargons that you’ll hear in this article, for detailed information please do Google search on them
Celery : We will use this to execute or asyncronous tasks
Celery Beat : Celery beat is used to schedule our task
RabbitMQ : This is used as message broker (in simple terms — forms links) to send task to celery
Redis : This works same as RabbitMQ, another message broker
Cronjob : Cron is a utility that schedule scripts or commands to be run automatically at specified time and date or intervals and these tasks or jobs are what we call as cron job
Crontab : It is a list of commands that you want to run on a regular schedule, mostly we will be using this to setup our task to run at every x interval, for detail on this follow this link of Crontab
Note — We can run scheduled task through either with cronjob or celery, but Celery gives are more control and that’s why it is mostly preferred with django
Now enough of talking lets start scheduling our task in django
Step 1 : Install Celery by running below command, and don’t forget to add that in requirements.txt (optional)
pip install celeryto verify installation of celery run command
pip show celeryit will show you output like this, if you get ouput same as below, you can move to step 2
Name: celery
Version: 4.4.6
Summary: Distributed Task Queue.
Home-page: http://celeryproject.org
Author: Ask Solem
Author-email: auvipy@gmail.com
License: BSD
Location: /home/my_pc/.local/lib/python3.6/site-packages
Requires: vine, billiard, kombu, pytz, future
Required-by:Step 2 : Create a file name as celery.py alongside with the settings.py and urls.py files as illustrated below, if you have created setting package then put this file inside settings package
my_app/
|-- my_app/
| |-- core/
| | |-- migrations/
| | |-- templates/
| | |-- apps.py
| | |-- models.py
| | +-- views.py
| |-- templates/
| |-- __init__.py
| |-- settings.py
| |-- celery.py <-- celery file goes here
| |-- urls.py
| +-- wsgi.py
|-- manage.py
+-- requirements.txtnow add the code below in celery.py file, read comments for better understanding
import os from celery import Celery
from os.path import dirname, basename# To set which setting you want to use by default
os.environ.setdefault('DJANGO_SETTINGS_MODULE','my_app.settings')# to initalize our Celery app
app = Celery(basename(dirname(__file__))) # to inform celery load settings which are specific to celery
app.config_from_object('django.conf:settings', namespace='CELERY') # to tell Celery that it should use the settings.INSTALLED_APPS list of applications to find defined Celery tasks.app.autodiscover_tasks()
Now we have to edit the init file shown below
my_app/
|-- my_app/
| |-- core/
| | |-- migrations/
| | |-- templates/
| | |-- apps.py
| | |-- models.py
| | +-- views.py
| |-- templates/
| |-- __init__.py <-- edit this init file
| |-- settings.py
| |-- celery.py
| |-- urls.py
| +-- wsgi.py
|-- manage.py
+-- requirements.txtAdd the code below in init file
from .celery import app as celery_app # This will make sure the app is always imported when
# Django starts so that shared_task will use this app.__all__ = ['celery_app']
Step 3 : Now we will add code which will be executed by our scheduler, create a file name as tasks.py shown below
my_app/
|-- my_app/
| |-- core/
| | |-- migrations/
| | |-- templates/
| | |-- apps.py
| | |-- tasks.py <-- create this file here
| | |-- models.py
| | +-- views.py
| |-- templates/
| |-- __init__.py
| |-- settings.py
| |-- celery.py
| |-- urls.py
| +-- wsgi.py
|-- manage.py
+-- requirements.txtNow add the code which you want scheduler to run, i have added the sample code below, our sample code update first user in database
# change import here according to your project name
from my_app.celery import app # <-- ⚡⚡⚡⚡⚡⚡⚡⚡ change here
from .models import User
import random@app.task()
def update_user():
dummy = str(random.randint(100, 1000))
User.objects.filter(id=1).update(username=dummy)
Step 4 : Now we will create schedule for this task to run at every 10 seconds, So now create a file name as celery_config.py file as show in below directory
my_app/
|-- my_app/
| |-- core/
| | |-- migrations/
| | |-- templates/
| | |-- apps.py
| | |-- tasks.py
| | |-- models.py
| | +-- views.py
| |-- templates/
| |-- __init__.py
| |-- settings.py
| |-- celery.py
| |-- celery_config.py <-- create your file here
| |-- urls.py
| +-- wsgi.py
|-- manage.py
+-- requirements.txtadd code below in celery_config.py file
CELERY_BEAT_SCHEDULE = {
# name for your scheduler
'update_my_user': {
# path for your task
'task': 'my_app.core.tasks.update_user',
# 10 defined here is in seconds
'schedule': 10,
},
}and don’t forget to import this file in your settings.py file as shown below
from .celery_config import *Step 5 : Here we will use redis as our message broker, which will be responsible for sending tasks to celery at periodic interval
Download redis from its offical page => Download
after downloading fire up server by opening terminal and execute
redis-serverYou can test that Redis is working properly by typing this into your terminal
redis-cli pingRedis should reply with PONG , You also need to add Redis as a dependency in the Django Project (optional), after doing this add the following setting to your setttings.py file
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_TIMEZONE = 'UTC'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'Step 6: Now finally we are done, we need to test our scheduler by running command below, run below commands in 2 separate terminal
celery -A my_app worker -l infohere,
- -A denote for app name which is added next to it in my case it is my_app, change accordingly your app name
- worker is what we want to run
- -l denote level of log we want, we are keeping this as info
In next terminal run
celery -A my_app beat -l infochange my_app to your app name, this command is used to beat schedule at specific time defined in celery_config.py
Finally if you didn’t get any error you should be able to see username changing for user with primary_key 1, every 10 seconds 🎉🎉🎉
Conclusion : Setting up celery is bit tricky make sure, you cross check settings twice after implementing steps from this article, and don’t add files in wrong directory, then you should be able to execute schedule task properly
Check out my article on :
