badass django with celery 🚬

In this article we will see how to schedule periodic task, like sending mail to managers every 2 hours or sending mail at 4 am daily

Before getting started let us understand some of Jargons that you’ll hear in this article, for detailed information please do Google search on them

Celery : We will use this to execute or asyncronous tasks

Celery Beat : Celery beat is used to schedule our task

RabbitMQ : This is used as message broker (in simple terms — forms links) to send task to celery

Redis : This works same as RabbitMQ, another message broker

Cronjob : Cron is a utility that schedule scripts or commands to be run automatically at specified time and date or intervals and these tasks or jobs are what we call as cron job

Crontab : It is a list of commands that you want to run on a regular schedule, mostly we will be using this to setup our task to run at every x interval, for detail on this follow this link of Crontab

Note — We can run scheduled task through either with cronjob or celery, but Celery gives are more control and that’s why it is mostly preferred with django

Now enough of talking lets start scheduling our task in django

Step 1 : Install Celery by running below command, and don’t forget to add that in requirements.txt (optional)

pip install celery

to verify installation of celery run command

pip show celery

it will show you output like this, if you get ouput same as below, you can move to step 2

Name: celery
Version: 4.4.6
Summary: Distributed Task Queue.
Home-page: http://celeryproject.org
Author: Ask Solem
Author-email: auvipy@gmail.com
License: BSD
Location: /home/my_pc/.local/lib/python3.6/site-packages
Requires: vine, billiard, kombu, pytz, future
Required-by:

Step 2 : Create a file name as celery.py alongside with the settings.py and urls.py files as illustrated below, if you have created setting package then put this file inside settings package

my_app/
 |-- my_app/
 |    |-- core/
 |    |    |-- migrations/
 |    |    |-- templates/
 |    |    |-- apps.py
 |    |    |-- models.py
 |    |    +-- views.py
 |    |-- templates/
 |    |-- __init__.py             
 |    |-- settings.py 
 |    |-- celery.py             <-- celery file goes here
 |    |-- urls.py
 |    +-- wsgi.py
 |-- manage.py
 +-- requirements.txt

now add the code below in celery.py file, read comments for better understanding

import os from celery import Celery
from os.path import dirname, basename
# To set which setting you want to use by default
os.environ.setdefault('DJANGO_SETTINGS_MODULE','my_app.settings')
# to initalize our Celery app
app = Celery(basename(dirname(__file__))) 
# to inform celery load settings which are specific to celery 
app.config_from_object('django.conf:settings', namespace='CELERY') 
# to tell Celery that it should use the settings.INSTALLED_APPS list  of applications to find defined Celery tasks.app.autodiscover_tasks()

Now we have to edit the init file shown below

my_app/
 |-- my_app/
 |    |-- core/
 |    |    |-- migrations/
 |    |    |-- templates/
 |    |    |-- apps.py
 |    |    |-- models.py
 |    |    +-- views.py
 |    |-- templates/
 |    |-- __init__.py              <-- edit this init file      
 |    |-- settings.py 
 |    |-- celery.py             
 |    |-- urls.py
 |    +-- wsgi.py
 |-- manage.py
 +-- requirements.txt

Add the code below in init file

from .celery import app as celery_app  
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
__all__ = ['celery_app']

Step 3 : Now we will add code which will be executed by our scheduler, create a file name as tasks.py shown below

my_app/
 |-- my_app/
 |    |-- core/
 |    |    |-- migrations/
 |    |    |-- templates/
 |    |    |-- apps.py
 |    |    |-- tasks.py              <-- create this file here
 |    |    |-- models.py
 |    |    +-- views.py
 |    |-- templates/
 |    |-- __init__.py
 |    |-- settings.py 
 |    |-- celery.py             
 |    |-- urls.py
 |    +-- wsgi.py
 |-- manage.py
 +-- requirements.txt

Now add the code which you want scheduler to run, i have added the sample code below, our sample code update first user in database

# change import here according to your project name
from my_app.celery import app             # <-- ⚡⚡⚡⚡⚡⚡⚡⚡ change here
from .models import User
import random
@app.task()
def update_user():
    dummy = str(random.randint(100, 1000))
    User.objects.filter(id=1).update(username=dummy)

Step 4 : Now we will create schedule for this task to run at every 10 seconds, So now create a file name as celery_config.py file as show in below directory

my_app/
 |-- my_app/
 |    |-- core/
 |    |    |-- migrations/
 |    |    |-- templates/
 |    |    |-- apps.py
 |    |    |-- tasks.py              
 |    |    |-- models.py
 |    |    +-- views.py
 |    |-- templates/
 |    |-- __init__.py
 |    |-- settings.py 
 |    |-- celery.py  
 |    |-- celery_config.py              <-- create your file here    
 |    |-- urls.py
 |    +-- wsgi.py
 |-- manage.py
 +-- requirements.txt

add code below in celery_config.py file

CELERY_BEAT_SCHEDULE = {
    # name for your scheduler
    'update_my_user': {
        # path for your task
        'task': 'my_app.core.tasks.update_user',
        # 10 defined here is in seconds
        'schedule': 10,
    },
}

and don’t forget to import this file in your settings.py file as shown below

from .celery_config import *

Step 5 : Here we will use redis as our message broker, which will be responsible for sending tasks to celery at periodic interval

Download redis from its offical page => Download

after downloading fire up server by opening terminal and execute

redis-server

You can test that Redis is working properly by typing this into your terminal

redis-cli ping

Redis should reply with PONG , You also need to add Redis as a dependency in the Django Project (optional), after doing this add the following setting to your setttings.py file

CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_TIMEZONE = 'UTC'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'

Step 6: Now finally we are done, we need to test our scheduler by running command below, run below commands in 2 separate terminal

celery -A my_app worker -l info

here,

• -A denote for app name which is added next to it in my case it is my_app, change accordingly your app name
• worker is what we want to run
• -l denote level of log we want, we are keeping this as info

In next terminal run

celery -A my_app beat -l info

change my_app to your app name, this command is used to beat schedule at specific time defined in celery_config.py

Finally if you didn’t get any error you should be able to see username changing for user with primary_key 1, every 10 seconds 🎉🎉🎉

Conclusion : Setting up celery is bit tricky make sure, you cross check settings twice after implementing steps from this article, and don’t add files in wrong directory, then you should be able to execute schedule task properly

Check out my article on :

SonarQube with pre commit hook ⚓