How to create Periodic Tasks in Celery?

Celery is a powerful, production-ready asynchronous job queue, which allows you to run time-consuming Python functions in the background. Celery is on the Python Package Index (PyPi), and can be easily installed with pip or easy_install and its dependencies.

Installation:

$ pip install django-celery && pip install redis

Celery uses a broker to pass messages between your application and Celery worker processes. We are having like rabbitmq, redis but we are using redis as a broker in this article. To initiate a task, an application will adds a message to the queue, which the broker then delivers to a worker.

So we can install redis using the following command:;

sudo apt-get install redis-server

Setting up celery in your project:

1. We need to add ‘djcelery’ into your project Installed Apps.

INSTALLED_APPS += (
        'djcelery',
    )

2. To configure celery, we are using redis broker which is easy to install, lightweight, fast:

import djcelery
    djcelery.setup_loader()
    BROKER_URL = 'redis://localhost:6379/1'

3. We need to tell celery from where tasks are being loaded using CELERY_IMPORTS command

CELERY_IMPORTS = ("testapp.tasks")

4. Celery Periodic Task means which runs at a regular intervals of time. These periodic tasks are scheduled by a celery beat which will executed by a worker.

• You can set the interval of time using crontab, timedelta. Using the time delta, we can execute a task at particular point of time like

timedelta(seconds=30)
     timedelta(minutes=30),
     timedelta(minutes=*/5) --> which will execute at every 5 minutes

• Like with cron, the tasks may overlap if the first task does not complete before the next.

crontab(0, 0, day_of_month='23') --> which will execute on the 23rd day of the every month.
     crontab(minute='*/5') --> which will execute at every 5 minutes
     crontab(hour='*', minute='5', day_of_week='mon') --> which will execute for every 5 minutes on mon

5. We can setup celery periodic tasks settings using celery beat schedular.

from celery.schedules import crontab
    CELERY_TIMEZONE = "Asia/Calcutta"
    CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
    CELERYBEAT_SCHEDULE = {
        # Executes for every 5 minutes on mon,tue,wed,thu,fri,sat
        'add-every-day-evening': {
            'task': 'testapp.tasks.load_messages_into_database',
            'schedule': crontab(hour='*', minute='5', day_of_week='mon,tue,wed,thu,fri,sat'),
        },
    }

You have to ensure only a single scheduler is running for a schedule at a time, otherwise you would end up with duplicate tasks.

The article was originally published at MicroPyramid blog.