Celery With Supervisor

Celery:

Celery is a task queue with a focus on real-time processing, while also supports task scheduling. Task queues are used as mechanisms to distribute work across multiple threads or machines. A task queues input is a unit of work called a task, dedicated worker processes and constantly monitor the queue for new work to perform. Celery communicates via messages using a broker to mediate between workers and clients.To initiate a task client puts a message on the queue, then the broker delivers that message to a worker.

Note: You can get more about celery http://celery.readthedocs.org/en/latest/django/first-steps-with-django.html

To run celery in virtual environment need to type the following command in your virtual environment

export C_FORCE_ROOT="true"

Running the worker with supervisor

In production, you will want to run the worker in the background as a daemon and sometimes there may be a chance of stopping of celery worker automatically then it should be restarted automatically. To do the tasks you need to use the tools provided like supervisord.

First, you need to install supervisor in your virtualenv and generate a configuration file.

$ pip install supervisor    $ cd /path/to/your/project    $ echo_supervisord_conf > supervisord.conf

Next, just add the following section in a configuration file:

[program:celeryd]    command=python manage.py celery worker -l info    stdout_logfile=/path/to/your/logs/celeryd.log    stderr_logfile=/path/to/your/logs/celeryd.log    autostart=true    autorestart=true    startsecs=10    stopwaitsecs=600

It’s a simplified version of the Celery supervisor configuration file, adapted to work with virtualenvs.

Usage

Just run supervisord in your project directory.

$ supervisord

Running supervisor during startup or booting time

create a file /etc/init.d/supervisord and configure your actual supervisord.conf in which celery is configured in DAEMON_ARGS as follows

DAEMON_ARGS="-c /path/to/supervisord.conf"

to run it

sudo chmod +x /etc/init.d/supervisord

and to automatically schedule it, do

sudo update-rc.d supervisord defaults

To Stop and Start the service

service supervisord stop    service supervisord start

Running supervisor during startup or booting time using upstart(For Ubuntu users)

Create a new file /etc/init/supervisor.conf. Its content should look like this:

description "supervisor"    start on runlevel [2345]    stop on runlevel [!2345]    respawn    chdir /path/to/supervisord    exec supervisord

Note that we’re using the same supervisord configuration file we used before. No changes there…

We can now start and stop supervisord with the following commands

$ sudo stop supervisor$ sudo start supervisor

--

--

--

Python, Django, Android and IOS, reactjs, react-native, AWS, Salesforce consulting & development company

Love podcasts or audiobooks? Learn on the go with our new app.

Reproducible experiments and benchmarks on SkyhookDM Ceph using Popper

Services in Linux

Tutorial: Creating security cameras in Unity

“Developing Phoenix apps with IEx.pry is da bomb!”

Communicating with Siemens S7–1500 Controller

A Complete Guide to Posting GitHub Issues.

How To Upload Image(Cloudinary/Asp.Net Core)

Advanced Tableau: Developing, Testing and Deploying Content in Tableau

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
MicroPyramid

MicroPyramid

Python, Django, Android and IOS, reactjs, react-native, AWS, Salesforce consulting & development company

More from Medium

Setting up Django in Windows

Expenses Tracker App using Django and Pandas.

Docker zing a Python Django Web App on Ubuntu

Django Plausible Proxy