29

I've define a Celery app in a module, and now I want to start the worker from the same module in its __main__, i.e. by running the module with python -m instead of celery from the command line. I tried this:

app = Celery('project', include=['project.tasks'])

# do all kind of project-specific configuration
# that should occur whenever this module is imported

if __name__ == '__main__':
    # log stuff about the configuration
    app.start(['worker', '-A', 'project.tasks'])

but now Celery thinks I'm running the worker without arguments:

Usage: worker <command> [options] 

Show help screen and exit.

Options:
  -A APP, --app=APP     app instance to use (e.g. module.attr_name)
[snip]

The usage message is the one you get from celery --help, as if it didn't get a command. I've also tried

app.worker_main(['-A', 'project.tasks'])

but that complains about the -A not being recognized.

So how do I do this? Or alternatively, how do I pass a callback to the worker to have it log information about its configuration?

Fred Foo
  • 342,876
  • 71
  • 713
  • 819

5 Answers5

22

using app.worker_main method (v3.1.12): 

± cat start_celery.py
#!/usr/bin/python

from myapp import app


if __name__ == "__main__":
    argv = [
        'worker',
        '--loglevel=DEBUG',
    ]
    app.worker_main(argv)
okocian
  • 468
  • 4
  • 11
13

Based on code from Django-Celery module you could try something like this:

from __future__ import absolute_import, unicode_literals

from celery import current_app
from celery.bin import worker


if __name__ == '__main__':
    app = current_app._get_current_object()

    worker = worker.worker(app=app)

    options = {
        'broker': 'amqp://guest:guest@localhost:5672//',
        'loglevel': 'INFO',
        'traceback': True,
    }

    worker.run(**options)
daniula
  • 6,790
  • 4
  • 31
  • 49
  • It looks like this is the deprecated way of starting the worker, but their new way is too tangled up in Django code for me to figure it out. I've gone with this solution, thanks. – Fred Foo May 01 '14 at 12:10
  • 2
    Why deprecated? This code is run when you call `python manage.py celeryd`. It doesn't throw any warnings. – daniula May 01 '14 at 19:07
  • The code says it's the old way of running the worker. – Fred Foo May 02 '14 at 11:44
  • 2
    Now I see it. Alternative solution has same concept. Take `djcelery/management/commands/celery.py` code and modify it, so it will always behave as `./manage.pu celery worker` is called. – daniula May 02 '14 at 20:32
12

Since Celery 5 things have been changed

The worker_main results now:

AttributeError: 'Celery' object has no attribute 'worker_main'

For Celery 5 do following:

app = celery.Celery(
    'project',
    include=['project.tasks']
)

if __name__ == '__main__':
    worker = app.Worker(
        include=['project.tasks']
    )
    worker.start()

See here celery.apps.worker and celery.worker.WorkController.setup_defaults for details (hope it will be documented better in the future).

Tomasz Hławiczka
  • 281
  • 2
  • 5
  • 3
    For anyone trying to do this in 2021 this is the approach that worked for me. (Celery version 5.1) – categulario Jun 17 '21 at 23:31
  • Nice, but can you reload the whoe thing? I tried worker.reload() it didn't work. Also tried worker.stop() and worker.start() but didn't work properly neither. – Nachokhan Apr 28 '22 at 16:11
8

worker_main was put back in celery 5.0.3 here: https://github.com/celery/celery/pull/6481

This worked for me on 5.0.4:

self.app.worker_main(argv = ['worker', '--loglevel=info', '--concurrency={}'.format(os.environ['CELERY_CONCURRENCY']), '--without-gossip'])
dap
  • 81
  • 1
  • 1
5

I think you are just missing wrapping the args so celery can read them, like: 

queue = Celery('blah', include=['blah'])
queue.start(argv=['celery', 'worker', '-l', 'info'])
nurieta
  • 1,545
  • 13
  • 6