Celery(2)定时任务

创建任务计划脚本

函数方式添加定时任务

1)在当前项目在创建

[root@linux-node1 celery_pro]# pwd
/soft/celery_pro
[root@linux-node1 celery_pro]# vim periodic_task.py 

cat periodic_task.py 

##这里导入的就是之前项目中的app, 你就无需再创建一个了，必须定格写
from __future__ import absolute_import, unicode_literals
from .celery import app
##


from celery.schedules import crontab


'''
只要celery一连接上就会执行这个函数
函数里面会自动带sender参数,这个sender参数，就是你app的实例
'''
@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    # 添加一个定时任务,每十秒钟执行
    #test.s:  这里的s也就是传参数
    #name:  起个名字
    
    sender.add_periodic_task(10.0, test.s('hello'), name='add every 10')

    # Calls test('world') every 30 seconds
    sender.add_periodic_task(30.0, test.s('world'), expires=10)

    # Executes every Monday morning at 7:30 a.m.
    sender.add_periodic_task(
       #什么时间段，执行
        crontab(hour=22, minute=55, day_of_week=1),
        test.s('Happy Mondays!'),
    )

@app.task
def test(arg):
    print(arg)

 

请记住！worker只是去队列中取任务,然后执行任务

所以当你写完定时任务程序,你就需要定时把这个任务扔到队列中

这个东西就是beat

2)beat

启动方式

[root@linux-node1 soft]# celery -A celery_pro beat -l debug
celery beat v4.0.2 (latentcall) is starting.
__    -    ... __   -        _
LocalTime -> 2017-07-11 04:13:17
Configuration ->
    . broker -> redis://localhost:6379//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%DEBUG
    . maxinterval -> 5.00 minutes (300s)

但是这样是不行的....哈哈 那是因为beat根本就没有检测到你的任务,所以你得换个方式启动

仔细看..我标红的地方，就是你定时的任务

[root@linux-node1 soft]# celery -A celery_pro.periodic_task beat -l debug
celery beat v4.0.2 (latentcall) is starting.
__    -    ... __   -        _
LocalTime -> 2017-07-11 04:18:48
Configuration ->
    . broker -> redis://localhost:6379//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%DEBUG
    . maxinterval -> 5.00 minutes (300s)
[2017-07-11 04:18:48,137: DEBUG/MainProcess] Setting default socket timeout to 30
[2017-07-11 04:18:48,139: INFO/MainProcess] beat: Starting...
[2017-07-11 04:18:48,149: DEBUG/MainProcess] Current schedule:
<ScheduleEntry: celery_pro.periodic_task.test('world') celery_pro.periodic_task.test('world') <freq: 30.00 seconds>
<ScheduleEntry: celery_pro.periodic_task.test('Happy Mondays!') celery_pro.periodic_task.test('Happy Mondays!') <crontab: 57 22 2 * * (m/h/d/dM/MY)>
<ScheduleEntry: add every 10 celery_pro.periodic_task.test('hello') <freq: 10.00 seconds>
[2017-07-11 04:18:48,149: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2017-07-11 04:18:48,151: DEBUG/MainProcess] beat: Waking up in 9.98 seconds.

但是！！！！  你这样做还是有点问题的,因为你的这个文件是后来写的。当你启动一个worker的时候你仔细看

[tasks]
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . celery_pro.tasks.add
  . celery_pro.tasks.mul
  . celery_pro.tasks.xsum
  . celery_pro.tasks2.number

 如果其中没有你写的periodic_task 任务,  那你就需要在你的配置文件中注册一下

from __future__ import absolute_import, unicode_literals
from celery import Celery

app = Celery('proj',
             broker='redis://localhost',
             backend='redis://localhost',
             include=['celery_pro.tasks','celery_pro.tasks2','celery_pro.periodic_task',])    #切记这里注册

# Optional configuration, see the application user guide.
app.conf.update(
    result_expires=3600,
)

if __name__ == '__main__':
    app.start()

当你注册完毕后,你在启动你的worker程序

celery -A celery_pro worker -l debug

然后在开启你的beat程序来定时放入任务

celery -A celery_pro.periodic_task beat -l debug

这样你在面板上就可以看到了

配置文件方式添加定时任务

from __future__ import absolute_import, unicode_literals
from celery import Celery

app = Celery('proj',
             broker='redis://localhost',
             backend='redis://localhost',
             include=['celery_pro.tasks','celery_pro.tasks2','celery_pro.periodic_task',])

# Optional configuration, see the application user guide.
app.conf.update(
    result_expires=3600,
)

app.conf.beat_schedule = {
    'add-every-30-seconds': {
        'task': 'celery_pro.tasks.add',          #项目地址/脚本.方法
        'schedule': 5.0,             #5秒钟执行
        'args': (16,16)            #给tasks.add 传递2个参数     
    },
}
app.conf.timezone = 'UTC'

if __name__ == '__main__':
    app.start()

然后重启你的beat程序....不需要重启worker

更复杂的定时配置　　

上面的定时任务比较简单，只是每多少s执行一个任务，但如果你想要每周一三五的早上8点给你发邮件怎么办呢？哈，其实也简单，用crontab功能，跟linux自带的crontab功能是一样的，可以个性化定制任务执行时间

from celery.schedules import crontab
 
app.conf.beat_schedule = {
    # Executes every Monday morning at 7:30 a.m.
    'add-every-monday-morning': {
        'task': 'tasks.add',
        'schedule': crontab(hour=7, minute=30, day_of_week=1),
        'args': (16, 16),
    },
}

###

测试情况

[root@linux-node1 celery_pro]# cat celery.py
from __future__ import absolute_import, unicode_literals
from celery import Celery

app = Celery('proj',
             broker='redis://localhost',
             backend='redis://localhost',
             include=['celery_pro.tasks','celery_pro.tasks2','celery_pro.periodic_task',])

# Optional configuration, see the application user guide.
app.conf.update(
    result_expires=3600,
)

app.conf.beat_schedule = {
    'add-every-30-seconds': {
        'task': 'celery_pro.tasks.add',
        'schedule': 5.0,
        'args': (16,16)
    },
}
app.conf.timezone = 'UTC'

if __name__ == '__main__':
    app.start()

[root@linux-node1 celery_pro]# cat tasks.py 

from __future__ import absolute_import, unicode_literals
from .celery import app


@app.task
def add(x, y):
    return x + y


@app.task
def mul(x, y):
    return x * y


@app.task
def xsum(numbers):
    return sum(numbers)

[root@linux-node1 celery_pro]# cat tasks2.py
from __future__ import absolute_import, unicode_literals
from .celery import app
import time,random

@app.task
def number(start,end):
    time.sleep(3)
    return random.randint(start,end)

[root@linux-node1 celery_pro]# cat periodic_task.py
from __future__ import absolute_import, unicode_literals
from .celery import app


from celery.schedules import crontab



@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    # Calls test('hello') every 10 seconds.
    sender.add_periodic_task(10.0, test.s('hello'), name='add every 10')

    # Calls test('world') every 30 seconds
    sender.add_periodic_task(30.0, test.s('world'), expires=10)

    # Executes every Monday morning at 7:30 a.m.
    sender.add_periodic_task(
        crontab(hour=22, minute=57, day_of_week=2),
        test.s('Happy Mondays!'),
    )

@app.task
def test(arg):
    print(arg)

 

更多配置方法

 

Example	Meaning
crontab()	Execute every minute.
crontab(minute=0, hour=0)	Execute daily at midnight.
crontab(minute=0, hour='*/3')	Execute every three hours: midnight, 3am, 6am, 9am, noon, 3pm, 6pm, 9pm.
crontab(minute=0, hour='0,3,6,9,12,15,18,21')	Same as previous.
crontab(minute='*/15')	Execute every 15 minutes.
crontab(day_of_week='sunday')	Execute every minute (!) at Sundays.
crontab(minute='*', hour='*',day_of_week='sun')	Same as previous.
crontab(minute='*/10', hour='3,17,22',day_of_week='thu,fri')	Execute every ten minutes, but only between 3-4 am, 5-6 pm, and 10-11 pm on Thursdays or Fridays.
crontab(minute=0,hour='*/2,*/3')	Execute every even hour, and every hour divisible by three. This means: at every hour except: 1am, 5am, 7am, 11am, 1pm, 5pm, 7pm, 11pm
crontab(minute=0, hour='*/5')	Execute hour divisible by 5. This means that it is triggered at 3pm, not 5pm (since 3pm equals the 24-hour clock value of “15”, which is divisible by 5).
crontab(minute=0, hour='*/3,8-17')	Execute every hour divisible by 3, and every hour during office hours (8am-5pm).
crontab(0, 0,day_of_month='2')	Execute on the second day of every month.
crontab(0, 0, day_of_month='2-30/3')	Execute on every even numbered day.
crontab(0, 0, day_of_month='1-7,15-21')	Execute on the first and third weeks of the month.
crontab(0, 0,day_of_month='11', month_of_year='5')	Execute on the eleventh of May every year.
crontab(0, 0, month_of_year='*/3')	Execute on the first month of every quarter.