FastAPI + Celery

FastAPI + Celery

https://derlin.github.io/introduction-to-fastapi-and-celery/03-celery/

What is Celery¶

From their documentation:

Task queues are used as a mechanism to distribute work across threads or machines. A task queue’s input is a unit of work called a task. Dedicated worker processes constantly monitor task queues for new work to perform.

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task the client adds a message to the queue, the broker then delivers that message to a worker.

A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling. [it] is written in Python, but the protocol can be implemented in any language [(current clients in NodeJS, PHP)].

In other words, the entities involved in Celery are:

• producers: also called clients, they are the ones requesting tasks and doing something with the results.
• broker: the broker is the message transport, used to send and receive messages between producers and workers. In other words, they store the task queue. Celery supports a myriad of message brokers, but currently only two are feature-complete:

Redis and

• RabbitMQ.
• workers: the workers are processes that constantly watch the task queue and execute tasks.
• result backend: a backend is only necessary when we want to keep track of the tasks' states or retrieve results from tasks. A result backend is optional but turned on by default, see Celery without a Results Backend.

See also the diagram in Understanding Celery's architecture

 

https://github.com/derlin/introduction-to-fastapi-and-celery

对于mkdown文档可以本地使用docker运行。

docker run --rm -it -p 8888:8000 -v ${PWD}:/docs squidfunk/mkdocs-material

 

完整的示例：

既有异步任务，也有定时任务。

https://github.com/fanqingsong/docker-fastapi-celery-redis

 

参考：

https://segmentfault.com/a/1190000039180016

https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html#introduction