笔记-python-standard library-17.7 queue

 笔记-python-standard library-17.7 queue

 

1.  queue

source code:Lib/queue.py

该模块实现了多生产者，多消费者队列。

此模块实现了所有的required locking semantics.

模块支持三种类型的队列，区别仅仅在于检索的顺序。

三种队列分别是FIFO,LIFO，优先级队列（使用heaq模块，优先抛出最小值）。

 

1.1.  模块中定义的类

class queue.Queue(maxsize=0)

class queue.LifoQueue(maxsize=0)

class queue.PriorityQueue(maxsize=0)

注意：优先级队列中优先抛出最小值。

 

exception queue.Empty

Exception raised when non-blocking get() (or get_nowait()) is called on a Queue object which is empty.

 

exception queue.Full

Exception raised when non-blocking put() (or put_nowait()) is called on a Queue object which is full.

 

1.2.  类方法

三种队列类都提供了以下方法：

1.   Queue.qsize()返回队列大致的大小，注意qsize()>0不代表get不会阻塞，同样，qsize()<maxsize不代表put不会阻塞

2.   Queue.empty()    

3.   Queue.full()

4.   Queue.put(item, block=True, timeout=None)

block决定是否阻塞，timeout决定最长阻塞时间

5.   Queue.put_nowait(item)

6.   Queue.get_nowait()： equal to get(False)

下面两个方法来跟踪队列中的任务是否被执行完。

7.   Queue.task_done()

8.   Queue.join()

阻塞直到队列中的所有任务完成。

 

其它属性

maxsize：队列最大长度，可以在初始化时给出，也可创建后手动设定。

 

1.1.    productor and consumer

queue实现生产者与消费者

q = queue.Queue()

def productor(arg):
    while True:
        if q.qsize() < 30:
            q.put(str(arg)+'banana')
        time.sleep(1)

def consumer(arg):
    while True:
        print('con {}, pro {}'.format(arg, q.get()))
        time.sleep(2)

for i in range(3):
    t = threading.Thread(target=productor, args=(i,))
    t.start()

for j in range(5):
    t = threading.Thread(target=consumer, args=(j,))
    t.start()

 

1.2.    task_done and join

文档如下：

    def task_done(self):

        '''Indicate that a formerly enqueued task is complete.

 

        Used by Queue consumer threads.  For each get() used to fetch a task,

        a subsequent call to task_done() tells the queue that the processing

        on the task is complete.

 

        If a join() is currently blocking, it will resume when all items

        have been processed (meaning that a task_done() call was received

        for every item that had been put() into the queue).

 

        Raises a ValueError if called more times than there were items

        placed in the queue.

        '''

        with self.all_tasks_done:

            unfinished = self.unfinished_tasks - 1

            if unfinished <= 0:

                if unfinished < 0:

                    raise ValueError('task_done() called too many times')

                self.all_tasks_done.notify_all()

            self.unfinished_tasks = unfinished

 

关键是如果不使用task_done，阻塞不会结束，

下面的代码把红色行删掉，线程会阻塞在q.join()处，

q = queue.Queue()
q.maxsize = 100

def productor(arg):
    while True:
        if q.qsize() >50:
            q.join()
        else:
            q.put(str(arg)+' banana')
        time.sleep(0.5)

def consumer(arg):
    while True:
        print('con {}, pro {}'.format(arg, q.get()))
        q.task_done()
        time.sleep(2)


def start_t():
    for i in range(4):
        t = threading.Thread(target=productor, args=(i,))
        t.start()

    for j in range(5):
        t = threading.Thread(target=consumer, args=(j,))
        t.start()

    time.sleep(1)
    while True:
        q_length = q.qsize()
        if q_length == 0:
            pass
            #break
        else:
            print("queue's size is {}".format(q_length))
            time.sleep(2)

start_t()
time.sleep(0.5)
print(r'end {}')
print(threading.enumerate())

 

2.      小结：

1. queue是线程安全的
2. FIFO实际使用的是dqueue，
3. 在使用join时一定要使用task_done