Python 线程使用模式

参考阅读：http://www.ibm.com/developerworks/aix/library/au-threadingpython/

一个小例子：

 import threading
 import datetime

 class ThreadClass(threading.Thread):
   def run(self):
     now = datetime.datetime.now()
     print "%s says Hello World at time: %s" %
     (self.getName(), now)

 for i in range(2):
   t = ThreadClass()
   t.start()

自己写的线程类要从threading.Thread继承，要实现run方法

Noah Gift推荐在使用python的线程时使用queue模式

#!/usr/bin/env python
import Queue
import threading
import urllib2
import time

hosts = ["http://yahoo.com", "http://google.com", "http://amazon.com",
"http://ibm.com", "http://apple.com"]

queue = Queue.Queue()

class ThreadUrl(threading.Thread):
"""Threaded Url Grab"""
def __init__(self, queue):
  threading.Thread.__init__(self)
  self.queue = queue

def run(self):
  while True:
    #grabs host from queue
    host = self.queue.get()

    #grabs urls of hosts and prints first 1024 bytes of page
    url = urllib2.urlopen(host)
    print url.read(1024)

    #signals to queue job is done
    self.queue.task_done()

start = time.time()
def main():

#spawn a pool of threads, and pass them queue instance
for i in range(5):
  t = ThreadUrl(queue)
  t.setDaemon(True)
  t.start()

#populate queue with data
  for host in hosts:
    queue.put(host)

#wait on the queue until everything has been processed
queue.join()

main()
print "Elapsed Time: %s" % (time.time() - start)

这个例子给出使用queue的模式：

1.用Queue.Queue()创建队列实例，然后用其操作数据

2.把该队列实例传入线程类中

3.生成守护线程池

4.每次从队列中取一个数据，在线程中使用改数据，使用run方法，完成工作

5.工作完成后，用queue.task_done()发送信号给队列，以表明任务已经结束

6.在queue上使用join()，这意味着一直等到queue为空时，再退出主程序

 

这里设置守护线程为真，是为了让主线程能够在只有守护线程时还在运行时退出，简化程序执行流程

 

链式处理：

 

import Queue
import threading
import urllib2
import time
from BeautifulSoup import BeautifulSoup

hosts = ["http://yahoo.com", "http://google.com", "http://amazon.com",
        "http://ibm.com", "http://apple.com"]

queue = Queue.Queue()
out_queue = Queue.Queue()

class ThreadUrl(threading.Thread):
    """Threaded Url Grab"""
    def __init__(self, queue, out_queue):
        threading.Thread.__init__(self)
        self.queue = queue
        self.out_queue = out_queue

    def run(self):
        while True:
            #grabs host from queue
            host = self.queue.get()

            #grabs urls of hosts and then grabs chunk of webpage
            url = urllib2.urlopen(host)
            chunk = url.read()

            #place chunk into out queue
            self.out_queue.put(chunk)

            #signals to queue job is done
            self.queue.task_done()

class DatamineThread(threading.Thread):
    """Threaded Url Grab"""
    def __init__(self, out_queue):
        threading.Thread.__init__(self)
        self.out_queue = out_queue

    def run(self):
        while True:
            #grabs host from queue
            chunk = self.out_queue.get()

            #parse the chunk
            soup = BeautifulSoup(chunk)
            print soup.findAll(['title'])

            #signals to queue job is done
            self.out_queue.task_done()

start = time.time()
def main():

    #spawn a pool of threads, and pass them queue instance
    for i in range(5):
        t = ThreadUrl(queue, out_queue)
        t.setDaemon(True)
        t.start()

    #populate queue with data
    for host in hosts:
        queue.put(host)

    for i in range(5):
        dt = DatamineThread(out_queue)
        dt.setDaemon(True)
        dt.start()


    #wait on the queue until everything has been processed
    queue.join()
    out_queue.join()

main()
print "Elapsed Time: %s" % (time.time() - start)

 

可见使用queue来使用线程真的是简单方便，而且还可以通过链式queue来扩展。上面的小程序可以看做是搜索引擎和数据挖掘的基础组成部分