基于Redis的background job系统:PHP Resque

Resque介绍

Resque是github开源的基于Redis和Ruby实现的background job系统.而PHP Resque则是Resque的PHP版本.

PHP Resque有两个角色: Job 和Worker.

Jobs

一个Job就是一个需要在后台完成的任务.
每个Job都是一个独立的PHP Class, 至少包含perform方法处理Job,还可以包含setUp和tearDown方法.
新增加的Job和相关参数(消息),保存到Redis实现的Queue当中.
Workers

Worker以守护进程的方式后台运行,从Queue中按顺序读取Job来执行,执行时会先fork出子进程.
一个Worker的一次循环,可以处理一个Queue,也可以处理多个Queue.
Queue的优先级别(Priority)可以在启动worker的时候,通过环境变量QUEUE来指定.优先级高的先执行.
Job执行失败不会自动重试,而是丢到fail的Queue里面.如有重试的需求,需要自行处理.

Worker的环境变量参数有:
- QUEUE: 必须的参数.它指定了Queue的优先级别,靠前的会优先执行.如:QUEUE=email,log 如果使用QUEUE=*表示执行所有任务.
- APP_INCLUDE:包含文件,比如APP_INCLUDE=./worker_jobs.php, worker_jobs.php里面包含Worker需要调用的Jobs的PHP Class.
- COUNT:设置Worker数量,默认是1. 如:COUNT=3
- REDIS_BACKEND:设置Redis的IP, PORT等,默认为localhost:6379
- VERBOSE: 显示简单的debug信息,使用VERBOSE=1即可.
- VVERBOSE: 显示详细的debug信息,使用VVERBOSE=1即可.
- INTERVAL: Worker检查Queue的时间间隔.默认是:INTERVAL=5
- PIDFILE:如果是只开单个Worker的话,可以将pid写入PIDFILE指定的文件. 如:PIDFILE=/var/run/phpresque.pid

PHP Resque安装

安装Redis

Redis的安装,猛击这里
安装PHP Resque

PHP Resque可以通过Composer来安装,也可以直接下载源码即可使用.
```
# git clone https://github.com/chrisboulton/php-resque.git
```

PHP Resque使用

编写Job

// ping_job.php
class Ping_Job {
    private function ping($host, $port, $timeout) {
        $start = microtime(true);
        $fp = fsockopen($host, $port, $errno, $errstr, $timeout);
        if (!$fp) { return "down"; }
        $end = microtime(true);
        return round((($end - $start) * 1000), 0)." ms";
    }

    public function perform() {
        echo $this->ping($this->args['host'],$this->args['port'],$this->args['timeout']);
    }
}

一个Job必须定义perform方法,worker才会执行它.如果同时也定义了setUp和tearDown方法的话,则前者会在perform方法运行之前执行,后者会在perform方法运行之后执行.任何的由Job直接抛出的Exception都会导致Job的失败.

将Job插入Queue

将Job插入Queue的代码:

// myqueue.php
$opts = getopt('h:p::t::');

if (empty($opts['h'])) {
    die("Please specify the host to ping \n");
}
$host = $opts['h'];
$port = empty($opts['p']) || !is_numeric($opts['p']) ? 80 : $opts['p'];
$timeout = empty($opts['t']) || !is_numeric($opts['t']) ? 10 : $opts['t'];


require '../lib/Resque.php';
date_default_timezone_set('Asia/Shanghai');
Resque::setBackend('192.168.1.241:6379');

$JobId = Resque::enqueue('ping', 'Ping_Job', compact('host','port', 'timeout'), true);

echo "The job id is : ", $JobId , PHP_EOL;

终端运行(这里Cli方式,也可以选择其它运行方式.):

# php myqueue.php -h www.zrwm.com
The job id is : 1f987949ce085c9762ddca9ab0d14d74

后台运行Worker

Autoloader:

// autoloader.php
spl_autoload_register(function($class){
    set_include_path(get_include_path() . PATH_SEPARATOR . __DIR__);
    $class_file = strtolower($class) . '.php';
    if (file_exists($class_file)) {
        require_once($class_file);
    }
});

Worker代码:

// myworker.php

require_once('./autoloader.php'); // autoloader

$QUEUE = getenv('QUEUE');
if(empty($QUEUE)) {
	die("Set QUEUE env var containing the list of queues to work.\n");
}
require_once '../lib/Resque.php';
require_once '../lib/Resque/Worker.php';

//$REDIS_BACKEND = getenv('REDIS_BACKEND');

$REDIS_BACKEND='192.168.1.241:6379';
if(!empty($REDIS_BACKEND)) {
	Resque::setBackend($REDIS_BACKEND);
}

$logLevel = 0;
$LOGGING = getenv('LOGGING');
$VERBOSE = getenv('VERBOSE');
$VVERBOSE = getenv('VVERBOSE');
if(!empty($LOGGING) || !empty($VERBOSE)) {
	$logLevel = Resque_Worker::LOG_NORMAL;
}
else if(!empty($VVERBOSE)) {
	$logLevel = Resque_Worker::LOG_VERBOSE;
}

$APP_INCLUDE = getenv('APP_INCLUDE');
if($APP_INCLUDE) {
	if(!file_exists($APP_INCLUDE)) {
		die('APP_INCLUDE ('.$APP_INCLUDE.") does not exist.\n");
	}

	require_once $APP_INCLUDE;
}

$interval = 5;
$INTERVAL = getenv('INTERVAL');
if(!empty($INTERVAL)) {
	$interval = $INTERVAL;
}

$count = 1;
$COUNT = getenv('COUNT');
if(!empty($COUNT) && $COUNT > 1) {
	$count = $COUNT;
}

if($count > 1) {
	for($i = 0; $i < $count; ++$i) {
		$pid = pcntl_fork();
		if($pid == -1) {
			die("Could not fork worker ".$i."\n");
		}
		// Child, start the worker
		else if(!$pid) {
			$queues = explode(',', $QUEUE);
			$worker = new Resque_Worker($queues);
			$worker->logLevel = $logLevel;
			fwrite(STDOUT, '*** Starting worker '.$worker."\n");
			$worker->work($interval);
			break;
		}
	}
}
// Start a single worker
else {
	$queues = explode(',', $QUEUE);
	$worker = new Resque_Worker($queues);
	$worker->logLevel = $logLevel;

	$PIDFILE = getenv('PIDFILE');
	if ($PIDFILE) {
		file_put_contents($PIDFILE, getmypid()) or
			die('Could not write PID information to ' . $PIDFILE);
	}

	fwrite(STDOUT, '*** Starting worker '.$worker."\n");
	$worker->work($interval);
}

后台运行Worker:

# QUEUE=ping php myworker.php >> myworker.log 2>&1 &

查看运行中的Worker:

# ps -ef | grep myworker
  501 13009  7368   0  3:48下午 ttys002    0:00.03 php myworker.php

查看Worker的日志:

# cat myworker.log
*** Starting worker Jose.local:13009:ping

留意到当前myworker的进程ID为13009.

查看Job的状态

状态查看代码:

// check_status.php
if(empty($argv[1])) {
	die('Specify the ID of a job to monitor the status of.');
}

require '../lib/Resque/Job/Status.php';
require '../lib/Resque.php';
date_default_timezone_set('GMT');
Resque::setBackend('192.168.1.241:6379');

$status = new Resque_Job_Status($argv[1]);
if(!$status->isTracking()) {
	die("Resque is not tracking the status of this job.\n");
}

echo "Tracking status of ".$argv[1].". Press [break] to stop.\n\n";
while(true) {
	fwrite(STDOUT, "Status of ".$argv[1]." is: ".$status->get()."\n");
	sleep(1);
}

查看Job的状态:

# php check_status.php 1f987949ce085c9762ddca9ab0d14d74
Tracking status of 1f987949ce085c9762ddca9ab0d14d74. Press [break] to stop.

Status of 1f987949ce085c9762ddca9ab0d14d74 is: 4
Status of 1f987949ce085c9762ddca9ab0d14d74 is: 4
Status of 1f987949ce085c9762ddca9ab0d14d74 is: 4
Status of 1f987949ce085c9762ddca9ab0d14d74 is: 4

状态说明:
1 : Resque_Job_Status::STATUS_WAITING – 表示Job仍然留在Queue中
2 : Resque_Job_Status::STATUS_RUNNING – 表示Job正在运行用
3 : Resque_Job_Status::STATUS_FAILED – 表示Job失败
4 : Resque_Job_Status::STATUS_COMPLETE -表示Job完成
false – Failed to fetch the status – 表示无法获取Job状态(可能输入的Job ID不正确)

Resque介绍

Jobs

Workers