输出、状态-hadoop源码TaskAttemptID TaskTrackerAction JobTracker，FileOutputCommitter相关-by小雨
首先声明，我是一个菜鸟。一下文章中出现技术误导情况盖不负责
1,TaskAttemptID代表task attempt，一个task attempt就是一个map/reduce task 的一个例实taskid，而个每TaskAttemptID由两部分构成：TaskID+task序列号
eg:
attempt_200707121733_0003_m_000005_0
代表2007年07月12日17点33分启动的第0003号作业（job）的第00005号map任务的第0号task attempt
2,TaskTrackerAction的类型有，LaunchTaskAction()，KillTaskAction()，KillJobAction()，ReinitTrackerAction()，CommitTaskAction();
/**
* Ennumeration of various 'actions' that the {@link JobTracker}
* directs the {@link TaskTracker} to perform periodically.
*
*/
public static enum ActionType {
/** Launch a new task. */
LAUNCH_TASK,
/** Kill a task. */
KILL_TASK,
/** Kill any tasks of this job and cleanup. */
KILL_JOB,
/** Reinitialize the tasktracker. */
REINIT_TRACKER,
/** Ask a task to save its output. */
COMMIT_TASK
};
/**
* A factory-method to create objects of given {@link ActionType}.
* @param actionType the {@link ActionType} of object to create.
* @return an object of {@link ActionType}.
*/
public static TaskTrackerAction createAction(ActionType actionType) {
TaskTrackerAction action = null;
switch (actionType) {
case LAUNCH_TASK:
{
action = new LaunchTaskAction();
}
break;
case KILL_TASK:
{
action = new KillTaskAction();
}
break;
case KILL_JOB:
{
action = new KillJobAction();
}
break;
case REINIT_TRACKER:
{
action = new ReinitTrackerAction();
}
break;
case COMMIT_TASK:
{
action = new CommitTaskAction();
}
break;
}
return action;
}
3,JobTracker中相干：
offerService()
|--taskScheduler.start();
|--recoveryManager.recover();
通过RecoveryManager的recover()法方恢复Job
|-- this.expireTrackersThread.start();
检测已失效的TaskTracker点节。
// Used to expire TaskTrackers that have gone down
|-- this.retireJobsThread.start();
除清那些已成完很长时光仍然存在队列中的job
|--expireLaunchingTaskThread.start();
停止那些在超时时光内未告报进度的Tasks。
A thread to timeout tasks that have been assigned to task trackers,
* but that haven't reported back yet.
|--completedJobsStoreThread.start();
|--this.interTrackerServer.start();/ start the inter-tracker server once the jt is ready
/**
* Run forever
*/
public void offerService() throws InterruptedException, IOException {
// Prepare for recovery. This is done irrespective of the status of restart
// flag.
while (true) {
try {
recoveryManager.updateRestartCount();
break;
} catch (IOException ioe) {
LOG.warn("Failed to initialize recovery manager. ", ioe);
// wait for some time
Thread.sleep(FS_ACCESS_RETRY_PERIOD);
LOG.warn("Retrying...");
}
}
taskScheduler.start();
// Start the recovery after starting the scheduler
try {
recoveryManager.recover();
} catch (Throwable t) {
LOG.warn("Recovery manager crashed! Ignoring.", t);
}
this.expireTrackersThread = new Thread(this.expireTrackers,
"expireTrackers");
this.expireTrackersThread.start();
this.retireJobsThread = new Thread(this.retireJobs, "retireJobs");
this.retireJobsThread.start();
expireLaunchingTaskThread.start();
if (completedJobStatusStore.isActive()) {
completedJobsStoreThread = new Thread(completedJobStatusStore,
"completedjobsStore-housekeeper");
completedJobsStoreThread.start();
}
// start the inter-tracker server once the jt is ready
this.interTrackerServer.start();
synchronized (this) {
state = State.RUNNING;
}
4, Job的五种状态
public static final int RUNNING = 1;
public static final int SUCCEEDED = 2;
public static final int FAILED = 3;
public static final int PREP = 4;
public static final int KILLED = 5;
5，jobStatus信息:
this.jobid = jobid;
this.setupProgress = setupProgress;
this.mapProgress = mapProgress;
this.reduceProgress = reduceProgress;
this.cleanupProgress = cleanupProgress;
this.runState = runState;
6，FileOutputCommitter相干
setupJob：Hadoop初始化时设置job的输出；
commitJob：当job成完时，除清job的输出，这个法方在馈反来回的job状态为SUCCEEDED时用调；
cleanupJob：job结束后除清job的输出；
// do the clean up of temporary directory
abortJob：当job的返回状态是FAILED或KILLED时，行执该数函，用于终止作业的输出；
setupTask：设置task的输出；
// FileOutputCommitter's setupTask doesn't do anything. Because the
// temporary task directory is created on demand when the
// task is writing.
needsTaskCommit：检测task否是须要交提；
commitTask：将task的输出移到作业的输出目录；
// Move the task outputs to their final place
// Delete the temporary task-specific output directory
abortTask：取消task的输出；
文章结束给大家分享下程序员的一些笑话语录：
IBM和波音777
波音777是有史以来第一架完全在电脑虚拟现实中设计制造的飞机，所用的设备完全由IBM公司所提供。试飞前，波音公司的总裁非常热情的邀请IBM的技术主管去参加试飞，可那位主管却说道:“啊，非常荣幸，可惜那天是我妻子的生日，So..”..
波音公司的总载一听就生气了:“胆小鬼，我还没告诉你试飞的日期呢!”
