Hadoop源码分析23:MapReduce的Job提交过程
命令为：
hadoop_debugjar /opt/hadoop-1.0.0/hadoop-examples-1.0.0.jar wordcount/user/admin/in/yellow.txt /user/admin/out/555
首先调用org.apache.hadoop.util.runJar.main
public static void main(String[]args){
// 加载Jar包 /opt/hadoop-1.0.0/hadoop-examples-1.0.0.jar
JarFile jarFile = new JarFile(fileName);
//根据META-INF得知主Class为org/apache/hadoop/examples/ExampleDriver
Manifest manifest = jarFile.getManifest();
if (manifest !=null){
mainClassName =manifest.getMainAttributes().getValue("Main-Class");
}
//建立本地临时文件夹 /tmp/hadoop-admin
File tmpDir = newFile(newConfiguration().get("hadoop.tmp.dir"));
tmpDir.mkdirs();
//建立本地工作文件夹 /tmp/hadoop-admin/hadoop-unjar4705742737164408087 finalFile workDir = File.createTempFile("hadoop-unjar", "", tmpDir);
workDir.delete();
workDir.mkdirs();
//JVM退出时将tmp/hadoop-admin/hadoop-unjar4705742737164408087删除
Runtime.getRuntime().addShutdownHook(newThread(){
publicvoidrun(){
try{
FileUtil.fullyDelete(workDir);
} catch(IOExceptione) {
}
}
});
//将Jar包解压到/tmp/hadoop-admin/hadoop-unjar4705742737164408087
unJar(file, workDir);
//将/tmp/hadoop-admin/hadoop-unjar4705742737164408087，/tmp/hadoop-admin/hadoop-unjar4705742737164408087/classes/,/tmp/hadoop-admin/hadoop-unjar4705742737164408087/lib全部添加到classpath
classPath.add(newFile(workDir+"/").toURL());
classPath.add(file.toURL());
classPath.add(newFile(workDir,"classes/").toURL());
File[] libs = newFile(workDir,"lib").listFiles();
if(libs!= null){
for(inti = 0;i ＜libs.length;i++) {
classPath.add(libs[i].toURL());
}
}
//运行主函数
main.invoke(null,newObject[]{ newArgs });
}
设置属性：
job.setJarByClass(WordCount.class); //mapred.jar
job.setMapperClass(WordCountMap.class); //mapreduce.map.class
job.setReducerClass(WordCountReduce.class); //mapreduce.reduce.class
job.setCombinerClass(WordCountReduce.class);//mapreduce.combine.class
job.setMapOutputKeyClass(Text.class); //mapred.mapoutput.key.class
job.setMapOutputValueClass(IntWritable.class);//mapred.mapoutput.value.class
job.setOutputKeyClass(Text.class); //mapred.output.key.class
job.setOutputValueClass(IntWritable.class); //mapred.output.value.class
job.setJobName("WordCount"); //mapred.job.name
FileInputFormat.addInputPath(job,input); //mapred.input.dir
FileOutputFormat.setOutputPath(job,output); //mapred.output.dir
job.submit()
publicvoidsubmit()throwsIOException,InterruptedException,
ClassNotFoundException {
......
// Connect tothe JobTracker and submit the job
connect();
info=jobClient.submitJobInternal(conf);
......
}
连接JobTracker：
privatevoidconnect()throwsIOException,InterruptedException {
......
jobClient=newJobClient((JobConf)getConfiguration());
......
}
其中：
publicJobClient(JobConfconf) throwsIOException{
......
init(conf);
}
publicvoidinit(JobConfconf) throwsIOException{
......
this.jobSubmitClient=createRPCProxy(JobTracker.getAddress(conf),conf);
}
privatestaticJobSubmissionProtocolcreateRPCProxy(InetSocketAddress addr,
Configuration conf) throwsIOException{
return(JobSubmissionProtocol)RPC.getProxy(JobSubmissionProtocol.class,
JobSubmissionProtocol.versionID,addr,
UserGroupInformation.getCurrentUser(), conf,
NetUtils.getSocketFactory(conf,JobSubmissionProtocol.class));
}
此时获得一个实现JobSubmissionProtocol的RPC调用，即JobTracker的代理。
获取job StagingArea
PathjobStagingArea =JobSubmissionFiles.getStagingDir(JobClient.this,
jobCopy);
RPC请求：JobSubmissionProtocol.getStagingAreaDir()
返回：hdfs://server1:9000/tmp/hadoop-admin/mapred/staging/Admin/.staging
RPC请求：ClientProtocol.getFileInfo(/tmp/hadoop-admin/mapred/staging/Admin/.staging)
返回：org.apache.hadoop.hdfs.protocol.HdfsFileStatus@5521691b，即存在
RPC请求：ClientProtocol.getFileInfo(/tmp/hadoop-admin/mapred/staging/Admin/.staging)
返回：org.apache.hadoop.hdfs.protocol.HdfsFileStatus@726c554，用以判断权限
获得 NewJobId
JobIDjobId = jobSubmitClient.getNewJobId();
RPC请求：JobSubmissionProtocol.getNewJobId()
返回：job_201404010621_0004
建立 submitJob Dir：
PathsubmitJobDir = newPath(jobStagingArea,jobId.toString());
hdfs://server1:9000/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004
复制Jar到HDFS
copyAndConfigureFiles(jobCopy,submitJobDir);
RPC请求：ClientProtocol.getFileInfo(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004)
返回：null
RPC请求：ClientProtocol.mkdirs(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004,rwxr-xr-x)
返回：true
RPC请求：ClientProtocol.setPermission(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004,rwx------)
返回：null
RPC请求：ClientProtocol.getFileInfo(/tmp/hadoop-admin/mapred/staging/admin/.staging/job_201404010621_0004/job.jar）
返回：null，即不存在
RPC请求：ClientProtocol.create(/tmp/hadoop-admin/mapred/staging/admin/.staging/job_201404010621_0004/job.jar,rwxr-xr-x, DFSClient_-1317833261, true, true, 3,67108864)
返回：输出流
RPC请求：ClientProtocol.addBlock(/tmp/hadoop-admin/mapred/staging/admin/.staging/job_201404010621_0004/job.jar,DFSClient_-1317833261, null)
返回：org.apache.hadoop.hdfs.protocol.LocatedBlock@1a9b701
Block：blk_6689254996395759186_2720
BlockToken：Ident: ,Pass: , Kind: , Service:
DataNode：[10.1.1.103:50010,10.1.1.102:50010]
RPC请求：ClientProtocol.complete(/tmp/hadoop-admin/mapred/staging/admin/.staging/job_201404010621_0004/job.jar,DFSClient_-1317833261）
返回：true
RPC请求：ClientProtocol.setReplication(/tmp/hadoop-admin/mapred/staging/admin/.staging/job_201404010621_0004/job.jar,10)
返回：true
RPC请求：ClientProtocol.setPermission(/tmp/hadoop-admin/mapred/staging/admin/.staging/job_201404010621_0004/job.jar,rw-r--r--)
返回：null
RPC请求：ClientProtocol.renewLease(DFSClient_-1317833261)
返回：null
此后有1个守护线程会不断发送 renewLease请求
此时本地文件/opt/hadoop-1.0.0/hadoop-examples-1.0.0.jar被复制到HDFS文件系统/tmp/hadoop-admin/mapred/staging/admin/.staging/job_201404010621_0004/job.xml
Reduce数目：
int reduces= jobCopy.getNumReduceTasks();
reduce数目为2
检查输出目录
RPC请求：ClientProtocol.getFileInfo(/user/admin/out/555)
返回：null，即不存在
获取输入分片信息：
int maps =writeSplits(context, submitJobDir);
其中：
private＜T extendsInputSplit＞ intwriteNewSplits(JobContextjob, Path jobSubmitDir) throwsIOException,
InterruptedException, ClassNotFoundException {
Configuration conf = job.getConfiguration();
InputFormat＜?, ?＞ input =
ReflectionUtils.newInstance(job.getInputFormatClass(),conf);
List＜InputSplit＞ splits =input.getSplits(job);
T[] array = (T[]) splits.toArray(newInputSplit[splits.size()]);
// sort thesplits into order based on size, so that the biggest
// gofirst
Arrays.sort(array, newSplitComparator());
JobSplitWriter.createSplitFiles(jobSubmitDir,conf,
jobSubmitDir.getFileSystem(conf), array);
returnarray.length;
}
其中：
publicList＜InputSplit＞getSplits(JobContextjob
) throwsIOException{
...........
}
RPC请求：ClientProtocol.getFileInfo(/user/admin/in/yellow.txt)
返回：path="hdfs://server1:9000/user/admin/in/yellow.txt",length=201000000,isdir=false,block_replication=3, blocksize=67108864, permission=rw-r--r--,owner=Admin, group=supergroup
RPC请求：ClientProtocol.getBlockLocations(/user/admin/in/yellow.txt,0, 201000000)
返回：3个BlockLocation
offset={0}, length={67108864}, hosts={server3,server2}, names={[10.1.1.102:50010, 10.1.1.103:50010]}, topologyPaths={[/default-rack/10.1.1.103:50010,/default-rack/10.1.1.102:50010]}
offset={67108864}, length={67108864}, hosts={server3,server2}, names={[10.1.1.102:50010, 10.1.1.103:50010]}, topologyPaths={[/default-rack/10.1.1.103:50010,/default-rack/10.1.1.102:50010]}
offset={134217728},length={66782272}, hosts={server3,server2}, names={[10.1.1.102:50010, 10.1.1.103:50010]},topologyPaths={[/default-rack/10.1.1.103:50010,/default-rack/10.1.1.102:50010]}
最终确定的分片信息 为3个Filespit
Filespit： file={hdfs://server1:9000/user/admin/in/yellow.txt},hosts={ [server3, server2] }, length={ 67108864 },start={0}
Filespit： file={hdfs://server1:9000/user/admin/in/yellow.txt},hosts={ [server3, server2] }, length={ 67108864 },start={67108864}
Filespit： file={hdfs://server1:9000/user/admin/in/yellow.txt},hosts={ [server3, server2] }, length={ 66782272}, start={134217728}
map数量为3
jobCopy.setNumMapTasks(maps);
建立分片文件：
JobSplitWriter.createSplitFiles(jobSubmitDir,conf,
jobSubmitDir.getFileSystem(conf), array);
RPC请求：ClientProtocol.create(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.split,rwxr-xr-x, DFSClient_-1317833261, true, true, 3,67108864);
返回：输出流
RPC请求：ClientProtocolsetPermission(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.split,rw-r--r--)
返回：null
RPC请求：ClientProtocol.setReplication(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.split,10)
返回：true
RPC请求：ClientProtocol.addBlock(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.split,DFSClient_-1317833261, null)
返回：LocatedBlock对象为
Block： blockid=-921399365952861077,generationStamp=2714，numBytes=0
BlockTokenIdentifier：Ident: ,Pass: , Kind: , Service:
DatanodeInfo[]：[10.1.1.103:50010,10.1.1.102:50010]
offset：0
RPC请求：ClientProtocol.complete(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.split,DFSClient_-1317833261)
返回：true
写入的 SplitMetaInfo为
[data-size :67108864 start-offset : 7 locations : server3 server2]
[data-size :67108864 start-offset : 116 locations: server2 server3]
[data-size :66782272 start-offset : 225 locations : server2 server3 ]
RPC请求：ClientProtocol.create(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.splitmetainfo,rwxr-xr-x, DFSClient_-1317833261, true, true, 3,67108864)
返回：输出流
RPC请求： ClientProtocol.setPermission(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.splitmetainfo,rw-r--r--)
返回：null
RPC请求：ClientProtocol.addBlock(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.splitmetainfo,DFSClient_-1317833261, null)
返回：LocatedBlock对象为
Block： blockid=789965327875207186,generationStamp=2715，numBytes=0
BlockTokenIdentifier：Ident: ,Pass: , Kind: , Service:
DatanodeInfo[]：[10.1.1.103:50010,10.1.1.102:50010]
offset：0
RPC请求：ClientProtocol.complete(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.splitmetainfo,DFSClient_-1317833261)
返回：true
设置AccessControl
RPC请求：JobSubmissionProtocol.getQueueAdmins(default)
返回：All usersare allowed
Write jobfile to JobTracker'sfs
RPC请求：ClientProtocol.create(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.xml,rwxr-xr-x, DFSClient_-1317833261, true, true, 3,67108864)
返回：输出流
RPC请求：ClientProtocol.setPermission(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.xml,rw-r--r--)
返回：null
RPC请求：ClientProtocol.addBlock(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.xml，DFSClient_-1317833261,null)
返回：LocatedBlock对象为
Block： blockid= -7725157033540829125,generationStamp= 2716，numBytes=0
BlockTokenIdentifier：Ident: ,Pass: , Kind: , Service:
DatanodeInfo[]：[10.1.1.103:50010,10.1.1.102:50010]
offset：0
RPC请求：ClientProtocol.complete(/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.xml,DFSClient_-1317833261)
返回：true
此时"/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/"下生成文件 job.xml，包含了所有的配置信息.
此时HDFS目录"/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/"下面文件为：
-rw-r--r-- 10 admin supergroup 142465 2014-04-08 00:20 job.jar
-rw-r--r-- 10 admin supergroup 334 2014-04-08 00:45 job.split
-rw-r--r-- 3 admin supergroup 80 2014-04-08 00:50 job.splitmetainfo
-rw-r--r-- 3 admin supergroup 20416 2014-04-08 00:55job.xml
job.jar 为运行的Jar包, job.split内容 为（FileSplit 对象）， job.splitmetainfo内容 为（SplitMetaInfo对象），job.xml 为job的配置文件
提交作业：
status= jobSubmitClient.submitJob(
jobId, submitJobDir.toString(),jobCopy.getCredentials());
RPC请求：JobSubmissionProtocol.submitJob(job_201404010621_0004,hdfs://server1:9000/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004,org.apache.hadoop.security.Credentials@70677770)
返回:JobStatus： setProgress=0，mapProgress=0，reduceProgress=0，cleanProgress=0，runstate=4，priority=NOMAL，..
RPC请求：JobSubmissionProtocol.getJobProfile(job_201404010621_0004）
返回：JobProfile：jobFile=hdfs://server1:9000/tmp/hadoop-admin/mapred/staging/Admin/.staging/job_201404010621_0004/job.xml，jobID=job_201404010621_0004，name=WordCount，queue=default，url=http://server1:50030/jobdetails.jsp?jobid=job_201404010621_0004，user=Admin
综合JobStatus和JobProfile
Job:job_201404010621_0004
file:hdfs://server1:9000/tmp/hadoop-admin/mapred/staging/admin/.staging/job_201404010621_0004/job.xml
tracking URL:http://server1:50030/jobdetails.jsp?jobid=job_201404010621_0004
map()completion: 0.0
reduce()completion: 0.0
监控Job状态：
jobClient.monitorAndPrintJob(conf,info);
RPC请求：JobSubmissionProtocol.getJobStatus(job_201404010621_0004)
返回： setProgress=1，mapProgress=1，reduceProgress=0.22222224，cleanProgress=1，runstate=1，priority=NOMAL
RPC请求：JobSubmissionProtocol.getJobStatus(job_201404010621_0004)
返回： setProgress=1，mapProgress=1，reduceProgress=1，cleanProgress=1，runstate=2，priority=NOMAL，
即map 100%reduce 100%
之后会多次发送JobSubmissionProtocol.getJobStatus(job_201404010621_0004)请求
RPC请求：JobSubmissionProtocol.getTaskCompletionEvents(job_201404010621_0004,0, 10)
返回： [Task Id :attempt_201404010621_0004_m_000004_0, Status : SUCCEEDED, Task Id :attempt_201404010621_0004_m_000002_0, Status : SUCCEEDED, Task Id :attempt_201404010621_0004_m_000000_0, Status : SUCCEEDED, Task Id :attempt_201404010621_0004_m_000001_0, Status : SUCCEEDED, Task Id :attempt_201404010621_0004_m_000000_1, Status : KILLED, Task Id :attempt_201404010621_0004_r_000000_0, Status : SUCCEEDED, Task Id :attempt_201404010621_0004_r_000001_0, Status : SUCCEEDED, Task Id :attempt_201404010621_0004_m_000003_0, Status :SUCCEEDED]
RPC请求：JobSubmissionProtocol.getJobCounters(job_201404010621_0004)
返回：OW[class=classorg.apache.hadoop.mapred.Counters,value=Counters: 29
Job Counters
Launched reduce tasks=2
SLOTS_MILLIS_MAPS=293879
Total time spent by all reduces waiting after reserving slots(ms)=0
Total time spent by all maps waiting after reserving slots(ms)=0
Launched map tasks=4
Data-local map tasks=4
SLOTS_MILLIS_REDUCES=74342
File Output Format Counters
Bytes Written=933
FileSystemCounters
FILE_BYTES_READ=316152
HDFS_BYTES_READ=201008521
FILE_BYTES_WRITTEN=370366
HDFS_BYTES_WRITTEN=933
File Input Format Counters
Bytes Read=201008194
Map-Reduce Framework
Map output materialized bytes=2574
Map input records=15600000
Reduce shuffle bytes=2574
Spilled Records=23025
Map output bytes=356000000
Total committed heap usage (bytes)=378023936
CPU time spent (ms)=158350
Combine input records=41011850
SPLIT_RAW_BYTES=327
Reduce input records=225
Reduce input groups=75
Combine output records=12075
Physical memory (bytes) snapshot=650371072
Reduce output records=75
Virtual memory (bytes) snapshot=5300277248
Map output records=41000000]
浙公网安备 33010602011771号