小记--------spark的Master的Application注册机制源码分析及Master的注册机制原理分析
Master类位置所在:spark-core_2.11-2.1.0.jar的org.apache.spark.deploy.master下的Master类
1 //截取了部分代码 2 //处理Application注册的请求 3 case RegisterApplication(description, driver) => 4 //如果master的状态是standby,也就是当前这个master,是standby master,不是active master 5 //那么Application来请求注册,什么都不会干 6 // TODO Prevent repeated registrations from some driver 7 if (state == RecoveryState.STANDBY) { 8 // ignore, don't send response 9 } else { 10 logInfo("Registering app " + description.name) 11 //用ApplicationDescription信息,创建ApplicationInfo 12 val app = createApplication(description, driver) 13 //注册Application 14 //将ApplicationInfo加入缓存,将Application加入等待调度的队列--waitingApps 15 registerApplication(app)//详细代码见:代码1 16 logInfo("Registered app " + description.name + " with ID " + app.id) 17 //使用持久化引擎,将ApplicationInfo进行持久化 18 persistenceEngine.addApplication(app) 19 //反向,向SparkDeploySchedulerBackend的APPClient的ClientActor,发送消息,也就是RegisteredApplication 20 driver.send(RegisteredApplication(app.id, self)) 21 schedule()//schedule 重新调度,--至此Application注册完成 22 } 23 代码1 24 private def registerApplication(app: ApplicationInfo): Unit = { 25 val appAddress = app.driver.address 26 if (addressToApp.contains(appAddress)) { 27 logInfo("Attempted to re-register application at same address: " + appAddress) 28 return 29 } 30 31 applicationMetricsSystem.registerSource(app.appSource) 32 33 //这里其实就是将APP的信息加入内存缓存中 34 apps += app 35 idToApp(app.id) = app 36 endpointToApp(app.driver) = app 37 addressToApp(appAddress) = app 38 39 //将APP加入等待调度的队列中 waitingApps就是一个ArrayBuffer 40 waitingApps += app 41 if (reverseProxy) { 42 webUi.addProxyTargets(app.id, app.desc.appUiUrl) 43 } 44 }
作者:于二黑
本文版权归作者和博客园共有,欢迎转载,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文连接,否则保留追究法律责任的权利。