Azkaban源码分析2-----项目的创建和提交

 

1.创建、提交项目

Azkaban的项目都是打包成zip文件上传到web页面上的，其实项目和工作流的信息都是存储在Mysql数据库中的

 在上传zip文件时，azkaban会对上传的文件进行解压缩，然后分析各个job组成的工作流并且申城DAG图

具体的工作流程：

azkaban.project.DirectoryFlowLoader.loadProjectFlow

// Load all the props files and create the Node objects
loadProjectFromDir(projectDir.getPath(), projectDir, null);

// Create edges and find missing dependencies
resolveDependencies();

// Create the flows.
buildFlowsFromDependencies();

// Resolve embedded flows
resolveEmbeddedFlows();

FlowLoaderUtils.checkJobProperties(project.getId(), this.props, this.jobPropsMap, this.errors);

1.1 loadProjectFromDir  ---加载project的目录文件

  private void loadProjectFromDir(final String base, final File dir, Props parent) {
    final File[] propertyFiles = dir.listFiles(new SuffixFilter(PROPERTY_SUFFIX));
    Arrays.sort(propertyFiles);
    //加载zip包中的配置文件的信息
    for (final File file : propertyFiles) {
      final String relative = getRelativeFilePath(base, file.getPath());
      try {
        parent = new Props(parent, file);
        parent.setSource(relative);

        final FlowProps flowProps = new FlowProps(parent);
        this.flowPropsList.add(flowProps);
      } catch (final IOException e) {
        this.errors.add("Error loading properties " + file.getName() + ":"
            + e.getMessage());
      }

      this.logger.info("Adding " + relative);
      this.propsList.add(parent);
    }

    //加载所有JOB信息如果重复就不加载
    final File[] jobFiles = dir.listFiles(new SuffixFilter(JOB_SUFFIX));
    for (final File file : jobFiles) {
      final String jobName = getNameWithoutExtension(file);
      try {
        if (!this.duplicateJobs.contains(jobName)) {
          if (this.jobPropsMap.containsKey(jobName)) {
            this.errors.add("Duplicate job names found '" + jobName + "'.");
            this.duplicateJobs.add(jobName);
            this.jobPropsMap.remove(jobName);
            this.nodeMap.remove(jobName);
          } else {
            final Props prop = new Props(parent, file);
            final String relative = getRelativeFilePath(base, file.getPath());
            prop.setSource(relative);//截取字符串作为job的名字
            //创建一个节点
            final Node node = new Node(jobName);
            final String type = prop.getString("type", null);
            if (type == null) {
              this.errors.add("Job doesn't have type set '" + jobName + "'.");
            }

            node.setType(type);

            node.setJobSource(relative);
            if (parent != null) {
              node.setPropsSource(parent.getSource());
            }

            // 使一个job变成一个root节点
            if (prop.getBoolean(CommonJobProperties.ROOT_NODE, false)) {
              this.rootNodes.add(jobName);
            }

            this.jobPropsMap.put(jobName, prop);
            this.nodeMap.put(jobName, node);
          }
        }
      } catch (final IOException e) {
        this.errors.add("Error loading job file " + file.getName() + ":"
            + e.getMessage());
      }
    }

    //如果有子文件夹，继续加载，说明支持多层文件夹
    for (final File file : dir.listFiles(new DirFilter())) {
      loadProjectFromDir(base, file, parent);
    }
  }

1.2 resolveDependencies()

    -----关联各个节点之间的依赖关系，主要是通过配置文件中的dependencies 项来确定依赖关系的

1.3 buildFlowsFromDependencies()　　

这个方法的主要功能就是依据依赖关系，创建工作流，修改数据库记录这些信息。 
a.先是查询出当前最大的版本号，然后+1赋值。 
b.上传工作流等相关的文件到数据库中，10M为1个单位，到project_files表中。 
c.然后修改project_flows，这张表主要记录了依据.job文件生成的工作流拓扑图。

1.4 resolveEmbeddedFlows()

---递归的判断每个依赖是否出现工作流的依赖

  private void resolveEmbeddedFlows() {
    for (final String flowId : this.flowDependencies.keySet()) {
      final HashSet<String> visited = new HashSet<>();
      resolveEmbeddedFlow(flowId, visited);
    }
  }

  private void resolveEmbeddedFlow(final String flowId, final Set<String> visited) {
    final Set<String> embeddedFlow = this.flowDependencies.get(flowId);
    if (embeddedFlow == null) {
      return;
    }

    visited.add(flowId);
    for (final String embeddedFlowId : embeddedFlow) {
      if (visited.contains(embeddedFlowId)) {
        this.errors.add("Embedded flow cycle found in " + flowId + "->"
            + embeddedFlowId);
        return;
      } else if (!this.flowMap.containsKey(embeddedFlowId)) {
        this.errors.add("Flow " + flowId + " depends on " + embeddedFlowId
            + " but can't be found.");
        return;
      } else {
        resolveEmbeddedFlow(embeddedFlowId, visited);
      }
    }

    visited.remove(flowId);
  }

2.提交工作流

　　在web端提交一个任务时如果没有指定提交到哪个Executor中，会交给选择器采用execID的hashcode值来确定由哪台executor执行

　　并且在 azkaban.executor.ExecutorManager 中构建提交任务的url

　　url= http://x.x.x.x:port/executor?execid=12&user