YARN executor user

The Shell action has the following caveats:

• Interactive commands are not supported.
• MR1:
  • In an unsecure cluster, everything is run as the user who started the TaskTracker where our shell script is running (mapred user); in a “Kerberized” cluster, it will run as the UNIX user of whomever submitted the workflow. This is in contrast to MapReduce-based actions, which, for the purposes of interaction with Hadoop, are run as the user who submitted the workflow –although the UNIX process for the task still runs as mapred.
• YARN/MR2:
  • The user everything is run as depends entirely on your ContainerExecutor YARN configurations (i.e.yarn.nodemanager.container-executor.class). If the DefaultContainerExecutor is being used, then everything will run as the user who started the NodeManagers (yarn user). If LinuxContainerExecutor is being used, then everything will run as whoever is configured in yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user (“nobody” by default). If LinuxContainerExecutor is being used and yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-user is set to “true”, then everything will run as the user who submitted the job. Alternatively, in a Kerberized cluster with LinuxContainerExecutor, everything will always run as the user who submitted the job. This can be a bit confusing, so I created this table:

    

• The Shell action is executed on an arbitrary node in the cluster.
• Different operating systems may have different versions of the same shell commands.

The implications of that third caveat are very important. Oozie executes the shell action in the same way it executes any of the other actions: as a MapReduce job. In this case, it’s a 1-mapper-0-reducer job, which is why it can be executed on any node in the cluster. This means that any command or script that we want to execute has to be available on that node; because we don’t know which node the shell action will be executed on, the command or script has to be available on all nodes! This is fine for typical built-in shell commands like echo or grep, but can be more problematic for programs such as matlab, which must not only be installed but may also require a license. Instead, we’ll be putting our script in the same directory as the workflow.xml and taking advantage of the<file>tag to have Oozie copy it to the proper node for us.

Even though two operating systems, or even two different versions of the same operating system, may have the same built-in commands or programs, they may behave differently or accept different arguments. For example, we’ll be using the tail command later; on Mac OS 10.7.5 we can specify the number of lines with the following arguments, but this won’t work properly on CentOS:

more:

http://blog.cloudera.com/blog/2013/03/how-to-use-oozie-shell-and-java-actions/