随笔分类 - hadoop
摘要：一个任务出现失败时，tasktracker会将此任务的失败信息报告给jobtracker，jobtracker会分配新的节点执行此任务。这种情况下不会影响整个作业的完整执行。但是如果容易任务出现多次失败，且失败次数超出失败的最大指定次数，那么作业会在未完成的情况下被终止。mapred-site.xm...
阅读全文
摘要：环境：centos6.4 64位；Hadoop2.2.0用编译好的Hadoop2.2.0中的$hadoop2.2.0/lib/native文件夹 替换现有集群上每个节点的$hadoop2.2.0/lib/native文件夹
阅读全文
摘要：环境centos6.4 64位软件下载地址:http://pan.baidu.com/s/1jGFnrJw参考资料：http://ju.outofmemory.cn/entry/64668参考资料：http://f.dataguru.cn/thread-189176-1-1.htmlHadoop2....
阅读全文
摘要：环境:centos 6.4 64位完整编译过程：http://www.cnblogs.com/orangeann/p/3695373.html1、[INFO] ----------------------------------------------------------------------...
阅读全文
摘要：环境：centos6.4 64位1、禁用selinux（所有节点)、关闭防火墙（所有节点必须做） 禁用selinux：$vim /etc/sysconfig/selinux将其中属性SELINUX修改为：SELINUX=disabled 关闭防火墙：$service iptables stop ...
阅读全文
摘要：mapreduce.jobhistory.address 0.0.0.0:10020 MapReduce JobHistory Server IPC host:port注释：历史作业ipc的ip端口。 mapreduce.jobhistory.webapp.address 0.0.0.0:19888 MapReduce JobHistory Server Web UI host:port注释：历史作业的web端访问ip地址 mapreduce.jobhistory.keytab Location of the kerberos keytab file for the ...
阅读全文
摘要：mapreduce.job.end-notification.url http://localhost:8080/jobstatus.php?jobId=$jobId&jobStatus=$jobStatus--> Indicates url which will be called on completion of job to inform end status of job. User can give at most 2 variables with URI : $jobId and $jobStatus. ...
阅读全文
摘要：mapreduce.job.counters.limit 120 Limit on the number of user counters allowed per job. 注释： mapreduce.framework.name local The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn. 注释： yarn.app.mapreduce.am.staging-dir /tmp/hadoop-yarn/staging The staging d...
阅读全文
摘要：mapreduce.tasktracker.healthchecker.script.path Absolute path to the script which is periodicallyrun by the node health monitoring service to determine if the node is healthy or not. If the value of this key is empty or the file does not exist in the location configured here, the node health...
阅读全文
摘要：mapreduce.jobtracker.taskcache.levels 2 This is the max level of the task cache. For example, if the level is 2, the tasks cached are at the host level and at the rack level. 注释： mapreduce.job.queuename default Queue to which a job is submitted. This must match one of the queues d...
阅读全文
摘要：mapreduce.task.io.sort.factor 10 The number of streams to merge at once while sorting files. This determines the number of open file handles.注释： mapreduce.task.io.sort.mb 100 The total amount of buffer memory to use while sorting files, in megabytes. By default, gives each merge stream 1...
阅读全文
摘要：mapreduce.jobtracker.jobhistory.location If job tracker is static the history files are stored in this single well known place. If No value is set here, by default, it is in the local file system at ${hadoop.log.dir}/history. 注释： mapreduce.jobtracker.jobhistory.task.numberprogresssplits ...
阅读全文
摘要：The kerberos principal for the proxy, if the proxy is not running as part of the RM. yarn.web-proxy.principal 注释： Keytab for WebAppProxy, if the proxy is not running as part of the RM. yarn.web-proxy.keytab 注释： The address for the web proxy as HOST:PORT, if this is...
阅读全文
摘要：yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler 注释： mapreduce.job.jar 注释： mapreduce.job.hdfs-servers ${fs.defaultFS} 注释：
阅读全文
摘要：address of node manager IPC. yarn.nodemanager.address 0.0.0.0:0 注释： Environment variables that should be forwarded from the NodeManager's environment to the container's. yarn.nodemanager.admin-env MALLOC_ARENA_MAX=$MALLOC_ARENA_MAX 注释： Environment variables that contai...
阅读全文
摘要：The address of the applications manager interface in the RM. yarn.resourcemanager.address 0.0.0.0:8032 注释： The number of threads used to handle applications manager requests. yarn.resourcemanager.client.thread-count 50 注释： The expiry interval for application master rep...
阅读全文
摘要：Factory to create client IPC classes. yarn.ipc.client.factory.class 注释： Type of serialization to use. yarn.ipc.serializer.type protocolbuffers 注释： Factory to create server IPC classes. yarn.ipc.server.factory.class 注释： Factory to create IPC exceptions. yarn.ipc...
阅读全文
摘要：hadoop.hdfs.configuration.version 1 version of this configuration file注释： dfs.namenode.logging.level info The logging level for dfs namenode. Other values are "dir"(trace namespace mutations), "block"(trace block under/over replications and blockcreations/deletions), or "all
阅读全文
摘要：hadoop.http.authentication.type simple Defines authentication used for Oozie HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME# hadoop.http.authentication.token.validity 36000 Indicates how long (in seconds) an authentication token is vali...
阅读全文
摘要：tfile.io.chunk.size 1048576 Value chunk size in bytes. Default to 1MB. Values of the length less than the chunk size is guaranteed to have known value length in read time (See also TFile.Reader.Scanner.Entry.isValueLengthKnown()). tfile.fs.output.buffer.size 262144 Buffer ...
阅读全文
浙公网安备 33010602011771号