05 测试hadoop自带词频统计demo

在了解了Hadoop中的存储组件HDFS之后，我们再来看一下Hadoop中另一个重要组件的计算MapReduce。HDFS搞定海量的存储，MapReduce搞定海量的计算。hadoop如其他优秀的开源组件一样，也提供了丰富的demo，下面我们就来看一下如何使用mapreduce自带demo进行词频统计。

1 系统、环境和约束条件

2 操作

  • 1 使用putty登录centos
  • 2 执行以下命令：
# 切换到家目录
cd 
# 进入hadoop的bin目录
cd hadoop-2.5.2/bin
#  vim word，在其中加入以下内容并保存退出，读者可以随意加入别的内容，这是我们待会要统计词频的文件
hello i am zhangli
hello i am xiaoli
hi i am ali
who are you
i am xiaoli
# 上传word文件
./hdfs dfs -put word /word
# 查看上传结果
./hdfs dfs -cat /word
# 开始统计，其中
# ./yarn是执行命令
# jar是表示执行的是jar包
# /root/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar 表示要执行的jar包
# wordcount  是要执行过程的名字
# /word 是我们上传的待分析的文件在HDFS中的路径
# /output 是我们分析之后结果的输出路径
./yarn jar /root/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /word /output
# 等待一阵子，会有以下输出
19/05/30 12:29:41 INFO client.RMProxy: Connecting to ResourceManager at hadoop1/192.168.100.192:8032
19/05/30 12:29:46 INFO input.FileInputFormat: Total input paths to process : 1
19/05/30 12:29:47 INFO mapreduce.JobSubmitter: number of splits:1
19/05/30 12:29:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1559056674360_0002
19/05/30 12:29:50 INFO impl.YarnClientImpl: Submitted application application_1559056674360_0002
19/05/30 12:29:51 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1559056674360_0002/
19/05/30 12:29:51 INFO mapreduce.Job: Running job: job_1559056674360_0002
19/05/30 12:30:19 INFO mapreduce.Job: Job job_1559056674360_0002 running in uber mode : false
19/05/30 12:30:19 INFO mapreduce.Job:  map 0% reduce 0%
19/05/30 12:30:36 INFO mapreduce.Job:  map 100% reduce 0%
19/05/30 12:30:46 INFO mapreduce.Job:  map 100% reduce 100%
19/05/30 12:30:49 INFO mapreduce.Job: Job job_1559056674360_0002 completed successfully
19/05/30 12:30:50 INFO mapreduce.Job: Counters: 49
	File System Counters
		FILE: Number of bytes read=111
		FILE: Number of bytes written=194141
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=156
		HDFS: Number of bytes written=65
		HDFS: Number of read operations=6
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters 
		Launched map tasks=1
		Launched reduce tasks=1
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=15451
		Total time spent by all reduces in occupied slots (ms)=7614
		Total time spent by all map tasks (ms)=15451
		Total time spent by all reduce tasks (ms)=7614
		Total vcore-seconds taken by all map tasks=15451
		Total vcore-seconds taken by all reduce tasks=7614
		Total megabyte-seconds taken by all map tasks=15821824
		Total megabyte-seconds taken by all reduce tasks=7796736
	Map-Reduce Framework
		Map input records=5
		Map output records=17
		Map output bytes=135
		Map output materialized bytes=111
		Input split bytes=89
		Combine input records=17
		Combine output records=10
		Reduce input groups=10
		Reduce shuffle bytes=111
		Reduce input records=10
		Reduce output records=10
		Spilled Records=20
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=697
		CPU time spent (ms)=8200
		Physical memory (bytes) snapshot=445980672
		Virtual memory (bytes) snapshot=4215586816
		Total committed heap usage (bytes)=322437120
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=67
	File Output Format Counters 
		Bytes Written=65
# 查看/output输出，在以下路径中会看到有两个文件，其中_SUCCESS代表成功，part-r-00000代表输出结果
./hdfs dfs -ls /output
以下为输出：
Found 2 items
-rw-r--r--   2 root supergroup          0 2019-05-30 12:30 /output/_SUCCESS
-rw-r--r--   2 root supergroup         65 2019-05-30 12:30 /output/part-r-00000
# 查看词频统计结果
./hdfs dfs -cat /output/part-r-00000
# 以下为输出
ali	1
am	4
are	1
hello	1
hi	1
i	4
who	1
xiaoli	2
you	1
zhangli	1

以上就是利用hadoop自带的词频统计demo进行统计并查看统计结果的过程。

posted @ 2020-03-29 21:10  张力的程序园  阅读(377)  评论(0)    收藏  举报