hadoop平台实现的小例子-wordcount单词计数

1.编写WordCount.java,包含Mapper类和Reducec类

2.编译WordCount.java,java -classpath #资料下载里面有

3.打包 jar -cvf WordCount.jar classes/*

4.作业提交 hadoop jar WordCount.jar WordCount input output

提交到hadoop中运行，指定输入文件 ，指定输出文件

javac -classpath /opt/hadoop-1.2.1/hadoop-core-1.2.1.jar:/opt/hadoop-1.2.1/lib/com

 

1.编写WordCount.java,包含Mapper类和Reducec类

#jps    // jps查看hadoop是否启动：5个进程

 

#cd /opt/examples/word_count

#touch WordCount.java   //新建文件

#vim WordCount.java    //编辑wordcount.java文件

 

 

2.编译WordCount.java,java -classpath  

#mkdir word_count_class

#javac -classpath /opt/hadoop-1.2.1/hadoop-core-1.2.1.jar:/opt/hadoop-1.2.1/lib/commons-cli-1.2.jar -d word_count_class/ WordCount.java  //执行WordCount.java文件在word_count_class文件夹中生成三个class文件

 

 

 

3.打包 jar -cvf WordCount.jar classes/*

#cd word_count_class

#jar –cvf wordcount.jar *.class  // jar -cvf 将所有class文件打包成wordcount.jar

 

 

4.作业提交 hadoop jar WordCount.jar WordCount input output

提交到hadoop中运行，指定输入文件 ，指定输出文件

javac -classpath /opt/hadoop-1.2.1/hadoop-core-1.2.1.jar:/opt/hadoop-1.2.1/lib/com

#mkdir input

#cd input

#touch file1

#touc h file2

#vim file1

Hello world

Hello hadoop

Hello file system

Hello java

Hello lady

Hello java home

 

 

#vim file2

world hello

Hello hadoop home

Hello file system

Hello java

Hello lady

Hello java home

Hello mapreduce

 

#cd ..

#hadoop fs –mkdit input_wordcount

#hadoop fs –put input/* input_wordcount/    // hadoop fs -put 把input文件夹里的所有文件放进hadoop里新建的input_wordcount文件夹

#hadoop fs –ls   // hadoop fs -ls 显示hadoop里目录下的文件或目录

#hadoop fs –cat input_wordcount/file1  //查看文件内容

#hadoop fs –cat input_wordcount/file2

#hadoop fs –ls input_wordcount

 

#hadoop jar word_count_class/wordcount.jar  WordCount  input_wordcount  output_wordcount

// hadoop jar 执行wordcount.jar包调用input_wordcount文件夹里的file1、file2文件，在新建的output_wordcount文件夹中输出, 执行过程中，先执行map后执行reduce

 

 

#hadoop fs –ls output_wordcount    //执行完后hadoop fs -ls 显示out_wordcount目录中有3个文件生成：_SUCCESS、_logs和part-r-00000

 

 

#hadoop fs –cat output_wordcount/part-r-00000   //行结果在part-r-00000中，hadoop fs –cat

查看该文件内容，可看到程序运行的结果：单词计数成功！