【Todo】找出共同好友 & Spark & Hadoop面试题
找了这篇文章看了一下面试题<Spark 和hadoop的一些面试题（准备）>
http://blog.csdn.net/qiezikuaichuan/article/details/51578743
其中有一道题目很不错，详见：
http://www.aboutyun.com/thread-18826-1-1.html
http://www.cnblogs.com/lucius/p/3483494.html
我觉得可以在Hadoop上面实际编程做一下。
我觉得第一篇文章里面下面这一段总结的很好：
简要描述你知道的数据挖掘算法和使用场景
（一）基于分类模型的案例
（1）垃圾邮件的判别 通常会采用朴素贝叶斯的方法进行判别
（2）医学上的肿瘤判断 通过分类模型识别
（二）基于预测模型的案例
（1）红酒品质的判断 分类回归树模型进行预测和判断红酒的品质
（2）搜索引擎的搜索量和股价波动
（三）基于关联分析的案例：沃尔玛的啤酒尿布
（四）基于聚类分析的案例：零售客户细分
（五）基于异常值分析的案例：支付中的交易欺诈侦测
（六）基于协同过滤的案例：电商猜你喜欢和推荐引擎
（七）基于社会网络分析的案例：电信中的种子客户
（八）基于文本分析的案例
（1）字符识别：扫描王APP
（2）文学著作与统计：红楼梦归属
上面的统计共同好友的题目。写了个程序试了一下。
在Intellij项目 HadoopProj里面。maven项目，依赖如下：
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.hadoop.my</groupId> <artifactId>hadoop-proj</artifactId> <version>1.0-SNAPSHOT</version> <dependencies> <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.7.3</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.7.3</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.7.3</version> </dependency> </dependencies> <repositories> <repository> <id>aliyunmaven</id> <url>http://maven.aliyun.com/nexus/content/groups/public/</url> </repository> </repositories> </project>
代码如下：
package com.hadoop.my; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import java.io.IOException; /** * Created by baidu on 16/12/3. */ public class HadoopProj { public static class CommonFriendsMapper extends Mapper<LongWritable, Text, Text, Text> { @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String[] split = line.split(":"); String person = split[0]; String[] friends = split[1].split(","); for (String f: friends) { context.write(new Text(f), new Text(person)); } } } public static class CommonFriendsReducer extends Reducer<Text, Text, Text, Text> { // 输入<B->A><B->E><B->F>.... // 输出 B A,E,F,J protected void reduce(Text friend, Iterable<Text> persons, Context context) throws IOException, InterruptedException { StringBuffer sb = new StringBuffer(); for (Text person: persons) { sb.append(person+","); } context.write(friend, new Text(sb.toString())); } } public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { //读取classpath下的所有xxx-site.xml配置文件，并进行解析 Configuration conf = new Configuration(); Job friendJob = Job.getInstance(conf); //通过主类的类加载器机制获取到本job的所有代码所在的jar包 friendJob.setJarByClass(HadoopProj.class); //指定本job使用的mapper类 friendJob.setMapperClass(CommonFriendsMapper.class); //指定本job使用的reducer类 friendJob.setReducerClass(CommonFriendsReducer.class); //指定reducer输出的kv数据类型 friendJob.setOutputKeyClass(Text.class); friendJob.setOutputValueClass(Text.class); //指定本job要处理的文件所在的路径 FileInputFormat.setInputPaths(friendJob, new Path(args[0])); //指定本job输出的结果文件放在哪个路径 FileOutputFormat.setOutputPath(friendJob, new Path(args[1])); //将本job向hadoop集群提交执行 boolean res = friendJob.waitForCompletion(true); System.exit(res?0:1); } }
打成Jar包之后，传到Hadoop机器m42n05上面。
在上面还要新建输入文件，内容：
A:B,C,D,F,E,O
B:A,C,E,K
C:F,A,D,I
D:A,E,F,L
E:B,C,D,M,L
F:A,B,C,D,E,O,M
G:A,C,D,E,F
H:A,C,D,E,O
I:A,O
J:B,O
K:A,C,D
L:D,E,F
M:E,F,G
O:A,H,I,J
命令：
$ hadoop fs -mkdir /input/frienddata $ hadoop fs -put text.txt /input/frienddata $ hadoop fs -ls /input/frienddata Found 1 items -rw-r--r-- 3 work supergroup 142 2016-12-03 17:12 /input/frienddata/text.txt
把hadoop-proj.jar 拷贝到 m42n05的/home/work/data/installed/hadoop-2.7.3/myjars
运行命令
$ hadoop jar /home/work/data/installed/hadoop-2.7.3/myjars/hadoop-proj.jar com.hadoop.my.HadoopProj /input/frienddata /output/friddata
报错：
$ hadoop jar /home/work/data/installed/hadoop-2.7.3/myjars/hadoop-proj.jar com.hadoop.my.HadoopProj /input/frienddata /outputtput/frienddata 16/12/03 17:19:52 INFO client.RMProxy: Connecting to ResourceManager at master.Hadoop/10.117.146.12:8032 /fri Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://master.Hadoop:8390/input/frienddata already exists
看起来是命令行后面参数的索引不对，注意代码里是这样写的。
//指定本job要处理的文件所在的路径 FileInputFormat.setInputPaths(friendJob, new Path(args[0])); //指定本job输出的结果文件放在哪个路径 FileOutputFormat.setOutputPath(friendJob, new Path(args[1]));
而Java里面，和C++不同，参数的确是从0开始的。程序名本身不占位。
所以可能是不需要输入类名。重新输入命令：
$ hadoop jar /home/work/data/installed/hadoop-2.7.3/myjars/hadoop-proj.jar /input/frienddata /output/frienddata
获得输出：
16/12/03 17:24:33 INFO client.RMProxy: Connecting to ResourceManager at master.Hadoop/10.117.146.12:8032 16/12/03 17:24:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 16/12/03 17:24:34 INFO input.FileInputFormat: Total input paths to process : 1 16/12/03 17:24:34 INFO mapreduce.JobSubmitter: number of splits:1 16/12/03 17:24:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1478254572601_0002 16/12/03 17:24:34 INFO impl.YarnClientImpl: Submitted application application_1478254572601_0002 16/12/03 17:24:34 INFO mapreduce.Job: The url to track the job: http://master.Hadoop:8320/proxy/application_1478254572601_0002/ 16/12/03 17:24:34 INFO mapreduce.Job: Running job: job_1478254572601_0002 16/12/03 17:24:40 INFO mapreduce.Job: Job job_1478254572601_0002 running in uber mode : false 16/12/03 17:24:40 INFO mapreduce.Job: map 0% reduce 0% 16/12/03 17:24:45 INFO mapreduce.Job: map 100% reduce 0% 16/12/03 17:24:49 INFO mapreduce.Job: map 100% reduce 100% 16/12/03 17:24:50 INFO mapreduce.Job: Job job_1478254572601_0002 completed successfully 16/12/03 17:24:50 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=348 FILE: Number of bytes written=238531 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=258 HDFS: Number of bytes written=156 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=2651 Total time spent by all reduces in occupied slots (ms)=2446 Total time spent by all map tasks (ms)=2651 Total time spent by all reduce tasks (ms)=2446 Total vcore-milliseconds taken by all map tasks=2651 Total vcore-milliseconds taken by all reduce tasks=2446 Total megabyte-milliseconds taken by all map tasks=2714624 Total megabyte-milliseconds taken by all reduce tasks=2504704 Map-Reduce Framework Map input records=14 Map output records=57 Map output bytes=228 Map output materialized bytes=348 Input split bytes=116 Combine input records=0 Combine output records=0 Reduce input groups=14 Reduce shuffle bytes=348 Reduce input records=57 Reduce output records=14 Spilled Records=114 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=111 CPU time spent (ms)=1850 Physical memory (bytes) snapshot=455831552 Virtual memory (bytes) snapshot=4239388672 Total committed heap usage (bytes)=342360064 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=142 File Output Format Counters Bytes Written=156
看一下输出文件的位置：
$ hadoop fs -ls /output/frienddata Found 2 items -rw-r--r-- 3 work supergroup 0 2016-12-03 17:24 /output/frienddata/_SUCCESS -rw-r--r-- 3 work supergroup 156 2016-12-03 17:24 /output/frienddata/part-r-00000 $ hadoop fs -cat /output/frienddata/part-r-00000 A I,K,C,B,G,F,H,O,D, B A,F,J,E, C A,E,B,H,F,G,K, D G,C,K,A,L,F,E,H, E G,M,L,H,A,F,B,D, F L,M,D,C,G,A, G M, H O, I O,C, J O, K B, L D,E, M E,F, O A,H,I,J,F,
当然，也可以把输出merge到本地文件：
$ hdfs dfs -getmerge hdfs://master.Hadoop:8390/output/frienddata /home/work/frienddatatmp $ cat frienddatatmp A I,K,C,B,G,F,H,O,D, B A,F,J,E, C A,E,B,H,F,G,K, D G,C,K,A,L,F,E,H, E G,M,L,H,A,F,B,D, F L,M,D,C,G,A, G M, H O, I O,C, J O, K B, L D,E, M E,F, O A,H,I,J,F,
上面这道题目，做完了。
