求uv

uv定义

在同一天内，UV指记录第一次进入网站的具有独立IP的访问者，在同一天内再次访问该网站则不计数.

代码实现:方法一

SparkConf conf = new SparkConf();
conf.setMaster("local").setAppName("uv");

JavaSparkContext jsc = new JavaSparkContext(conf);
JavaRDD<String> lineRDD = jsc.textFile("data/pvuvdata.txt");
lineRDD.flatMap(new FlatMapFunction<String, Tuple2<String, String>>() {
    @Override
    public Iterator<Tuple2<String, String>> call(String s) throws Exception {
        String[] str = s.split(" ");
        Tuple2<String, String> tuple2= new Tuple2<String, String>(str[5], str[0]);
        List<Tuple2<String, String>> list = new ArrayList<>();
        list.add(tuple2);
        return list.iterator();
    }
}).distinct()
.mapToPair(new PairFunction<Tuple2<String, String>, String, Integer>() {
    @Override
    public Tuple2<String, Integer> call(Tuple2<String, String> tuple2) throws Exception {
        return new Tuple2<String, Integer>(tuple2._1, 1);
    }
})
.reduceByKey(new Function2<Integer, Integer, Integer>() {
    @Override
    public Integer call(Integer integer, Integer integer2) throws Exception {
        return integer + integer2;
    }
}).foreach(new VoidFunction<Tuple2<String, Integer>>() {
    @Override
    public void call(Tuple2<String, Integer> stringIntegerTuple2) throws Exception {
        System.out.println(stringIntegerTuple2);
    }
});

jsc.stop();

代码实现:方法二

使用countByKey