Flume+Kafka+Storm+Hbase+HDSF+Poi整合

Flume+Kafka+Storm+Hbase+HDSF+Poi整合

需求:

针对一个网站,我们需要根据用户的行为记录日志信息,分析对我们有用的数据。

举例:这个网站www.hongten.com(当然这是一个我虚拟的电商网站),用户在这个网站里面可以有很多行为,比如注册,登录,查看,点击,双击,购买东西,加入购物车,添加记录,修改记录,删除记录,评论,登出等一系列我们熟悉的操作。这些操作都被记录在日志信息里面。我们要对日志信息进行分析。

本文中,我们对购买东西和加入购物车两个行为进行分析。然后生成相应的报表,这样我们可以通过报表查看用户在什么时候喜欢购买东西,什么时候喜欢加入购物车,从而,在相应的时间采取行动,激烈用户购买东西,推荐商品给用户加入购物车(加入购物车,这属于潜在购买用户)。

毕竟网站盈利才是我们希望达到的目的,对吧。

 

1.抽象用户行为

    // 用户的action
    public static final String[] USER_ACTION = { "Register", "Login", "View", "Click", "Double_Click", "Buy", "Shopping_Car", "Add", "Edit", "Delete", "Comment", "Logout" };

 

2.日志格式定义

115.19.62.102    海南    2018-12-20    1545286960749    1735787074662918890    www.hongten.com    Edit
27.177.45.84    新疆    2018-12-20    1545286962255    6667636903937987930    www.hongten.com    Delete
176.54.120.96    宁夏    2018-12-20    1545286962256    6988408478348165495    www.hongten.com    Comment
175.117.33.187    辽宁    2018-12-20    1545286962257    8411202446705338969    www.hongten.com    Shopping_Car
17.67.62.213    天津    2018-12-20    1545286962258    7787584752786413943    www.hongten.com    Add
137.81.41.9    海南    2018-12-20    1545286962259    6218367085234099455    www.hongten.com    Shopping_Car
125.187.107.57    山东    2018-12-20    1545286962260    3358658811146151155    www.hongten.com    Double_Click
104.167.205.87    内蒙    2018-12-20    1545286962261    2303468282544965471    www.hongten.com    Shopping_Car
64.106.149.83    河南    2018-12-20    1545286962262    8422202443986582525    www.hongten.com    Delete
138.22.156.183    浙江    2018-12-20    1545286962263    7649154147863130337    www.hongten.com    Shopping_Car
41.216.103.31    河北    2018-12-20    1545286962264    6785302169446728008    www.hongten.com    Shopping_Car
132.144.93.20    广东    2018-12-20    1545286962265    6444575166009004406    www.hongten.com    Add

日志格式:

//log fromat
String log = ip + "\t" + address + "\t" + d + "\t" + timestamp + "\t" + userid + "\t" + Common.WEB_SITE + "\t" + action;

 

3.系统架构

 

 

4.报表样式

由于我采用的是随机生成数据,所有,我们看到的结果呈现线性增长

这里我只是实现了一个小时的报表,当然,也可以做一天,一个季度,全年,三年,五年的报表,可以根据实际需求实现即可。

 

5.组件分布情况

我总共搭建了4个节点node1,node2,node3,node4(注: 4个节点上面都要有JDK)

Zookeeper安装在node1,node2,nod3

Hadoop集群在node1,node2,nod3,node4

Hbase集群在node1,node2,nod3,node4

Flume安装在node2

Kafka安装在node1,node2,node3

Storm安装在node1,node2,node3

 

6.具体实现

6.1.配置Flume

--从node2
cd flumedir

vi flume2kafka

--node2配置如下
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = node2
a1.sources.r1.port = 41414

# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = all_my_log
a1.sinks.k1.brokerList = node1:9092,node2:9092,node3:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 10000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

:wq

 

6.2.启动Zookeeper

--关闭防火墙node1,node2,node3,node4
service iptables stop

--启动Zookeeper,在node1,node2,node3
zkServer.sh start

 

6.3.启动Kafka

--启动kafka
--分别进入node1,node2,node3
cd /root/kafka/kafka_2.10-0.8.2.2
./start-kafka.sh

 

6.4.启动Flume服务

--进入node2,启动
cd /root/flumedir
flume-ng agent -n a1 -c conf -f flume2kafka -Dflume.root.logger=DEBUG,console

 

6.5.产生日志信息并写入到Flume

运行java 代码,产生日志信息并写入到Flume服务器

package com.b510.big.data.flume.client;

import java.nio.charset.Charset;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Random;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.api.RpcClient;
import org.apache.flume.api.RpcClientFactory;
import org.apache.flume.event.EventBuilder;

/**
 * @author Hongten
 * 
 *         功能: 模拟产生用户日志信息,并且向Flume发送数据
 */
public class FlumeClient {

    public static void main(String[] args) {
        ExecutorService exec = Executors.newCachedThreadPool();
        exec.execute(new GenerateDataAndSend2Flume());

        exec.shutdown();
    }

}

class GenerateDataAndSend2Flume implements Runnable {

    FlumeRPCClient flumeRPCClient;
    static Random random = new Random();

    GenerateDataAndSend2Flume() {
        // 初始化RPC客户端
        flumeRPCClient = new FlumeRPCClient();
        flumeRPCClient.init(Common.FLUME_HOST_NAME, Common.FLUME_PORT);
    }

    @Override
    public void run() {
        while (true) {
            Date date = new Date();
            SimpleDateFormat simpleDateFormat = new SimpleDateFormat(Common.DATE_FORMAT_YYYYDDMM);
            String d = simpleDateFormat.format(date);
            Long timestamp = new Date().getTime();
            // ip地址生成
            String ip = random.nextInt(Common.MAX_IP_NUMBER) + "." + random.nextInt(Common.MAX_IP_NUMBER) + "." + random.nextInt(Common.MAX_IP_NUMBER) + "." + random.nextInt(Common.MAX_IP_NUMBER);
            // ip地址对应的address(这里是为了构造数据,并没有按照真实的ip地址,找到对应的address)
            String address = Common.ADDRESS[random.nextInt(Common.ADDRESS.length)];

            Long userid = Math.abs(random.nextLong());
            String action = Common.USER_ACTION[random.nextInt(Common.USER_ACTION.length)];
            // 日志信息构造
            // example : 199.80.45.117 云南 2018-12-20 1545285957720 3086250439781555145 www.hongten.com Buy
            String data = ip + "\t" + address + "\t" + d + "\t" + timestamp + "\t" + userid + "\t" + Common.WEB_SITE + "\t" + action;
            //System.out.println(data);

            // 往Flume发送数据
            flumeRPCClient.sendData2Flume(data);

            try {
                TimeUnit.MICROSECONDS.sleep(random.nextInt(1000));
            } catch (InterruptedException e) {
                flumeRPCClient.cleanUp();
                System.out.println("interrupted exception : " + e);
            }
        }
    }
}

class FlumeRPCClient {

    private RpcClient client;
    private String hostname;
    private int port;

    public void init(String hostname, int port) {
        this.hostname = hostname;
        this.port = port;
        this.client = getRpcClient(hostname, port);
    }

    public void sendData2Flume(String data) {
        Event event = EventBuilder.withBody(data, Charset.forName(Common.CHAR_FORMAT));

        try {
            client.append(event);
        } catch (EventDeliveryException e) {
            cleanUp();
            client = null;
            client = getRpcClient(hostname, port);
        }
    }

    public RpcClient getRpcClient(String hostname, int port) {
        return RpcClientFactory.getDefaultInstance(hostname, port);
    }

    public void cleanUp() {
        // Close the RPC connection
        client.close();
    }
}

// 所有的常量定义
class Common {
    public static final String CHAR_FORMAT = "UTF-8";

    public static final String DATE_FORMAT_YYYYDDMM = "yyyy-MM-dd";

    // this is a test web site
    public static final String WEB_SITE = "www.hongten.com";

    // 用户的action
    public static final String[] USER_ACTION = { "Register", "Login", "View", "Click", "Double_Click", "Buy", "Shopping_Car", "Add", "Edit", "Delete", "Comment", "Logout" };

    public static final int MAX_IP_NUMBER = 224;
    // ip所对应的地址
    public static String[] ADDRESS = { "北京", "天津", "上海", "广东", "重庆", "河北", "山东", "河南", "云南", "山西", "甘肃", "安徽", "福建", "黑龙江", "海南", "四川", "贵州", "宁夏", "新疆", "湖北", "湖南", "山西", "辽宁", "吉林", "江苏", "浙江", "青海", "江西", "西藏", "内蒙", "广西", "香港", "澳门", "台湾", };

    // Flume conf
    public static final String FLUME_HOST_NAME = "node2";
    public static final int FLUME_PORT = 41414;

}

 

6.6.监听Kafka

--进入node3,启动kafka消费者
cd /home/kafka-2.10/bin
./kafka-console-consumer.sh --zookeeper node1,node2,node3 --from-beginning --topic all_my_log

 

运行效果:

168.208.193.207    安徽    2018-12-20    1545287646527    5462770148222682599    www.hongten.com    Login
103.143.79.127    新疆    2018-12-20    1545287646529    3389475301916412717    www.hongten.com    Login
111.208.80.39    山东    2018-12-20    1545287646531    535601622597096753    www.hongten.com    Shopping_Car
105.30.86.46    四川    2018-12-20    1545287646532    7825340079790811845    www.hongten.com    Login
205.55.33.74    新疆    2018-12-20    1545287646533    4228838365367235561    www.hongten.com    Logout
34.44.60.134    安徽    2018-12-20    1545287646536    702584874247456732    www.hongten.com    Double_Click
154.169.15.145    广东    2018-12-20    1545287646537    1683351753576425036    www.hongten.com    View
126.28.192.28    湖南    2018-12-20    1545287646538    8319814684518483148    www.hongten.com    Edit
5.140.156.73    台湾    2018-12-20    1545287646539    7432409906375230025    www.hongten.com    Logout
72.175.210.95    西藏    2018-12-20    1545287646540    5233707593244910849    www.hongten.com    View
121.25.190.25    广西    2018-12-20    1545287646541    268200251881841673    www.hongten.com    Buy

 

6.7.在Kafka创建Topic

--进入node1,创建一个topic:filtered_log
--设置3个partitions
--replication-factor=3
./kafka-topics.sh --zookeeper node1,node2,node3 --create --topic filtered_log --partitions 3 --replication-factor 3

 

6.8.Storm清洗数据

  • Storm从Kafka消费数据
  • Storm对数据进行筛选(Buy-已经购买,Shopping_Car-潜在购买)
  • Storm把筛选的数据放入到Kafka
package com.b510.big.data.storm.process;

import java.util.ArrayList;
import java.util.List;
import java.util.Properties;

import storm.kafka.KafkaSpout;
import storm.kafka.SpoutConfig;
import storm.kafka.StringScheme;
import storm.kafka.ZkHosts;
import storm.kafka.bolt.KafkaBolt;
import storm.kafka.bolt.mapper.FieldNameBasedTupleToKafkaMapper;
import storm.kafka.bolt.selector.DefaultTopicSelector;
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.generated.AlreadyAliveException;
import backtype.storm.generated.InvalidTopologyException;
import backtype.storm.spout.SchemeAsMultiScheme;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;

public class LogFilterTopology {

    public static void main(String[] args) {

        ZkHosts zkHosts = new ZkHosts(Common.ZOOKEEPER_QUORUM);
        //Spout从'filtered_log' topic里面获取数据
        SpoutConfig spoutConfig = new SpoutConfig(zkHosts, Common.ALL_MY_LOG_TOPIC, Common.ZOOKEEPER_ROOT, Common.ZOOKEEPER_ID);
        List<String> zkServers = new ArrayList<>();
        for (String host : zkHosts.brokerZkStr.split(",")) {
            zkServers.add(host.split(":")[0]);
        }

        spoutConfig.zkServers = zkServers;
        spoutConfig.zkPort = Common.ZOOKEEPER_PORT;
        spoutConfig.forceFromStart = true;
        spoutConfig.socketTimeoutMs = 60 * 60 * 1000;
        spoutConfig.scheme = new SchemeAsMultiScheme(new StringScheme());

        // 创建KafkaSpout
        KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);

        TopologyBuilder builder = new TopologyBuilder();
        // Storm从Kafka消费数据
        builder.setSpout(Common.KAFKA_SPOUT, kafkaSpout, 3);
        // Storm对数据进行筛选(Buy-已经购买,Shopping_Car-潜在购买)
        builder.setBolt(Common.FILTER_BOLT, new FilterBolt(), 8).shuffleGrouping(Common.KAFKA_SPOUT);

        // 创建KafkaBolt
        @SuppressWarnings({ "unchecked", "rawtypes" })
        KafkaBolt kafkaBolt = new KafkaBolt().withTopicSelector(new DefaultTopicSelector(Common.FILTERED_LOG_TOPIC)).withTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper());

        // Storm把筛选的数据放入到Kafka
        builder.setBolt(Common.KAFKA_BOLT, kafkaBolt, 2).shuffleGrouping(Common.FILTER_BOLT);

        Properties props = new Properties();
        props.put("metadata.broker.list", Common.STORM_METADATA_BROKER_LIST);
        props.put("request.required.acks", Common.STORM_REQUEST_REQUIRED_ACKS);
        props.put("serializer.class", Common.STORM_SERILIZER_CLASS);

        Config conf = new Config();
        conf.put("kafka.broker.properties", props);

        conf.put(Config.STORM_ZOOKEEPER_SERVERS, zkServers);

        if (args == null || args.length == 0) {
            // 本地方式运行
            LocalCluster localCluster = new LocalCluster();
            localCluster.submitTopology("storm-kafka-topology", conf, builder.createTopology());
        } else {
            // 集群方式运行
            conf.setNumWorkers(3);
            try {
                StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
            } catch (AlreadyAliveException | InvalidTopologyException e) {
                System.out.println("error : " + e);
            }
        }
    }
}

class FilterBolt extends BaseBasicBolt {

    private static final long serialVersionUID = 1L;

    @Override
    public void execute(Tuple input, BasicOutputCollector collector) {
        String logStr = input.getString(0);
        // 只针对我们感兴趣的关键字进行过滤
        // 这里我们过滤包含'Buy', 'Shopping_Car'的日志信息
        if (logStr.contains(Common.KEY_WORD_BUY) || logStr.contains(Common.KEY_WORD_SHOPPING_CAR)) {
            collector.emit(new Values(logStr));
        }
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields(FieldNameBasedTupleToKafkaMapper.BOLT_MESSAGE));
    }
}

class Common {
    public static final String ALL_MY_LOG_TOPIC = "all_my_log";
    public static final String FILTERED_LOG_TOPIC = "filtered_log";
    
    public static final String DATE_FORMAT_YYYYDDMMHHMMSS = "yyyyMMddHHmmss";
    public static final String DATE_FORMAT_HHMMSS = "HHmmss";
    public static final String DATE_FORMAT_HHMMSS_DEFAULT_VALUE = "000001";

    public static final String HBASE_ZOOKEEPER_LIST = "node1:2888,node2:2888,node3:2888";
    public static final int ZOOKEEPER_PORT = 2181;
    public static final String ZOOKEEPER_QUORUM = "node1:" + ZOOKEEPER_PORT + ",node2:" + ZOOKEEPER_PORT + ",node3:" + ZOOKEEPER_PORT + "";
    public static final String ZOOKEEPER_ROOT = "/MyKafka";
    public static final String ZOOKEEPER_ID = "MyTrack";

    public static final String KAFKA_SPOUT = "kafkaSpout";
    public static final String FILTER_BOLT = "filterBolt";
    public static final String PROCESS_BOLT = "processBolt";
    public static final String HBASE_BOLT = "hbaseBolt";
    public static final String KAFKA_BOLT = "kafkaBolt";

    // Storm Conf
    public static final String STORM_METADATA_BROKER_LIST = "node1:9092,node2:9092,node3:9092";
    public static final String STORM_REQUEST_REQUIRED_ACKS = "1";
    public static final String STORM_SERILIZER_CLASS = "kafka.serializer.StringEncoder";

    // key word
    public static final String KEY_WORD_BUY = "Buy";
    public static final String KEY_WORD_SHOPPING_CAR = "Shopping_Car";
    
    //hbase
    public static final String TABLE_USER_ACTION = "t_user_actions";
    public static final String COLUMN_FAMILY = "cf";
    //间隔多少秒写入Hbase一次
    public static final int WRITE_RECORD_TO_TABLE_PER_SECOND = 1;
    public static final int TABLE_MAX_VERSION = (60/WRITE_RECORD_TO_TABLE_PER_SECOND) * 60 * 24;
}

 

6.9.监听Kafka

--进入node3,启动kafka消费者
cd /home/kafka-2.10/bin
./kafka-console-consumer.sh --zookeeper node1,node2,node3 --from-beginning --topic filtered_log

 

效果:

87.26.135.185    黑龙江    2018-12-20    1545290594658    7290881731606227972    www.hongten.com    Shopping_Car
60.96.96.38    青海    2018-12-20    1545290594687    6935901257286057015    www.hongten.com    Shopping_Car
43.159.110.193    江苏    2018-12-20    1545290594727    7096698224110515553    www.hongten.com    Shopping_Car
21.103.139.11    山西    2018-12-20    1545290594693    7805867078876194442    www.hongten.com    Shopping_Car
139.51.213.184    广东    2018-12-20    1545290594729    8048796865619113514    www.hongten.com    Buy
58.213.148.89    河北    2018-12-20    1545290594708    5176551342435592748    www.hongten.com    Buy
36.205.221.116    湖南    2018-12-20    1545290594715    4484717918039766421    www.hongten.com    Shopping_Car
135.194.103.53    北京    2018-12-20    1545290594769    4833011508087432349    www.hongten.com    Shopping_Car
180.21.100.66    贵州    2018-12-20    1545290594752    5270357330431599426    www.hongten.com    Buy
167.71.65.70    山西    2018-12-20    1545290594790    275898530145861990    www.hongten.com    Buy
125.51.21.199    宁夏    2018-12-20    1545290594814    3613499600574777198    www.hongten.com    Buy

 

 

6.10.Storm再次消费Kafka数据处理后保存数据到Hbase

  • Storm再次从Kafka消费数据
  • Storm对数据进行统计(Buy-已经购买人数,Shopping_Car-潜在购买人数)
  • Storm将数据写入到Hbase
package com.b510.big.data.storm.process;

import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;
import java.util.Properties;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HConnection;
import org.apache.hadoop.hbase.client.HConnectionManager;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.Put;

import storm.kafka.KafkaSpout;
import storm.kafka.SpoutConfig;
import storm.kafka.StringScheme;
import storm.kafka.ZkHosts;
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.generated.AlreadyAliveException;
import backtype.storm.generated.InvalidTopologyException;
import backtype.storm.spout.SchemeAsMultiScheme;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.IBasicBolt;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;

public class LogProcessTopology {

    public static void main(String[] args) {

        ZkHosts zkHosts = new ZkHosts(Common.ZOOKEEPER_QUORUM);
        //Spout从'filtered_log' topic里面获取数据
        SpoutConfig spoutConfig = new SpoutConfig(zkHosts, Common.FILTERED_LOG_TOPIC, Common.ZOOKEEPER_ROOT, Common.ZOOKEEPER_ID);
        List<String> zkServers = new ArrayList<>();
        for (String host : zkHosts.brokerZkStr.split(",")) {
            zkServers.add(host.split(":")[0]);
        }

        spoutConfig.zkServers = zkServers;
        spoutConfig.zkPort = Common.ZOOKEEPER_PORT;
        spoutConfig.forceFromStart = true;
        spoutConfig.socketTimeoutMs = 60 * 60 * 1000;
        spoutConfig.scheme = new SchemeAsMultiScheme(new StringScheme());

        // 创建KafkaSpout
        KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);

        TopologyBuilder builder = new TopologyBuilder();
        // Storm再次从Kafka消费数据
        builder.setSpout(Common.KAFKA_SPOUT, kafkaSpout, 3);
        // Storm对数据进行统计(Buy-已经购买人数,Shopping_Car-潜在购买人数)
        builder.setBolt(Common.PROCESS_BOLT, new ProcessBolt(), 3).shuffleGrouping(Common.KAFKA_SPOUT);
        // Storm将数据写入到Hbase
        builder.setBolt(Common.HBASE_BOLT, new HbaseBolt(), 3).shuffleGrouping(Common.PROCESS_BOLT);

        Properties props = new Properties();
        props.put("metadata.broker.list", Common.STORM_METADATA_BROKER_LIST);
        props.put("request.required.acks", Common.STORM_REQUEST_REQUIRED_ACKS);
        props.put("serializer.class", Common.STORM_SERILIZER_CLASS);

        Config conf = new Config();
        conf.put("kafka.broker.properties", props);

        conf.put(Config.STORM_ZOOKEEPER_SERVERS, zkServers);

        if (args == null || args.length == 0) {
            // 本地方式运行
            LocalCluster localCluster = new LocalCluster();
            localCluster.submitTopology("storm-kafka-topology", conf, builder.createTopology());
        } else {
            // 集群方式运行
            conf.setNumWorkers(3);
            try {
                StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
            } catch (AlreadyAliveException | InvalidTopologyException e) {
                System.out.println("error : " + e);
            }
        }
        
    }
}

class ProcessBolt extends BaseBasicBolt {

    private static final long serialVersionUID = 1L;

    @Override
    public void execute(Tuple input, BasicOutputCollector collector) {
        String logStr = input.getString(0);
        if (logStr != null) {
            String infos[] = logStr.split("\\t");
            //180.21.100.66    贵州    2018-12-20    1545290594752    5270357330431599426    www.hongten.com    Buy
            collector.emit(new Values(infos[2], infos[6]));
        }
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("date", "user_action"));
    }
}

class HbaseBolt implements IBasicBolt {
    private static final long serialVersionUID = 1L;

    HBaseDAO hBaseDAO = null;
    
    SimpleDateFormat simpleDateFormat = null;
    SimpleDateFormat simpleDateFormatHHMMSS = null;
    
    int userBuyCount = 0;
    int userShoopingCarCount = 0;
    
    //这里要考虑避免频繁写入数据到hbase
    int writeToHbaseMaxNum = Common.WRITE_RECORD_TO_TABLE_PER_SECOND * 1000;
    long begin = System.currentTimeMillis();
    long end = 0;
    
    @SuppressWarnings("rawtypes")
    @Override
    public void prepare(Map map, TopologyContext context) {
        hBaseDAO = new HBaseDAOImpl();
        simpleDateFormat = new SimpleDateFormat(Common.DATE_FORMAT_YYYYDDMMHHMMSS);
        simpleDateFormatHHMMSS = new SimpleDateFormat(Common.DATE_FORMAT_HHMMSS);
        hBaseDAO.createTable(Common.TABLE_USER_ACTION, new String[]{Common.COLUMN_FAMILY}, Common.TABLE_MAX_VERSION);
    }

    @Override
    public void execute(Tuple input, BasicOutputCollector collector) {
        // 如果时间是第二天的凌晨1s
        // 需要对count做清零处理
        //不过这里的判断不是很准确,因为在此时,可能前一天的数据还没有处理完
        if (simpleDateFormatHHMMSS.format(new Date()).equals(Common.DATE_FORMAT_HHMMSS_DEFAULT_VALUE)) {
            userBuyCount = 0;
            userShoopingCarCount = 0;
        }
        
        if (input != null) {
            // base one ProcessBolt.declareOutputFields()
            String date = input.getString(0);
            String userAction = input.getString(1);

            if (userAction.equals(Common.KEY_WORD_BUY)) {
                //同一个user在一天之内可以重复'Buy'动作
                userBuyCount++;
            }

            if (userAction.equals(Common.KEY_WORD_SHOPPING_CAR)) {
                userShoopingCarCount++;
            }

            end = System.currentTimeMillis();
            if ((end - begin) > writeToHbaseMaxNum) {
                System.out.println("hbase_key: " + Common.KEY_WORD_BUY + "_" + date + " , userBuyCount: " + userBuyCount + ", userShoopingCarCount :" + userShoopingCarCount);
                
                //往hbase中写入数据
                String quailifer = simpleDateFormat.format(new Date());
                hBaseDAO.insert(Common.TABLE_USER_ACTION , 
                        Common.KEY_WORD_BUY + "_" + date, 
                        Common.COLUMN_FAMILY, 
                        new String[] { quailifer },
                        new String[] { "{user_buy_count:" + userBuyCount + "}" }
                        );
                hBaseDAO.insert(Common.TABLE_USER_ACTION , 
                        Common.KEY_WORD_SHOPPING_CAR + "_" + date, 
                        Common.COLUMN_FAMILY, 
                        new String[] { quailifer },
                        new String[] { "{user_shopping_car_count:" + userShoopingCarCount + "}" }
                        );
                begin = System.currentTimeMillis();
            }
        }
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {

    }

    @Override
    public Map<String, Object> getComponentConfiguration() {
        return null;
    }

    @Override
    public void cleanup() {

    }
}

interface HBaseDAO {
    public void createTable(String tableName, String[] columnFamilys, int maxVersion);
    public void insert(String tableName, String rowKey, String family, String quailifer[], String value[]);
}

class HBaseDAOImpl implements HBaseDAO {

    HConnection hConnection = null;
    static Configuration conf = null;

    public HBaseDAOImpl() {
        conf = new Configuration();
        conf.set("hbase.zookeeper.quorum", Common.HBASE_ZOOKEEPER_LIST);
        try {
            hConnection = HConnectionManager.createConnection(conf);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
    
    public void createTable(String tableName, String[] columnFamilys, int maxVersion) {
        try {
            HBaseAdmin admin = new HBaseAdmin(conf);
            if (admin.tableExists(tableName)) {
                System.err.println("table existing in hbase.");
            } else {
                HTableDescriptor tableDesc = new HTableDescriptor(TableName.valueOf(tableName));
                for (String columnFamily : columnFamilys) {
                    HColumnDescriptor hColumnDescriptor = new HColumnDescriptor(columnFamily);
                    hColumnDescriptor.setMaxVersions(maxVersion);
                    tableDesc.addFamily(hColumnDescriptor);
                }

                admin.createTable(tableDesc);
                System.err.println("table is created.");
            }
            admin.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
    
    @Override
    public void insert(String tableName, String rowKey, String family, String quailifer[], String value[]) {
        HTableInterface table = null;
        try {
            table = hConnection.getTable(tableName);
            Put put = new Put(rowKey.getBytes());
            for (int i = 0; i < quailifer.length; i++) {
                String col = quailifer[i];
                String val = value[i];
                put.add(family.getBytes(), col.getBytes(), val.getBytes());
            }
            table.put(put);
            System.err.println("save record successfuly.");
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                table.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

}

 

Storm处理逻辑:

1.每秒向Hbase写入数据

2.明天凌晨会重置数据

如果,我们一直运行上面的程序,那么,系统就会一直往Hbase里面写入数据,那么这样,我们就可以采集到我们生成报表的数据了。

那么下面就是报表实现

 

6.11.读取Hbase数据通过POI生成Excel Report

  • 读取Hbase数据
  • 通过POI生成Excel报表
package com.b510.big.data.poi;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.CellUtil;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HConnection;
import org.apache.hadoop.hbase.client.HConnectionManager;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.Result;
import org.apache.poi.xssf.usermodel.XSSFCell;
import org.apache.poi.xssf.usermodel.XSSFSheet;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;

public class ReportUtil {

    public static void main(String[] args) throws Exception {

        String year = "2018";
        String month = "12";
        String day = "21";
        String hour = "14";

        generateReport(year, month, day, hour);
    }

    private static void generateReport(String year, String month, String day, String hour) {
        HBaseDAO hBaseDAO = new HBaseDAOImpl();
        // format: yyyyMMddHH
        String begin = year + month + day + hour;
        String[] split = generateQuailifers(begin);

        List<Integer> userBuyCountList = getData(hBaseDAO, year, month, day, split, Common.KEY_WORD_BUY);
        List<Integer> userShoppingCarCountList = getData(hBaseDAO, year, month, day, split, Common.KEY_WORD_SHOPPING_CAR);

        //System.err.println(userBuyCountList.size());
        //System.err.println(userShoppingCarCountList.size());

        writeExcel(year, month, day, hour, userBuyCountList, userShoppingCarCountList);
    }

    private static void writeExcel(String year, String month, String day, String hour, List<Integer> userBuyCountList, List<Integer> userShoppingCarCountList) {
        try {
            File file = new File(Common.REPORT_TEMPLATE);
            InputStream in = new FileInputStream(file);
            XSSFWorkbook wb = new XSSFWorkbook(in);
            XSSFSheet sheet = wb.getSheetAt(0);
            if (sheet != null) {
                XSSFCell cell = null;

                cell = sheet.getRow(0).getCell(0);
                cell.setCellValue("One Hour Report-" + year + "-" + month + "-" + day + " From " + hour + ":00 To " + hour + ":59");

                putData(userBuyCountList, sheet, 3);
                putData(userShoppingCarCountList, sheet, 7);

                FileOutputStream out = new FileOutputStream(Common.REPORT_ONE_HOUR);
                wb.write(out);
                out.close();
                System.err.println("done.");
            }
        } catch (Exception e) {
            System.err.println("Exception" + e);
        }
    }

    private static void putData(List<Integer> userBuyCountList, XSSFSheet sheet, int rowNum) {
        XSSFCell cell;
        if (userBuyCountList != null && userBuyCountList.size() > 0) {
            for (int i = 0; i < userBuyCountList.size(); i++) {
                cell = sheet.getRow(rowNum).getCell(i + 1);
                cell.setCellValue(userBuyCountList.get(i));
            }
        }
    }

    private static List<Integer> getData(HBaseDAO hBaseDAO, String year, String month, String day, String[] split, String preKey) {
        List<Integer> list = new ArrayList<Integer>();
        Result rs = hBaseDAO.getOneRowAndMultiColumn(Common.TABLE_USER_ACTION, preKey + "_" + year + "-" + month + "-" + day, split);
        for (Cell cell : rs.rawCells()) {
            String value = new String(CellUtil.cloneValue(cell)).split(":")[1].trim();
            value = value.substring(0, value.length() - 1);
            list.add(Integer.valueOf(value));
        }
        return list;
    }

    private static String[] generateQuailifers(String begin) {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < 60;) {

            if (i == 0 || i == 5) {
                sb.append(begin).append("0").append(i).append("00").append(",");
            } else {
                sb.append(begin).append(i).append("00").append(",");
            }
            i = i + 5;
        }
        sb.append(begin).append("5959");
        String sbStr = sb.toString();
        String[] split = sbStr.split(",");
        return split;
    }
}

interface HBaseDAO {
    Result getOneRowAndMultiColumn(String tableName, String rowKey, String[] cols);
}

class HBaseDAOImpl implements HBaseDAO {

    HConnection hConnection = null;
    static Configuration conf = null;

    public HBaseDAOImpl() {
        conf = new Configuration();
        conf.set("hbase.zookeeper.quorum", Common.HBASE_ZOOKEEPER_LIST);
        try {
            hConnection = HConnectionManager.createConnection(conf);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    @Override
    public Result getOneRowAndMultiColumn(String tableName, String rowKey, String[] cols) {
        HTableInterface table = null;
        Result rsResult = null;
        try {
            table = hConnection.getTable(tableName);
            Get get = new Get(rowKey.getBytes());
            for (int i = 0; i < cols.length; i++) {
                get.addColumn(Common.COLUMN_FAMILY.getBytes(), cols[i].getBytes());
            }
            rsResult = table.get(get);
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                table.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        return rsResult;
    }

}

class Common {

    // report
    public static final String REPORT_TEMPLATE = "./resources/report.xlsx";
    public static final String REPORT_ONE_HOUR = "./resources/one_report.xlsx";

    public static final String DATE_FORMAT_YYYYDDMMHHMMSS = "yyyyMMddHHmmss";

    public static final String HBASE_ZOOKEEPER_LIST = "node1:2888,node2:2888,node3:2888";

    // key word
    public static final String KEY_WORD_BUY = "Buy";
    public static final String KEY_WORD_SHOPPING_CAR = "Shopping_Car";

    // hbase
    public static final String TABLE_USER_ACTION = "t_user_actions";
    public static final String COLUMN_FAMILY = "cf";

}

 

7.源码下载

Source Code:Flume_Kafka_Storm_Hbase_Hdfs_Poi_src.zip

相应的Jar文件,由于so big,自己根据import *信息加入。

 

8.总结

学习Big Data一段时间了,通过自己的学习和摸索,实现自己想要的应用,还是很有成就感的哈....当然,踩地雷也是一种不错的体验...:)

 

========================================================

More reading,and english is important.

I'm Hongten

 

大哥哥大姐姐,觉得有用打赏点哦!你的支持是我最大的动力。谢谢。
Hongten博客排名在100名以内。粉丝过千。
Hongten出品,必是精品。

E | hongtenzone@foxmail.com  B | http://www.cnblogs.com/hongten

========================================================

 

posted @ 2018-12-21 16:50  Hongten  阅读(1032)  评论(0编辑  收藏
Fork me on GitHub