02 使用Storm的本地模式完成词频统计

前面我们已经安装了storm，storm有两种模式，一是本地模式，主要用于学习和测试，另一个是集群模式，实际生产中使用这种模式。本节将阐述如何使用本地模式的storm进行词频统计。

1 系统、软件以及前提约束

• CentOS 7 64 工作站 作者的机子ip是192.168.100.200，请读者根据自己实际情况设置
• idea 2018.1

2 操作

• 1 在idea中创建一个maven项目
• 2 修改pom.xml，在其中加入以下依赖

    <dependencies>
        <dependency>
            <!--spark依赖-->
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        <!--scala依赖-->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.11.8</version>
        </dependency>
        <!--hadoop依赖-->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.6.0-cdh5.7.0</version>
        </dependency>
        <!--hbase依赖-->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>2.0.0-cdh6.0.1</version>
        </dependency>
        <!--storm依赖-->
        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-core</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>log4j-over-slf4j</artifactId>
                </exclusion>
            </exclusions>
            <version>1.2.1</version>
        </dependency>
    </dependencies>

等待下载jar包完毕。

• 3 在src/main/java中加入RandomSentenceSpout.java做数据源

import java.util.Map;
import java.util.Random;

import org.apache.storm.spout.SpoutOutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseRichSpout;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Values;

public class RandomSentenceSpout extends BaseRichSpout {
	SpoutOutputCollector _collector;
	Random _rand;

	public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
		_collector = collector;
		_rand = new Random();
	}

	public void nextTuple() {
		String[] sentences = new String[] { "the cow jumped over the moon", "an apple a day keeps the doctor away" };
		String sentence = sentences[_rand.nextInt(sentences.length)];
		_collector.emit(new Values(sentence));
	}

	public void ack(Object id) {
	}

	public void fail(Object id) {
	}

	public void declareOutputFields(OutputFieldsDeclarer declarer) {
		declarer.declare(new Fields("word"));
	}
}

• 4 在src/main/java中加入SplitSentenceBolt.java做数据分割

import org.apache.storm.topology.BasicOutputCollector;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseBasicBolt;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;

public class SplitSentenceBolt extends BaseBasicBolt {

	private static final long serialVersionUID = -1L;

	public void execute(Tuple input, BasicOutputCollector collector) {
		String sentence = input.getString(0);
		String[] words = sentence.split(" ");
		for (String word : words) {
			word = word.trim();
			if (!word.isEmpty()) {
				word = word.toLowerCase();
				collector.emit(new Values(word));
			}
		}
	}

	public void declareOutputFields(OutputFieldsDeclarer declarer) {
		declarer.declare(new Fields("word"));
	}
}

• 5 在src/main/java中添加WordCountBolt.java做单词统计

import org.apache.storm.topology.BasicOutputCollector;
import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.topology.base.BaseBasicBolt;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Tuple;

import java.util.HashMap;
import java.util.Map;

public class WordCountBolt extends BaseBasicBolt {
	private static final long serialVersionUID = -1L;
	private Map<String, Integer> counts = new HashMap<String, Integer>();


	public void execute(Tuple tuple, BasicOutputCollector collector) {
		String word = tuple.getString(0);
		Integer count = counts.get(word);
		if (count == null) {
			count = 0;
		}
		count++;
		counts.put(word, count);
		System.out.println(Thread.currentThread().getId() + "=========== word : " + word + " count: " + count);
	}

	public void declareOutputFields(OutputFieldsDeclarer declarer) {
		declarer.declare(new Fields("word", "count"));
	}
}

• 6 在src/main/java中添加WordCountSub.java做流程约束和拓扑提交

import org.apache.storm.Config;
import org.apache.storm.LocalCluster;
import org.apache.storm.StormSubmitter;
import org.apache.storm.topology.TopologyBuilder;
import org.apache.storm.tuple.Fields;

public class WordCountSub{
	public static void main(String[] args) throws Exception {

		TopologyBuilder builder = new TopologyBuilder();

		builder.setSpout("spout", new RandomSentenceSpout(), 5);

		builder.setBolt("split", new SplitSentenceBolt(), 8).shuffleGrouping("spout");
		builder.setBolt("count", new WordCountBolt(), 12).fieldsGrouping("split", new Fields("word"));

		Config conf = new Config();
		conf.setNumWorkers(3);
		conf.setNumAckers(1);
		LocalCluster localCluster= new LocalCluster();
		localCluster.submitTopology("test",conf,builder.createTopology());
	}

}

• 7 鼠标右键执行WordCountSub.java，等待一阵子【也许较长，此过程很耗内存】，会在控制台看到输出。
  以上就是使用storm的本地模式进行词频统计。