Storm 01之 Storm基本概念及第一个demo

2.1 Storm基本概念

在运行一个Storm任务之前，需要了解一些概念：

Topologies ：[tə'pɑ:lədʒɪ]拓扑结构
Streams
Spouts：[spaʊt]喷出; 喷射; 滔滔不绝地讲; 把…典当掉;
Bolts:[bəʊlt]筛选; 囫囵吞下; （把门、窗等）闩上; 突然说出，脱口说出;
Stream groupings
Reliability
Tasks
Workers
Configuration
supervisor:英[ˈsu:pəvaɪzə(r)]监督者，管理者; 镇长;
nimbus：英[ˈnɪmbəs]美[ˈnɪmbəs]（人和物产生的）灵气，（神像头上的）光轮;

Storm集群和Hadoop集群表面上看很类似。但是Hadoop上运行的是MapReduce jobs，而在Storm上运行的是拓扑（topology），这两者之间是非常不一样的。一个关键的区别是：一个MapReduce job最终会结束，而一个topology永远会运行（除非你手动kill掉）。

在Storm的集群里面有两种节点：控制节点（master node）和工作节点（worker node）。控制节点上面运行一个叫Nimbus(nimbus[英][ˈnɪmbəs][美][ˈnɪmbəs]

n.（人和物产生的）灵气，（神像头上的）光轮; )后台程序，它的作用类似Hadoop里面的JobTracker。Nimbus负责在集群里面分发代码，分配计算任务给机器，并且监控状态。

每一个工作节点上面运行一个叫做Supervisor的节点。Supervisor会监听分配给它那台机器的工作，根据需要启动/关闭工作进程。每一个工作进程执行一个topology的一个子集；一个运行的topology由运行在很多机器上的很多工作进程组成。

Nimbus和Supervisor之间的所有协调工作都是通过Zookeeper集群完成。另外，Nimbus进程和Supervisor进程都是快速失败（fail-fast)和无状态的。所有的状态要么在zookeeper里面，要么在本地磁盘上。这也就意味着你可以用kill -9来杀死Nimbus和Supervisor进程，然后再重启它们，就好像什么都没有发生过。这个设计使得Storm异常的稳定。

2.1.1 Topologies

一个topology是spouts和bolts组成的图，通过stream groupings将图中的spouts和bolts连接起来，如下图：

一个topology会一直运行直到你手动kill掉，Storm自动重新分配执行失败的任务，并且Storm可以保证你不会有数据丢失（如果开启了高可靠性的话）。如果一些机器意外停机它上面的所有任务会被转移到其他机器上。

运行一个topology很简单。首先，把你所有的代码以及所依赖的jar打进一个jar包。然后运行类似下面的这个命令：

storm jar all-my-code.jar backtype.storm.MyTopology arg1 arg2

这个命令会运行主类: backtype.strom.MyTopology, 参数是arg1, arg2。这个类的main函数定义这个topology并且把它提交给Nimbus。storm jar负责连接到Nimbus并且上传jar包。

Topology的定义是一个Thrift结构，并且Nimbus就是一个Thrift服务，你可以提交由任何语言创建的topology。上面的方面是用JVM-based语言提交的最简单的方法。

2.1.2 Streams

消息流stream是storm里的关键抽象。一个消息流是一个没有边界的tuple序列，而这些tuple序列会以一种分布式的方式并行地创建和处理。通过对stream中tuple序列中每个字段命名来定义stream。在默认的情况下，tuple的字段类型可以是：integer，long，short， byte，string，double，float，boolean和byte array。你也可以自定义类型（只要实现相应的序列化器）。

每个消息流在定义的时候会被分配给一个id，因为单向消息流使用的相当普遍， OutputFieldsDeclarer定义了一些方法让你可以定义一个stream而不用指定这个id。在这种情况下这个stream会分配个值为‘default’默认的id 。

Storm提供的最基本的处理stream的原语是spout和bolt。你可以实现spout和bolt提供的接口来处理你的业务逻辑。

2.1.3 Spouts

消息源spout是Storm里面一个topology里面的消息生产者。一般来说消息源会从一个外部源读取数据并且向topology里面发出消息：tuple。Spout可以是可靠的也可以是不可靠的。如果这个tuple没有被storm成功处理，可靠的消息源spouts可以重新发射一个tuple，但是不可靠的消息源spouts一旦发出一个tuple就不能重发了。

消息源可以发射多条消息流stream。使用OutputFieldsDeclarer.declareStream来定义多个stream，然后使用SpoutOutputCollector来发射指定的stream。

Spout类里面最重要的方法是nextTuple。要么发射一个新的tuple到topology里面或者简单的返回如果已经没有新的tuple。要注意的是nextTuple方法不能阻塞，因为storm在同一个线程上面调用所有消息源spout的方法。

另外两个比较重要的spout方法是ack和fail。storm在检测到一个tuple被整个topology成功处理的时候调用ack，否则调用fail。storm只对可靠的spout调用ack和fail。

2.1.4 Bolts

所有的消息处理逻辑被封装在bolts里面。Bolts可以做很多事情：过滤，聚合，查询数据库等等。

Bolts可以简单的做消息流的传递。复杂的消息流处理往往需要很多步骤，从而也就需要经过很多bolts。比如算出一堆图片里面被转发最多的图片就至少需要两步：第一步算出每个图片的转发数量。第二步找出转发最多的前10个图片。（如果要把这个过程做得更具有扩展性那么可能需要更多的步骤）。

Bolts可以发射多条消息流，使用OutputFieldsDeclarer.declareStream定义stream，使用OutputCollector.emit来选择要发射的stream。

Bolts的主要方法是execute, 它以一个tuple作为输入，bolts使用OutputCollector来发射tuple，bolts必须要为它处理的每一个tuple调用OutputCollector的ack方法，以通知Storm这个tuple被处理完成了，从而通知这个tuple的发射者spouts。一般的流程是： bolts处理一个输入tuple, 发射0个或者多个tuple, 然后调用ack通知storm自己已经处理过这个tuple了。storm提供了一个IBasicBolt会自动调用ack。

2.1.5 Stream groupings

定义一个topology的其中一步是定义每个bolt接收什么样的流作为输入。stream grouping就是用来定义一个stream应该如果分配数据给bolts上面的多个tasks。

Storm里面有7种类型的stream grouping（7种stream grouping策略）

Shuffle Grouping: 随机分组，随机派发stream里面的tuple，保证每个bolt接收到的tuple数目大致相同。
Fields Grouping：按字段分组，比如按userid来分组，具有同样userid的tuple会被分到相同的Bolts里的一个task，而不同的userid则会被分配到不同的bolts里的task。
All Grouping：广播发送，对于每一个tuple，所有的bolts都会收到。
Global Grouping：全局分组，这个tuple被分配到storm中的一个bolt的其中一个task。再具体一点就是分配给id值最低的那个task。
Non Grouping：不分组，这个分组的意思是说stream不关心到底谁会收到它的tuple。目前这种分组和Shuffle grouping是一样的效果，有一点不同的是storm会把这个bolt放到这个bolt的订阅者同一个线程里面去执行。
Direct Grouping：直接分组，这是一种比较特别的分组方法，用这种分组意味着消息的发送者指定由消息接收者的哪个task处理这个消息。只有被声明为Direct Stream的消息流可以声明这种分组方法。而且这种消息tuple必须使用emitDirect方法来发射。消息处理者可以通过TopologyContext来获取处理它的消息的task的id （OutputCollector.emit方法也会返回task的id）。
Local or shuffle grouping：如果目标bolt有一个或者多个task在同一个工作进程中，tuple将会被随机发生给这些tasks。否则，和普通的Shuffle Grouping行为一致。

2.1.6 Reliability

Storm保证每个tuple会被topology完整的执行。Storm会追踪由每个spout tuple所产生的tuple树（一个bolt处理一个tuple之后可能会发射别的tuple从而形成树状结构），并且跟踪这棵tuple树什么时候成功处理完。每个topology都有一个消息超时的设置，如果storm在这个超时的时间内检测不到某个tuple树到底有没有执行成功，那么topology会把这个tuple标记为执行失败，并且过一会儿重新发射这个tuple。

为了利用Storm的可靠性特性，在你发出一个新的tuple以及你完成处理一个tuple的时候你必须要通知storm。这一切是由OutputCollector来完成的。通过emit方法来通知一个新的tuple产生了，通过ack方法通知一个tuple处理完成了。

Storm的可靠性我们在第四章会深入介绍。

2.1.7 Tasks

每一个spout和bolt会被当作很多task在整个集群里执行。每一个executor对应到一个线程，在这个线程上运行多个task，而stream grouping则是定义怎么从一堆task发射tuple到另外一堆task。你可以调用TopologyBuilder类的setSpout和setBolt来设置并行度（也就是有多少个task）。

2.1.8 Workers

一个topology可能会在一个或者多个worker（工作进程）里面执行，每个worker是一个物理JVM并且执行整个topology的一部分。比如，对于并行度是300的topology来说，如果我们使用50个工作进程来执行，那么每个工作进程会处理其中的6个tasks。Storm会尽量均匀的工作分配给所有的worker。

2.1.9 Configuration

Storm里面有一堆参数可以配置来调整Nimbus, Supervisor(supervisor[英][ˈsu:pəvaɪzə(r)][美][ˈsu:pərvaɪzə(r)]n.监督者，管理者; 镇长; )以及正在运行的topology的行为，一些配置是系统级别的，一些配置是topology级别的。default.yaml里面有所有的默认配置。你可以通过定义个storm.yaml在你的classpath里来覆盖这些默认配置。并且你也可以在代码里面设置一些topology相关的配置信息（使用StormSubmitter）。

2.2 构建Topology

1. 实现的目标：

我们将设计一个topology，来实现对一个句子里面的单词出现的频率进行统计。这是一个简单的例子，目的是让大家对于topology快速上手，有一个初步的理解。

2. 设计Topology结构：

在开始开发Storm项目的第一步，就是要设计topology。确定好你的数据处理逻辑，我们今天将的这个简单的例子，topology也非常简单。整个topology如下：

整个topology分为三个部分：

Input(Spout):数据源，负责发送sentence

Splitsentence(Bolt):负责将sentence切分

Wordcount(Bolt):负责对单词的频率进行累加

3. 设计数据流

这个topology从Input读取句子,并把句子划分成单词,然后汇总每个单词出现的次数,一个tuple负责读取句子,每一个tuple分别对应计算每一个单词出现的次数,大概样子如下所示：

4. 代码实现：

maven的pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>study</groupId>
  <artifactId>dxz</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>dxz</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-core</artifactId>
      <version>0.9.3</version>
      <scope>provided</scope>
    </dependency>
  </dependencies>
</project>

java代码：

package com.stormdemo.bolt;

import java.util.HashMap;
import java.util.Map;
import java.util.Random;

import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.generated.AlreadyAliveException;
import backtype.storm.generated.InvalidTopologyException;
import backtype.storm.spout.SpoutOutputCollector;
import backtype.storm.task.OutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.IRichBolt;
import backtype.storm.topology.IRichSpout;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;
import backtype.storm.utils.Utils;


public class WordCount {

    public static class WordProduct implements IRichSpout {
        private static final long serialVersionUID = 1L;
        private SpoutOutputCollector collector;
        Random rand;
        
        /**
         * 这是第一个方法要执行的方法（只执行一次），常用于初始化的（如打开文件、打开连接等）
         * 里面接收了三个参数，第一个是创建Topology时的配置， 
         * 第二个是所有的Topology数据，第三个是用来把Spout的数据发射给bolt 
         */
        @Override
        public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
            System.out.println("in open()*****************");
            this.collector = collector;
            rand = new Random();
        }

        public void close() {
            // TODO Auto-generated method stub
            
        }

        public void activate() {
            // TODO Auto-generated method stub
            
        }

        public void deactivate() {
            // TODO Auto-generated method stub
            
        }

        /**
         * 这个方法会不断被调用，为了降低它对CPU的消耗，当任务完成时让它sleep一下 
         */
        @Override
        public void nextTuple() {
            Utils.sleep(10000);
            String[] sentences = new String[] {
                    "this is a test",
                    "hello world",
                    "I am fine",
                    "china duan"
            };
            String s = sentences[rand.nextInt(sentences.length)];
            
            System.out.println("in nextTuple():" + s);
            collector.emit(new Values(s));
        }

        @Override  
        public void ack(Object msgId) {  
            System.out.println("OK:" + msgId);  
        } 
        
        @Override  
        public void fail(Object msgId) {  
            System.out.println("FAIL:" + msgId);  
        }  

        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            declarer.declare(new Fields("sentence"));
        }

        public Map<String, Object> getComponentConfiguration() {
            // TODO Auto-generated method stub
            return null;
        }
        
    }
    public static class SplitSentence implements IRichBolt {
        private OutputCollector collector;
        
        /**
         * Topology执行完毕的清理工作，比如关闭连接、释放资源等操作都会写在这里 
         * 因为这只是个Demo，我们用它来打印我们的计数器
         */
        @Override
        public void cleanup() {
            System.out.println("cleanup()");
        }

        /**
         * 这是bolt中最重要的方法，每当接收到一个tuple时，此方法便被调用
         * 这个方法的作用就是把文本文件中的每一行切分成一个个单词，并把这些单词发射出去（给下一个bolt处理）
         */
        @Override
        public void execute(Tuple input) {
            String s = input.getString(0);
            String[] words = s.split(" ");
            for(String w : words) {
                w = w.trim();
                if(!w.isEmpty()) {
                    collector.emit(new Values(w));
                }
            }
            //确认成功处理一个tuple
            collector.ack(input);
        }

        public void prepare(Map arg0, TopologyContext arg1, OutputCollector arg2) {
            this.collector = arg2;
        }

        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            declarer.declare(new Fields("word"));
            
        }

        public Map<String, Object> getComponentConfiguration() {
            // TODO Auto-generated method stub
            return null;
        }
        
    }
    public static class WordCounter implements IRichBolt {
        private OutputCollector collector;
        Map<String,Integer> counts = new HashMap<String, Integer>();
        
        /**
         * Topology执行完毕的清理工作，比如关闭连接、释放资源等操作都会写在这里 
         * 因为这只是个Demo，我们用它来打印我们的计数器
         */
        @Override
        public void cleanup() {
            System.out.println("cleanup()");
        }

        /**
         * 这是bolt中最重要的方法，每当接收到一个tuple时，此方法便被调用
         * 这个方法的作用就是统计单词次数
         */
        @Override
        public void execute(Tuple input) {
            String w = input.getString(0);
            Integer count = counts.get(w);
            if(count==null) {
                count = 1;
            } else {
                count++;
            }
            counts.put(w, count);
            collector.emit(new Values("word",count));
            
            System.out.println(w + ":" + Integer.toString(count));
            //确认成功处理一个tuple 
            collector.ack(input);
        }

        public void prepare(Map arg0, TopologyContext arg1, OutputCollector arg2) {
            this.collector = arg2;
        }

        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            declarer.declare(new Fields("word","count"));
        }

        public Map<String, Object> getComponentConfiguration() {
            // TODO Auto-generated method stub
            return null;
        }
        
    }
    
    public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException {
        //定义一个Topology
        TopologyBuilder builder = new TopologyBuilder();
        
        //为topology设置spout(节点)
        builder.setSpout("input", new WordProduct(),1);
        
        //为topology设置bolt(节点)
        builder.setBolt("bolt_sentence", new SplitSentence(),1).shuffleGrouping("input");
        builder.setBolt("bolt_wordcounter", new WordCounter(),1).fieldsGrouping("bolt_sentence", new Fields("word"));
        
        //配置
        Config config = new Config();
        //TOPOLOGY_DEBUG(setDebug), 当它被设置成true的话， storm会记录下每个组件所发射的每条消息。这在本地环境调试topology很有用， 但是在线上这么做的话会影响性能的。
        config.setDebug(true); 
        //定义你希望集群分配多少个工作进程给你来执行这个topology. 
        config.setNumWorkers(2); 
        
        if(args!=null && args.length>0) {
            config.setNumWorkers(3);
            StormSubmitter.submitTopology(args[0], config, builder.createTopology());
        } else {
            //创建一个本地模式cluster
            LocalCluster cluster = new LocalCluster();
            System.out.println("start word count");
            cluster.submitTopology("word count", config, builder.createTopology());
            //运行60s
            Utils.sleep(60000);
            
            System.out.println("begin kill topology");
            cluster.killTopology("word count"); //终止topology任务
            cluster.shutdown(); 
        }
    }

}

结果：

5693 [main-EventThread] INFO  org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED
5701 [main] INFO  backtype.storm.daemon.supervisor - Starting supervisor with id 1a1baffd-69d0-42f9-89de-35a60496fbd1 at host SF0001107252A.sf.com
start word count
5768 [main] INFO  backtype.storm.daemon.nimbus - Received topology submission for word count with conf {"topology.max.task.parallelism" nil, "topology.acker.executors" nil, "topology.kryo.register" nil, "topology.kryo.decorators" (), "topology.name" "word count", "storm.id" "word count-1-1484476433", "topology.workers" 2, "topology.debug" true}
5805 [main] INFO  backtype.storm.daemon.nimbus - Activating word count: word count-1-1484476433
5927 [main] INFO  backtype.storm.scheduler.EvenScheduler - Available slots: (["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1027] ["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1028] ["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1029] ["1989826c-4520-4bbc-bb34-e09f5155ce91" 1024] ["1989826c-4520-4bbc-bb34-e09f5155ce91" 1025] ["1989826c-4520-4bbc-bb34-e09f5155ce91" 1026])
5951 [main] INFO  backtype.storm.daemon.nimbus - Setting new assignment for topology id word count-1-1484476433: #backtype.storm.daemon.common.Assignment{:master-code-dir "C:\\Users\\01107252\\AppData\\Local\\Temp\\12d6d452-2f70-4f4f-8451-e21c36371261\\nimbus\\stormdist\\word count-1-1484476433", :node->host {"1989826c-4520-4bbc-bb34-e09f5155ce91" "SF0001107252A.sf.com", "1a1baffd-69d0-42f9-89de-35a60496fbd1" "SF0001107252A.sf.com"}, :executor->node+port {[4 4] ["1989826c-4520-4bbc-bb34-e09f5155ce91" 1024], [3 3] ["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1027], [2 2] ["1989826c-4520-4bbc-bb34-e09f5155ce91" 1024], [1 1] ["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1027], [5 5] ["1a1baffd-69d0-42f9-89de-35a60496fbd1" 1027]}, :executor->start-time-secs {[5 5] 1484476433, [3 3] 1484476433, [1 1] 1484476433, [4 4] 1484476433, [2 2] 1484476433}}
6637 [Thread-3] INFO  backtype.storm.daemon.supervisor - Downloading code for storm id word count-1-1484476433 from C:\Users\01107252\AppData\Local\Temp\12d6d452-2f70-4f4f-8451-e21c36371261\nimbus\stormdist\word count-1-1484476433
6749 [Thread-5] INFO  backtype.storm.daemon.supervisor - Downloading code for storm id word count-1-1484476433 from C:\Users\01107252\AppData\Local\Temp\12d6d452-2f70-4f4f-8451-e21c36371261\nimbus\stormdist\word count-1-1484476433
7255 [Thread-5] INFO  backtype.storm.daemon.supervisor - Finished downloading code for storm id word count-1-1484476433 from C:\Users\01107252\AppData\Local\Temp\12d6d452-2f70-4f4f-8451-e21c36371261\nimbus\stormdist\word count-1-1484476433
7313 [Thread-3] INFO  backtype.storm.daemon.supervisor - Finished downloading code for storm id word count-1-1484476433 from C:\Users\01107252\AppData\Local\Temp\12d6d452-2f70-4f4f-8451-e21c36371261\nimbus\stormdist\word count-1-1484476433
7317 [Thread-6] INFO  backtype.storm.daemon.supervisor - Launching worker with assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "word count-1-1484476433", :executors ([3 3] [1 1] [5 5])} for this supervisor 1a1baffd-69d0-42f9-89de-35a60496fbd1 on port 1027 with id 7c37d53d-735c-43ee-8b1f-0ea0bf3ab564
7321 [Thread-6] INFO  backtype.storm.daemon.worker - Launching worker for word count-1-1484476433 on 1a1baffd-69d0-42f9-89de-35a60496fbd1:1027 with id 7c37d53d-735c-43ee-8b1f-0ea0bf3ab564 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\01107252\\AppData\\Local\\Temp\\ad8f635a-51d8-41a8-bd21-722220739f15", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1027 1028 1029), "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
7322 [Thread-6] INFO  backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
7322 [Thread-6] INFO  org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting
7323 [Thread-6] INFO  org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2000 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@51480654
7327 [Thread-6-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000. Will not attempt to authenticate using SASL (unknown error)
7329 [Thread-6-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Socket connection established to 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000, initiating session
7330 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /0:0:0:0:0:0:0:1:51109
7330 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /0:0:0:0:0:0:0:1:51109
7332 [SyncThread:0] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x159a1b0b919000a with negotiated timeout 20000 for client /0:0:0:0:0:0:0:1:51109
7332 [Thread-6-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000, sessionid = 0x159a1b0b919000a, negotiated timeout = 20000
7333 [Thread-6-EventThread] INFO  org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7333 [Thread-6-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
7335 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b919000a
7337 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /0:0:0:0:0:0:0:1:51109 which had sessionid 0x159a1b0b919000a
7337 [Thread-6] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b919000a closed
7337 [Thread-6-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
7338 [Thread-6] INFO  backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
7339 [Thread-6] INFO  org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting
7339 [Thread-6] INFO  org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@14f9cec1
7341 [Thread-6-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
7343 [Thread-6-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2000, initiating session
7343 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:51112
7344 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:51112
7346 [SyncThread:0] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x159a1b0b919000b with negotiated timeout 20000 for client /127.0.0.1:51112
7347 [Thread-6-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2000, sessionid = 0x159a1b0b919000b, negotiated timeout = 20000
7347 [Thread-6-EventThread] INFO  org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7374 [Thread-4] INFO  backtype.storm.daemon.supervisor - Launching worker with assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id "word count-1-1484476433", :executors ([4 4] [2 2])} for this supervisor 1989826c-4520-4bbc-bb34-e09f5155ce91 on port 1024 with id f5fc123f-fb15-4515-a3c5-d658e983f9e3
7375 [Thread-4] INFO  backtype.storm.daemon.worker - Launching worker for word count-1-1484476433 on 1989826c-4520-4bbc-bb34-e09f5155ce91:1024 with id f5fc123f-fb15-4515-a3c5-d658e983f9e3 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\01107252\\AppData\\Local\\Temp\\4a0fd3f4-bc1c-41b0-8319-2271783a8e8e", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.environment" nil, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
7375 [Thread-4] INFO  backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
7376 [Thread-4] INFO  org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting
7376 [Thread-4] INFO  org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2000 sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@65667fc0
7380 [Thread-4-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2000. Will not attempt to authenticate using SASL (unknown error)
7381 [Thread-4-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2000, initiating session
7381 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /127.0.0.1:51115
7381 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /127.0.0.1:51115
7383 [SyncThread:0] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x159a1b0b919000c with negotiated timeout 20000 for client /127.0.0.1:51115
7384 [Thread-4-SendThread(127.0.0.1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2000, sessionid = 0x159a1b0b919000c, negotiated timeout = 20000
7384 [Thread-4-EventThread] INFO  org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7384 [Thread-4-EventThread] INFO  backtype.storm.zookeeper - Zookeeper state update: :connected:none
7386 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b919000c
7388 [Thread-4] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b919000c closed
7388 [Thread-4-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
7389 [Thread-4] INFO  backtype.storm.utils.StormBoundedExponentialBackoffRetry - The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
7389 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:51115 which had sessionid 0x159a1b0b919000c
7390 [Thread-4] INFO  org.apache.storm.curator.framework.imps.CuratorFrameworkImpl - Starting
7390 [Thread-4] INFO  org.apache.storm.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2000/storm sessionTimeout=20000 watcher=org.apache.storm.curator.ConnectionState@6d895a70
7393 [Thread-4-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000. Will not attempt to authenticate using SASL (unknown error)
7394 [Thread-4-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Socket connection established to 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000, initiating session
7394 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /0:0:0:0:0:0:0:1:51118
7394 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Client attempting to establish new session at /0:0:0:0:0:0:0:1:51118
7396 [SyncThread:0] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - Established session 0x159a1b0b919000d with negotiated timeout 20000 for client /0:0:0:0:0:0:0:1:51118
7396 [Thread-4-SendThread(0:0:0:0:0:0:0:1:2000)] INFO  org.apache.storm.zookeeper.ClientCnxn - Session establishment complete on server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2000, sessionid = 0x159a1b0b919000d, negotiated timeout = 20000
7396 [Thread-4-EventThread] INFO  org.apache.storm.curator.framework.state.ConnectionStateManager - State change: CONNECTED
7401 [Thread-4] INFO  backtype.storm.daemon.worker - Reading Assignments.
7834 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor __acker:[2 2]
7845 [Thread-4] INFO  backtype.storm.daemon.task - Emitting: __acker __system ["startup"]
7845 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks __acker:[2 2]
7863 [Thread-4] INFO  backtype.storm.daemon.executor - Timeouts disabled for executor __acker:[2 2]
7863 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor __acker:[2 2]
7871 [Thread-8-__acker] INFO  backtype.storm.daemon.executor - Preparing bolt __acker:(2)
7879 [Thread-8-__acker] INFO  backtype.storm.daemon.executor - Prepared bolt __acker:(2)
7888 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor bolt_wordcounter:[4 4]
7889 [Thread-4] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter __system ["startup"]
7890 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks bolt_wordcounter:[4 4]
7897 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor bolt_wordcounter:[4 4]
7900 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Preparing bolt bolt_wordcounter:(4)
7901 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Prepared bolt bolt_wordcounter:(4)
7907 [Thread-4] INFO  backtype.storm.daemon.executor - Loading executor __system:[-1 -1]
7908 [Thread-4] INFO  backtype.storm.daemon.task - Emitting: __system __system ["startup"]
7909 [Thread-4] INFO  backtype.storm.daemon.executor - Loaded executor tasks __system:[-1 -1]
7911 [Thread-4] INFO  backtype.storm.daemon.executor - Finished loading executor __system:[-1 -1]
7911 [Thread-4] INFO  backtype.storm.daemon.worker - Launching receive-thread for 1989826c-4520-4bbc-bb34-e09f5155ce91:1024
7911 [Thread-12-__system] INFO  backtype.storm.daemon.executor - Preparing bolt __system:(-1)
7929 [Thread-4] INFO  backtype.storm.daemon.worker - Worker has topology config {"storm.id" "word count-1-1484476433", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\01107252\\AppData\\Local\\Temp\\4a0fd3f4-bc1c-41b0-8319-2271783a8e8e", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "word count", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1024 1025 1026), "topology.environment" nil, "topology.debug" true, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 2, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
7929 [Thread-4] INFO  backtype.storm.daemon.worker - Worker f5fc123f-fb15-4515-a3c5-d658e983f9e3 for storm word count-1-1484476433 on 1989826c-4520-4bbc-bb34-e09f5155ce91:1024 has finished loading
7930 [Thread-13-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Starting receive-thread: [stormId: word count-1-1484476433, port: 1024, thread-id: 0 ]
7937 [Thread-12-__system] INFO  backtype.storm.daemon.executor - Prepared bolt __system:(-1)
8355 [Thread-6] INFO  backtype.storm.daemon.worker - Reading Assignments.
8394 [Thread-6] INFO  backtype.storm.daemon.executor - Loading executor bolt_sentence:[3 3]
8395 [Thread-6] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence __system ["startup"]
8395 [Thread-6] INFO  backtype.storm.daemon.executor - Loaded executor tasks bolt_sentence:[3 3]
8398 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Preparing bolt bolt_sentence:(3)
8398 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Prepared bolt bolt_sentence:(3)
8398 [Thread-6] INFO  backtype.storm.daemon.executor - Finished loading executor bolt_sentence:[3 3]
8408 [Thread-6] INFO  backtype.storm.daemon.executor - Loading executor input:[5 5]
8410 [Thread-6] INFO  backtype.storm.daemon.task - Emitting: input __system ["startup"]
8410 [Thread-6] INFO  backtype.storm.daemon.executor - Loaded executor tasks input:[5 5]
8421 [Thread-18-input] INFO  backtype.storm.daemon.executor - Opening spout input:(5)
8422 [Thread-6] INFO  backtype.storm.daemon.executor - Finished loading executor input:[5 5]
in open()*****************
8541 [Thread-18-input] INFO  backtype.storm.daemon.executor - Opened spout input:(5)
8544 [Thread-18-input] INFO  backtype.storm.daemon.executor - Activating spout input:(5)
8589 [Thread-6] INFO  backtype.storm.daemon.executor - Loading executor __system:[-1 -1]
8590 [Thread-6] INFO  backtype.storm.daemon.task - Emitting: __system __system ["startup"]
8591 [Thread-6] INFO  backtype.storm.daemon.executor - Loaded executor tasks __system:[-1 -1]
8593 [Thread-6] INFO  backtype.storm.daemon.executor - Finished loading executor __system:[-1 -1]
8595 [Thread-20-__system] INFO  backtype.storm.daemon.executor - Preparing bolt __system:(-1)
8595 [Thread-20-__system] INFO  backtype.storm.daemon.executor - Prepared bolt __system:(-1)
8609 [Thread-6] INFO  backtype.storm.daemon.executor - Loading executor __acker:[1 1]
8611 [Thread-6] INFO  backtype.storm.daemon.task - Emitting: __acker __system ["startup"]
8611 [Thread-6] INFO  backtype.storm.daemon.executor - Loaded executor tasks __acker:[1 1]
8613 [Thread-6] INFO  backtype.storm.daemon.executor - Timeouts disabled for executor __acker:[1 1]
8613 [Thread-6] INFO  backtype.storm.daemon.executor - Finished loading executor __acker:[1 1]
8613 [Thread-6] INFO  backtype.storm.daemon.worker - Launching receive-thread for 1a1baffd-69d0-42f9-89de-35a60496fbd1:1027
8614 [Thread-6] INFO  backtype.storm.daemon.worker - Worker has topology config {"storm.id" "word count-1-1484476433", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 0, "topology.skip.missing.kryo.registrations" true, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 50, "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "C:\\Users\\01107252\\AppData\\Local\\Temp\\ad8f635a-51d8-41a8-bd21-722220739f15", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "storm.meta.serialization.delegate" "backtype.storm.serialization.DefaultSerializationDelegate", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "localhost", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2000, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["localhost"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "word count", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 300, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" (1027 1028 1029), "topology.environment" nil, "topology.debug" true, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 2, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "local", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144, "topology.classpath" nil}
8616 [Thread-6] INFO  backtype.storm.daemon.worker - Worker 7c37d53d-735c-43ee-8b1f-0ea0bf3ab564 for storm word count-1-1484476433 on 1a1baffd-69d0-42f9-89de-35a60496fbd1:1027 has finished loading
8615 [Thread-22-__acker] INFO  backtype.storm.daemon.executor - Preparing bolt __acker:(1)
8617 [Thread-22-__acker] INFO  backtype.storm.daemon.executor - Prepared bolt __acker:(1)
8618 [Thread-23-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Starting receive-thread: [stormId: word count-1-1484476433, port: 1027, thread-id: 0 ]
in nextTuple():china duan
18545 [Thread-18-input] INFO  backtype.storm.daemon.task - Emitting: input default [china duan]
18555 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Processing received message source: input:5, stream: default, id: {}, [china duan]
18555 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [china]
18560 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [duan]
18564 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [china]
18564 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
china:1
18565 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [duan]
18565 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
duan:1
in nextTuple():this is a test
28559 [Thread-18-input] INFO  backtype.storm.daemon.task - Emitting: input default [this is a test]
28559 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Processing received message source: input:5, stream: default, id: {}, [this is a test]
28560 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [this]
28560 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [is]
28560 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [a]
28560 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [test]
28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [this]
28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
this:1
28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [is]
28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
is:1
28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [a]
28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
a:1
28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [test]
28560 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
test:1
in nextTuple():I am fine
38561 [Thread-18-input] INFO  backtype.storm.daemon.task - Emitting: input default [I am fine]
38562 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Processing received message source: input:5, stream: default, id: {}, [I am fine]
38562 [Thread-18-input] INFO  backtype.storm.daemon.executor - Processing received message source: __system:-1, stream: __tick, id: {}, [30]
38562 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [I]
38562 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [am]
38562 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [fine]
38563 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [I]
38563 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
I:1
38563 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [am]
38563 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
am:1
38564 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [fine]
38564 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 1]
fine:1
in nextTuple():I am fine
48564 [Thread-18-input] INFO  backtype.storm.daemon.task - Emitting: input default [I am fine]
48565 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Processing received message source: input:5, stream: default, id: {}, [I am fine]
48565 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [I]
48565 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [am]
48566 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [I]
48566 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 2]
I:2
48566 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [fine]
48567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [am]
48567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 2]
am:2
48567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [fine]
48567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 2]
fine:2
in nextTuple():china duan
58566 [Thread-18-input] INFO  backtype.storm.daemon.task - Emitting: input default [china duan]
58566 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.executor - Processing received message source: input:5, stream: default, id: {}, [china duan]
58566 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [china]
58566 [Thread-16-bolt_sentence] INFO  backtype.storm.daemon.task - Emitting: bolt_sentence default [duan]
58567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [china]
58567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 2]
china:2
58567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.executor - Processing received message source: bolt_sentence:3, stream: default, id: {}, [duan]
58567 [Thread-10-bolt_wordcounter] INFO  backtype.storm.daemon.task - Emitting: bolt_wordcounter default [word, 2]
duan:2
begin kill topology
65992 [main] INFO  backtype.storm.daemon.nimbus - Delaying event :remove for 30 secs for word count-1-1484476433
65999 [main] INFO  backtype.storm.daemon.nimbus - Updated word count-1-1484476433 with status {:type :killed, :kill-time-secs 30}
65999 [main] INFO  backtype.storm.daemon.nimbus - Shutting down master
66002 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b9190001
66004 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b9190001 closed
66004 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
66005 [main] INFO  backtype.storm.daemon.nimbus - Shut down master
66012 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN  org.apache.storm.zookeeper.server.NIOServerCnxn - caught end of stream exception
org.apache.storm.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read additional data from client sessionid 0x159a1b0b9190001, likely client has closed socket
    at org.apache.storm.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) ~[storm-core-0.9.3.jar:0.9.3]
    at org.apache.storm.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) [storm-core-0.9.3.jar:0.9.3]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_31]
66012 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /0:0:0:0:0:0:0:1:51081 which had sessionid 0x159a1b0b9190001
66013 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b9190003
66014 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b9190003 closed
66014 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:51087 which had sessionid 0x159a1b0b9190003
66015 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
66016 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b9190005
66018 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b9190005 closed
66018 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] WARN  org.apache.storm.zookeeper.server.NIOServerCnxn - caught end of stream exception
org.apache.storm.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read additional data from client sessionid 0x159a1b0b9190005, likely client has closed socket
    at org.apache.storm.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) ~[storm-core-0.9.3.jar:0.9.3]
    at org.apache.storm.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) [storm-core-0.9.3.jar:0.9.3]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_31]
66019 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:51093 which had sessionid 0x159a1b0b9190005
66019 [main] INFO  backtype.storm.daemon.supervisor - Shutting down 1989826c-4520-4bbc-bb34-e09f5155ce91:f5fc123f-fb15-4515-a3c5-d658e983f9e3
66020 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
66020 [main] INFO  backtype.storm.process-simulator - Killing process a311407c-b041-4fc2-a42b-52a959bf5f15
66021 [main] INFO  backtype.storm.daemon.worker - Shutting down worker word count-1-1484476433 1989826c-4520-4bbc-bb34-e09f5155ce91 1024
66021 [main] INFO  backtype.storm.daemon.worker - Shutting down receive thread
66021 [main] INFO  backtype.storm.messaging.loader - Shutting down receiving-thread: [word count-1-1484476433, 1024]
66021 [main] INFO  backtype.storm.messaging.loader - Waiting for receiving-thread:[word count-1-1484476433, 1024] to die
66021 [Thread-13-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Receiving-thread:[word count-1-1484476433, 1024] received shutdown notice
66023 [main] INFO  backtype.storm.messaging.loader - Shutdown receiving-thread: [word count-1-1484476433, 1024]
66023 [main] INFO  backtype.storm.daemon.worker - Shut down receive thread
66023 [main] INFO  backtype.storm.daemon.worker - Terminating messaging context
66023 [main] INFO  backtype.storm.daemon.worker - Shutting down executors
66023 [main] INFO  backtype.storm.daemon.executor - Shutting down executor __acker:[2 2]
66023 [Thread-7-disruptor-executor[2 2]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
66023 [Thread-8-__acker] INFO  backtype.storm.util - Async loop interrupted!
66024 [main] INFO  backtype.storm.daemon.executor - Shut down executor __acker:[2 2]
66024 [main] INFO  backtype.storm.daemon.executor - Shutting down executor bolt_wordcounter:[4 4]
66024 [Thread-10-bolt_wordcounter] INFO  backtype.storm.util - Async loop interrupted!
66024 [Thread-9-disruptor-executor[4 4]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
cleanup()
66025 [main] INFO  backtype.storm.daemon.executor - Shut down executor bolt_wordcounter:[4 4]
66025 [main] INFO  backtype.storm.daemon.executor - Shutting down executor __system:[-1 -1]
66025 [Thread-12-__system] INFO  backtype.storm.util - Async loop interrupted!
66025 [Thread-11-disruptor-executor[-1 -1]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
66025 [main] INFO  backtype.storm.daemon.executor - Shut down executor __system:[-1 -1]
66025 [main] INFO  backtype.storm.daemon.worker - Shut down executors
66026 [main] INFO  backtype.storm.daemon.worker - Shutting down transfer thread
66026 [Thread-14-disruptor-worker-transfer-queue] INFO  backtype.storm.util - Async loop interrupted!
66026 [main] INFO  backtype.storm.daemon.worker - Shut down transfer thread
66026 [main] INFO  backtype.storm.daemon.worker - Shutting down default resources
66027 [main] INFO  backtype.storm.daemon.worker - Shut down default resources
66036 [main] INFO  backtype.storm.daemon.worker - Disconnecting from storm cluster state context
66037 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b919000d
66038 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /0:0:0:0:0:0:0:1:51118 which had sessionid 0x159a1b0b919000d
66039 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b919000d closed
66039 [main] INFO  backtype.storm.daemon.worker - Shut down worker word count-1-1484476433 1989826c-4520-4bbc-bb34-e09f5155ce91 1024
66039 [Thread-4-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
66316 [main] INFO  backtype.storm.daemon.supervisor - Shut down 1989826c-4520-4bbc-bb34-e09f5155ce91:f5fc123f-fb15-4515-a3c5-d658e983f9e3
66316 [main] INFO  backtype.storm.daemon.supervisor - Shutting down supervisor 1989826c-4520-4bbc-bb34-e09f5155ce91
66317 [Thread-3] INFO  backtype.storm.event - Event manager interrupted
66317 [Thread-4] INFO  backtype.storm.event - Event manager interrupted
66318 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b9190007
66319 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b9190007 closed
66319 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:51099 which had sessionid 0x159a1b0b9190007
66319 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
66321 [main] INFO  backtype.storm.daemon.supervisor - Shutting down 1a1baffd-69d0-42f9-89de-35a60496fbd1:7c37d53d-735c-43ee-8b1f-0ea0bf3ab564
66321 [main] INFO  backtype.storm.process-simulator - Killing process 36f44b82-647b-4812-8f60-70aa7aac4fa7
66321 [main] INFO  backtype.storm.daemon.worker - Shutting down worker word count-1-1484476433 1a1baffd-69d0-42f9-89de-35a60496fbd1 1027
66322 [main] INFO  backtype.storm.daemon.worker - Shutting down receive thread
66322 [main] INFO  backtype.storm.messaging.loader - Shutting down receiving-thread: [word count-1-1484476433, 1027]
66322 [Thread-23-worker-receiver-thread-0] INFO  backtype.storm.messaging.loader - Receiving-thread:[word count-1-1484476433, 1027] received shutdown notice
66322 [main] INFO  backtype.storm.messaging.loader - Waiting for receiving-thread:[word count-1-1484476433, 1027] to die
66322 [main] INFO  backtype.storm.messaging.loader - Shutdown receiving-thread: [word count-1-1484476433, 1027]
66322 [main] INFO  backtype.storm.daemon.worker - Shut down receive thread
66322 [main] INFO  backtype.storm.daemon.worker - Terminating messaging context
66322 [main] INFO  backtype.storm.daemon.worker - Shutting down executors
66323 [main] INFO  backtype.storm.daemon.executor - Shutting down executor bolt_sentence:[3 3]
66323 [Thread-16-bolt_sentence] INFO  backtype.storm.util - Async loop interrupted!
66323 [Thread-15-disruptor-executor[3 3]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
cleanup()
66323 [main] INFO  backtype.storm.daemon.executor - Shut down executor bolt_sentence:[3 3]
66323 [main] INFO  backtype.storm.daemon.executor - Shutting down executor input:[5 5]
66324 [Thread-17-disruptor-executor[5 5]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
66324 [Thread-18-input] INFO  backtype.storm.util - Async loop interrupted!
66324 [main] INFO  backtype.storm.daemon.executor - Shut down executor input:[5 5]
66324 [main] INFO  backtype.storm.daemon.executor - Shutting down executor __system:[-1 -1]
66324 [Thread-20-__system] INFO  backtype.storm.util - Async loop interrupted!
66324 [Thread-19-disruptor-executor[-1 -1]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
66324 [main] INFO  backtype.storm.daemon.executor - Shut down executor __system:[-1 -1]
66324 [main] INFO  backtype.storm.daemon.executor - Shutting down executor __acker:[1 1]
66325 [Thread-22-__acker] INFO  backtype.storm.util - Async loop interrupted!
66325 [Thread-21-disruptor-executor[1 1]-send-queue] INFO  backtype.storm.util - Async loop interrupted!
66325 [main] INFO  backtype.storm.daemon.executor - Shut down executor __acker:[1 1]
66325 [main] INFO  backtype.storm.daemon.worker - Shut down executors
66325 [main] INFO  backtype.storm.daemon.worker - Shutting down transfer thread
66325 [Thread-24-disruptor-worker-transfer-queue] INFO  backtype.storm.util - Async loop interrupted!
66325 [main] INFO  backtype.storm.daemon.worker - Shut down transfer thread
66326 [main] INFO  backtype.storm.daemon.worker - Shutting down default resources
66326 [main] INFO  backtype.storm.daemon.worker - Shut down default resources
66330 [main] INFO  backtype.storm.daemon.worker - Disconnecting from storm cluster state context
66330 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b919000b
66332 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b919000b closed
66332 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:51112 which had sessionid 0x159a1b0b919000b
66332 [main] INFO  backtype.storm.daemon.worker - Shut down worker word count-1-1484476433 1a1baffd-69d0-42f9-89de-35a60496fbd1 1027
66332 [Thread-6-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
66594 [main] INFO  backtype.storm.daemon.supervisor - Shut down 1a1baffd-69d0-42f9-89de-35a60496fbd1:7c37d53d-735c-43ee-8b1f-0ea0bf3ab564
66595 [main] INFO  backtype.storm.daemon.supervisor - Shutting down supervisor 1a1baffd-69d0-42f9-89de-35a60496fbd1
66595 [Thread-5] INFO  backtype.storm.event - Event manager interrupted
66595 [Thread-6] INFO  backtype.storm.event - Event manager interrupted
66596 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x159a1b0b9190009
66597 [main] INFO  org.apache.storm.zookeeper.ZooKeeper - Session: 0x159a1b0b9190009 closed
66598 [main-EventThread] INFO  org.apache.storm.zookeeper.ClientCnxn - EventThread shut down
66598 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxn - Closed socket connection for client /0:0:0:0:0:0:0:1:51105 which had sessionid 0x159a1b0b9190009
66598 [main] INFO  backtype.storm.testing - Shutting down in process zookeeper
66598 [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2000] INFO  org.apache.storm.zookeeper.server.NIOServerCnxnFactory - NIOServerCnxn factory exited run method
66598 [main] INFO  org.apache.storm.zookeeper.server.ZooKeeperServer - shutting down
66598 [main] INFO  org.apache.storm.zookeeper.server.SessionTrackerImpl - Shutting down
66598 [main] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - Shutting down
66599 [ProcessThread(sid:0 cport:-1):] INFO  org.apache.storm.zookeeper.server.PrepRequestProcessor - PrepRequestProcessor exited loop!
66599 [main] INFO  org.apache.storm.zookeeper.server.SyncRequestProcessor - Shutting down
66599 [SyncThread:0] INFO  org.apache.storm.zookeeper.server.SyncRequestProcessor - SyncRequestProcessor exited!
66599 [main] INFO  org.apache.storm.zookeeper.server.FinalRequestProcessor - shutdown of request processor complete
66599 [main] INFO  backtype.storm.testing - Done shutting down in process zookeeper
66599 [main] INFO  backtype.storm.testing - Deleting temporary path C:\Users\01107252\AppData\Local\Temp\12d6d452-2f70-4f4f-8451-e21c36371261
66614 [main] INFO  backtype.storm.testing - Deleting temporary path C:\Users\01107252\AppData\Local\Temp\00f65db7-e527-4998-b253-110ce0e5ab71
66618 [main] INFO  backtype.storm.testing - Unable to delete file: C:\Users\01107252\AppData\Local\Temp\00f65db7-e527-4998-b253-110ce0e5ab71\version-2\log.1
66618 [main] INFO  backtype.storm.testing - Deleting temporary path C:\Users\01107252\AppData\Local\Temp\4a0fd3f4-bc1c-41b0-8319-2271783a8e8e
66660 [main] INFO  backtype.storm.testing - Deleting temporary path C:\Users\01107252\AppData\Local\Temp\ad8f635a-51d8-41a8-bd21-722220739f15
68397 [SessionTracker] INFO  org.apache.storm.zookeeper.server.SessionTrackerImpl - SessionTrackerImpl exited loop!

关键的方法是 execute方法。正如你可以看到，它将句子拆分成单词，并发出每个单词作为一个新的元组。另一个重要的方法是declareOutputFields，其中宣布bolts输出元组的架构。在这里宣布，它发出一个域为word的元组。

setBolt的最后一个参数是你想为bolts的并行量。 SplitSentence bolts 是10个并发，这将导致在storm集群中有十个线程并行执行。你所要做的的是增加bolts的并行量在遇到topology的瓶颈时。

setBolt方法返回一个对象，用来定义bolts的输入。例如，SplitSentence螺栓订阅组件“1”使用随机分组的输出流。 “1”是指已经定义KestrelSpout。我将解释在某一时刻的随机分组的一部分。到目前为止，最要紧的是，SplitSentence bolts会消耗KestrelSpout发出的每一个元组。

storm的运行有两种模式: 本地模式和分布式模式.

1) 本地模式：

storm用一个进程里面的线程来模拟所有的spout和bolt. 本地模式对开发和测试来说比较有用。你运行storm-starter里面的topology的时候它们就是以本地模式运行的，你可以看到topology里面的每一个组件在发射什么消息。

2) 分布式模式：

storm由一堆机器组成。当你提交topology给master的时候，你同时也把topology的代码提交了。master负责分发你的代码并且负责给你的topolgoy分配工作进程。如果一个工作进程挂掉了， master节点会把认为重新分配到其它节点。

首先，这个代码定义通过定义一个LocalCluster对象来定义一个进程内的集群。提交topology给这个虚拟的集群和提交topology给分布式集群是一样的。通过调用submitTopology方法来提交topology，它接受三个参数：要运行的topology的名字，一个配置对象以及要运行的topology本身。

topology的名字是用来唯一区别一个topology的，这样你然后可以用这个名字来杀死这个topology的。前面已经说过了，你必须显式的杀掉一个topology，否则它会一直运行。

Conf对象可以配置很多东西，下面两个是最常见的：

TOPOLOGY_WORKERS(setNumWorkers) 定义你希望集群分配多少个工作进程给你来执行这个topology. topology里面的每个组件会被需要线程来执行。每个组件到底用多少个线程是通过setBolt和setSpout来指定的。这些线程都运行在工作进程里面. 每一个工作进程包含一些节点的一些工作线程。比如，如果你指定300个线程，60个进程，那么每个工作进程里面要执行6个线程，而这6个线程可能属于不同的组件(Spout, Bolt)。你可以通过调整每个组件的并行度以及这些线程所在的进程数量来调整topology的性能。

TOPOLOGY_DEBUG(setDebug), 当它被设置成true的话， storm会记录下每个组件所发射的每条消息。这在本地环境调试topology很有用，但是在线上这么做的话会影响性能的。

结论：

本章从storm的基本对象的定义，到广泛的介绍了storm的开发环境，从一个简单的例子讲解了topology的构建和定义。希望大家可以从本章的内容对storm有一个基本的理解和概念，并且已经可以构建一个简单的topology！！

posted on 2015-08-03 22:38 duanxz 阅读(828) 评论(0) 编辑收藏举报