201707172345复习-storm篇
一、
0、storm介绍
实时，分布式，流式，系统
低延迟，高性能，无数据丢失，有状态的
1、storm下载，安装，配置，启动
以下是storm/conf/storm.yaml文件内容
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
- "s129"
- "s130"
- "s131"
# nimbus的地址
nimbus.seeds: ["s129"]
# storm 存储一些少量状态的目录：如jars，confs
storm.local.dir: "/home/ubuntu/storm"
# supervisor启动的工作进程Worker的占用端口号：4个端口号表示有4个worker，
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
#
# ##### These may optionally be filled in:
#
## List of custom serializations
# topology.kryo.register:
# - org.mycompany.MyType
# - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
# - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
# - "server1"
# - "server2"
## Metrics Consumers
# topology.metrics.consumer.register:
# - class: "org.apache.storm.metric.LoggingMetricsConsumer"
# parallelism.hint: 1
# - class: "org.mycompany.MyMetricsConsumer"
# parallelism.hint: 1
# argument:
# - endpoint: "metrics-collector.mycompany.org"
启动命令storm nimbus ， storm supervisor ， storm ui
2、strom 命令行命令 http://storm.apache.org/releases/1.1.0/Command-line-client.html， 最长用的也就是jar了，其他的基本没啥用，在ui上都能够看得到。
3、storm 核心概念
storm任务：称为topology， topology由spout和bolt组成，一个spout或一个bolt称为一个task
作用：一个任务，描述了任务的相关信息，被nimbus解析成各种任务并发布。
spout：数据的源头，可以读取一个文件，可以是kafka的消费者
bolt：接受spout传来的数据，并做处理，处理完毕可以持久化到磁盘上，也可以继续发送给下一个bolt
tuple：数据传输的最小单元，以逗号分割的多个值
4、storm架构：主从架构
主：nimbus，
作用：分配任务到zk上，上传配置信息，和jar包到zk上
从：supervisor， 在从节点上启动java进程Supervisor
作用：监听zk上的节点，随时从zk上接受任务，启动相应的进程来处理任务
运行时，supervisor的节点启动Worker进程来运行task，可以为spout和bolt设置并行度，即有几个线程来运行同时运行spout和bolt，一个task就是spout或bolt的其中一个线程 ， Worker进程是一个java进程，由supervisor进程启动。
作用：一个worker进程可以启动多个线程来运行这个task，每个线程称为Executor
--------------心跳信息-------------
nimbus和supervisor每个一定时间向zk发送一次心跳；supervisor和nimbus之间并没有直接交互
supervisor启动的worker也隔一定时间向zk发送心跳
Worker之间通过netty传递数据
5、nimbus启动过程解析
这是启动strom nimbus的过程：就做了一件事儿:和zk建立连接 2017-07-18 19:08:02.075 main o.a.s.d.nimbus [INFO] Starting Nimbus with conf {"topology.builtin.metrics.bucket.size.secs" 60, "nimbus.childopts" "-Xmx1024m", "ui.filter.params" nil, "storm.cluster.mode" "distributed", "storm.messaging.netty.client_worker_threads" 1, "logviewer.max.per.worker.logs.size.mb" 2048, "supervisor.run.worker.as.user" false, "topology.max.task.parallelism" nil, "topology.priority" 29, "zmq.threads" 1, "storm.group.mapping.service" "org.apache.storm.security.auth.ShellBasedGroupsMapping", "transactional.zookeeper.root" "/transactional", "topology.sleep.spout.wait.strategy.time.ms" 1, "scheduler.display.resource" false, "topology.max.replication.wait.time.sec" 60, "drpc.invocations.port" 3773, "supervisor.localizer.cache.target.size.mb" 10240, "topology.multilang.serializer" "org.apache.storm.multilang.JsonSerializer", "storm.messaging.netty.server_worker_threads" 1, "nimbus.blobstore.class" "org.apache.storm.blobstore.LocalFsBlobStore", "resource.aware.scheduler.eviction.strategy" "org.apache.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy", "topology.max.error.report.per.interval" 5, "storm.thrift.transport" "org.apache.storm.security.auth.SimpleTransportPlugin", "zmq.hwm" 0, "storm.group.mapping.service.params" nil, "worker.profiler.enabled" false, "storm.principal.tolocal" "org.apache.storm.security.auth.DefaultPrincipalToLocal", "supervisor.worker.shutdown.sleep.secs" 3, "pacemaker.host" "localhost", "storm.zookeeper.retry.times" 5, "ui.actions.enabled" true, "zmq.linger.millis" 5000, "supervisor.enable" true, "topology.stats.sample.rate" 0.05, "storm.messaging.netty.min_wait_ms" 100, "worker.log.level.reset.poll.secs" 30, "storm.zookeeper.port" 2181, "supervisor.heartbeat.frequency.secs" 5, "topology.enable.message.timeouts" true, "supervisor.cpu.capacity" 400.0, "drpc.worker.threads" 64, "supervisor.blobstore.download.thread.count" 5, "task.backpressure.poll.secs" 30, "drpc.queue.size" 128, "topology.backpressure.enable" false, "supervisor.blobstore.class" "org.apache.storm.blobstore.NimbusBlobStore", "storm.blobstore.inputstream.buffer.size.bytes" 65536, "topology.shellbolt.max.pending" 100, "drpc.https.keystore.password" "", "nimbus.code.sync.freq.secs" 120, "logviewer.port" 8000, "topology.scheduler.strategy" "org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy", "topology.executor.send.buffer.size" 1024, "resource.aware.scheduler.priority.strategy" "org.apache.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy", "pacemaker.auth.method" "NONE", "storm.daemon.metrics.reporter.plugins" ["org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter"], "topology.worker.logwriter.childopts" "-Xmx64m", "topology.spout.wait.strategy" "org.apache.storm.spout.SleepSpoutWaitStrategy", "ui.host" "0.0.0.0", "storm.nimbus.retry.interval.millis" 2000, "nimbus.inbox.jar.expiration.secs" 3600, "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.acker.executors" nil, "topology.fall.back.on.java.serialization" true, "topology.eventlogger.executors" 0, "supervisor.localizer.cleanup.interval.ms" 600000, "storm.zookeeper.servers" ["s129" "s130" "s131"], "nimbus.thrift.threads" 64, "logviewer.cleanup.age.mins" 10080, "topology.worker.childopts" nil, "topology.classpath" nil, "supervisor.monitor.frequency.secs" 3, "nimbus.credential.renewers.freq.secs" 600, "topology.skip.missing.kryo.registrations" false, "drpc.authorizer.acl.filename" "drpc-auth-acl.yaml", "pacemaker.kerberos.users" [], "storm.group.mapping.service.cache.duration.secs" 120, "topology.testing.always.try.serialize" false, "nimbus.monitor.freq.secs" 10, "storm.health.check.timeout.ms" 5000, "supervisor.supervisors" [], "topology.tasks" nil, "topology.bolts.outgoing.overflow.buffer.enable" false, "storm.messaging.netty.socket.backlog" 500, "topology.workers" 1, "pacemaker.base.threads" 10, "storm.local.dir" "/home/ubuntu/storm", "worker.childopts" "-Xmx%HEAP-MEM%m -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump", "storm.auth.simple-white-list.users" [], "topology.disruptor.batch.timeout.millis" 1, "topology.message.timeout.secs" 30, "topology.state.synchronization.timeout.secs" 60, "topology.tuple.serializer" "org.apache.storm.serialization.types.ListDelegateSerializer", "supervisor.supervisors.commands" [], "nimbus.blobstore.expiration.secs" 600, "logviewer.childopts" "-Xmx128m", "topology.environment" nil, "topology.debug" false, "topology.disruptor.batch.size" 100, "storm.messaging.netty.max_retries" 300, "ui.childopts" "-Xmx768m", "storm.network.topography.plugin" "org.apache.storm.networktopography.DefaultRackDNSToSwitchMapping", "storm.zookeeper.session.timeout" 20000, "drpc.childopts" "-Xmx768m", "drpc.http.creds.plugin" "org.apache.storm.security.auth.DefaultHttpCredentialsPlugin", "storm.zookeeper.connection.timeout" 15000, "storm.zookeeper.auth.user" nil, "storm.meta.serialization.delegate" "org.apache.storm.serialization.GzipThriftSerializationDelegate", "topology.max.spout.pending" nil, "storm.codedistributor.class" "org.apache.storm.codedistributor.LocalFileSystemCodeDistributor", "nimbus.supervisor.timeout.secs" 60, "nimbus.task.timeout.secs" 30, "drpc.port" 3772, "pacemaker.max.threads" 50, "storm.zookeeper.retry.intervalceiling.millis" 30000, "nimbus.thrift.port" 6627, "storm.auth.simple-acl.admins" [], "topology.component.cpu.pcore.percent" 10.0, "supervisor.memory.capacity.mb" 3072.0, "storm.nimbus.retry.times" 5, "supervisor.worker.start.timeout.secs" 120, "storm.zookeeper.retry.interval" 1000, "logs.users" nil, "worker.profiler.command" "flight.bash", "transactional.zookeeper.port" nil, "drpc.max_buffer_size" 1048576, "pacemaker.thread.timeout" 10, "task.credentials.poll.secs" 30, "drpc.https.keystore.type" "JKS", "topology.worker.receiver.thread.count" 1, "topology.state.checkpoint.interval.ms" 1000, "supervisor.slots.ports" [6700 6701 6702 6703], "topology.transfer.buffer.size" 1024, "storm.health.check.dir" "healthchecks", "topology.worker.shared.thread.pool.size" 4, "drpc.authorizer.acl.strict" false, "nimbus.file.copy.expiration.secs" 600, "worker.profiler.childopts" "-XX:+UnlockCommercialFeatures -XX:+FlightRecorder", "topology.executor.receive.buffer.size" 1024, "backpressure.disruptor.low.watermark" 0.4, "nimbus.task.launch.secs" 120, "storm.local.mode.zmq" false, "storm.messaging.netty.buffer_size" 5242880, "storm.cluster.state.store" "org.apache.storm.cluster_state.zookeeper_state_factory", "worker.heartbeat.frequency.secs" 1, "storm.log4j2.conf.dir" "log4j2", "ui.http.creds.plugin" "org.apache.storm.security.auth.DefaultHttpCredentialsPlugin", "storm.zookeeper.root" "/storm", "topology.tick.tuple.freq.secs" nil, "drpc.https.port" -1, "storm.workers.artifacts.dir" "workers-artifacts", "supervisor.blobstore.download.max_retries" 3, "task.refresh.poll.secs" 10, "storm.exhibitor.port" 8080, "task.heartbeat.frequency.secs" 3, "pacemaker.port" 6699, "storm.messaging.netty.max_wait_ms" 1000, "topology.component.resources.offheap.memory.mb" 0.0, "drpc.http.port" 3774, "topology.error.throttle.interval.secs" 10, "storm.messaging.transport" "org.apache.storm.messaging.netty.Context", "topology.disable.loadaware.messaging" false, "storm.messaging.netty.authentication" false, "topology.component.resources.onheap.memory.mb" 128.0, "topology.kryo.factory" "org.apache.storm.serialization.DefaultKryoFactory", "worker.gc.childopts" "", "nimbus.topology.validator" "org.apache.storm.nimbus.DefaultTopologyValidator", "nimbus.seeds" ["s129"], "nimbus.queue.size" 100000, "nimbus.cleanup.inbox.freq.secs" 600, "storm.blobstore.replication.factor" 3, "worker.heap.memory.mb" 768, "logviewer.max.sum.worker.logs.size.mb" 4096, "pacemaker.childopts" "-Xmx1024m", "ui.users" nil, "transactional.zookeeper.servers" nil, "supervisor.worker.timeout.secs" 30, "storm.zookeeper.auth.password" nil, "storm.blobstore.acl.validation.enabled" false, "client.blobstore.class" "org.apache.storm.blobstore.NimbusBlobStore", "supervisor.childopts" "-Xmx256m", "topology.worker.max.heap.size.mb" 768.0, "ui.http.x-frame-options" "DENY", "backpressure.disruptor.high.watermark" 0.9, "ui.filter" nil, "ui.header.buffer.bytes" 4096, "topology.min.replication.count" 1, "topology.disruptor.wait.timeout.millis" 1000, "storm.nimbus.retry.intervalceiling.millis" 60000, "topology.trident.batch.emit.interval.millis" 500, "storm.auth.simple-acl.users" [], "drpc.invocations.threads" 64, "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "ui.port" 8080, "storm.exhibitor.poll.uripath" "/exhibitor/v1/cluster/list", "storm.messaging.netty.transfer.batch.size" 262144, "logviewer.appender.name" "A1", "nimbus.thrift.max_buffer_size" 1048576, "storm.auth.simple-acl.users.commands" [], "drpc.request.timeout.secs" 600} 2017-07-18 19:08:02.083 main o.a.s.n.NimbusInfo [INFO] Nimbus figures out its name to s129 2017-07-18 19:08:02.606 main o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2017-07-18 19:08:02.642 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2017-07-18 19:08:02.645 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:host.name=s129 2017-07-18 19:08:02.645 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.version=1.8.0_131 2017-07-18 19:08:02.645 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation 2017-07-18 19:08:02.645 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.home=/soft/jdk1.8.0_131/jre 2017-07-18 19:08:02.645 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/soft/apache-storm-1.0.3/lib/storm-rename-hack-1.0.3.jar:/soft/apache-storm-1.0.3/lib/log4j-over-slf4j-1.6.6.jar:/soft/apache-storm-1.0.3/lib/servlet-api-2.5.jar:/soft/apache-storm-1.0.3/lib/clojure-1.7.0.jar:/soft/apache-storm-1.0.3/lib/reflectasm-1.10.1.jar:/soft/apache-storm-1.0.3/lib/log4j-core-2.1.jar:/soft/apache-storm-1.0.3/lib/asm-5.0.3.jar:/soft/apache-storm-1.0.3/lib/minlog-1.3.0.jar:/soft/apache-storm-1.0.3/lib/log4j-slf4j-impl-2.1.jar:/soft/apache-storm-1.0.3/lib/objenesis-2.1.jar:/soft/apache-storm-1.0.3/lib/kryo-3.0.3.jar:/soft/apache-storm-1.0.3/lib/slf4j-api-1.7.7.jar:/soft/apache-storm-1.0.3/lib/log4j-api-2.1.jar:/soft/apache-storm-1.0.3/lib/disruptor-3.3.2.jar:/soft/apache-storm-1.0.3/lib/storm-core-1.0.3.jar:/soft/storm/conf 2017-07-18 19:08:02.645 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib 2017-07-18 19:08:02.645 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp 2017-07-18 19:08:02.645 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA> 2017-07-18 19:08:02.646 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux 2017-07-18 19:08:02.646 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64 2017-07-18 19:08:02.646 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-117-generic 2017-07-18 19:08:02.646 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.name=ubuntu 2017-07-18 19:08:02.646 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.home=/home/ubuntu 2017-07-18 19:08:02.650 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.dir=/home/ubuntu/storm 2017-07-18 19:08:02.652 main o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=s129:2181,s130:2181,s131:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@13275d8 2017-07-18 19:08:02.791 main-SendThread(s129:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server s129/192.168.40.129:2181. Will not attempt to authenticate using SASL (unknown error) 2017-07-18 19:08:02.817 main o.a.s.b.FileBlobStoreImpl [INFO] Creating new blob store based in /home/ubuntu/storm/blobs 2017-07-18 19:08:02.942 main o.a.s.d.nimbus [INFO] Using default scheduler 2017-07-18 19:08:02.956 main o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2017-07-18 19:08:02.978 main o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=s129:2181,s130:2181,s131:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@77ec6a3d 2017-07-18 19:08:02.998 main o.a.s.n.NimbusInfo [INFO] Nimbus figures out its name to s129 2017-07-18 19:08:03.015 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server s131/192.168.40.131:2181. Will not attempt to authenticate using SASL (unknown error) 2017-07-18 19:08:03.105 main o.a.s.n.NimbusInfo [INFO] Nimbus figures out its name to s129 2017-07-18 19:08:03.256 main o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2017-07-18 19:08:03.261 main o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=s129:2181,s130:2181,s131:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@2f00f851 2017-07-18 19:08:03.289 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server s131/192.168.40.131:2181. Will not attempt to authenticate using SASL (unknown error) 2017-07-18 19:08:03.730 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to s131/192.168.40.131:2181, initiating session 2017-07-18 19:08:03.730 main-SendThread(s129:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to s129/192.168.40.129:2181, initiating session 2017-07-18 19:08:03.733 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to s131/192.168.40.131:2181, initiating session 2017-07-18 19:08:03.879 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server s131/192.168.40.131:2181, sessionid = 0x35d555c1d7c0000, negotiated timeout = 20000 2017-07-18 19:08:03.881 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server s131/192.168.40.131:2181, sessionid = 0x35d555c1d7c0001, negotiated timeout = 20000 2017-07-18 19:08:03.885 main-SendThread(s129:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server s129/192.168.40.129:2181, sessionid = 0x15d555c0c470000, negotiated timeout = 20000 2017-07-18 19:08:03.902 main-EventThread o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2017-07-18 19:08:03.902 main-EventThread o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2017-07-18 19:08:03.906 main-EventThread o.a.s.zookeeper [INFO] Zookeeper state update: :connected:none 2017-07-18 19:08:03.910 main-EventThread o.a.s.zookeeper [INFO] Zookeeper state update: :connected:none 2017-07-18 19:08:03.910 main-EventThread o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2017-07-18 19:08:03.971 Curator-Framework-0 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting 2017-07-18 19:08:03.988 main o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x35d555c1d7c0000 closed 2017-07-18 19:08:03.992 main-EventThread o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down 2017-07-18 19:08:04.003 main o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2017-07-18 19:08:04.010 main o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=s129:2181,s130:2181,s131:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@312b34e3 2017-07-18 19:08:04.031 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server s131/192.168.40.131:2181. Will not attempt to authenticate using SASL (unknown error) 2017-07-18 19:08:04.033 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to s131/192.168.40.131:2181, initiating session 2017-07-18 19:08:04.040 main o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2017-07-18 19:08:04.042 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server s131/192.168.40.131:2181, sessionid = 0x35d555c1d7c0002, negotiated timeout = 20000 2017-07-18 19:08:04.044 main-EventThread o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2017-07-18 19:08:04.045 main o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=s129:2181,s130:2181,s131:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@21079a12 2017-07-18 19:08:04.057 main-SendThread(s129:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server s129/192.168.40.129:2181. Will not attempt to authenticate using SASL (unknown error) 2017-07-18 19:08:04.059 main-SendThread(s129:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to s129/192.168.40.129:2181, initiating session 2017-07-18 19:08:04.093 main-SendThread(s129:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server s129/192.168.40.129:2181, sessionid = 0x15d555c0c470001, negotiated timeout = 20000 2017-07-18 19:08:04.094 main-EventThread o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2017-07-18 19:08:04.443 main o.a.s.zookeeper [INFO] Queued up for leader lock. 2017-07-18 19:08:04.640 main-EventThread o.a.s.zookeeper [INFO] s129 gained leadership, checking if it has all the topology code locally. 2017-07-18 19:08:04.723 main o.a.s.d.m.MetricsUtils [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter 2017-07-18 19:08:04.726 main-EventThread o.a.s.zookeeper [INFO] active-topology-ids [] local-topology-ids [] diff-topology [] 2017-07-18 19:08:04.728 main o.a.s.d.m.r.JmxPreparableReporter [INFO] Preparing... 2017-07-18 19:08:04.731 main-EventThread o.a.s.zookeeper [INFO] Accepting leadership, all active topology found localy. 2017-07-18 19:08:04.817 main o.a.s.d.common [INFO] Started statistics report plugin... 2017-07-18 19:08:04.992 main o.a.s.d.nimbus [INFO] Starting nimbus server for storm version '1.0.3'
6、supervisor启动过程解析
这是storm supervisor的启动过程，也只是和zk联系而已。 2017-07-18 19:08:11.570 main o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2017-07-18 19:08:11.625 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2017-07-18 19:08:11.625 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:host.name=s130 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.version=1.8.0_131 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.home=/soft/jdk1.8.0_131/jre 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/soft/apache-storm-1.0.3/lib/storm-rename-hack-1.0.3.jar:/soft/apache-storm-1.0.3/lib/log4j-over-slf4j-1.6.6.jar:/soft/apache-storm-1.0.3/lib/servlet-api-2.5.jar:/soft/apache-storm-1.0.3/lib/clojure-1.7.0.jar:/soft/apache-storm-1.0.3/lib/reflectasm-1.10.1.jar:/soft/apache-storm-1.0.3/lib/log4j-core-2.1.jar:/soft/apache-storm-1.0.3/lib/asm-5.0.3.jar:/soft/apache-storm-1.0.3/lib/minlog-1.3.0.jar:/soft/apache-storm-1.0.3/lib/log4j-slf4j-impl-2.1.jar:/soft/apache-storm-1.0.3/lib/objenesis-2.1.jar:/soft/apache-storm-1.0.3/lib/kryo-3.0.3.jar:/soft/apache-storm-1.0.3/lib/slf4j-api-1.7.7.jar:/soft/apache-storm-1.0.3/lib/log4j-api-2.1.jar:/soft/apache-storm-1.0.3/lib/disruptor-3.3.2.jar:/soft/apache-storm-1.0.3/lib/storm-core-1.0.3.jar:/soft/storm/conf 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA> 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.version=3.13.0-117-generic 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.name=ubuntu 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.home=/home/ubuntu 2017-07-18 19:08:11.626 main o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.dir=/home/ubuntu/storm 2017-07-18 19:08:11.627 main o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=s129:2181,s130:2181,s131:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@1f6d27cc 2017-07-18 19:08:11.781 main-SendThread(s130:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server s130/192.168.40.130:2181. Will not attempt to authenticate using SASL (unknown error) 2017-07-18 19:08:12.345 main-SendThread(s130:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to s130/192.168.40.130:2181, initiating session 2017-07-18 19:08:12.505 main-SendThread(s130:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server s130/192.168.40.130:2181, sessionid = 0x25d555c0c4f0000, negotiated timeout = 20000 2017-07-18 19:08:12.537 main-EventThread o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2017-07-18 19:08:12.591 Curator-Framework-0 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting 2017-07-18 19:08:12.628 main o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x25d555c0c4f0000 closed 2017-07-18 19:08:12.630 main-EventThread o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down 2017-07-18 19:08:12.634 main o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2017-07-18 19:08:12.640 main o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=s129:2181,s130:2181,s131:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@1cb37ee4 2017-07-18 19:08:12.662 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server s131/192.168.40.131:2181. Will not attempt to authenticate using SASL (unknown error) 2017-07-18 19:08:12.664 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to s131/192.168.40.131:2181, initiating session 2017-07-18 19:08:12.677 main-SendThread(s131:2181) o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server s131/192.168.40.131:2181, sessionid = 0x35d555c1d7c0003, negotiated timeout = 20000 2017-07-18 19:08:12.679 main-EventThread o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2017-07-18 19:08:12.741 main o.a.s.l.Localizer [INFO] Reconstruct localized resource: /home/ubuntu/storm/supervisor/usercache 2017-07-18 19:08:12.745 main o.a.s.l.Localizer [WARN] No left over resources found for any user during reconstructing of local resources at: /home/ubuntu/storm/supervisor/usercache 2017-07-18 19:08:12.812 main o.a.s.d.s.Supervisor [INFO] Starting supervisor for storm version '1.0.3'. 2017-07-18 19:08:12.815 main o.a.s.d.s.Supervisor [INFO] Starting Supervisor with conf {storm.messaging.netty.min_wait_ms=100, storm.zookeeper.auth.user=null, storm.messaging.netty.buffer_size=5242880, storm.exhibitor.port=8080, pacemaker.auth.method=NONE, ui.filter=null, worker.profiler.enabled=false, ui.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, topology.bolts.outgoing.overflow.buffer.enable=false, supervisor.supervisors.commands=[], logviewer.cleanup.age.mins=10080, topology.tuple.serializer=org.apache.storm.serialization.types.ListDelegateSerializer, drpc.port=3772, topology.max.spout.pending=null, topology.transfer.buffer.size=1024, logviewer.port=8000, worker.childopts=-Xmx%HEAP-MEM%m -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump, topology.component.cpu.pcore.percent=10.0, storm.daemon.metrics.reporter.plugins=[org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter], drpc.childopts=-Xmx768m, nimbus.task.launch.secs=120, logviewer.childopts=-Xmx128m, storm.zookeeper.servers=[s129, s130, s131], topology.disruptor.batch.timeout.millis=1, storm.messaging.transport=org.apache.storm.messaging.netty.Context, storm.messaging.netty.authentication=false, topology.kryo.factory=org.apache.storm.serialization.DefaultKryoFactory, worker.heap.memory.mb=768, storm.network.topography.plugin=org.apache.storm.networktopography.DefaultRackDNSToSwitchMapping, supervisor.slots.ports=[6700, 6701, 6702, 6703], resource.aware.scheduler.eviction.strategy=org.apache.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy, topology.stats.sample.rate=0.05, storm.local.dir=/home/ubuntu/storm, pacemaker.host=localhost, storm.messaging.netty.max_retries=300, topology.testing.always.try.serialize=false, storm.principal.tolocal=org.apache.storm.security.auth.DefaultPrincipalToLocal, java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib, worker.gc.childopts=, storm.group.mapping.service.cache.duration.secs=120, zmq.linger.millis=5000, topology.multilang.serializer=org.apache.storm.multilang.JsonSerializer, drpc.request.timeout.secs=600, zmq.threads=1, nimbus.blobstore.class=org.apache.storm.blobstore.LocalFsBlobStore, topology.state.synchronization.timeout.secs=60, topology.worker.shared.thread.pool.size=4, topology.executor.receive.buffer.size=1024, supervisor.monitor.frequency.secs=3, storm.nimbus.retry.times=5, transactional.zookeeper.port=null, storm.auth.simple-white-list.users=[], topology.scheduler.strategy=org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy, storm.zookeeper.port=2181, storm.zookeeper.retry.intervalceiling.millis=30000, storm.cluster.state.store=org.apache.storm.cluster_state.zookeeper_state_factory, nimbus.thrift.port=6627, nimbus.thrift.threads=64, supervisor.supervisors=[], nimbus.seeds=[s129], storm.auth.simple-acl.admins=[], topology.min.replication.count=1, ui.http.x-frame-options=DENY, nimbus.blobstore.expiration.secs=600, storm.group.mapping.service=org.apache.storm.security.auth.ShellBasedGroupsMapping, storm.nimbus.retry.interval.millis=2000, topology.max.task.parallelism=null, drpc.https.keystore.password=, supervisor.heartbeat.frequency.secs=5, nimbus.credential.renewers.freq.secs=600, storm.thrift.transport=org.apache.storm.security.auth.SimpleTransportPlugin, storm.auth.simple-acl.users=[], storm.zookeeper.auth.password=null, ui.port=8080, drpc.authorizer.acl.strict=false, topology.message.timeout.secs=30, topology.error.throttle.interval.secs=10, drpc.https.keystore.type=JKS, supervisor.memory.capacity.mb=3072.0, drpc.authorizer.acl.filename=drpc-auth-acl.yaml, topology.builtin.metrics.bucket.size.secs=60, storm.local.mode.zmq=false, ui.header.buffer.bytes=4096, topology.shellbolt.max.pending=100, drpc.max_buffer_size=1048576, storm.codedistributor.class=org.apache.storm.codedistributor.LocalFileSystemCodeDistributor, worker.profiler.childopts=-XX:+UnlockCommercialFeatures -XX:+FlightRecorder, nimbus.supervisor.timeout.secs=60, topology.worker.max.heap.size.mb=768.0, storm.zookeeper.root=/storm, topology.disable.loadaware.messaging=false, zmq.hwm=0, topology.sleep.spout.wait.strategy.time.ms=1, nimbus.topology.validator=org.apache.storm.nimbus.DefaultTopologyValidator, worker.heartbeat.frequency.secs=1, storm.messaging.netty.max_wait_ms=1000, topology.max.error.report.per.interval=5, nimbus.thrift.max_buffer_size=1048576, pacemaker.max.threads=50, supervisor.blobstore.download.max_retries=3, topology.enable.message.timeouts=true, storm.messaging.netty.transfer.batch.size=262144, supervisor.run.worker.as.user=false, storm.messaging.netty.client_worker_threads=1, topology.tasks=null, storm.group.mapping.service.params=null, drpc.http.port=3774, transactional.zookeeper.root=/transactional, supervisor.blobstore.download.thread.count=5, pacemaker.kerberos.users=[], topology.spout.wait.strategy=org.apache.storm.spout.SleepSpoutWaitStrategy, storm.blobstore.inputstream.buffer.size.bytes=65536, supervisor.worker.timeout.secs=30, topology.worker.receiver.thread.count=1, logviewer.max.sum.worker.logs.size.mb=4096, nimbus.file.copy.expiration.secs=600, pacemaker.port=6699, topology.worker.logwriter.childopts=-Xmx64m, drpc.http.creds.plugin=org.apache.storm.security.auth.DefaultHttpCredentialsPlugin, storm.blobstore.acl.validation.enabled=false, ui.filter.params=null, topology.workers=1, topology.environment=null, drpc.invocations.port=3773, topology.disruptor.batch.size=100, nimbus.cleanup.inbox.freq.secs=600, client.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, topology.fall.back.on.java.serialization=true, storm.nimbus.retry.intervalceiling.millis=60000, logviewer.appender.name=A1, ui.users=null, pacemaker.childopts=-Xmx1024m, storm.messaging.netty.server_worker_threads=1, scheduler.display.resource=false, storm.auth.simple-acl.users.commands=[], ui.actions.enabled=true, storm.zookeeper.connection.timeout=15000, topology.tick.tuple.freq.secs=null, nimbus.inbox.jar.expiration.secs=3600, topology.debug=false, storm.zookeeper.retry.interval=1000, worker.log.level.reset.poll.secs=30, storm.exhibitor.poll.uripath=/exhibitor/v1/cluster/list, storm.zookeeper.retry.times=5, nimbus.code.sync.freq.secs=120, topology.component.resources.offheap.memory.mb=0.0, topology.state.checkpoint.interval.ms=1000, topology.priority=29, supervisor.localizer.cleanup.interval.ms=600000, storm.health.check.dir=healthchecks, topology.disruptor.wait.timeout.millis=1000, supervisor.cpu.capacity=400.0, topology.classpath=null, backpressure.disruptor.high.watermark=0.9, supervisor.localizer.cache.target.size.mb=10240, topology.worker.childopts=null, drpc.https.port=-1, topology.max.replication.wait.time.sec=60, topology.acker.executors=null, supervisor.worker.start.timeout.secs=120, supervisor.worker.shutdown.sleep.secs=3, logviewer.max.per.worker.logs.size.mb=2048, topology.trident.batch.emit.interval.millis=500, task.heartbeat.frequency.secs=3, supervisor.enable=true, supervisor.blobstore.class=org.apache.storm.blobstore.NimbusBlobStore, topology.backpressure.enable=false, drpc.worker.threads=64, resource.aware.scheduler.priority.strategy=org.apache.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy, storm.messaging.netty.socket.backlog=500, nimbus.queue.size=100000, drpc.queue.size=128, topology.eventlogger.executors=0, pacemaker.base.threads=10, nimbus.childopts=-Xmx1024m, nimbus.monitor.freq.secs=10, topology.executor.send.buffer.size=1024, transactional.zookeeper.servers=null, nimbus.task.timeout.secs=30, logs.users=null, ui.host=0.0.0.0, pacemaker.thread.timeout=10, storm.meta.serialization.delegate=org.apache.storm.serialization.GzipThriftSerializationDelegate, dev.zookeeper.path=/tmp/dev-storm-zookeeper, backpressure.disruptor.low.watermark=0.4, topology.skip.missing.kryo.registrations=false, drpc.invocations.threads=64, storm.zookeeper.session.timeout=20000, storm.workers.artifacts.dir=workers-artifacts, topology.component.resources.onheap.memory.mb=128.0, storm.log4j2.conf.dir=log4j2, task.backpressure.poll.secs=30, storm.cluster.mode=distributed, ui.childopts=-Xmx768m, task.refresh.poll.secs=10, supervisor.childopts=-Xmx256m, task.credentials.poll.secs=30, storm.health.check.timeout.ms=5000, storm.blobstore.replication.factor=3, worker.profiler.command=flight.bash} 2017-07-18 19:08:13.011 main o.a.s.d.s.Slot [WARN] SLOT s130:6700 Starting in state EMPTY - assignment null 2017-07-18 19:08:13.013 main o.a.s.d.s.Slot [WARN] SLOT s130:6701 Starting in state EMPTY - assignment null 2017-07-18 19:08:13.020 main o.a.s.d.s.Slot [WARN] SLOT s130:6702 Starting in state EMPTY - assignment null 2017-07-18 19:08:13.021 main o.a.s.d.s.Slot [WARN] SLOT s130:6703 Starting in state EMPTY - assignment null # 清除不使用的topology， 2017-07-18 19:08:13.030 main o.a.s.l.AsyncLocalizer [INFO] Cleaning up unused topologies in /home/ubuntu/storm/supervisor/stormdist # 当前supervisor节点有个唯一id，保存在zk上 2017-07-18 19:08:13.054 main o.a.s.d.s.Supervisor [INFO] Starting supervisor with id 0515ec6e-0cdb-401e-be0f-c13a90e68daa at host s130. 2017-07-18 19:08:13.084 main o.a.s.d.m.MetricsUtils [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter 2017-07-18 19:08:13.090 main o.a.s.d.m.r.JmxPreparableReporter [INFO] Preparing... 2017-07-18 19:08:13.128 main o.a.s.m.StormMetricsRegistry [INFO] Started statistics report plugin...
8、storm api
package com.practice.storm2; import java.util.HashMap; import java.util.Map; import org.apache.storm.Config; import org.apache.storm.StormSubmitter; import org.apache.storm.generated.AlreadyAliveException; import org.apache.storm.generated.AuthorizationException; import org.apache.storm.generated.InvalidTopologyException; import org.apache.storm.generated.StormTopology; import org.apache.storm.topology.TopologyBuilder; public class MyFirstTopology { public static void main(String[] args) { // 1、做配置对象 Map<String, Object> conf = new HashMap<String, Object>(); conf.put(Config.TOPOLOGY_WORKERS, Integer.parseInt(args[0])); // 指定当前这个topology使用几个Worker进程 conf.put(Config.TOPOLOGY_DEBUG, true); conf.put(Config.STORM_ZOOKEEPER_SESSION_TIMEOUT, 4000000); // 2、生成builder对象,并且组装及配置各种组件 // 指定组件名，组件的类，组件的并行度，如果是bolt，还需要配置从哪个spout取数据，是怎么取数据 TopologyBuilder topologyBuilder = new TopologyBuilder(); topologyBuilder.setSpout("spout", new FakeCallLogReaderSpout(), Integer.parseInt(args[1])); // 并行度：同时又几个线程(Executor)来运行spout topologyBuilder.setBolt("bolt1", new CallLogCreatorBolt(), Integer.parseInt(args[2])).shuffleGrouping("spout"); // 指定这个bolt的多个task // 3、使用builder对象，生成topology对象 StormTopology topology = topologyBuilder.createTopology(); // 4、本地运行：所有的内容都在同一个jvm中运行：生成一个LocalCluster对象,提交这个topology,以及他的配置,以及这个topology的名称 /*LocalCluster cluster = new LocalCluster(); cluster.submitTopology("topolygyName", conf, topology); // 5、服务器去运行,且服务器运行结束前,客户端不能关闭 /*try { // 用于让当前进程睡眠10秒 Thread.sleep(20000); } catch (InterruptedException e) { e.printStackTrace(); }*/ // 6、服务器运行结束,关闭程序,即退出运行这个程序的jvm？ /*cluster.shutdown();*/ // 7、在集群上运行 try { StormSubmitter.submitTopology(args[3], conf , topology); } catch (AlreadyAliveException e) { e.printStackTrace(); } catch (InvalidTopologyException e) { e.printStackTrace(); } catch (AuthorizationException e) { e.printStackTrace(); } } }
package com.practice.storm2; import java.util.Map; import org.apache.storm.spout.SpoutOutputCollector; import org.apache.storm.task.TopologyContext; import org.apache.storm.topology.IRichSpout; import org.apache.storm.topology.OutputFieldsDeclarer; import org.apache.storm.tuple.Fields; import org.apache.storm.tuple.Values; public class FakeCallLogReaderSpout implements IRichSpout { /** * 由于IRichSpout实现了Serializable类,后续的子类也能够被序列化,序列化时使用到serialVersionUID * 因为spout有多个并发度，在不同的Worker进程内有spout的task，这些task就需要先反序列化spout，然后执行代码。 */ private static final long serialVersionUID = -7452365435481231737L; private int index = 0; // 声明在外头,当open方法被调用时,初始化这些值,以便在别的方法中也能使用 private TopologyContext conetxt; private SpoutOutputCollector collector; @SuppressWarnings("rawtypes") public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) { this.conetxt = context; this.collector = collector; } public void close() { try { Util.log(this, "这是spout的close方法被调用"); } catch (Exception e) { e.printStackTrace(); } } // 当spout被暂停掉，该方法被调用 public void deactivate() { try { Util.log(this, "这是spout的deactivate方法被调用：当spout被暂停掉，该方法被调用，在未来可以再继续使用该spout"); } catch (Exception e) { e.printStackTrace(); } } // 当spout被继续使用时调用该方法 public void activate() { try { Util.log(this, "这是spout的activate方法被调用：spout曾经被暂停过，现在又起来了"); } catch (Exception e) { e.printStackTrace(); } } /** * 模拟数据,发送10次,每次100条tuple 可以写个kafka的consumer,来消费kafka中指定topic的内容 */ public void nextTuple() { for (int i = 0; i < 10 && index < 10; i++) { try { Util.log(this, i + ""); } catch (Exception e) { e.printStackTrace(); } index ++; collector.emit(new Values(this.hashCode(), i)); } } // spout发出的数据到达bolt时，bolt会返回一个ack，ack方法被自动调用 public void ack(Object msgId) { try { Util.log(this, "这是spout的ack方法被调用:bolt已经成功接收标识为：" + msgId + "的消息"); } catch (Exception e) { e.printStackTrace(); } } // 当spout发出的数据没有达到bolt时，会自动调用这个方法。 public void fail(Object msgId) { try { // 记录日志：可以在做一个全局变量：Map<标识，消息>， 如果进入了ack，那么就从这个map中删除该标识，如果失败，该标识需要被重新发送 Util.log(this, "这是spout的fail方法被调用：说明spout发出的标识为:" + msgId + " 的消息没有被bolt成功接收"); Util.log(this, "可能原因：网络超时，或者bolt宕机"); } catch (Exception e) { e.printStackTrace(); } } // 申明输出的tuple中的各个值得字段名 public void declareOutputFields(OutputFieldsDeclarer declarer) { // 指定emit中的变量的别名 declarer.declare(new Fields("spoutHashcode", "value")); } public Map<String, Object> getComponentConfiguration() { return null; } }
package com.practice.storm2; import java.util.Map; import org.apache.storm.task.OutputCollector; import org.apache.storm.task.TopologyContext; import org.apache.storm.topology.IRichBolt; import org.apache.storm.topology.OutputFieldsDeclarer; import org.apache.storm.tuple.Tuple; public class CallLogCreatorBolt implements IRichBolt { private static final long serialVersionUID = 5747611056459882987L; private TopologyContext context; private OutputCollector collector; public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) { this.context = context; this.collector = collector; } // 接收到了spout中发来的tuple public void execute(Tuple input) { // 先用上游spout或者上游bolt中的declareOutputFields方法锁定义的字段名来取tuple中的内容 int spoutHashcode = input.getIntegerByField("spoutHashcode"); int value = input.getIntegerByField("value"); // 取到内容，做业务处理 try { Util.log(this, spoutHashcode + ","+value); } catch (Exception e) { e.printStackTrace(); } collector.ack(input); } public void cleanup() { try { Util.log(this, "这是bolt的cleanup方法被调用"); } catch (Exception e) { e.printStackTrace(); } } // 若果没有更下游的bolt了，那么这个可以不必写 public void declareOutputFields(OutputFieldsDeclarer declarer) { // TODO Auto-generated method stub } public Map<String, Object> getComponentConfiguration() { return null; } }
package com.practice.storm2; import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; import java.lang.management.ManagementFactory; import java.lang.management.RuntimeMXBean; import java.net.InetAddress; /** * 辅助类，用来查看spout和bolt中的数据 */ public class Util { /*public static void main(String[] args) throws Exception { Runtime runtime = Runtime.getRuntime(); Process exec = runtime.exec("nc localhost 8888"); OutputStream outputStream = exec.getOutputStream(); for (int i = 0; i < 100; i++) { outputStream.write(("hello" + i).getBytes()); } outputStream.close(); }*/ public static void log(Object o, String message) throws Exception { // spou发送:[主机名@ip, 进程名@进程号, 线程名@线程号, this.hashcode, value] // bolt收到:[主机名@ip, 进程名@进程号, 线程名@线程号, this.hashdoe, spoutHashcode, value] String info = ""; if(message.contains(",")){ info += "bolt收到:"; }else { info += "spou发送:"; } // 获取ip, String hostName = InetAddress.getLocalHost().getHostName(); String hostip = InetAddress.getLocalHost().getHostAddress(); // 获取进程 RuntimeMXBean runtimeMXBean = ManagementFactory.getRuntimeMXBean(); String processName = runtimeMXBean.getName(); // 获取线程 String tname = Thread.currentThread().getName(); long tid = Thread.currentThread().getId(); // 当前对象的hashcode int hashcode = o.hashCode(); info += "[" + hostName + "@" + hostip + " , " + processName + " , " + tname + "@" + tid + " , " + hashcode + "] " + message + "\n"; BufferedWriter bw = new BufferedWriter(new FileWriter(new File("/home/ubuntu/stormShuffle/text.txt"), true)); bw.write(info); bw.flush(); bw.close(); } }
9、spout原理，代码分析，继承
10、bolt 原理，代码分析，继承，最后如何存储数据
11、代码结构 http://storm.apache.org/releases/1.1.0/Structure-of-the-codebase.html
12、topology生命周期 http://storm.apache.org/releases/1.1.0/Lifecycle-of-a-topology.html
启动：
监视：
杀死：
13、消息传递 http://storm.apache.org/releases/1.1.0/Message-passing-implementation.html
14、storm的stream grouping
15、数据保证性 http://storm.apache.org/releases/1.1.0/Guaranteeing-message-processing.html
上图的总结：
1、storm 提供了几个不同级别的保证消息处理，包括：best effort， at least once， exactly once(只能通过Trident编程实现)。
2、从spout来了一个tuple，会触发产生上千个基于这个tuple的新的tuple：
例如单词统计：来了一个tuple，内容是一行sentence，切割每个单词后，又产生了更多的tuple
3、元组树：树的根节点是从spout发出来的第一个tuple，叶子节点是由这个tuple衍生出来的新的tuple
storm认为:当元组树上的所有的tuple被成功处理，称为 ：初始元组被“完全处理”。 这里的元组是指从spout中出来的最初元组。
这颗元组树，在30秒内没有处理完，则这个元组是失败的。超时时间可配置： Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS
4、当消息被完全处理或者没有被完全处理发生了什么？
首先，storm调用spout的nextTuple方法请求一个tuple，spout使用open方法里提供的SpoutOutPutCollector发射tuple到outputStream。
当发射tuple时，Spout提供一个"message id" ，这个消息id用来标识tuple。
然后，storm会跟踪这个tuple tree， 如果storm检测到这个tuple 是被完全处理的，storm 会在原始的spout task上调用ack method，并返回message id。
同样：如果tuple 没有被完全处理，超时了， 那么原始spout上的fail方法将会被调用。
spout是并行运行，多个spout task同时都在发射tuple，这个tuple是否被完全处理，ack和fail方法只会在这个tuple所处的task上调用，不会影响别的spout task
5、为了使用storm的可靠性能力，你需要做两件事儿：
1、当你在元组数中创建一个新的连接时，你应该告诉storm
2、当你完成一个tuple的处理时，你应该告诉storm
做以上两件事，storm可以检测到什么时候tuple是已经被完全处理了，并且能够适当地ack或者fail 那个spout task。
6、在tuple树中指定一个连接被称为"锚定"，锚定在你发射一个tuple时同时完成。
新的tuple被锚定，如果这个新的tuple在后续处理中失败了，那么原始tuple会被重新发送一次。
7、collector.emit(new Values(word)); 这种方式会导致新的tuple取消锚定，当新tuple失败时，不会重新发送原始tuple，
是否需要锚定，看你的需求，有时候不锚定也是很好的。
8、一个新的tuple也可锚定到多个原始tuple，因为多个原始tuple进来合并生成的新的tuple，当新的tuple失败时，原始的多个tuple都要重新发送。 这个在做join时很有用。
多锚定可能会破坏tuple tree 并且生成有向无环图(DAG:Directed Acyclic Graph, 任意一条边有方向，且不存在环路的图)
storm既可以处理tuple tree 也可以处理DAG (之前的释放版只能处理tuple tree， 而DAG就会卡住)，现在storm都能处理了，用户就不用管了
你也可以使用fail方法，明确地指定tuple是失败的，例如在spout中从数据库取数据，取数据失败，你用一个try catch捕获到这个exception，在catch中明确指定当前tuple是失败的，让spout重新去取一次。不要等到它后续的失败或者超时失败在重新取数据。
每一个tuple处理完，都需要被确认(acked)或者被失败(failed),strom使用内存跟踪每个tuple，如果你不这么干，storm会认为tuple一直失败，一直重新发送，终究会导致内存溢出
9、如果你implements IRichBolt，你就需要在execute末尾手动ack，但是如果你是继承BaseBasicBolt，那么bolt会在你处理完后自动给你ack
10、如果是多锚定，那么会有延迟ack，直到基于tuple 的一个分支上得到计算结果。
11、怎么样使我的应用程序能够重新发送tuple呢?
如果想要exactly once ，只能使用Trident api，
大多数情况下，大数据分析，丢掉一部分数据也是可以的，可以通过配置 Config.TOPOLOGY_ACKERS = 0 ,
如果想要at least once ，如果所有的操作时幂等的或者在之后能够发生deduping (此处不太懂)
12、storm 怎么有效地实现可靠性?
storm 有一个特殊的acker任务跟踪DAG。当一个DAG完成了，它发送一个消息给spout task，用户可以设置ack的值 Config.TOPOLOGY_ACKERS，
每一个tuple，不管是在spout还是bolt里，都有一个随机的64位id，这个id被acker线程用来跟踪每一个spout tuple
每一个tuple 都知道它所在的元组树上的所有的tuple id， 但你发射一个新的tuple到一个bolt里时，spout tuple ids 被锚定的新的tuple 拷贝一份，
当一个tuple是acked，tuple告诉一个合适的acker线程tuple tree状态已经改变。一般地，tuple 告诉acker ： "I am now completed within the tree for this spout tuple, and here are the new tuples in the tree that were anchored to me".
13、一些更详细的acker 跟踪tuple tree ：
acker 并不是随时都跟踪tuple ，对于一个有成千上万的节点的集群来说，追踪所有的tuple，会导致内存压力
有一个很好的策略是，acker每当spout发射出20字节的内容时才跟踪一次。以下是ack算法：
一个acker task 保存一个map,第一个值是spout task id， 第二各值是64位的ack val， ack val 代表整个tuple tree 的状态
当acker 看到ack val 变成0了，然后他就知道tupe tree 是处理完了
14、现在来看一下，storm是如何避免数据丢失的：
1、tuple 没有acked由于task died ： 这种情况下跟这个tuple在一条链上的spout tuple 会被重新发送一次
2、acker task died ： 这种请况下，所有的spuout tuple 都会超时，并被重新发送
3、spout task died ： 这种情况下，spout关联的资源重新发送数据
15、正如您所看到的，Storm的可靠性机制是完全分布式的、可伸缩的和容错的。
16、调节可靠性：
acker task 是轻量级的，你不需要太多，你可以通过storm ui来跟踪它的性能，如果吞吐量看起来有问题，你需要增加更多的acker tasks
如果可靠性对于你是不重要的，你不需要更关心丢失tuple的状态，你可以减少acker task 来提高性能，不跟踪tuple，网络间数据传输量将减少一半.
有3种方式去掉消息可靠性：
第一种：设置Config.TOPOLOGY_ACKERS = 0， 这种方式，strom在spout 发射tuple之前就调用了ack方法，tuple 将不被跟踪
第二种：针对没一个个消息移除可靠性，你可以关闭跟踪为一个单独的tuple，即SpoutOutputCollector.emit发射时，不要给定message id，
第三种：不要锚定
16、storm 高级封装1：Trident Trident is an alternative interface to Storm. It provides exactly-once processing, "transactional" datastore persistence, and a set of common stream analytics operations.
1、高级抽象
2、storm 无状态， Trident 有状态，可以做到exactly once
3、以小批次进行处理，不再使用bolt，而是使用filter ，aggregate，function，and states
4、Trident Tuple ： trident 的处理单元 ： 每个tuple 由预定义的字段列表构成，字段类型可以是bytes ， character， integer， long， float， double ， boolean， byte array
5、类结构：
Topology：TridentTopology tridentTopology = new TridentTopology();
Spout：
stream grouping：
bolt：（在Trident中没有bolt，但是有等价于bolt的：function， filter，aggregate）
17、storm 高级封装2：SQL The Storm SQL integration allows users to run SQL queries over streaming data in Storm. // TODO 后续使用到了继续补充
18、DRPC：Distrivuted RPC
对上面的总结：
1、将输入流作为函数的参数，函数调用作为输出流
2、DRPC 没有storm的那么些特征，例如stream，spout，bolt，topology
DRPC 已经打包在storm里
3、高级预览
DRPC由DRPC Server来协调：DRPC收到一个请求，将请求发送到storm的topology，然后从storm topology得到结果，并将结果返回到client
从client角度来看，DRPC就像一个常规的RPC调用:常规RPC远端只是一台机器，而DRPC远端是一个集群。
首先，client向server 发送函数名和参数，topology使用DRPCSpout从DRPC Server接受一个函数调用流。
在DRPC Server端会将(唯一id,函数调用)标记起来
然后，topology计算结果，并调用ReturnResults去连接DRPC Server，把结果给了相应的函数调用id。
DRPC Server将结果发送给调用该函数的那个client。
4、线性DRPCTopologyBuilder
storm有一个LinearDRPCTopologyBuilder类，几乎能够自动化地完成所有的DPRC 步骤：
1、Setting up the spout 设置spout
2、Returning the results to the DRPC server 返回结果给DRPC Server
3、Providing functionality to bolts for doing finite aggregations over groups of tuples 为bolt提供功能：在有限的tuple组上进行聚合
你只需要关心业务部分，LinearDRPCTopologyBuilder为你做了连接DRPC Server和发送返回结果的部分。
5、Local mode DRPC
可以运行在本地模式下
6、Remote mode DRPC
在一个真实的集群上使用只需要三步：
1.Launch DRPC server(s) 启动DRPC Server
命令：storm drpc
2.Configure the locations of the DRPC servers 在storm.yaml 中配置远程服务器地址
drpc.servers:
- "drpc1.foo.com"
- "drpc2.foo.com"
3.Submit DRPC topologies to Storm cluster 提交DRPC topology到Strom 集群
StormSubmitter.submitTopology("exclamation-drpc", conf, builder.createRemoteTopology());
7、A more complex example
上面的例子只是玩具，现在看一个更复杂的例子：计算访问Twitter url的次数
8、非线性DRPCTopologyBuilder
9、线性DRPCTopologyBuilder是怎么工作的
drpcspout发出[参数，返回的信息]。返回的信息是主机和服务器端的性能以及由该服务器生成的ID
拓扑结构包括：
drpcspout
PrepareRequest（生成一个请求ID和创建一个返回的信息并为args流）
coordinatedbolt包装并且直接分组
JoinResult（加入结果返回信息）
returnresult（连接到该服务器并返回结果）
10、LinearDRPCTopologyBuilder is a good example of a higher level abstraction built on top of Storm's primitives
LinearDRPCTopologyBuilder是一个更高层次的抽象之上的Storm的一个很好的例子
19、事务性topology
20、状态检查点
上面的总结：
0、状态检查点：
Storm core 对于bolt有一些抽象概念，用来保存和恢复bolt的操作。
有一个默认的基于内存的状态实现，还有一个基于redis的实现，用来提供状态的持久化
1、检查点管理
bolt 如果要求它的状态能够被管理并且被框架持久化，应该实现IStatefulBolt接口或者继承BaseStatefulBolt并且实现里面的initState(T State)方法。
initState被框架调用，在bolt被初始化时，初始化时和之前保存的状态一起被初始化。
这个方法在prepare方法之后，execute方法(处理tuple)之前处理
只支持一种仅有的状态，就是被KeyValueState提供的kv状态
wordcount bolt 的状态保存样例：
1.继承BaseStatefulBolt并且输入你想要保存的kv对的类型。
2.bolt在init方法中和之前的状态一起被初始化，这将会包含上次运行期间框架所提交的word counts(单词个数)
3.在execute方法中，更新单词个数
框架定期地检查状态，默认是每1秒钟检查一次。这个频率值可以通过topology.state.checkpoint.interval.ms来配置。
状态持久化，通过设置topology.state.provider来持久化状态，例如在storm.yaml中配置使用基于redis的kv状态实现：
topology.state.provider: org.apache.storm.redis.state.RedisKeyValueStateProvider， 注意：记得将jar包放在类路径下
这个提供状态持久化的类的属性可以使用topology.state.privider.config来配置。例如redis的配置：
{
"keyClass": "Optional fully qualified class name of the Key type.",
"valueClass": "Optional fully qualified class name of the Value type.",
"keySerializerClass": "Optional Key serializer implementation class.",
"valueSerializerClass": "Optional Value Serializer implementation class.",
"jedisPoolConfig": {
"host": "localhost",
"port": 6379,
"timeout": 2000,
"database": 0,
"password": "xyz"
}
}
2、检查点机制
检查点在一个指定的时间上被一个内部的检查点spout触发。这个指定的时间使用topology.state.checkpoint.interval.ms来配置。
如果在topology中有至少有一个"有状态bolt"，那么checkpoint spout会自动被topology builder增加。
有状态的topology 结构，topology builder 在StatefulBoltExecutor中包装你的"有状态bolt"，这个StatefulBoltExecutor会收到并处理状态提交。
无状态的bolt被CheckpointTupleForwarder包装，这个CheckpointTupleForwarder仅仅定向到checkpoint tuples ， 这样checkpoint tuples 能够通过topoloby DAG.
在两个检查点期间，checkpoint tuple被checkpoint spout发射出。在接受checkpoint tuples时，保存bolt的状态，然后将checkpoint tuple发给下一个组件。
每个bolt等待检查点到达它的所有输入流，然后保存它状态，是状态和topology的状态一直。一旦checkpoint spout 从所有的bolt的收到ack，整个tuple被"完全处理"，并且事务被被记录作为保证。
topology.state.checkpoint.interval.ms应该低于topology.message.timeout.secs。检查间隔应该小于超时时间。如果时间还没到达检查点，但是所有的bolt已经acked了，那么spout也就acked了。
状态提交工作像一个有3个阶段的协议，有准备阶段和提交阶段。这样能使topology的状态总是保持一致。
2.1、恢复
当topology第一次启动时，恢复阶段就被触发了，如果之前的事务没有被成功地准备好，一个回滚消息被发送，这样bolt的一些准备事务就可以被丢弃了。
如果之前的状态被成功地准备好但是没有被提交，一个个提交消息被发送，这样准备好的事务能够被发送。
在上面的步骤完成后，bolt能够有状态地被初始化。
如果一个bolt未能确认检查点消息，或者和谋改革worker进程死掉了，那么恢复就会被触发。
因此，当个worker进程重新启动后，检查点机制确认bolt之前的状态并从这个位置继续。
2.2、保证性
storm依靠ack机制能够在失败时重新发送tuple。
这是有可能的：状态被提交了，但是worker进程在ack之前失败了，这种情况下tuple被重发，但是导致了重复的状态更新。
目前的StatefulBoltExecutor 继续处理新发送的tuple，这也能导致重复的状态更新。
状态更新斌没有消除重复的赋值，目前只提供at-least保证。
为了提供at-least保证，有状态topology的所有的bolt都需要锚定，并且一旦完成处理就ack
对于没有状态的bolt，锚定/ack 能够自动地被管理。
有状态的bolt将被期望手动锚定tuple并且ack the tuple 在处理完后。
2.3、有状态bolt的钩子(回调)
有状态的bolt提供钩子方法能够实现一些自定义的动作，在框架提交状态前，在preCommit()方法中可以做自己的操作。
21、与kafka集成
22、与HBase集成 http://storm.apache.org/releases/1.0.3/storm-hbase.html
23、与HDFS集成 http://storm.apache.org/releases/1.0.3/storm-hdfs.html
24、与hive集成 http://storm.apache.org/releases/1.0.3/storm-hive.html
25、与jdbc集成 http://storm.apache.org/releases/1.0.3/storm-jdbc.html
27、storm on yarn 略
28、storm常见问题：
nimbus宕机
supervisor宕机
worker进程失败
29、storm集群优化，spout配置优化，bolt配置优化
浙公网安备 33010602011771号