kafka 安装部署配置
版本为2.2.1：kafka_2.11_2.2.1
第一步：下载
地址： https://kafka.apache.org/downloads
选择 Binary downloads:
- Scala 2.11 - kafka_2.11-2.2.1.tgz (asc, sha512)
第二步：上传到服务器并解压
$ tar -xzf kafka_2.13-3.1.0.tgz $ cd kafka_2.13-3.1.0
第三步：修改server.properties文件
listeners=PLAINTEXT://10.10.10.10:9092 改成服务器的实际ip
note：这里如果配置的是localhost，控制台的--bootstrap-server 需要用localhost:9092，如果是配置的实际的ip（假设10.10.1.0.10），控制台就用10.10.10.10:9092
log文件存储位置改为自己想指定的位置，这里强烈建议不要放在/tmp文件夹下面，这个/tmp文件夹是linux下的一个特殊文件目录
log.dirs=/tmp1/kafka-logs
第四步：启动kafka服务（java8+）先启动zk
bin/kafka-server-start.sh -daemon config/server.properties
查看pid或9092端口
ps -ef |grep kafka or netstat -anp|grep 9092
没起来，因为还没起zookeeper，执行
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
查看zk端口
netstat -anp |grep 2181
正常，再启动kafka，查看9092端口或pid，正常
第五步：测试生产和消费
查看topic列表
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
新装的，啥也没有，创建一个topic
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test001
再查看，test001创建成功，向test001发送一条消息"hello"，注意这里的生产者用的参数是 broker-list 而不是bootstrap-server
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test001
另开一个terminal，执行后台消费查看是否收到"hello"消息
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test001 --from-beginning
收到消息"hello"，验证结束
停止kafka服务
bin\kafka-server-stop.sh
kafka服务日志查看位置
./logs 下的kafkaServer.out
附录server.properties配置参数说明
官网： https://kafka.apache.org/documentation/#configuration
############################# Server Basics ############################# # The id of the broker. This must be set to a unique integer for each broker. # 节点的ID，必须与其它节点不同,建议值为ip三位 broker.id=0 ############################# Socket Server Settings ############################# # The address the socket server listens on. It will get the value returned from # java.net.InetAddress.getCanonicalHostName() if not configured. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 # 套接字服务器监听的地址。如果没有配置，就使用java.net.InetAddress.getCanonicalHostName()的返回值 #listeners=PLAINTEXT://:9092 # Hostname and port the broker will advertise to producers and consumers. If not set, # it uses the value for "listeners" if configured. Otherwise, it will use the value # returned from java.net.InetAddress.getCanonicalHostName(). #advertised.listeners=PLAINTEXT://your.host.name:9092 # Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details #listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL # The number of threads that the server uses for receiving requests from the network and sending responses to the network # 接受网络请求的线程数 num.network.threads=3 # The number of threads that the server uses for processing requests, which may include disk I/O # 进行磁盘IO的线程数 num.io.threads=8 # The send buffer (SO_SNDBUF) used by the socket server # 套接字服务器使用的发送缓冲区大小 socket.send.buffer.bytes=102400 # The receive buffer (SO_RCVBUF) used by the socket server # 套接字服务器使用的接收缓冲区大小 socket.receive.buffer.bytes=102400 # The maximum size of a request that the socket server will accept (protection against OOM) # 单个请求最大能接收的数据量 socket.request.max.bytes=104857600 ############################# Log Basics ############################# # A comma separated list of directories under which to store log files # 一个逗号分隔的目录列表，用来存储日志文件 log.dirs=/tmp/kafka-logs # The default number of log partitions per topic. More partitions allow greater # parallelism for consumption, but this will also result in more files across # the brokers. # 每个主题的日志分区的默认数量。更多的分区允许更大的并行操作，但是它会导致节点产生更多的文件 num.partitions=1 # The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. # This value is recommended to be increased for installations with data dirs located in RAID array. # 每个数据目录中的线程数，用于在启动时日志恢复，并在关闭时刷新。 num.recovery.threads.per.data.dir=1 ############################# Internal Topic Settings ############################# # The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state" # 内部主题“_consumer_offsets”和“_transaction_state”的复制因子 # For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3. # 对于开发测试以外的任何情况，建议使用大于1的值来确保可用性，比如3。 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 ############################# Log Flush Policy ############################# # Messages are immediately written to the filesystem but by default we only fsync() to sync # the OS cache lazily. The following configurations control the flush of data to disk. # 消息直接被写入文件系统，但是默认情况下我们仅仅调用fsync()以延迟的同步系统缓存 # There are a few important trade-offs here: # 这些有一些重要的权衡 # 1. Durability: Unflushed data may be lost if you are not using replication. # 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush. # 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks. # 1. 持久性:如果不使用复制，未刷新的数据可能会丢失。 # 2. 延迟:非常大的刷新间隔可能会在刷新时导致延迟，因为将会有大量数据刷新。 # 3. 吞吐量:刷新通常是最昂贵的操作，而一个小的刷新间隔可能会导致过多的搜索。 # The settings below allow one to configure the flush policy to flush data after a period of time or # every N messages (or both). This can be done globally and overridden on a per-topic basis. # 下面的设置允许你去配置刷新策略，每隔一段时间刷新或者一次N个消息（或者两个都配置）。这可以在全局范围内完成，并在每个主题的基础上重写。 # The number of messages to accept before forcing a flush of data to disk # 在强制刷新数据到磁盘之前允许接收消息的数量 #log.flush.interval.messages=10000 # The maximum amount of time a message can sit in a log before we force a flush # 在强制刷新之前，消息可以在日志中停留的最长时间 #log.flush.interval.ms=1000 ############################# Log Retention Policy ############################# # The following configurations control the disposal of log segments. The policy can # be set to delete segments after a period of time, or after a given size has accumulated. # 以下的配置控制了日志段的处理。策略可以配置为每隔一段时间删除片段或者到达一定大小之后。 # A segment will be deleted whenever *either* of these criteria are met. Deletion always happens # from the end of the log. # 当满足这些条件时，将会删除一个片段。删除总是发生在日志的末尾。 # The minimum age of a log file to be eligible for deletion due to age # 一个日志的最小存活时间 log.retention.hours=168 # A size-based retention policy for logs. Segments are pruned from the log unless the remaining # segments drop below log.retention.bytes. Functions independently of log.retention.hours. # 一个基于大小的日志保留策略。只要剩下的部分不低于log.retention.bytes，段将从日志中删除。 #log.retention.bytes=1073741824 # The maximum size of a log segment file. When this size is reached a new log segment will be created. # 每一个日志段大小的最大值。当到达这个大小时，会生成一个新的片段。 log.segment.bytes=1073741824 # The interval at which log segments are checked to see if they can be deleted according # to the retention policies # 检查日志段的时间间隔，看是否可以根据保留策略删除它们 log.retention.check.interval.ms=300000 ############################# Zookeeper ############################# # Zookeeper connection string (see zookeeper docs for details). # This is a comma separated host:port pairs, each corresponding to a zk # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". # You can also append an optional chroot string to the urls to specify the # root directory for all kafka znodes. # 配置zookeeper地址，多个以逗号分隔，还可以在最后指定kafka存储的namespace：eg："127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002/kafka" zookeeper.connect=localhost:2181 # Timeout in ms for connecting to zookeeper # 连接到Zookeeper的超时时间 zookeeper.connection.timeout.ms=6000 ############################# Group Coordinator Settings ############################# # The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance. # 指定GroupCoordinator延迟初始消费者再平衡的时间(以毫秒为单位) # The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms. # The default value for this is 3 seconds. # 默认值是3秒 # We override this to 0 here as it makes for a better out-of-the-box experience for development and testing. # However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup. # 在生产环境中，3秒的默认值更合适 group.initial.rebalance.delay.ms=0
生产者producer.properties
# list of brokers used for bootstrapping knowledge about the rest of the cluster # format: host1:port1,host2:port2 ... bootstrap.servers=localhost:9092 # 指定数据压缩格式：none、gzip、snappy、lz4、zstd compression.type=none # name of the partitioner class for partitioning events; default partition spreads data randomly # 分区策略，默认随机 #partitioner.class= # the maximum amount of time the client will wait for the response of a request # 客户端等待请求响应的最大时间，即在向 producer 发送 ack 之前，broker允许等待的最大时间 #request.timeout.ms= # how long `KafkaProducer.send` and `KafkaProducer.partitionsFor` will block for # `KafkaProducer.send` and `KafkaProducer.partitionsFor`阻塞的时间 #max.block.ms= # the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together # 生产者将等待一个延迟，以便和其他的消息组合成一个批次发出，减少发送的请求数 #linger.ms= # the maximum size of a request in bytes # 请求的最大字节数 #max.request.size= # the default batch size in bytes when batching multiple records sent to a partition # 在批处理消息发送到一个分区的多个记录时的批处理大小，默认16k #batch.size= # the total bytes of memory the producer can use to buffer records waiting to be sent to the server # 生产者可以用来缓存消息的缓冲区大小 #buffer.memory= # 默认配置文件只有一部分，比如下面的acks,更多配置可参考： http://kafka.apache.org/documentation/#producerconfigs # 在请求完成之前，生产者要求leader已收到确认消息的数量。 # 0：生产者不会等待服务器的ack响应，不保证服务器收到消息，返回的偏移量总是-1 # 1：leader将消息写入本地日志，但不会管其他follower是否写入。 # all：leader和所有的follower都成功同步消息才会返回ack，保证消息不会丢失。all等价于-1. acks=
brokers集群部署
> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties
然后配置：
broker.id是两个broker的唯一标志，port需要修改为不同的两个
config/server-1.properties:
broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs-1
config/server-2.properties:
broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/tmp/kafka-logs-2
启动两个节点：
> bin/kafka-server-start config/server-1.properties &
> bin/kafka-server-start config/server-2.properties &
杀死其中一个broker：用来测试容错性
ps | grep server-1.properties
kill -9 pid
