网站关键词更换了,河北网站建设推广,siteserver cms网站访问量,网站备案代理公司前言
最近做项目#xff0c;需要使用kafka进行通信#xff0c;且只能使用kafka#xff0c;笔者没有测试集群#xff0c;就自己搭建了kafka集群#xff0c;实际上笔者在很早之前就搭建了#xff0c;因为当时还是zookeeper#xff08;简称ZK#xff09;注册元数据#…前言
最近做项目需要使用kafka进行通信且只能使用kafka笔者没有测试集群就自己搭建了kafka集群实际上笔者在很早之前就搭建了因为当时还是zookeeper简称ZK注册元数据现在新版kafka3.0.0开始已经自带了元数据能力使用raft协议减少了kafka对zk的依赖性。笔者在查询资料发现说jdk至少jdk11实测jdk8也能运行且并不需要网上说的34节点33即可当然理论上broker节点越多越好但是元数据节点建议3、5个最合适raft的过半一致性和容错性的综合取舍。
准备
准备kafka安装包Apache Kafka 笔者使用的kafka 3.5.0和scala 2.13采用3台虚拟机当然容器也不是不行注意持久化pv pvc和配置的管理ip换成域名dns的切换支持中间件建议使用虚拟机可以降低很多容错性。
jdk使用open jdk配置java_home和path以Ubuntu为例 sudo apt install openjdk-8-jdk-headless 以macOS为例创建一个ubuntu-server 最小安装的虚拟机vmware毕竟个人使用不要钱然后安装openssh 和 openjdk然后shutdown now 网络选择桥接相当于一台“真实在”网络上的一台物理机 这样就得到了
192.168.0.108
192.168.0.107
192.168.0.106
3台虚拟机
步骤
先看kafka集群的架构图实际上安装的过程就是架构图的执行过程 从图中可以看出已经没有zk的存在了从kafka节点自己管理元数据通过raft协议选主的方式。
1. kafka的准备
上传kafka安装包必须是二进制安装包不要源码包编译比较麻烦然后解压 tar -zxvf kafka_2.13-3.5.0.gz 查看配置目录会发现 明显多了kraft的配置目录那么如果使用kafka raft元数据中心则需要修改kraft目录启动时指定kraft目录的配置
2. 配置修改
raft协议实际上跟zk差不多使用raft协议的中间件就太多了但是本质上每个节点都需要一个唯一idzk也是如此所以kafka kraft就相当于集成的zk。
在kraft下的有3个文件文件其中启动相关的是server.properties中 执行配置修改 # The role of this server. Setting this puts us in KRaft mode
process.rolesbroker,controller# The node id associated with this instances roles
node.id1# The connect string for the controller quorum
controller.quorum.voters1localhost:9093############################# Socket Server Settings ############################## The address the socket server listens on.
# Combined nodes (i.e. those with process.rolesbroker,controller) must list the controller listener here at a minimum.
# If the broker listener is not defined, the default listener will use a host name that is equal to the value of java.net.InetAddress.getCanonicalHostName(),
# with PLAINTEXT listener name, and port 9092.
# FORMAT:
# listeners listener_name://host_name:port
# EXAMPLE:
# listeners PLAINTEXT://your.host.name:9092
listenersPLAINTEXT://:9092,CONTROLLER://:9093# Name of listener used for communication between brokers.
inter.broker.listener.namePLAINTEXT# Listener name, hostname and port the broker will advertise to clients.
# If not set, it uses the value for listeners.
advertised.listenersPLAINTEXT://localhost:9092# A comma-separated list of the names of the listeners used by the controller.
# If no explicit mapping set in listener.security.protocol.map, default will be using PLAINTEXT protocol
# This is required if running in KRaft mode.
controller.listener.namesCONTROLLER# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
listener.security.protocol.mapCONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads3# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads8# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes102400# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes102400# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes104857600############################# Log Basics ############################## A comma separated list of directories under which to store log files
log.dirs/tmp/kraft-combined-logs# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions1# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir1############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics __consumer_offsets and __transaction_state
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor1
transaction.state.log.replication.factor1
transaction.state.log.min.isr1############################# Log Flush Policy ############################## Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages10000# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms1000############################# Log Retention Policy ############################## The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours168# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes1073741824# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes1073741824# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms300000每一行都有注释重点关注
笔者设定
192.168.0.106 nodeid 1
192.168.0.107 nodeid 2
192.168.0.108 nodeid 3 至此配置基本上完成同理一个节点可以同时是controller和broker也可以仅仅是controller或者broker因为controller的负载比较轻所以一般是和broker一起。其中有个log.dir这个的路径是下面元数据生成的路径选主和数据事务日志索引日志的存储目录
3. 启动
1. 生成uuid 任意找一个节点执行 ./kafka-storage.sh random-uuid 每次执行uuid会不一样这个uuid标识是一个集群所以所有节点公用一个uuid不要每个节点重新生成会识别不了 然后执行format如下标红是我生成的这个每次不是固定的 ./kafka-storage.sh format -t gZzkfRm4T1y8wSAY-ZNG5Q -c ../config/kraft/server.properties 格式化配置文件同步其他节点 配置文件有什么变化在日志配置的目录下出现 关键还是meta的文件有集群id和节点id版本号这个对启动至关重要。
即在上面的log.dir的目录生成所以尽量不能使用临时目录
2. 启动
启动就很简单了使用刚刚配置的server.properties执行启动即可 ./kafka-server-start.sh -daemon ../config/kraft/server.properties 不过为了方便查看启动日志建议执行日志的console文件输出 先看事务日志和索引 验证
验证很简单查看bin同级目录下的日志即可 日志带有[2025-02-08 08:34:12,286] INFO [KafkaRaftServer nodeId1] Kafka Server started (kafka.server.KafkaRaftServer)
如果生成用途可以安装kafka的控制台kafka-ui不过我这里就不安装了因为docker安装比较容易。
总结
kafka从3.0.0开始推出了raft模式的元数据中心实际上类似zkkafka自己命名kraft。使用这种方式搭建kafka集群将不再需要zk同理kafka的集群的每个节点可以同时是broker和controller以前zk充当也可以是单独的brokercontroller负载不重不建议单独controller跟zk没区别官方说明需要jdk11及以上实测jdk8可以运行但是生成建议严格按照官方标定的jdk执行jdk是向下兼容的但是不确定是否会涉及新api或新特性的使用。
另外实际使用中可能会涉及使用iptables做nat限制kafka的连接方比如在kafka节点通过iptables限制发送者或者消费端的ip
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 9093 -j DNAT --to-destination kafkaxxx:9093
kafkaxxx --- 指定的是 Kafka 服务所在的机器地址
如果kafka是对接方提供则在nat打通时需要客户端连接的服务器也执行iptables否则可能出现连接kafka正常但是不能消费。
iptables -t nat -A POSTROUTING -p tcp -m tcp --dport 9093 -j SNAT --to-source natxxx
natxxx --- 指定的是配置 iptables 的本机的地址
