docker如何搭建kafka集群
导读:本文共4916.5字符,通常情况下阅读需要16分钟。同时您也可以点击右侧朗读,来听本文内容。按键盘←(左) →(右) 方向键可以翻页。
摘要: 一、原生Docker命令1. 删除所有dangling数据卷(即无用的Volume,僵尸文件)dockervolumerm$(dockervolumels-qfdangling=true)2. 删除所有dangling镜像(即无tag的镜像)dockerrmi$(dockerimages|grep"^<none>"|awk&quo... ...
目录
(为您整理了一些要点),点击可以直达。一、原生Docker命令
1. 删除所有dangling数据卷(即无用的Volume,僵尸文件)
dockervolumerm$(dockervolumels-qfdangling=true)
2. 删除所有dangling镜像(即无tag的镜像)
dockerrmi$(dockerimages|grep"^<none>"|awk"{print$3}"
3. 删除所有关闭的容器
dockerps-a|grepExit|cut-d''-f1|xargsdockerrm
二、镜像选择
环境为M1版本的mbp:
Zookeeper采用zookeeper
Kafka采用wurstmeister/kafka
Kafka-Manager采用scjtqs/kafka-manager
Mysql采用mysql/mysql-server
三、集群规划
1. 新建docker网络
dockernetworkcreatedocker-net--subnet172.20.10.0/16
dockernetworkls
2. 集群规划
四、Zookeeper集群安装
1. 新建文件zk-docker-compose.yml
version:'3.4'services:zook1:image:zookeeper:latestrestart:alwayshostname:zook1container_name:zook1#容器名称,方便在rancher中显示有意义的名称ports:-2183:2181#将本容器的zookeeper默认端口号映射出去volumes:#挂载数据卷-"/Users/konsy/Development/volume/zkcluster/zook1/data:/data"-"/Users/konsy/Development/volume/zkcluster/zook1/datalog:/datalog"-"/Users/konsy/Development/volume/zkcluster/zook1/logs:/logs"environment:ZOO_MY_ID:1#即是zookeeper的节点值,也是kafka的brokerid值ZOO_SERVERS:server.1=zook1:2888:3888;2181server.2=zook2:2888:3888;2181server.3=zook3:2888:3888;2181networks:docker-net:ipv4_address:172.20.10.11zook2:image:zookeeper:latestrestart:alwayshostname:zook2container_name:zook2#容器名称,方便在rancher中显示有意义的名称ports:-2184:2181#将本容器的zookeeper默认端口号映射出去volumes:-"/Users/konsy/Development/volume/zkcluster/zook2/data:/data"-"/Users/konsy/Development/volume/zkcluster/zook2/datalog:/datalog"-"/Users/konsy/Development/volume/zkcluster/zook2/logs:/logs"environment:ZOO_MY_ID:2#即是zookeeper的节点值,也是kafka的brokerid值ZOO_SERVERS:server.1=zook1:2888:3888;2181server.2=zook2:2888:3888;2181server.3=zook3:2888:3888;2181networks:docker-net:ipv4_address:172.20.10.12zook3:image:zookeeper:latestrestart:alwayshostname:zook3container_name:zook3#容器名称,方便在rancher中显示有意义的名称ports:-2185:2181#将本容器的zookeeper默认端口号映射出去volumes:-"/Users/konsy/Development/volume/zkcluster/zook3/data:/data"-"/Users/konsy/Development/volume/zkcluster/zook3/datalog:/datalog"-"/Users/konsy/Development/volume/zkcluster/zook3/logs:/logs"environment:ZOO_MY_ID:3#即是zookeeper的节点值,也是kafka的brokerid值ZOO_SERVERS:server.1=zook1:2888:3888;2181server.2=zook2:2888:3888;2181server.3=zook3:2888:3888;2181networks:docker-net:ipv4_address:172.20.10.13networks:docker-net:external:name:docker-net
2. 执行脚本部署zookeeper至Docker:
dockercompose-f./zk-docker-compose.ymlup-d
五、Kafka集群安装
1. 新建文件kafka-docker-compose.yml
version:'2'services:kafka1:image:docker.io/wurstmeister/kafkarestart:alwayshostname:kafka1container_name:kafka1ports:-9093:9093-9193:9193environment:KAFKA_BROKER_ID:1KAFKA_LISTENERS:INSIDE://:9093,OUTSIDE://:9193#KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094SKAFKA_ADVERTISED_LISTENERS:INSIDE://kafka1:9093,OUTSIDE://localhost:9193KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXTKAFKA_INTER_BROKER_LISTENER_NAME:INSIDEKAFKA_ZOOKEEPER_CONNECT:zook1:2181,zook2:2181,zook3:2181ALLOW_PLAINTEXT_LISTENER:'yes'JMX_PORT:9999#开放JMX监控端口,来监测集群数据volumes:-/Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka-/Users/konsy/Development/volume/kafka/kafka1/kafka:/kafkaexternal_links:-zook1-zook2-zook3networks:docker-net:ipv4_address:172.20.10.14kafka2:image:docker.io/wurstmeister/kafkarestart:alwayshostname:kafka2container_name:kafka2ports:-9094:9094-9194:9194environment:KAFKA_BROKER_ID:2KAFKA_LISTENERS:INSIDE://:9094,OUTSIDE://:9194#KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094KAFKA_ADVERTISED_LISTENERS:INSIDE://kafka2:9094,OUTSIDE://localhost:9194KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXTKAFKA_INTER_BROKER_LISTENER_NAME:INSIDEKAFKA_ZOOKEEPER_CONNECT:zook1:2181,zook2:2181,zook3:2181ALLOW_PLAINTEXT_LISTENER:'yes'JMX_PORT:9999#开放JMX监控端口,来监测集群数据volumes:-/Users/konsy/Development/volume/kafka/kafka2/wurstmeister/kafka:/wurstmeister/kafka-/Users/konsy/Development/volume/kafka/kafka2/kafka:/kafkaexternal_links:-zook1-zook2-zook3networks:docker-net:ipv4_address:172.20.10.15kafka3:image:docker.io/wurstmeister/kafkarestart:alwayshostname:kafka3container_name:kafka3ports:-9095:9095-9195:9195environment:KAFKA_BROKER_ID:3KAFKA_LISTENERS:INSIDE://:9095,OUTSIDE://:9195#KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094KAFKA_ADVERTISED_LISTENERS:INSIDE://kafka3:9095,OUTSIDE://localhost:9195KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXTKAFKA_INTER_BROKER_LISTENER_NAME:INSIDEKAFKA_ZOOKEEPER_CONNECT:zook1:2181,zook2:2181,zook3:2181ALLOW_PLAINTEXT_LISTENER:'yes'JMX_PORT:9999#开放JMX监控端口,来监测集群数据volumes:-/Users/konsy/Development/volume/kafka/kafka3/wurstmeister/kafka:/wurstmeister/kafka-/Users/konsy/Development/volume/kafka/kafka3/kafka:/kafkaexternal_links:-zook1-zook2-zook3networks:docker-net:ipv4_address:172.20.10.16networks:docker-net:external:name:docker-net
2. 执行脚本部署kafka至Docker:
dockercompose-f./kafka-docker-compose.ymlup-d
3. listeners 和 advertised.listeners
listeners: 学名叫监听器,其实就是告诉外部连接者要通过什么协议访问指定主机名和端口开放的 Kafka 服务。
advertised.listeners:和 listeners 相比多了个 advertised。Advertised 的含义表示宣称的、公布的,就是说这组监听器是 Broker 用于对外发布的。
比如说:
listeners:INSIDE://172.17.0.10:9092,OUTSIDE://172.17.0.10:9094advertised_listeners:INSIDE://172.17.0.10:9092,OUTSIDE://<公网ip>:端口kafka_listener_security_protocol_map:"INSIDE:SASL_PLAINTEXT,OUTSIDE:SASL_PLAINTEXT"kafka_inter_broker_listener_name:"INSIDE"
advertised_listeners 监听器会注册在 zookeeper 中;
当我们对 172.17.0.10:9092 请求建立连接,kafka 服务器会通过 zookeeper 中注册的监听器,找到 INSIDE 监听器,然后通过 listeners 中找到对应的 通讯 ip 和 端口;
同理,当我们对 <公网 ip>:端口 请求建立连接,kafka 服务器会通过 zookeeper 中注册的监听器,找到 OUTSIDE 监听器,然后通过 listeners 中找到对应的 通讯 ip 和 端口 172.17.0.10:9094;
总结:advertised_listeners 是对外暴露的服务端口,真正建立连接用的是 listeners。
4.安装kafka-manager
4.1 新建文件kafka-manager-docker-compose.yml
version:'2'services:kafka-manager:image:scjtqs/kafka-manager:latestrestart:alwayshostname:kafka-managercontainer_name:kafka-managerports:-9000:9000external_links:#连接本compose文件以外的container-zook1-zook2-zook3-kafka1-kafka2-kafka3environment:ZK_HOSTS:zook1:2181,zook2:2181,zook3:2181KAFKA_BROKERS:kafka1:9093,kafka2:9094,kafka3:9095APPLICATION_SECRET:letmeinKM_ARGS:-Djava.net.preferIPv4Stack=truenetworks:docker-net:ipv4_address:172.20.10.10networks:docker-net:external:name:docker-net
4.2 执行脚本部署kafka-manager至Docker:
dockercompose-f./kafka-manager-docker-compose.ymlup-d
4.3 配置Cluster
5. 测试kafka
5.1连接容器
进入kafka容器的命令行:
dockerexec-tikafka/bin/bash
进入kafka所在目录:
cdopt/kafka_2.13-2.8.1/
5.2 创建Replication为2,Partition为2的topic
bin/kafka-topics.sh--create--zookeeperzook1:2181--replication-factor2--partitions2--topicpartopic
5.3 查看topic的状态
在kafka容器中的 opt/kafka_2.12-1.1.0/ 目录下输入
bin/kafka-topics.sh--describe--zookeeperzook1:2181--topicpartopic
</div> <div class="zixun-tj-product adv-bottom"></div> </div> </div> <div class="prve-next-news">
docker如何搭建kafka集群的详细内容,希望对您有所帮助,信息来源于网络。