使用带有 Helm 安装的 Kafka/Confluent 的连接器

我使用 Helm charts 在本地 Minikube 上安装了 Kafka

我想连接 MySQL 和 Kafka，使用连接器之一(如 例如 Debezium MySQL CDC).在说明中，它说:

<块引用>

安装连接器

使用 Confluent Hub 客户端安装这个连接器:

confluent-hub install debezium/debezium-connector-mysql:0.9.2

听起来不错，除了 1) 我不知道在哪个 pod 上运行此命令，2) 似乎没有一个 pod 有可用的 confluent-hub 命令.

问题:

1. confluent-hub 不是通过这些 Helm 图表安装的吗?
2. 我必须自己安装 confluent-hub 吗?
3. 如果是这样，我必须将它安装在哪个 pod 上?

解决方案

理想情况下，这应该作为 helm 脚本的一部分进行配置，但不幸的是，目前还没有.解决此问题的一种方法是从 Confluent 的 Kafka Connect Docker 映像构建一个新的 Docker.手动下载连接器并将内容解压缩到一个文件夹中.将其内容复制到容器中的路径.像下面这样的东西.

Dockerfile 的内容

FROM confluentinc/cp-kafka-connect:5.2.1复制<连接器目录>/usr/share/java

/usr/share/java 是 Kafka Connect 寻找插件的默认位置.您还可以在 helm 安装期间使用不同的位置并提供新位置 (plugin.path).

构建此映像并将其托管在可访问的地方.您还必须在 helm 安装期间提供/覆盖图像和标签详细信息.

Here> 是 values.yaml 文件的路径.您可以在此处找到 image 和 plugin.path 值.

I have installed Kafka on a local Minikube by using the Helm charts https://github.com/confluentinc/cp-helm-charts following these instructions https://docs.confluent.io/current/installation/installing_cp/cp-helm-charts/docs/index.html like so:

helm install -f kafka_config.yaml confluentinc/cp-helm-charts --name kafka-home-delivery --namespace cust360

The kafka_config.yaml is almost identical to the default yaml, with the one exception being that I scaled it down to 1 server/broker instead of 3 (just because I'm trying to conserve resources on my local minikube; hopefully that's not relevant to my problem).

Also running on Minikube is a MySQL instance. Here's the output of kubectl get pods --namespace myNamespace:

I want to connect MySQL and Kafka, using one of the connectors (like Debezium MySQL CDC, for instance). In the instructions, it says:

Install your connector

Use the Confluent Hub client to install this connector with:

confluent-hub install debezium/debezium-connector-mysql:0.9.2

Sounds good, except 1) I don't know which pod to run this command on, 2) None of the pods seem to have a confluent-hub command available.

Questions:

1. Does confluent-hub not come installed via those Helm charts?
2. Do I have to install confluent-hub myself?
3. If so, which pod do I have to install it on?

解决方案

Ideally this should be configurable as part of the helm script, but unfortunately it is not as of now. One way to work around this is to build a new Docker from Confluent's Kafka Connect Docker image. Download the connector manually and extract the contents into a folder. Copy the contents of this to a path in the container. Something like below.

Contents of Dockerfile

FROM confluentinc/cp-kafka-connect:5.2.1
COPY <connector-directory> /usr/share/java

/usr/share/java is the default location where Kafka Connect looks for plugins. You could also use different location and provide the new location (plugin.path) during your helm installation.

Build this image and host it somewhere accessible. You will also have to provide/override the image and tag details during the helm installation.

Here is the path to the values.yaml file. You can find the image and plugin.path values here.