https://ops.docs.72602.space/tags/kafka/index.html Hugo en Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/knative/eventing/plugin/eventing-kafka-broker/prepare_dev_environment/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/knative/eventing/plugin/eventing-kafka-broker/prepare_dev_environment/index.html update go -> 1.24 install ko -> 1.8.0 go install github.com/google/ko@latest # wget https://github.com/ko-build/ko/releases/download/v0.18.0/ko_0.18.0_Linux_x86_64.tar.gz # tar -xzf ko_0.18.0_Linux_x86_64.tar.gz -C /usr/local/bin/ko # cp /usr/local/bin/ko/ko /root/bin protoc PB_REL="https://github.com/protocolbuffers/protobuf/releases" curl -LO $PB_REL/download/v30.2/protoc-30.2-linux-x86_64.zip # mkdir -p ${HOME}/bin/ mkdir -p /usr/local/bin/protoc unzip protoc-30.2-linux-x86_64.zip -d /usr/local/bin/protoc cp /usr/local/bin/protoc/bin/protoc /root/bin # export PATH="$PATH:/root/bin" rm -rf protoc-30.2-linux-x86_64.zip protoc-gen-go -> 1.5.4 go install google.golang.org/protobuf/cmd/protoc-gen-go@latest export GOPATH=/usr/local/go/bin copy some code mkdir -p ${GOPATH}/src/knative.dev cd ${GOPATH}/src/knative.dev git clone git@github.com:knative/eventing.git # clone eventing repo git clone git@github.com:AaronYang0628/eventing-kafka-broker.git cd eventing-kafka-broker git remote add upstream https://github.com/knative-extensions/eventing-kafka-broker.git git remote set-url --push upstream no_push export KO_DOCKER_REPO=docker-registry.lab.zverse.space/data-and-computing/ay-dev https://ops.docs.72602.space/kubernetes/serverless/knative/eventing/async_prediction_flow/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/knative/eventing/async_prediction_flow/index.html Flow flowchart LR A[User Curl] -->|HTTP| B{ISVC-Broker:Kafka} B -->|Subscribe| D[Trigger1] B -->|Subscribe| E[Kserve-Triiger] B -->|Subscribe| F[Trigger3] E --> G[Mnist Service] G --> |Kafka-Sink| B Setps 1. Create Broker Setting kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: kafka-broker-config namespace: knative-eventing data: default.topic.partitions: "10" default.topic.replication.factor: "1" bootstrap.servers: "kafka.database.svc.cluster.local:9092" #kafka service address default.topic.config.retention.ms: "3600" EOF 2. Create Broker kubectl apply -f - <<EOF apiVersion: eventing.knative.dev/v1 kind: Broker metadata: annotations: eventing.knative.dev/broker.class: Kafka name: isvc-broker namespace: kserve-test spec: config: apiVersion: v1 kind: ConfigMap name: kafka-broker-config namespace: knative-eventing EOF 3. Create Trigger kubectl apply -f - << EOF apiVersion: eventing.knative.dev/v1 kind: Trigger metadata: name: kserve-trigger namespace: kserve-test spec: broker: isvc-broker filter: attributes: type: prediction-request-udf-attr # you can change this subscriber: uri: http://prediction-and-sink.kserve-test.svc.cluster.local/v1/models/mnist:predict EOF 4. Create InferenceService 1kubectl apply -f - <<EOF 2apiVersion: serving.kserve.io/v1beta1 3kind: InferenceService 4metadata: 5 name: prediction-and-sink 6 namespace: kserve-test 7spec: 8 predictor: 9 model: 10 modelFormat: 11 name: pytorch 12 storageUri: gs://kfserving-examples/models/torchserve/image_classifier/v1 13 transformer: 14 containers: 15 - image: docker-registry.lab.zverse.space/data-and-computing/ay-dev/msg-transformer:dev9 16 name: kserve-container 17 env: 18 - name: KAFKA_BOOTSTRAP_SERVERS 19 value: kafka.database.svc.cluster.local 20 - name: KAFKA_TOPIC 21 value: test-topic # result will be saved in this topic 22 - name: REQUEST_TRACE_KEY 23 value: test-trace-id # using this key to retrieve preidtion result 24 command: 25 - "python" 26 - "-m" 27 - "model" 28 args: 29 - --model_name 30 - mnist 31EOF Expectd Output root@ay-k3s01:~# kubectl -n kserve-test get pod NAME READY STATUS RESTARTS AGE prediction-and-sink-predictor-00001-deployment-f64bb76f-jqv4m 2/2 Running 0 3m46s prediction-and-sink-transformer-00001-deployment-76cccd867lksg9 2/2 Running 0 4m3s Expectd Output Source code of the docker-registry.lab.zverse.space/data-and-computing/ay-dev/msg-transformer:dev9 could be found 🔗here https://ops.docs.72602.space/kubernetes/serverless/knative/eventing/broker/install_steps/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/knative/eventing/broker/install_steps/index.html About Source, curl, kafkaSource, Broker Trigger Sink: ksvc, isvc Install a Channel (messaging) layer kubectl apply -f https://github.com/knative-extensions/eventing-kafka-broker/releases/download/knative-v1.18.0/eventing-kafka-controller.yaml Expectd Output configmap/kafka-broker-config created configmap/kafka-channel-config created customresourcedefinition.apiextensions.k8s.io/kafkachannels.messaging.knative.dev created customresourcedefinition.apiextensions.k8s.io/consumers.internal.kafka.eventing.knative.dev created customresourcedefinition.apiextensions.k8s.io/consumergroups.internal.kafka.eventing.knative.dev created customresourcedefinition.apiextensions.k8s.io/kafkasinks.eventing.knative.dev created customresourcedefinition.apiextensions.k8s.io/kafkasources.sources.knative.dev created clusterrole.rbac.authorization.k8s.io/eventing-kafka-source-observer created configmap/config-kafka-source-defaults created configmap/config-kafka-autoscaler created configmap/config-kafka-features created configmap/config-kafka-leader-election created configmap/kafka-config-logging created configmap/config-namespaced-broker-resources created configmap/config-tracing configured clusterrole.rbac.authorization.k8s.io/knative-kafka-addressable-resolver created clusterrole.rbac.authorization.k8s.io/knative-kafka-channelable-manipulator created clusterrole.rbac.authorization.k8s.io/kafka-controller created serviceaccount/kafka-controller created clusterrolebinding.rbac.authorization.k8s.io/kafka-controller created clusterrolebinding.rbac.authorization.k8s.io/kafka-controller-addressable-resolver created deployment.apps/kafka-controller created clusterrole.rbac.authorization.k8s.io/kafka-webhook-eventing created serviceaccount/kafka-webhook-eventing created clusterrolebinding.rbac.authorization.k8s.io/kafka-webhook-eventing created mutatingwebhookconfiguration.admissionregistration.k8s.io/defaulting.webhook.kafka.eventing.knative.dev created mutatingwebhookconfiguration.admissionregistration.k8s.io/pods.defaulting.webhook.kafka.eventing.knative.dev created secret/kafka-webhook-eventing-certs created validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.kafka.eventing.knative.dev created deployment.apps/kafka-webhook-eventing created service/kafka-webhook-eventing created kubectl apply -f https://github.com/knative-extensions/eventing-kafka-broker/releases/download/knative-v1.18.0/eventing-kafka-channel.yaml Expectd Output configmap/config-kafka-channel-data-plane created clusterrole.rbac.authorization.k8s.io/knative-kafka-channel-data-plane created serviceaccount/knative-kafka-channel-data-plane created clusterrolebinding.rbac.authorization.k8s.io/knative-kafka-channel-data-plane created statefulset.apps/kafka-channel-dispatcher created deployment.apps/kafka-channel-receiver created service/kafka-channel-ingress created Install a Broker layer kubectl apply -f https://github.com/knative-extensions/eventing-kafka-broker/releases/download/knative-v1.18.0/eventing-kafka-broker.yaml Expectd Output configmap/config-kafka-broker-data-plane created clusterrole.rbac.authorization.k8s.io/knative-kafka-broker-data-plane created serviceaccount/knative-kafka-broker-data-plane created clusterrolebinding.rbac.authorization.k8s.io/knative-kafka-broker-data-plane created statefulset.apps/kafka-broker-dispatcher created deployment.apps/kafka-broker-receiver created service/kafka-broker-ingress created Reference for more information, you can check 🔗https://knative.dev/docs/eventing/brokers/broker-types/kafka-broker/ https://ops.docs.72602.space/kubernetes/serverless/knative/eventing/broker/display_broker_msg/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/knative/eventing/broker/display_broker_msg/index.html Flow flowchart LR A[Curl] -->|HTTP| B{Broker} B -->|Subscribe| D[Trigger1] B -->|Subscribe| E[Trigger2] B -->|Subscribe| F[Trigger3] E --> G[Display Service] Setps 1. Create Broker Setting kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: kafka-broker-config namespace: knative-eventing data: default.topic.partitions: "10" default.topic.replication.factor: "1" bootstrap.servers: "kafka.database.svc.cluster.local:9092" #kafka service address default.topic.config.retention.ms: "3600" EOF 2. Create Broker kubectl apply -f - <<EOF apiVersion: eventing.knative.dev/v1 kind: Broker metadata: annotations: eventing.knative.dev/broker.class: Kafka name: first-broker namespace: kserve-test spec: config: apiVersion: v1 kind: ConfigMap name: kafka-broker-config namespace: knative-eventing EOF deadletterSink: https://ops.docs.72602.space/installation/database/kafka/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/installation/database/kafka/index.html Installation Install By Helm✔️ ArgoCD✔️ Docker Compose✔️ Preliminary 1. Kubernetes is installed; if not, check 🔗link 2. Helm binary is installed; if not, check 🔗link 1.get helm repo Details helm repo add bitnami oci://registry-1.docker.io/bitnamicharts/kafka helm repo update 2.install chart https://ops.docs.72602.space/kubernetes/serverless/knative/eventing/broker/kafka_broker_invoke_isvc/index.html Thu, 07 Mar 2024 15:00:59 +0800 https://ops.docs.72602.space/kubernetes/serverless/knative/eventing/broker/kafka_broker_invoke_isvc/index.html 1. Prepare RBAC create cluster role to access CRD isvc kubectl apply -f - <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: kserve-access-for-knative rules: - apiGroups: ["serving.kserve.io"] resources: ["inferenceservices", "inferenceservices/status"] verbs: ["get", "list", "watch"] EOF create rolebinding and grant privileges kubectl apply -f - <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kafka-controller-kserve-access roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kserve-access-for-knative subjects: - kind: ServiceAccount name: kafka-controller namespace: knative-eventing EOF 2. Create Broker Setting kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: kafka-broker-config namespace: knative-eventing data: default.topic.partitions: "10" default.topic.replication.factor: "1" bootstrap.servers: "kafka.database.svc.cluster.local:9092" #kafka service address default.topic.config.retention.ms: "3600" EOF 3. Create Broker kubectl apply -f - <<EOF apiVersion: eventing.knative.dev/v1 kind: Broker metadata: annotations: eventing.knative.dev/broker.class: Kafka name: isvc-broker namespace: kserve-test spec: config: apiVersion: v1 kind: ConfigMap name: kafka-broker-config namespace: knative-eventing delivery: deadLetterSink: ref: apiVersion: serving.knative.dev/v1 kind: Service name: event-display EOF 4. Create InferenceService Reference you can create isvc first-tourchserve service, by following 🔗link