Skip to content

kafka.adoc

Latest commit

 

History

History
215 lines (157 loc) · 6.02 KB

kafka.adoc

File metadata and controls

215 lines (157 loc) · 6.02 KB

Core Kafka installation and operations

Create Kafka broker and topic

Add Streams for Apache Kafka operator to your OpenShift environment.

Create a Broker with metrics enabled:

  • consumer lag

  • consumer offsets

oc new-project my-kafka
oc apply -f k8s/basics/01-metrics-configmap.yaml
oc apply -f k8s/basics/02-kafka.yaml
oc apply -f k8s/basics/03-kafkatopic.yaml

Enabling user workload monitoring

Note
 This section relies on the cluster-admin role.
Important

This workshop is designed to be run in the my-kafka namespace. To change the namespace, you need to update accordingly the following files:

  • k8s/monitor/clusterrolebinding.patch.yaml

  • k8s/monitor/podmonitor.patch.yaml

The following command add many resources in OpenShift through kustomizer:

oc apply -k k8s/monitor

Specifically it does the following:

  • adds a ConfigMap named cluster-monitoring-config in the openshift-monitoring namespace: it triggers the deployment of Prometheus and Thanos. Grafana uses the Thanos Querier which works as Prometheus proxy to scrape the metrics.

  • creates pod monitoring rules for kafka resources

  • creates prometheus rules

  • creates the grafana service account and the related access token

  • creates a ClusterRoleBinding which grants permissions to grafana-serviceaccount

Optionally, check that the prometheus-operator, prometheus-user-workload and thanos-ruler-user-workload pods are running in the openshift-user-workload-monitoring project.

oc -n openshift-user-workload-monitoring get pod

Deploy Grafana

The following script creates a datasource configuration for Grafana to scrape the metrics. It points to Thanos Querier which acts as a Prometheus proxy and uses the access token in from grafana-serviceaccount.

k8s/monitor/01-create-datasource.sh

Grafana deployment:

oc apply -f k8s/monitor/05-grafana.yaml

Expose grafana:

oc create route edge --service=grafana

Login with the default credentials (admin/admin) and then change the password.

Load the dashboard definitions from grafana-dashboards folder:

  • strimzi-kafka.json

  • strimzi-kafka-exporter.json

Install the consumer and producer applications

mvn -f kafka-consumer package -Dquarkus.kubernetes.deploy=true -DskipTests
mvn -f kafka-producer package -Dquarkus.kubernetes.deploy=true -DskipTests

Further information about the applications:

Demo routines

AMQ Streams High Availability

  1. Show consumer logs

    oc logs -f deployments/kafka-consumer

  2. Show producer logs

    oc logs -f deployments/kafka-producer

  3. Show the partitions distribution

    oc exec -it my-cluster-controller-3 -- \
    bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 \
    --describe --topic event

  4. Test the Kafka’s resilience and consistency by forcefully shutting down one of brokers' pod.

    Tip
    		 Use the following command: oc delete --force pod <pod-name>
    Important
    		 Watching at the consumer log you should notice that it temporarily stops processing some messages (missing messages). This is expected! In fact, Kafka promotes consistency over availability, so until a new partition leader is elected you cannot write or consume messages on that partition. Eventually, the new leader will become available and the missing messages will be caught up.

  5. Show again the topic distribution on the cluster members

  6. Show the dashboard

Clean up

To temporarily stop the demo , you can use the following command:

oc scale deployment kafka-producer --replicas 0
oc scale deployment kafka-consumer --replicas 0
oc scale deployment grafana --replicas 0
oc delete kafkanodepool broker controller
oc delete kafka my-cluster
oc delete pvc -l app=my-kafka-demo

To restart the demo, you can use the following command:

oc apply -f k8s/basics/02-kafka.yaml
oc scale deployment kafka-producer --replicas 1
oc scale deployment kafka-consumer --replicas 1
oc scale deployment grafana --replicas 1

To permanently clean up the demo, you can use the following command:

oc delete all -l app=kafka-clients
oc delete all -l app=kafka-monitor
oc delete kafkatopic event
oc delete kafkanodepool broker controller
oc delete kafka my-cluster
oc delete pvc -l app=my-kafka-demo
oc delete project my-kafka

Appendix

Refresh Grafana token

Prometheus access token lasts 30 days.

To refresh it:

oc delete configmap grafana-config
k8s/monitor/01-create-datasource.sh
oc delete pod --selector name=grafana

Full Grafana clean up

Delete Grafana deployment:

oc delete all -l app=kafka-monitor
oc delete podmonitor -l app=kafka-monitor
oc delete sa -l app=kafka-monitor
oc delete pvc -l app=kafka-monitor
oc delete configmap grafana-config

Entities segregation

Segregating cluster configuration from daily operational tasks, such as creating users and topics, improves security and simplifies management. This separation is implemented by establishing dedicated namespaces, like kafka-users for user management and kafka-topics for topic management. By assigning appropriate Role-Based Access Control (RBAC) permissions, users can manage these resources without requiring access to the core Kafka infrastructure. The Kafka CRD’s watchedNamespace property, within the entityOperator section, restricts the Topic and User Operators to their designated namespaces.

For instance:

  entityOperator:
    topicOperator:
      watchedNamespace: kafka-users
    userOperator:
      watchedNamespace: kafka-topics

Following this configuration, create Topic and User Custom Resource Definitions (CRDs) within the corresponding kafka-topics and kafka-users namespaces, respectively.
⚡