Enable the consumer lag predictor for Aiven for Apache Kafka®#
The consumer lag predictor in Aiven for Apache Kafka® provides visibility into the time between message production and consumption, allowing for improved cluster performance and scalability.
Consumer Lag Predictor for Aiven for Apache Kafka® is a limited availability feature. If you’re interested in trying out this feature, contact the sales team at sales@Aiven.io.
Before you start, ensure you have the following:
Enable via Aiven Console#
In Aiven Console, select your project and then choose your Aiven for Apache Kafka® service.
Click Service settings on the sidebar.
Scroll to the Advanced configuration section, and select Configure.
In the Advanced configuration screen, select Add configuration options.
In the add configuration options:
Find and set
kafka_lag_predictor.enabledto Enabled position. This enables the lag predictor to compute predictions for all consumer groups across all topics.
kafka_lag_predictor.group_filtersand enter the desired consumer group pattern. This specifies which consumer groups to consider during lag prediction calculations.
By default, the consumer lag predictor calculates the lag of all consumer groups. To restrict the calculation to specific groups, use the
Select Save configurations to save your changes and enable consumer lag prediction.
Enable via Aiven CLI#
Follow these steps to enable the consumer lag predictor for your Aiven for Apache Kafka service using Aiven CLI.
Retrieve the project information using the following command:
avn project details
If you need details for a specific project, use:
avn project details --project <your_project_name>
Get the name of the Aiven for Apache Kafka service for which you want to enable the consumer lag predictor by using the following command:
avn service list
Make a note of the
SERVICE_NAMEcorresponding to your Aiven for Apache Kafka service.
Enable the consumer lag predictor for your service:
avn service update <SERVICE_NAME> -c kafka_lag_predictor.enabled=true
Replace <SERVICE_NAME> with your actual service name.
This enables the lag predictor to compute predictions for all consumer groups across all topics.
If you wish to specify which consumer groups should be considered when calculating the lag prediction, you can set the
avn service update <SERVICE_NAME> \ -c kafka_lag_predictor.group_filters=\ '["example_consumer_group_1", "example_consumer_group_2"]'
<SERVICE_NAME>with the actual name or ID of your Aiven for Apache Kafka® service.
example_consumer_group_2with your actual consumer group names.
Monitor metrics with Prometheus#
After enabling the consumer lag predictor, you can use Prometheus to access and monitor detailed metrics that offer insights into your Kafka cluster’s performance. Here are the specific metrics and what they represent:
Represents the total count of records produced.
Represents the total count of records consumed.
Represents the estimated time lag, in seconds, for a consumer group to catch up to the latest message.