Get the best from Apache Kafka® Connect#
We recommend to follow these best practices to ensure that your Apache Kafka® Connect service is fast and reliable.
Pay attention to
tasks.max for connector configurations#
By default, connectors run a maximum of 1 task, which usually leads
to under-utilization for large Apache Kafka Connect services (unless you have
many connectors with one task each). In general, it is good to keep the
cluster CPUs occupied with connector tasks without overloading them. If
your connector is under-performing, you can try increasing
to match the number of partitions.
Consider a standalone Apache Kafka Connect service#
You can run Apache Kafka Connect as part of your existing Aiven for Apache Kafka service (for business-4 and higher service plans). While this allows you to try out Apache Kafka Connect by enabling the feature within your existing service, for heavy usage we recommend that you enable a standalone Apache Kafka Connect service to run your connectors. This allows you to scale the Apache Kafka service and the connector service independently and offers more CPU time and memory for the Apache Kafka Connect service.
Having a dedicated cluster that moves the data and does not add load to the main nodes goes according to the “separation of concerns” paradigm.