Performance test kafka consumer/producer
Alberto last edited by
I've developed a consumer kafka that does the following: - get an event on a kafka topic. - carries out some operations - emits one or more events on kafka topics
I would like to find a solution to measure the performance of this component (execution time between the reception of messages and the transmission of other messages).
I have looking for a solution using Jmeter or Gateling. But I have not found any solution.
Do you have any idea, or any link to help me?
If the component you've developed is consuming, processing and emitting messages than it is both Kafka consumer and producer.
If you want to test your component in isolation, i.e., measure the time it takes for your component to process a message, you can mock Kafka, mock Kafka consuming and producing parts or write your code in such a way the processing part is Kafka-agnostic and can be tested in isolation. You can read more here:
- Kafka: Are there are examples on how to use Mockito for unit testing Kafka?
- Python: how to mock a kafka topic for unit tests?
Once you have your component isolated, you may measure processing time with any performance framework.
If you, however, want to measure performance of Kafka configuration then I suggest reading: Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines).
There are also Gatling plugins for Kafka producer protocol.