Kafka Performance Benchmarking with OpenMessaging Framework

A complete guide for running standardized Kafka performance benchmarks using the OpenMessaging Benchmark framework with cluster-friendly configurations.

Overview

This guide provides a containerized solution for running Kafka performance benchmarks that are:

Cluster-friendly - Won't overwhelm your Kafka infrastructure
Customer-ready - Professional interactive menu for Engineers
Reproducible - Dockerized environment with consistent results
Flexible - Multiple test scenarios from validation to throughput testing

Prerequisites

Docker and Docker Compose installed
Access to your Vast DATA cluster

Quick Start

# 1. Clone the repository
git clone https://github.com/openmessaging/openmessaging-benchmark.git
cd openmessaging-benchmark

# 2. Add the benchmark files (see Setup section below)

# 3. Configure for your Kafka cluster
chmod +x setup-kafka-config.sh
./setup-kafka-config.sh

# 4. Build and start
chmod +x build-and-start.sh
./build-and-start.sh

# 5. Run benchmarks
chmod +x run-external-benchmarks.sh
./run-external-benchmarks.sh

Complete Setup

Step 1: Repository Setup

Clone the OpenMessaging benchmark repository:

git clone https://github.com/openmessaging/openmessaging-benchmark.git
cd openmessaging-benchmark

Step 2: Create Project Files

Copy the following files in the project root directory:
Dockerfile
docker-compose.yaml
setup-kafka-config.sh
build-and-start.sh
run-external-benchmarks.sh

Step 3: Make Scripts Executable

chmod +x setup-kafka-config.sh
chmod +x build-and-start.sh
chmod +x run-external-benchmarks.sh

Usage Guide

For Engineers - Quick Demo

5-Minute Customer Demo:

./run-external-benchmarks.sh
# Choose option 1 (Validation)
# Show results in real-time

15-Minute Customer Presentation:

./run-external-benchmarks.sh
# Choose option 6 (Run All Tests)
# Professional comprehensive analysis

For Performance Testing

Custom Testing:

./run-external-benchmarks.sh
# Choose option 7 (Custom Test)
# Configure specific rates and message sizes

Results Analysis:

# Start web viewer for interactive results
docker-compose --profile viewer up -d
# Open http://localhost:8088 in browser

# Or examine JSON results directly
ls -la ./results/
cat ./results/latest-result.json | jq '.workloads[0]'

Benchmark Types

1. Validation (2 minutes)

Purpose: Quick functionality verification
Settings: 5 msgs/sec, 1KB messages, single partition
Use Case: Pre-demo validation, basic connectivity test

2. Throughput (3 minutes)

Purpose: Demonstrate sustained throughput capabilities
Settings: 50 msgs/sec, 1KB messages, 4 partitions
Use Case: Customer demos, capacity planning

3. Latency (3 minutes)

Purpose: Show low-latency performance characteristics
Settings: 10 msgs/sec, 1KB messages, single partition
Use Case: Real-time application discussions

4. Small Message Test (3 minutes)

Purpose: IoT/sensor data simulation
Settings: 20 msgs/sec, 100B messages, single partition
Use Case: IoT customer scenarios

5. Multi-Partition Test (3 minutes)

Purpose: Demonstrate scaling across partitions
Settings: 25 msgs/sec, 1KB messages, 4 partitions
Use Case: High-throughput application design

Configuration Details

Cluster-Friendly Settings

Our configuration is specifically designed to be gentle on Kafka clusters:

Producer Configuration:

acks=1 - Balanced durability and performance
batch.size=8192 - Small batches reduce memory pressure
compression.type=none - No CPU overhead
enable.idempotence=false - Simpler transaction handling
max.in.flight.requests.per.connection=1 - Sequential processing

Consumer Configuration:

enable.auto.commit=true - Simplified offset management
max.poll.records=10 - Small poll batches
max.partition.fetch.bytes=32768 - Controlled fetch sizes

Rate Limiting:

Maximum producer rate: 50 msgs/sec
Test durations: 2-3 minutes
Single broker configuration to reduce coordination overhead

Troubleshooting

Common Issues

1. "Broker may not be available"

# Test connectivity
docker exec kafka-benchmark /usr/local/bin/health-check.sh

# Verify broker IP in configuration
docker exec kafka-benchmark cat /opt/benchmark/configs/kafka.yaml

2. "Topic creation failed"

# Check if topic configurations are supported
# Configurations are intentionally minimal to avoid compatibility issues

3. "Container not running"

# Restart the benchmark environment
docker-compose down
./build-and-start.sh

4. "Out of memory errors"

# Adjust memory settings in docker-compose.yml
# Default: 4GB heap, increase if needed

Performance Tuning

For Higher Throughput (use carefully):

Increase producerRate in workload files
Adjust batch.size and linger.ms in Kafka config
Add more partitions to topics

For Lower Latency:

Reduce linger.ms to 0
Use acks=1 instead of acks=all
Minimize max.poll.records

For Cluster Stability:

Keep producer rates under 100 msgs/sec
Use single broker configuration
Monitor cluster health during tests

Results Interpretation

Key Metrics

Throughput Metrics:

publishRate - Messages published per second
consumeRate - Messages consumed per second
backlog - Accumulated message backlog

Latency Metrics:

aggregatedEndToEndLatency.p50 - Median latency
aggregatedEndToEndLatency.p95 - 95th percentile latency
aggregatedEndToEndLatency.p99_9 - 99.9th percentile latency

Example Results Analysis:

{
  "workload": "validation",
  "publishRate": 4.98,
  "aggregatedEndToEndLatency": {
    "p50": 2.1,
    "p95": 8.3,
    "p99_9": 15.7
  }
}

Customer Presentation Tips

For Demos:

Start with Validation to show basic functionality.
Run Throughput to demonstrate sustained performance.
Use Latency for real-time application discussions.
Emphasize the cluster-friendly nature of the tests.
Show the professional web interface for viewing results.

Key Talking Points:

Low-latency capabilities (sub-10ms p95).
Sustained throughput with minimal resource usage.
Partition scalability demonstration.
Compatibility with standard Kafka tooling.

Maintenance

Regular Updates

Update Docker Images:

docker-compose down
docker-compose build --no-cache
docker-compose up -d

Update Workload Configurations:

Modify files in /workloads/ directory.
Adjust producer rates based on cluster capacity.
Update test durations as needed.

Clean Up Results:

# Archive old results
tar -czf kafka-benchmark-results-$(date +%Y%m%d).tar.gz results/
rm -f results/*.json

Security Considerations

The benchmark uses network_mode: host for simplicity.
No authentication is configured by default.
Ensure proper network isolation in production environments.
Consider using TLS/SSL for secure Kafka connections.

Support

For questions or issues with this benchmark setup:

Check the troubleshooting section above.
Review Docker and container logs.
Consult the OpenMessaging Benchmark documentation.
Contact the VAST Support (support@vastdata.com) team for Kafka cluster-specific issues.

Version: 1.0
Tested With: OpenMessaging Benchmark 0.0.1-SNAPSHOT, Kafka 3.6.1

Documentation Index