Kafka vs RabbitMQ for IoT: Choosing the Right Message Queue for High-Volume Telemetry

Kafka vs RabbitMQ for IoT: Choosing the Right Message Queue for High-Volume Telemetry

MQTT is the right protocol between your devices and your cloud entry point. But once that data arrives at scale — 10,000 devices publishing every 5 seconds — you need something between the MQTT broker and your application services. That's where Kafka and RabbitMQ come in.

Both are battle-tested message queues used by companies processing billions of events daily. But they solve different problems, and choosing the wrong one will either over-engineer your system or leave you unable to scale.

The IoT Data Ingestion Problem

At 10,000 devices publishing telemetry every 5 seconds, you're handling 2,000 messages per second continuously. Your data pipeline must:

Neither a raw database insert nor a simple REST API handles all four requirements at scale.

Apache Kafka: Built for Throughput

Kafka is a distributed commit log. Messages are written to partitioned topics and retained for a configurable period (days, weeks, forever). Consumers read at their own pace using offsets.

Architecture for IoT

IoT Devices → MQTT Broker (AWS IoT Core / Mosquitto)
                    ↓ MQTT Bridge / Kafka Connect
              Kafka Topic: iot.telemetry
                    ↓                        ↓
         InfluxDB Consumer          Alerts Engine Consumer
         (Time-series storage)      (Threshold detection)
                                          ↓
                                    ML Pipeline Consumer
                                    (Anomaly detection)

Kafka MQTT Bridge

Use Kafka Connect with the MQTT Source Connector:

{
  "name": "mqtt-source-connector",
  "config": {
    "connector.class": "io.confluent.connect.mqtt.MqttSourceConnector",
    "mqtt.server.uri": "ssl://your-broker:8883",
    "mqtt.topics": "devices/+/telemetry",
    "kafka.topic": "iot.telemetry",
    "mqtt.qos": "1",
    "transforms": "ExtractDeviceId",
    "transforms.ExtractDeviceId.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
    "transforms.ExtractDeviceId.field": "deviceId"
  }
}

Kafka Consumer in Node.js

import { Kafka } from 'kafkajs'
const kafka = new Kafka({ brokers: ['kafka:9092'] })
const consumer = kafka.consumer({ groupId: 'influxdb-writer' })
await consumer.subscribe({ topic: 'iot.telemetry', fromBeginning: false })
await consumer.run({
  eachBatch: async ({ batch }) => {
    const points = batch.messages.map(msg => {
      const data = JSON.parse(msg.value.toString())
      return new Point('telemetry')
        .tag('device_id', data.deviceId)
        .floatField('temperature', data.temperature)
        .floatField('humidity', data.humidity)
        .timestamp(new Date(data.ts))
    })
    await influxWriteApi.writePoints(points)
  },
})

Batching is critical — writing 500 points at once is 100× faster than 500 individual writes.

Kafka's strengths for IoT

| Feature | Kafka | |---------|-------| | Throughput | 1M+ messages/sec per broker | | Message retention | Days to forever (configurable) | | Replay | Yes — rewind consumer offset to any point | | Fan-out | Multiple consumer groups, each reads independently | | Ordering | Per partition (use device ID as partition key) | | Latency | ~5–15ms (tunable) |

When Kafka is overkill

RabbitMQ: Flexible Routing, Lower Complexity

RabbitMQ is a traditional message broker built on AMQP. Messages are pushed to queues via exchanges with flexible routing rules. Once consumed and acknowledged, messages are gone.

Architecture for IoT

MQTT Broker → RabbitMQ (MQTT Plugin)
                    ↓
             Exchange: iot.telemetry (topic exchange)
            /          |           \
    Queue:           Queue:         Queue:
  influxdb-writer  alert-engine   ml-pipeline

RabbitMQ MQTT Plugin

RabbitMQ ships with a built-in MQTT plugin — your devices connect directly:

rabbitmq-plugins enable rabbitmq_mqtt

# rabbitmq.conf
mqtt.listeners.tcp.default = 1883
mqtt.listeners.ssl.default = 8883
mqtt.ssl_cert_login       = true  # Use device cert as username
mqtt.default_user         = guest
mqtt.default_pass         = guest
mqtt.vhost                = iot
mqtt.exchange             = amq.topic
mqtt.subscription_ttl     = 1800000
mqtt.prefetch             = 10

Topic Exchange Routing

// Node.js consumer
import amqplib from 'amqplib'
const conn    = await amqplib.connect('amqps://rabbitmq:5671')
const channel = await conn.createChannel()
await channel.assertExchange('iot.telemetry', 'topic', { durable: true })
const q = await channel.assertQueue('influxdb-writer', { durable: true })
await channel.bindQueue(q.queue, 'iot.telemetry', 'devices.*.telemetry')
channel.consume(q.queue, async (msg) => {
  if (!msg) return
  const data = JSON.parse(msg.content.toString())
  await writeToInflux(data)
  channel.ack(msg)
})

RabbitMQ's strengths for IoT

| Feature | RabbitMQ | |---------|----------| | Throughput | ~100K messages/sec (clustered) | | Routing | Flexible topic/fanout/header exchanges | | Message TTL | Configurable per queue | | Replay | Not natively (use Shovel for DLX) | | Setup complexity | Low — single binary, great UI | | MQTT native | Yes — built-in plugin |

Head-to-Head: Which to Choose

Choose Kafka when:

Choose RabbitMQ when:

The hybrid approach (common in production)

Many production IoT systems use both:

IoT Devices → MQTT Broker
                   ↓
              RabbitMQ        ← command routing, small-scale telemetry
                   ↓
               Kafka          ← high-volume telemetry ingestion
                   ↓
        InfluxDB / S3 / Redshift

MQTT → RabbitMQ for device commands and low-volume sensor data. RabbitMQ → Kafka for high-volume telemetry that needs replay and long-term retention.

Performance Benchmarks

Internal tests on AWS (3-node Kafka cluster vs 3-node RabbitMQ cluster, c5.2xlarge):

| Scenario | Kafka | RabbitMQ | |----------|-------|----------| | Peak throughput | 1.2M msg/sec | 180K msg/sec | | p99 latency @ 50K/sec | 8ms | 12ms | | p99 latency @ 200K/sec | 15ms | 45ms | | Consumer group fan-out (5 consumers) | No overhead | 5× broker load | | Replay 24h of data | Instant | Not supported |

For most IoT projects below 50,000 msg/sec, RabbitMQ is sufficient and simpler to operate. Kafka becomes necessary at higher volumes or when replay is non-negotiable.

Operational Considerations

Kafka requires:

RabbitMQ requires:

Both are operationally manageable on AWS MSK (managed Kafka) or CloudAMQP (managed RabbitMQ) if you want to avoid running them yourself.

Conclusion

The decision framework: if you're under 100K msg/sec and don't need message replay, RabbitMQ is simpler and entirely adequate. Above 100K msg/sec, or if your architecture requires replay, fan-out to many independent consumers, or long-term message retention, Kafka is the right investment.

For new IoT products at early scale, start with RabbitMQ. Build your abstraction layer cleanly so you can migrate to Kafka as you grow without rewriting your consumers.

Need help designing your IoT data pipeline? [Contact Code Caracal](/contact) — we've built both at scale.