Streaming & Messaging - Hikube

Kafka, RabbitMQ, NATS.
kubectl apply - that's it.

Three fully managed messaging services on Hikube. Distributed event streaming, AMQP task queues, ultra-light pub/sub. Automatic replication, coordinated ZooKeeper, JetStream included.

TRY FOR FREE DOCUMENTATION

3 services

Kafka - RabbitMQ - NATS

Native HA

Automatic replication + quorum

<1 ms

NATS latency (same datacenter)

0 config

ZooKeeper, Sentinel, Raft - managed
Comparison

The right service for every pattern

Kafka, RabbitMQ and NATS meet distinct needs. The choice depends on volume, delivery model and durability.

SERVICE
PROTOCOL
MODEL
PERSISTENCE
USE CASES
Kafka
Kafka
TCP binary
Pub/Sub with distributed log
Yes - configurable retention
Event streaming, data pipelines, real-time analytics
RabbitMQ
RabbitMQ
AMQP
Queue with routing
Yes - quorum queues (Raft)
Task queues, RPC, complex routing, asynchronous workflows
Frame 11502-2
NATS
TCP text
Pub/Sub + Request/Reply
Optional - JetStream
Microservices, IoT, edge, lightweight real-time communication
SERVICES

Each service in detail

Kafka

Strimzi Operator ZooKeeper Distributed log

Kafka organises messages into topics split into partitions, distributed across several brokers. Each partition is replicated according to the replicationFactor. If a broker fails, partitions are redirected to the available replicas. ZooKeeper coordinates leader election and stores cluster metadata.

Unlike RabbitMQ, Kafka keeps messages after they are consumed. Retention is configurable per topic (retention.ms). Several consumer groups can read the same topic independently, each keeping its own offset.

  • Topics declared in the manifest
    Partitions, replication factor, retention and cleanup policy configured directly in YAML. No need for external tools to create topics.
  • Managed ZooKeeper
    3 ZooKeeper instances coordinate the cluster automatically. The odd quorum is guaranteed — a ZK node failure does not interrupt the service.
  • Retention and compaction
    cleanup.policy: delete deletes messages after retention.ms. cleanup.policy: compact keeps only the last message per key, ideal for reference tables.
  • Optional external exposure
    external: true creates one LoadBalancer per broker for off-cluster clients. Avoid without authentication configured.
yaml
kind: Kafka 
  metadata:
name: example
spec:
  external: false
  kafka:
    replicas: 3
    resourcesPreset: small
    size: 10Gi
    storageClass: replicated
  zookeeper:
    replicas: 3 # Toujours impair
    resourcesPreset: small
    size: 5Gi
    storageClass: replicated
  topics:
    - name: my-topic
      partitions: 3
      replicas: 3
      config:
        retention.ms: "604800000" # 7 jours
        cleanup.policy: "delete" 


 
Why managed

What you no longer have to do

Kafka, RabbitMQ and NATS are three different distributed systems, but they share the same self-hosted problem: each requires dedicated expertise to run correctly in production. Sizing nodes, configuring quorum, testing failover, managing version upgrades without interrupting the message flow is engineering time that produces no business value.

On Hikube, the Kubernetes operator absorbs this load for all three services. You declare what you want - number of nodes, topics or vhosts, resources - and the operator provisions, replicates and falls over when something breaks. Changing the configuration is just changing a value in a YAML file and applying it.

markup-cropped-1

Automatic quorum and consensus

ZooKeeper for Kafka, Raft for RabbitMQ and NATS JetStream. Leader-election mechanisms are configured and monitored without manual intervention.

markup-cropped-1

Topics, vhosts and streams with the cluster

Partitions, retention, virtual hosts, JetStream streams declared in YAML in the same place. No separate initialisation script to maintain.

markup-cropped-1

Scaling without interruption

Change the number of nodes in the manifest and apply. The operator adds the instances as a rolling update with no manual rebalancing and no downtime.

markup-cropped-1

Auto-generated credentials

Passwords are created in a Kubernetes Secret at provisioning. Reference the Secret in your pods, no plaintext keys in your manifests or Git repositories.

Use cases

Which service for which need?

The three services don't compete, they complement each other. Here are the concrete patterns in where each one stands out.

Application log pipeline

Ingest millions of log events from 50 microservices. 7-day retention, several independent consumers (Elasticsearch, ClickHouse, alerting). Each consumer reads from its own offset.

3 brokers - 3 partitions - retention.ms: 604800000

Event sourcing and audit trail

Each user action is published on a topic with cleanup.policy: compact. The current state is the sum of the events. The compact topic guarantees retention of the last state per key.

cleanup.policy: compact - replicas: 3 - min.insync: 2

Email processing queue

Orders trigger a message in an emails.pending queue. Workers read the queue, send the email, and ACK the message. If a worker drops, the message is re-queued automatically.

replicas: 3 - quorum queues - ACK/NACK

Multi-destination event routing

A topic exchange routes orders.* events to the billing queue, orders.shipped events to the logistics queue, and # to an audit queue. One publish, many destinations.

exchange: topic - bindings - 3 tails

Inter-microservice communication

Service A calls service B via NATS request/reply rather than HTTP. No service discovery, no load balancer. NATS routes directly to an available subscriber in the queue group.

Request/Reply - queue groups - <1ms latency

IoT telemetry with JetStream

10,000 sensors publish metrics on sensors.>. JetStream stores 24 hours of data. Analysis services connect and replay history at startup. Useful after maintenance.

JetStream - 24h retention - replay on connect
Managed Streaming & Messaging

Event-driven, the foundation of modern architectures

Event-driven architectures have become the standard for modern distributed systems. Where synchronous APIs create fragile couplings, message brokers let services communicate without direct dependency. Each service produces or consumes events at its own pace.

Kafka structures high-volume data pipelines - logs, metrics, business events - with long retention and event replay to simplify auditing and disaster recovery. RabbitMQ ensures reliable routing of critical tasks: each message is processed exactly once, with guaranteed delivery. NATS reduces friction between microservices thanks to lightweight communication, microsecond latency and no heavy infrastructure.

In the managed version, your teams adopt these patterns without starting from scratch on the infrastructure. Same technology, same SDKs, same tools, deployed and operated by Hikube.

markup-cropped-1

Decoupling and application resilience

One failing service doesn't drag down the others. Messages accumulate in the broker and are processed on recovery, with no loss or manual intervention.

markup-cropped-1

IoT and edge computing

NATS ingests millions of sensor events with a minimal memory footprint. JetStream ensures data is not lost if a processing service restarts.

markup-cropped-1

Data streaming and real-time analytics

Feed ClickHouse, Elasticsearch or your Grafana dashboards continuously from Kafka. The same topic can feed several independent consumers simultaneously.

markup-cropped-1

Compatible with your stack

Kafka Connect, Confluent SDKs, pika, nats.go, nats.py. All existing clients work without modification. Point to the Kubernetes service, the rest is the same.

FAQ

Question about Managed Streaming & Messaging

Kafka or RabbitMQ, how to choose?

Kafka is made for high-volume streaming and long retention. Messages stay available after consumption, and several consumers can read the same topic independently. It's the natural choice for data pipelines, event sourcing and real-time analytics.

RabbitMQ is made for task queues and complex routing. Messages are consumed and deleted, with guaranteed acknowledgement. It's the natural choice for background processing, asynchronous workflows and systems where every message must be processed exactly once.

When should you use NATS rather than Kafka or RabbitMQ?

NATS is the right choice when you need light, fast communication between microservices, without the complexity of a heavyweight broker. Its sub-millisecond latency and minimal memory footprint (<10 MB per instance) make it the best choice for edge architectures, IoT, and request/reply communications between services.

If you need long-lasting persistence and large-scale event replay, Kafka is still more suitable. If you need complex routing with guaranteed ACK, RabbitMQ is more appropriate.

Why does ZooKeeper require an odd number of replicas?

ZooKeeper uses a quorum-based consensus algorithm. A decision (leader election, writing) is validated when a majority of nodes approve it. With 3 nodes, quorum is 2: losing one node does not interrupt service. With 4 nodes, quorum is also 2: you still only tolerate the loss of a single node. Going from 3 to 4 does not improve fault tolerance, but increases costs. That's why we recommend 3 or 5, never 4.

Is NATS JetStream enabled by default?

Yes, JetStream is enabled by default on Hikube NATS clusters. Disable it only if you need a purely ephemeral pub/sub without persistence: you'll reduce the memory footprint and eliminate the need for a persistent volume. In production, keep JetStream activated to benefit from persistence, replay and durable consumer groups.

Note: even with JetStream enabled, messages are only persisted if a stream is configured to capture the relevant subjects.

What's the difference between partitions and replicationFactor in Kafka?

Partitions define parallelism: more partitions let more consumers read in parallel. Each partition is an ordered sequence of messages on a broker.

The replicationFactor defines durability: each partition is copied to this number of brokers. If a broker goes down, its partitions are served by the replicas. The replicationFactor cannot exceed the number of brokers.

Can I expose my cluster outside the Kubernetes cluster?

Yes, for Kafka and RabbitMQ via external: true, and for NATS via external: true. Kafka creates a LoadBalancer per broker, RabbitMQ exposes the AMQP (5672) and Management (15672) ports, NATS exposes the client port (4222).

External exposure makes the service reachable over the Internet: make sure you use strong passwords. Kafka has no default authentication on Hikube, so we strongly advise against exposing it without additional security configuration.