Streaming & Messaging - Hikube

Kafka, RabbitMQ, NATS.
kubectl apply - that's it.

Three fully managed messaging services on Hikube. Distributed event streaming, AMQP task queues, ultra-light pub/sub. Automatic replication, coordinated ZooKeeper, JetStream included.

CONTACT OUR EXPERTS DOCUMENTATION

3 services

Kafka - RabbitMQ - NATS

Native HA

Automatic replication + quorum

>1 ms

NATS latency (same datacenter)

0 config

ZooKeeper, Sentinel, Raft - managed
Comparison

The right service for every pattern

Kafka, RabbitMQ and NATS meet distinct needs. The choice depends on volume, delivery model and durability.

SERVICE
PROTOCOL
MODEL
PERSISTENCE
USE CASES
Kafka
Kafka
TCP binary
Pub/Sub with distributed log
Yes - configurable retention
Event streaming, data pipelines, real-time analytics
RabbitMQ
RabbitMQ
AMQP
Queue with routing
Yes - quorum cues (Raft)
Task queues, RPC, complex routing, asynchronous workflows
Frame 11502-2
NATS
TCP text
Pub/Sub + Request/Reply
Optional - JetStream
Microservices, IoT, edge, real-time light communication
SERVICES

Each service in detail

Kafka

Strimzi Operator ZooKeeper Log distributed

Kafka organizes messages into topics divided into partitions, distributed across several brokers. Each partition is replicated according to the replicationFactor. In the event of a broker failure, partitions are redirected to available replicas. ZooKeeper coordinates the election of leaders and stores cluster metadata.

Unlike RabbitMQ, Kafka retains messages after they have been consumed. The retention time is configurable per topic (retention.ms). Several consumer groups can read the same topic independently, each maintaining its own offset.

  • Topics declared in the manifest
    Partitions, replication factor, retention and cleanup policy configured directly in YAML. No need for external tools to create topics.
  • Managed ZooKeeper
    3 ZooKeeper instances coordinate the cluster automatically. Odd quorum is guaranteed - a ZK node failure does not interrupt service.
  • Retention and compaction
    cleanup.policy: delete deletes messages after retention.ms. cleanup.policy: compact retains only the last message per key, ideal for reference tables.
  • Optional external exposure
    external: true creates one LoadBalancer per broker for off-cluster clients. Avoid without authentication configured.
yaml
apiVersion: apps.cozystack.io/v1alpha1
kind: Kafka 
  metadata:
name: example
spec:
  external: false
  kafka:
    replicas: 3
    resourcesPreset: small
    size: 10Gi
    storageClass: replicated
  zookeeper:
    replicas: 3 # Toujours impair
    resourcesPreset: small
    size: 5Gi
    storageClass: replicated
topics:
  - name: my-topic
    partitions: 3
    replicas: 3
    config:
      retention.ms: "604800000" # 7 jours
      cleanup policy: "delete" 


 
Why managed

What you don't have to do anymore

Kafka, RabbitMQ and NATS are three different distributed systems, but they share the same self-hosted problem: each requires dedicated expertise to operate correctly in production. Sizing nodes, configuring quorum, testing failover, managing version upgrades without interrupting message flow represents engineering time that doesn't produce business value.

On Hikube, the Kubernetes operator absorbs this load for all three services. You declare what you want - number of nodes, topics or vhosts, resources - and the operator provisions, replicates and switches in the event of failure. Modifying the configuration is as simple as changing a value in a YAML file and applying it.

markup-cropped-1

Automatic quorum and consensus

ZooKeeper for Kafka, Raft for RabbitMQ and NATS JetStream. Leader election mechanisms are configured and monitored without manual intervention.

markup-cropped-1

Topics, vhosts and streams with the cluster

Partitions, retention, virtual hosts, JetStream streams declared in YAML in the same place. No separate initialization script to maintain.

markup-cropped-1

Scaling without interruption

Modify the number of nodes in the manifest and apply. The operator adds rolling update instances without manual rebalancing or downtime.

markup-cropped-1

Self-generated credentials

Passwords are created in a Kubernetes Secret at provisioning. Reference the Secret in your pods, no unencrypted keys in your manifests or Git repositories.

Use cases

Which service for which need?

The three services don't compete with each other, they complement each other. Here are the concrete patterns in which each one comes into its own.

Application log pipeline

Ingestion of millions of log events from 50 microservices. 7-day retention, several independent consumers (Elasticsearch, ClickHouse, alerting). Each consumer reads from its own offset.

3 brokers - 3 partitions - retention.ms: 604800000

Event sourcing and audit trail

Each user action is published on a topic with cleanup.policy: compact. The current state is the sum of all events. The compact topic guarantees preservation of the last state per key.

cleanup.policy: compact - replicas: 3 - min.insync: 2

Email processing queue

Commands trigger a message in an emails.pending queue. Workers read the queue, send the email, and ACK the message. If a worker drops out, the message is re-queued automatically.

replicas: 3 - quorum queues - ACK/NACK

Multi-destination event routing

A topic exchange routes orders.* events to the billing queue, orders.shipped events to the logistics queue, and # events to an audit queue. One publish, many destinations.

exchange: topic - bindings - 3 tails

Inter-microservice communication

Service A calls service B via NATS request/reply rather than HTTP. No service discovery, no load balancer. NATS routes directly to an available subscriber in the queue group.

Request/Reply - queue groups - <1ms latency

Telemetry IoT with JetStream

10,000 sensors publish metrics on sensors.>. JetStream stores 24 hours of data. Analysis services connect and replay history at startup. Useful after maintenance.

JetStream - 24h retention - replay on connect
Managed Streaming & Messaging

Event-driven, the foundation of modern architectures

Event-driven architectures have become the standard for modern distributed systems. Where synchronous APIs create fragile couplings, message brokers enable services to communicate without direct dependency. Each service produces or consumes events at its own pace.

Kafka structures pipelines of high-volume data - logs, metrics, business events - with long retention and event replay to simplify auditing and disaster recovery. RabbitMQ ensures reliable routing of critical tasks: each message is processed exactly once, with guaranteed delivery. NATS reduces friction between microservices thanks to lightweight communication, microsecond latency and no heavy infrastructure.

In the managed version, your teams adopt these patterns without starting from scratch on the infrastructure. Same technology, same SDKs, same tools, deployed and operated by Hikube.

markup-cropped-1

Decoupling and application resilience

One service out of order doesn't drag down the others. Messages accumulate in the broker and are processed on resumption, without loss or manual intervention.

markup-cropped-1

IoT and edge computing

NATS ingests millions of sensor events with a minimal memory footprint. JetStream ensures that data is not lost if a processing service restarts.

markup-cropped-1

Data streaming and real-time analytics

Feed ClickHouse, Elasticsearch or your Grafana dashboards continuously from Kafka. The same topic can feed several independent consumers simultaneously.

markup-cropped-1

Compatible with your stack

Kafka Connect, Confluent SDKs, pika, nats.go, nats.py. All existing clients work without modification. Point to the Kubernetes service, the rest is the same.

FAQ

Question about Managed Streaming & Messaging

Kafka or RabbitMQ, what's the best choice?

Kafka is made for high-volume streaming and long retention - messages remain available after consumption, and several consumers can read the same topic independently. It's the natural choice for data pipelines, event sourcing and real-time analytics.

RabbitMQ is made for task queues and complex routing - messages are consumed and discarded, with guaranteed acknowledgement. It's the natural choice for background processing, asynchronous workflows and systems where every message has to be processed exactly once.

When should you use NATS rather than Kafka or RabbitMQ?

NATS is the right choice when you need light, fast communication between microservices - without the complexity of a heavyweight broker. Its sub-millisecond latency and minimal memory footprint (<10 MB per instance) make it the best choice for edge architectures, IoT, and request/reply communications between services.

If you need long-lasting persistence and large-scale event replay, Kafka is more suitable. If you need complex routing with guaranteed ACK, RabbitMQ is more appropriate.

Why does ZooKeeper require an odd number of replicas?

ZooKeeper uses a quorum-based consensus algorithm - a decision (leader election, writing) is validated when a majority of nodes approve it. With 3 nodes, quorum is 2 - the loss of a node does not interrupt service. With 4 nodes, quorum is also 2 - you always tolerate the loss of a single node. Going from 3 to 4 does not improve fault tolerance, but does increase costs. That's why we recommend 3 or 5, never 4.

Is NATS JetStream enabled by default?

Yes, JetStream is enabled by default on Hikube NATS clusters. Disable it only if you need a purely ephemeral pub/sub without persistence - you'll reduce the memory footprint and eliminate the need for a persistent volume. In production, keep JetStream activated to benefit from persistence, replay and durable consumer groups.

Please note: even with JetStream enabled, messages are only persisted if a stream is configured to capture the subjects concerned.

What's the difference between partitions and replicationFactor in Kafka?

Partitions define parallelism - more partitions allow more consumers to read in parallel. Each partition is an ordered sequence of messages on a broker.

The replicationFactor defines durability - each partition is copied to this number of brokers. If a broker goes down, its partitions are served by replicas. The replicationFactor cannot exceed the number of brokers.

Can I expose my cluster outside the Kubernetes cluster?

Yes, for Kafka and RabbitMQ via external: true, and for NATS via external: true. Kafka creates a LoadBalancer per broker, RabbitMQ exposes the AMQP (5672) and Management (15672) ports, NATS exposes the client port (4222).

External exposure makes the service accessible over the Internet - make sure you use strong passwords. Kafka has no default authentication on Hikube, so we strongly advise against exposing it without additional security configuration.