Real-time systems fail in the gaps between services. A durable event log closes those gaps.
Why Kafka, and not another message queue
Traditional brokers such as RabbitMQ or IBM MQ delete a message once it is read. Apache Kafka keeps events for as long as you configure, as an ordered, durable log. That one difference changes what you can build:
- Several teams can read the same stream independently, at their own pace.
- A new service can replay history instead of waiting for new events.
- Replication across brokers means a server can fail without losing data.
Amazon MSK: Kafka without the undifferentiated work
Amazon MSK runs open-source Kafka as a managed service. Standard Kafka tools and clients keep working, and AWS takes on node provisioning and operating-system patching. You pay for infrastructure, with no proprietary licence on top.
It also fits naturally into the rest of AWS: IAM for access, KMS for encryption, PrivateLink for private connectivity, and Lambda, S3, Glue and DynamoDB as producers and consumers. Typical uses include IoT ingestion, event-driven microservices, and feeding data lakes and analytics.
Red Hat OpenShift: one platform for the services around the stream
The producers and consumers around Kafka need a home too. OpenShift is enterprise Kubernetes with CI/CD and GitOps built in, security and compliance controls, automated updates, and the same operating model across cloud, on-premises and bare metal.
How we put it together
- Design for replay. Retention, partitioning and keys are chosen so any consumer can rebuild its state.
- Make every write idempotent. Duplicate events are expected in distributed systems; downstream stores must not care.
- Catch what fails. Dead-letter topics and alerts, so a bad message never silently blocks a partition.
- Keep it private. Brokers inside private networks, IAM-based access and encryption in transit and at rest.