# Amazon MSK (managed Kafka)

> Apache Kafka, run by AWS inside the client's account. For when Kafka itself is the requirement, or when many consumers read a big stream. From about $560 a month before the first message.

- Verdict: Use when it fits
- Vendor: AWS
- Layer: [APIs, queues and events](https://cloudlesslabs.com/stack/layers/api-events/index.md). Job: Managed Apache Kafka
- Last reviewed: 1 November 2026, by Nicolas Dao (Cloudless Labs, Sydney)
- Web page: https://cloudlesslabs.com/stack/aws-msk/. The whole stack in one file: https://cloudlesslabs.com/stack/brief.md

Kafka is the stream most enterprises already run. Apache says more than 80% of the Fortune 100 use it (the project's own claim), and the reason isn't speed: Kinesis handles hundreds of MB a second too. It's three features and an ecosystem.

- **Compacted topics.** Kafka can keep the latest value for every key for good, which is what event sourcing and Kafka Streams' state rely on. Kinesis only expires records by age.
- **Transactions across topics.** An idempotent producer and transactions give exactly-once processing inside Kafka. Kinesis delivers at least once, and the consumer removes duplicates.
- **Consumer groups for free.** Any number of groups read the same topic at their own pace, and adding one costs almost nothing. On Kinesis, every consumer pays per GB read, and enhanced fan-out stops at 20 consumers a stream.
- **The ecosystem.** Kafka Connect and its connectors, Debezium's change data capture from Postgres, MySQL, SQL Server or Oracle, Kafka Streams and Flink, schema registries, and a protocol that every cloud, every on-premises data centre and many vendors speak.

I haven't run Kafka in production. Everything on this page is research and arithmetic, checked on 1 November 2026, and it's why I start with Kinesis on AWS: for a stream with a few consumers and no Kafka requirement, Kinesis is cheaper and has nothing to run. MSK earns its price when one of the four points above is the requirement, or when many consumers read a big stream for a long time. The numbers, with a chart of where the lines cross, are in [Queue, bus or stream?](https://cloudlesslabs.com/stack/guides/queue-or-stream/index.md) and [Kinesis or Kafka on AWS?](https://cloudlesslabs.com/stack/guides/kinesis-or-kafka/index.md)

### Three ways to buy it

| | Standard brokers | Express brokers | Serverless |
|---|---|---|---|
| What you size | Brokers and their disks | Brokers only; storage is pay-per-use | Nothing |
| Smallest production cluster in Sydney | 3 × kafka.m7g.large, about $558 a month | 3 × express.m7g.large, about $1,117 a month | About $684 a month before any data |
| Storage | $0.12/GB-month on every broker (three copies); tiered storage $0.0652/GB-month (one copy) | $0.12/GB-month (one copy), plus $0.0125 per GB written | $0.12/GB-month, plus $0.125/GB in and $0.0625/GB out |
| Limits to know | 16,384 GiB per broker; storage never shrinks | 3 AZs and M7g only | 200 MB/s in and 400 MB/s out per cluster; IAM only |
| Pick it when | The default: cheapest at steady volume, with tiered storage for history | You'd rather pay more than manage disks and scaling | A small, irregular need that must speak Kafka |

Sources, checked on 1 November 2026: [Amazon MSK pricing](https://aws.amazon.com/msk/pricing/) and the AWS Price List for Sydney, [MSK quotas](https://docs.aws.amazon.com/msk/latest/developerguide/limits.html).

Running Kafka yourself on EC2 isn't a fourth way I'd pick: on AWS it costs more than MSK once replication between AZs is counted, and you own every patch and broken broker ([Kafka on EC2](https://cloudlesslabs.com/stack/kafka-on-ec2/index.md)).

## Use it when

- Kafka itself is the requirement: compacted topics (the latest value per key, kept for good), transactions across topics, or an existing Kafka Connect, Debezium or Kafka Streams setup.
- Many consumers reading the same big stream. Kafka's consumer groups cost nothing extra, where every Kinesis consumer pays per GB read: with ten consumers and a week of history, MSK is the cheapest stream from about 1,500 events a second.
- Hybrid or multi-cloud: the Kafka protocol runs on-premises, on every cloud and inside many vendors' products.
- A team that already knows and runs Kafka, or history kept longer than Kinesis's 365 days.

## Don't use it when

- Small or moderate volumes: the cluster costs about $560 a month before the first message. SNS, SQS or EventBridge cost a few dollars; Kinesis starts at about $14.
- A stream with a few consumers and no Kafka requirement: Kinesis provisioned stays cheaper, even at 100 MB/s with three consumers.
- Consumers that can't sit in the VPC: the brokers are private, so a Lambda reading them needs a VPC connection and a NAT gateway or PrivateLink.

## Pricing

Sydney: 3 Standard brokers (kafka.m7g.large, $0.255 an hour each) about $558 a month, the production minimum. Broker storage $0.12/GB-month, provisioned on every broker (three copies); tiered storage $0.0652/GB-month (one copy). Express brokers $0.51 an hour each, plus $0.0125 per GB written and $0.12/GB-month stored. Serverless $0.9375 per cluster-hour (about $684 a month) plus $0.125/GB in and $0.0625/GB out. Replication between brokers is free; client traffic that crosses an AZ costs $0.01/GB on each side. Checked 1 November 2026.

## Gotchas, with fixes

- **Gotcha:** Client traffic that crosses an AZ is billed, $0.01/GB on each side in Sydney: with three AZs, two reads in three cross by default.
  **Fix:** Turn on fetch-from-follower (the consumer's client.rack set to its AZ), so consumers read the replica in their own AZ. Replication between brokers is already free on MSK.
- **Gotcha:** Broker storage only grows: auto-scaling raises it at most once every six hours, and it can never shrink.
  **Fix:** Start small and keep history in tiered storage (one copy, $0.0652/GB-month), or use Express brokers, whose storage is pay-per-use.
- **Gotcha:** A cluster on a Kafka version past its end of support is upgraded by AWS, at any time and without notice. Kafka 3.7 reached end of support on 1 September 2026.
  **Fix:** Run 3.9 (AWS's recommended version) or 4.x, and plan each upgrade yourself before the end-of-support date.
- **Gotcha:** T3 brokers are cheap (about $127 a month for three) and not for production: AWS says to test them under your load first.
  **Fix:** Treat kafka.m7g.large as the production floor, and T3 as a development cluster.
- **Gotcha:** MSK Serverless caps a cluster at 200 MB/s in and 400 MB/s out, accepts IAM authentication only, and per GB costs several times a provisioned cluster.
  **Fix:** Keep Serverless for small, irregular Kafka needs; anything steady goes on Standard brokers.
- **Gotcha:** A Lambda reading MSK in on-demand mode needs a NAT gateway or PrivateLink endpoints in the VPC (about $57 a month); provisioned mode bills its pollers at $0.231 an hour each (about $169 a month).
  **Fix:** Budget the VPC plumbing with the cluster, and size provisioned pollers on the consumer lag you can accept.

## Provisioning

Official Pulumi package · Official Terraform provider

aws.msk.Cluster (Express brokers: leave out storageInfo), aws.msk.ServerlessCluster and, since February 2026, aws.msk.Topic in @pulumi/aws; the same as aws_msk_cluster and aws_msk_topic in the hashicorp/aws provider. The Kafka provider (@pulumi/kafka) also manages topics and ACLs, but it must reach the brokers inside the VPC.

## Instead, consider

- [Amazon Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md): An ordered stream with a few consumers and no Kafka requirement: no brokers, IAM on every call, from about $14 a month.
- [Amazon SNS](https://cloudlesslabs.com/stack/aws-sns/index.md): Each consumer needs each message once, at moderate volume: fan-out to SQS queues, from $0.
- [Kafka on EC2, run yourself](https://cloudlesslabs.com/stack/kafka-on-ec2/index.md): Only for a Kafka version, plugin or setting MSK doesn't offer, with a team that already runs Kafka.

## Decision guides

- [Queue, bus or stream? SQS, EventBridge, Kinesis or Kafka](https://cloudlesslabs.com/stack/guides/queue-or-stream/index.md): Events move between my services. Are SQS, SNS and EventBridge enough, or do I need a stream like Kinesis or Kafka, and at what scale does the answer change?
- [Kinesis or Kafka on AWS?](https://cloudlesslabs.com/stack/guides/kinesis-or-kafka/index.md): I need a real stream on AWS. Kinesis Data Streams, or Kafka on Amazon MSK, and is running Kafka myself ever worth it?

## The concept behind it

- [Messaging or streaming?](https://cloudlesslabs.com/concepts/messaging-or-streaming/index.md): Should the pipe remember a message after it has been delivered?
- [At least once, and idempotency](https://cloudlesslabs.com/concepts/delivery-guarantees/index.md): Every pipe on AWS can deliver the same message twice. How does the consumer stay correct when it does?

## Where it fits

- [Event-driven plumbing](https://cloudlesslabs.com/stack/use-cases/events/index.md): pick 6. Only when Kafka itself is required (compacted topics, transactions, Connect or Debezium), or many consumers read a big stream for a week or more.

## Sources

- [Amazon MSK pricing](https://aws.amazon.com/msk/pricing/)
- [MSK broker sizes](https://docs.aws.amazon.com/msk/latest/developerguide/broker-instance-sizes.html)
- [MSK Express brokers](https://docs.aws.amazon.com/msk/latest/developerguide/msk-broker-types-express.html)
- [MSK tiered storage](https://docs.aws.amazon.com/msk/latest/developerguide/msk-tiered-storage.html)
- [MSK version support policy](https://docs.aws.amazon.com/msk/latest/developerguide/version-support.html)
- [MSK quotas, Serverless included](https://docs.aws.amazon.com/msk/latest/developerguide/limits.html)
- [Apache Kafka documentation](https://kafka.apache.org/documentation/)

## Related

- [APIs, queues and events](https://cloudlesslabs.com/stack/layers/api-events/index.md): every tool in this layer, with its verdict
- [The whole stack in one file](https://cloudlesslabs.com/stack/brief.md): every use case and tool, for questions that span many
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)