# Kinesis or Kafka on AWS?

> I need a real stream on AWS. Kinesis Data Streams, or Kafka on Amazon MSK, and is running Kafka myself ever worth it? Kinesis by default on AWS: no brokers, IAM on every call, Lambda in batches, and the cheapest stream up to tens of MB a second with a few consumers. Kafka on MSK when Kafka itself is the requirement (compacted topics, transactions across topics, Connect and Debezium, hybrid) or when many consumers and long history make every Kinesis consumer's per-GB read the biggest cost. Running Kafka yourself on EC2 costs more than MSK once replication between AZs is counted.

- A choice between tools from The Stack by Nicolas Dao (Cloudless Labs, Sydney): when to pick each, and when to run both
- Last reviewed: 1 November 2026, by Nicolas Dao
- Web page: https://cloudlesslabs.com/stack/guides/kinesis-or-kafka/. The whole stack in one file: https://cloudlesslabs.com/stack/brief.md
- Tools: [Amazon Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md) (Use when it fits), [Amazon MSK (managed Kafka)](https://cloudlesslabs.com/stack/aws-msk/index.md) (Use when it fits), [Kafka on EC2, run yourself](https://cloudlesslabs.com/stack/kafka-on-ec2/index.md) (Skip), [Amazon Data Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md) (My default), [AWS Lambda](https://cloudlesslabs.com/stack/aws-lambda/index.md) (My default)

## When to pick which

Pick Kinesis when:

- No requirement for Kafka itself, and the consumers are Lambda, Firehose or Flink.
- A few consumers, history up to 365 days, and consumers that ignore duplicates (delivery is at least once).
- You want no brokers, no VPC plumbing and IAM on every call.

Pick Amazon MSK when:

- Compacted topics, transactions across topics, or Kafka Connect, Debezium or Kafka Streams.
- Many consumer groups on the same data, or history kept longer than a year.
- Hybrid or multi-cloud, or a team that already runs Kafka.

Pick Kafka on EC2 when:

- Only a Kafka version, plugin or broker setting MSK doesn't offer, with a team that has run Kafka through upgrades and failures.

Both, when:

- Not two streams in one system: pick one. Across a portfolio, yes: Kinesis for the AWS-native systems with a few consumers, MSK where Kafka's ecosystem is the point.
- A move from Kinesis to MSK runs both for a while: producers write to both, and consumers move one at a time.

Questions that change the answer:

- Do you need compacted topics or transactions across topics? Then it's Kafka.
- How many consumers read each event, and how long is history kept? Many and long favour Kafka; few and short favour Kinesis.
- Is there an existing Kafka estate, a Connect pipeline or Debezium? Then MSK.
- Can every consumer sit in the VPC? MSK's brokers are private; Kinesis is reached anywhere over IAM.

## The short version

| | Kinesis Data Streams | Kafka (on Amazon MSK) |
|---|---|---|
| My verdict | Use when it fits: my stream by default on AWS | Use when it fits: when Kafka is the requirement |
| Unit of scale | A shard: 1 MB/s or 1,000 records a second in, 2 MB/s out | A partition, bounded by the brokers' hardware |
| Order | Per shard, by partition key | Per partition, by key |
| Longest history | 365 days | Unlimited, with tiered storage or compaction |
| Compacted topics (latest value per key) | No | Yes |
| Delivery | At least once: consumers remove duplicates | At least once, or exactly once inside Kafka with transactions |
| Consumers | Shared reads (2 MB/s a shard, five reads a second), or up to 20 enhanced fan-out consumers a stream | Any number of consumer groups, at almost no extra cost |
| Progress tracking | Your consumers' checkpoints (KCL keeps them in a DynamoDB table) | Committed offsets, kept by the brokers |
| Biggest record | 1 MiB by default, 10 MiB when switched on | About 1 MB by default, configurable |
| Access | IAM on every call, from anywhere | Private brokers in a VPC; IAM, TLS or SASL |
| Portability | AWS only | The Kafka protocol: MSK, Confluent, Redpanda, Azure Event Hubs, Google Cloud, on-premises |
| What you run | Nothing | Brokers, sized and upgraded (by AWS on MSK) |
| Minimum per month, Sydney | About $13 (one provisioned shard) | About $558 (three brokers) |

Sources, checked on 1 November 2026: [Kinesis quotas](https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html), [Apache Kafka design](https://kafka.apache.org/documentation/#design), [Amazon MSK pricing](https://aws.amazon.com/msk/pricing/), [Kinesis Data Streams pricing](https://aws.amazon.com/kinesis/data-streams/pricing/).

## What Kafka has that Kinesis doesn't

Throughput isn't the difference: both carry hundreds of MB a second. Three features are.

- **Compacted topics.** Kafka can keep the latest value for every key for good, and delete a key with a tombstone. That's how a topic becomes the system of record for an entity's current state, rebuilds a cache, or feeds event sourcing. Kinesis only expires records by age.
- **Transactions across topics.** An idempotent producer never duplicates on retry, and a transaction writes to several topics and commits the consumer's offsets atomically: exactly-once processing inside Kafka. Kinesis is at least once; every consumer must ignore what it has already processed.
- **Consumers for free.** Each consumer group keeps its own offsets on the brokers, so the tenth consumer costs what the second did. On Kinesis, each consumer pays per GB read, and enhanced fan-out stops at 20 consumers a stream.

And Kinesis has what Kafka doesn't: nothing to run, IAM on every call from anywhere (a Lambda needs no VPC), on-demand scaling with no capacity planning, batches straight into Lambda, and, since August 2026, delivery straight into S3 and Iceberg tables without Firehose.

## The ecosystem, and how much of it reaches Kinesis

| Kafka's ecosystem | On AWS without Kafka |
|---|---|
| Kafka Connect and its connectors | EventBridge Pipes (Kinesis, SQS, DynamoDB streams as sources), Firehose for S3 and Redshift |
| Debezium change data capture (Postgres, MySQL, SQL Server, Oracle and more) | Debezium Server writes to Kinesis directly; AWS DMS replicates a database into Kinesis |
| Kafka Streams, ksqlDB | Managed Service for Apache Flink, which reads Kinesis |
| Schema registries | AWS Glue Schema Registry (Kinesis support from Java only) |
| Compacted topics | No equivalent |
| The protocol everyone speaks | No equivalent: Kinesis is AWS only |

Sources, checked on 1 November 2026: [Debezium connectors](https://debezium.io/documentation/reference/stable/connectors/index.html), [Debezium Server](https://debezium.io/documentation/reference/operations/debezium-server.html), [DMS with Kinesis as a target](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Kinesis.html).

So the ecosystem argues for Kafka when compaction or the protocol is the point, or when the team already has Connect pipelines. A single change-data-capture feed into an AWS system doesn't need Kafka.

## Kafka on AWS: three ways to buy it, and one I'd skip

| | MSK, Standard brokers | MSK, Express brokers | MSK Serverless | Kafka on EC2, run yourself |
|---|---|---|---|---|
| My verdict | Use when it fits | Use when it fits | Use when it fits, small only | Skip |
| You size and run | Brokers and disks; AWS patches | Brokers only | Nothing | Everything |
| Minimum per month, Sydney | About $558 | About $1,117 | About $684 | About $388, plus your team |
| Storage | Three copies on broker disks, or tiered storage (one copy) | Pay-per-use, one copy | Pay-per-use | Three copies on your disks |
| Replication between AZs | Free | Free | Free | $0.02 per GB, twice per GB written |
| Pick it when | The default: cheapest at steady volume | You'd rather pay more than manage disks and scaling | A small, irregular need that must speak Kafka | A version, plugin or setting MSK doesn't offer |

The full entries: [Amazon MSK](https://cloudlesslabs.com/stack/aws-msk/index.md) and [Kafka on EC2](https://cloudlesslabs.com/stack/kafka-on-ec2/index.md).

## Priced on four workloads

Sydney, USD a month, 1 KB events, three AZs. The four workloads: 10 KB/s with one consumer and 24 hours kept; 1 MB/s with two consumers, 24 hours; 10 MB/s with three consumers, seven days; 100 MB/s with five consumers, seven days.

| Option | 10 KB/s | 1 MB/s | 10 MB/s | 100 MB/s |
|---|---|---|---|---|
| Kinesis, provisioned | about 14 | about 78 | about 1,146 | about 24,570 |
| Kinesis, on-demand | about 39 | about 557 | about 7,371 | about 103,963 |
| MSK, Standard brokers | about 566 | about 643 | about 1,753 | about 16,379 |
| MSK, Express brokers | about 1,118 | about 1,195 | about 2,522 | about 22,982 |
| MSK Serverless | about 694 | about 1,360 | about 9,639 | over one cluster's limit |
| Kafka on EC2, run yourself | about 390 | about 559 | about 4,272 | about 21,163 |

Sources, checked on 1 November 2026: the AWS Price List for Sydney ([Kinesis](https://aws.amazon.com/kinesis/data-streams/pricing/), [MSK](https://aws.amazon.com/msk/pricing/), [EC2](https://aws.amazon.com/ec2/pricing/on-demand/)); the arithmetic and sizing are mine, in the site's research notes. MSK and EC2 assume fetch-from-follower; MSK keeps history beyond 24 hours in tiered storage, and the EC2 cluster uses an S3 tier at 100 MB/s. The EC2 figures leave out the people who run it.

What the table says:

- **Up to 10 MB/s with a few consumers, Kinesis provisioned is the cheapest stream by far.** Its minimum is a fraction of any Kafka cluster's.
- **At 100 MB/s with five consumers, MSK is cheaper.** Kinesis's enhanced fan-out alone costs about $13,000 a month there: every consumer pays per GB. Batching small records with the Kinesis producer library takes about $4,900 off Kinesis's bill, which doesn't close the gap.
- **On-demand Kinesis is the expensive way to run a big stream.** It's the right start while the volume is unknown, and the wrong place to stay.
- **Kafka on EC2 never wins once volume is real,** because it pays for replication between AZs that MSK doesn't charge.

Where exactly the lines cross depends on the number of consumers and the history kept, more than on throughput. Both are drawn in [Queue, bus or stream?](https://cloudlesslabs.com/stack/guides/queue-or-stream/index.md)

## Other managed Kafka on AWS

I haven't used these, so they get no verdict here. They're worth knowing because they run on AWS and some keep the data in the client's account.

| Service | Where it runs | In Sydney | Worth a look when |
|---|---|---|---|
| Confluent Cloud | Confluent's AWS accounts; PrivateLink on the Enterprise and Dedicated tiers | Yes (Basic, Standard, Enterprise, Dedicated) | Managed connectors, Flink and governance in one product; the only one here with an IRAP PROTECTED assessment |
| WarpStream (Confluent) | Stateless agents in your own AWS account, data in your S3 | Control plane in Sydney | Very high volume where cost matters more than latency (about 0.5–1 s end to end) |
| Redpanda | Its cloud, or in your account (BYOC) | BYOC and Dedicated | A Kafka-compatible engine without the JVM |

IBM closed its purchase of Confluent, WarpStream included, on 17 March 2026. Sources, checked on 1 November 2026: [Confluent Cloud pricing](https://www.confluent.io/confluent-cloud/pricing/), [WarpStream pricing](https://www.warpstream.com/pricing), [Redpanda pricing](https://www.redpanda.com/pricing).

## Who moved, and why

Two public migrations went from Kinesis to Kafka on MSK. [Atlassian](https://www.atlassian.com/blog/how-we-build/scaling-streamhub-transitioning-from-kinesis-to-kafka-for-145-billion-daily-events) moved about 145 billion events a day in 2026: Kinesis's cost grew in step with traffic, extended retention and fan-out were expensive, and reliability had plateaued. Its own lessons are frank about the other side: "Managed does not mean unlimited", storage can't shrink, and rebalancing is slow. [Kong](https://konghq.com/blog/engineering/from-kinesis-to-kafka) moved in 2023 because Kinesis can't write a transaction across several topics.

Both had what this guide calls the Kafka cases: huge volume with many consumers, and transactions across topics. Below that, Kinesis is less to run and costs less.

## Related

- [Amazon Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md): the full entry (verdict, pricing, gotchas)
- [Amazon MSK (managed Kafka)](https://cloudlesslabs.com/stack/aws-msk/index.md): the full entry (verdict, pricing, gotchas)
- [Kafka on EC2, run yourself](https://cloudlesslabs.com/stack/kafka-on-ec2/index.md): the full entry (verdict, pricing, gotchas)
- [Amazon Data Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md): the full entry (verdict, pricing, gotchas)
- [AWS Lambda](https://cloudlesslabs.com/stack/aws-lambda/index.md): the full entry (verdict, pricing, gotchas)
- [Messaging or streaming?](https://cloudlesslabs.com/concepts/messaging-or-streaming/index.md): the concept behind it
- [Every decision guide, the choices between tools first](https://cloudlesslabs.com/stack/guides/index.md)
- [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)