[AWS - DA] Messaging - Kinesis

Overview

Different from SQS, once data into Kinesis, it cannot be deleted. SQS will be deleted after processed
Kinesis is regional

Kinesis Producers

Data record consists of:
- Sequence number (unique per partition key within shard)
- Partition key 🔑
- Data Blocl (up to 1 MB)
Write throughput: 1 MB/sec or 1000 records/sec per shard
API: PutRecord

The same idea as DynamoDB for hot key issue.

There are three ways to handle peak time:

1. Increase shards, nothing wrong with that, just pay more

2. Exponential Backoff

3. Using highly distributed partiton key

Uing SQS to buffer is not a good idea, Kinesis is designed for real time processing... what's the point if you add buffer? Kinesis doesn't polling the queue as well. So this is not a valid option.

Consumers

Two modes:

Shared (Classic) Fan-out Consumer:
- pull mode
- number of Consumers will affect the speed
- API: GetRecords
Enhanced Fan-out Consumer
- push mode
- each consumer is 2MB/Sec, number of consumer will not affect speed
- API: SubscribeToShard