Performance Tuning Guide

This guide provides detailed recommendations for optimizing zae-limiter performance, covering DynamoDB capacity planning, Lambda configuration, and cost optimization strategies.

1. DynamoDB Capacity Planning

Understanding RCU/WCU Costs

Each zae-limiter operation has specific DynamoDB capacity costs. Use this table for capacity planning:

Operation	RCUs	WCUs	Notes
acquire()	1	1	O(1) regardless of limit count (composite bucket items)
acquire() with cascade	2	4	Entity + parent bucket reads and writes (TransactWriteItems, 2 WCU per item)
acquire() speculative success	0	1	Skips read; conditional UpdateItem (issue #315)
acquire() speculative success + cascade (sequential)	0	2	Child then parent speculative UpdateItem
acquire() speculative success + cascade (parallel)	0	2	Concurrent child + parent via entity cache (issue #318)
acquire() speculative fast rejection	0	1	Exhausted bucket; the failed conditional write is charged 1 WCU, its ALL_OLD image is free
acquire() repeat rejection inside the rejection-cache TTL	0	0	No DynamoDB call: the last state seen already shows every shard short (ADR-147, see Rejection Cache)
acquire() speculative fallback (non-cascade)	2.5	2	Failed speculative (1 WCU) + normal path: META and bucket read (1 RCU), the uncached disabled walk over up to 3 config items (1.5 RCU), the write (1 WCU)
acquire() speculative cascade fallback (parent refill helps)	1	3	Child stays consumed; parent-only read of the parent's bucket and META (1 RCU, #684) + single-item write (1 WCU)
acquire() retry (contention)	0	1	ADD-based writes don't require re-read
acquire() with adjustments	0	+1 per entity	Independent writes via write_each() (1 WCU each)
acquire() rollback (on exception)	0	+1 per entity	Independent compensating writes (1 WCU each)
Aggregator bucket refill (per active bucket)	0	1	Proactive refill via Lambda; 1 WCU even if the lock is lost
acquire(limits=None) with config cache miss	+3	0	+3 GetItem operations for config hierarchy
acquire() slow-path disabled walk (per entity)	+1.5	0	ADR-125 gate: uncached 3-key BatchGetItem (entity, entity _default_, resource config). Cascade pays it per entity; the speculative fast path pays 0
check_availability()	1 + shards/4	0	Read-only. GSI3 KEYS_ONLY query + one BatchGetItem over the entity's shards (GHSA-76rv); both answers come from that one snapshot
available() / time_until_available()	1 + shards/4	0	Thin wrappers over check_availability(), so the cost is the same and the two can never disagree
get_limits()	1	0	Query operation
set_limits()	1	N+1	Query + N PutItems
delete_entity()	1 per query page	batched	Queries every page of the entity's PK and of GSI3, then BatchWrite in 25-item chunks, resending unprocessed deletes

O(1) Cost Optimization (v0.7.0)

ADR-114 (Composite Bucket Items) and ADR-115 (ADD-Based Writes) reduced acquire() costs from O(N) to O(1) where N is the number of limits. All limits for an entity+resource are stored in a single DynamoDB item, and ADD operations eliminate the need for read-modify-write cycles on contention retries.

Namespace Overhead

Namespace-prefixed keys (e.g., {ns}/ENTITY#id) add a few bytes per item but have no measurable impact on RCU/WCU costs. All operations in the table above apply identically regardless of namespace.

Capacity Validation

These costs are validated by automated tests. Run uv run pytest tests/benchmark/test_capacity.py -v to verify.

Capacity Estimation Formula

Use these formulas to estimate hourly capacity requirements:

Hourly RCUs = requests/hour × (1 + 1.5 × (1 + cascade_pct) + cascade_pct + config_cache_miss_pct × 3)
Hourly WCUs = requests/hour × (1 + cascade_pct × 3)

The 1.5 × (1 + cascade_pct) term is the ADR-125 disabled walk, which runs once per entity on every slow-path acquire. It is unconditional there — passing limits explicitly short-circuits config resolution but not the disabled gate.

With speculative writes enabled (speculative_writes=True), the steady-state formula changes:

Hourly RCUs = requests/hour × (fallback_pct × (1 + 1.5 × (1 + cascade_pct)) + cascade_pct × 0.5 + config_cache_miss_pct × 3)
Hourly WCUs = requests/hour × (1 + cascade_pct)

The disabled walk is scaled by fallback_pct here: steady-state speculative traffic never reaches it, because the fast path enforces disabled with an attribute_not_exists guard on the bucket item instead (ADR-125).

Where fallback_pct is the fraction of requests that fall back to the slow path (typically <5% for pre-warmed buckets).

O(1) Scaling (v0.7.0+)

Costs no longer scale with the number of limits per request. Composite bucket items store all limits in a single DynamoDB item, so 1 limit and 10 limits cost the same.

With the Lambda aggregator enabled (2 windows: hourly, daily):

Additional WCUs = requests/hour × 2

Example Calculations

Scenario 1: Simple API Rate Limiting

• 10,000 requests/hour
• 2 limits per request (rpm, tpm)
• No cascade, config cached

Calculation:

RCUs = 10,000 × 1.0 = 10,000 RCUs/hour
WCUs = 10,000 × 1.0 = 10,000 WCUs/hour

Limit count doesn't affect cost

With composite bucket items (v0.7.0+), whether you track 1 limit or 10 limits, the DynamoDB cost is identical—all limits are stored in a single item.

Scenario 2: Hierarchical LLM Limiting

• 10,000 requests/hour
• 2 limits per request
• 50% use cascade (API key → project)
• 2% config cache miss rate

Calculation:

RCUs = 10,000 × (1 + 0.5 + 0.02×3) = 10,000 × 1.56 = 15,600 RCUs/hour
WCUs = 10,000 × (1 + 0.5×3) = 10,000 × 2.5 = 25,000 WCUs/hour

Billing Mode Selection

Mode	Best For	Trade-offs
PAY_PER_REQUEST (default)	Variable traffic, new deployments	Higher per-request cost, no planning needed
Provisioned	Steady traffic >100 TPS	Lower cost at scale, requires planning
Provisioned + Reserved	High-volume production	Lowest cost, 1-year commitment

Migration Guidance

Start with PAY_PER_REQUEST. Once traffic patterns stabilize (typically 2-4 weeks), analyze CloudWatch metrics to determine optimal provisioned capacity. Switch when monthly on-demand costs exceed provisioned + 20% buffer.

---

2. Lambda Concurrency Settings

The aggregator Lambda processes DynamoDB Stream events to maintain usage snapshots.

Default Configuration

Setting	Default	Range	Impact
Memory	256 MB	128-3008 MB	Higher = faster, more expensive
Timeout	60 seconds	1-900 seconds	Should be 2× typical duration
Reserved Concurrency	None	1-1000	Limits parallel executions

Memory Tuning

Lambda CPU scales linearly with memory allocation:

Memory	vCPUs	Best For
128 MB	~0.08	Minimal workloads (testing only)
256 MB	~0.15	Most workloads (default)
512 MB	~0.30	High-throughput streams
1024 MB	~0.60	Rarely needed

Guidance based on batch size:

• <50 records/batch: 128-256 MB sufficient
• 50-100 records/batch: 256-512 MB recommended
• Peak streams: Monitor Lambda duration; increase memory if >50% of timeout

Concurrency Management

DynamoDB Streams creates one shard per 1000 WCU (or ~3000 writes/sec). Each shard invokes one Lambda instance.

Recommendations:

Volume	Reserved Concurrency	Notes
<1000 writes/sec	None	Default scaling sufficient
1000-10000/sec	10-50	Prevents runaway scaling
>10000/sec	Expected shards + 20%	Based on table monitoring

Error Handling

Configure error handling for production reliability:

# Deploy with DLQ and alarms
zae-limiter deploy --name my-app \
  --alarm-sns-topic arn:aws:sns:us-east-1:123456789012:alerts

• Retries: Failed records retry 3 times within the same batch
• DLQ: Persistent failures go to Dead Letter Queue (if configured)
• Duration Alarm: Triggers at 80% of timeout (48s default)

---

3. Batch Operation Patterns

Transaction Limits

DynamoDB enforces these limits:

Constraint	Limit	Impact
TransactWriteItems	100 items max	Affects multi-limit updates
BatchWriteItem	25 items per request	Entity deletion is chunked
Optimistic locking	Entire transaction fails	Causes retry on contention

Efficient Patterns

Multi-Limit Acquisition

# Efficient: Single lease for multiple limits
async with limiter.acquire(
    "entity-id",
    "llm-api",
    {"rpm": 1, "tpm": 0},  # 1 request; token cost is not known yet
    limits=[rpm_limit, tpm_limit],
) as lease:
    # 1 BatchGetItem + 1 UpdateItem (1 WCU, single composite bucket)
    response = await call_llm()
    await lease.adjust(tpm=response.usage.total_tokens)
    # Adjustment: +1 UpdateItem via write_each (1 WCU)

# Inefficient: Separate acquisitions
async with limiter.acquire("entity-id", "llm-api", {"rpm": 1}, limits=[rpm_limit]):
    async with limiter.acquire("entity-id", "llm-api", {"tpm": 100}, limits=[tpm_limit]):
        # 2 reads + 2 writes (doubles cost!)
        pass

Name every limit you intend to adjust, even at 0. consume declares which limits the lease covers: a limit left out of it has no lease entry, so adjust() against it is silently ignored. An estimate of 0 is the way to say "this limit is in play, the cost is not known yet" — it costs nothing extra, since the write is the same single composite item either way.

Cascade Optimization

# Entity without cascade (default) — saves 1 GetEntity + parent bucket operations
await limiter.create_entity(entity_id="api-key", parent_id="project-1")

async with limiter.acquire("api-key", "llm-api", {"rpm": 1}, limits=limits):
    pass  # Only checks api-key's limits

# Entity with cascade — checks and updates parent limits too
await limiter.create_entity(entity_id="api-key", parent_id="project-1", cascade=True)

async with limiter.acquire("api-key", "llm-api", {"rpm": 1}, limits=limits):
    pass  # Checks both api-key AND project-1 limits

Write Sharding (Automatic Pre-Shard Buckets)

Starting with v0.9.0 (GHSA-76rv), zae-limiter automatically handles DynamoDB hot partition mitigation via pre-shard buckets. Each bucket item lives on its own DynamoDB partition key (PK={ns}/BUCKET#{id}#{resource}#{shard}), and an auto-injected wcu:1000 infrastructure limit tracks per-partition write pressure.

How it works:

1. Every bucket starts with shard_count=1 (shard 0)
2. An internal wcu limit (capacity: 1000 millitokens) is auto-injected on every bucket
3. When wcu is exhausted on a speculative write, the client doubles shard_count via a conditional write on shard 0 (source of truth) and moves to one of the new shards
4. The client creates the new shard's bucket item itself on its next slow-path acquire, with capacity / shard_count tokens — the aggregator is not required for sharding to engage (ADR-133)
5. The Lambda aggregator, when deployed, is a backstop: it can also double shards when the stored wcu balance, projected to now, is below 20% of capacity (never past 32), though it rarely does so before the client has (step 3 is what protects a hot shard). It pre-creates the new shard items so clients skip that one-time slow path
6. Shard count changes on shard 0 are propagated to all other shards by the aggregator
7. Clients pick a random shard from the entity cache: random.randrange(shard_count)
8. If application limits are exhausted on one shard but the entity has multiple shards, the client retries on up to 2 other randomly chosen shards, creating any it finds missing

Shard-aware capacity: The aggregator's refill, the client's slow-path refill and the client's shard creation all divide capacity and refill amount by shard_count, so each shard holds its proportional share and the entity admits at most its configured capacity per refill window. A doubling does not rewrite shard 0's balance, but the refill calculation clamps tokens to the shard's share on every pass — including the passes that add no tokens — so shard 0 is trimmed to its new share by the next refiller to touch it (the client slow path, or the aggregator, which writes the trim as a negative delta). There is no transient above the configured capacity.

A quota (Limit.quota()) never refills, so it is handled differently: a new shard is funded by an atomic move off the shard whose grant covers it — or, when no shard covers it, by a fresh share — and each shard's ceiling is the share it was granted, not the current share. A doubling therefore neither creates nor destroys quota allowance (ADR-145). A shard creation that moves tokens reads the entity's other shards once and writes one transaction carrying the new shard, the donor's debit and the owner check (6 WCU), once per shard, then reads shard 0's count once (1 RCU) to catch a doubling that landed while it was being created.

Works without the aggregator: Deployments using --no-aggregator get the same write-sharding behaviour; the only difference is that each new shard costs one slow-path acquire (about 2.5 RCU + 2 WCU) to create, once, instead of being pre-created from the stream.

Known limitation — a single request cannot exceed capacity // shard_count. No shard ever holds more than its share, so once a bucket has sharded, a request larger than one share is rejected on every shard even while the entity is well under its configured limit. For example, an entity limited to rpm:1000 that has doubled to 4 shards admits at most 250 per call. RateLimitExceeded reports the per-shard capacity and refill rather than the undivided configuration, so the status is honest about what the shard can hold, and MAX_SHARD_COUNT (32) bounds how small a share can get. If your workload issues single requests of comparable size to a limit's capacity (large tpm estimates, most commonly), either raise the limit or keep the entity unsharded by lowering its write rate. Surfacing this occurrence as an event or CloudWatch metric is tracked in #475.

No application code changes required. Pre-shard buckets are transparent to users. The wcu limit is filtered from all user-facing output (bucket states, exceptions, usage snapshots).

When automatic sharding is insufficient:

For extreme high-fanout cascade scenarios (1000+ children with cascade=True writing to the same parent), automatic bucket sharding handles the per-bucket partition pressure. However, if you need to distribute traffic across multiple logical parents for application-level load balancing, you can still use manual entity sharding:

# Manual entity sharding for application-level distribution
# (Only needed for extreme cascade fan-out beyond what pre-shard handles)
num_shards = 10
api_key_id = "api-key-12345"
shard_id = hash(api_key_id) % num_shards
parent_id = f"project-1-shard-{shard_id}"

await limiter.create_entity(
    entity_id=api_key_id,
    parent_id=parent_id,
    cascade=True,
)

Session Quotas

A session quota (Limit.quota(..., reset_after=...), ADR-139) keeps every shard of an entity on one window (ADR-140), and that coherence is the only cost it adds. Where S is the entity's shard_count and L the number of reset_after limits on the bucket:

Event	Cost	How often
Acquire inside a window	Unchanged: 1 WCU, 0 reads on the speculative path. The fast path's condition is byte-identical to any other limit's	Every request
A new window opening	One slow-path acquire per shard — the first request to reach each shard after the rollover re-materialises it, as at any schedule boundary	S per window per entity
The fan-out by the request that opened the window	(S − 1) × L conditional writes, one per (sibling shard, window limit)	Once per window per entity
Opening a window, unsharded entity (S = 1)	One slow-path acquire; no fan-out writes and no requests issued	Once per window per entity
Creating shard N > 0	+1 RCU: one strongly consistent GetItem of shard 0's window start, projected to the window attributes, on top of the ADR-133 create cost	Once per shard

At MAX_SHARD_COUNT (32) with one session limit, a rollover is 31 extra WCU (~$19 per million rollovers) — paid once per window, not per request. The fan-out writes a scalar (the window start) and never tokens: each sibling restores its own balance on its next materialising pass.

The shard-creation read is strongly consistent on purpose: an eventually consistent read could return shard 0's pre-rollover window, which looks ended, and the new shard would open a window of its own at a full share on top of the one shard 0 just opened.

A cascade slow path that creates a parent shard reads the parent's own window (parent and child windows are independent), so it pays the same +1 RCU once per parent shard.

Stored Limits Optimization

# Config caching reduces RCUs (60s TTL by default)
repo = await Repository.open(config_cache_ttl=60)  # seconds (0 to disable)
# Repeat rejections are answered locally: rejection_cache_ttl (1.0s, 0 to disable)
# and rejection_cache_size (10,000) sit beside it (see Rejection Cache)
limiter = RateLimiter(repository=repo)

# Pass explicit limits to skip config resolution entirely
async with limiter.acquire(
    entity_id="user-123",
    resource="api",
    limits=[Limit.per_minute("rpm", 100)],  # No config lookup
    consume={"rpm": 1},
) as lease:
    ...

Bulk Operations

# Efficient bulk limit setup
await limiter.set_limits("entity-1", [rpm_limit, tpm_limit], resource="llm-api")
await limiter.set_limits("entity-2", [rpm_limit, tpm_limit], resource="llm-api")
# Runs 2 Queries + 2×2 PutItems

# Entity deletion (automatically batched in 25-item chunks)
await limiter.delete_entity("entity-2")
# Runs 1 Query + BatchWrite (up to 25 WCUs per chunk)

---

4. Expected Latencies

Operation Latencies

Latencies vary by environment and depend on network conditions, DynamoDB utilization, and operation complexity.

Operation	Moto p50	LocalStack p50	AWS (external) p50	AWS (in-region) p50
acquire() - single limit	9ms	43ms	38ms	15-20ms
acquire() - two limits	11ms	43ms	36ms	15-20ms
acquire() with cascade	22ms	47ms	48ms	25-35ms
available() check	1ms	7ms	10ms	1-3ms

Environment Differences

• Moto: In-memory mock, measures code overhead only
• LocalStack: Docker-based, includes local network latency (varies by host)
• AWS (external): From outside AWS, includes internet latency (~8-14ms per round-trip)
• AWS (in-region): From EC2/Lambda in same region (~0.5-1ms per round-trip)

In-Region Performance

When running inside AWS (same region as DynamoDB), latency drops significantly because network round-trips take <1ms instead of 8-14ms. For a typical LLM API call, rate limit overhead is ~4% (20ms / 500ms) vs ~7% when calling from external networks.

Latency Breakdown

Typical acquire() latency breakdown for a single limit (non-cascade):

acquire() latency breakdown (external client):
├── Network to AWS               ~8-10ms   (internet latency)
├── DynamoDB GetItem             ~3-5ms    (server-side processing)
├── Network back                 ~8-10ms
├── UpdateItem                   ~3-5ms    (single-item API, 1 WCU)
└── Network back                 ~8-10ms
                                 ─────────
                         Total:  ~30-40ms

acquire() latency breakdown (in-region):
├── Network to DynamoDB          ~0.5-1ms  (VPC internal)
├── DynamoDB GetItem             ~3-5ms
├── Network back                 ~0.5-1ms
├── UpdateItem                   ~3-5ms    (single-item API, 1 WCU)
└── Network back                 ~0.5-1ms
                                 ─────────
                         Total:  ~10-15ms

With speculative writes enabled (speculative_writes=True), the read round trip is eliminated on success:

speculative acquire() latency breakdown (in-region, success):
├── Network to DynamoDB          ~0.5-1ms  (VPC internal)
├── Conditional UpdateItem       ~3-5ms    (1 WCU, skips read)
└── Network back                 ~0.5-1ms
                                 ─────────
                         Total:  ~5-8ms

Single-item vs Transaction writes

Non-cascade acquire() writes a single composite bucket item, so transact_write() dispatches it as a plain UpdateItem (1 WCU). Cascade mode with 2 items uses TransactWriteItems (2 WCU per item). Adjustments and rollbacks always use independent single-item writes via write_each() (1 WCU each).

Environment Selection

Environment	Use Case	Latency Factor
Moto	Unit tests, CI/CD	1× (baseline)
LocalStack	Integration tests, local dev	4-5×
AWS (external)	Development, testing	4×
AWS (in-region)	Production	2×

Run benchmarks to measure your specific environment:

# Moto benchmarks (fast)
uv run pytest tests/benchmark/test_latency.py -v --benchmark-json=latency.json

# LocalStack benchmarks (requires Docker)
docker compose up -d
export AWS_ENDPOINT_URL=http://localhost:4566
uv run pytest tests/benchmark/test_localstack.py -v --benchmark-json=latency.json

# AWS benchmarks (requires credentials)
uv run pytest tests/benchmark/test_aws.py --run-aws -v

---

5. Throughput Benchmarks

Maximum Throughput

Theoretical and practical throughput limits depend on contention patterns:

Scenario	Moto TPS	AWS TPS	Bottleneck
Sequential, single entity	95	28	Network round-trip
Sequential, multiple entities	76	26	Network round-trip
Concurrent, separate entities	85	176	Scales with parallelism
Concurrent, single entity	88	—	Optimistic locking contention
Cascade sequential	27	19	Parent bucket operations
Cascade concurrent	28	91	Parent bucket contention

AWS Concurrent Performance

AWS concurrent throughput (176 TPS) exceeds sequential (28 TPS) because parallel requests to separate entities eliminate serialization. In-region performance would be ~2× higher due to reduced network latency.

Contention Analysis

When multiple requests update the same bucket concurrently, DynamoDB's optimistic locking causes transaction retries:

Concurrent updates to same bucket:
├── Request A: Read bucket version=1
├── Request B: Read bucket version=1
├── Request A: Write with condition version=1 → SUCCESS, version=2
├── Request B: Write with condition version=1 → FAIL (ConditionalCheckFailed)
└── Request B: Retry with version=2 → SUCCESS

Each retry adds ~10-30ms latency.

Mitigation Strategies

# Strategy 1: Higher capacity (reduces contention per request)
rpm_limit = Limit.per_minute("rpm", 1000)

# Strategy 2: Distribute load across entities
# Instead of one shared entity, use sharded entities:
shard = hash(request_id) % 10
entity_id = f"api-key-shard-{shard}"

# Strategy 3: Client-side rate limiting before acquire
# Reduce concurrent requests to the same entity

Running Benchmarks

Use the automated benchmark runner:

# Run all benchmarks (moto + LocalStack)
python scripts/run_benchmarks.py

# Include AWS benchmarks
python scripts/run_benchmarks.py --run-aws

# Skip LocalStack (moto only)
python scripts/run_benchmarks.py --skip-localstack

# Custom output directory
python scripts/run_benchmarks.py --output-dir ./results

Or run individual test suites:

# Throughput tests
uv run pytest tests/benchmark/test_throughput.py -v

# Analyze results
python -c "import json; print(json.load(open('benchmark.json'))['benchmarks'])"

---

6. Cost Optimization Strategies

DynamoDB Cost Breakdown

Costs vary by region. Using us-east-1 as reference:

Component	On-Demand Cost	Notes
Write Request Units	$0.625 per million	Each WCU = one write
Read Request Units	$0.125 per million	Each RCU = one read
Storage	$0.25 per GB/month	Usually minimal
Streams	$0.02 per 100K reads	Lambda polling
Lambda	$0.20 per million + duration	Aggregator function

Cost Estimation Examples

O(1) Costs (v0.7.0+)

With composite bucket items, costs no longer multiply by number of limits. Whether you track 2 limits or 10 limits per request, DynamoDB costs are the same.

Low Volume: 10K requests/day

DynamoDB:
  Writes: 10K × 30 days = 300K WCUs            = $0.19
  Reads:  10K × 30 days = 300K RCUs            = $0.04
  Streams: 300K events                         = $0.06
Lambda: 300K invocations                       ≈ $0.06 + duration
Storage: ~10 MB                                = negligible
─────────────────────────────────────────────────────────
Total: ~$0.35/month

Medium Volume: 1M requests/day

DynamoDB:
  Writes: 1M × 30 = 30M WCUs                   = $18.75
  Reads:  1M × 30 = 30M RCUs                   = $3.75
  Streams: 30M events                          = $6.00
Lambda: 30M invocations                        ≈ $6.00 + duration
─────────────────────────────────────────────────────────
Total (on-demand): ~$35/month
Total (provisioned with auto-scaling): ~$22/month

Cost Reduction Strategies

1. Disable Unused Features

# Create entity without cascade if not needed (saves 1-2 WCUs per request)
await limiter.create_entity(entity_id="entity", parent_id="project-1")  # cascade=False by default
async with limiter.acquire("entity", "api", limits, {"rpm": 1}):
    pass

# Disable stored limits if static (saves 2 RCUs per request)
repo = await Repository.open()
limiter = RateLimiter(repository=repo)

2. Optimize TTL Settings

# Shorter TTL = faster cleanup = less storage
# bucket_ttl is configured via builder or CloudFormation
repo = await Repository.open()
limiter = RateLimiter(repository=repo)

3. Reduce Snapshot Granularity

# Deploy without aggregator if usage tracking not needed
zae-limiter deploy --name my-app --no-aggregator

4. Switch to Provisioned at Scale

• Break-even: ~5M operations/month
• Use auto-scaling with 70% target utilization
• Consider reserved capacity for >20M ops/month

5. Batch Similar Operations

# Combine multiple limits into single acquire
async with limiter.acquire(
    entity_id="entity",
    resource="api",
    consume={"rpm": 1},  # 1 transaction vs 3
    limits=[rpm_limit, tpm_limit, daily_limit],
):
    pass

Cost Monitoring

Set up CloudWatch metrics for cost tracking:

DynamoDB Metrics:

• ConsumedReadCapacityUnits
• ConsumedWriteCapacityUnits
• AccountProvisionedReadCapacityUtilization
• AccountProvisionedWriteCapacityUtilization

Lambda Metrics:

• Invocations
• Duration
• ConcurrentExecutions

Recommended Alerts:

# Deploy with alarms for cost anomalies
zae-limiter deploy --name my-app \
  --alarm-sns-topic arn:aws:sns:us-east-1:123456789012:billing-alerts

# Set AWS Budgets alert at 80% of expected monthly cost
aws budgets create-budget \
  --account-id 123456789012 \
  --budget file://budget.json \
  --notifications-with-subscribers file://notifications.json

---

7. Config Cache Tuning

The config cache reduces DynamoDB reads by caching system defaults, resource defaults, and entity limits. This section covers tuning the cache for your workload.

Cache Configuration

from zae_limiter import RateLimiter, Repository

# Default: 60-second TTL (recommended for most workloads)
repo = await Repository.open(config_cache_ttl=60)
limiter = RateLimiter(repository=repo)

# High-frequency updates: Shorter TTL for faster propagation
repo = await Repository.open(config_cache_ttl=10)
limiter = RateLimiter(repository=repo)

# Disable caching: For testing or when config changes must be immediate
repo = await Repository.open(config_cache_ttl=0)
limiter = RateLimiter(repository=repo)

The rejection cache has its own options beside config_cache_ttl: rejection_cache_ttl (seconds, default 1.0, 0 disables) and rejection_cache_size (default 10_000 entries). See Rejection Cache.

Cost Impact

Without caching, each acquire() call performs 3 DynamoDB reads to resolve limits:

1. Entity-level config lookup (1 RCU)
2. Resource-level config lookup (1 RCU)
3. System-level config lookup (1 RCU)

With caching (default):

Traffic Rate	Cache Hit Rate	Amortized RCU/request
1 req/sec	98.3%	0.05 RCU
10 req/sec	99.8%	0.005 RCU
100 req/sec	99.98%	0.0005 RCU

Negative caching also helps: When an entity has no custom config (95%+ of entities typically), the cache remembers this to avoid repeated lookups.

Automatic Cache Eviction

Config-modifying methods (set_limits(), delete_limits()) automatically evict relevant cache entries. Manual invalidation is only needed after external changes (e.g., direct DynamoDB writes or changes from another process).

Manual Invalidation

After external config changes, force immediate refresh:

await repo.invalidate_config_cache()

Without manual invalidation, changes propagate within the TTL period (max 60 seconds by default).

Monitoring Cache Performance

# Get cache statistics
stats = repo.get_cache_stats()
total = stats.hits + stats.misses
print(f"Cache hit rate: {stats.hits / total:.1%}" if total else "No requests yet")
print(f"Cache entries: {stats.size}")
print(f"TTL: {stats.ttl_seconds}s")
print(f"Local rejections: {stats.local_rejections}")  # rejection cache (ADR-147)

TTL Selection Guidelines

Scenario	Recommended TTL	Rationale
Production (stable config)	60s (default)	Best cost/latency trade-off
Development/testing	10-30s	Faster config iteration
Compliance-critical	10-30s	Minimizes staleness
Testing with frequent changes	0 (disabled)	Immediate visibility
High-traffic APIs (>100 req/s)	60-120s	Maximize cache hits

---

8. Speculative Writes

Speculative writes (issue #315) enable a fast path for acquire() that skips the read round trip by issuing a conditional UpdateItem directly. This is most effective for pre-warmed buckets with sufficient capacity.

Configuring Speculative Writes

from zae_limiter import RateLimiter, Repository

repo = await Repository.open()
limiter = RateLimiter(repository=repo, speculative_writes=True)  # Default

How It Works

Instead of the normal read-then-write flow (BatchGetItem + UpdateItem), the speculative path attempts a conditional UpdateItem first.

First acquire (sequential, populates entity cache):

acquire(entity_id, resource, consume)
|
+- Speculative UpdateItem (condition: bucket exists, is not disabled, has not
|  crossed a schedule boundary, and has enough tokens)
   |
   +- SUCCEEDS -> read cascade/parent_id from ALL_NEW, populate entity cache
   |  +- cascade=False -> DONE (1 RT, 0 RCU, 1 WCU)
   |  +- cascade=True -> Parent speculative UpdateItem (sequential)
   |     +- SUCCEEDS -> DONE (2 RT, 0 RCU, 2 WCU)
   |     +- FAILS -> [parent failure handling, see below]
   |
   +- FAILS (ConditionalCheckFailedException)
      +- No ALL_OLD (bucket missing) -> SLOW PATH (creates bucket)
      +- Missing limit in ALL_OLD -> SLOW PATH
      +- Schedule boundary crossed (vu passed) -> SLOW PATH (re-materialises)
      +- Refill would help -> SLOW PATH
      +- Refill won't help -> RateLimitExceeded (0 RCU, 1 WCU)

Subsequent acquires (parallel, issue #318):

When the entity cache contains (cascade=True, parent_id) from a prior acquire, child and parent speculative writes are issued concurrently:

acquire(entity_id, resource, consume)   [cache hit: cascade=True, parent_id known]
|
+- asyncio.gather(child_speculative, parent_speculative)
   |
   +- BOTH SUCCEED -> DONE (1 RT, 0 RCU, 2 WCU)
   +- CHILD FAILS, PARENT SUCCEEDS -> Compensate parent, check child ALL_OLD
   |  +- [same child failure handling as sequential path]
   +- CHILD SUCCEEDS, PARENT FAILS -> Check parent ALL_OLD (child stays consumed)
   |  +- No ALL_OLD (missing) -> Compensate child, SLOW PATH
   |  +- Missing limit -> Compensate child, SLOW PATH
   |  +- Refill won't help -> Compensate child, RateLimitExceeded
   |  +- Refill would help -> Parent-only slow path (keep child)
   |     +- Parent acquire succeeds -> DONE (2 RT, 0.5 RCU, 3 WCU)
   |     +- Parent acquire fails -> Compensate child, SLOW PATH
   +- BOTH FAIL -> Check child ALL_OLD, fall back or fast-reject

The ReturnValuesOnConditionCheckFailure=ALL_OLD response provides the current bucket state on failure, allowing the limiter to determine whether refill would help without an additional read.

Deferred cascade compensation: When the child speculative write succeeds but the parent fails with "refill would help", the child's consumption is kept in place while a parent-only slow path is attempted. This avoids compensating the child (1 WCU), re-reading it (0.5 RCU), and using TransactWriteItems for the full cascade write (4 WCU). Instead, only the parent is read (0.5 RCU) and written via a single-item UpdateItem (1 WCU). Compensation only happens when the parent-only path also fails.

Entity metadata cache (issue #318): Repository._entity_cache stores {entity_id: (cascade, parent_id)} as immutable metadata with no TTL. After the first acquire populates the cache (from ALL_NEW on speculative success or from the entity META record on slow path), subsequent cascade acquires fire child and parent speculative writes concurrently via asyncio.gather inside speculative_consume(). This reduces cascade latency from 2 sequential round trips to 1 parallel round trip while maintaining the same WCU cost. In sync mode, asyncio.gather is transformed to self._run_in_executor(lambda: a, lambda: b) using a lazy ThreadPoolExecutor(max_workers=2) for true parallel execution.

Cost Comparison

Scenario	Round Trips	RCU	WCU	Cost per 1M
Normal path (non-cascade)	2	1	1	$0.75
Speculative success (non-cascade)	1	0	1	$0.625
Speculative fast rejection (exhausted)	1	0	1	$0.625
Repeat rejection (rejection cache, inside the TTL)	0	0	0	$0.00
Speculative fallback (refill helps)	3	2.5	2	$1.5625
Speculative fallback (schedule boundary)	3	2.5	2	$1.5625
Normal path (cascade)	3	2	4	$1.75
Speculative success (cascade, sequential)	2	0	2	$1.25
Speculative success (cascade, parallel)	1	0	2	$1.25
Speculative cascade fallback (parent refill helps)	2+	1	3	$2.00
Speculative cascade fast rejection (parent exhausted)	1	0	3	$1.875
Repeat cascade rejection (parent known short, rejection cache)	0	0	0	$0.00

A scheduled limit pays the boundary fallback once per bucket per boundary — every request in flight at that instant fails the condition together, then serialises on the refill lock — so four boundaries a day across 10,000 active buckets is roughly 40,000 extra slow-path passes, about $0.05.

When speculative writes save money

The speculative path is cheaper than the normal path when most requests succeed without needing refill. If a high percentage of requests fall back to the slow path (new entities, near-capacity buckets, frequent config changes), the extra WCU from the failed speculative write makes it more expensive.

Latency Comparison

Scenario	Round Trips	Expected Latency (in-region)
Normal path (non-cascade)	2	10-15ms
Speculative success (non-cascade)	1	5-8ms
Speculative fast rejection (exhausted)	1	5-8ms
Speculative fallback (refill helps)	3	15-22ms
Normal path (cascade)	3	15-22ms
Speculative success (cascade, sequential)	2	8-12ms
Speculative success (cascade, parallel)	1	5-8ms
Speculative cascade fallback (parent refill helps)	2+	12-20ms
Speculative cascade fast rejection (parent exhausted)	1	5-8ms

Rejection Cache

A fast rejection still costs 1 WCU, because DynamoDB charges a failed conditional write. To avoid paying that on every repeat 429, the repository keeps the last bucket state each speculative response returned, per shard (ADR-147). Before the next speculative write, the limiter projects that state to now with the same refill arithmetic as a fast rejection:

• Every shard known short: RateLimitExceeded is raised with no DynamoDB call (0 RCU, 0 WCU, no round trip).
• Some shards known short: the write goes to a shard not known short.
• No usable state: the write goes ahead as before.

The cache only rejects; admission always needs a successful write, so it cannot over-admit. A state is not used once it is older than rejection_cache_ttl, past a schedule boundary (vu) or its bucket TTL, stamped disabled, or when acquire() passes limits=. A cascading child is rejected locally only when its own cached state says it cascades and the parent's cached shards are all short too, or on its own shortfall while a trusted parent state shows the parent is not disabled. Refunds, releases and rollbacks in this process forget the shard, and every admin write through the repository, or invalidate_config_cache(), clears the cache.

Example: a client looping on 429s at 1,000 req/s from one process drops from ~1,000 WCU/s of rejections to ~1 WCU/s — one real write per TTL to refresh the state. That is per process and counts rejections only: admissions still cost a write each, and N processes on the same hot entity pay N real writes per TTL. A process that sees an entity less often than once per TTL saves nothing, because its state has expired by the next request.

Trade-off: for up to rejection_cache_ttl per process, a request can be rejected after tokens came back by a route this process cannot see: another process's refund, an admin change made elsewhere, or another process doubling the shard count. Each process pays its own real write per TTL.

Known behaviour, each bounded by rejection_cache_ttl in one process:

• A resource or parent disabled by another process can get a 429 here instead of ResourceDisabled (403), until the cached state expires.
• An entity deleted by another process can be rejected instead of admitted to a fresh bucket.
• During a DynamoDB outage, a bucket seen exhausted in the last TTL raises RateLimitExceeded rather than taking the on_unavailable path (stricter, not looser).
• A client that retries a 429 without waiting spins on CPU: about 10,000 retries per second against about 100 when every rejection went to DynamoDB. Other tasks keep running (#704); honour retry_after_seconds.

The cache is bounded at rejection_cache_size entries. Two per-process maps beside it are not bounded by it: the cascade-policy cache (one entry per entity and resource seen) and the entity cache (one per entity, as in earlier releases).

from zae_limiter import RateLimiter, Repository

# Default: trust a cached state for 1 second, keep at most 10,000 shards
repo = await Repository.open(rejection_cache_ttl=1.0, rejection_cache_size=10_000)

# Disable: every rejection reaches DynamoDB (1 WCU each)
repo = await Repository.open(rejection_cache_ttl=0)
limiter = RateLimiter(repository=repo)

print(repo.get_cache_stats().local_rejections)  # rejections answered without DynamoDB

The builder takes the same options: .rejection_cache_ttl(seconds), .rejection_cache_size(entries).

Aggregator-Assisted Refill (Issue #317)

When the Lambda aggregator is enabled, it proactively refills token buckets for active entities between client requests. This keeps speculative writes on the fast path (1 RT, 0 RCU, 1 WCU) by ensuring buckets have sufficient tokens, reducing fallback to the slow path (3 RT, 1 RCU, 2 WCU).

How it works:

1. The aggregator processes DynamoDB Stream events for bucket modifications
2. For each active (entity, resource) bucket, it aggregates consumption deltas from the batch
3. If projected tokens after natural refill are insufficient to cover the observed consumption rate, it writes a proactive refill
4. The refill uses ADD (commutative with concurrent speculative writes) and an optimistic lock on rf to prevent double-refill

Cost: 1 WCU per refill attempted, including one that loses the lock because another writer updated rf first (a failed conditional write is still charged). The cost is amortized across all stream records in a batch, so high-throughput workloads see fewer refills per request.

Aggregator refill + speculative writes

The combination of aggregator-assisted refill and speculative writes provides the best latency and cost profile: the aggregator keeps buckets warm so speculative writes rarely fall back, achieving ~5-8ms p50 latency at $0.625/M requests (non-cascade).

When to Use Speculative Writes

Good fit:

• High-throughput workloads with pre-warmed buckets
• Buckets that rarely exhaust capacity (high capacity relative to request rate)
• Latency-sensitive applications where saving one round trip matters
• Cascade entities with repeated acquires (entity cache enables parallel writes after first acquire)
• Deployments with the Lambda aggregator enabled (aggregator keeps buckets warm for speculative success)

Poor fit:

• New entities that have never been seen before (first acquire always falls back)
• Near-capacity buckets that frequently exhaust (high fallback rate)
• Workloads with frequent config changes (missing limits trigger fallback)
• One-shot entities that are only acquired once (entity cache provides no benefit)

Monitoring Speculative Effectiveness

Track the ratio of speculative successes to fallbacks to determine if speculative writes are beneficial for your workload:

# Speculative writes work transparently with acquire()
# Monitor DynamoDB ConsumedWriteCapacityUnits to observe:
# - Lower WCU = more speculative successes
# - Higher WCU = more fallbacks (consider disabling)

Disabling speculative writes

If most requests are from new entities or near-capacity buckets, disable with speculative_writes=False to avoid the extra WCU from failed speculative attempts.

---

9. Load Testing with Locust

For realistic, multi-user load testing against a live DynamoDB stack, zae-limiter provides a Locust integration module (zae_limiter.locust). It exposes RateLimiterUser and RateLimiterSession, analogous to Locust's built-in HttpUser and HttpSession, so that every acquire(), available(), and management call fires Locust request events with timing.

Installation

Install the [bench] extra to pull in Locust and its dependencies:

# pip install zae-limiter[bench]

Quick Start

from locust import task
from zae_limiter.locust import RateLimiterUser

class MyUser(RateLimiterUser):
    stack_name = "my-limiter"

    @task
    def do_acquire(self):
        with self.client.acquire(
            entity_id="user-123",
            resource="gpt-4",
            consume={"rpm": 1, "tpm": 500},
            name="gpt-4/baseline",
        ):
            pass  # simulate work

Run with:

locust -f locustfile.py --host <stack-name>

Key Design Points

• Shared limiter: A single SyncRateLimiter instance is shared across all Locust user greenlets (thread-safe via boto3).
• Connection pool: _configure_boto3_pool() automatically enlarges the boto3 connection pool (default 1000, override with BOTO3_MAX_POOL env var) to prevent pool exhaustion under high concurrency.
• Event types: ACQUIRE, COMMIT, RATE_LIMITED, AVAILABLE, and management operations (SET_SYSTEM_DEFAULTS, CREATE_ENTITY, etc.) appear as distinct request types in the Locust UI.
• Multi-resource acquire: self.client.acquire(..., also={"budget": {"budget_rpm": 1}}) passes also= through to SyncRateLimiter.acquire() (ADR-148). One lease is one ACQUIRE event, named for the primary resource (or name= when given).
• Rate limit handling: RateLimitExceeded is tracked as RATE_LIMITED (not counted as a failure), so Locust statistics cleanly separate infrastructure errors from expected rate limiting.

Example Scenarios

Pre-built locustfiles are available in examples/locust/locustfiles/:

Scenario	File	Description
Simple	simple.py	Single resource, single limit, basic acquire
Max RPS	max_rps.py	Zero-wait back-to-back acquire for throughput ceiling
Multi-resource	multi_resource.py	One acquire(..., also=...) over a cascading search and a per-user budget, vs two nested acquires. Compare the ITERATION multi-resource row, which times a whole iteration in both classes; the per-resource rows are informational (the nested class has two per iteration)
LLM Gateway	llm_gateway.py	8 LLM models with RPM + TPM and lease adjustments
LLM Production	llm_production.py	Weighted tasks with custom daily/spike load shapes
Stress	stress.py	16K entities with whale/spike/power-law traffic patterns

See examples/locust/README.md for full usage instructions including distributed execution on AWS.

---

Summary

Optimization Area	Key Recommendations
Capacity	Start with on-demand, switch to provisioned at 5M+ ops/month
Latency	Expect 15-20ms p50 in-region, 35-45ms external; network is the dominant factor
Throughput	Distribute load across entities to avoid contention
Cost	Disable cascade/stored_limits when not needed
Config Cache	Use default 60s TTL; invalidate manually for immediate changes
Speculative Writes	Enable for pre-warmed high-throughput workloads; saves 1 round trip on success; cascade entities get parallel writes after first acquire
Load Testing	Use zae_limiter.locust with RateLimiterUser for realistic multi-user load tests; see examples/locust/
Monitoring	Set up CloudWatch alerts for capacity and cost anomalies

For detailed benchmark data, run:

python scripts/run_benchmarks.py --run-aws