Skip to content

Rate Limit Operations

This guide covers troubleshooting rate limit enforcement issues and procedures for adjusting limits at runtime.

Decision Tree

flowchart TD START([Rate Limit Issue]) --> Q1{What's happening?} Q1 -->|Unexpected RateLimitExceeded| A1[Check bucket state] Q1 -->|Limits not enforcing| A2[Check entity exists] Q1 -->|Cascade not working| A3[Check parent entity] Q1 -->|Need to adjust limits| A4[Runtime adjustment] A1 --> CHECK1[Query bucket in DynamoDB] A2 --> CHECK2[Query entity metadata] A3 --> CHECK3[Verify parent_id and entity cascade setting] CHECK1 --> FIX1{Bucket state?} FIX1 -->|Depleted| WAIT[Wait for refill or reset] FIX1 -->|Has capacity| CONFIG[Check limit configuration] CHECK2 --> FIX2{Entity exists?} FIX2 -->|No| CREATE[Create entity first] FIX2 -->|Yes| LIMITS[Check stored limits config] click A1 "#unexpected-ratelimitexceeded" "Diagnose unexpected limits" click A2 "#limits-not-enforcing" "Check entity setup" click A3 "#cascade-not-working" "Check parent setup" click A4 "#adjust-limits-at-runtime" "Adjust limits" click WAIT "#reset-bucket-state" "Reset bucket" click CHECK1 "#debug-bucket-state" "Query bucket state"

Troubleshooting

Symptoms

  • Requests succeed when they should be rate limited
  • RateLimitExceeded raised unexpectedly
  • Cascade to parent entity not working
  • Bucket state appears incorrect

Diagnostic Steps

Check entity and bucket state:

# Query entity metadata
aws dynamodb get-item --table-name <name> \
  --key '{"PK": {"S": "ENTITY#<entity_id>"}, "SK": {"S": "#META"}}'

# Query bucket state for a specific resource (shard 0)
# v0.9.0+ bucket key format: PK={ns}/BUCKET#{entity}#{resource}#{shard}, SK=#STATE
aws dynamodb get-item --table-name <name> \
  --key '{"PK": {"S": "{ns}/BUCKET#<entity_id>#<resource>#0"}, "SK": {"S": "#STATE"}}'

Verify stored limits:

aws dynamodb query --table-name <name> \
  --key-condition-expression "PK = :pk AND begins_with(SK, :sk)" \
  --expression-attribute-values '{":pk": {"S": "ENTITY#<entity_id>"}, ":sk": {"S": "#LIMIT#"}}'

Unexpected RateLimitExceeded

Possible causes:

Cause Diagnosis Solution
Bucket depleted Check bucket tokens value Wait for refill or increase limit
Limit too restrictive Compare limit capacity vs usage Increase capacity
Cascade triggered Check violations array Increase parent limit
Clock skew Compare server time with bucket last_update Sync NTP
Over-sharded entity shard_count above 32 on the bucket item See Over-sharded entities

Check bucket state:

# v0.9.0+ bucket key format
aws dynamodb get-item --table-name <name> \
  --key '{"PK": {"S": "{ns}/BUCKET#<entity_id>#<resource>#0"}, "SK": {"S": "#STATE"}}' \
  --projection-expression "b_<limit_name>_tk, rf, b_<limit_name>_cp, shard_count"

Interpret the response:

  • tokens: Current token count (in millitokens, divide by 1000)
  • last_update: Unix timestamp of last access
  • capacity: Maximum bucket capacity (in millitokens)

Over-sharded entities

Before #745, the aggregator's proactive sharding could double an entity's shard_count on light traffic, with no upper bound: one entity went from 1 to 2048 shards over 11 aggregator invocations. Each shard holds capacity // shard_count, so at 2048 shards almost every request is larger than one shard's share.

Symptom: RateLimitExceeded while the entity is far below its configured limit. In one measurement, 1 request in 200 was admitted.

How to spot it:

  • The bucket item's shard_count is above 32 (MAX_SHARD_COUNT). Use the get-item command above; shard_count is in the projection.
  • Or the GSI3 discovery in Debug Bucket State returns more than 32 bucket items for one entity and resource.

Upgrading does not repair it. A current version never doubles past 32 and never propagates a count above 32, but nothing lowers a shard_count that is already there. A repair command is tracked in #747; until then, use the manual procedure below. reset_bucket() / entity reset cannot do it: it writes every shard in one transaction, sized for at most 32 shards.

Find every over-sharded bucket

aws dynamodb scan --table-name <name> \
  --filter-expression "SK = :state AND shard_count > :cap" \
  --expression-attribute-values '{":state": {"S": "#STATE"}, ":cap": {"N": "32"}}' \
  --projection-expression "PK, shard_count"

Each PK is {ns}/BUCKET#{entity}#{resource}#{shard}; one (entity, resource) pair usually shows up once per shard. The scan reads the whole table (about 0.5 RCU per 4 KB), so run it once and work from its output.

Repair one entity and resource

Stop the clients first

Every client process caches each entity's shard_count in memory and only ever raises it. A process still running with the old count recreates the deleted shards at that count the next time it acquires for the entity (reproduced on moto: 20 acquires from such a process after the repair created 20 shard items at shard_count = 2048). A restarted process reads the count afresh.

  1. Upgrade first: zae-limiter upgrade --name <name>, so the aggregator is a version that can no longer run away (#745).
  2. Stop every process that calls acquire() for the entity (the application, workers, anything holding a Repository). If you cannot stop them, restart all of them right after step 3 and expect to repeat it once (step 5).

  3. Delete every bucket item of the pair, with credentials that can delete items (FullAccessPolicy):

    import boto3
from boto3.dynamodb.conditions import Key

TABLE = "my-app"  # the stack name
NAMESPACE_ID = "..."  # zae-limiter namespace show <namespace> --name <name>
ENTITY = "user-123"
RESOURCE = "gpt-4"

table = boto3.resource("dynamodb").Table(TABLE)
prefix = f"{NAMESPACE_ID}/BUCKET#{ENTITY}#{RESOURCE}#"

# The shards GSI3 knows about...
keys = set()
query = {
    "IndexName": "GSI3",
    "KeyConditionExpression": Key("GSI3PK").eq(f"{NAMESPACE_ID}/ENTITY#{ENTITY}")
    & Key("GSI3SK").begins_with(f"BUCKET#{RESOURCE}#"),
}
while True:
    page = table.query(**query)
    keys.update(item["PK"] for item in page["Items"])
    if "LastEvaluatedKey" not in page:
        break
    query["ExclusiveStartKey"] = page["LastEvaluatedKey"]

# ...and every slot below the highest count, which GSI3 can lag behind
shard0 = table.get_item(
    Key={"PK": f"{prefix}0", "SK": "#STATE"},
    ConsistentRead=True,
    ProjectionExpression="shard_count",
).get("Item", {})
highest = max(
    [int(shard0.get("shard_count", 1))]
    + [int(pk.rsplit("#", 1)[1]) + 1 for pk in keys]
)
keys.update(f"{prefix}{shard}" for shard in range(highest))

with table.batch_writer() as batch:  # resends unprocessed deletes
    for pk in keys:
        batch.delete_item(Key={"PK": pk, "SK": "#STATE"})
print(f"deleted the bucket items of {highest} shard slots")

  4. Start the clients again. The next acquire creates the bucket at one shard, from the current config.

  5. Check: run the scan again. If the pair is listed, a process that was still running recreated it — restart it and repeat step 3.

On moto, a fresh process admitted 2 of 200 requests on a bucket at 2048 shards before the repair and 200 of 200 after it.

What the repair costs you. It deletes state, not config:

Lost Effect
Balances The recreated bucket starts full. A rate limit can admit a burst of up to one capacity; a quota gets its whole allowance again for the current period, whatever was already spent; a session quota starts a fresh window
Consumption counters on the bucket items The usage snapshots already written stay; the aggregator ignores deletes
disabled / bypass / soft stamps, owner stamps Nothing: the first acquire after the repair takes the slow path, which reads disabled, bypass and soft from config (uncached) and the owner from the entity record, as for any new bucket

Entity metadata, stored limits, and reset / top-up operation records are not touched.

Client clock skew can still double up to 32. The aggregator projects a bucket's stored wcu balance from its rf timestamp to the aggregator's own clock before deciding to double. rf is stamped from the client's clock, so when a client runs about 2 seconds ahead, the projection adds no refill, a stale low balance still reads as a hot partition, and the aggregator can double the entity again on following invocations. The walk is bounded: it stops at MAX_SHARD_COUNT (32) and never goes past it. Keep client and Lambda clocks in sync (NTP; Lambda's clock is already synced) to avoid it.

Limits Not Enforcing

Common causes:

Cause Solution
Entity not created Create entity before rate limiting: await limiter.create_entity(...)
No stored limits configured Set limits via set_system_defaults(), set_resource_defaults(), or set_limits()
Stale bucket state Bucket refills over time; tokens may have refilled
Limit configuration mismatch Verify limit capacity and refill_rate match expectations

Verify entity exists:

aws dynamodb get-item --table-name <name> \
  --key '{"PK": {"S": "ENTITY#<entity_id>"}, "SK": {"S": "#META"}}'

Resource or Entity Disabled

acquire() raises ResourceDisabled instead of RateLimitExceeded when the resource or entity has been explicitly turned off (ADR-125). Unlike a throttling error, retrying will never succeed until the disable is cleared — treat it as a 403, not a 429.

Check the resolved state:

repo = await Repository.open()
disabled, level = await repo.resolve_disabled("<entity_id>", "<resource>")
print(f"disabled={disabled} (decided at {level} level)")

level is "entity", "entity_default", "resource", or None when nothing along the walk sets disabled explicitly (so it resolves to enabled).

Re-enable:

# Clear the resource-level disable (reverts to inherit — currently always enabled,
# since there is no system-level disable)
zae-limiter resource clear-disabled <resource>

# Or carve out a single entity while the resource stays disabled for everyone else
zae-limiter entity enable <entity_id> --resource <resource>

Because disabling stamps every existing bucket, an enable/clear-disabled call also re-stamps eagerly before returning, so the fix takes effect on the next request, not after a TTL or cache delay. It does not necessarily touch every bucket: resource enable/ clear-disabled skip any bucket whose entity has its own disagreeing override (that carve-out is left alone, by design), and the unscoped entity clear-disabled (no --resource) restamps every bucket the entity has to its own freshly re-resolved value rather than to a single directive. See CLI: Disabling Resources and Entities and ADR-125.

Cascade Not Working

If cascade to parent is not enforced:

Step 1: Verify parent entity exists:

aws dynamodb get-item --table-name <name> \
  --key '{"PK": {"S": "ENTITY#<parent_id>"}, "SK": {"S": "#META"}}'

Step 2: Verify child has parent_id set:

aws dynamodb get-item --table-name <name> \
  --key '{"PK": {"S": "ENTITY#<child_id>"}, "SK": {"S": "#META"}}'
# Check the "parent_id" attribute in response

Step 3: Check whether a cascade policy decides it for this resource. Cascade is decided per resource (ADR-146): a policy on the entity's config for the resource, then on its entity-wide (_default_) config, then on the resource wins over the entity's own flag. get-limits / get-defaults print a Cascade: line for a level that sets one:

zae-limiter entity get-limits <child_id> --resource <resource>
zae-limiter entity get-limits <child_id> --resource _default_
zae-limiter resource get-defaults <resource>
# "Cascade: off (explicit)" on any of these turns cascade off for this resource

Step 4: With no policy anywhere, the entity's own flag applies:

# Check the entity's cascade setting in its metadata
aws dynamodb get-item --table-name <name> \
  --key '{"PK": {"S": "ENTITY#<child_id>"}, "SK": {"S": "#META"}}'
# Check the "cascade" attribute in response — should be true

To turn cascade on without recreating the entity, set a policy — for one resource, or for every resource of the entity — which restamps its existing buckets immediately:

zae-limiter entity set-cascade <child_id> on --resource <resource>
zae-limiter entity set-cascade <child_id> on

If the stack was deployed before 0.16.0, run zae-limiter upgrade first; on such a stack the only option is to recreate the entity with create_entity(..., cascade=True).

Verification

Test that rate limiting is working correctly:

from zae_limiter import Repository, RateLimiter, Limit, RateLimitExceeded

repo = await Repository.open()
limiter = RateLimiter(repository=repo)

# Consume all capacity
for i in range(100):
    try:
        async with limiter.acquire(
            entity_id="test-entity",
            resource="test",
            limits=[Limit.per_minute("rpm", 100)],
            consume={"rpm": 1},
        ):
            pass
    except RateLimitExceeded as e:
        print(f"Rate limited after {i} requests, retry_after={e.retry_after_seconds}s")
        break

Procedures

Adjust Limits at Runtime

Programmatic Adjustment

Update stored limits for an entity without redeployment:

from zae_limiter import Repository, RateLimiter, Limit

repo = await Repository.open()
limiter = RateLimiter(repository=repo)

# Update limits for an entity
await limiter.set_limits(
    entity_id="api-key-123",
    resource="gpt-4",  # Resource these limits apply to
    limits=[
        Limit.per_minute("rpm", 1000),  # Requests per minute
        Limit.per_minute("tpm", 100_000),  # Tokens per minute
    ],
)

print("Limits updated successfully")

Direct DynamoDB Update

Advanced

Direct DynamoDB updates bypass validation. Use programmatic API when possible.

# Update a stored limit
aws dynamodb put-item --table-name <name> \
  --item '{
    "PK": {"S": "ENTITY#<entity_id>"},
    "SK": {"S": "#LIMIT#<resource>#<limit_name>"},
    "capacity": {"N": "1000000"},
    "refill_amount": {"N": "1000000"},
    "refill_period": {"N": "60"}
  }'

Reset Bucket State

Give an entity its full allowance back on one resource, on every shard, now:

zae-limiter entity reset <entity_id> --resource <resource>
# or one limit only
zae-limiter entity reset <entity_id> -r <resource> --limit <limit>

await repo.reset_bucket("user-123", "gpt-4")

The reset is written in place: the consumption counter, the disabled flag, the shard count and the quota grant record survive, and a quota starts a new period (a session quota's window ends, so the next request opens a fresh one). Every shard takes one slow pass on its next acquire. It is audited as bucket_reset. See ADR-149.

Do not delete bucket items

Deleting a bucket item loses its consumption counter (usage snapshots go wrong), its disabled stamp and, for a quota, its grant record, and a sharded quota whose shards are not all deleted together can over-admit. Use entity reset.

Grant More Allowance

A purchase, a goodwill credit, or the difference after a plan upgrade:

zae-limiter entity top-up <entity_id> -r <resource> --add <limit>:<tokens>

A quota gains exactly that many tokens for the current period (stored as b_{limit}_tu); a dripping limit gains at most the room below its ceiling. Needs Lambdas at 0.17.0 or later for a quota. Audited as bucket_topped_up.

Debug Bucket State

Query all buckets for an entity using GSI3:

# Discover all bucket PKs via GSI3 (KEYS_ONLY)
aws dynamodb query --table-name <name> \
  --index-name GSI3 \
  --key-condition-expression "GSI3PK = :gsi3pk" \
  --expression-attribute-values '{":gsi3pk": {"S": "{ns}/ENTITY#<entity_id>"}}' \
  --output json

Then fetch individual bucket items using the PKs returned above.

Interpret bucket values:

All token values are stored as millitokens (multiply by 1000). Bucket attributes use the b_{limit_name}_{field} naming convention:

Field Attribute Description Example
Tokens b_rpm_tk Current available tokens × 1000 50000 = 50 tokens
Capacity b_rpm_cp Base bucket size × 1000, before schedule scaling and shard division 100000 = 100 tokens
Refill timestamp rf Last refill (epoch ms) 1705312800000
Shard count shard_count Number of shards for this bucket 2
Valid until vu Next instant the effective parameters change (epoch ms). Absent means nothing here is scheduled. vu at or before now is why a bucket is taking the slow path 1705334400000
Schedule sched Item-level schedule, compact encoding (leading digits are the encoding version) 1h9-17w1-5s500
Reset schedule rsched Item-level reset schedule, compact encoding 1m0h0
Timezone sched_tz IANA zone shared by every schedule on this item America/New_York
Per-limit overrides b_rpm_sched, b_rpm_rsched That limit's own schedule, when it differs from the item default. A literal - means that limit has no schedule; a missing attribute means it inherits the item default 1h0-6c2000 or -
Session window start b_session_ws A session quota's current window start (epoch ms). The window ends at ws + rsa × 1000 1705312800000
Session window applied b_session_wa The window start this shard's balance reflects. A ws later than wa means a new window has been opened and this shard has not yet restored its balance; an item written before the marker existed carries none, and rf is compared instead 1705312800000
Session window snapshot b_session_wtc The consumption counter as it stood when another shard's new window reached this one. While the window is pending (ws later than wa), the shard restores its share less whatever it consumed since, so nothing spent in between is forgiven 4000
Session window length b_session_rsa The session window's length in seconds, copied from config so readers need no config lookup 18000
Quota grant count b_rpd_gc Quotas only (calendar and session): the shard count this shard's current-period share was sized at (ADR-145). The shard's ceiling is cp // gc, and its grant covers every shard j with j % gc == this shard % gc. An item written before the attribute existed carries none and reads as gc = shard_count — unless it holds more than one share at that count, in which case it is read at the smaller count whose share covers its balance (the grant that balance implies) 2
Quota top-up b_rpd_tu Quotas only: allowance an operator topped this shard up by this period (millitokens, ADR-149). The shard's ceiling is cp // gc + tu; the next reset or window roll removes it 5000000

The ceiling actually enforced is not b_rpm_cp. The schedule in force at the current instant is applied to the base first, and the result is then divided by shard_count — or, for a quota, by b_{name}_gc. An entity on a scale: 0.5 window across 2 shards admits cp × 0.5 / 2 per shard. Compare observed rejections against that, not against the stored number.

Bucket Expiry

Bucket items carry a DynamoDB TTL, and whether they do depends on where their limits came from:

Limits resolved from TTL
Entity config, for that resource None — the item persists
Entity config, entity-wide _default_ None — the item persists
Resource defaults recovery_horizon × multiplier
System defaults recovery_horizon × multiplier
A limits= override passed to acquire() recovery_horizon × multiplier

The multiplier is bucket_ttl_refill_multiplier, default 7; set it to 0 to disable expiry everywhere.

For resource- and system-level limits this expiry is also the propagation mechanism: changing those defaults does not rewrite live buckets, so a bucket picks up the new numbers when it expires and is recreated. Entity-level changes fan out immediately and need no expiry.

The recovery horizon is the longest a limit could need to come back:

  • A limit that drips uses time-to-fill, (capacity / refill_amount) × refill_period_seconds, taken at its slowest across the base parameters and every schedule window. An entry that overrides refill_amount or refill_period_seconds to something slower therefore lengthens the TTL for the whole item.
  • A quota uses its reset period — the cycle its reset cron repeats on, rounded up (31 days for a monthly pattern, 366 for an annual one).
  • A session quota uses its reset_after exactly — 5 hours for a 5-hour window, so 35 hours at the default multiplier. An expired item simply means the next request opens a fresh window, which is what an idle entity would get anyway.

Across limits sharing one item the largest horizon wins.

Verification After Changes

After adjusting limits, verify:

# Check available capacity
available = await limiter.available(
    entity_id="<entity_id>",
    resource="<resource>",
    limits=[Limit.per_minute("<limit_name>", <capacity>)],
)

print(f"Available: {available}")
# Output: {'<limit_name>': <available_tokens>}

DynamoDB Key Patterns

Pattern Key Description
Entity metadata PK={ns}/ENTITY#<id>, SK=#META Entity configuration
Bucket state (v0.9.0+) PK={ns}/BUCKET#<id>#<resource>#<shard>, SK=#STATE Token bucket (per shard)
Entity config PK={ns}/ENTITY#<id>, SK=#CONFIG#<resource> Stored limit config
Resource config PK={ns}/RESOURCE#<resource>, SK=#CONFIG Resource default limits
System config PK={ns}/SYSTEM#, SK=#CONFIG System default limits