Handling S3 Throttling During High-Throughput Binlog Archiving #

A high-throughput MySQL 8.0 instance can rotate binary logs at multiple gigabytes per minute during peak OLTP windows, and when that stream is pushed to object storage the archiver eventually trips a partition-level request ceiling. The failure signature is unambiguous: Amazon S3 returns HTTP 503 SlowDown, while Google Cloud Storage returns 429 RateLimitExceeded or 503 BackendError. These are not transient network anomalies — they are explicit capacity signals telling you that request velocity has saturated the storage partition or the egress quota. Left unhandled, a default HTTP client retries the 503 immediately, compounds the failure into a request storm, forces local disk retention past its safe threshold, and ultimately fractures the continuous recovery chain that Point-in-Time Recovery depends on. This page resolves that exact scenario: how to detect throttling, pace the uploader so it stops provoking it, and enforce backpressure so a 503 window degrades gracefully instead of dropping segments.

Visual Overview #

Context & Prerequisites #

Throttling is a failure mode of the transport layer, so this page assumes the surrounding machinery already exists and focuses only on the rate-limit scenario. The retry classification, idempotency gating, and dead-letter routing that decide whether a throttled segment recovers or vanishes are defined in the parent cluster, Error Handling & Retry Logic — a 503 SlowDown must be classified as transient/retryable there, never as fatal. The provider-abstracted upload path that actually issues the PutObject calls lives in AWS S3 & GCS Sync Pipelines, and the bounded worker pool that feeds it is covered in Async Processing & Queue Management. You need boto3>=1.28 (or google-cloud-storage), Python 3.10+, and a MySQL 8.0 source with performance_schema enabled so generation velocity can be measured against upload acknowledgment rate.

Step-by-Step Implementation #

1. Isolate the bottleneck before tuning anything #

The primary diagnostic for storage throttling is a measurable divergence between binlog rotation cadence and successful upload acknowledgments. Enable wire-level SDK tracing so the exact retry headers and request identifiers are captured:

# boto3 / botocore debug tracing
export AWS_SDK_LOAD_CONFIG=1
export BOTO3_LOG_LEVEL=DEBUG

For GCS pipelines, attach a debug handler to the client namespace:

import logging

logging.getLogger("google.cloud").setLevel(logging.DEBUG)

Cross-reference the exposed x-amz-request-id / x-goog-request-id values against provider metrics (5xxErrors, ThrottledRequests) to prove where the constraint lives — bucket prefix, VPC endpoint egress, or IAM rate limit. On the database side, establish a baseline generation velocity so you can quantify the gap. PITR relevance: this delta is the leading indicator of a forming recovery hole — once local rotation outruns remote acknowledgment past your buffer, unarchived segments are one purge away from being unrecoverable.

-- MySQL 8.0.0+ : bytes written to the binlog since server start
SELECT SUM_NUMBER_OF_BYTES_WRITE
FROM performance_schema.file_summary_by_event_name
WHERE EVENT_NAME = 'wait/io/file/sql/binlog';

2. Classify the throttle response deterministically #

A retry controller must react to 503 SlowDown differently from a 403 AccessDenied. Route the response through an explicit classifier using structural pattern matching so only genuine capacity signals are paced-and-retried:

from enum import Enum

class UploadDisposition(Enum):
    RETRY_PACED = "retry_paced"     # throttle: back off, respect Retry-After
    RETRY_PLAIN = "retry_plain"     # generic 5xx: normal backoff
    FATAL = "fatal"                 # auth/policy: escape immediately

def classify(status: int, code: str) -> UploadDisposition:
    # MySQL binlog archiving: throttle codes across S3 and GCS
    match (status, code):
        case (503, "SlowDown") | (429, _) | (503, "BackendError"):
            return UploadDisposition.RETRY_PACED
        case (500 | 502 | 504, _):
            return UploadDisposition.RETRY_PLAIN
        case (401 | 403, _):
            return UploadDisposition.FATAL
        case _:
            return UploadDisposition.RETRY_PLAIN

PITR relevance: misclassifying 503 SlowDown as fatal dead-letters a perfectly recoverable segment; misclassifying 403 as retryable hammers a broken credential 12 times before anyone is paged. Both leave gaps.

3. Configure adaptive, server-aware retries #

Fixed exponential backoff is insufficient for sustained high-throughput workloads because it ignores the server’s own Retry-After guidance. Override the default botocore configuration with adaptive mode, which throttles the client based on observed service feedback:

import boto3
from botocore.config import Config

session = boto3.Session()
s3_client = session.client(
    "s3",
    config=Config(
        retries={"max_attempts": 12, "mode": "adaptive"},
        max_pool_connections=25,
    ),
)

For GCS, use google.api_core.retry with jitter so a fleet decorrelates rather than retrying in synchronized waves:

from google.api_core import retry

retry_policy = retry.Retry(
    initial=1.0,
    maximum=30.0,
    multiplier=2.0,
    predicate=retry.if_transient_error,
    deadline=120.0,
)

PITR relevance: adaptive mode prevents the retry storm that turns a two-second throttle into a multi-minute outage, keeping upload latency bounded and the recovery chain contiguous.

4. Shard the key prefix to spread partition load #

S3 scales throughput per key prefix. Archiving every segment to a flat bucket path guarantees all requests land on one partition and contend for one ceiling. Route segments across independent partitions using date/hour granularity so the provider allocates capacity per shard:

from datetime import datetime, timezone

def sharded_key(server_uuid: str, binlog_name: str) -> str:
    now = datetime.now(timezone.utc)
    # e.g. mysql-binlogs/<uuid>/2026/07/04/08/mysql-bin.000042.zst.enc
    return (
        f"mysql-binlogs/{server_uuid}/"
        f"{now:%Y/%m/%d/%H}/{binlog_name}.zst.enc"
    )

Keep the key deterministic and derivable from the segment so idempotent re-runs resolve to the same path. PITR relevance: the compress/encrypt transform that produces the .zst.enc suffix is owned by Compression & Encryption Workflows; seal each segment once before the retry loop so every paced retry uploads byte-identical content and the stored checksum never drifts.

5. Enforce bounded-queue backpressure #

The decisive control is architectural: decouple MySQL I/O generation from upload velocity with a strictly bounded queue, so a sustained 503 window applies backpressure upstream instead of exhausting the heap.

import asyncio
from dataclasses import dataclass

@dataclass(slots=True)
class BinlogSegment:
    filename: str
    size_bytes: int
    path: str

class ArchiverPipeline:
    def __init__(self, max_queue_depth: int = 50, concurrency: int = 8):
        self.queue: asyncio.Queue[BinlogSegment] = asyncio.Queue(maxsize=max_queue_depth)
        self.semaphore = asyncio.Semaphore(concurrency)  # cap concurrent uploads

    async def enqueue(self, segment: BinlogSegment) -> None:
        if self.queue.full():
            # Hard backpressure: pause rotation cadence / raise an alert,
            # never silently drop the segment.
            raise RuntimeError("Upload queue saturated — throttling detected.")
        await self.queue.put(segment)

    async def drain(self) -> None:
        async def process_one(seg: BinlogSegment) -> None:
            async with self.semaphore:
                await asyncio.to_thread(self._upload_sync, seg)

        tasks = []
        while not self.queue.empty():
            tasks.append(asyncio.create_task(process_one(self.queue.get_nowait())))
        if tasks:
            await asyncio.gather(*tasks)

    def _upload_sync(self, segment: BinlogSegment) -> None:
        """Synchronous S3/GCS PutObject, invoked from a thread pool."""
        raise NotImplementedError

PITR relevance: a bounded queue makes throttling visible and safe — the pipeline signals backpressure at a known depth rather than accumulating unarchived segments in memory until the process is OOM-killed and the recovery timeline is cut mid-stream.

Configuration Reference #

Minimal, copy-pasteable knobs for the throttling scenario. Reduce concurrency first when a 503 window opens; widen it only after acknowledgment latency recovers.

# my.cnf — MySQL 8.0.1+ : retention wide enough to survive a throttle window
[mysqld]
binlog_expire_logs_seconds = 172800   # 48h, ≥ 2× worst-case archive lag
max_binlog_size            = 1073741824

Verification Checklist #

• Throttle responses (503 SlowDown, 429) are classified RETRY_PACED, never FATAL, and never max_attempts=1.
• retries.mode is adaptive (S3) or jittered google.api_core.retry (GCS) — confirmed in captured wire logs.
• Object keys are prefix-sharded by date/hour and resolve deterministically for idempotent re-runs.
• The upload queue is bounded and raises visible backpressure at max_queue_depth instead of dropping segments.
• ThrottledRequests / 5xxErrors decay to zero after concurrency is reduced.
• Local disk usage stays below 85% for the duration of the throttle window.
• Every archived object passes mysqlbinlog --verify-binlog-checksum after download.
• A dry-run replay to a staging MySQL 8.0 instance confirms binlog sequence continuity and GTID consistency across the throttled interval.

Gotchas & Version-Specific Caveats #

• Never disable retries or set max_attempts=1. A silently failed upload during a throttle window creates an unrecoverable PITR gap that surfaces only when a recovery drill fails.
• Do not confuse CPU latency with network throttling. Compression and encryption can stall the upload thread and mimic a 503 backlog; prefer zstd or lz4 over gzip, and encrypt in streaming AES-256-GCM so multi-gigabyte segments never load fully into memory. Sealing per-retry also breaks idempotency by producing a new checksum each attempt.
• Never block the event loop. Wrap synchronous PutObject calls in asyncio.to_thread() or a ThreadPoolExecutor; a blocking call inside the loop starves every other upload and worsens the backlog.
• botocore adaptive vs standard mode: standard retries the correct error set but does not rate-limit the client, so it will still storm a throttled partition. Only adaptive applies client-side token-bucket throttling — use it for this scenario specifically.
• MySQL 8.0 vs 8.4: binlog_expire_logs_seconds is the retention control on both; the legacy expire_logs_days was deprecated in 8.0 and removed in 8.4, so a config carried forward from 5.7 will fail to start on 8.4. Size retention off worst-case archive lag, not a fixed day count.
• Prefix migration is a live remedy, not a permanent one. Moving active uploads to a fresh prefix forces new partition capacity and clears an acute throttle, but only sustained prefix sharding prevents recurrence.

Frequently Asked Questions #

Related #

• Error Handling & Retry Logic — the parent classification, idempotency, and dead-letter layer this scenario plugs into.
• AWS S3 & GCS Sync Pipelines — the provider-abstracted upload path that issues the throttled requests.
• Building a Python Script to Sync Binlogs to S3 with Boto3 — a focused single-cloud uploader with multipart and credential tuning.
• Async Processing & Queue Management — the bounded worker pool that applies the backpressure described here.
• The Binary Log — MySQL 8.0 Reference Manual — canonical documentation for segment rotation and status surfaces.

Back to Error Handling & Retry Logic.