Binlog Archiving

Tiering Archived Binlogs with S3 Lifecycle Rules

The lifecycle rule that saves the most money is also the one most likely to delete your recovery capability, and the two failure modes are subtle: a Transition that pushes the recovery window into Glacier adds hours to RTO, and an Expiration that fires inside the recovery window silently amputates the chain. This page writes S3 lifecycle rules that tier archived binlogs down for cost while keeping every segment the recovery window needs both present and fast, using precise prefix scoping, a guarded expiration that stays behind the base-backup overlap, and explicit noncurrent-version handling so a versioned bucket does not quietly retain a second copy of everything. It is the concrete implementation of the policy in Object Storage Lifecycle & Cost Management for Archived Binlogs.

Context & Prerequisites

This implements the guarded tiering from Object Storage Lifecycle & Cost Management; read that for the recovery-horizon inequality this must not violate. You need an S3 bucket holding the archive laid out per AWS S3 & GCS Sync Pipelines, Python 3.10+ with boto3, and IAM permission for s3:PutLifecycleConfiguration. The expiry days must exceed your base-backup overlap window from Base Backup Integration for PITR.

Step-by-Step Implementation

1. Scope the rule to the segment prefix only

Filter the rule to the segments/ prefix so it never touches manifests or metadata.

{
  "Rules": [{
    "ID": "binlog-segments-tiering",
    "Filter": { "Prefix": "segments/" },
    "Status": "Enabled",
    "Transitions": [
      { "Days": 30, "StorageClass": "STANDARD_IA" },
      { "Days": 90, "StorageClass": "GLACIER" }
    ],
    "Expiration": { "Days": 400 }
  }]
}

PITR relevance: scoping to segments/ keeps the manifest — the index a recovery reads first — permanently in fast storage, so resolving a recovery plan is never delayed by a cold retrieval.

2. Set expiry behind the base-backup overlap, with margin

Choose Expiration.Days strictly greater than the oldest base backup you honor plus a safety margin, so no expiry can outrun the chain.

# Python 3.10+
def safe_expiry_days(oldest_base_backup_age_days: int, margin_days: int = 30) -> int:
    # Expiry must never fire before the oldest recoverable backup + margin.
    return oldest_base_backup_age_days + margin_days

PITR relevance: deriving expiry from the actual backup horizon — not a round number — is what guarantees the archive always overlaps every recoverable base backup.

3. Keep the recovery window in a fast tier

Ensure the Days before the first Transition exceeds your RTO-scoped recovery window, so no segment you might replay under RTO lives in Glacier.

# Python 3.10+
def first_transition_days(recovery_window_days: int, rto_buffer_days: int = 3) -> int:
    return recovery_window_days + rto_buffer_days

PITR relevance: a recovery that must fetch a Glacier segment waits minutes to hours; keeping the whole RTO window in Standard/IA keeps retrieval off the critical path.

4. Handle noncurrent versions on versioned buckets

If the bucket is versioned, add a NoncurrentVersionExpiration so superseded objects (e.g. a re-archived truncated segment) do not accumulate cost forever.

{
  "ID": "binlog-noncurrent-cleanup",
  "Filter": { "Prefix": "segments/" },
  "Status": "Enabled",
  "NoncurrentVersionExpiration": { "NoncurrentDays": 30 }
}

PITR relevance: the current version is always the authoritative segment; noncurrent versions are superseded copies safe to expire after a short retention, and leaving them unmanaged doubles storage silently.

Configuration Snippet & Reference Table 

# Python 3.10+  — apply the lifecycle configuration
import boto3, json

def put_lifecycle(bucket: str, rules: dict) -> None:
    boto3.client("s3").put_bucket_lifecycle_configuration(
        Bucket=bucket, LifecycleConfiguration=rules)
Rule elementValueWhy it matters for PITR
Filter.Prefixsegments/Keeps manifests in fast storage.
first Transition.Days> RTO windowRecovery window stays fast.
GLACIER transitioncompliance tailCheap retention beyond RTO scope.
Expiration.Days> base-backup overlap + marginNever amputates the chain.
NoncurrentVersionExpirationshort (e.g. 30d)Stops superseded copies accruing cost.

Verification Checklist

Gotchas & Version-Specific Caveats

Expiration is asynchronous and silent. S3 may take up to 48 hours after the expiry threshold to delete, and it emits no failure — so an over-aggressive rule removes objects with no error. Only manifest reconciliation catches the resulting gap.

Glacier retrieval is not instant. A GLACIER (or Deep Archive) object requires a restore request and wait before it can be read; never place the RTO-scoped window there.

Lifecycle applies to current versions by default. On a versioned bucket, Expiration and Transition act on current versions; noncurrent copies need their own NoncurrentVersion* actions or they linger.

Object Lock can block expiry. If segments are under a retention lock for compliance, expiry cannot remove them until the lock elapses — intended, but budget for the storage until then.

Frequently Asked Questions

How do I pick the expiry days without risking the chain?

Derive it, do not guess it. Take the age of the oldest base backup you intend to recover from, add a safety margin (a week or a month), and set Expiration.Days to that. Because the number tracks your actual backup retention, the archive always overlaps every recoverable backup. Re-derive it whenever backup retention changes, and let manifest reconciliation catch any drift.

Should the manifest be tiered too?

No. Keep the manifest in Standard storage permanently. It is tiny relative to the segments and it is the first thing a recovery reads to resolve which objects to fetch — putting it in a cold tier would add retrieval latency to every recovery for negligible savings. Scope lifecycle rules to the segments/ prefix so the manifest is untouched.

Back to Object Storage Lifecycle & Cost Management.