Skip to content

Design Records

Design records document the engineering decisions behind PGSTY SILO: root causes, S3 and IAM protocol invariants, trade-offs against rejected alternatives, operational limits, and regression coverage.

November 9: Console Languages

November 6: Found in Release Acceptance

November 4: Release-Blocking Lifecycle Fixes

  • Multi-Pool Lifecycle Expiry: Decide on the Whole Object (#278, #279): Why an expiry decided on the versions of one pool hides, deletes, or resurrects versions in another pool, and why a recheck on the merged version list under the object lock is the minimum complete repair.
  • Restored Copies: Expire the Copy, Not the Object (#281, #282): Why the expiry of a restored current version hid the object or stripped an older null version, and which storage and native-restore safeguards ship with the exact-version fix.
  • Native Restore in Two Phases (#284): Why a native restore held the object lock for its whole tier download, why the first #281 safeguards also stranded failed ordinary uploads, and how staging outside the lock with a short, checked commit replaces the private upload.
  • Audit Webhooks After a Peer Reload (#259): Why a runtime audit_webhook change leaves every server but the serving one with a stopped target, which logger subsystems share it, and why building logger targets on the server context plus opening the queue store first is the minimum complete repair.
  • Failed Writes and Phantom Empty Prefixes (#285, #286): Why a write whose staged data vanished before the remote rename leaves an empty object directory that later lists as an empty prefix, and why removing it on both rename failure paths is sufficient.
  • Heal Dry Run Must Not Delete Data (from #280): Why a dry-run heal could delete the last metadata copies of a dangling object and run a real bucket heal on remote nodes, and why an early return plus forwarding the dry-run flag to peers closes both paths.
  • Lifecycle Delete-All Must Stay on One Key (found during review of #278): Why ExpiredObjectAllVersions, DelMarkerExpiration, and the default batch-expiration purge delete every other key under the expired key’s name, and why purging the key’s own versions by ID after a recheck is the minimum safe repair.
  • Bucket Quota in the Console: List Display and Numeric Validation (silo-console#72, silo-console#73): Why a Console-set quota showed as “-” in the bucket list, why bucket creation could turn a huge quota into one byte, where both came from, and what the merged Console fix covers.
  • Lock Domains of Large Single-Pool Clusters: Why DELETE and CopyObject fail to exclude PUT on the same key in single-pool clusters of 18 or more hosts, which layouts that covers, and the routing change that closes it.

November 1: Pool Migration Integrity Review

October 24: FULL_OBJECT Checksum Replication

October 24: Multi-Pool Deletion and Placement

  • Multi-Pool Deletion: Preserve the Latest Copy on Failure (#256, #257): Why an online-drive preflight cannot protect against later I/O failures, how staged cleanup preserves each item’s authoritative copy, and the retry, tier-reference, performance, and compatibility boundaries of the merged fix.

  • Overwrite Pool Affinity: Why #245 Was Not Adopted (#245): The original placement problem, request-aware quorum and health improvements, unversioned-only scope, two rounds of m3 performance measurements, and the decision to stop after small-object overwrite regressions remained. The experimental changes were not merged.

October 23: Alias Regions

  • Alias regions: Persistent per-alias regions, client-cache isolation, and upgrade, downgrade and credential-rotation boundaries, reconciled with mcli 20261109.

October 23 IAM & KMS Design Notes

  • LDAP Long DNs: Restoring Login Without Hiding Revocation Errors (#250): How \xx hex escaping in NormalizeDN triggers the 255-byte single-component filename limit on the object IAM backend, and why loadIAMConfigBytesWithMetadata classifies only impossible pure-ASCII path components (> 255 bytes) as errConfigNotFound.
  • IAM Policy Reloads: Lock Scope, Cancellation, and Time Budgets (#251): Moving policy reload and KMS decryption outside the authentication-cache lock while retaining writeMu, threading request cancellation end-to-end to KES.Decrypt, and keeping etcd 30-second RPC deadlines separate from total decryption budgets.

Server 20261018 Key Design Notes

For the Server 20261018 release, see these core architecture notes:

  • IAM Policy Reloads: Lock Scope, Cancellation, and Time Budgets

    In Design

    DesignIAMKMSS3Review

    IAM Policy Reloads: Lock Scope, Cancellation, and Time Budgets

    Commit 1b5477f3d (pgsty/silo#251) resolves two coupled failure modes under slow KMS/KES responses: Authentication-cache lock contention: ListPolicies and ListPolicyDocs now load and decrypt policy replacements outside the in-memory …

    Commit 1b5477f3d (pgsty/silo#251) resolves two coupled failure modes under slow KMS/KES responses: Authentication-cache lock contention: ListPolicies and ListPolicyDocs now load and decrypt policy replacements outside the in-memory …

  • Alias regions: a small configuration change with compatibility boundaries

    In Design

    DesignmcliS3Compatibility

    Alias regions: a small configuration change with compatibility boundaries

    Decision, 2026-10-23: mcli PR #44 is merged on main at 91006a587. We accepted the per-alias region design, kept its small production implementation and supplemented the regression coverage. This records the original design decision. The capability is …

    Decision, 2026-10-23: mcli PR #44 is merged on main at 91006a587. We accepted the per-alias region design, kept its small production implementation and supplemented the regression coverage. This records the original design decision. The capability is …

  • SILO, KES and OpenBao: Key Ownership and Recovery

    In Design

    DesignKESSecurityOperations

    SILO, KES and OpenBao: Key Ownership and Recovery

    SILO integrates with external key-management systems through PGSTY KES (MINIO_KMS_KES_ENDPOINT). The current release is KES 20261026 (built with Go 1.27.3 and updated cloud dependencies while retaining the API, configuration, and key formats …

    SILO integrates with external key-management systems through PGSTY KES (MINIO_KMS_KES_ENDPOINT). The current release is KES 20261026 (built with Go 1.27.3 and updated cloud dependencies while retaining the API, configuration, and key formats …

  • LIST Recovery: Confirm Only Uncertain Names and Bound the Waiting Cost

    In Design

    DesignS3ListObjectsPerformanceReview

    LIST Recovery: Confirm Only Uncertain Names and Bound the Waiting Cost

    Shipped in SILO Server 20261018, PR #239 (merged as fbcb51197, closing #218 and retaining mr javad seydi’s original signed commits from #221) and the October 18 directory/common-prefix follow-ups (3dd53eb81 and 2080b3d03) resolve silent object …

    Shipped in SILO Server 20261018, PR #239 (merged as fbcb51197, closing #218 and retaining mr javad seydi’s original signed commits from #221) and the October 18 directory/common-prefix follow-ups (3dd53eb81 and 2080b3d03) resolve silent object …

  • IAM Admin API Authorization Boundaries

    In Design

    DesignIAMSecurityCompatibility

    IAM Admin API Authorization Boundaries

    Server 20261018 ships Server PR #242 (commit 2616d43fb on baseline 00a375c4e, merged as 4e55fc7db) and the bulk self-listing follow-up 995439260 to close SN-2026-017. This design note records the problem scope, handler-level authorization contract, …

    Server 20261018 ships Server PR #242 (commit 2616d43fb on baseline 00a375c4e, merged as 4e55fc7db) and the bulk self-listing follow-up 995439260 to close SN-2026-017. This design note records the problem scope, handler-level authorization contract, …

  • Clearing Stale Fields in Legacy Metadata and Scanner Responses

    In Design

    DesignReliabilityStorageScanner

    Clearing Stale Fields in Legacy Metadata and Scanner Responses

    This article records the design and verification of PR #233 (tested head 86c5a92eef3d, merged as d94b3b3c9251), which shipped in Server 20261018. It fixes stale fields carried over when legacy xl.meta version headers and remote namespace-scanner …

    This article records the design and verification of PR #233 (tested head 86c5a92eef3d, merged as d94b3b3c9251), which shipped in Server 20261018. It fixes stale fields carried over when legacy xl.meta version headers and remote namespace-scanner …

  • Object Lock Retention Bypass and Version-Overwrite Repair Design (SN-2026-015 / SN-2026-016)

    In Design

    SILOSecurityObject LockWORMS3IAM

    Object Lock Retention Bypass and Version-Overwrite Repair Design (SN-2026-015 / SN-2026-016)

    This document records the verification, Git provenance, repair design, critical-path evaluation, and candidate trade-offs for two Object Lock (WORM) security vulnerabilities identified through external reports and an internal write-path audit: …

    This document records the verification, Git provenance, repair design, critical-path evaluation, and candidate trade-offs for two Object Lock (WORM) security vulnerabilities identified through external reports and an internal write-path audit: …

  • Pinned TLS Parameters and Handshake Compatibility

    In Design

    DesignTLSOIDCOperationsCompatibility

    Pinned TLS Parameters and Handshake Compatibility

    After the Go TLS default repair 48e184652 shipped in Server 20260916, the reporter of issue #154 retested on that release and still saw connection reset by peer on OIDC discovery. Meanwhile, upstream Go issues reported the exact same symptom and …

    After the Go TLS default repair 48e184652 shipped in Server 20260916, the reporter of issue #154 retested on that release and still saw connection reset by peer on OIDC discovery. Meanwhile, upstream Go issues reported the exact same symptom and …

  • A Listing Must Not Drop a Null Version That Still Has Quorum

    In Design

    DesignS3ListObjectsReview

    A Listing Must Not Drop a Null Version That Still Has Quorum

    This record covers the listing omission investigated on 2026-09-16: an unversioned bucket under concurrent overwrites returned HTTP 200 listings that lacked keys the same endpoint could still GET. It describes the confirmed mechanism, the targeted …

    This record covers the listing omission investigated on 2026-09-16: an unversioned bucket under concurrent overwrites returned HTTP 200 listings that lacked keys the same endpoint could still GET. It describes the confirmed mechanism, the targeted …

  • Go 1.27 TLS Defaults and OIDC Discovery Failure Modes

    In Design

    DesignTLSOIDCOperations

    Go 1.27 TLS Defaults and OIDC Discovery Failure Modes

    After SILO’s toolchain moved to Go 1.27, the Server TLS repair 48e184652 (“fix(tls): honor Go key exchange defaults across transports”, shipped in Server 20260916) removed explicit curve overrides across eight Server TLS configuration points. This …

    After SILO’s toolchain moved to Go 1.27, the Server TLS repair 48e184652 (“fix(tls): honor Go key exchange defaults across transports”, shipped in Server 20260916) removed explicit curve overrides across eight Server TLS configuration points. This …

  • Request-Header Deadlines: Absolute Header Limits and Rolling Body Idle Timeouts

    In Design

    DesignHTTPS3Review

    Request-Header Deadlines: Absolute Header Limits and Rolling Body Idle Timeouts

    This article records the repair of Server’s HTTP read deadlines, merged in PR #196 (fix 055030ea5) and shipped in Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a). The investigation was prompted by a slow-HTTP DoS scanner report (#183) and …

    This article records the repair of Server’s HTTP read deadlines, merged in PR #196 (fix 055030ea5) and shipped in Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a). The investigation was prompted by a slow-HTTP DoS scanner report (#183) and …

  • Replica Metadata Normalization: What a Trusted Copy May Not Re-Inject

    In Design

    DesignReplicationS3Review

    Replica Metadata Normalization: What a Trusted Copy May Not Re-Inject

    Shipped in SILO Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a), PR #194 (commit 4fcdf37ce, merged as 9f3037e941, retaining Mikhail Khadarenka’s original implementation from PR #187) fixes a regression where a trusted replication receiver …

    Shipped in SILO Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a), PR #194 (commit 4fcdf37ce, merged as 9f3037e941, retaining Mikhail Khadarenka’s original implementation from PR #187) fixes a regression where a trusted replication receiver …

  • Replicated Tag Ordering: Revision Timestamps, Tombstones, and Resurrection

    In Design

    DesignReplicationS3Review

    Replicated Tag Ordering: Revision Timestamps, Tombstones, and Resurrection

    Shipped in SILO Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a), PR #193 (commit 03027727d) and PR #196 (commit 680eac66e) fix two defects in how object tags preserve ordering across bucket replication: SSE-KMS destination copies silently …

    Shipped in SILO Server 20260916 (40220bd836cbd066ca424fa4dc5dbb90057fb55a), PR #193 (commit 03027727d) and PR #196 (commit 680eac66e) fix two defects in how object tags preserve ordering across bucket replication: SSE-KMS destination copies silently …

  • SSE-C Replica Integrity

    In Design

    DesignS3Operations

    SSE-C Replica Integrity

    This article records the SSE-C replica integrity contract formed by #122, #123, #124, #126, and #134, which shipped in Server 20260916, building on the earlier zero-byte/read-attribute authentication and destination-key checksum repairs from Server …

    This article records the SSE-C replica integrity contract formed by #122, #123, #124, #126, and #134, which shipped in Server 20260916, building on the earlier zero-byte/read-attribute authentication and destination-key checksum repairs from Server …

  • Object Lock Replication Ordering

    In Design

    DesignS3Operations

    Object Lock Replication Ordering

    Initial Object Lock replication timestamp ordering (f4c1286c9) shipped in SILO Server 20260903 (recorded in the security advisory ledger). Subsequent fixes for timestamp-only removals, SSE-C replica retransmission, and multi-pool Object Lock …

    Initial Object Lock replication timestamp ordering (f4c1286c9) shipped in SILO Server 20260903 (recorded in the security advisory ledger). Subsequent fixes for timestamp-only removals, SSE-C replica retransmission, and multi-pool Object Lock …

  • Multi-Pool Object Consistency

    In Design

    DesignS3Operations

    Multi-Pool Object Consistency

    Shipped in SILO Server 20260916, PR #178 repairs multi-pool mutation ordering, PR #207 extends current-object selection to conditional PUT (closing #133 and #144), and commit fced86303 preserves object tags across pool rebalance and decommission. …

    Shipped in SILO Server 20260916, PR #178 repairs multi-pool mutation ordering, PR #207 extends current-object selection to conditional PUT (closing #133 and #144), and commit fced86303 preserves object tags across pool rebalance and decommission. …

  • Federated CopyObject: Preserve the Destination Contract

    In Design

    DesignS3Operations

    Federated CopyObject: Preserve the Destination Contract

    Server 20260916 repairs the legacy etcd bucket-federation CopyObject path across PR #157, PR #159, PR #163, PR #177, and PR #179. When CopyObject forwards a copy across federated deployments as a remote PutObject, the proxy streams logical source …

    Server 20260916 repairs the legacy etcd bucket-federation CopyObject path across PR #157, PR #159, PR #163, PR #177, and PR #179. When CopyObject forwards a copy across federated deployments as a remote PutObject, the proxy streams logical source …

  • Durable IAM Revocations

    In Design

    DesignS3Operations

    Durable IAM Revocations

    SILO retains the revision of every deleted IAM record so an offline site cannot resurrect a deleted identity or policy grant when it reconnects. This mechanism covers built-in users, service accounts, groups, policy documents, and policy mappings …

    SILO retains the revision of every deleted IAM record so an offline site cannot resurrect a deleted identity or policy grant when it reconnects. This mechanism covers built-in users, service accounts, groups, policy documents, and policy mappings …

  • Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence

    In Design

    DesignReplicationS3

    Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence

    #77 is a site-replication correctness defect repaired in Server 20260916 (PR #180, commit 48ec10312). Before this repair, a receiving site overwrote the peer’s source timestamp with local arrival time (UTCNow) and could then reject a genuinely newer …

    #77 is a site-replication correctness defect repaired in Server 20260916 (PR #180, commit 48ec10312). Before this repair, a receiving site overwrote the peer’s source timestamp with local arrival time (UTCNow) and could then reject a genuinely newer …

  • Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation

    In Design

    DesignReplicationS3Review

    Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation

    This article records the root-cause analysis, design tradeoffs, and validation across three rounds of replication reliability repairs shipped in Server 20260916: Round 1 (PR #162, merged as d1105bbb): addresses #153, #152, and #137, covering …

    This article records the root-cause analysis, design tradeoffs, and validation across three rounds of replication reliability repairs shipped in Server 20260916: Round 1 (PR #162, merged as d1105bbb): addresses #153, #152, and #137, covering …