Skip missing metadata files gracefully instead of retrying forever

[minguyen9988]

Problem

When downloading a backup, downloadTableMetadata() retries indefinitely on missing metadata files. If a table was dropped or renamed during upload, its .json metadata file may not exist on remote storage, but the retry loop keeps trying with exponential backoff — wasting minutes before ultimately failing the entire download.

Current behavior

// pkg/backup/download.go — downloadTableMetadata()
retry := retrier.New(retrier.ExponentialBackoff(b.cfg.General.RetriesOnFailure, ...), b)
err := retry.RunCtx(ctx, func(ctx context.Context) error {
    tmReader, err := b.dst.GetFileReader(ctx, remoteMetadataFile)
    if err != nil {
        return errors.Wrapf(err, "can't GetFileReader(%s) error", remoteMetadataFile)
    }
    // ...
})

When GetFileReader returns "object doesn't exist" / "NoSuchKey" / "StatusCode 404", this error is transient-looking to the retrier, so it retries RetriesOnFailure times with exponential backoff. But a 404 is permanent — the object will never appear.

Proposed Fix

Detect permanent "not found" errors and break out of the retry loop immediately:

retry := retrier.New(retrier.ExponentialBackoff(b.cfg.General.RetriesOnFailure, ...), b)
err := retry.RunCtx(ctx, func(ctx context.Context) error {
    tmReader, err := b.dst.GetFileReader(ctx, remoteMetadataFile)
    if err != nil {
        // "object doesn't exist" is permanent — flag it and stop retrying
        if strings.Contains(err.Error(), "doesn't exist") || 
           strings.Contains(err.Error(), "key not found") || 
           strings.Contains(err.Error(), "NoSuchKey") || 
           strings.Contains(err.Error(), "StatusCode 404") {
            notFoundErr = true
            return nil // break out of retry loop
        }
        return errors.Wrapf(err, "can't GetFileReader(%s) error", remoteMetadataFile)
    }
    // ...
})

// After retry loop:
if notFoundErr {
    log.Warn().Str("remoteMetadataFile", remoteMetadataFile).
        Msg("metadata file not found on remote, skipping table")
    continue
}

When this happens

• Table dropped during backup upload (backup started, table dropped, metadata never uploaded)
• Incremental backup where base backup's metadata was cleaned up
• Partial upload failure where some tables' metadata was never written
• Table renamed between backup create and upload

Impact

Without this fix, a single missing metadata file causes the entire download to hang for RetriesOnFailure × exponential_backoff duration (typically 5-10 minutes) before failing. With this fix, the missing table is skipped in milliseconds and the rest of the backup downloads successfully.

[minguyen9988]

Reproduction Steps (against master branch, commit 1a5b06b)

Environment

• S3-compatible storage
• ClickHouse with at least one table
• clickhouse-backup built from master branch

Steps

The easiest way to reproduce is to manually remove a metadata file from remote storage after upload:

# 1. Upload a backup
clickhouse-backup create test_skip
clickhouse-backup upload test_skip

# 2. Manually remove one table's metadata from remote (simulating corruption/partial upload)
# For MinIO:
mc rm myminio/mybucket/test_skip/metadata/mydb/sometable.json
# For AWS S3:
aws s3 rm s3://mybucket/test_skip/metadata/mydb/sometable.json

# 3. Download — this table's metadata is permanently missing
clickhouse-backup download test_skip
# HANGS for RetriesOnFailure × exponential_backoff duration, then fails

This also happens naturally when a table is dropped between backup creation and download on a DR target, or when an upload was interrupted after some tables' data was uploaded but before their metadata was written.

What happens (current behavior)

The download hangs retrying the missing metadata file. With default retries_on_failure: 5 and exponential backoff starting at 500ms:

• Retry 1: wait 500ms
• Retry 2: wait 1s
• Retry 3: wait 2s
• Retry 4: wait 4s
• Retry 5: wait 8s
• Total: ~16 seconds wasted, then the ENTIRE download fails

With higher retry counts (common in production configs), this can waste minutes.

Root cause in code

pkg/backup/download.go lines 489-508 on master (inside downloadTableMetadata()):

retry := retrier.New(retrier.ExponentialBackoff(b.cfg.General.RetriesOnFailure,
    common.AddRandomJitter(b.cfg.General.RetriesDuration, b.cfg.General.RetriesJitter)), b)
err := retry.RunCtx(ctx, func(ctx context.Context) error {
    tmReader, err := b.dst.GetFileReader(ctx, remoteMetadataFile)
    if err != nil {
        // BUG: 404/NoSuchKey is treated as transient, retried forever
        return errors.Wrapf(err, "can't GetFileReader(%s) error", remoteMetadataFile)
    }
    // ...
})

The error from GetFileReader for a 404 (object not found) contains strings like "doesn't exist", "NoSuchKey", or "StatusCode 404", but the retry loop treats it as a transient error and retries with exponential backoff. A 404 is permanent — the object will never appear.

Expected behavior

WARN: metadata file not found on remote, skipping table  remoteMetadataFile=test_skip/metadata/mydb/sometable.json

The download should skip the missing table and continue with the remaining tables. This is safe because:

• If the table was dropped, there's nothing to restore
• If the upload was partial, the table's data files also won't exist
• The rest of the backup is still valid and should be downloadable

Note

There IS existing handling for .sql files (line 515-517 on master):

if err != nil && strings.HasSuffix(localMetadataFile, ".sql") {
    log.Warn().Str("localMetadataFile", localMetadataFile).Err(err).Send()
    continue
}

But this only applies to .sql metadata in embedded backups, NOT to the primary .json metadata files. The fix should detect 404 errors and break out of the retry loop for ALL metadata file types.