feature: add more object storage backends#346
Conversation
QuakeWang
left a comment
There was a problem hiding this comment.
LGTM
@liujiwen-up please fix the CI check
| Ok(Operator::new(builder)?.finish()) | ||
| } | ||
|
|
||
| fn normalize_config(props: HashMap<String, String>) -> HashMap<String, String> { |
There was a problem hiding this comment.
nit: The config normalization and mirrored-key logic are duplicated across the backend modules. Maybe a small shared helper could be useful if we keep adding storage backends.
|
Non-blocking: could we add a test for listing an ADLS filesystem root without a trailing slash, e.g. |
| feature = "storage-gcs", | ||
| feature = "storage-obs" | ||
| ))] | ||
| fn bucket_and_relative_path<'a>( |
There was a problem hiding this comment.
nit: This generalized bucket_and_relative_path(storage_name, allowed_schemes) duplicates the logic of the existing OSS/S3-specific version above. Consider unifying into one method — OSS/S3 callers just pass their own allowed_schemes slice.
Purpose
Linked issue: close #xxx
Add OpenDAL-backed support for more object storage backends in Paimon Rust, covering COS, Azure Data Lake Storage Gen2, OBS, and GCS.
Brief change log
Added storage feature flags for storage-cos, storage-azdls, storage-obs, and storage-gcs.
Added storage-all coverage for the new backends.
Implemented backend config parsing and OpenDAL operator construction for:
Added scheme aliases such as cosn, gs, abfs, and abfss.
Added object storage FileIO path tests to verify scheme detection and relative path handling.
Extended GCS config alias support for Hadoop/OpenDAL-style option keys.
Updated docs to list and show usage examples for the new storage backends.
Tests
API and Format
This change adds optional Cargo feature flags and storage backend support. It does not change the Paimon table format or existing storage format.
Documentation
Updated documentation to include the newly supported storage backends and basic configuration examples.