What
Make ParquetReader generic over FloatElem so it can read both Float32 and Float64 Parquet columns, with zero-copy for same-dtype and an Arrow cast for cross-dtype reads.
Why
ParquetReader::new rejects non-Float64 columns at schema check (readers/parquet.rs ~line 91), making f32 Parquet files completely unusable today. This blocks any f32 file source pipeline.
How
- Introduce
ParquetReader<T: FloatElem> accepting List<Float32> and List<Float64> (and FixedSizeList variants)
- Same dtype → zero-copy path; different dtype →
arrow::compute::cast with safe options (document NaN/Inf/overflow behavior)
- Apply the same policy to
ParquetStreamingReader
- New test file
qdp-core/tests/parquet_f32.rs covering: f32 column read as f32, f64 column cast to f32, unsupported column type returns InvalidInput with dtype in message
Out of scope: pipeline_runner wiring
Acceptance criteria:
What
Make
ParquetReadergeneric overFloatElemso it can read bothFloat32andFloat64Parquet columns, with zero-copy for same-dtype and an Arrow cast for cross-dtype reads.Why
ParquetReader::newrejects non-Float64 columns at schema check (readers/parquet.rs~line 91), making f32 Parquet files completely unusable today. This blocks any f32 file source pipeline.How
ParquetReader<T: FloatElem>acceptingList<Float32>andList<Float64>(and FixedSizeList variants)arrow::compute::castwith safe options (document NaN/Inf/overflow behavior)ParquetStreamingReaderqdp-core/tests/parquet_f32.rscovering: f32 column read as f32, f64 column cast to f32, unsupported column type returnsInvalidInputwith dtype in messageOut of scope: pipeline_runner wiring
Acceptance criteria:
ParquetReader::<f32>succeeds on f32 columnInvalidInputerror with column dtype in messageqdp-core/tests/parquet_f32.rs