Dynamic file loading

[derekperkins]

I'm working on a few different ways to get data loaded into DuckDB from other data sources. One idea I had was to dynamically generate parquet files. By having some sort of path params in the file name, I could get data from an arbitrary data source, package it as a parquet file, then send it across the wire.

data:
  walk:
    file: https://my-api.com/{tenant_id}-{start_date}-{end_date}-{type}.parquet

Would it be make sense to add arrow ipc support for performance reasons?
https://duckdb.org/2025/05/23/arrow-ipc-support-in-duckdb.html

[jheer]

Great question. In Mosaic, the main use case for Arrow has been for fast, unidirectional interoperability by sending query results to JavaScript as Arrow buffers. We've been more inclined to use Parquet for storage and loading of external data for DuckDB itself. For larger datasets being sent over the wire, I suspect that Parquet's compression will win out in terms of overall performance / experience. But I haven't tested this. I'd be curious to see actual performance numbers, including for small-to-medium size data, comparing the two.