Manifests table scan should return iceberg schema rather arrow schema

[liurenjie1024]

          I think we should return iceberg schema here, and user could easily convert it to arrow schema.

Originally posted by @liurenjie1024 in #861 (comment)

[Xuanwo]

Makes sense to me.

[flaneur2020]

👍 let me fix this

[rshkv]

Also happy to help with this but not sure why / if we want this.

Looking at the DataFusion integration, it seems useful for the metadata TableProvider to be able to provide the Arrow schema.

If we need to expose the Arrow schema for 1) the DataFusion integration and 2) to build RecordBatch internally, it might be preferable for the metadata tables to have schema() return Arrow. If necessary, clients can still convert to Iceberg. Unlike the other way round, where we know conversion would be necessary.

[liurenjie1024]

The reason I suggest returning iceberg schema is that metadata table is a concept in iceberg library, not only in datafusion integration. The difference is that, iceberg library will be used by more engines like datafuse, polars, etc. The reason we provide an record batch stream is for convenience, IMO we should provid similar plan files api so that other engines could consume it, but since arrow is the defact standard for in memory data exchange, I'm fine with keeping the scan api.