How to read multiple files?

[eitsupi]

It would be great if the syntax for reading multiple files as a single DataFrame in place.

[drizk1]

To make sure I understand, would the goal be to give multiple file paths and then they are all appended/stacked into the same dataframe?

[eitsupi]

I want something like these:
https://duckdb.org/docs/data/multiple_files/overview.html

I gave up using Julia for today because there didn't seem to be an easy way to do this (of course duckdb would do it, but it didn't seem like there was any point in using Julia if using duckdb)

[drizk1]

We can definitely get something like the below in place for you

read_csv(['flights1.csv', 'flights2.csv'], arguments = here);

[drizk1]

As I get that added for you hopefully within the week, the only other thing I'll mention is that if you'd like you can use TidierDB which give you TidierData and Tidier Syntax on duckdb

[eitsupi]

I'll mention is that if you'd like you can use TidierDB which give you TidierData and Tidier Syntax on duckdb

Thank you. Of course I tried installing it, but it seemed like too many dependencies to use for the duckdb read function only.

Perhaps it would be better to use package extensions and not make it the default dependency except for dependencies that are used by any backend?

[drizk1]

Moving to package extensions is definitely something I want to do to lighten up the dep, I just haven't found the time to do yet unfortunately.

Will definitely get multiple file paths up for you soon tho

[drizk1]

Alright, v.1.3 will be headed to the registry shortly (~30mins). It supports reading multiple files at once when passed as a vector for
-read_csv
-read_csv2
-read_delim
-read_tsv
-read_parquet
here is an example. please let us know if there are other issues/feature you would like addressed

path ="https://gist.githubusercontent.com/seankross/a412dfbd88b3db70b74b/raw/5f23f993cd87c283ce766e7ac6b329ee7cc2e1d1/mtcars.csv";
read_csv([path, path], col_select = [:model, :mpg, :hp], skip = 10)

44×3 DataFrame
 Row │ model                mpg      hp    
     │ String31             Float64  Int64 
─────┼─────────────────────────────────────
   1 │ Merc 280C               17.8    123
   2 │ Merc 450SE              16.4    180
   3 │ Merc 450SL              17.3    180
   4 │ Merc 450SLC             15.2    180
   5 │ Cadillac Fleetwood      10.4    205
   6 │ Lincoln Continental     10.4    215
   7 │ Chrysler Imperial       14.7    230
  ⋮  │          ⋮              ⋮       ⋮
  38 │ Fiat X1-9               27.3     66
  39 │ Porsche 914-2           26.0     91
  40 │ Lotus Europa            30.4    113
  41 │ Ford Pantera L          15.8    264
  42 │ Ferrari Dino            19.7    175
  43 │ Maserati Bora           15.0    335
  44 │ Volvo 142E              21.4    109
                            30 rows omitted