Convert RecordLoader.loadArchives to a Spark Data Source

[ruebot]

Since we're pivoting to full DataFrame support (#223, #190), we should convert/migrate RecordLoader.loadArchives, and any other related functions to a Spark Data Source. That way we could do things like:

spark.read.format("webArchive")
  .option("mode", "FAILFAST")
  .option("inferSchema", "true")
  .option("/path/to/files")
  .schema(someSchema)
  .load()

Then, we could, (since it's an open issue #147) write WARCs that way too? 🤷‍♂️

spark.write.format("webArchive")
  .option("mode", "OVERWRITE")
  .option("/path/to/files")
  .save()

These are the Spark core data sources:

• CSV
• JSON
• Parquet
• ORC
• JDBC/ODBC
• Plain-text
• Avro

Community implemented data sources:

• Cassandra
• HBase
• MongoDB
• AWS Redshift
• XML

[ruebot]

Some helpful links:

• https://mapr.com/blog/spark-data-source-api-extending-our-spark-sql-query-engine/
• http://blog.madhukaraphatak.com/spark-datasource-v2-part-3/
• https://databricks.com/session/how-to-connect-spark-to-your-own-datasource

[sepastian]

https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-DataSourceV2.html
https://github.com/spirom/spark-data-sources

[ruebot]

Cassandra example

[ruebot]

I'm thinking this is out of scope for this project given the work being done on #494 now. So, I'm going to close it as won't fix.