port download code to use the batch endpoint

[missinglink]

this draft PR updates the OA downloader to use the batch.openaddresses.io endpoint.

a few things worth mentioning:

• the API and downloads require an authentication token
• each 'file' is now looked up against the OA API to get the latest 'run' (ie. latest version)
• downloads are now gzip compressed rather than zip
• the 'global downloads' remain unchanged, still pointing to the old domain, last time I looked there weren't equivalent files on the batch domain
• I haven't done any testing of the import process, only the download process so far

[missinglink]

This is ready for testing, there are some features I would have liked to add but will have to wait for a subsequent PR:

• do not decompress after downloading, instead write the .gz and do the decompression on import (saves disk space)
• source pattern matching, it would be ideal to add us/{ca,ny}/* to the config file and have the API figure it out

I still need to clarify a couple things with the OA team:

• rate-limiting, the code enforces "current policy is 10 request per minute", is this still relevant and will there be an issue with the shared community key?
• what support is there for the 'global downloads' (openaddr-collected-global.zip & openaddr-collected-global-sa.zip)?

[Joxit]

LGTM

👍 for decompression on import and source pattern matching for a next PR.

Maybe you can add a logger.error for all missing files even if errorsFatal is false. This allows you to know all the errors at once.

openaddresses/utils/download_filtered.js

Lines 24 to 32 in 60256bd

	const files = config.get('imports.openaddresses.files', []);
	const sources = await getSources(files);
	const validSources = sources.filter(source => source.url);
	// respect 'imports.openaddresses.missingFilesAreFatal' setting
	if (errorsFatal && (sources.length !== validSources.length)) {
	callback(sources.find(source => source.error)); // return first error
	return;
	}

[missinglink]

Yeah good idea about the logging, I'll go through and make sure everything is logged, even if it's at the debug log level, I also want to log the output of these child process shell commands if anything goes wrong.

I have a chat going with the OA team at the moment to clarify some of the final questions:

• rate limiting: one concurrency per-user is perfect for now, they are looking to migrate to Cloudflare+R2 so hopefully the bandwidth restrictions will be reduced/removed at some point soon.
• global downloads: there is a 'collection' named collection-global.zip which contains everything, addresses, parcels etc. I'd prefer to have a new collection with only the addresses, and in .tar.gz rather than .zip but we will see what's possible.

[vberten]

Hi @Joxit, @missinglink,

Any update on this? Is that planned to make this available in the pelias/docker repo?

Thanks!

[reverload]

Hi @Joxit, @missinglink,

This looks like a great improvement—thank you for all the work on this so far! 😊 I wanted to check in and ask how things are going with this PR. If there’s anything I can do to help, I’d be more than happy to assist!

I’m currently trying to set up Pelias Docker for a Spain build, and I noticed that the countrywide data for Spain from OpenAddresses is only available on the new repositories. This update would be really helpful for my use case, so I’m happy to contribute or test anything you might need to move things forward.

Thanks again for all your efforts! Let me know if I can pitch in.

Cheers