Export data as CSV and JSON

[SelmaGuedidi]

Closes #5

[idris]

@SelmaGuedidi can you please remove the output/yaml directories? Those are old and not tied to the actual data. I think it was just a sample format from before that's no longer relevant.

[idris]

Thanks for the PR! Looks mostly good! But unfortunately you used the wrong source data. Sorry for the confusion!

[idris]

Sorry, the data in the output/yaml directory is an old format that should have been deleted.

You need to read the data from data/, and follow the schemas there.

For the output, I think CSV should output two files: one for companies, one for brands. The JSON should output one file, which looks like:

{
  "brands": [ {...}, {...} ],
  "companies": [ {...}, {...} ]
}

Also, please add an id to each brand in both the CSV and JSON. The id is just the filename (without the extension).

[idris]

Nice! Thanks for adding this check

[idris]

FYI: requirements.txt uses pyyaml, not pyaml

[SelmaGuedidi]

@idris I made the changes , hope it's fixed :)

• Removed data in output/yaml
• Updated data retrieval to follow schemas in data/
• CSV output generates separate files for companies and brands
• JSON output consolidated into a single file with "brands" and "companies" keys
• Updated .pre-commit-config.yaml to use pyyaml instead of pyaml

[THM222]

Can we add timestamps on the exports? createdAt, updatedAt, utc timezone maybe yyyy-mm-ddTHH:mm:ss.SSSZ?

For now they can be the same, and lets raise a new issue to edit the updateAt correctly.

[idris]

The format of dictionaries in the CSV was weird, so I figure JSON is a good format, and it works well for both lists and dicts:

Suggested change

	if isinstance(value, list):
	return ', '.join(map(str, value))
	if isinstance(value, list) or isinstance(value, dict):
	return json.dumps(value)

[SelmaGuedidi]

@idris

   if isinstance(value, list):
        return ', '.join(map(str, value))

brands.csv:

   if isinstance(value, list) or isinstance(value, dict):
        return json.dumps(value)

brands.csv:

I used ', '.join(map(str, value)) to remove [] and "" in the csv files.
and I think dict format does not exist here because isinstance(value, dict) is always false

[idris]

Hm, I think you're right that the ', '.join(...) version is better for lists.

If you look at the sabra row, there is a dict for the stakeholders column, which shows up like this:
"{'id': 'pepsico', 'type': 'owner', 'percent': 50}, {'id': 'strauss-group', 'type': 'owner', 'percent': 50}"

We're not really sure how the stakeholders bit will pan out, so I'm just going to leave your join as-is, and remove stakeholders from the CSV export for now.

[idris]

oooh the plot thickens.. it's an array of objects. Even more reason to ignore that for now.

[idris]

Thanks @SelmaGuedidi ! I added one suggestion above, but let me know if you disagree.

@THM222 I think we can handle both timestamps in a separate issue. I created that here: #10