RFC Data Pipeline - Data Monitoring Squad

[yeiror]

Greetings for everyone. We'll appreciate any feedback related with this RFC.

https://github.com/Platzi-Master-C9/knowledge-base/pull/21

[gh0stl1m]

@yeiror I'm not sure if the PR was changed by the Pull request is throwing 404, so I can't check the proposal

[yeiror]

It's really weird what I see. Anyway you can check the PR # 21 in the knowledge base repo.

https://github.com/Platzi-Master-C9/knowledge-base/pull/21/commits

[gh0stl1m]

@yeiror I'm seeing it now but let's bear on mind that the RFC should be into the accepted folder because right now it is outside

[larispardo]

I have a couple of comments as discussed on the meeting.

• we need to make sure the API Rest is consistent with such as the team wants to use. As the team has more knowledge of python I would suggest we use a framework in this language, but let's discuss it as a team.

• I would love to see more alternatives to the proposed pipeline in case we get into some problems with the ones proposed. Specially for Apache Airflow since it will be something new for the rest of the team.

[jhonny3e7]

We choose the Python side on RFC for API, but at the end those who gonna work on the API, will make the final decision.

[jhonny3e7]

About Airflow I was thinking about it and make the same conclusion, it´s a better aproach for now, to do the pipelines with the cron jobs. thanks for comments.

[YaelRmz]

I thought the team of pipeline it was going to work with the API as part of the extraction. @larispardo wich is the team that work with the API?

[larispardo]

Yes, it is, but you should unify this technology, it would be inconsistent that we have one RFC saying we are going to use FastAPI and another Fastify 🤔. That is what I am trying to say

[YaelRmz]

Aaa ok. RFC about Data collection focused more in the structure of data and the routes for the API. We don't have problem to change the library to FastAPI, actually we add this library to Alternatives section in RFC