Home

Metadata and Data lineage

• Netflix, metacat

  • https://github.com/Netflix/metacat
  • https://medium.com/netflix-techblog/metacat-making-big-data-discoverable-and-meaningful-at-netflix-56fb36a53520

• https://medium.com/netflix-techblog/building-and-scaling-data-lineage-at-netflix-to-improve-data-infrastructure-reliability-and-1a52526a7977

• LinkedIn, WhereHows

  • https://github.com/linkedin/WhereHows
  • https://engineering.linkedin.com/blog/2016/03/open-sourcing-wherehows--a-data-discovery-and-lineage-portal

• Lyft, Amundsen

  • https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d27254fbb9
  • https://www.youtube.com/watch?v=m1B-ptm0Rrw
  • https://www.datacouncil.ai/hubfs/DataEngConf/Data%20Council/Slides%20SF%2019/Amundsen%20-%20A%20Data%20Discovery%20Platform%20from%20Lyft.pdf

---

https://medium.com/@garyogasawara/hadoop-performance-benchmark-results-comparing-on-premise-s3-vs-hdfs-cf7a9ea3baa3

http://info.bluedata.com/rs/693-TGY-247/images/IDC-Perspective-Decoupling-Compute-Storage.pdf

https://github.com/rstacruz/cheatsheets

Apache Airflow:

• https://github.com/etsy/boundary-layer

zeppelin, centos7, 971748464

pypi mirror, https://stackoverflow.com/questions/50348248/creating-a-full-replica-offline-copy-of-the-public-pypi-repository

• https://github.com/pypa/bandersnatch

Readings:

• Architecting Modern Data Platforms, https://learning.oreilly.com/library/view/architecting-modern-data/9781491969267/
• Stream Processing with Apache Flink, https://learning.oreilly.com/library/view/stream-processing-with/9781491974285/
• Learning Spark Streaming, https://learning.oreilly.com/library/view/learning-spark-streaming/9781491944233/
• http://book.mixu.net/distsys/