Project 2

• Project 2 is a Spark application that processes COVID data and using Zeppelin (or Tableau or other visualization software for graphics and visuals) for showing trends and data analysis.

MVP:

• Create a Spark Application that processes COVID data
• Involves some analysis of COVID data using 10 queries
• Produce one or more .jar files for the analysis. Then run application using spark-submit
• Find a trend
• Implement logging (with Spark)
• Use Zeppelin (or Tableau or other visualization software for graphics and visuals) for showing trends and data analysis
• Implement Agile Scrum methodology for project work
• JIra Software for task managment

Stretch Goals:

• Utilize Apache Airflow for workflow scheduling
• Exporting results
• Encrypting password

Tech Stack:

• Apache Spark
• Spark SQL
• YARN
• HDFS and/or S3
• SBT
• Scala 2.12 (or 2.13)
• Git + GitHub
• Zeppelin (or Tableau or other visualization software)

Development LifeCycle

1. Plan
2. Design
3. Code
4. Test
5. Deploy

Project Management:

• IntelliJ IDEA 2022.1.2

Generalizations

• IDE Used: IntelliJ IDEA 2022.1.2
• Scala Package Used: Winutils
• Visualization Programs: Apache Zeppelin

Extra Data Used

Provisional COVID-19 Deaths by Sex and Age

• https://data.cdc.gov/NCHS/Provisional-COVID-19-Deaths-by-Sex-and-Age/9bhg-hcku

Average monthly temperature in the U.S. from January 2019 to May 2022 (in Fahrenheit)

• https://www.statista.com/statistics/513628/monthly-average-temperature-in-the-us-fahrenheit/

Collaborators:

• Oscar Garcia
• Jordi Icetch
• Joseph Kim
• Thuvarakan Nakarajah
• Edwin Castano