Skip to content

Latest commit

 

History

History
113 lines (94 loc) · 4.62 KB

README.md

File metadata and controls

113 lines (94 loc) · 4.62 KB
Raw

Neighborhoods

On DUMBO HPC:

To Generate School Data in Spark:

A. Load school data into HDFS
  • SCHOOL_LOCATIONS_2014_2015_JSON.json
  • SCHOOL_LOCATIONS_2015_2016_JSON.json
  • SCHOOL_LOCATIONS_2016_2017_JSON.json
  • HS_SQR_2014_2015_Summary.csv
  • HS_SQR_2015_2016_Summary.csv
  • HS_SQR_2016_2017_Summary.csv
  • EMS_SQR_2014_2015_Summary.csv
  • EMS_SQR_2015_2016_Summary.csv
  • EMS_SQR_2016_2017_Summary.csv
B. Modify HDFS paths in School_Locations.scala, Updated_EMS_HS.scala, Updated_Join_Latitude_Longitude.scala
C. Enter into command line
spark-shell --packages com.databricks:spark-csv_2.10:1.5.0
:load School_Locations.scala
:load Updated_EMS_HS.scala
:load Updated_Join_Latitude_Longitude.scala

To Generate Crime Data in Spark:

A. Load crime data into HDFS
  • resources/crime/RawData/misdemeanor-offenses-by-precinct-2000-2017.csv
  • resources/crime/RawData/non-seven-major-felony-offenses-by-precinct-2000-2017.csv
  • resources/crime/RawData/seven-major-felony-offenses-by-precinct-2000-2017.csv
  • resources/crime/RawData/violation-offenses-by-precinct-2000-2017.csv
B. Modify HDFS paths in src/crime/CrimeDataETL.scala
C. Enter into command line
spark-shell --packages com.databricks:spark-csv_2.10:1.5.0
:load CrimeDataETL.scala

To Generate Housing Data in Spark:

A. Load housing data into HDFS

All files are in resources/housing/rawData

or from Dumbo

  • /user/sc2936/housingSalesRaw/rollingsales_bronx.csv
  • /user/sc2936/housingSalesRaw/rollingsales_brooklyn.csv
  • /user/sc2936/housingSalesRaw/rollingsales_manhattan.csv
  • /user/sc2936/housingSalesRaw/rollingsales_queens.csv
  • /user/sc2936/housingSalesRaw/rollingsales_statenisland.csv

These last five folders for "borough"_sales_prices each contain seperate files of data per year 2005-2016

  • /user/sc2936/housingSalesRaw/bronx_sales_prices
  • /user/sc2936/housingSalesRaw/brooklyn_sales_prices
  • /user/sc2936/housingSalesRaw/manhattan_sales_prices
  • /user/sc2936/housingSalesRaw/queens_sales_prices
  • /user/sc2936/housingSalesRaw/staten_island_sales_prices
B.1 Modify HDFS paths in src/housing/Recent-And-2017-Sales.scala
  • You must run Recent-And-2017-Sales.scala first as it will generate the input file "housingSalesClean/new2017_5.1.18" for Historical-Housing.scala
  • It will also generate "summary_2017_2018_5.1.2018", which is an input of recent sale prices for the model
B.2 Modify HDFS paths in src/housing/Historical-Housing.scala
  • Historical-Housing.scala will generate one of the input files for the model "housingSalesClean/historical_all_buildingType_5.1.18"
C. Enter into command line
spark-shell --packages com.databricks:spark-csv_2.10:1.5.0
:load Recent-And-2017-Sales.scala
:load Historical-Housing.scala

To Run the model in Spark:

A. Load output Crime, School, and Housing data from above into HDFS

output files can be found at:

  • resources/housing/historical_all_buildingType_5.1.18.csv
  • resources/housing/summary_2017_2018_5.2.2018
  • resources/crime/CrimeWithPrediction.csv
  • resources/school/Final_Output/NYC_School_Data.csv
B. Modify HDFS paths in src/Model_PoC/Model.scala
The output of the model will generate the input for the geo json builder -> resources/model/Output_H.0.35_C.0.4_S.0.25_6.csv

The weights for the model are hard coded within the file and changed for each iteration.
The best wieghts are currently in use:
val housing_weight = .35
val crime_weight = .4
val school_weight = .25

The model will also output a file to manually review goodness -> resources/model/Goodness_H.0.35_C.0.4_S.0.25_6.csv

C. Enter into command line
spark-shell --packages com.databricks:spark-csv_2.10:1.5.0
:load Model.scala

To Convert Model Result to D3 Compatible GeoJSON

A. Run the RebuildJSON.java with two input files
  • model result: resources/model/Output_H.0.35_C.0.4_S.0.25_6.csv
  • NYC GeoJSON: resources/nyc.geojson
  • Result: resources/model/JsonBuilderResult.json

To view map of data follow these steps:

A. In your command line:
  • python -m SimpleHTTPServer
  • open index.html or go to 0.0.0.0:8000

To interact with the map:

A. Hover over a neighborhood to view the name of that neighborhood
B. Select your budget and home type
  • The red areas reveal the top five neighborhoods that are about to pop with respect to your budget and home needs (some budget ranges will reveal fewer than five)
  • These areas will update as you change your budget and home type
  • If you would like to see the data relating to the top five ranks, view the console log.