Skip to content

Latest commit

 

History

History
66 lines (43 loc) · 1.88 KB

README.md

File metadata and controls

66 lines (43 loc) · 1.88 KB
Raw

Spark stat analyzer

POC to generate consolidated statistics from json files (generated by navitia-stat-logger or navitia-stat-exporter)

Pre-requisites

  • Spark 1.5+ (may work with previous versions, but untested)
  • A repository with exported statistics files where stat files are stored in a tree like
  |
  \- <year>
      |
      \- <month>
          |
          \- <day>
              |
              \- <files>.json.log(.gz)

The files are json logs (one json per line). The files may be compressed using gzip.

Create config.py file from template config.py.dist and adapt them

Usage

  • For requests_calls consolidation
<path/to/spark>/bin/spark-submit  --conf spark.ui.showConsoleProgress=true --master='local[3]' manage.py -a <analyzer> -i <your_export_directory> -s <start_date> -e <end_date>

where:

  • analyzer: analyzer name, possible value : token_stats, users, requests_calls, error_stats, coverage_stop_areas, coverage_modes, coverage_journeys_transfers or coverage_journeys_requests_params

  • start_date and end_date is in YYYY-MM-DD format

  • For coverage_journeys consolidation

<path/to/spark>/bin/spark-submit  --conf spark.ui.showConsoleProgress=true --master='local[3]' coverage_journeys.py <your_export_directory> <start_date> <end_date>

where:

  • start_date and end_date is in YYYY-MM-DD format

Note that the results are stored in the export dir

Shell ZSH users

Launching 'spark-submit' command , if you get error :

zsh: no matches found

You can replace in command line --master='local[3]' by --master=local, or use bash.

Tests

To run tests, you can build your own image and launch them using:

docker build -f Dockerfile.test -t stat-analyser:test .
docker run -e USER_ID=$(id -u) -it -v $(pwd):/srv/spark-stat-analyzer stat-analyser:test sh -c './run_test.sh'

The docker image is also hosted on our registry under the same name:tag