Support CSV output in report/log/aggregate (jazzband#281)

- New option --csv/-s in report, log and aggregate commands, mutually
exclusive with --json/-j.
- New hidden option --plain to handle the default output format in
report, aggregate and log commands (i.e. plain text). This option is
not intended to be used by the final user.
- Add support in MutuallyExclusiveOption for options with an internal
name different from the ones permitted in the command line.

CHANGELOG.md

@@ -7,9 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Add CSV output format support for `report`, `log` and `aggregate` commands
+  using the `--csv/-s` command line option flag (#281).
+
 ### Fixed
 
-- Update zsh shell completion (#264)
+- Update zsh shell completion (#264).
 
 ### Removed
 

docs/user-guide/commands.md

@@ -52,8 +52,10 @@ projects or tags to the report.
 If you are outputting to the terminal, you can selectively enable a pager
 through the `--pager` option.
 
-You can change the output format for the report from *plain text* to *JSON*
-by using the `--json` option.
+You can change the output format from *plain text* to *JSON* using the
+`--json` option or to *CSV* using the `--csv` option. Only one  of these
+two options can be used at once.
+
 
 Example:
 
@@ -86,6 +88,21 @@ Example:
     Wed 21 November 2018 - 01m 17s
       watson - 01m 17s
             [docs     01m 17s]
+
+    $ watson aggregate --csv
+    from,to,project,tag,time
+    2018-11-14 00:00:00,2018-11-14 23:59:59,watson,,20542.0
+    2018-11-14 00:00:00,2018-11-14 23:59:59,watson,features,2046.0
+    2018-11-14 00:00:00,2018-11-14 23:59:59,watson,docs,18496.0
+    2018-11-19 00:00:00,2018-11-19 23:59:59,watson,,21532.0
+    2018-11-19 00:00:00,2018-11-19 23:59:59,watson,features,4323.0
+    2018-11-19 00:00:00,2018-11-19 23:59:59,watson,docs,17209.0
+    2018-11-20 00:00:00,2018-11-20 23:59:59,watson,,10235.0
+    2018-11-20 00:00:00,2018-11-20 23:59:59,watson,features,917.0
+    2018-11-20 00:00:00,2018-11-20 23:59:59,watson,docs,5863.0
+    2018-11-20 00:00:00,2018-11-20 23:59:59,watson,website,3455.0
+    2018-11-21 00:00:00,2018-11-21 23:59:59,watson,,77.0
+    2018-11-21 00:00:00,2018-11-21 23:59:59,watson,docs,77.0
 
 ### Options
 
@@ -96,7 +113,8 @@ Flag | Help
 `-t, --to DATE` | The date at which the report should stop (inclusive). Defaults to tomorrow.
 `-p, --project TEXT` | Reports activity only for the given project. You can add other projects by using this option several times.
 `-T, --tag TEXT` | Reports activity only for frames containing the given tag. You can add several tags by using this option multiple times
-`-j, --json` | Format the report in JSON instead of plain text
+`-j, --json` | Format output in JSON instead of plain text
+`-s, --csv` | Format output in CSV instead of plain text
 `-g, --pager / -G, --no-pager` | (Don't) view output through a pager.
 `--help` | Show this message and exit.
 
@@ -232,6 +250,10 @@ You can limit the log to a project or a tag using the `--project` and
 `--tag` options. They can be specified several times each to add multiple
 projects or tags to the log.
 
+You can change the output format from *plain text* to *JSON* using the
+`--json` option or to *CSV* using the `--csv` option. Only one  of these
+two options can be used at once.
+
 Example:
 
 
@@ -260,6 +282,14 @@ Example:
     Wednesday 16 April 2014 (5h 19m 18s)
             02cb269  09:53 to 12:43   2h 50m 07s  apollo11  [wheels]
             1070ddb  13:48 to 16:17   2h 29m 11s  voyager1  [antenna, sensors]
+
+    $ watson log --from 2014-04-16 --to 2014-04-17 --csv
+    id,start,stop,project,tags
+    a96fcde,2014-04-17 09:15,2014-04-17 09:43,hubble,"lens, camera, transmission"
+    5e91316,2014-04-17 10:19,2014-04-17 12:59,hubble,"camera, transmission"
+    761dd51,2014-04-17 14:42,2014-04-17 15:54,voyager1,antenna
+    02cb269,2014-04-16 09:53,2014-04-16 12:43,apollo11,wheels
+    1070ddb,2014-04-16 13:48,2014-04-16 16:17,voyager1,"antenna, sensors"
 
 ### Options
 
@@ -276,7 +306,8 @@ Flag | Help
 `-a, --all` | Reports all activities.
 `-p, --project TEXT` | Logs activity only for the given project. You can add other projects by using this option several times.
 `-T, --tag TEXT` | Logs activity only for frames containing the given tag. You can add several tags by using this option multiple times
-`-j, --json` | Format the log in JSON instead of plain text
+`-j, --json` | Format output in JSON instead of plain text
+`-s, --csv` | Format output in CSV instead of plain text
 `-g, --pager / -G, --no-pager` | (Don't) view output through a pager.
 `--help` | Show this message and exit.
 
@@ -439,7 +470,8 @@ If you are outputting to the terminal, you can selectively enable a pager
 through the `--pager` option.
 
 You can change the output format for the report from *plain text* to *JSON*
-by using the `--json` option.
+using the `--json` option or to *CSV* using the `--csv` option. Only one
+of these two options can be used at once.
 
 Example:
 
@@ -507,6 +539,15 @@ Example:
             "to": "2016-02-28T23:59:59.999999-08:00"
         }
     }
+
+    $ watson report --from 2014-04-01 --to 2014-04-30 --project apollo11 --csv
+    from,to,project,tag,time
+    2014-04-01 00:00:00,2014-04-30 23:59:59,apollo11,,48140.0
+    2014-04-01 00:00:00,2014-04-30 23:59:59,apollo11,brakes,28421.0
+    2014-04-01 00:00:00,2014-04-30 23:59:59,apollo11,module,27701.0
+    2014-04-01 00:00:00,2014-04-30 23:59:59,apollo11,reactor,30950.0
+    2014-04-01 00:00:00,2014-04-30 23:59:59,apollo11,steering,38017.0
+    2014-04-01 00:00:00,2014-04-30 23:59:59,apollo11,wheels,36695.0
 
 ### Options
 
@@ -523,7 +564,8 @@ Flag | Help
 `-a, --all` | Reports all activities.
 `-p, --project TEXT` | Reports activity only for the given project. You can add other projects by using this option several times.
 `-T, --tag TEXT` | Reports activity only for frames containing the given tag. You can add several tags by using this option multiple times
-`-j, --json` | Format the report in JSON instead of plain text
+`-j, --json` | Format output in JSON instead of plain text
+`-s, --csv` | Format output in CSV instead of plain text
 `-g, --pager / -G, --no-pager` | (Don't) view output through a pager.
 `--help` | Show this message and exit.
 

tests/test_utils.py

@@ -2,7 +2,10 @@
 """Unit tests for the 'utils' module."""
 
 import arrow
+import collections as co
+import csv
 import functools
+import json
 import os
 import datetime
 
@@ -20,8 +23,12 @@
 
 from watson.utils import (
     apply_weekday_offset,
+    build_csv,
     confirm_project,
     confirm_tags,
+    flatten_report_for_csv,
+    frames_to_csv,
+    frames_to_json,
     get_start_time_for_period,
     make_json_writer,
     safe_save,
@@ -228,3 +235,107 @@ def test_confirm_tags_reject_raises_abort(confirm):
     watson_tags = ['a', 'b']
     with pytest.raises(Abort):
         confirm_project(tags, watson_tags)
+
+
+# build_csv
+
+def test_build_csv_empty_data():
+    assert build_csv([]) == ''
+
+
+def test_build_csv_one_col():
+    lt = csv.get_dialect('excel').lineterminator
+    data = [{'col': 'value'}, {'col': 'another value'}]
+    result = lt.join(['col', 'value', 'another value']) + lt
+    assert build_csv(data) == result
+
+
+def test_build_csv_multiple_cols():
+    lt = csv.get_dialect('excel').lineterminator
+    dm = csv.get_dialect('excel').delimiter
+    data = [
+        co.OrderedDict([('col1', 'value'),
+                        ('col2', 'another value'),
+                        ('col3', 'more')]),
+        co.OrderedDict([('col1', 'one value'),
+                        ('col2', 'two value'),
+                        ('col3', 'three')])
+    ]
+    result = lt.join([
+        dm.join(['col1', 'col2', 'col3']),
+        dm.join(['value', 'another value', 'more']),
+        dm.join(['one value', 'two value', 'three'])
+        ]) + lt
+    assert build_csv(data) == result
+
+
+# frames_to_csv
+
+def test_frames_to_csv_empty_data(watson):
+    assert frames_to_csv(watson.frames) == ''
+
+
+def test_frames_to_csv(watson):
+    watson.start('foo', tags=['A', 'B'])
+    watson.stop()
+
+    result = frames_to_csv(watson.frames)
+
+    read_csv = list(csv.reader(StringIO(result)))
+    header = ['id', 'start', 'stop', 'project', 'tags']
+    assert len(read_csv) == 2
+    assert read_csv[0] == header
+    assert read_csv[1][3] == 'foo'
+    assert read_csv[1][4] == 'A, B'
+
+
+# frames_to_json
+
+def test_frames_to_json_empty_data(watson):
+    assert frames_to_json(watson.frames) == '[]'
+
+
+def test_frames_to_json(watson):
+    watson.start('foo', tags=['A', 'B'])
+    watson.stop()
+
+    result = json.loads(frames_to_json(watson.frames))
+
+    keys = {'id', 'start', 'stop', 'project', 'tags'}
+    assert len(result) == 1
+    assert set(result[0].keys()) == keys
+    assert result[0]['project'] == 'foo'
+    assert result[0]['tags'] == ['A', 'B']
+
+
+# flatten_report_for_csv
+
+def test_flatten_report_for_csv(watson):
+    now = arrow.utcnow().ceil('hour')
+    watson.add('foo', now.shift(hours=-4), now, ['A', 'B'])
+    watson.add('foo', now.shift(hours=-5), now.shift(hours=-4), ['A'])
+    watson.add('foo', now.shift(hours=-7), now.shift(hours=-5), ['B'])
+
+    start = now.shift(days=-1)
+    stop = now
+    result = flatten_report_for_csv(watson.report(start, stop))
+
+    assert len(result) == 3
+
+    assert result[0]['from'] == start.format('YYYY-MM-DD 00:00:00')
+    assert result[0]['to'] == stop.format('YYYY-MM-DD 23:59:59')
+    assert result[0]['project'] == 'foo'
+    assert result[0]['tag'] == ''
+    assert result[0]['time'] == (4 + 1 + 2) * 3600
+
+    assert result[1]['from'] == start.format('YYYY-MM-DD 00:00:00')
+    assert result[1]['to'] == stop.format('YYYY-MM-DD 23:59:59')
+    assert result[1]['project'] == 'foo'
+    assert result[1]['tag'] == 'A'
+    assert result[1]['time'] == (4 + 1) * 3600
+
+    assert result[2]['from'] == start.format('YYYY-MM-DD 00:00:00')
+    assert result[2]['to'] == stop.format('YYYY-MM-DD 23:59:59')
+    assert result[2]['project'] == 'foo'
+    assert result[2]['tag'] == 'B'
+    assert result[2]['time'] == (4 + 2) * 3600