f/modified air call (#149)

* New query builder in old aircall connector

We want generic routes

* New helper functions for handling Aircall related data

- Added helper for setting up team/user-related data
- Added unit tests for the helper handling team/user-related data

* Crude code for testing aircall API

- concatenates data and uses a promise all for obtaining /teams data
(includes users data as well)

- setting up the tests, at the moment they don't test for anything but
were used to trigger code in the connector

- getting ready for panda data frame

* Quick save before starting day of code

- added raw df creation (needs improvement)
- put through jq filter

* Request now calls /calls and /tags

- Rough data in /tags and /calls
- Added formatting to /teams /users joint call
- Added pandas data frame to both types of calls

* started on data reformat for /calls

* quick save of very crude code

* added a dictionary linked to dataset

* Code refactor and added calls case

- jq filters done for all requests
- two pandas dataframe results are finished

* Raw code; pandas df's generated for all calls

- tags, users and calls are all async
- calls dataframes now integrating both /teams and /calls data
- have to tweak order of columns in latter df

* tidied up code in connector: right function doing right thing

* added new constant

* Got pagination working

All calls are now bulk fetches

* Added basic limit handler

Only handles up to 10 passes

* quick commit before merge

* Cleaned up code

- gave up on running chunked code
- might not be enough to handle all cases

* Added helper test for dataframe builder

* finished testing the dataframe builder

* Added new test for empty dataframe builder

* Updated tests for jq filters

- added default jq filters if dataset is bad even though this should be
caught beforehand
- changed default call to 'users' from 'teams' in Enum

* quick commit before merge

* Added tests for _get_data()

- Tests for a tuple-of-lists-generating call
- Tests for a single list-generating call

* changed name of bulk_fetch function

* Added new tests

- main retrieve data call
- divied up remaining get data calls so that two different tests treat
each call separately
- added tests to check order of columns and JSON properties

* added a 'passonable' limit

* tweaked generation of order of columns

* quick 'save' before installing new dependencies

* Added async tests in new file

- Tested basic functionality of recursive fetch
- Removed to new file because won't work with old tests

* Improved aircall connector

- added code so that does not fail if no meta data returned
- cut up functions that were too complex

* save for evening; broken test

* Added a new test for absence of meta in response

- The real code was already added in a prior commit
- The response is still returned but recursive function does not throw
error

* Function for handling df merge and tests

- last test for fetch_page function
- added side effects in test
- added function to check data in calls call to avoid error

* preparation for proper PR

* TEMP more logging during CI tests

* Fixing async tests and added loop tests

- loop functions inside AirCall tested; temporary code to see if CI
tests pass
- tests for loop generator functions inside AirCall connector

* trying a potential fix for wootric tests

* trying out pytest asyncio event loop for tests

* fixed lint issue

* added decorators to tests

* put back in run fetches tests

* Use event_loop fixture to avoid loop leak

* Distinct patching of coroutines before/after 3.8

* linted version of fixed tests

* purified and normalized test helpers

* Finished test and improved two others

- finished default limit test
- removed JS-like conditional for arrays in two tests
- reset max runs constant to 60 (for API limit)

* linted files

* fixed problem with python version in test

* removed old E2E tests for mocked versions

* linted file

* Replaced hardcoded API tokens with bearer

- added bearer back to main function
- added bearer to recursive fetcher

* Updated tests to include bearer code

- in this commit, most tests in aircall tests are skipped due to missing
token
- mocked bearer token in a function

* linted files

* console log for recursive func endpoint

* more console logs

* more console logs

* trying out fix for bad endpoint

* trying another potential solution

* removed bad func signature in call

* test

* Got generic fetch working with Bearer API

- removed previous bugged code
- got requests working with generic fetch function and Bearer API
- put back in Toucan-style config without completely removing local
config for local testing in dev
- fixed tests to pass with on local machine with Bearer workflow

* linted files

* got rid of a pytest option used for debugging something

* Refactored test helpers

- removed redundant function
- replaced verbose code with a list comprehension

* added new teams and users filtered data

* Added new filtered mock data

- This should improve the durability of the tests
- Had two different styles of 'datasets' in mock data so updated
supposed filtered results

* added more fake data

* Removed some tests and cleaned up others

- removed unnecessary print expressions
- removed unnecessary tests due to refactoring helpers

* Removed unnecessary helper

- added FILTER_DICTIONARY to the connector
- removed try catches for loop in hopes that it won't mess things up

* linter bugged out

* more linting problems

* Put back in try/catch in loop functions

- E2E manual test with browser calling the AirCall requests fails
without try/catch

* fixed an issue with recursive function

* New and updated tests and refactoring

- centralized loop functionality
- generic function for checking python version
- changed how mock futures are built
- removed jq filters from AirCall code
- created new tests for filtering functions
- updated tests to reflect changes

* reset limit back to 60

* added test to cover no user in calls response

* No user case in calls, changed default to calls

- Calls call now default call rather than users
- Added conditionals to check for user in filter function

* fixed test that failed

* fixed bug with user name not displaying

* Added infinite limit and limit of zero

- -1 is now infinite limit and validation starts from -1 on up
- NO TEAM replaces a team of null
- limit of 0 blocks fetches
- updated tests to reflect changes
- added new test to make sure limit of 0 works correctly

* removed a print

* Updated code based on feedback in review

- changed build future helper to include check for Python version
- added a helper to handle mock fetches based on Python version
- some small changes

* fixed format of async function tests

* added some more revisions

Co-authored-by: David Nowinsky <[email protected]>
Co-authored-by: David Nowinsky <[email protected]>

requirements-testing.txt

@@ -5,6 +5,8 @@ docker >= 3.1.1
 flake8 >= 3.5.0
 flake8-quotes
 isort
+pytest-aiohttp >= 0.3.0
+pytest-asyncio
 pytest-cov >= 2.5.1
 pytest >= 3.4.2
 pytest-mock >= 1.9.0

tests/aircall/helpers.py

@@ -0,0 +1,42 @@
+"""Helpers functions for setting up tests for AirCall connector"""
+
+from tests.general_helpers import check_py_version
+from toucan_connectors.aircall.aircall_connector import AircallConnector, AircallDataSource
+
+# we want this module's code to be checked against Python3.8
+# versions prior to Python3.8 don't handle mocks same way for async functions
+PY_VERSION_TO_CHECK = (3, 8)
+
+is_py_version_older = check_py_version(PY_VERSION_TO_CHECK)
+
+
+def build_con_and_ds(dataset: str):
+    """
+    Builds test connector and test datasource for testing with API key
+
+    Leave this function in if ever want to run tests without skipping
+    due to there being no Bearer tokens
+    How to use:
+    Replace build_ds function with this one in test_aircall file
+    Be sure to also replace the endpoints inside the aircall connector file
+    """
+    con = AircallConnector(name='mah_test', bearer_auth_id='abc123efg')
+    ds = AircallDataSource(name='mah_ds', domain='test_domain', dataset=dataset, limit=1,)
+
+    return con, ds
+
+
+def assert_called_with(fake_func, expected_params=[], expected_count=None):
+    """Tests mock function with called/awaited depending on Python version"""
+    if is_py_version_older:
+        if expected_count is not None:
+            assert fake_func.call_count == expected_count
+        if expected_params:
+            fake_func.assert_called_with(*expected_params)
+    else:
+        if expected_count is not None:
+            assert fake_func.await_count == expected_count
+        if expected_params:
+            fake_func.assert_awaited_with(*expected_params)
+        else:
+            fake_func.assert_awaited_once()