🎛️ ControlHelper Rewrite #1052

the-bay-kay · 2023-10-04T17:22:35Z

See Issue 994 for context!

In order to fix the 'Download JSON Dump' feature, we need to rework the ControlHelper service. The goal is to separate this service into its own file (e.g., controlHelper.ts). In the process of converting this from an angular service to a TS component, I'll be using luxon instead of moment.js, as discussed in the issue.

As per issue e-mission#994, fixed the `Download JSON Dump`. The previous date-picker formatted the input as 'dd MMM yyyy', used luxon to format the object. This is a bit hacky - going to rewrite the 'ControlHelper' service, and completely remove the `moment()` altogether.

`js/services.js` has been rewritten so that it no longer relies on the legacy `moments.js` library, and instead uses `luxon`.

the-bay-kay · 2023-10-05T17:47:59Z

It seems that moment.js and luxon handle the .endOf('day') function differently - one ends on the minute (59:00), and the other the seconds (59:59). This has been causing issues, since we need to get the exact unix number for the server call, I'm looking into how to emulate the old method.

the-bay-kay · 2023-10-05T18:46:05Z

Update:
Using an Epoch Converter, I've found some odd things about the methods.
It seems moment.js's "End of day" function takes the given date, and "fast forwards" by a day, minus a second - e.g.,
Thursday, April 13, 2023 7:00:00 AM --> Friday, April 14, 2023 6:59:59 AM
(Local Time: '...' --> Thursday, April 13, 2023 11:59:59 PM GMT-07:00 DST)

Wheras luxon, oddly, is not changing the date at all:
Thursday, April 13, 2023 7:00:00 AM --> Friday, April 14, 2023 6:59:59 AM
(Local Time: '...' --> Thursday, April 13, 2023 12:00:00 AM GMT-07:00 DST)

My guess is that the issue lies within the local time - it seems Luxon uses that for the endOf() method. I'm going to go skim through both of these libraries, and see if I can find the code itself. If anyone has sample data of an overnight travel (e.g., spanning two days), I would really appreciate if I could borrow that for testing! The timestamps seem to be pretty finicky, so I'd like to test my code with as many types of data as possible.

See comments on [the PR Draft](e-mission#1052). Luxon and Moment.js seem to handle the `endOf('day')` slightly differently. The remedy for this was to simulate the old Unix Integer conversion for the end date, but use the contemporary one for the start date. This adjustment fixed the download, and worked for the multiple timelines tested.

Adjusted `js/services.js` such that it only uses the constants exported from `js/plugins/logger.ts`. Removed the member function `writeFile` from the controlHelper, as it was not being used.

the-bay-kay

In 2e9173b, I got the "startTime" and "endTime" values mixed. To clarify: the startTime must be processed through the old conversion method getUnixNum() and the endTime is fetched by running luxon's .toUnixInteger() .The other way around does not work, I'll update the comment in my next commit.

the-bay-kay · 2023-10-06T22:28:01Z

After reading up on the requestFileSystem() method within the getMyData() member function, it seems that Mozilla considers this to be a deprecated feature. I'm currently looking at rewriting this section with the more contemporary Web File System API!

controlHelper no longer utilizes Angular services. The internal functions were separated into their own exports. File write is confirmed to be working, need to write tests and verify all other functions are working as intended.

the-bay-kay · 2023-10-10T22:51:22Z

Update: After some wrangling with the androidSDK, the rewrite is confirmed to work on android! Since the iOS version relies on the built-in app, which is not included on the simulator, further testing is needed. As discussed in our meeting today, I'm going to see if we can utilize the "share" feature of iOS, rather than the mail app. This would allow us to test with the simulator and have greater flexibility when sending our timelines.

(I'll update this comment later with a video of the email working - unfortunately, my usual screen recording options on Linux (Flameshot, OBS) aren't working with this fresh install, so I need to do a bit more troubleshooting before I have full videos of Android!)

Date range	...	....

the-bay-kay · 2023-10-11T18:21:22Z

Update: The code is not committed, but I've got a variation of the JSONDump written that utilizes the Social Sharing plugin, similar to how the QR Code is shared. Below are a few of my notes on the current status of this:

The download is confirmed to work on iOS: I have not been able to test an email specifically, but as shown in the video below, the file downloads to 'files' successfully.
An interesting side-effect of using the 'socialSharing' library, is that the "message" portion of the emails is also downloaded as a separate file. I'm looking to see if this can be disabled for local downloads
It seems that there are some issues on Android - upon my first few tests, the files downloaded returned empty. Luckily, at least "message" and "subject" lines work as intended when emailing.

DownloadWorking.mp4

the-bay-kay · 2023-10-11T20:51:10Z

After further digging, it seems that the socialSharing library doesn't have a means to stop the extra "message" file from being downloaded. Furthermore, I haven't made much progress on the android side of things. Before continuing down this path further, I think it's worth asking for other folks' opinions! As I see it, we've got two possible ways of moving forward:

I can continue to work with the socialSharing library, and make it worth with both Android and iOS. If we really want it working soon, could case on the O.S., using socialSharing for iOS, and the openEmail() function for Android (this would mean controlHelper relies on two separate email plugins, which doesn't seem ideal). Either way, this would give more flexibility for iOS sharing, with the caveat that the "direct-to-files" seems to dump an extra text file in certain contexts.
We can stick to the current code - I still need to test on hardware, but the current version is confirmed to work with Android, and will presumably also work for iOS - however, this would mean the user still relies on the built-in mail app.

Any thoughts @shankari @JGreenlee ? I think this boils down to a question of "Is the extra flexibility worth complicating the share process". Would my time be better spent pivoting to other parts of the rewrite, or is it worth changing this while I'm already updating it?

For the time being, I'll keep working on setting up socialSharing on Android!

shankari · 2023-10-11T21:37:30Z

@the-bay-kay I don't have a strong preference wrt functionality. To my knowledge, the only people who use this button are:

developers
people testing out the app who don't yet have access to an environment with an admin dashboard aka people who have signed the MOU but are waiting for NREL to sign it as well

So I don't think it is a huge deal to them to configure the default mail app. But it would obviously be better if that wasn't required. So I would say that we spend a time-bounded effort on this. If you can get the android code to work in (say) one day, let us clean it up. If not, let us revisit.

Changed all variables that do not mutate to const.

the-bay-kay · 2023-10-11T23:52:14Z

Success!! Got it working on Android. First thing tomorrow I'll go ahead and test on a few other versions before committing and setting up the PR for review.
Screencast from 2023-10-11 19-47-00.webm

See [PR 1052](e-mission#1052). Using the socialShare library, as we do in the OpCode components, allows iOS users to share through apps other than the defauolt mail app.

the-bay-kay · 2023-10-12T15:55:41Z

A quick note on the force-push; I accidentally added code from master, and wanted to make sure this branch was as separate when merging. I don't believe the commit would've changed much (it was just a prior update I did), but I figure better to keep them separate for now.

the-bay-kay · 2023-10-12T17:06:44Z

Here's an email of the JSONDump working on both Android and iOS!! I tested on a couple different versions, so this should be good to go!
(Also, for anyone else wanting to document on an XOrg Linux machine - I found vokoScreen to be the simplest and best screen recording software!)

android_working.mp4

iOS.Working.mp4

JGreenlee · 2023-10-13T04:41:06Z

Great work! Before I review this, can you change the target of this PR to some other branch, then change it back to service_rewrite_2023?

(#1040 is merged but my commits are still showing up in this PR. I've discovered that toggling the branch forces Github to update the diff)

the-bay-kay · 2023-10-13T15:12:55Z

Updated! Good to know you can clean the git logs like that, I'll go ahead and do the same thing for my next PR!

JGreenlee

The code looks clean - I just have a few substitutions I recommend.

I am unsure how, or even if, we would go about testing controlHelper.
I don't think we can test the email/share menu with unit tests. We might be able to write a test validating the contents of the dump file. But this would require us to mock the retrieval of entries, return some fake entries, and mock the filesystem write.
If you break getMyData into smaller functions you can probably validate the 'dump' without worrying about mocking filesystem. Function A creates the dump and function B writes the dump to filesystem; function A calls function B. You would test only function A.

www/js/controlHelper.ts

Per discussion in PR, made adjustments to debug logs and funciton names.

By extracting the `writeFile` and `shareData` methods, we can now write unit tests for controlHelper; specifically, we are able to test the `writeFile` portion, as the `shareData` method relies on external plugins that are difficult to mock.

the-bay-kay · 2023-10-16T23:33:47Z

... If you break getMyData into smaller functions you can probably validate the 'dump' without worrying about mocking filesystem. Function A creates the dump and function B writes the dump to filesystem; function A calls function B. You would test only function A.

Quick update on this! I split apart the two functions in commit 92cd6fb, and they seem to (upon manual testing) be functioning fine! My current issue, however, is with how i18nextInit.ts (used bycontrolHelper) interacts with Jest. I've run into two small hiccups:

i18nextInit.ts relies on the global variable, __DEV__: This can be set in jest.config.json, or set up during the individual tests. Is there any reason we shouldn't set this globally? Jest will only be running in a dev environment, and I can't find a reason why we'd want it disabled. I set it in the config file for now, but can move it into the test if we'd prefer.
I've run into an issue with the use of react-native-paper-dates . When Jest reaches line 73 of i18nextInit.ts, it throws a syntax error as follows:

    Details:

    /Users/katier/Documents/e-mission-phone/node_modules/react-native/Libraries/Utilities/codegenNativeComponent.js:18
        paperComponentName ?  : string,
                              ^

    SyntaxError: Unexpected token ':'

      71 |
      72 | // Next, register the translations for react-native-paper-dates
    > 73 | import { en, es, fr, it, registerTranslation } from 'react-native-paper-dates';
         | ^
      74 | const rnpDatesLangs = {
      75 |   en, es, fr, it,
      76 |   lo: loJson['react-native-paper-dates'] /* Lao translations are not included in the library,

I still need to do further investigation as to why exactly this issue is happening! As I discussed with Abby, some of the issues with the i18next_ utils stem from this branch being somewhat out of date, compared to the e-mission-translate repository. Could this be part of the issue?

The only function that we need to test is the createWriteFile() method, which doesn't rely on i18nextInit.ts. I'd be curious to see if we could test the function in isoluation, without dealing with i18next. That being said, I don't want to "kick the can down the road" and wait until something else breaks, so I'll go ahead and keep searching for fixes!

JGreenlee · 2023-10-17T04:44:14Z

See @Abby-Wheelis's PR #1063 for ideas on how to address both of these.

(1. You can define __DEV__ as a global in the jest config and 2. You can pull in the changes from #1049 and this will allow Jest to not get tripped up on the React code)

the-bay-kay · 2023-11-10T16:57:01Z

Also, what is the status of this PR? Did we ever figure out how to test it? If not, can we get this out now and worry about that later?

I'm pretty stumped on how to write tests for this. The issue lies in the fileEntry.onwriteend() and .onerror() methods. I've tried assigning them to a global getFileWriter object, and then calling the passed mocks from .write(dataObj), but this didn't work.

My understanding of the fileEntry object is that the the two "onend" methods are event listeners, but I'm unsure how to mock these. A normal jest.fn() doesn't work, because we need to call the resolve() method that's in scope for these functions. Normally I'd pass the resolve in as a higher order function, but none of the functions under window['resolveLocalFileSystemURL'] can accept a function like this.

I'll go ahead and clean up the current PR, resolve the conflicts, and push what I have! I've manually tested on iOS & Android, with a few emulators and OPCodes, and everything seems to run smoothly.

the-bay-kay · 2023-11-17T00:40:16Z

In manual testing, I've run into another issue with this rewrite. When loading the JSON files, all seems to be well when loading files without usercache data: however, if a given timeline file has usercache data, the timeline appears blank, and does not have any data loaded.

If anyone has a .timeline file that contains usercache data, I would greatly appreciate a copy to test! It seems that none of the files in the server's emission/tests/data/real_examples folder have usercache data. I believe this issue occurs within the intake-sinigle-user phase of loading the server data. I generated my timeline file using this PR, using my open_survey opcode - I would be curious to see if older timelines still have this issue, that would help me narrow down the issue!

the-bay-kay · 2023-11-17T19:09:47Z

Update: the UserCache data doesn't seem to be the issue. After pre-processing the JSON dump and filtering out the userCache data, the timeline loads, with the following error. Interestingly, the intake-single-user errors don't occur when using this data. I'll spend some more time trying to hunt down the issue!

the-bay-kay · 2023-11-17T19:17:38Z

Update: the UserCache data doesn't seem to be the issue. After pre-processing the JSON dump and filtering out the userCache data, the timeline loads, with the following error. Interestingly, the intake-single-user errors don't occur when using this data.

Interestingly, this issue doesn't occur when the file is loaded into the Android emulator. Here is a side-by-side of the same timelines loaded on both iOS and android - on iOS, the aforementioned error is thrown, and there are no detected modes. On android, however, the file loads without any issue. Here is a copy of the timeline file (download) - I would be curious to see if other folks are having this same android/ios discrepancy!

UPDATE: It seems that all of the current OBJDumps seem to load fine on my Linux/Android setup... I'm beginning to suspect there's an issue with my current e-mission-server repo on MacOS, will look into re-cloning.

Cleaned up branch for review

- Android was having issues with the new fileIO - since we are not mocking these functions, we can switch back for now.

JGreenlee · 2023-11-17T22:07:32Z

If anyone has a .timeline file that contains usercache data, I would greatly appreciate a copy to test! It seems that none of the files in the server's emission/tests/data/real_examples folder have usercache data.

emission/tests/data/real_examples/shankari_2016-06-20.user_inputs has some mode and purpose labels, is that what you're looking for?

Found a fix for `['resolveLocalFileSystemURL']` method of fileIO, that works with Android and iOS. Reverting back to this method and updating, to make this consistent with `uploadService.ts` This reverts commit dbe76d1.

the-bay-kay · 2023-11-27T22:13:29Z

emission/tests/data/real_examples/shankari_2016-06-20.user_inputs has some mode and purpose labels, is that what you're looking for?

Not exactly! When loading this data (using the load_timeline_for_day_and_user.py), I get the following output at the end:

Finished loading 0 entries into the usercache and 17 entries into the timeseries

When loading my personal data, the script shows that 9 "usercache" entries have been loaded:

Finished loading 9 entries into the usercache and 29231 entries into the timeseries

~~My current theory is that this usercache data may be the issue.~~ I get multiple AttributeError: 'NoneType' object has no attribute 'data' errors. (E.x. : analysis_timeseries_queries.py throws an AttributeError on line 48, when called by line 200 of section_segmentation.py). When I go ahead and load the data, I am met by the error shown in the screenshots, and the timelines fail to load the predicted modes.

When running the JSONDump with timelines loaded from emission/tests/data/real_examples/, on iOS or Android, I haven't encountered this issue - this only occurs on my recent data, collected in the open_access survey. I've tested a few days from my opcode, and have gotten similar errors.

I'm pretty stumped on this bug, so I'm putting this into review for a second set of eyes - I'm not terribly familiar with the server, so I may start by combing through the server errors and trying to understand what's missing.

the-bay-kay · 2023-11-27T22:46:26Z

OK -- peculiarly, I loaded a timeline for Nov 24 from my live OPCode, and it seems to have loaded without any issue...? Looking into this further

EDIT: On my open survey opcode, Nov 25th and 23rd have the aforementioned bug, but Nov 24th doesn't -- with my initial inspection, I cannot find any discernible difference between these days (beyond the superficial difference in trips). Notably, this timeline has the usercache data loaded, and the size of the datasets is similar - I was barking up the wrong tree, that isn't the cause. It seems the errors occur most frequently during the segmenting into sections portion of the intake, I'll continue investigating...

the-bay-kay · 2023-11-28T21:19:30Z

After some discussion in a meeting today, myself and others have determined this error may be similar to that documented in issue #936 . Below is a full copy of the errors found when running the intake on a faulty timeline file.
(Side Note: somewhere within this script, there is a zen of python somewhere in the script, which I've removed from this output.)

(emission) $ ./e-mission-py.bash bin/debug/intake_single_user.py -e nrelop_dev-emulator-program_katieBroken

storage not configured, falling back to sample, default configuration
URL not formatted, defaulting to "Stage_database"
Connecting to database URL localhost
analysis.debug.conf.json not configured, falling back to sample, default configuration
google maps key not configured, falling back to nominatim
nominatim not configured either, place decoding must happen on the client
overpass not configured, falling back to default overleaf.de
transit stops query not configured, falling back to default
[ Zen of Python Text ]
analysis.trip_model.conf.json not configured, falling back to sample, default configuration
expectations.conf.json not configured, falling back to sample, default configuration
ERROR:root:habitica not configured, game functions not supported
Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/net/ext_service/habitica/proxy.py", line 22, in <module>
    key_file = open('conf/net/ext_service/habitica.json')
FileNotFoundError: [Errno 2] No such file or directory: 'conf/net/ext_service/habitica.json'
2023-11-28T13:04:51.247143-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: moving to long term**********
New entry count == 0 and skip_if_no_new_data = False, continuing
2023-11-28T13:04:51.266319-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: updating incoming user inputs**********
2023-11-28T13:04:51.366549-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: filter accuracy if needed**********
2023-11-28T13:04:51.372261-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: segmenting into trips**********
2023-11-28T13:04:52.045295-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: segmenting into sections**********
Sectioning failed for user d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c
Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/segmentation/section_segmentation.py", line 52, in segment_current_sections
    segment_trip_into_sections(user_id, trip_entry, trip_entry.data.source)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/segmentation/section_segmentation.py", line 66, in segment_trip_into_sections
    distance_from_place = _get_distance_from_start_place_to_end(trip_entry)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/segmentation/section_segmentation.py", line 200, in _get_distance_from_start_place_to_end
    start_place = esda.get_object(esda.RAW_PLACE_KEY, start_place_id)
  File "/Users/jrischpa/Documents/e-mission-server/emission/storage/decorations/analysis_timeseries_queries.py", line 48, in get_object
    return get_entry(key, object_id).data
AttributeError: 'NoneType' object has no attribute 'data'
2023-11-28T13:04:52.059446-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: smoothing sections**********
Marking smoothing as failed
Traceback (most recent call last):
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'filter'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/location_smoothing.py", line 116, in filter_current_sections
    filter_jumps(user_id, section.get_id())
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/location_smoothing.py", line 142, in filter_jumps
    is_ios = section_points_df["filter"].dropna().unique().tolist() == ["distance"]
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'filter'
2023-11-28T13:04:52.077582-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: cleaning and resampling timeline**********
Cleaning and resampling failed for user d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c
Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py", line 88, in clean_and_resample
    last_raw_place = save_cleaned_segments_for_ts(user_id, time_query.startTs, time_query.endTs)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py", line 114, in save_cleaned_segments_for_ts
    return save_cleaned_segments_for_timeline(user_id, tl)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py", line 140, in save_cleaned_segments_for_timeline
    (last_cleaned_place, filtered_tl) = create_and_link_timeline(tl, user_id, trip_map)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py", line 951, in create_and_link_timeline
    curr_cleaned_start_place = get_filtered_place(tl.first_place())
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/clean_and_resample.py", line 246, in get_filtered_place
    _copy_non_excluded(old_data=raw_place.data,
AttributeError: 'NoneType' object has no attribute 'data'
2023-11-28T13:04:52.628696-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: inferring transportation mode**********
Error while inferring modes, timestamp is unchanged
Traceback (most recent call last):
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/ssl.py", line 1074, in _create
    self.do_handshake()
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/ssl.py", line 1343, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lz4.overpass-api.de', port=443): Max retries exceeded with url: /api/interpreter (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 24, in predict_mode
    mip.runPredictionPipeline(user_id, time_query)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 55, in runPredictionPipeline
    self.predictedProb = self.predictModesStep()
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 74, in predictModesStep
    predictedProb.append(get_prediction(i, section_entry))
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 113, in get_prediction
    return get_motorized_prediction(i, section_entry)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 140, in get_motorized_prediction
    predicted_transit_mode = _get_transit_prediction(i, section_entry)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 175, in _get_transit_prediction
    start_transit_stops = enetm.get_stops_near(section_entry.data.start_loc, start_radius)
  File "/Users/jrischpa/Documents/e-mission-server/emission/net/ext_service/transit_matching/match_stops.py", line 124, in get_stops_near
    stops = get_public_transit_stops(lat - bbox_delta, lon - bbox_delta, lat + bbox_delta, lon + bbox_delta)
  File "/Users/jrischpa/Documents/e-mission-server/emission/net/ext_service/transit_matching/match_stops.py", line 81, in get_public_transit_stops
    call_return = make_request_and_catch(overpass_query)
  File "/Users/jrischpa/Documents/e-mission-server/emission/net/ext_service/transit_matching/match_stops.py", line 30, in make_request_and_catch
    response = requests.post(url + "api/interpreter", data=overpass_query)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='lz4.overpass-api.de', port=443): Max retries exceeded with url: /api/interpreter (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))
2023-11-28T13:04:53.186132-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: inferring labels**********
2023-11-28T13:04:53.196621-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: populating expectations**********
2023-11-28T13:04:53.205540-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: creating confirmed objects **********
Error while creating confirmed objects, timestamp is unchanged
Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/userinput/matcher.py", line 96, in create_confirmed_objects
    confirmed_tl = create_and_link_timeline(ts, timeline, last_confirmed_place)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/userinput/matcher.py", line 124, in create_and_link_timeline
    curr_confirmed_start_place = create_confirmed_entry(ts,
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/userinput/matcher.py", line 226, in create_confirmed_entry
    confirmed_object_data = copy.copy(tce["data"])
TypeError: 'NoneType' object is not subscriptable
2023-11-28T13:04:53.226488-08:00**********UUID d1f18eb8-8aed-4bcc-8b34-b6b589c78b1c: creating composite objects **********

the-bay-kay · 2023-11-28T21:51:10Z

For reference, here's the output from running the intake pipeline on another JSON dump from my open survey, one that loads without errors.

(emission) $ ./e-mission-py.bash bin/debug/intake_single_user.py -e nrelop_dev-emulator-program_katieWorking
storage not configured, falling back to sample, default configuration
URL not formatted, defaulting to "Stage_database"
Connecting to database URL localhost
analysis.debug.conf.json not configured, falling back to sample, default configuration
google maps key not configured, falling back to nominatim
nominatim not configured either, place decoding must happen on the client
overpass not configured, falling back to default overleaf.de
transit stops query not configured, falling back to default
[ Zen of Python ]
analysis.trip_model.conf.json not configured, falling back to sample, default configuration
expectations.conf.json not configured, falling back to sample, default configuration
ERROR:root:habitica not configured, game functions not supported
Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/net/ext_service/habitica/proxy.py", line 22, in <module>
    key_file = open('conf/net/ext_service/habitica.json')
FileNotFoundError: [Errno 2] No such file or directory: 'conf/net/ext_service/habitica.json'
2023-11-28T13:43:48.249504-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: moving to long term**********
Got error None while saving entry AttrDict({'_id': ObjectId('65665f072bec2a55ca5f625c'), 'metadata': {'time_zone': 'America/Los_Angeles', 'plugin': 'none', 'write_ts': 1700876988.3018951, 'platform': 'ios', 'read_ts': 0, 'key': 'statemachine/transition', 'type': 'message'}, 'data': {'currState': 'STATE_ONGOING_TRIP', 'transition': None, 'ts': 1700876988.301783}, 'user_id': UUID('eada86fb-40ec-4cd1-8ca8-7862db147718')}) -> None
New entry count == 4991 and skip_if_no_new_data = False, continuing
2023-11-28T13:43:53.339275-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: updating incoming user inputs**********
2023-11-28T13:43:53.360571-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: filter accuracy if needed**********
2023-11-28T13:43:53.365240-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: segmenting into trips**********
2023-11-28T13:43:55.445780-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: segmenting into sections**********
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/segmentation/section_segmentation_methods/flip_flop_detection.py:59: RuntimeWarning: invalid value encountered in scalar divide
  sm.update({"trip_pct": (curr_section_time * 100)/total_trip_time})
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/segmentation/section_segmentation_methods/flip_flop_detection.py:59: RuntimeWarning: invalid value encountered in scalar divide
  sm.update({"trip_pct": (curr_section_time * 100)/total_trip_time})
2023-11-28T13:43:56.780488-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: smoothing sections**********
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
Caught error index -1 is out of bounds for axis 0 with size 0 while processing section, skipping...
Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/location_smoothing.py", line 195, in get_points_to_filter
    filtering_algo.filter(with_speeds_df)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/jump_smoothing.py", line 240, in filter
    self.find_segments()
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/jump_smoothing.py", line 127, in find_segments
    segmentation_points = self.get_segmentation_points_ios()
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/jump_smoothing.py", line 155, in get_segmentation_points_ios
    jump_to = self.with_speeds_df[(self.with_speeds_df.index < jump) & (
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 5320, in __getitem__
    return getitem(key)
IndexError: index -1 is out of bounds for axis 0 with size 0
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
Caught error index -1 is out of bounds for axis 0 with size 0 while processing section, skipping...
Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/location_smoothing.py", line 195, in get_points_to_filter
    filtering_algo.filter(with_speeds_df)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/jump_smoothing.py", line 240, in filter
    self.find_segments()
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/jump_smoothing.py", line 127, in find_segments
    segmentation_points = self.get_segmentation_points_ios()
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/jump_smoothing.py", line 155, in get_segmentation_points_ios
    jump_to = self.with_speeds_df[(self.with_speeds_df.index < jump) & (
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 5320, in __getitem__
    return getitem(key)
IndexError: index -1 is out of bounds for axis 0 with size 0
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
Caught error index -1 is out of bounds for axis 0 with size 0 while processing section, skipping...
Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/location_smoothing.py", line 195, in get_points_to_filter
    filtering_algo.filter(with_speeds_df)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/jump_smoothing.py", line 240, in filter
    self.find_segments()
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/jump_smoothing.py", line 127, in find_segments
    segmentation_points = self.get_segmentation_points_ios()
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/jump_smoothing.py", line 155, in get_segmentation_points_ios
    jump_to = self.with_speeds_df[(self.with_speeds_df.index < jump) & (
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 5320, in __getitem__
    return getitem(key)
IndexError: index -1 is out of bounds for axis 0 with size 0
2023-11-28T13:43:57.438092-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: cleaning and resampling timeline**********
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
/Users/jrischpa/Documents/e-mission-server/emission/analysis/intake/cleaning/cleaning_methods/speed_outlier_detection.py:27: FutureWarning: The default value of numeric_only in DataFrame.quantile is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
  quartile_vals = df_to_use.quantile([0.25, 0.75]).speed
2023-11-28T13:43:58.963839-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: inferring transportation mode**********
Error while inferring modes, timestamp is unchanged
Traceback (most recent call last):
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/ssl.py", line 1074, in _create
    self.do_handshake()
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/ssl.py", line 1343, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='lz4.overpass-api.de', port=443): Max retries exceeded with url: /api/interpreter (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 24, in predict_mode
    mip.runPredictionPipeline(user_id, time_query)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 55, in runPredictionPipeline
    self.predictedProb = self.predictModesStep()
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 74, in predictModesStep
    predictedProb.append(get_prediction(i, section_entry))
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 113, in get_prediction
    return get_motorized_prediction(i, section_entry)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 140, in get_motorized_prediction
    predicted_transit_mode = _get_transit_prediction(i, section_entry)
  File "/Users/jrischpa/Documents/e-mission-server/emission/analysis/classification/inference/mode/rule_engine.py", line 175, in _get_transit_prediction
    start_transit_stops = enetm.get_stops_near(section_entry.data.start_loc, start_radius)
  File "/Users/jrischpa/Documents/e-mission-server/emission/net/ext_service/transit_matching/match_stops.py", line 124, in get_stops_near
    stops = get_public_transit_stops(lat - bbox_delta, lon - bbox_delta, lat + bbox_delta, lon + bbox_delta)
  File "/Users/jrischpa/Documents/e-mission-server/emission/net/ext_service/transit_matching/match_stops.py", line 81, in get_public_transit_stops
    call_return = make_request_and_catch(overpass_query)
  File "/Users/jrischpa/Documents/e-mission-server/emission/net/ext_service/transit_matching/match_stops.py", line 30, in make_request_and_catch
    response = requests.post(url + "api/interpreter", data=overpass_query)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jrischpa/miniconda-23.1.0/envs/emission/lib/python3.9/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='lz4.overpass-api.de', port=443): Max retries exceeded with url: /api/interpreter (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))
2023-11-28T13:43:59.500719-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: inferring labels**********
2023-11-28T13:43:59.551722-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: populating expectations**********
2023-11-28T13:43:59.573080-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: creating confirmed objects **********
While getting section summary, section length = 0. This should never happen, but let's not crash if it does
While getting section summary, section length = 0. This should never happen, but let's not crash if it does
While getting section summary, section length = 0. This should never happen, but let's not crash if it does
While getting section summary, section length = 0. This should never happen, but let's not crash if it does
2023-11-28T13:43:59.978388-08:00**********UUID eada86fb-40ec-4cd1-8ca8-7862db147718: creating composite objects **********

JGreenlee · 2023-12-04T04:57:10Z

@shankari What do we expect to happen when we dump, and then intake, a day that begins with a trip (meaning the start_place is on the previous day)?

Does the dump need to include the place? Because if not, the start_place for that trip won't exist when it gets processed

JGreenlee · 2023-12-04T05:02:00Z

My current theory is that this is why some days exhibit this issue and some don't.

If the day started with a place (meaning you were staying there at midnight), that place gets included in the dump. When the first trip of the day gets processed, it has a start_place.
But if the day started with a trip (meaning the trip was ongoing at midnight), the start_place of that trip is missing.

JGreenlee · 2023-12-04T05:07:29Z

Or, alternatively, would we expect the cleaned and confirmed objects to be re-created anyway when we intake? And only the raw locations would be used from the dump?

shankari · 2023-12-04T06:08:11Z

Or, alternatively, would we expect the cleaned and confirmed objects to be re-created anyway when we intake? And only the raw locations would be used from the dump?

If we reset the pipeline (delete all analysis objects) and re-run the intake, cleaned and confirmed objects will be re-created.
If we don't reset the pipeline, then the objects will not be recreated. The pipeline is incremental and only works on new data for scalability - we don't want to be re-creating the entire timeline every single time.

@shankari What do we expect to happen when we dump, and then intake, a day that begins with a trip (meaning the start_place is on the previous day)?

I checked the code, and we don't specify the ``time_key`, so we get all entries that were written that day.
This should generally be fairly close to the time that the trip was taken since we push at the end of the trip and run the pipeline once a day, but it can be wrong if we reset + recreate the pipeline.
We should really fix this after checking what we did before the migration.

But to answer the original question, we will not return the first place for the day since it would have been created while processing the data for the last trip on the previous day.

      getRawEntries(null, startMoment.unix(), endMoment.unix())
        .then(writeDumpFile)
        .then(emailData)
        .then(function () {
          Logger.log('Email queued successfully');
        })
        .catch(function (error) {
          Logger.displayError('Error emailing JSON dump', error);
        });

JGreenlee · 2023-12-04T06:31:03Z

I don't think @the-bay-kay was resetting the pipeline as part of this process, so there should have been no new timeline data to process and the pipeline shouldn't have done anything to the timeline.

Yet it threw an error in the section segmentation code. If there should have been nothing new, why was it even re-segmenting at all?

JGreenlee · 2023-12-04T06:40:44Z

Also, I don't see any reason to think this is related to the migration.

@the-bay-kay can confirm whether the same thing happens on master for Nov 23, 24, and 25, but we have been observing issues with the JSON dump since before any of this was rewritten. If anything, the goal here was to make it functional again, which appears to have been at least mostly successful

If we can confirm that the issue affecting Nov 23 and 25 is not a regression, I suggest we open a separate issue for it and merge this in the meantime.

shankari · 2023-12-04T06:43:15Z

I'm going to have to look carefully at the data and potentially try to re-run it locally to debug further. So yes, if this happens on master, I am fine with investigating this issue later.

shankari · 2023-12-04T06:52:09Z

Can @the-bay-kay first confirm that this is not a regression? I am happy to merge fairly soon after that.

the-bay-kay · 2023-12-04T16:47:48Z

I don't think @the-bay-kay was resetting the pipeline as part of this process, so there should have been no new timeline data to process and the pipeline shouldn't have done anything to the timeline.

Can @the-bay-kay first confirm that this is not a regression?

Can confirm both! I just double checked, and loading this timeline onto a fresh clone of master has the same issue. If anyone is interested getting a copy of my opcode or these timelines for testing, LMK and I can send them over!

shankari

I am merging this in the interests of unblocking the final set of changes for the rewrite. However, although I see some expanded mocks, I don't actually see any new tests here. I would suggest that you add tests similar to the ones proposed in the discussion starting at
#1097 (comment)

I have also added some polishing, both large and small, for future fixes.

shankari · 2023-12-06T03:59:01Z

www/index.html

@@ -15,4 +15,4 @@
    <div id="appRoot" class="fill-container" style="width: 100vw; height: 100vh;"></div>
  </body>
  <script src="dist/bundle.js"></script>
-</html>
+</html>


extraneous whitespace

shankari · 2023-12-06T04:04:07Z

www/js/services/controlHelper.ts

+  // window['cordova'].file.cacheDirectory is not guaranteed to free up memory,
+  // so it's good practice to remove the file right after it's used!
+  const localClearData = function () {
+    return new Promise<void>(function (resolve, reject) {


Woo! Good addition to the existing functionality!

shankari · 2023-12-06T04:05:37Z

www/js/services/controlHelper.ts

+  const localWriteFile = function (result: ServerResponse<any>) {
+    const resultList = result.phone_data;
+    return new Promise<void>(function (resolve, reject) {


can this be combined with the similar function that saves loggerDB and userCacheDB?
Maybe create a new fileWriterService?

shankari · 2023-12-06T04:16:33Z

www/js/types/diaryTypes.ts

-  end_confirmed_place: ConfirmedPlace;
+  end_confirmed_place: ServerData<ConfirmedPlace>;


dumb typescript question for my edification: what does this change in aid of?

the-bay-kay added 2 commits October 4, 2023 09:40

Replaced moment objects with DateTime

1130508

`js/services.js` has been rewritten so that it no longer relies on the legacy `moments.js` library, and instead uses `luxon`.

the-bay-kay added 2 commits October 5, 2023 14:24

Updated Logger usage, cleaned up controlHelper

8825d06

Adjusted `js/services.js` such that it only uses the constants exported from `js/plugins/logger.ts`. Removed the member function `writeFile` from the controlHelper, as it was not being used.

the-bay-kay commented Oct 6, 2023

View reviewed changes

Rewrote controlHelper as a separate TS file

46832c7

controlHelper no longer utilizes Angular services. The internal functions were separated into their own exports. File write is confirmed to be working, need to write tests and verify all other functions are working as intended.

Small code-quality updates to controlHelper.ts

4960803

Changed all variables that do not mutate to const.

Email now uses socialShare library

6fd2e73

See [PR 1052](e-mission#1052). Using the socialShare library, as we do in the OpCode components, allows iOS users to share through apps other than the defauolt mail app.

the-bay-kay force-pushed the controlHelper-rewrite branch from 22d365a to 46832c7 Compare October 12, 2023 15:52

the-bay-kay marked this pull request as ready for review October 12, 2023 17:07

the-bay-kay changed the base branch from service_rewrite_2023 to master October 13, 2023 15:11

the-bay-kay changed the base branch from master to service_rewrite_2023 October 13, 2023 15:11

the-bay-kay mentioned this pull request Oct 13, 2023

🏗️🗃️ UnifiedDataLoader rewrite #1064

Merged

JGreenlee requested changes Oct 16, 2023

View reviewed changes

www/js/controlHelper.ts Outdated Show resolved Hide resolved

www/js/controlHelper.ts Outdated Show resolved Hide resolved

www/js/controlHelper.ts Outdated Show resolved Hide resolved

www/js/controlHelper.ts Outdated Show resolved Hide resolved

the-bay-kay added 2 commits October 16, 2023 10:54

Updated debug statements, format changes

3420577

Per discussion in PR, made adjustments to debug logs and funciton names.

Split getData into multiple functions

92cd6fb

By extracting the `writeFile` and `shareData` methods, we can now write unit tests for controlHelper; specifically, we are able to test the `writeFile` portion, as the `shareData` method relies on external plugins that are difficult to mock.

the-bay-kay added 4 commits November 10, 2023 10:56

Merge branch 'type_interfaces' into controlHelper-rewrite

9247efe

index.html fix

05b287a

Merge branch 'service_rewrite_2023' into controlHelper-rewrite

bc90720

Merge branch 'service_rewrite_2023' into controlHelper-rewrite

2177000

the-bay-kay added 2 commits November 17, 2023 11:31

Ran prettier, removed broken tests

2c02e37

Cleaned up branch for review

Switched to old fileIO, fixes issue with android

dbe76d1

- Android was having issues with the new fileIO - since we are not mocking these functions, we can switch back for now.

the-bay-kay added 2 commits November 27, 2023 13:25

Revert "Switched to old fileIO, fixes issue with android"

a232e3f

Found a fix for `['resolveLocalFileSystemURL']` method of fileIO, that works with Android and iOS. Reverting back to this method and updating, to make this consistent with `uploadService.ts` This reverts commit dbe76d1.

Changed fileIO directory to fix android issue

369b3ad

shankari approved these changes Dec 6, 2023

View reviewed changes

shankari merged commit db19a5b into e-mission:service_rewrite_2023 Dec 6, 2023
2 checks passed

the-bay-kay mentioned this pull request Dec 21, 2023

Download JSON Dump returning blank e-mission/e-mission-docs#994

Open

shankari mentioned this pull request Feb 3, 2024

🎀 Final change of the service rewrite #1126

Merged

		end_confirmed_place: ConfirmedPlace;
		end_confirmed_place: ServerData<ConfirmedPlace>;

🎛️ ControlHelper Rewrite #1052

🎛️ ControlHelper Rewrite #1052

Conversation

the-bay-kay commented Oct 4, 2023

the-bay-kay commented Oct 5, 2023

the-bay-kay commented Oct 5, 2023

the-bay-kay left a comment • edited Loading

Choose a reason for hiding this comment

the-bay-kay commented Oct 6, 2023

the-bay-kay commented Oct 10, 2023 • edited by shankari Loading

the-bay-kay commented Oct 11, 2023

the-bay-kay commented Oct 11, 2023

shankari commented Oct 11, 2023

the-bay-kay commented Oct 11, 2023

the-bay-kay commented Oct 12, 2023

the-bay-kay commented Oct 12, 2023

JGreenlee commented Oct 13, 2023

the-bay-kay commented Oct 13, 2023

JGreenlee left a comment • edited Loading

Choose a reason for hiding this comment

the-bay-kay commented Oct 16, 2023

JGreenlee commented Oct 17, 2023

the-bay-kay commented Nov 10, 2023

the-bay-kay commented Nov 17, 2023

the-bay-kay commented Nov 17, 2023

the-bay-kay commented Nov 17, 2023 • edited Loading

JGreenlee commented Nov 17, 2023

the-bay-kay commented Nov 27, 2023 • edited Loading

the-bay-kay commented Nov 27, 2023 • edited Loading

the-bay-kay commented Nov 28, 2023 • edited Loading

the-bay-kay commented Nov 28, 2023

JGreenlee commented Dec 4, 2023 • edited Loading

JGreenlee commented Dec 4, 2023

JGreenlee commented Dec 4, 2023

shankari commented Dec 4, 2023

JGreenlee commented Dec 4, 2023

JGreenlee commented Dec 4, 2023

shankari commented Dec 4, 2023

shankari commented Dec 4, 2023

the-bay-kay commented Dec 4, 2023

shankari left a comment

Choose a reason for hiding this comment

shankari Dec 6, 2023

Choose a reason for hiding this comment

shankari Dec 6, 2023

Choose a reason for hiding this comment

shankari Dec 6, 2023

Choose a reason for hiding this comment

shankari Dec 6, 2023

Choose a reason for hiding this comment

the-bay-kay left a comment •

edited

Loading

the-bay-kay commented Oct 10, 2023 •

edited by shankari

Loading

JGreenlee left a comment •

edited

Loading

the-bay-kay commented Nov 17, 2023 •

edited

Loading

the-bay-kay commented Nov 27, 2023 •

edited

Loading

the-bay-kay commented Nov 27, 2023 •

edited

Loading

the-bay-kay commented Nov 28, 2023 •

edited

Loading

JGreenlee commented Dec 4, 2023 •

edited

Loading