Skip to content

Commit

Permalink
store: Retry registerQueue on failure
Browse files Browse the repository at this point in the history
In addition to the test this adds, I wanted to manually test it
end-to-end, to help be sure this covered the scenario where this
retry is known to be needed in the wild:
 * The app is offline for a while, perhaps because the device
   is asleep.
 * The app comes online, tries polling, and finds the event queue
   has expired, so it attempts a re-register.
 * Before that completes (which after all takes several seconds
   if the realm is a large one), the app goes offline again.
 * That request's response therefore never reaches the app,
   and so when it eventually comes back online it needs to retry.

Step 1 is annoying to carry out literally, because it means
waiting 10 minutes for the event queue to expire.  To work around
that, I sabotaged the getEvents binding function to use a wrong
`queue_id` value:

    'queue_id': RawParameter('wrong' + queueId),

so that the server would always respond with a BAD_EVENT_QUEUE_ID
error, just the same as if the queue had expired.

Then to take the app offline and online again, I just turned
airplane mode on and off on my device.  Because I used a
physical device connected to my computer over USB, that caused
no interference to my watching the logs on the console.

In my manual testing, the retries worked perfectly: no matter
how many times I turned airplane mode on and off, or with what
timing, the app always returned to getting a fresh queue and
polling it for events.

Fixes: #556
  • Loading branch information
gnprice committed Mar 12, 2024
1 parent 3901aa7 commit b19154a
Show file tree
Hide file tree
Showing 2 changed files with 46 additions and 2 deletions.
18 changes: 17 additions & 1 deletion lib/model/store.dart
Original file line number Diff line number Diff line change
Expand Up @@ -605,7 +605,7 @@ class UpdateMachine {
final connection = globalStore.apiConnectionFromAccount(account);

final stopwatch = Stopwatch()..start();
final initialSnapshot = await registerQueue(connection); // TODO retry
final initialSnapshot = await _registerQueueWithRetry(connection);
final t = (stopwatch..stop()).elapsed;
assert(debugLog("initial fetch time: ${t.inMilliseconds}ms"));

Expand All @@ -628,6 +628,22 @@ class UpdateMachine {
final String queueId;
int lastEventId;

static Future<InitialSnapshot> _registerQueueWithRetry(
ApiConnection connection) async {
BackoffMachine? backoffMachine;
while (true) {
try {
return await registerQueue(connection);
} catch (e) {
assert(debugLog('Error fetching initial snapshot: $e\n'
'Backing off, then will retry…'));
// TODO tell user if initial-fetch errors persist, or look non-transient
await (backoffMachine ??= BackoffMachine()).wait();
assert(debugLog('… Backoff wait complete, retrying initial fetch.'));
}
}
}

Completer<void>? _debugLoopSignal;

/// In debug mode, causes the polling loop to pause before the next
Expand Down
30 changes: 29 additions & 1 deletion test/model/store_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,6 @@ void main() {
addTearDown(() => UpdateMachine.debugEnableRegisterNotificationToken = true);
}

// ignore: unused_element
void checkLastRequest() {
check(connection.takeLastRequest()).isA<http.Request>()
..method.equals('POST')
Expand All @@ -174,6 +173,35 @@ void main() {
users.map((expected) => (it) => it.fullName.equals(expected.fullName)));
}));

test('retries registerQueue on NetworkError', () => awaitFakeAsync((async) async {
await prepareStore();

// Try to load, inducing an error in the request.
connection.prepare(exception: Exception('failed'));
final future = UpdateMachine.load(globalStore, eg.selfAccount.id);
bool complete = false;
future.whenComplete(() => complete = true);
async.flushMicrotasks();
checkLastRequest();
check(complete).isFalse();

// The retry doesn't happen immediately; there's a timer.
check(async.pendingTimers).length.equals(1);
async.elapse(Duration.zero);
check(connection.lastRequest).isNull();
check(async.pendingTimers).length.equals(1);

// After a timer, we retry.
final users = [eg.selfUser, eg.otherUser];
connection.prepare(json: eg.initialSnapshot(realmUsers: users).toJson());
final updateMachine = await future;
updateMachine.debugPauseLoop();
check(complete).isTrue();
// checkLastRequest(); TODO UpdateMachine.debugPauseLoop was too late; see comment above
check(updateMachine.store.users.values).unorderedMatches(
users.map((expected) => (it) => it.fullName.equals(expected.fullName)));
}));

// TODO test UpdateMachine.load starts polling loop
// TODO test UpdateMachine.load calls registerNotificationToken
});
Expand Down

0 comments on commit b19154a

Please sign in to comment.