1.8.9.1 ECHOWORDS mode to encourage fluent reading

[JackMostow]

This issue was originally part of Revise READ UI/UX tracking #157:

Problem: get READ’s “blame the kid” strict left-to-right policy not to encourage kids to, read, one, word, at, a, time, unlike Project LISTEN’s Reading Tutor, which used “chase the kid” to track the kid through the text.

Solution:

1. Rename RoboTutor's current ECHO as REREAD because it rereads the sentence after the kid reads it.
2. Add an ECHO mode that works more like its namesake, Project LISTEN's Reading Tutor's ECHO mode.
3. Relax the strict left-to-right policy to tolerate occasional rejected or skipped words.

Currently RoboTutor underlines the word it expects next. It turns words green if accepted and red if not.
Requiring acceptance of each word before advancing to the next word discourages fluent reading due to false rejections.

Change this behavior as follows:

1. Modify position tracking to tolerate an individual missed word but not 2 in a row.
   a. MultiMatch aligns ASR output to text to minimize total penalties for mismatches and jumps. It may or may not be the right mechanism to tolerate isolated missed words, e.g. by aligning only to i, i+1, or i+2.
   b. The policy means that the expected next word can skip from word i to i+2 but not further.
   c. Underline the expected next word to indicate the current position i.
   d. The current position should not move backwards from word i to j < i because it would be confusing.
   e. The current position can skip from i to i+2 if the ASR output ends with ... word i-1 word i+1.

2. Color words green when accepted, red when rejected, and (still) black when skipped.
   a. Or should skipped words turn red?
   b. It's ok to visibly accept and credit a previously rejected word, but not to rescind credit once granted.

3. When the reader pauses (0.5 sec) before reading word i, echo word(s) j..i-1, where word j-1 was the last echoed word, and turn each word green after echoing it. When RoboTutor reads just one word, use its isolated narration for clarity. Otherwise use the appropriate subsequence of its human narration. Note that:
   a. Reading, word, by, word will echo a word 0.5 sec after it's spoken, but wait patiently for the next word.
   b. Reading a phrase at a time will echo the phrase 0.5 sec after it's spoken.

4. Treat a tap almost like reading the current word:
   a. A tap after a hesitation will just read (and credit) the current word i.
   b. A tap with no hesitation before word i will reread the word(s) j..i. I don't expect this case to occur often.

Notes:

1. Project LISTEN's Reading Tutor changed the background color of each word to yellow while reading it, to show its position. This would be a nice feature if there's time, to make clear which word is being read.

2. Project LISTEN's Reading Tutor deferred crediting words green rather than turn them green right away. We can implement that behavior if the behavior above causes kids to wait for each word to turn green before reading the next word.

3. Replacing underlining with RoboFinger would be cute but is not worth the effort, especially if it requires representing the position of each text word on the screen.

[JackMostow]

Actually we can dispense with REREAD and just use ECHO, provided we modify ECHO to reread the entire sentence if the kid read it disfluently, in which case rereading it scaffolds comprehension.

[JackMostow]

@octavpo - can you implement ECHO by August 28 when I return?

[JackMostow]

@octavpo - Let me rephrase that request more emphatically:

XPRIZE weights reading 60%, ECHO tries to addess severe problems in RoboTutor's current ASR-driven assisted oral reading, and is important to kid-test ASAP.

So please implement ECHO in time for me to try out on August 29. If you have any questions before then, please email them to me.

I'll be offline Friday and Saturday PST, but will reply ASAP. I'll be travelling all day Tuesday, August 28.

Thanks. - Jack

[JackMostow]

Octav email of Saturday, September 01, 2018 6:29 PM:

... I'll try to put more details on the Github issue, but in short I found that in my tests most cases when a word is marked red are due to the fact that works are recognized and lost rather than because they're not recognized correctly. So I'm trying to fix that first.

Jack email of Sat 9/1/2018 9:47 PM:

“Words are recognized and lost” is very interesting. Based just on your own testing, or can you tell from the verbose logs from use in the field? How?

Please send a real example. From your brief description, I suspect a bug in the MultiMatch process that aligns ASR output against the text to read – not in MultiMatch itself, but perhaps something like omitting utterances from the speech aligned against the text if newer utterances have come in.

[JackMostow]

@octavpo - #357 said:
READ often fails to respond to oral reading miscues. Why?
a. The speech is not loud enough to pass the detection threshold, perhaps due to a noisy environment.
b. The ASR recognizes something but RoboTutor doesn't respond.

Note that a word turns red only if RoboTutor detected speech, i.e. not in case a.

[JackMostow]

@judithodili suggested that lowering the speech detection threshold would make ASR hallucinate more.
@judithodili and @amyogan - What if RoboTutor detects no speech for [3] seconds?

Options:

a. Don't supply the word automatically, but cue the kid to tap, e.g. have RoboFinger tap the audio icon
+: keeps RoboTutor from advancing without the kid doing anything
+: clearer than just displaying the audio icon
+: simple

b. Treat the word as misread, i.e. turn it red.
+: if kid attempted the word, responds visibly if not immediately
-: if kid was silent, might cause confusion or frustration due to too much time pressure

c. If RoboTutor still detects no speech, invoke the 2-strike rule, i.e. supply the word.
+: alleviate frustration caused by RoboTutor failing to detect kid's attempts to read the word
-: lets RoboTutor keep reading even if the kid says nothing

I favor option a. Do you concur?

[amyogan]

Yes, I like keeping the kids actively involved!

[JackMostow]

@octavpo - Please implement option a.

[octavpo]

Please send a real example. From your brief description, I suspect a bug in the MultiMatch process that aligns ASR output against the text to read – not in MultiMatch itself, but perhaps something like omitting utterances from the speech aligned against the text if newer utterances have come in.

The problem is not the MultiMatch, which is pretty much disabled, but rather a combination of factors, the main one being that after each word the listening process is restarted and any extra words or parts that might have been recognized before are erased. So for instance in a sequence like 'na mitsu' it might send only 'na' (because of a requirement that words are stable for 300ms) and then 'mitsu' is skipped.

I tried to fix the issue by removing the restart but I wasn't able to make it work yet. So Ive switched and now I'm working on implementing the features requested in this issue.

[JackMostow]

@octavpo - Discussing how Listener works would have been much more timely months ago, and might already be too late.

The above description is frustratingly vague -- e.g. that MultiMatch is "pretty much disabled." A "real example" would be well-chosen to illustrate an issue and include:

• the input text sentence
• the .wav input, if possible
• the ASR output, preferably the stream of partial hypotheses
• the MultiMatch output
• the resulting tutor response

Unless Listener has changed drastically since the Reading Tutor, you may (or may not) be confusing two requirements:

1. A 300 ms silence threshold to terminate an utterance
2. A shorter threshold to consider a word stable.

Also, it's possible that ASR output is supposed to be concatenated over multiple utterances as input to MultiMatch, rather than input to MultiMatch one utterance at a time to track the reader through the text.
Unfortunately I don't recall if we implemented the concatenative approach or merely contemplated it.
I'm not sure if it's equivalent to what you called "removing the restart".

Now that you're back, we should meet ASAP to look at a real example and trace the code involved, but I won't be in till 9/26. For right now, please post a real "real example" as I requested on 9/1/2018, with a detailed explanation including links to the specific relevant locations in source files.

[JackMostow]

From: Octav Popescu [email protected]
Sent: Thursday, September 27, 2018 6:39 PM

I have returned from vacation on 9/17, but I've returned to LA. I'm coming back to Pittsburgh on 10/16.
The skipping aspect of issue #348 (parts 1 and 2) do have the effect to compensate the issue with dropping words, although maybe not in all cases. I've implemented it to happen in all modes that involve listening to students, not just ECHO mode, I don't see why it would be restricted to that. The aspect related specifically to ECHO mode (parts 3 and 4) probably won't have any influence.
Octav

[JackMostow]

So are these the old -> new behavior changes of the modes that listen to students, where "sentence" means a text segment as long as a sentence or as short as one word?

• PARROT: RoboTutor reads sentence, then kid reads entire sentence fluently -> RoboTutor reads sentence fluently, then kid repeats with RoboTutor echoing each word/phrase

• ECHO: Kid reads sentence, then RoboTutor reads entire sentence fluently -> Kid reads sentence with RoboTutor echoing each word/phrase, then RoboTutor reads entire sentence fluently

• READ: Kid reads sentence -> Kid reads sentence, with RoboTutor echoing each word/phrase

• REVEAL: Kid recites sentence with RoboTutor showing each word as kid says it -> Kid recites sentence with RoboTutor showing each word and echoing each word/phrase

[octavpo]

No, not sure why you think it would work like this. If it's related to my previous message, I was trying to say that the word skipping behavior described in the first two points under changes in the first message is common to all modes, not the new echo mode.

[JackMostow]

Thanks for clarifying. I misunderstood 1-4 as applying to all modes, not just 1-2 as you in fact said.

[JackMostow]

@octavpo (@kevindeland, @judithodili , @eyarzebinski , @amyogan take note) -
I need your update by this Wednesday 10/3 for it to have any chance of getting into the upcoming beta test.

[JackMostow]

@octavpo - When will your update be ready? - Jack

[octavpo]

I'm trying to finish it but it's no easy. The skipping code works fine. The echo mode still has a few issues, not sure how long I need to finish it.

[octavpo]

Btw one issue I found and I can't solve on my own, probably familiar to you, is that the segmentation timing is not very accurate. In my testing it seems it cuts off the last word, so I added another 100ms after it. Which seems to be fine for the few examples I was running, but probably not a good general solution.

[JackMostow]

@octavpo - Can you TODAY (Wed. 10/3 your time) make a stable branch with just the skipping code, so we can try it ASAP?

@kevindeland needs all code by Thursday night 10/4 or Friday (I forget which) to prepare to deploy Monday.

Please point me to an APK. I assume you know how to generate them in general, but ask @kevindeland if you need RoboTutor-specific help (e.g. regarding assets).
Please turn on skipping by default but include a switch (e.g. a configuration parameter) to disable it easily.
Ask @ealanhill how to use his configuration parameter mechanism.
Make sure it behaves reasonably for 1-word "sentences".
Does it behave visibly as if accepting individual skipped words? That is, does it fix most false rejections?
If so I want it even if you don't fix echo mode in time -- though I still hope that you will, in which case it too should have a switch. One lesson from Project LISTEN is that a feature is something you can turn off.

[octavpo]

It took a while to merge it with the current development branch, but I think it's fine now. The PR #386 includes the previous change about having different reading modes for successive sentences (which hadn't been merged into development for some reason), since they're in the same tutor.

Not sure about APKs, where do you guys put them? Not sure where I can find a tutor with 1-word sentence to test, I haven't tried that.

There is a switch in the code to enable/disable skipping, but not in configurations, I'll see about that. It does accept skipped words but leaves them black, as requested. Not sure if it fixes most false rejections, it can still happen that the next word is again wrong, and then it's rejected.

[JackMostow]

@octavpo - I wish you had just put the apk somewhere instead of incurring further delay by waiting for me to say where. Please put it in Downloads from any team member, and post a link to it here so that I'll get notified.

Any of the word reading activities have lists of individual words to read.

@kevindeland - Any requests/suggestions to Octav for changes to his test apk, e.g. where or how to include it under the activity matrix or CUSTOM menu in the debugger menu if it doesn't simply affect all story reading activities with modes that listen to the kid (PARROT, ECHO, READ, REVEAL)?

Thanks. - Jack

[eyarzebinski]

@octavpo story_4 as it appears in apk 2.3.0.1 (note that there was a story resort, so this might not be story_4 in old apks) has a 1 word sentence:
first sentence: Anaitwa Bi Simu.
second sentence: Mwangalie!

[ealanhill]

@octavpo Looks like something went wrong with the build process, you're missing an important class in the APK. Here's the full stack trace:

2018-10-05 11:24:17.708 6887-6887/cmu.xprize.robotutor E/AndroidRuntime: FATAL EXCEPTION: main
    Process: cmu.xprize.robotutor, PID: 6887
    java.lang.RuntimeException: Unable to instantiate activity ComponentInfo{cmu.xprize.robotutor/cmu.xprize.robotutor.RoboTutor}: java.lang.ClassNotFoundException: Didn't find class "cmu.xprize.robotutor.RoboTutor" on path: DexPathList[[zip file "/data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/base.apk"],nativeLibraryDirectories=[/data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/lib/x86, /system/fake-libs, /data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/base.apk!/lib/x86, /system/lib, /vendor/lib]]
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2718)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2892)
        at android.app.ActivityThread.-wrap11(Unknown Source:0)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1593)
        at android.os.Handler.dispatchMessage(Handler.java:105)
        at android.os.Looper.loop(Looper.java:164)
        at android.app.ActivityThread.main(ActivityThread.java:6541)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:767)
     Caused by: java.lang.ClassNotFoundException: Didn't find class "cmu.xprize.robotutor.RoboTutor" on path: DexPathList[[zip file "/data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/base.apk"],nativeLibraryDirectories=[/data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/lib/x86, /system/fake-libs, /data/app/cmu.xprize.robotutor-AJ1Oqx9PbF2tGDB-CLxsfw==/base.apk!/lib/x86, /system/lib, /vendor/lib]]
        at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:93)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:379)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
        at android.app.Instrumentation.newActivity(Instrumentation.java:1173)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2708)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2892) 
        at android.app.ActivityThread.-wrap11(Unknown Source:0) 
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1593) 
        at android.os.Handler.dispatchMessage(Handler.java:105) 
        at android.os.Looper.loop(Looper.java:164) 
        at android.app.ActivityThread.main(ActivityThread.java:6541) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240) 
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:767) 

In order to avoid this, please follow the directions found in the link Kevin provided (https://www.quora.com/How-do-I-build-an-APK-in-Android-studio).

If you have the lastet version of Android Studio installed, there are a few slight differences. Go to Build > Generate Signed Bundle / APK, select the APK radio option and click Next. From there follow the directions in the link to generate a signing key and the APK.

@kevindeland @JackMostow if there is a branch that @octavpo has been working off of to create this, I might be able to generate a quick APK if y'all provide the branch name/link.

[JackMostow]

@ealanhill - Judging from https://github.com/RoboTutorLLC/RoboTutor/branches/active:
reading_modes updated 6 hours ago by @octavpo
remote_configuration updated 3 days ago by @ealanhill

[ealanhill]

@JackMostow I uploaded a debug build to the same folder: https://drive.google.com/open?id=1ZUIHwtaGq44rrgrfF3loIcBX0PErcurS

It has all the classes, but I'm concerned it won't work because it's a debug version that is signed by my debug cert, so it may throw a fit when you install it on a device. If it doesn't, great!

The version code looks to be off from the one @octavpo created, but it is built from the latest on the reading_modes branch. Here's to hoping it works and provides you what you need to complete the testing.

[JackMostow]

Just saw your post. Will try immediately!

[JackMostow]

Good news: installs, opens, loads assets.
@octav - Bad news: story.hear::story_54__it takes at least several seconds to load with no sign of progress, just a gray screen.
It keeps repeating the sentence "Hapo zamani ... blanketi" but the first word stays underlined.
story.parrot::story_4 also takes a long time to load and then keeps repeating the first sentence.
story.echo::story_4 listens to first sentence but then keeps repeating it.
story.parrot:: (parrot letter A) also takes a long time to load and then keeps repeating the first sentence.

Good news: not the fault of the apk building!

[JackMostow]

@ealanhill - Thanks for cracking the apk issue.
Can you clarify "The version code looks to be off from the one @octavpo created"?
Does it have anything to do with the version number being 2.0.0.0?

@octavpo - story.read::story_22 doesn't have the bug above, but it has as much trouble as ever recognizing my voice.

RoboTutor wrote the recording as RoboTutor_debug.2.0.0.0_2018.10.05.13.59.56_6105001158.wav, but it sounds severely sped-up.

[ealanhill]

@JackMostow yes, it says that it's version 2.0.0.0, and I think Robotutor is much higher than that

[octavpo]

The code on reading modes is 2.3.0.1, so I don't know what built you guys are using, but it's not from the current branch code.

I was trying to build an apk myself, but since @ealanhill was saying I should be on the latest version and I had ignored Studio's message to update for the last week, I thought it would be a good idea to do so. But then the update went wrong, so I had to reinstall it. That worked, but now when I try to build I get a message saying I need to select Android SDK. No idea how to do that. Any help? I'm an Android 3.2.

[octavpo]

@JackMostow Before trying to update Studio I had run story.hear::story_54 and story.read::story_22, and they both work fine. 'parrot' and 'echo' tutors are not available on the matrix generated by this branch, as they're not on development either. That shows again you're not running my code.

[octavpo]

Ok it looks like I needed to do some sync-ing, now it compiles. I'll build an apk soon.

[JackMostow]

@octavpo - 5pm today is the deadline to submit code to Kevin after our testing shows that it works properly. You were off-line for 8 hours. Meanwhile, @ealanhill was kind enough to find and fix the bug in your code and build an apk I could finally run.
The audio it recorded is bad, and the ASR seemed worse than without your changes. Under the circumstances I decided we had to do without them.
Your apk should use what 2.4 or Kevin's next APK will. No promises.

[octavpo]

What you're saying about is not true, @ealanhill hasn't fixed anything and gave you an apk that's not the current state of the reading modes branch. And up to now it wasn't my job to create apks, so no surprise i didn't know how to do it. I do need to sleep from time to time.

Anyway, I've built an apk with the code from the branch and put it in the download area if you still want to test it, as you wish. It's still 2.3 at this point, I'll work on merging the latest development code next. I don't have a problem if you don't want to include it, but the speech recognition is working definitely much better in my testing.

[JackMostow]

I’m recording Leonora till ~5 but will then try your apk.
Are you sure Alan’s apk didn’t include your branch?

[octavpo]

I'm sure it wasn't the current version on the branch, since it was showing 2.0 instead of 2.3 and it had a different activity matrix.

I've rebased my branch to the latest version on development, 2.5.0.0, and put a new apk in the download area. I don't get an activity selector matrix since the merge though. I thought it was because I had chosen a release version, but now I chose debug and I still don't see it.

[octavpo]

It seems the missing activity selector matrix is because Kevin has just pushed the external configuration facility to the development branch. So the code now defaults to a release version unless you have a configuration file that says otherwise. So to get it back you need to push the config.json file I just put in the Google drive download area to /sdcard/Download.

[JackMostow]

octav_robotutor.release.2.3.0.1.apk had the same repeated-sentence problem and no debugger menu so I couldn't test ASR. I just installed octav_robotutor.debug.2.5.0.0.apk.

[JackMostow]

Despite "debug" in the name, it has no debugger menu. Is there a configuration file to enable it?

[octavpo]

See my message above.

[JackMostow]

octav_robotutor.debug.2.5.0.0.apk also takes a long time to display a story and then has the repeated-sentence bug.
Just saw your configuration file post.

[octavpo]

I don't see the issues you're seeing, it works fine on my tablet. Maybe I need to build it for an older API? What Android version are you running on your tablet?

[JackMostow]

The configuration file enabled the debugger menu, thanks.
story.read_41:;story_41.vdebug_sw takes a very long time to load.
It can tolerate skips but has such a high false rejection rate that it doesn't help much.
It still gets stuck on the last word of a sentence, which I gather can't be skipped.

[JackMostow]

I'm running Android 7.1.2 on a Google Pixel C. You?

[octavpo]

Now I do see an error when running the signed version, which I'm not getting when running from Studio. And it does slow down tutor starts, that's probably what you're getting. It's something about class access, I'll see if I can do anything about it. It might be an issue with gradle built files, which I don't know much about.

I don't see the high false rejection rate, in my tests I'd say it has a high false acceptance rate. That's due to the very constraint language model (basically just the current sentence), I can't do much about. For me very rarely it's rejecting a word. Maybe it has something to do with your American accent.

I did see that sometimes it gets stuck on the last word and I needed to repeat it a few times. It looks like that's something with the recognizer itself, I don't see it returning any words. The last word can be skipped, but only if the recognizer returns something. Not sure I can fix that, I'll see.

[octavpo]

I'm running Android 8.1.0. Now that I saw the error I'm guessing it has nothing to do with this.

[octavpo]

I fixed the error that was slowing tutor load down. It was an issue left from the reading mode changes, very strange that it wasn't showing when running from Studio. So I put a new 2.5 apk in downloads if you want to try it.

About the high rejection rate, is it possible to have it tested by a native speaker? Or maybe at least a non-native English speaker? I definitely see a big improvement in my testing.

[octavpo]

I probably don't need to say this, but of course it also matters if you're testing in a quiet environment or with a headset. It still takes other noises and voices and tries to match them, and then because of the very restricted language model it will probably succeed to return a word that's not the right one.

[JackMostow]

XPRIZE doesn't allow headsets or any peripheral devices. Were you using a headset mic?
I've tested in quiet environments, but some environments we saw in Bagamoyo were anything but, due to 5 kids using RoboTutor at a time in a limited space.
What if any sort of silence compression does your session audio recording do, and when -- while recording or when writing to a file? The playback sounds sped-up.
Does the recording include enough metadata about which speech was recorded when so that it can play back the recording at the original rate to synchronize with screen video?
The ideal test would be on speech extracted from session audio recorded at our beta sites.

[JackMostow]

@octavpo - The slowdown error was in the reading_modes branch not yet absorbed into the development branch, right? Your changes were too late to test much, so I'm skeptical that they'll make it into the beta.

[octavpo]

I was just wondering if ambient noise can explain why you're getting a different experience than me. I'm using a headset when there's noise around, but I've tested without one too and I'm getting similar results.

I'm not doing any silence compression. I just retested and I don't see any playback speed-up on my tablet. Could it be some setting on your tablet? Might help to restart it.

All changes are on the same branch.