Chart Generation Taking a Long Time (or not happening at all)

[ryanleesipes]

Describe the bug
Generating charts for my inbox seem to take a very long time (I have yet to see the charts actually display). Have waited for charts to appear, assuming it is because of my large inbox. But have waited for likely an hour with the tab open at this point and still no charts. The spinner is spinning so I assume something is happening, but I'm not sure if I should continue to wait or if it's a bug,

To Reproduce

1. I click on the button to display the stats dashboard.
2. The top-line stats display, such as: Mails total, Mails unread, Mails received, etc
3. None of the charts display information. Just black where the charts should show up, the titles however are there.

Expected behavior
Expected the charts to appear and update as more information was processed, or the charts to show up faster.

Screenshots

System (please complete the following information):

• OS: elementaryOS 5.1.7
• Thunderbird Version 78.3.1 (64-bit)

[devmount]

This definitely is a bug. I have an inbox with ~30k emails which takes 13s to generate the charts. This is still quite long, so performance should definitely be improved, but never finishing chart generation is a bug.

Do you have multiple email accounts? Does the popup loading spinner finish (before clicking your account and opening the stats page)?

[ryanleesipes]

Just one account.

The spinner does not seem to stop if I let it sit before opening the page (in the pop over menu).

[devmount]

Ok, thank you for your feedback! I'll try to reproduce this and come back as soon as I have a solution.

[ryanleesipes]

I can look at the console or anywhere you think there might be useful information.

[devmount]

I set up a virtual environment with elementaryOS 5.1.7 and Thunderbird Version 78.3.1 (64-bit) and wasn't able to reproduce your issue. So I assume it must be something with your emails or folder structure.

Can you please do the following:

• open the console (add-ons > gear > debug add-ons > scroll to ThirdStats > Inspect)
• clear console (small trash icon on the upper left)
• open (or right click > reload if its already open) the Stats tab
• make a screenshot or copy any errors/warnings and paste it here

Thank you!

[devmount]

Plus:
I just released v0.4.0 with improved loading indication (you shouldn't see empty charts anymore now). Maybe this changes anything for you, @ryanleesipes?

[ryanleesipes]

Here is the screenshot you requested.

I do like the new indicator:

[devmount]

Thank you for the screenshot! I created a possible fix. To test it, please do the following:

1. Clone this repo git clone https://github.com/devmount/third-stats
2. Change into the created directory cd third-stats
3. Switch branch git checkout fix-infinite-loading
4. Open Thunderbird and uninstall ThirdStats, if you already installed it
5. Open Thunderbird > main menu > add-ons > gear menu > debug add-ons > load temporary add-on > choose the manifest file in the cloned repo's directory
6. Check, if charts are loaded now. If not, please create a screenshot of the console again.

Thank you very much @ryanleesipes for taking the time to test and debug this issue.

[ryanleesipes]

So two things to note (one might be the issue).

I installed Thunderbird on this machine via Flatpak so I wonder if that is causing issues. When attempting to load the temporary add-on I get the following message:

Extension is invalid
Loading locale file _locales/en/messages.json: Error: Error while loading 'file:///run/user/1000/doc/c28881a6/_locales/en/messages.json' (NS_ERROR_FILE_NOT_FOUND)

This could be because Flatpaks have their own filesystem, I'm troubleshooting.

[devmount]

Interesting, is that file actually existing on that location? Looks like the root directory of ThirdStats was not properly recognized... I don't have any experience with Flatpak, so thank you for troubleshooting!

[koobs]

I'm seeing the same issue on:

• Windows: 10 Pro x64 (2004, 19041.572)
• Thunderbird: 82.0b3 (64-bit) (Installed via AMO)
• Third-Stats: 0.5.0

Additional symptoms to report:

• The processing hangs after the same number of messages are parsed. In my case: 82,562 (now 82,564 after 2 new messages are received)
• I see the same errors in debug addons console output as @ryanleesipes

However, when I view the standard console output (not via debug addons), I see some additional errors that dont appear to show up during an 'addon debug' context. This is with the full-page third-stats loading, not just stats via the menu icon.

Full Console Log: console-export-2020-10-20_21-33-0.txt

Looking at the process stack (process explorer, see screenshot), the only potentially interesting thing I can see is en elevated number of calls to RegExprParser::* and configthreadlocale activity after the mail parsing process hangs, which continues until closing the third-stats tab. The number of calls to TargetNTUnmapViewofSection is at least two orders of magnitude higher too (baseline is <= 0.03 CPU when the issue is visible)

Additional information. Unsure if relevent or related:

• The processing (count) seems to restart at zero each time the toolbar icon is opened -> closed -> opened (may be intended)

[devmount]

Thank you @koobs for this detailed description and error analysis! 👏🏻 This definitely helps. Have you checked the possible fix I created? Does it change anything?

Thanks to your error log I can do some additional tests and see if I can find what's causing this issue.

The processing (count) seems to restart at zero each time the toolbar icon is opened -> closed -> opened (may be intended)

A click of the toolbar icon always reloads the popup content, so the counts are processed again - I can see if it's possible to implement a storage for already processed email/folder counts. If this is a feature you would like, you can create a feature request for that.

[tonewarper]

I use TB 78.4 on MacOS Catalina 10.15.7 … and experiencing the exact same behaviors. You’re seeing this on 3 OS’s , so it does not seem OS-related.

[devmount]

Thank you @tonewarper for confirming this issue. Unfortunately I cannot reproduce this on my own machines, so I'm waiting for someone to test the possible fix. I can provide the fix as XPI file, if that helps testing it.

[devmount]

Next version of ThirdStats contains a possible (!) fix, let's see if this issue persists.

[devmount]

@tonewarper @koobs @ryanleesipes The possible fix is now public in the Thunderbird add-on repository. Please update ThirdStats to v0.7.0 and let me know, if you still have this issue. Thank you all for testing and improving this add-on!

[koobs]

@devmount Success! Thank you 👍 Now we can focus on overwhelming you with feature requests ;D

Which commit was the likely fix for the issue? (Edit: Im guessing 9750684 ?)

[devmount]

Great! I'm very happy that it works for you now 👏🏻

Now we can focus on overwhelming you with feature requests ;D

Go ahead, I'm looking forward to your ideas 💡👍🏻😅

Yes, it was 9750684

[tonewarper]

I am the bearer of unpleasant news… on my end, this very-same behavior still persists on v0.70 , even after restarting TB twice. Perhaps it worked for @koobs on Windows while I’m on MacOS? Or maybe there is another variable at play. Most of my mails are fetched via IMAP on Gmail servers. It’s peculiar that Thunderbird Stats is able to produce a count of the mails but doesn’t generate the charting. The counter counts up while calculating, and at some point the counting stops while the wheel keeps spinning and the ‘Fetching’ loader remains where the charts should be… forever.

[devmount]

@tonewarper Sorry that it didn't work for you. Can you share your console output (as described here)?

[devmount]

So for @koobs is the fix working, for @tonewarper not. How about you @ryanleesipes? Did 0.7.0 fix this issue for you?
@tonewarper Did you have some time to look into the console?

[6Sim9]

Hi,
Same issue here:
TB 78.4.3 (64b) / W10

It works great with my local folder...

but fail with my main account:

I hope it helps...

[devmount]

Thank you @arndissler for investigating this issue further 👏🏻

[devmount]

It looks like it could be caused by this bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1529791

@6Sim9 & @tonewarper can you somehow determine if you use virtual folders?

[koobs]

@devmount For what its worth, I'm running (successfully, without errors) ThirdStats on a Gmail backed account too and not running any special settings or folder subscribe settings. I mention this as "[Gmail]" is mentioned in the the bug mentioned in the previous comment. See screenshots below:

You can also see a "MFH Pending" folder (a 'Saved Search'), which might also be classed a virtual folder (i'm not sure)

[devmount]

@koobs thank you. Another user just confirmed, that he is successfully using ThirdStats with virtual folders, so that might not be the problem here. I'll take a look what other folder types exist and if there is any type currently not supported by the webextension API.

[arndissler]

Is anyone who's facing this issue here up for testing a "daily build" of Thunderbird? The testing procedure will be as follows:

1. clone (duplicate) your Thunderbird profile where the error occurs
2. download a specific Daily build of Thunderbird
3. extract the ZIP (Windows) or open the DMG (macOS)
4. start the Thunderbird from the ZIP/DMG via use the cloned profile from step 1., so Thunderbird should be started from the command line with the "profile" option set
5. open the Debugger as described in this comment, step 1. to 4.
6. run the scripts which are provided by devmount in this comment
7. there should be still errors in the console, but they should mention the exact folder, e.g. Error listing messages in folder 'your-foldername-here' in account 'account-name'

The Daily build containing the patch are available for Windows 32bit, Windows 64bit and macOS

To start Thunderbird Daily with the cloned profile, run Thunderbird.exe --profile C:\path\to\profile on windows or /Volumes/Thunderbird\ Daily/Thunderbird\ Daily.app/Contents/MacOS/thunderbird --profile /path/to/profile on macOS, ensure to modify the pathnames according to your local system.

[koobs]

@arndissler Probably not the best place for this, but relevent nonetheless given the request. If I (and users) had control over Telemetry (the pref is locked True on beta/nightlies), we'd use nightly (and I use to). I'd be more than happy and in fact, I'd prefer, to run nightly again, particularly to help support extension developers ramp up after the mass webextensions exodus

Edit: Thank you for the bug fix / patch upstream :)

[tonewarper]

I’m not sure if I use virtual folders, but what I do know for sure is that I’ve configured my IMAP folders in TB to sync all Gmails and store on my hard drive too for faster search-speed, so I have a massive profile size (I still use PST/Mbox not Maildir, if that makes a difference). I can also now confirm that the “?” folder path I mentioned earlier was indeed not a TB bug implicated in the null result… it was relabelled that way inside the Gmail interface.

While I’d love to dedicate to troubleshooting and deep-testing, what I’m able to commit to is occasional quick validation/verification over the long haul. I’ll keep checking in on this project! I’m happy that at least you’ve gotten to some degree of pinpointing the problem for some users like me, and hope that in time more users experiencing it will chime in here to get us over to the promised land.

[arndissler]

Four your information: it seems like the fix for this behaviour will be available with Thunderbird 85.

[devmount]

Thank you Arnd 👏🏻

[6Sim9]

Hi!
I don't know if it's help.
I copied my email folder from d: to c: and created a new account with this local directory.

And it works!

Initial account still doesn't work...

[devmount]

I copied my email folder from d: to c: and created a new account with this local directory.

This confirms, that it's a problem with IMAP folders. I guess that converting the IMAP account to a local account creates "real" folders only which the WebExtensions API can handle properly. We will wait for Thunderbird 85 to be released to test this issue again.

[6Sim9]

OK ... I'm sorry ... I realize I didn't respond when you asked me about IMAP or POP.
My server setting is "POP mail server".

[devmount]

No problem. It seems to be the same for POP mail server. There are special folder types on your mail server Thunderbird can't handle properly. Let's hope, that TB85 gets released soon.

[devmount]

Hello everyone,

I am happy to announce that Thunderbird 91 is now available as a beta (direct download). It will be only a month now, that Thunderbird 91 becomes stable and gets released.

So if anyone still affected by this (@ryanleesipes @tonewarper @6Sim9 ?) already wants to test it, please go ahead. Otherwise I hope you can confirm the fix in a month at latest, when Thunderbird 91 gets released, so we can close this issue. Finally! 😅

[ryanleesipes]

It worked! And it works on 78 now, great job @devmount!

[devmount]

Great news! Glad to hear that it now works on 78 too 👏🏻

[devmount]

Now that Thunderbird 91 got released, I can finally close this 👏🏻 If some of you still have issues with the generation of the stats data, please open a new issue.