Never got to a result

[tfederici]

Hi, I installed ThirdStats more than 3 years ago because it looked really useful for me
since then, I've tried dozens and dozens of times, even with different machines, it never got to provide any kind of result
I left it working for days, I chose a single account and a single folder, a short time period, whatever, but never succeeded
the tool starts and it looks to idle when approaching around 80% of advancement on the top bar
never got any error message

this is my usual configuration (I tried also others)

System (please complete the following information):

• Windows 10
• Thunderbird Version [102.6.1 (64-Bit)]

thank you for your support
tommaso

[devmount]

Hi,
wow I'm really sorry that you tried over such a long time. Too see what's possibly causing this on your end, we need to take a look into your Thunderbird console. Can you please do the following:

1. Open the console (Main menu > Add-ons > Gear menu > Debug add-ons > Scroll to ThirdStats > Inspect)
2. Clear console (small trash icon on the upper left)
3. Open the stats page (or right click into the stats page > reload if its already open), click the Refresh button if applicable
4. Switch back to the console tab and make a screenshot or copy any errors/warnings and paste it here

This issue is not uncommon (e.g. #19 or #298).

[devmount]

@tfederici Any news on this?

[tfederici]

Hi Andreas, I immediatly replied via mail, sorry for this
however, I ollowed your instructions and what I see is this

then, following the first message, I went to check if javascript were enabled or not, and it was (I was already pretty sure...)
I think instead that the two cases you pointed out are not very much similar to this (but I had only a quick look...)
thank you again for your support!
ciao
tommaso

[tfederici]

The issue is still there like it was in my first message, and Thirdstats has never worked so far
could anyone help me?
thanks a lot

[devmount]

Ok, let's see

An unexpected error occured

This is unfortunately a very meaningless message. But maybe we can narrow down the issue, like we did in this thread. Please paste the following snippet in your console:

let accounts = await messenger.accounts.list(); let allfolders = [];
accounts.map(a => { function traverse(folders) { if (!folders) return; for (let f of folders) { allfolders.push(f); traverse(f.subFolders); }} traverse(a.folders);})
console.log(allfolders);

This should print a list of all folders of all accounts. Does this throw any error messages?

[tfederici]

Hi Andreas,
I've done what you asked and the result is this:

if I open each arrow below Array, I find a list of a subset of my folders, like this:
129: Object { accountId: "account1", name: "NAHIFI", path: "/Università/NAHIFI", … }

if instead I launch the add-on, I see numbers increasing until the percentage of elaboration reaches around 70%, then it idles
let me know
thanks

[devmount]

Ok, now that we know, that the folder list can by correctly retrieved, let's take it a step further and count all messages of all folders of all accounts. Please run the following snippet, if you have the console still open:

let count = 0;
await Promise.all(allfolders.map(async f => { console.log(f); let page = await messenger.messages.list(f); page.messages.map(m => count++); while (page.id) { page = await messenger.messages.continueList(page.id); page.messages.map(m => count++); } })).then(() => { console.log(count); });

If you have a new console, here is the full code snippet:

let accounts = await messenger.accounts.list(); let allfolders = []; let count = 0;
accounts.map(a => { function traverse(folders) { if (!folders) return; for (let f of folders) { allfolders.push(f); traverse(f.subFolders); }} traverse(a.folders);})
await Promise.all(allfolders.map(async f => { console.log(f); let page = await messenger.messages.list(f); page.messages.map(m => count++); while (page.id) { page = await messenger.messages.continueList(page.id); page.messages.map(m => count++); } })).then(() => { console.log(count); });

The output can take some time, according to the number of messages you have. Please let me know, if you now have any error messages. The last folder that is displayed should be the one causing trouble.

[tfederici]

Dear Andreas,
I made what you had asked and after some time this is the message I got

unfortunately, once again, an unexpected error, but located this time
let me know what to do next
many thanks

[devmount]

Can you do it once again with the snippet above? I edited it after posting and added a console statement to find the folder causing trouble... (sorry for that)

[tfederici]

just done
it lists (apparently) all my mail folders, starting with this

and ending with the last folder followed by the usual error

I'm here for other tests

[devmount]

Interesting, it seems to be possible to iterate over all folders, but then there is an issue processing the results 🤔 I modified the snippet slightly, maybe this gives us a better error message.

Same console:

count = 0;
await Promise.all(allfolders.map(async f => { console.log(f); let page = await messenger.messages.list(f); page.messages.map(m => count++); while (page.id) { page = await messenger.messages.continueList(page.id); page.messages.map(m => count++); } })).then(() => { console.log(count); }).catch((error) => { console.error(error.message); });

New console:

let accounts = await messenger.accounts.list(); let allfolders = []; let count = 0;
accounts.map(a => { function traverse(folders) { if (!folders) return; for (let f of folders) { allfolders.push(f); traverse(f.subFolders); }} traverse(a.folders);})
await Promise.all(allfolders.map(async f => { console.log(f); let page = await messenger.messages.list(f); page.messages.map(m => count++); while (page.id) { page = await messenger.messages.continueList(page.id); page.messages.map(m => count++); } })).then(() => { console.log(count); }).catch((error) => { console.error(error.message); });

[tfederici]

we've been not lucky
unfortunately, it ends in the same way of last time

it processes all folders then it gives an unexplained error

[devmount]

That's indeed unfortunate. I'm afraid I cannot do anything to solve this, since it seems to be a problem with Thunderbird itself. I created a corresponding thread in Thunderbirds topicbox, let's hope somebody can help here.

https://thunderbird.topicbox.com/groups/addons/Ta524ece5e13e06ab/unexpected-error-when-iterating-over-all-messages

[tfederici]

Thank you Andreas for your support
let's see now what it'll happen
please, inform me if you have any novelties
many thanks again

[devmount]

@tfederici Do you use any virtual folders (see topicbox thread)?

I extended the snippet once more with a try/catch block for the messages list declaration:

let accounts = await messenger.accounts.list(); let allfolders = []; let count = 0;
accounts.map(a => { function traverse(folders) { if (!folders) return; for (let f of folders) { allfolders.push(f); traverse(f.subFolders); }} traverse(a.folders);})
await Promise.all(allfolders.map(async f => { console.log(f); try { let page = await messenger.messages.list(f); page.messages.map(m => count++); while (page.id) { page = await messenger.messages.continueList(page.id); page.messages.map(m => count++); } } catch(error) { console.error(error); } })).then(() => { console.log(count); }).catch((error) => { console.error(error.message); });

Does this bring any useful error message?

[tfederici]

Andreas, I also saw the answer in the other thread, and I don't think to have virtual box, or at least I'm not aware of that
how can I check it?
however, this is what I obtain with the new test

after pages and pages of the same message of error, I see this:

for the first time there is a number, however!

let me know what to do next, if you have another test
thanks

[devmount]

how can I check it?

Hard to say, since I don't know, which email provider you have. Virtual folders are all folders, that are "generated" (e.g. folders for filter or search results, unified inboxes or similar).

for the first time there is a number, however!

Nice! That means we have found a way to work around the issue (which we cannot solve). I will soon (hopefully in the next couple of days) provide a patch update for ThirdStats. With that, you should be able to see at least stats for 83412 messages (which I assume are not all).

[tfederici]

great! even if I think to have many more messages stored

together with several other "normal" accounts, I have one gmail (that of my university) and two outlook acounts (other universities)
but no (visible) folder is specifically related to those accounts
the only chance is that when I configured such accounts I merged their Inbox and Trash folders into the general one
yet, why the error is given only after having analyzed without errors all the existing folders (includng Inbox)?
and why this time has been given hundred of times?
hope this may suggest you something
thanks again

[devmount]

I can only guess, but I think the folders itself can be retrieved (name, location), but their content cannot be read / processed by Thunderbirds webextension APIs. This time there are hundreds of error messages, because I added a try/catch block for each attempt to read paginated folder content. This is also why there is an actual result now, because those error throwing pages are not blocking the overall message retrieval anymore.

[tfederici]

actually, looking at the list more carefully, you can see several repeated names of folder, with each one related to a specific account (and 3 of them are gmail or outlook), while all other hundred of folders have as accountId always "account1"

[devmount]

I see, but is that an issue? This should match the accounts / folders you see in your Thunderbird. Every account has e.g. an inbox or a trash folder.

[tfederici]

not at all, rather it's a clue that the so-called virtual folders, even though not usually visible, are there
if they are really the source of the problem, I have such kind of folders

btw, I've observed by chance a strange thing:
if I launch ThirdStats it reports 1 Total mail and 1 Received Mail (all other figures are 0)
but if I click repeatedly on the button to confirm the period selected period, the numbers of mail increase (also that of Sent mail) until around the same number we obtained with your last snippet
however, other figures are 0 and charts are empty
I don't know if this may be of interest

[devmount]

I see. Thanks for the additional information, will take that into account. Haven't had time to provide a patch yet.

[devmount]

I added the patch now. If you like, you can try the latest build and test, if stats are (at least partially) shown to you. I will release a new ThirdStats version, as soon as you confirmed that it works.

[tfederici]

wowww! first time I see something there!
thank you Andreas for your effort!

however, only for your knowledge, there are still some issues:

• based on the count I made, there are around 27.000 messages not considered
• as you can see in the top right chart something doesn't seem to work (is it anything I should configure?)
• in the bottom right chart, the reported addresses as most received from/sent to are definitely some of the least frequent
  but I don't know if such issues can be fixed
  whatever the case, thanks again

[devmount]

Thats great 👍🏻 Thank you for the screenshot.

1. Yes, it was to be expected that some messages are missing. The 27k are most likely in those unprocessable folders.
2. Yeah this is strange. Is this shown after filtering (e.g. time range) or right from the beginning? What if you right-click > reload the page?
3. In the add-on options, try to set the number of frequent contacts to something low like 20. Also due to the missing messages the results might be not accurate.

[tfederici]

Hi Andreas, I'm answering below by point:
2. it looks always that way, even when reloading, after having emptied the cache, and also restarting Thunderbird, but what I noticed is that is so only when looking at the last year: if I step back, the charts look regular
3. did so, and now it shows addresses more credible: thks

[devmount]

2. Ok, does that only happen on accounts, where you also have processing errors? I suspect it depends on those existing but unprocessable folders. What happens if you set a date range of the current year, e.g. 2023-01-01 to 2023-03-27 ?
3. You're welcome

[tfederici]

in the last attempts, not only the chart blue (received) but also the pink one below appeared as in the provious image, also after restart/repetition etc.
now I've tried with your request and it looks like you can see

additionally, I notice now that the lower little charts (Time distribution) look weird, with colums of weekdays with the same date (Monday 10, Tuesday 10, Wednesday 10 etc.), starting from one column of weekdays with 0 date (see tooltip)

as for the missing messages, I've tried to check where they actually are (here a different format of the output would have been much helpful, see below) and I saw that missing messages are dispersed among several folders, which however contain other messages reported in the stats

as for the output, I've a kind observation: the json file cannot be easily used in many ways, because it contain tons of data altogether under a same tag
let me suggest that a csv file reporting well separated the number of messages received and sent in each folder, or by each address, would be of a paramount utility, for example in documenting the effort on a specific project/activity (actually, this was my original interest for ThirdStats...)

I'm available for further tests, if it would be useful
thanks a lot!

[devmount]

Thank you for this additional report. Since this is another issue, I added #405 for further discussion.