Convert Sentence length and paragraph length to use HTML parser and enable AI button for both assessments

[FAMarfuaty]

Context

• We want to convert the paragraph length and sentence length assessments to use HTML Parser for the following reasons:
  • To improve the robustness of the assessments' analysis processing
  • To improve the accuracy of the highlighting
  • To make it possible to output the client ID of the blocks that are long or the blocks that contain long sentences
    • This is especially necessary for implementing the Yoast AI Optimize for both assessments

Summary

This PR can be summarized in the following changelog entry:

• [yoastseo] Converts sentence length and paragraph length assessments to use HTML parser in their analysis.
• [yoastseo] Converts sentence length and paragraph length assessments highlighting approach from search and replace to position-based approach.
• [yoastseo] Enables Yoast AI Optimize button for sentence length and paragraph length assessments.
• Introduces more robust HTML processing and highlighting for the sentence length and paragraph length assessments.
• [shopify-seo] Introduces more robust HTML processing and highlighting for the sentence length and paragraph length assessments.

Relevant technical choices:

Test instructions

Test instructions for the acceptance test before the PR gets merged

This PR can be acceptance tested by following these steps:

Optimize AI button in Free

• Install and activate Yoast SEO
• Go through the steps as described in this scenario
  for sentence length and paragraph length assessments
• Confirm that you get the expected result

For the scenarios below, also install and activate Yoast SEO Premium. If building the plugin, pair this PR with this PR in Premium: https://github.com/Yoast/wordpress-seo-premium/pull/4527

Paragraph length assessment

• Follow the testing steps as described in this document to test the paragraph length analysis
• Run the test in WordPress and in Shopify

Sentence length assessment

• Follow the testing steps as described in this document to test the sentence length analysis
• Run the test in WordPress and in Shopify

Relevant test scenarios

• Changes should be tested with the browser console open
• Changes should be tested on different posts/pages/taxonomies/custom post types/custom taxonomies
• Changes should be tested on different editors (Default Block/Gutenberg/Classic/Elementor/other)
• Changes should be tested on different browsers
• Changes should be tested on multisite

Test instructions for QA when the code is in the RC

• QA should use the same steps as above.

QA can test this PR by following these steps:

Impact check

This PR affects the following parts of the plugin, which may require extra testing:

• No further impact other than what is already covered in the testing instructions.

UI changes

• This PR changes the UI in the plugin. I have added the 'UI change' label to this PR.

Other environments

• This PR also affects Shopify. I have added a changelog entry starting with [shopify-seo], added test instructions for Shopify and attached the Shopify label to this PR.

Documentation

• I have written documentation for this change. For example, comments in the Relevant technical choices, comments in the code, documentation on Confluence / shared Google Drive / Yoast developer portal, or other.

Quality assurance

• I have tested this code to the best of my abilities.
• During testing, I had activated all plugins that Yoast SEO provides integrations for.
• I have added unit tests to verify the code works as intended.
• If any part of the code is behind a feature flag, my test instructions also cover cases where the feature flag is switched off.
• I have written this PR in accordance with my team's definition of done.
• I have checked that the base branch is correctly set.

Innovation

• No innovation project is applicable for this PR.
• This PR falls under an innovation project. I have attached the innovation label.
• I have added my hours to the WBSO document.

Fixes https://github.com/Yoast/lingo-other-tasks/issues/114
Fixes https://github.com/Yoast/lingo-other-tasks/issues/471
Fixes https://github.com/Yoast/wordpress-seo-premium/issues/4524

[mhkuu]

CR: Looks great, nice to see all this simplified code 😸

[mhkuu]

ACC: I started testing and found the following problems with sentence/paragraph length highlighting (in English):

Across all editors: The sentence length highlighting is off by one character, as it includes the space before the sentence. It would be good to fix that.

In Classic Editor and Elementor: For sentence length highlighting, if the first word has HTML mark-up, only that word is highlighted.

In Block Editor: I don't understand why we can not highlight longer sentences/paragraphs in Classic blocks, but it's a pre-existing issue, as we also do not support that for keyphrase density.

In Block Editor: Paragraph length highlighting does not work in Yoast blocks. was already fixed, just not on the zip I was working with

In the Yoast blocks, there seems to be a problem with the highlighting in general, as it's off by a few characters, see screenshot below for keyphrase density (keyphrase: communities).

Update:
Almost through with the testing steps. Still to do: both assessments in Shopify.

[FAMarfuaty]

Across all editors: The sentence length highlighting is off by one character, as it includes the space before the sentence. It would be good to fix that.

Fixed this with the latest commit :)

In Classic Editor and Elementor: For sentence length highlighting, if the first word has HTML mark-up, only that word is highlighted.

This problem turns out to be quite complex to solve in this PR. This is also an existing issue, see https://github.com/Yoast/lingo-other-tasks/issues/236.

In Block Editor: I don't understand why we can not highlight longer sentences/paragraphs in Classic blocks, but it's a pre-existing issue, as we also do not support that for keyphrase density.

As I understand it, we used to use the markTinyMCE functionality for applying highlighting in classic block. But I think there might be changes from Gutenberg that now we are using the gutenberg.js file for applying the highlighting. But of course it won't work since we didn't list "core/freeform" as supported for annotation.
The solution is not as simple as adding "core/freeform" to the array of supported block for annotation though
The issue: https://github.com/Yoast/lingo-other-tasks/issues/491

In the Yoast blocks, there seems to be a problem with the highlighting in general, as it's off by a few characters, see screenshot below for keyphrase density (keyphrase: communities).

If this is not the case for paragraph and sentence length assessment. I think we can tackle this separately.

cc @mhkuu

[mhkuu]

100% agree with your observations above 💯

I'll do a quick check in Shopify and then this PR can be merged! 😸

[coveralls]

Pull Request Test Coverage Report for Build aee61f4e5aa1d61c810817d23201a461ce299516

Details

• 61 of 61 (100.0%) changed or added relevant lines in 6 files are covered.
• 3 unchanged lines in 2 files lost coverage.
• Overall coverage increased (+20.0%) to 54.617%

---

Files with Coverage Reduction	New Missed Lines	%
packages/yoastseo/src/values/AssessmentResult.js	1	85.85%
packages/yoastseo/src/scoring/assessments/readability/ParagraphTooLongAssessment.js	2	97.59%

Totals
Change from base Build 870b6c8a8759df89487e8bbc077b82dd90e470a4:	20.0%
Covered Lines:	30084
Relevant Lines:	55453

---

💛 - Coveralls

[mhkuu]

Quick check in Shopify, everything seems to work as supposed (not 100% confident in myself though 😅). So I'm going to merge this one! 🎉