Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2916: Add tts on android #2950

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open

2916: Add tts on android #2950

wants to merge 32 commits into from

Conversation

bahaaTuffaha
Copy link
Contributor

@bahaaTuffaha bahaaTuffaha commented Oct 5, 2024

Short description

Adding text-to-speech functionality.

Proposed changes

  • TtsPlayer.tsx : deals with html content and it passes it to react-native-tts as sentences. used debounce function from lodash to handle rapid volume change. I added an expandPlayer state just to keep player expanded when changes from sentence to sentence. This component located at app level where it wraps around the NavigationContainer to be able to be floating at the bottom of the screen while scrolling.

  • useTtsPlayer.ts: Is a hook used to pass data (content, title, visibility, volume) from across the app to TtsPlayer.tsx using useContext.

  • Categories.tsx, News.tsx, Events.tsx each one is using the useTtsPlayer.ts hook to pass content and title.

  • Created a component Slider.tsx instead of adding more libraires and used the ones we have: react-native-reanimated & react-native-gesture-handler to create slider that can be used anywhere other than the volume.

  • Added ReadAloud HeaderButtonTitle to be selected from the kebab menu to show the player. This button will show up only in Integreat/IntegreatTestCms and at third level pages.

  • The TtsPlayer will stop when the app is in the background also if you back out from a 3rd level page.

  • Notes:

    • Not Tested on iOS !
    • Why I passed sentences to tts.speak()? due to react-native-tts doesn't have a functional pause also passing word by word will sound more robotic.
    • Persian is not supported at react-native-tts.
    • Pausing on words is not possible cause there is no way to know when its finished reading word and pausing is not supported for android.

Side effects

  • Categories.tsx, News.tsx, Events.tsx, Header.tsx

Testing

  • At integreat : Go to any 3rd level content in categories or news.
  • Open the kebab menu on the right and select Read aloud.
  • The player will show up press play to make it start reading content.

Resolved issues

Fixes: #2916


Copy link
Contributor

@LeandraH LeandraH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works pretty well so far! A couple of points about the functionality:

  • If I change the volume while it's playing, it starts re-reading the current fragment. If possible, I think it should keep reading at the same spot.
  • In using, I thought the volume bar was for updating the speed of reading. Maybe that's just me but maybe not, we should see if other people have that problem.
  • For Persian (and any other languages where it doesn't work), I think we should hide the button to read the text aloud, and close the overlay if it's open when someone switches languages.
  • In emulated iOS, no sound comes out and I get this warning: Excessive number of pending callbacks: 501. Some pending callbacks that might have leaked by never being called from native code: {"1000":{},"1001":{},"1002":{},"1003":{},"1004":{},"1005":{},"1006":{},"1007":{},"1008":{},"1009":{},"1010":{},"1011":{},"1012":{},"1013":{},"1014":{},"1015":{},"1016":{},"1017":{},"1018":{},"1019":{},"1020":{},"1021":{},"1022":{},"1023":{},"1024":{},"1025":{},"1026":{},"1027":{},"1028":{},"1029":{},"1030":{},"1031":{},"1032":{},"1033":{},"1034":{},"1035":{},"1036":{},"1037":{},"1038":{},"1039":{},"1040":{},"1041":{},"1042":{},"1043":{},"1044":{},"1045":{},"1046":{},"1047":{},"1048":{},"1049":{},"...(truncated keys)...":451} But I guess you did say that it's not yet tested for iOS :D

As discussed, I only half-checked the PR, let me know when it's ready for review again :)

assets/icons/no-sound.svg Outdated Show resolved Hide resolved
native/src/utils/TtsPlayerUtils.ts Outdated Show resolved Hide resolved
native/src/utils/TtsPlayerUtils.ts Outdated Show resolved Hide resolved
release-notes/unreleased/2916-Add-TTS-on-android.yml Outdated Show resolved Hide resolved
native/src/components/TtsPlayer.tsx Show resolved Hide resolved
native/src/components/TtsPlayer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsPlayer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsPlayer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsPlayer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsPlayer.tsx Outdated Show resolved Hide resolved
assets/icons/no-sound.svg Outdated Show resolved Hide resolved
Copy link
Contributor

@f1sh1918 f1sh1918 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i couldn't test it since its crashing for me on emulator when i click play on Android 13

native/src/components/Header.tsx Outdated Show resolved Hide resolved
native/src/components/Header.tsx Outdated Show resolved Hide resolved
native/src/components/News.tsx Outdated Show resolved Hide resolved
native/src/components/RemoteContent.tsx Outdated Show resolved Hide resolved
Copy link
Contributor

@f1sh1918 f1sh1918 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me on android. good work.
Since its a draft some improvements are missing i guess

image
Shouldn't here stand the title of what will be read?
Even add shadows/elevation for the container and the play button
image
The icons should switch i guess since the direction arrows are wrong

image

I think it would be better to create a fixed area for this player instead of overlapping content, but this may be also done in an additional tasks, since it will need some effort.

native/src/components/Slider.tsx Outdated Show resolved Hide resolved
Copy link

codeclimate bot commented Oct 17, 2024

Code Climate has analyzed commit 042e2ac and detected 3 issues on this pull request.

Here's the issue category breakdown:

Category Count
Complexity 3

The test coverage on the diff in this pull request is 75.9% (50% is the threshold).

This pull request will bring the total coverage in the repository to 74.2%.

View more on Code Climate.

@LeandraH
Copy link
Contributor

It unfortunately also doesn't play any sound on real iOS, there is just this warning that pops up:

IMG_023107052847-1

@bahaaTuffaha
Copy link
Contributor Author

It unfortunately also doesn't play any sound on real iOS, there is just this warning that pops up:

Maybe I should define iosVoiceId: '', for ios
https://github.com/ak1394/react-native-tts?tab=readme-ov-file#speaking

@LeandraH
Copy link
Contributor

It unfortunately also doesn't play any sound on real iOS, there is just this warning that pops up:

Maybe I should define iosVoiceId: '', for ios https://github.com/ak1394/react-native-tts?tab=readme-ov-file#speaking

Hmm, putting in the example 'com.apple.ttsbundle.Moira-compact' didn't change the behavior.

There is also a very minor bug that I found: when opening the TTS player, the line with the last update jumps down, and back up when you close it. I attached a video of the behavior.

Simulator.Screen.Recording.-.iPhone.16.-.2024-10-28.at.15.34.08.mp4

@bahaaTuffaha
Copy link
Contributor Author

It unfortunately also doesn't play any sound on real iOS, there is just this warning that pops up:

Maybe I should define iosVoiceId: '', for ios https://github.com/ak1394/react-native-tts?tab=readme-ov-file#speaking

Hmm, putting in the example 'com.apple.ttsbundle.Moira-compact' didn't change the behavior.

There is also a very minor bug that I found: when opening the TTS player, the line with the last update jumps down, and back up when you close it. I attached a video of the behavior.

This behavior is intended to leave space for the player to prevent it from covering the content.
I think iosVoiceId should be taken from voices()
Tts.voices().then(voices => console.log(voices));
The voices should be filtered by the language and passed to speak.

@LeandraH
Copy link
Contributor

This behavior is intended to leave space for the player to prevent it from covering the content.

That's a good idea but leads to jumping in this case. The padding should probably be added under the last update stamp instead of before.

@LeandraH
Copy link
Contributor

I think iosVoiceId should be taken from voices()
Tts.voices().then(voices => console.log(voices));
The voices should be filtered by the language and passed to speak.

I unfortunately still don't get any sound out. But the iOS can be fixed in a separate issue, I think you can for now work on getting the PR ready for reviewing for Android :)

@bahaaTuffaha bahaaTuffaha marked this pull request as ready for review November 13, 2024 10:50
Copy link
Member

@steffenkleinle steffenkleinle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, looks quite good already and I like the implementation. I just think that the TtsPlayer component should be split up to separate concerns and increase readability.

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" fill="none" viewBox="0 0 24 24"><path fill="currentColor" d="M9.878 19.125c-.188 0-.365-.04-.53-.12-.167-.081-.342-.21-.524-.387l-3.428-3.251a.29.29 0 0 0-.201-.072h-2.31c-.541 0-.957-.148-1.247-.443-.29-.295-.434-.735-.434-1.32V10.54c0-.58.144-1.017.434-1.312.29-.3.706-.45 1.247-.45h2.31a.29.29 0 0 0 .201-.073l3.428-3.218c.204-.199.38-.341.531-.427.15-.086.32-.129.507-.129.274 0 .496.094.668.282a.938.938 0 0 1 .257.668v12.344a.874.874 0 0 1-.257.644.867.867 0 0 1-.652.257zm3.774-3.677a.605.605 0 0 1-.273-.435.729.729 0 0 1 .152-.523c.231-.327.41-.703.54-1.127a4.65 4.65 0 0 0 .193-1.343c0-.467-.065-.915-.193-1.344a3.538 3.538 0 0 0-.54-1.127.688.688 0 0 1-.16-.515.637.637 0 0 1 .281-.442c.14-.097.293-.13.459-.097a.636.636 0 0 1 .41.282c.311.407.553.896.724 1.464.177.569.266 1.162.266 1.779a5.95 5.95 0 0 1-.266 1.778 4.563 4.563 0 0 1-.724 1.464.636.636 0 0 1-.41.282.627.627 0 0 1-.459-.096zm2.994 2.1a.565.565 0 0 1-.25-.403.726.726 0 0 1 .137-.506 7.6 7.6 0 0 0 1.014-2.133c.247-.8.37-1.628.37-2.486a8.55 8.55 0 0 0-.362-2.487 7.338 7.338 0 0 0-1.022-2.132.695.695 0 0 1-.145-.5.597.597 0 0 1 .258-.41.61.61 0 0 1 .474-.104.636.636 0 0 1 .41.281c.521.698.923 1.516 1.208 2.455a9.94 9.94 0 0 1 .426 2.897c0 .992-.145 1.955-.434 2.888-.285.934-.684 1.755-1.2 2.463a.636.636 0 0 1-.41.281.658.658 0 0 1-.474-.104zm3.017 2.132a.568.568 0 0 1-.257-.418.699.699 0 0 1 .136-.5 12.144 12.144 0 0 0 1.778-4.257c.167-.798.25-1.627.25-2.485 0-.864-.083-1.693-.25-2.487-.165-.8-.402-1.553-.707-2.261-.3-.714-.657-1.379-1.07-1.996a.726.726 0 0 1-.137-.507.563.563 0 0 1 .257-.41.556.556 0 0 1 .467-.097.626.626 0 0 1 .418.298c.462.681.861 1.42 1.2 2.213.337.789.597 1.623.78 2.503.182.88.273 1.794.273 2.744 0 .95-.09 1.864-.273 2.744-.177.88-.435 1.716-.773 2.51a12.15 12.15 0 0 1-1.207 2.205.663.663 0 0 1-.418.298.556.556 0 0 1-.467-.097z" style="stroke-width:1.09867"/></svg>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Are these all icons from a design on figma?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you noticed this icon in Figma has size 28x28 so I had to change it to 24x24 and clean it up...
some Icons like the play button is not separated (black circle, play icon) so I had to improvise 😅.
I use Inkscape and svgomg for cleaning.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be cool to request cleaned up ones from UI/UX instead as they have some weird or most likely unnecessary syntax/clipping rules.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

build-configs/integreat/index.ts Outdated Show resolved Hide resolved
translations/translations.json Outdated Show resolved Hide resolved
release-notes/unreleased/2916-Add-TTS-on-android.yml Outdated Show resolved Hide resolved
native/src/components/RemoteContent.tsx Outdated Show resolved Hide resolved
native/src/routes/Events.tsx Outdated Show resolved Hide resolved
native/src/hooks/useTtsPlayer.ts Outdated Show resolved Hide resolved
native/src/components/News.tsx Outdated Show resolved Hide resolved
native/src/components/Categories.tsx Outdated Show resolved Hide resolved
native/src/components/RemoteContent.tsx Outdated Show resolved Hide resolved
Comment on lines 19 to 21
themeContrast: string
grayBackgroundColor: string
slightlyDarkGray: string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 Do we really need three new colors? I'd prefer to reuse existing ones (at least where applicable).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes ... and these shades could make a difference in the look.
image

native/src/components/Page.tsx Outdated Show resolved Hide resolved
native/src/utils/TtsPlayerUtils.ts Outdated Show resolved Hide resolved
native/src/hooks/useTtsPlayer.ts Outdated Show resolved Hide resolved
native/src/hooks/useTtsPlayer.ts Show resolved Hide resolved
native/src/components/TtsContainer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsContainer.tsx Show resolved Hide resolved
native/src/components/TtsPlayer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsPlayer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsPlayer.tsx Outdated Show resolved Hide resolved
Copy link

@codescene-delta-analysis codescene-delta-analysis bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: OK

Change in average Code Health of affected files: +0.08 (9.70 -> 9.78)

View detailed results in CodeScene

Copy link
Member

@steffenkleinle steffenkleinle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just tested it with english and this works really good, nice job and thank you for taking on this big thing!

Nevertheless, I think your code still needs some polishing and cleaning up. Especially the concerns should be clear and listeners and useEffects should only be used if they are really necessary (in my opinion this should only be the case to initialize everything and for the tts-finish listener to continue reading. If you have any questions about my comments or need some help, just let me know!

native/src/hooks/useTtsPlayer.ts Outdated Show resolved Hide resolved
native/src/components/Categories.tsx Outdated Show resolved Hide resolved
native/src/components/Header.tsx Outdated Show resolved Hide resolved
native/src/components/Header.tsx Outdated Show resolved Hide resolved
native/src/components/TtsContainer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsContainer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsContainer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsContainer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsContainer.tsx Outdated Show resolved Hide resolved
native/src/components/TtsContainer.tsx Outdated Show resolved Hide resolved
Copy link
Member

@steffenkleinle steffenkleinle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good already, thanks for taking care of all those comments! A little more to go but we are getting there :)

@@ -83,6 +85,7 @@ const Header = ({
// Save route/canGoBack to state to prevent it from changing during navigating which would lead to flickering of the title and back button
const [previousRoute] = useState(navigation.getState().routes[navigation.getState().routes.length - 2])
const [canGoBack] = useState(navigation.canGoBack())
const { enabled: isTtsActive, setVisible: setTtsPlayerVisible } = useTtsPlayer()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙃 Please keep the naming consistent

Suggested change
const { enabled: isTtsActive, setVisible: setTtsPlayerVisible } = useTtsPlayer()
const { enabled: isTtsEnabled, setVisible: setTtsPlayerVisible } = useTtsPlayer()

setSentences: (sentences: string[]) => void
}

export const ttsContext = createContext<TtsContextType>({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 So far we are using upper case names for contexts, please change accoringly

Suggested change
export const ttsContext = createContext<TtsContextType>({
export const TtsContext = createContext<TtsContextType>({

native/src/components/TtsContainer.tsx Show resolved Hide resolved
Comment on lines +125 to +129
useEffect(() => {
if (sentences.length <= 1) {
stop()
}
}, [sentences])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 This useEffect should be removed by only passing a modified setSentences down that both sets the sentences and stops playing.

Comment on lines +169 to +173
handleBackward={playPrevious}
handleForward={playNext}
handleClose={close}
pauseReading={pause}
startPlaying={play}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧

Suggested change
handleBackward={playPrevious}
handleForward={playNext}
handleClose={close}
pauseReading={pause}
startPlaying={play}
playPrevious={playPrevious}
playNext={playNext}
close={close}
pause={pause}
play={play}

Comment on lines +167 to +168
setIsPlaying={setIsPlaying}
sentenceIndex={sentenceIndex}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 Both setIsPlaying and sentenceIndex should not be necessary anymore, could you adjust TtsPlayer accordingly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing sentenceIndex is needed for playPrevious(sentenceIndex) and playNext(sentenceIndex)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no need to pass sentence index to these functions, they can just directly read and use it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I pass it directly it won't keep playing.. it will stop at an index for some reason.

)

const playPrevious = (index = sentenceIndex) => {
const prevIndex = Math.max(0, index - 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ The max should not be needed anymore, right?

Suggested change
const prevIndex = Math.max(0, index - 1)
const previousIndex = index - 1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will go minus index when you keep pressing playPrevious

Copy link
Member

@steffenkleinle steffenkleinle Dec 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but you handle that in the if in the next line, right?
Edit: Oh, the if in the next line should be:

    if (previousIndex >= 0) {
      setSentenceIndex(prevIndex)
      play(prevIndex)
    }

native/src/components/TtsContainer.tsx Outdated Show resolved Hide resolved
const [isPlaying, setIsPlaying] = useState(false)
const [sentenceIndex, setSentenceIndex] = useState(0)
const [visible, setVisible] = useState(false)
const [enabled, setEnabled] = useState(false)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ There is no state needed for this as it does need to be manually set (enabled is just dependent on other states/constant values). See comments #2950 (comment) and #2950 (comment)
This should be done here and only enabled should be passed down in the context and exposed by useTtsPlayer.

Comment on lines +31 to +41
useEffect(() => {
if (sentences) {
tts.setSentences(sentences)
}
tts.setEnabled(enabled)
return () => {
tts.setSentences([])
tts.setEnabled(false)
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [enabled, sentences, languageCode])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 There is no need to manually set enabled, it should only be defined by other values.

Suggested change
useEffect(() => {
if (sentences) {
tts.setSentences(sentences)
}
tts.setEnabled(enabled)
return () => {
tts.setSentences([])
tts.setEnabled(false)
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [enabled, sentences, languageCode])
useEffect(() => {
setSentences(sentences)
return () => setSentences([])
}, [setSentences, sentences])

@steffenkleinle
Copy link
Member

steffenkleinle commented Dec 5, 2024

One more thing: Personally I'd be in favor of always showing the header item read aloud if the feature is available (that is, the current language is supported and it is enabled in the build config). So in contrast to now, we would also show the header item even if there is nothing to read. The player would then just say nothing to read. That would keep the UI mostly consistent (except if switching to an unsupported language).
@hauf-toni do you have a different opinion on this? At the moment the header item read aloud is only shown if the current page also has something to read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add TTS on android
5 participants