The deaf and lip reading difficulties

[LONDONER]

Watching the news this morning I saw an interview with a young deaf man who was explaining the difficulties that those who depend on lip reading have now that people are wearing face coverings.  He went on to explain that to add to the difficulties they already face, not being able to see people’s lips make for even more social isolation.

I am no IT technician but it occurred to me that it shouldn’t be beyond the capabilities of a really tech savvy IT technician to design a programme that would translate the spoken word in to a image of a mouth that would articulate what was big said.  This could be spoken in to a smart ”˜phone. and the image would appear on the screen that could be turned towards the deaf person who could then lip read.

Is this a feasible idea? 

The idea is mine but I willingly throw it open to anyone who could develop it.

[James]

There are Programs that turn spoken word into text, don't know if it is available on phones however, take care, stay safe

[LONDONER]

There are Programs that turn spoken word into text, don't know if it is available on phones however, take care, stay safe

Thanks James.  The interviewee said tjhat he had to resort to writing things down and that slowed down things considerably.  Given the advanced stae of animated films these days I think that my idea is not impossible.  It definitely needs a visual of a mouth, not necessarily a whole face.

[InbetweenDreams]

I think @James is referring to is speech to text and yes there are apps for that and yes it is definitely going to be slower than reading lips. @LONDONER It certainly is within the realm of possibilities to create a program that would essentially work the same way speech to text does except generating animations for various words. I'm no linguist but I would think that there are subtle movements between various vowel sounds that would need special care. Of course the big hurdle is getting the correct words, then to break it down by syllable... Perhaps emphasis makes a difference too.

I'm not sure how many different "sounds" exist in the English language (not to mention various accents, I'm sure people from the Southern US sound quite amusing across the pond) but if there is a "standardized" means to identify all the various sounds in a given word. For each sound an animation would be needed, as well as being aware of the next sound.

Ultimately, after thinking on this while writing I think a display, say at restaurants and other public locations, would be more effective. Just something at counters that is listening and converting speech to text all the time. I imagine reading words is easier than reading lips....

[Emiliano]

I’ve seen several versions of masks with a clear insert like these.

[Cardiganwearer]

I used occasionally work with deaf people. Most of them used BSL and a couple lip read. The thing that struck me about the lip readers was their rapt attention to your face, if you were speaking but also to everything around the speaker and in the room. Their concentration was intense. Eye movement studies on deaf people have confirmed this, they look around much more than hearing people, scouring their surroundings for information. I used to think that they must be really knackered at the end of the day having had to concentrate that hard at all times (and having to work with me couldn't have helped). Lip readers use much more than just the movement of the lips, facial expression and changes in posture feed into the sense they extract from situations. Take a look at signers, there is a lot more than hands going on. Deaf communication by hand or by mouth involves much more than just watching one tightly defined aspect of communication.

Having deaf people look at a screen which contains only one narrow channel of information will likely disrupt the strategies they have developed to exploit information from all around them. Also, have you ever looked at auto generated subtitles? They're often rubbish, even from clean sources with little background noise. Machine interpretation of speech is not very good, turning this into an animated mouth may well be a hindrance rather than a help. The mask with a clear insert idea is much better, you get to see the whole face and all the information that can be extracted from that and you're looking at the person speaking in context rather than a device, so you get all their postural cues and there won't be any machine misinterpretation to make life more difficult. 

Reminds me of this...

 

...technology without context.

[LONDONER]

I’ve seen several versions of masks with a clear insert like these.

Yes E miliano, I know they exist, but it limits a deaf person's scope.  He or she would only be able to speak to someone wearing one of those whereas, if he/she had a smart 'phone. with an app. as I have envisaged, he/she could speak to anyone.

---

I used occasionally work with deaf people. Most of them used BSL and a couple lip read. The thing that struck me about the lip readers was their rapt attention to your face, if you were speaking but also to everything around the speaker and in the room. Their concentration was intense. Eye movement studies on deaf people have confirmed this, they look around much more than hearing people, scouring their surroundings for information. I used to think that they must be really knackered at the end of the day having had to concentrate that hard at all times (and having to work with me couldn't have helped). Lip readers use much more than just the movement of the lips, facial expression and changes in posture feed into the sense they extract from situations. Take a look at signers, there is a lot more than hands going on. Deaf communication by hand or by mouth involves much more than just watching one tightly defined aspect of communication.

Having deaf people look at a screen which contains only one narrow channel of information will likely disrupt the strategies they have developed to exploit information from all around them. Also, have you ever looked at auto generated subtitles? They're often rubbish, even from clean sources with little background noise. Machine interpretation of speech is not very good, turning this into an animated mouth may well be a hindrance rather than a help. The mask with a clear insert idea is much better, you get to see the whole face and all the information that can be extracted from that and you're looking at the person speaking in context rather than a device, so you get all their postural cues and there won't be any machine misinterpretation to make life more difficult. 

Reminds me of this...

 

...technology without context.

Valid points Cardiganwearer, but see my reply to Emiliano.