Description : Ear switch as assistive communication and control device and Human Computer Interface

An in-ear switch, incorporated in a hearing aid or worn as a smart earphone. The switch is triggered by voluntary movement of the ear drum, by voluntary contraction of the tensor tympani muscle in the middle ear. This would be designed as a switch for communication aids for people with severe communication disabilities, but also may be used to control user interfaces for smart phones/computers and control of hearing aid functions.

<—Malleus

Earphone/ <—Tensor tympani sensor —>

Figure: Cross section of ear canal showing ear-phone sensor, ear drum, malleus (ossicular bone) and tensor tympani muscle

Figure: Cross section of ear canal showing tensor tympani muscle (4) (Didier Descouens/ available under CC BY- SA 3.0)

Background:

The tensor tympani is a small muscle in the middle ear that is attached to the malleus (a small bone that runs along the centre of the ear drum). Contraction of this muscle is thought to help muffle the hearing during chewing, and possibly to protect the hearing in response to loud noises. It has been noted that some people have voluntary control of this muscle and contraction can be heard by the individual as a rumbling noise. It’s effect can also be triggered during strong eyelid closing and yawning. I have recorded videos, using a USB video auroscope, of the effect of voluntary contraction of the muscle showing that the lower end of the handle of the malleus and adjacent ear drum moves backwards.

75% Prevalence of Tensor Tympani Control

Medical literature has reported series of people who have voluntary control. There is no formal medical literature on the prevalence of control but there is widespread acknowledgment of the ability of a significant number of people who have control, evidenced by the Reddit social media forum for this having 65,800 members (https:// www.reddit.com/r/earrumblersassemble/).

However survey results show 75% of people reporting tensor tympani contraction: SurveyMonkey surveys were completed by a general population of 100 established workers on Mechanical Turk (Crowd employment platform), showing: 75% reporting tensor tympani control and 17% already reported isolated tensor tympani control (ie without moving other muscles).

Movement of the handle of the malleus can be measured and tracked using openly available software imaging program. Movement detection software can detect movement of the ear drum due to voluntary tensor tympani contraction and is able to trigger the Grid3 assistive communication software and also Microsoft Windows On Screen Keyboard, to generate text output using purely voluntary tensor tympani movement

Figure: USB video auroscope Figure: Tracking video analysis movement Teslong Mini Otoscope, of malleus: Vernier Video Physics USB Ear Otoscope Inspection Camera

Figure: Analysis of tracked movement of tympanic membrane (showing regular voluntary contractions) -using Vernier Video Physics: Vernier Software & Technology Figures below: Image analysis showing change in distance with voluntary tensor tympani contraction, between a tracked point on drum and drum margin using Tracker - open source software (Doug Brown: physlets.org/tracker)

Figure below: Analysis of data from “Tracker” software of ear drum movement measurements demonstration On/Off data output

Proof of Concept

I have been able to trigger the on-screen Windows keyboard using intentional eardrum movement picked by a video usb auroscope. The screen shot below is of a video of movement detection software (“iSpyconnect”) which generates an F12 keyboard output in response to intentional movement of the ear drum, which selects the highlighted keys on a scanning keyboard. The central window is a live feed from the auroscope, with the light blue bar beneath showing the detection of movement. This has selected the highlighted options on the scanning keyboard. This generated the shown text output on a Word document.

This has demonstrated control of Microsoft on-screen keyboard for assistive technology users. However, this could easily be extrapolated to control hearing aid functions, and other technologies as outlined above. iSpy open source video surveillance software (iSpyconnect.com)

Control of Grid3 Communication Software;

The technique above has been successfully used to control the “Grid3” software; the leading assistive technology software in the UK: (From Grid3 (thinksmartbox.com) Assistive Technology Software and iSpy open source video surveillance software (iSpyconnect.com)

This has demonstrated control of an on screen keyboard for assistive technology users. However, this could easily be extrapolated to control hearing aid functions, and other technologies as outlined above.

Left Click selection of Gaphical User Interfaces; The Earswitch has also been used as a successful “click to select” graphical user interfaces eg icons and typing on the on-screen keyboard, as an adjunct to headtracking (eViaCam) cursor control. This could easily be used as an adjunct for eyetracking control making a it a widespread useful rapid user interface.

Application:

Communication Aid for People with Communication Disabilities (Primary Aim)

The ear-switch allows for a new assistive technology switch to control speech synthesis and communication & assistive technologies in people with severe communication disabilities due to neurological conditions. These people currently rely on simple switches to control scanning keyboards and other scanning graphical interfaces , which select the highlighted icon/ word or letter. Intel developed the ACAT software that Stephen Hawking used, and in the UK Grid3 technology provides assistive graphical user interfaces.

Figures: the late Professor Stephen Hawking (image by NASA/PAUL E. ALERS under Public Domain) using ACAT technology, which utilises scanning graphical keyboard selection using a switch sensor that detected cheek movements (image by Failedwizard at en.wikipedia • CC BY-SA 3.0)

Some people with these severe disabilities cannot reliably control existing switch technologies; they may not have enough muscle control to control a current switch. Also the switches can be difficult to position for reliable sensing of voluntary movement, or switches can be adversely affected by involuntary movements.

The ear switch gives an alternative communication switch, and also an assistive control switch eg for wheelchairs etc in people with neck injuries (cervical spine injuries) and some brainstem injuries (including strokes). Because the trigeminal nerve, which controls the tensor tympani muscle, arises high in the brain stem, the ear switch should not be affected by neck injuries or lower brainstem strokes, and so the ear switch may still be useable. Figures showing trigeminal nerve (light blue) that supplies tensor tympani muscle arises high in the brainstem (images by Grant, John Charles Boileu, 1962 available under CC BY-SA 3.0, and Patrick J. Lynch, medical illustrator derivative work: Beao available under CC BY 2.5)

<-Trigeminal nerve (Vth Cranial Nerve) supplies tensor tympani->

H o w e v e r, t h i s i s a d e v e l o p m e n t i n i t s infancy and it has not yet been used in these situations.

Secondary commercial applications may include:

1) A new Human Computer Interface; Handsfree/ invisible and silent control of all screen based technology (eg smart phones/ laptops /PCs etc), particularly if used in partnership with eyetracking technology. 2) Controller for smart earphone technology

3) Games controller 4) Hearing aid control Etc

Detecting drum movement Has been shown by visual imaging with current technology, as shown above.

Alternatively, infrared imaging may give similar results to visual imaging but without the need for a light source.

Visual imaging has the advantage that appropriately sized cameras are already easily available, and the above data confirms that the relevant movement is detectable and can trigger suitable user interfaces.

Prototype Development:

The next step may be incorporation of a CMOS camera module with LED, within an easily available earphone shell designed to be worn in-ear by performers; the wire going over the top of the ear helping to anchor the earphone. This would give a stable and comfortable earphone type Earswitch, as it is a well developed configuration for earphones. The existing CMOS camera module used with the auroscope could be used in the shell, however smaller new generation CMOS video sensors, such as Naneye (1mm sq), & the Fujikura/OmniVision Micro-Camera Module System: OV6948 or 6946, and fibre optic cameras (eg the 0.35mm. MilliScope II™ Fiberscope) would fit easily in these earphone shells. This would provide stable and comfortable (& proven) anchoring of the Earswitch within the ear/ irrespective of head movements. Similar wireless miniaturised camera technology with wireless image transmission is already used in capsule enteroscopy (eg cameras within pill type capsules that are swallowed to film the bowel in patients), and may provide lessons/ useable technology for the Earswitch.

Further generations of prototype could move to bluetooth (TM) transmission from wireless Earswitch earphone devices, either directly from the earphones, or from control units attached to the earphones, eg similar to some sports audio and bone conducting headphones.

The smaller 1mm sq CMOS sensors and fibre-optic cameras could be incorporated into existing hearing aids/ smart earphones without affecting their primary function, but with adding handsfree control.

Current Position:

I have proof of concept and a prototype consisting of a wired USB otoscope. I developed the idea originally to improve communication for people with severe communication disabilities, but I have applied for a patent, as I realised that there may be commercial applications. Proof of concept in allowing a person with communication disabilities to communicate would be the primary endpoint but would also provide evidence of the applicability of the concept, and hopefully stimulate more interest.

Also, development of specific earphone “Ear-switch Games Controller”, as a wired USB controller could show proof of demand and help stimulate development of the earswitch for assistive users etc

Nick G Updated 25.12.19