The Voice for Camera Phones with Java ME

The vOICe MIDlet

For information about installing the software, please first visit the web page

http://www.seeingwithsound.com/midlet.htm

In reading this manual, it is assumed that you have already succeeded in installing1 and running23 The vOICe seeing-with-sound MIDlet. The purpose of this manual is mostly to discuss the available key commands in more detail. You may wish to use a headset in order not annoy other nearby people with the rather unusual and hence attention- drawing sounds from your phone. Moreover, a stereo headset is advised for use with phones that offer stereo sound capabilities4. The vOICe’s options menu - called up with the phone’s Options key - contains a “Channels” entry that allows you to select mono (default), stereo or 3D audio channels. The stereo and 3D audio options ease the perception of The vOICe’s visual left-to-right scanning, but they may give severely distorted mono sound on phones that lack stereo capabilities. Just check what works for your phone. Also, some phones contain built-in stereo speakers on their side5, and the “View rotation” entry in the options menu can be used to correspondingly adapt the camera view when holding the phone with left and right speaker on the left and right, respectively.

Note that while using a screen reader it may be necessary to first mute The vOICe (with key "0") to hear the screen reader speak all menu and submenu items under the phone’s Options key. After changing settings, you can then unmute The vOICe again with key "0".

Once started, The vOICe MIDlet continuously grabs and sounds live snapshots from your phone camera. There are no connection costs while using it, because The vOICe MIDlet runs off-line. Each camera snapshot is sounded via a left-to-right scan through the view, while associating height with pitch and brightness with loudness. By default, a black-and-white camera view is sounded in just one second. For example, a bright rising line on a dark background sounds as a rising pitch sweep, a small bright spot sounds as a short beep, and a bright filled rectangle sounds as a noise burst. The vOICe’s

1 The vOICe MIDlet runs on MIDP-2.0 and MMAPI compliant camera phones. Note that on many phones you must have the "Warning tones" setting in your active profile turned On, or else The vOICe may not sound properly, if at all. Also, with many Nokia phones you should after installation (also of an upgrade) select Tools | (App) Manager | The vOICe | Options | (Suite) Settings | Multimedia | Ask first time, or else user permission may be asked for every single camera snapshot. Other phones may hold similar settings. Depending on the phone, the application Manager may be found in the main application Menu or under Menu | Tools. 2 Some phones automatically launch their built-in camera application upon opening the camera cover, and in that case you first need to close that other camera application before The vOICe can access the camera. This is because only one application can access the camera at any given time. Of course you should not forget to slide open any lens cover in the first place, or else the view will remain “black” because no light can enter the camera. 3 On some phones, user permission to control (and turn off) the camera shutter sound will be requested after startup of The vOICe. 4 For example the Nokia 6620, Nokia 6630 and Nokia 9500 camera phones have stereo sound capabilities. 5 For example the Nokia N82. The N82 may be held at head level with the two speakers facing upwards. © Peter B.L. Meijer 2010. All rights reserved. simplest application is as a light probe, but it is actually far more powerful because its changing polyphonic visual sounds or “soundscapes” now also track position and shape of objects, even with multiple objects within your camera view. Thus it allows you to locate light sources, recognize basic image patterns such as stripes and various textures, find borders, identify shapes, and so on. In addition, The vOICe MIDlet offers a number of color detection features, and includes a talking color identifier.

Many settings are accessible through the application’s Option menu if you have a phone screen reader, but there is also built-in speech support for the main features. For compatibility with phone screen readers, The vOICe supports two menu styles: the "Textual" style for the submenus is only advised for use with the old Talks 1.40.1 (to avoid crashes), while the "Normal" style is advised with Mobile Speak as well as Talks 2.0 and later. In addition, a number of keyboard shortcuts exist for direct access to various features, and we begin with a brief overview of the main key commands. At the end of this document a table overview is given.

The "0" key toggles the muted state. Pressing this key twice in rapid succession, like “00”, toggles a muted paused state that minimizes CPU load while releasing the camera resource for best responsiveness when accessing the menus with a phone screen reader. The "1" key toggles the negative video mode, which can help to see/find small or thin dark items on a bright background. The "3" key toggles the built-in speech off and on. The "7" key toggles a mode that helps prevent visual sound stuttering and buzzing on devices that cannot handle simultaneous visual sound rendering and playing. You can try it to find out what works best with your phone. The "9" key cycles over different contrast enhancement modes. The "*" (star, asterisk) key toggles the talking color identifier on and off. The "#" (pound, hash) key cycles over different sound volume levels when not muted. Other settings are controlled with the joystick. The default audio sample rate is 16 kHz, but lower sample rates can be selected by using the "DOWN" key (joystick down), and higher sample rates can be selected by using the "UP" key (joystick up). Available sample rates are 8 kHz, 11 kHz, 16 kHz and 22 kHz, but phones need not support all of these sample rates. Lower sample rates give lower sound quality, but may make the phone more responsive. The "RIGHT" key doubles the visual sound duration to at most two seconds, while the "LEFT" key halves the visual sound duration to at least half a second. Note that on some phones the "UP", "DOWN", "LEFT" and "RIGHT" keys may be mapped through the "2", "8", "4" and "6" numeric keys, respectively. Many of the program settings persist across multiple runs of The vOICe MIDlet.

Now more about the color detection features. As was stated above, the "*" (star) key toggles the talking color identifier on and off. This mobile color recognizer speaks the color of whatever shows at the center of your camera view, while alternating with the visual sound of the camera view that tells you about the shape and brightness of items in your view. If you prefer to only hear the talking color identifier, simply press the "*" (star) key twice in rapid succession, much like a double-click, and you will then only get to hear the color names. So “*” toggles color identification alternating with visual sounds, © Peter B.L. Meijer 2010. All rights reserved. while “**” toggles color identification without the visual sounds. Pressing the joystick “Fire” button will speak the color name once, even if The vOICe was muted, and on suitable phones6 it will use the built-in flash. In any case, recognized colors include (dark, normal, and light) red, green, blue, cyan, yellow, orange and magenta, as well as combination colors such as red- orange. Black, grey and white are also identified, bringing the total number of identified colors and shades to 47. Beware that the choice of color names can be culturally biased: cyan is a color in between green and blue, while magenta is basically the same as the color purple. Also, light-magenta and light-red make for the color pink or very similar colors, while dark-red-orange, dark- orange and dark-orange-yellow appear as various shades of brown. Dark yellow-green makes for olive-green.

Results of color recognition inevitably depend on ambient light and camera quality. Try to use good lighting whenever possible, preferably broad daylight. Still, under relatively low light conditions, better results may be obtained by first calibrating The vOICe for the given visual environment. To do this, point the camera to a known white surface (such as a white sheet of paper) near the object of which you want to identify the color, and apply the “Calibrate white” entry in The vOICe’s options menu7, which will basically tell The vOICe that this surface really is white or light grey rather than its actual grey or dark grey appearance. In fact it will also correct for the yellowish colors from incandescent lighting and many other sources of color bias. Next you can point the camera to other items of interest to identify their colors. Apply the calibration option with care: only apply it when you are certain that the full camera view is indeed white and relatively bright, or else you may get very poor color identification results due to a badly skewed color calibration! Calibration settings do not persist across runs to avoid unintended continued use of a calibration that would no longer match changing ambient light conditions. The vOICe does not normally need calibration in broad daylight conditions, but if applied with care, it can yield significantly more accurate color recognition results under relatively low light conditions. The calibration process takes only about a second and applies for the duration of the run unless you recalibrate or reset The vOICe via its menus.

The color identifier tells you the color at the center of the camera view, but sometimes you may wish to know where items of a given color are. Rather than pointing the camera around until the color identifier finally “hits” the object with the color of interest, you can tell The vOICe to sound the entire camera view but only sound items of the color that you specified. This is done either via the color filter options in the menus or by keying the first letter of the supported color name, being “r” for red, “g” for green, “b” for blue, “c” for cyan, “y” for yellow, “o” for orange and “m” for magenta. Now you need to know how

6 E.g., the Nokia N70, N90 and N91 support the required Advanced Multimedia Supplements (AMMS, JSR-234). 7 You can also apply long-press "*" (long-press star key) as a shortcut. © Peter B.L. Meijer 2010. All rights reserved. to enter these letters, unless your phone includes a QWERTY keyboard8. As you may know, letters are associated with keys 2 through 9 on your phone. In particular, key “2” holds the associated letters “a”, “b” and “c”, or “abc” for short. If you press key “2” once in The vOICe, you specify the digit “2”, but if you press key “2” multiple times in rapid succession, you get to the letters “a”, “b” and “c”. Pressing key “2” twice means “a”, pressing key “2” three times means “b” (which toggles the blue-only color filter), and pressing key “2” four times means “c” (which toggles the cyan-only color filter). The same principle applies to the other numeric keys. Key “3” holds “def”, key “4” holds “ghi”, key “5” holds “jkl”, key “6” holds “mno”, key “7” holds “pqrs”, key “8” holds “tuv”, and key “9” holds “wxyz”. Therefore, if you want to see and find only green items in your view, you press key “4” twice to specify “g” for green, or if you want to see and find only red items in your view, you press key “7” four times to specify “r” for red. These functions act like a toggle, so applying the same one another time turns the color filter off to return to the normal mode of operation. (Alternatively, you may also press key “9” twice to apply “w” for white which is equivalent to having no color filter.)

If you want to run a more complete analysis of what items of what shape and of what color show where in your camera view, you can press key “2” twice to toggle “a” for “Analyze”, which will then cycle over all available color filters for finding any objects and shapes that are red, green, blue, cyan, yellow, orange or magenta.

The combination of color filters with the visual sound bitmaps implies that over 4000 (namely 64×64) different locations for colored items can be represented, while at the same time including shape, shading and texture information - in just one or two seconds of sound. The general image-to-sound mapping makes that top left gives high pitch early in the visual sound, bottom left gives low pitch early in the visual sound, top right gives high pitch late in the visual sound, and bottom right gives low pitch late in the visual sound, with other positions giving intermediate positions in pitch and time.

Let’s consider an example where the general scanning of the visual sounds is combined with color filters to solve a practical problem. Suppose you want to know the color of something small or thin, say a thin electrical wire. Then it is extremely difficult to orient the camera such that the center of the camera view points exactly at this item of interest to get the color identification right. However, by using the visual sounds of the full view along with the "Analyze" submenu option for filtering colors (keyboard shortcut "a"), The vOICe will filter for each color in turn, such that at some point it only sounds any red

8 The Nokia 9500 camera phone includes a QWERTY keyboard. © Peter B.L. Meijer 2010. All rights reserved. items in the view along with saying the color name "red", and any red wire will appear as a single tone going up or down in pitch depending on its visual orientation. Of course such advanced uses may require some practice depending on the exact nature of what you are trying to accomplish.

On suitable phones9, pressing key “p” will save a snapshot picture to the memory card. The resulting JPEG10 format image file contains a numeric timestamp11 in the filename, e.g., as in "vOICe_1155477325843.jpg". The timestamp ensures that each snapshot automatically receives a unique filename. You may have to give several permissions while saving, depending on the phone’s security limitations. The saved image file may subsequently be used for many purposes, such as OCR (optical character recognition), or for sharing with friends. The file location is either the Images folder or the root of the memory card, depending on the type of phone. After saving the image, which may take several seconds, normal operation resumes.

When using the phone camera outside in the sunshine, color readings can be badly affected by glare if there is direct sunlight on the phone. In such situations, try to cup one hand over the phone without blocking the camera view, such that your hand acts like a sunshade - much like a hat can keep your face out of direct sunlight.

Finally, there is support for an additional very special color: skin. Pressing key “1” twice, or “11”, will toggle the skin-only color filter. This will in principle only sound any exposed skin in your view, such as faces and hands, which might find uses in for instance determining how many people are nearby or locating empty chairs in a conference room. The skin color filter also takes into account typical racial differences. However, certain materials such as wood can have a color that is very similar to skin, in which case you need to also take into account apparent shape and size in the visual sounds to try and determine for yourself if results of the skin-only filter only show skin. The best way to start and learn is to experiment. There should be a difference with and without clothes on.

In all uses, please stay aware that pointing the phone’s camera at people who do not know you or The vOICe, in public places or elsewhere, might trigger hostile reactions, for instance because people may think that you are taking their photograph without their permission or otherwise invading their privacy. Similar issues may apply when pointing the camera at certain properties.

Have fun!

9 The file I/O standard JSR-75 must be supported, e.g., as with the Nokia 6630 and Nokia 6680. Otherwise the screen will display show a “File I/O not supported” error message (may not be detectable with screen reader). 10 On phones that do not support JPEG, The vOICe will try to save in PNG format. 11 Timestamped filenames can if desired be converted to human-readable dates and times using an online timestamp converter, because The vOICe timestamps are in fact the number of (milli)seconds since January 1, 1970. © Peter B.L. Meijer 2010. All rights reserved.

Key Action Default 0 Toggles muted state Off 1 Toggles negative video Off 3 Toggles speech feedback On 7 Toggles "anti-stutter/buzzing" mode Off 9 Cycles contrast enhancement 100% * Toggles talking color identifier Off # Cycles sound volume levels 50% UP Higher sample rate, up to 22 kHz 16 kHz DOWN Lower sample rate, down to 8 kHz 16 kHz LEFT 0.5 or 1 second visual sound 1 second RIGHT 1 or 2 second visual sound 1 second FIRE [Flash and] say color Off 00 Mute and pause (low CPU load) Off ** Color identifier, no visual sounds Off ## Toggle blinders (narrow view) Off r Red-only color filter Off g Green-only color filter Off b Blue-only color filter Off c Cyan-only color filter Off y Yellow-only color filter Off o Orange-only color filter Off m Magenta-only color filter Off 11 or s Skin-only color filter Off a Analyze colors by cycling filters Off p Save snapshot picture to memory card Overview of The vOICe key commands © Peter B.L. Meijer 2010. All rights reserved.

Quirks mode

You can also experiment with a “bat call” quirks mode toggled by a long-press of the FIRE button: this gives you two loud but very brief high-pitched chirps in rapid succession, much like an audible version of the clicks or sound flashes emitted by bats during echolocation. The double sound flashes may thus be used with the phone’s built-in speaker to detect nearby obstacles from any echoes that you hear. The sound flash patterns are repeated with the same interval used for the visual sounds, every second by default. If you prefer, you can toggle use of single sound flashes by pressing the “1” key while in the bat call mode. You can also independently cycle the audio volume of the bat calls by pressing the "#" (pound, hash) key while in the bat call mode. If necessary, use your hand to form a cone over the phone’s speaker for improved directionality of the sound flashes, and hold the phone in a position that is consistently aligned with your ears, for instance in front of your face.

Please note that in general palate clicks made with your tongue may work better, because you can more readily adapt these to your current situation. Since these tongue clicks originate from your mouth they are also always consistently aligned with your ears, such that you can better train for very subtle changes as needed for echolocation purposes. On the other hand, your mouth may run dry after a few minutes of intensive tongue clicking, so use whatever suits you best.