DFS: Diabetes Forecast System

DFS: Diabetes Forecast System Usability Tests

Overall Summary

The most definitive statement that can be made about usability tests is that they take more time than expected. Between finishing the prototype and working around schedules this deliverable took almost three times as long as expected. Tight and solid scheduling is important because one delay can have a domino effect that leaves the group with dry periods of waiting followed by late nights spent furiously scampering to get things done.

We felt we got a lot out of our usability tests, with a number of problems coming to light that seem blatantly obvious in hindsight. The most common and noticeable problem was the ambiguity of the icons on our toolbar, which resulted in almost every user either wasting time pondering over their usage and meaning or wasting clicks navigating down digital dead-ends. The second most common issue is almost embarrassing: the way we display our predictions is far too vague to be truly useful. That prediction is the major piece of innovation with our product and fixing this issue is of the highest importance to us. We also discovered bugs in our implementation and were offered some very useful suggestions about how to streamline data entry. We plan on implementing as many of the suggestions as possible because we found we agreed with the users in most cases.

The actual running of the tests was a learning experience. We were able to some able to run some very fun tests with the technology at our disposal, including running the prototype right off a webpage and doing an entire test with someone in another country thought text messaging and video conferencing! All of the user’s were eager to help, but it is very important to start getting as much feedback as soon as possible. We found that as the experiments wore on the users got more and more comfortable expressing their opinions to us and offering suggestions. If given the opportunity to run the tests again, we’d add an emphasis to verbal questions and answers at the beginning of the test and devise some critical thinking or creativity evoking questions just to open communication lines as much as possible before we move onto questions and tasks that we consider critical.

Results of Usability testing

Below are the overall results of our testing. They are answers that we received after the tests were over and we rated them in terms of importance and their complexity (measured by time it took users to overcome the problems).

“Importance” indicates the overall severity of the problem, and is rated on a scale from 1 (unimportant) to 5 (most important). “Time to solve” is an indication of how long it took subjects to deal with the listed problem, and is rated on a scale from 1 (not time consuming) to 5 (very time consuming). Problems Importance Time to solve Suggestions to improve them from users

Confusing icons. 5 5 Color and symbolic meaning are misleading. Maybe use fewer colors and more standard icons. Text pop-ups on roll-over

Buttons that don't work. 5 3 Buttons don't have any functionality and don't give any useful feedback.

Prediction is vague. 5 2 Prediction is not specific and gives no advice to help prevent a predicted severe BGL.

Help button 3 2 Users had trouble finding the help button and rarely used it. We don’t offer helpful instructions on navigation

Entering food (or new food type) data. 4 5 Adding a food to the list of foods or just recording foods eaten.

Send users to history screen after data entry. 2 3 Currently, users go to home screen after data entry.

Eliminate extra steps 2 2 Send users to list editing screens from exercise and diet instead of routing it through settings.

Test Summaries

**Please note that all names have been changed to protect confidentiality.

James Jim is a 54 year old High School principal with Type II diabetes. English is his native language. He’s had diabetes for the past 2.5 years. He says the most difficult part of dealing with diabetes is the continuous monitoring needed from day to day. Before the test he showed interest in using a tool that could help him predict his daily BGL. He averages less than 3 hours of exercise a week.

The test took place at Jim’s house, at his home computer. I was pleasantly surprised to find that he had difficulty with almost every task. As I watched him perform the tasks I noticed that each task (or sub-task for larger ones) could be broken down into two sections. First, navigation. Second, manipulation. Navigation being the process needed to start the task and manipulation being the task itself. The manipulation portion of the test went off without a hitch and he expressed pleasure with much of the interface with regards to data entry and task completion (with a few minor exceptions that were more the result of an unfinished product than faulty interface design). However, the navigation portion of the tasks was atrocious. Our main mode of navigation is a toolbar across the top of the screen. Jim expressed extreme displeasure with this; not with the idea, but rather with our specific implementation. Whereas before the usability test all the icons seemed unique and descriptive, afterwards I’m left wondering how we could not have seen the obvious pitfalls. He felt there is ambiguity between several icons and not a satisfactory description of what they do. He frequently commented that the icons needed descriptive text boxes to appear upon rollover, because icons are more prone to misinterpretation. There is also the case of the “edit list” buttons from the diet and exercise screens; these buttons take you to settings, from which you have to take one more step to edit the lists. They should take you straight to the screens to edit the lists. He also expressed dissatisfaction with the vagueness of the prediction.

Dave is a 21 year old computer science student with Type I diabetes, and he uses an insulin pump. English is his native language. He has had diabetes for 18 years. When first diagnosed he found it most difficult not eating the same things that other kids do. He is not a vegetarian. He wrote that he knows how to deal with his diabetes "very well." Dave exercises only a little bit through regular activities. He records his blood glucose levels (BGL's) a few times per week, and that it is relevant to him to know his BGL. He uses his own glucometer mostly to transfer readings to his insulin pump, but uses the BGL memory feature least of all. Dave considers himself "techno-savvy" and would be interested in using a device that forecasts blood glucose levels over a period of several hours. His native language is English.

We met at the Student Union on the University of Maryland, College Park campus, in a study room which was quiet, to do the test. He finished the tasks quickly and without asking any questions. It took him about five minutes. He is a talented programmer and familiar with usability testing so I think he understood what he needed to do and was able to work it out with no problems. He was satisfied by the icons and the text (size and font) in the interface, and was able to navigate the menus easily. The most useful part of the device in his opinion was the history feature and the graph of the most recent BGL's. The least useful feature was the slide bar approach on the insulin screen. Dave liked the device and gave it a '7' out of a scale from 1 to 10 (10 being "most satisfied" with the interface). The easiest task for him was getting a BGL and the hardest ones were entering food that he had eaten and recording exercise. The predictions he found informative but they don't "suggest what to do" to prevent a high or low reading. He suggested that the system go right to the history screens once data is entered about diet, exercise, or insulin doses.

Steve is a 38 year old man with Type II diabetes, and he has Down Syndrome. English is his native language. He has had diabetes for 5 or 6 years. When he was first diagnosed, he said that he had trouble taking off extra weight, and that it was hard to exercise regularly. Currently, he goes to the gym almost everyday for 30 to 45 minutes. He is not vegetarian. Steve knows his condition fairly well, saying that the food he eats turns into sugar, and that he tries not to eat a lot of sugary snacks. When asked if he is "techno-savvy", he said he's "so-so, not that great" with using computers. When I met with him, before we started, he was using his computer. I noticed during the test that he had no problems using the mouse or keyboard to complete the tasks. He said that he would like to have a device that predicted his BGL's, but only tests his BGL three times a week, which is typical for someone with Type II diabetes. Despite this, he felt it very important to know his BGL, because if it was too high, he would be rushed to the hospital.

I met Steve at his home and we did the test at the table in his dining room. Steve had a lot of questions. He seemed lost at many points during the testing. I verbally assisted him with each task. He did have a good memory about the screens after learning what each one did. I think that he felt he was not doing what he should have been doing when he attempted to complete a task correctly, but faulty design on the part of the interface prevented this. For example, in task 7, we request that users enter a new exercise type, then save it. The 'save' button does not give users any feedback, however, and it doesn't actually save the exercise to the database. This was an oversight on the designer's part. Steve finished the testing in a little less than 15 minutes. He liked the device, but felt it was time consuming. The icons were "straightforward" in their meaning but the text was a little small. There were a lot of different screens in his opinion. He was okay with the feedback presented and thought that he would be able to fix mistakes if he had made any. His confidence was high about his ability to remember how to do the tasks we asked him to perform. If the tasks had taken any longer I think Steve would have become bored and may have asked to stop. It seemed that he was becoming very impatient toward the end of the task list, which made me wonder about the usefulness of such a device to someone who has a cognitive disability.

Julia is a 26 year old doctoral student in genetics, and she doesn't have diabetes so has never used a glucometer. English is her native language. She is familiar with computers, but says that she is not "techno-savvy". On a scale from 1 to 10 (10 being "most satisfied" with the interface), she rated the system a "6, with high potential. I feel when all pulled together and all functions working it will be a very nice interface." She found data entry easy to do and efficient. The graphics and icons seemed appropriate, except that the icon for the BGL screen was gray, which she took to be a feature that wasn't available, because it was "grayed out". The prediction was felt to be too vague, she would like to get a numerical value. She was satisfied with the number of screens that had to be worked through to get to the right one, but didn't appreciate one screen which didn't provide any feedback after she tried saving a new type of food to the food list. Julia couldn't tell whether or not she would be able to easily fix mistakes, because she didn't make any she told me.

During the test I noticed that she hunted around for the right screens to complete the tasks. She asked a few questions of me and I responded after letting her look a little more carefully to discover the answer herself. I think once she learned what the icons meant, or what screens they led her to, she remembered them well. She spoke to herself as she worked and I was able to hear that she was looking for the right answer but was a little stuck. She didn't quite understand the insulin screen, but for someone who doesn't have diabetes, that response seemed appropriate. Working on our tasks took her about 10 minutes.

Father Barry

Father Barry is a 31 year old Franciscan priest who has Type I diabetes. He was diagnosed 23 years ago. He said that he started giving himself injections when he was 8 (many children have their parents do this). His hardest struggle with diabetes is sticking to a diet low in carbohydrates. English is his native language. He does not consider himself "techno-savvy". In regards to his knowledge about his own diabetes, he told me, "I chase a low blood sugar with sugar, and use Humalog [insulin] when readings are high." Considering that our system makes a BGL prediction, he said that he would look at the prediction and then do a test. If the test matched the prediction on several occasions he would probably use it.

He finished the tasks in about 10 minutes and had questions throughout the tasks. When he began, he tried touching the screen, but the device is only interactive through the mouse or touchpad on my laptop. He used the touchpad. He found it somewhat difficult understanding the icons and asked about these. The icons representing the settings screen and the history screen were confusing to him, the others he found to be "somewhat ambiguous". I didn't realize that some of the tasks and interface features did not match up. For example, one task asks the users to press the 'forecast' button, but the button on the screen that should have been pressed was 'save'. I realized that better communication across our team would have prevented confusion like this. I wondered if Fr. Barry was becoming frustrated because of interface - task mismatches. As the person I tested with the greatest experience with glucometers (23 years), I took his opinion seriously when he gave the interface a 4 or 5 on a scale from 1 to 10 (10 being "most satisfied" with the device). He said data entry was complex and time consuming, and the feedback after saving it was "somewhat unclear". The BGL prediction he found a little vague. Asked about whether he would be able to save changes, he said that he wasn't sure because he hadn't made any data entry errors during the testing.

Celia is a 30 year-old single woman who learned of her condition of type I diabetes not too long ago. She is originally from El Salvador, however she has been leaving in the Washington DC area for the past 15 years. Although she speaks English fluently, she appreciates any sensitive information (related to her health) in Spanish. It was a very unexpected experience to learn of her condition and she is willing to learn and use everything she can in order to take care of her body.

Prior to her learning of her condition, Celia enjoyed life doing different things such as dancing (her preferred hobby) as well as helping others doing community service. As she started to feel extremely tired very often, she decided to start some tests. Blood tests gave notice of her condition. At this point she decided to learn as much as she can about proper and balanced diet. She also tried to work closely with doctors in order to keep herself in good shape since she has great goals in her life.

Celia is a school teacher for an elementary school in Montgomery County. She enjoys working with children and would like to keep her condition as secret as possible, however she understand that things are going to change for her for now on. Although she find technology tools (computers, PDAs, ipods, etc) fascinating, she does not consider herself computer savvy. Despite this, she feel very comfortable in using basic tools commonly found on the internet (such as pressing button, data entry, etc). She find different cultures very fascinating and enjoys spending time with people from other latin american countries and learn from them. These has led her to learn about different types of food. She prefers desserts more than anything and has a great taste when preparing them. This, however has been a great barrier to cross since she knows she needs to be more careful now that she needs to eat more balanced from now on.

Celia expressed her impression about the idea of forecasting. She enjoyed very much the testing process, although she would have hope for complete functionality. She also found the “diet” record feature very neat (with arrows to add to your daily intake). She clearly noted that feature as the most userfriendly and easier to use part of the system.

Monica

Monica is an administrative assistant for a Contruction Company. She handles incoming calls as well as some book-keeping for the company. She is very familiar with two-three friends who have type I diabetes. Although she is not personally familiar with diabetes, she is very mindful of the condition of her friends and at times has conversed with them to learn more about it. One thing that called her attention was the fact that people with type I diabetes had to learn to be more disciplined with their diet.

Monica showed that it was a useful tool as long as the user is willing to keep on entering data constantly. She understood pretty quickly that DFS depends on human interaction to provide an accurate reading and found that the interface (in general) was pretty well organized and made a lot of sense.

Jennifer

Jennifer is a college freshman in Taiwan. She was born with type I diabetes. When she was young, she did not have to worry too much about her BGL and insulin since most of the issues were taken care of by her parents. After she went to middle school, she began to learn to use BGL meter herself and calculate how much corresponding insulin she had to take.

Jennifer seems to be very confident about the device even though it is her first time of using it. There was no occasion that she needed to seek out help from the device or from the observer. The flow of her operation over different menus were very smooth. She did not have any problem performing the assigned tasks. It seems that the user was satisfied with the interface of the device.

Her case showed that for some users the interface for the DFS system was designed properly such that it is easy to follow and operate different functions. Even though her first language is not English, the words chosen for the menu were straightforward and easy to understand that she did not mistake any content or feedback. Because the user does not live in America, this usability test was conducted through presenting the interface online and combine instant messaging program along with a video camera and microphone to provide the observer chance to monitor the real reaction of the user.