Bulletin number 32 Delivering and Assessing Learning on Mobile Devices: Report on a pilot project by SQA, May and June 2007

June 2008

Published by the Scottish Qualifications Authority The Optima Building, 58 Robertson Street, Glasgow G2 8DQ Ironmills Road, Dalkeith, Midlothian EH22 1LE

www.sqa.org.uk

The information in this publication may be reproduced in support of SQA qualifications. If it is reproduced, SQA should be clearly acknowledged as the source. If it is to be used for any other purpose, then written permission must be obtained from the Publishing Team at SQA. It must not be reproduced for trade or commercial purposes.

© Scottish Qualifications Authority 2008 SQA is committed to the use of robust evidence in the development and evaluation of policy and its implementation, and carries out or commissions research across a range of topics to support this.

The publication of Research Bulletins allows us to disseminate the results of our research activity to practitioners, policy makers, parents, academics and anyone else who has an interest in the key role that qualifications play in economic growth and social inclusion in Scotland.

Disclaimer This report is the result of a small-scale research project. The information it contains reflects the opinions and experiences of the researcher and should not be viewed as a comparison of products by the Scottish Qualifications Authority. Products mentioned are examples of devices that could be used for e-learning and e-assessment — there are many others. The inclusion of a product should not be viewed as a recommendation by the Scottish Qualifications Authority, its staff or appointees. Some of these products, including their costs, may also have changed since the report was compiled or published.

Contents

Executive summary 1 1 Introduction 3 2 Hardware platforms 4 3 Formatting teaching and learning materials 6 4 Using the materials 10 5 Using mobile devices for assessment 12 6 Candidate and teacher experience 14 7 Vendor technologies 15 8 Conclusions and recommendations 17

Executive summary This report presents the findings of a small-scale pilot project undertaken by the Scottish Qualifications Authority between May and June 2007. The remit for the project was to take a preliminary look the prospects for using mobile devices, such as mobile phones and , for the delivery and assessment of learning.

Three devices were selected as broadly representing the types of device in current use for mobile access. There are other products which provide the same functions and similar features. The three devices we selected were:

♦ The T-Mobile Vario II MDA: a Windows Mobile-based . ♦ The N800 Internet Tablet: for standalone WiFi internet access. ♦ The Nokia 6151 : a typical budget mobile telephone.

The teaching and learning materials selected were for the SQA Internet Safety Unit1. Although these materials render well on full-sized screens, there are several factors that cause problems when trying to display them on mobile devices. These include the requirement for a lot of scrolling, redundant navigational features, the amount of white space, the size of graphics, and the use of add-on technologies, such as Java and Flash.

Changes were made to the materials themselves, including removing or reducing graphics, using bolding, avoiding add-ons and keeping pages short.

The materials were developed using Course Genie2. Changes were made to the Course Genie settings to simplify navigation, reduce the amount of white space displayed, and to remove the logo from the top of each page. The cumulative effect of these changes was to produce clear and readable pages which can be navigated with a minimal amount of downward scrolling.

All the devices tested displayed the materials in a usable format and permitted effective navigation, although the devices with larger screens provided the best user experience. Download speeds were excellent using the latest 3G technology and the slightly older GPRS technology. Excellent results were also obtained using WiFi access.

Although there appears to be great potential for using mobile devices for assessment, the mobile assessment aspect of this project was not particularly successful. The online assessments for the SQA Internet Safety qualification are hosted on the SOLAR website, but most of the devices used could not get past the first page, probably because subsequent pages appear to be Flash-based. Only the Internet Tablet displayed the SOLAR pages correctly.

1 An Intermediate 1 National Unit designed to enable candidates to make safe and legal use of the Internet — http://www.sqa.org.uk/files/nu/F0H510.pdf

2 Course Genie has since been renamed Wimba Create.

1 Course Genie questions, used in the online materials, worked well on devices running the latest Windows Mobile 6, but not on earlier versions. They also worked on mobile phones running the Opera Mini browser.

It may be useful to investigate alternative methods for online assessment, such as assessment via (SMS). Mobile devices may also be suitable for collecting audio and video evidence for a portfolio-based assessment and uploading it to a website, where an assessor or verifier could examine it online.

The project included a brief trial of the Mobile version of the SQA Internet Safety materials with a small group of secondary pupils, using relatively modern mobile telephones with GPRS access to the internet via proprietary browsers. The pupils enjoyed the experience and the trial highlighted a number of operational difficulties which will inform future developments.

During the project, we encountered two potentially useful vendor-specific technologies: M-Learning and Zoho Challenge. These could usefully be investigated further in future projects.

The preliminary study has highlighted three major points:

♦ It is possible, and even relatively easy, to adapt existing learning materials so that they can be displayed on a variety of mobile devices. ♦ Assessment on mobile devices is not nearly as straightforward, largely due to the fact that the mechanisms used make use of technologies, such as Flash, which are poorly implemented on mobile devices at present. ♦ In general, things worked better on the newer and more sophisticated devices than on the older and simpler ones.

Future developments would probably be best restricted to specific platforms, particularly devices running the Windows Mobile 6 and mobile phones which support the Opera Mini browser.

The outlook for mobile delivery is positive and there is already a large amount of material available which could be converted. The picture regarding mobile assessment is more complex and requires further investigation, but it may improve as technological developments take place.

2 1 Introduction The initial remit for the project was:

♦ Re-format teaching and learning materials for rendering on mobile devices. ♦ Explore suitable formats for rendering materials on mobile devices. ♦ Explore potential of mobile devices for assessment (using the existing SQA Internet Safety item bank). ♦ Research candidates’ and teachers’ views on mobile learning and mobile assessment. ♦ Research candidates’ and teachers’ experience of using the SQA Internet Safety materials and assessments on mobile devices. ♦ Explore vendor technologies to support mobile learning/mobile assessment.

The project was very brief and was very much a preliminary look at the prospects and potential pitfalls in this area. Decisions had to be made early in the project about the teaching materials and assessments to be used and the hardware platforms to be covered.

The teaching materials selected were those developed recently for the Internet Safety Unit, which had been piloted in the latter half of 2006. These were selected because both teaching and assessment materials were already available in online format.

3 2 Hardware platforms There is a vast range of different types of hardware available, so we decided to restrict coverage to three types of devices. The devices were selected because they broadly represent the types of device in current use for mobile internet access.

♦ The T-Mobile Vario II MDA is an HTC Hermes in T-Mobile livery. Similar phones are available, with slightly different styling, from several other manufacturers.3 It represents the current state-of-the-art in Windows Mobile- based smartphones. At the outset of this study the current release was Windows Mobile 5, but during the study Windows Mobile 6 was released, bringing significant improvements. The operating system and associated is very similar to that used on many standalone PDAs (ie, those which are not also telephones), so information obtained about this platform should also apply to PDAs. ♦ The Nokia N800 Internet Tablet is a standalone device for internet access. It runs the operating system, a variant of , and uses the Opera browser. Unlike the other devices tested, it is not a mobile phone, but accesses the internet via or WiFi. It is mainly of interest due to its large screen size (800 x 400 pixels). ♦ The Nokia 6151 Mobile Phone is a typical budget mobile telephone which supports 3G internet access. It is not Windows-based, but has its own proprietary browser. It also runs the Opera Mini browser from Opera Software, which works on a wide range of current phones. The 6151 was recently recommended as a ‘Best Buy’ by Which? magazine.

3 When this report was written the HTC Hermes variants were among only a few Smartphones offering 3G access. At the time of publication there are many others.

4 More about the selected hardware platforms T-Mobile Vario II MDA The Vario II is a compact smartphone with a slide-out QWERTY keyboard. It is 3G and HSDPA enabled, allowing internet access at broadband speeds up to 1.8Mb. It incorporates a 400MHz Samsung processor, 64MB dynamic memory, Microsoft Word and Excel editing, plus a PowerPoint and PDF viewer. Connectivity includes Bluetooth, infrared and USB2. It has a 2.1 megapixel camera with flash and video capture and a secondary video-conferencing camera.

Nokia N800 Internet Tablet The Nokia N800 features a high- resolution widescreen display and supports the Opera 8 browser and Flash 9. It allows internet calling with integrated web camera and and has an e-mail client and a full-screen finger keyboard. It has 128MB RAM and 256MB Flash Storage and supports Bluetooth and WiFi 802.11b/g.

Nokia 6151 3G Mobile Phone The Nokia 6151 is a budget 3G phone which allows for high data-transfer speed. The phone measures 108 x 47 x 19mm and weighs 98g.

The Nokia 6151 is equipped with a 1.3 megapixel camera, 30MB internal memory, a microSD memory card slot, and a stereo FM radio.

Connectivity features include tri-band GSM networks support plus 3G, EDGE, and Bluetooth support.

5 3 Formatting teaching and learning materials The teaching and learning materials selected for the project were those developed for the Internet Safety Unit, available online at:

www.e-learning-computing.com/netsafe.

This screenshot shows a typical page, introducing the topic of Trojans:

Although these materials render well on full-sized screens, there are several factors that cause problems when trying to display them on mobile devices. Many of these cause screens to be cluttered, and can reduce usability.

♦ Considerable scrolling, both horizontal and vertical, is required. This makes it difficult to get an idea of the overall content of the screen. ♦ Redundant navigational features (eg back/forward arrows as well as explicit hyperlinks and menu choices) take up a lot of room. These can be useful when plenty of screen area is available, but not when space is at a premium. ♦ There is a lot of white space. Although white space can improve readability, it can be wasteful when there is limited screen area available. ♦ Graphics are too large, meaning that they take a long time to download and take up a lot of space on the screen. ♦ Many web pages rely on add-on technologies, such as Flash (for animated graphics) or Java (for user interaction). These are not always fully implemented on mobile devices.

The screenshots given below illustrate these problems. They show different parts of the same page, initially displayed unmodified on a Vario II:

6

The following screenshots show how the same parts of the page appear on a mobile phone running Opera Mini. (Note that Opera Mini ‘folds’ the menus so that they appear at the top of the screen rather than the side.)

Note: All screenshots are about 50% larger than in reality. The relative sizes of the screens are just about right.

Fortunately, changes can be made in several areas to make pages render better on mobile devices.

First of all, changes can be made to the materials themselves, eg:

♦ Remove graphics and logos, or reduce size ♦ Use bolding for emphasis ♦ Avoid use of add-ons ♦ Keep pages short

7 The Internet Safety materials, like most of the other e-learning materials commissioned recently by SQA, were produced using Course Genie (http://www.wimba.com/products/coursegenie/), a development tool that allows e-learning materials to be generated from a Microsoft Word document.

Changes can be made to the Course Genie settings, eg Navigation Buttons and Navigation Columns can be removed by clearing the relevant check boxes on the Settings Page.

Course Genie makes extensive use of Cascading Style Sheets (CSS). The format of each course is controlled by a Course Genie Scheme, which has several style sheets. Values in these style sheets can be amended to reduce the amount of white space displayed and to remove the logo from the top of each page.

The screenshots below show the cumulative effect of these changes in producing clear, readable pages that are easy to navigate with a minimal amount of downward scrolling.

8

9 4 Using the materials The modified materials worked surprisingly well on all the devices tried. A few further screenshots from the Vario II and Opera Mini are shown below.

10 All of the materials displayed exceptionally well on the Nokia N800 Internet Tablet, but this is hardly surprising, given the size of the screen (800 x 480). Indeed, even the normal version (not converted for mobile devices) of the materials displayed well on this screen. Unfortunately, though, it was not possible to take screenshots on this device.

The materials also displayed surprisingly well on the Nokia 6151, despite the small size of the screen, and all navigation features appeared to work correctly.

In general, all the devices tested displayed the materials in a usable format and permitted effective navigation, although the devices with larger screens provided the best user experience.

At present there are three different technologies in use for accessing the internet from mobile devices: 3G/UTMS, GPRS/GSM and WAP. Download speeds using 3G/UTMS were excellent, although not quite up to broadband standard. Speeds using GPRS were acceptable — more comparable to a good dial-up connection. No tests were carried out using WAP, but given that this older technology is known to be significantly slower than GPRS, it is unlikely to produce satisfactory results.

Some of the devices (the Vario II and the Nokia Internet Tablet) also supported WiFi access, where a suitable network was available.4 Download speeds using this technology were also acceptable, although this is obviously going to vary, depending on the characteristics of the network.

4 When this report was written the Nokia Internet Tablet was one of the few mobile internet access devices of its type. Apple has since introduced the iPod Touch and the IPhone, both of which support WiFi internet access. This may expand the market for this type of device. The iPhone also offers telephone access to the internet but, at the time of publication, not 3G, although this has been promised for the future.

11 5 Using mobile devices for assessment In principle, there appears to be great potential for using mobile devices for assessment, particularly as they can also be used to contact a candidate via telephone or Voice-over-Internet-Protocol (VoIP), and can therefore also serve an authentication function.

However, the mobile assessment aspect of this project was not particularly successful. The online assessments generally used for Internet Safety are hosted on the SOLAR website (www.solarproject.org). Most of the devices tested could not get past the first page, probably because subsequent pages appear to be Flash- based. Only the Nokia N800 Internet Tablet, which supports Flash 9, displayed the SOLAR pages correctly.

Course Genie allows a whole range of question types, including multiple choice questions, completion items, short response questions, etc to be included in online course materials. For security and authentication reasons, these are not particularly well-suited to summative assessment via a mobile, but they do provide an excellent medium for formative assessment, with immediate feedback to the user.

Most of the testing for this report was carried out using Windows Mobile 5 on the MDA Vario II. The Course Genie online questions did not work well under Windows Mobile 5: the questions appeared to display all right, but when the ‘Check the Answer’ button was clicked, nothing happened. According to the support staff at Wimba (the publisher of Course Genie), this is because they rely on Java, which is not currently well supported on mobile devices.

However, just before the final version of this report was produced, Windows Mobile 6 was released, offering an improved implementation of Java. When the Course Genie questions were tried on this release, they worked perfectly, so the problem appears to be solved. Within the next few months, all new Windows Mobile devices (both smartphones and PDAs) will be supplied with Windows Mobile 6, and some older devices (although only the most recent ones) will be upgradeable.

The Course Genie online questions worked well under Opera Mini, as you can see from the following screenshot.

12

Other online assessment solutions appear to suffer from similar problems. For example, with Zoho Challenge (www.zohochallenge.com) questions and answers appear to work, but the results are not saved. The developers of this product believe that this may be a Javascript problem and are looking to see if it can be resolved.

It may be that the future of online assessment lies with different forms of assessment. For example, M-Learning (see Section 7 ‘Vendor technologies’) offers an assessment tool based on text messaging (SMS), which avoids many of the technological difficulties discussed earlier.

Many mobile devices would also be highly suited to collecting evidence for a portfolio-based assessment and uploading it to a mobile . This could be done using systems such as moblog.co.uk or mobiblogr, which are specifically designed to work with mobile devices. These systems allow photos, videos or audio recordings to be captured on a mobile device and be uploaded instantly to a website where an assessor or verifier could examine them online.

13 6 Candidate and teacher experience During June a local secondary school carried out a very brief trial of the mobile version of the Internet Safety materials with a number of students. The devices used were relatively modern mobile telephones which gave GPRS access to the internet via proprietary browsers. The students encountered the following problems:

♦ The internet connection was of a poor standard and frequently dropped. ♦ Many of the links would not open, particularly Wikipedia. ♦ Pages would often open correctly, then stop loading after a few seconds and refresh in double line spacing with links missing (O2 and Vodaphone). ♦ Quizzes appeared OK on some phones and answers appeared to be accepted, but the ‘correct answer reveal’ function did not work (O2 and Orange). ♦ Quiz answers occasionally appeared vertically on the page, making reading them a bit tricky! ♦ Large margin down left side left little room on the page for text. ♦ Closing the page at the end of each section to return to the home page did not work as there was only one window open at a time.

Many of these problems appear to be due to the fact that the students were using relatively unsophisticated telephones with proprietary browsers. None of these problems appeared when testing the materials with, for example, the Nokia 6151, which, although relatively cheap, is a very recent device. Results could probably have been improved by using the Opera Mini browser.

The students also made a few comments on their experience of accessing the course from mobile phones:

♦ It takes a long time to find out how to connect to the correct page at first. ♦ Students do not use the internet on their phones — it's too expensive! ♦ Eyes feel 'tired' after a while. ♦ It’s a bit awkward to hold the phone for so long just reading. ♦ The opening page has to be scrolled through every time to get to the links — the same action all the time! ♦ When holding the 'down' button to scroll through a page, you often go past the part you want and have to scroll back up. ♦ It's possible to use shortcuts to return to the page you were on (once you learn how to create shortcuts). ♦ WAP browsers are quite difficult to use and have few instructions.

Again, many of these problems result from the basic nature of the phones and browsers being used. The students tended to use ‘pay-as-you-go’, rather than contract phones, greatly increasing the cost of internet access. However, there are lessons to be learned, particularly in terms of separating introductory and navigational material and in putting in shortcuts to ease navigation.

14 7 Vendor technologies During the project, two potentially useful vendor-specific technologies were encountered: M-Learning and Zoho Challenge.

M-Learning (www.m-learning.org/index.htm) offers a set of tools for mobile learning:

♦ mediaBoard is similar to a VLE or Content Management system, and allows learners to send text (SMS) or picture (MMS) messages from their phones to personal and collaborative websites. ♦ SMS quizzes let learners send a simple text message with the answer and get an instant reply. There is an online authoring tool for creating SMS quizzes. ♦ MyLearning author for Pocket PC can be used to create media-rich mobile learning resources.

The complete suite costs £2,500 + VAT for an annual licence that allows 50 tutors and 1,000 students to use the system. This includes only a limited number of SMS messages (400), so if the system was used extensively this could be a significant additional cost. A trial version is available which allows three months access to all the tools for £250.

15 Zoho, producers of the well-regarded Zoho Office application, also produce an online testing tool called Zoho Challenge (www.zohochallenge.com), which offers a number of useful features:

♦ Support for multiple choice questions and descriptive questions. ♦ Results can be displayed immediately (this is configurable). ♦ Time limit can be specified for a question paper. ♦ Difficulty levels calculated automatically. ♦ Online access to candidates' details is available. ♦ Send e-mails to candidates, giving date and time of test and specifying a one- time user ID and password.

It allows users to:

♦ create multiple exams ♦ add candidates ♦ schedule tests up to years ahead ♦ view candidates' performance ♦ view a ‘report card’ ♦ view graphical representations of the overall results of your tests

Zoho Challenge is free to use for up to 25 candidates, and is modestly priced for higher numbers of candidates.

Zoho Challenge appears to work well on mobile devices running Windows Mobile 6 or Opera Mini, but the final results are not reported back to the server and recorded. The developers are aware of this problem and are hoping to resolve it.

16 8 Conclusions and recommendations This preliminary study has highlighted three major points:

♦ It is possible, and even relatively easy, to adapt existing learning materials so that they can be displayed on a variety of mobile devices. ♦ Assessment on mobile devices is not nearly as straightforward, largely due to the fact that existing ways of delivering assessment use technologies, such as Flash, that are poorly implemented on mobile devices at present. ♦ In general, things worked better on the newer and more sophisticated devices than on the older and simpler ones.

The second point may not be as important as it appears, as the ability of mobile devices to support complex technologies seems to be improving, eg with Windows Mobile 6.0.

Opera is reported to be working on an alternative to Flash as an add-on for the Opera Mini browser — they believe that Flash itself is too memory- and CPU- hungry to be used on mobile devices.

It looks as if there could be two approaches to any further developments: go for a lowest common-denominator approach that will work on as wide a range of devices as possible; or accept that a minimum standard of device is required to access future developments. A suitable standard might be to target Windows Mobile devices, both smartphones and PDAs.

Hardware Further thought needs to be given to the whole area of hardware. It may be OK to exclude older (eg non GPRS/3G) mobile phones, but what about other major types of device, such as Palm-based devices, Blackberrys and the new Apple iPhone?

Devices like the Nokia N800 may be very much in the minority, but there are other devices of this nature appearing, eg the Datawind Pocketsurfer (http://www.reghardware.co.uk/2007/07/13/preview_datawind_pocketsurfer_2/).

Technologies Techniques for carrying out mobile assessment also need to be considered. The M-Learning platform may be worth further investigation. It may be useful to obtain a copy of the trial version and investigate the tools, although mediaBoard may simply duplicate functions that are readily available elsewhere, such as mobile blogging tools. The SMS quizzes certainly look intriguing, and it may be that many potential students, especially younger ones, would find them interesting. It would also be interesting to compare the MyLearning authoring tool with Course Genie, to see if it provides any additional functionality, or is easier to use.

17 Most of the existing problems in mobile assessment appear to arise from the inability of mobile devices to fully support technologies like Java and Flash. These problems may vanish as support for these technologies improves. It may also be worth examining the ability of mobile devices to support portfolio-based assessment, via the uploading of artefacts to some kind of online repository.

Service providers Consideration also needs to be given to the ability of different mobile service providers to support mobile learning. At present, some providers allow access only to a limited range of internet facilities, whilst others make very high charges for internet access. At present, subscription-based services from T-Mobile and 3 seem to be the best option, but the picture is changing constantly. T-Mobile also provides a pay-as-you-go option which is capped at £1.00 per day.

Learners The findings of the pilot study of school-based learners suggests that they may not be the best target for mobile learning, due to their tendency to use the more basic models of mobile phone and to rely on pay-as-you-go, rather than subscription- based services. Some of the difficulties here could be overcome by the adoption of Opera Mini as a common browser platform, which provides a reliable experience on many different varieties of hardware and service. Mobile learning may be more appropriate to a corporate market, where more sophisticated devices are more common, or to colleges and training centres which can provide their students with suitable devices.

Further research Further research and projects in online delivery and assessment would be valuable. The following objectives could be incorporated:

♦ Produce a guide to producing or converting online materials for mobile delivery to provide guidance to future developers. ♦ Try a wider-ranging pilot in an area with good mobile phone connectivity, targeted solely on devices running Opera Mini or Windows Mobile 6. ♦ Assess the capability of products from other vendors (eg M-Learning) for delivering and assessing online courses. ♦ Assess the usefulness of mobile devices in collecting evidence for portfolio- based assessment. ♦ Assess the usefulness of as a means of delivering learning online. ♦ Develop sufficient material to allow a Group Award to be delivered.

18 The future In summary, the outlook for mobile delivery is positive, and there is already a large amount of material available that could be converted without undue difficulty.

The picture regarding mobile assessment is more complex and requires further investigation, but it may improve as technological developments take place.

There is also scope for using mobile technology to support portfolio-based assessment. The whole area deserves further study.

19