© 2019 published by FH JOANNEUM Gesellschaft mbH

Cover image by Mathias Knoll Layout by Mathias Knoll

FH JOANNEUM Gesellschaft mbH Publishing Alte Poststraße 149 A-8020 Graz www.fh-joanneum.at

Last modification: Tuesday 12th March, 2019 at 14:01. 10.1 and Swift Version 4.2.1. ISBN eBook: 978-3-902103-83-3

The work is published under a Creative Commons Attribution-ShareAlike 3.0 Austria license. CC BY-SA 3.0 AT

https://creativecommons.org/licenses/by-sa/3.0/at

ii Contents

1 iOS Development – The Big Picture 1 1.1 Introduction ...... 1 1.2 WIIFM – What is In It For Me ...... 1 1.3 Overall Concepts ...... 3 1.3.1 Philosophy ...... 3 1.3.2 App Design on Storyboards ...... 3 1.3.3 Selected Frameworks ...... 3 1.4 Development Tools ...... 3 1.4.1 Finding Help ...... 4 1.4.2 Xcode ...... 4 1.4.3 Deployment on Real Devices ...... 5 1.4.4 Deployment ...... 6 1.4.5 Package Manager ...... 6

2 Programming watchOS 15 2.1 watchOS ...... 15 2.2 The ...... 15 2.3 Apples Philosophy ...... 16 2.4 Overall Architecture ...... 16 2.5 Basic Concepts ...... 18 2.5.1 Views and Navigation ...... 18 2.5.2 The Navigation Flow ...... 20 2.5.3 Watch Kit Interface Elements ...... 20 2.5.4 Interaction ...... 20 2.5.5 Views and Controllers ...... 21 2.5.6 The Life Cycle of an Interface Controller . . . . . 22 2.5.7 Programming Navigation with/without Segues . . 24 2.5.8 Controls and Views ...... 25

iii 2.5.9 Complications ...... 26 2.5.10 Notifications ...... 26 2.5.11 Features ...... 27 2.5.12 The Life Cycle of a Watch Application ...... 27 2.5.13 Communication with iOS ...... 29 2.5.14 Security and Privacy ...... 34 2.6 Technical Data ...... 35

3 Swift I Programming Basics 39 3.1 Introduction ...... 39 3.1.1 Programming in Swift ...... 39 3.1.2 Swift Scripts ...... 40 3.1.3 REPL in the Playground ...... 40 3.1.4 Documentation in Playgrounds ...... 40 3.2 The Swift Language ...... 42 3.2.1 Coding Style Guides and Documentation . . . . . 43 3.2.2 Type Inference ...... 43 3.2.3 Optionals ...... 44 3.2.4 Functions and Parameters ...... 44 3.2.5 Closures ...... 46 3.3 Data Types ...... 46 3.3.1 Map and Reduce ...... 47 3.4 Object-Oriented Programming ...... 48 3.4.1 Inheritance, Polymorphism ...... 49 3.4.2 Extensions ...... 50 3.4.3 Protocols ...... 50 3.4.4 Protocol Extensions ...... 51 3.5 Further Hints ...... 51 3.6 Completely Unnecessary Knowledge ...... 51

4 Swift II Advanced Programming 55 4.1 Introduction ...... 55 4.2 Advanced Programming in Swift ...... 56 4.2.1 ARC – Automatic Reference Counting ...... 56 4.2.2 LLVM and LLDB – Compiler and Debugger . . . 57 4.3 Advanced OOP ...... 57 4.3.1 Properties ...... 57 4.3.2 Operator-Overloading ...... 58

iv 4.4 Asserts, Exceptions and Preconditions ...... 59 4.5 Nested Functions and Function Pointers ...... 61 4.6 Generics ...... 62 4.7 Structs and Mutating Functions ...... 63 4.8 Design Patterns in Swift ...... 63 4.9 Limitations of Swift ...... 64 4.10 The Future of Swift ...... 64

5 Programming iOS 69 5.1 Introduction ...... 69 5.2 Coding for iOS ...... 69 5.3 General App Design ...... 70 5.3.1 Performance ...... 70 5.3.2 Energy Consumption ...... 70 5.4 App Life Cycle ...... 70 5.5 Unit Tests ...... 71 5.6 Widgets ...... 73 5.7 Accessibility ...... 75 5.8 UI Tests ...... 75 5.9 Auto Layout ...... 76 5.10 Navigation Flow ...... 78 5.11 Interaction Design for Multitouch Input ...... 80 5.12 Internationalisation ...... 81

6 Concurrency 85 6.1 Introduction ...... 85 6.2 Concurrency ...... 85 6.3 Dispatch Queues ...... 86 6.4 Fetching Data ...... 86 6.5 Web Services ...... 90 6.6 Deserialisation ...... 91

7 Persistency 99 7.1 Introduction ...... 99 7.2 Persistency ...... 99 7.2.1 Default Configurations ...... 99 7.2.2 Persistency with Shared Preferences ...... 100 7.2.3 Cloud-Based KeyValue Store ...... 101

v 7.2.4 Object/Relational Mapping (O/R-M) ...... 101 7.2.5 Persistency with ...... 101 7.2.6 Alternatives to Core Data ...... 106 7.2.7 Secure Persistency ...... 106

8 Location-Based Services 109 8.1 Introduction ...... 109 8.2 Location-Based Services ...... 109 8.2.1 Users Context ...... 110 8.2.2 GPS and GLONASS Location ...... 110 8.2.3 Geofencing and Location-Based Services . . . . . 112 8.2.4 Maps ...... 112 8.2.5 Overlays on Maps ...... 114 8.2.6 Geocoding and Reverse Geocoding ...... 114 8.2.7 Remarks for Enterprises ...... 114

9 Animation 119 9.1 Introduction ...... 119 9.2 Animation ...... 119 9.2.1 Game Loop ...... 120 9.2.2 2D Drawing ...... 120 9.2.3 SpriteKit and SceneKit ...... 123

10 Sensors 127 10.1 Introduction ...... 127 10.2 Sensors and Actuators ...... 127 10.2.1 Actuators ...... 128 10.2.2 Sensors ...... 128 10.2.3 Sample Code ...... 131 10.2.4 More Frameworks ...... 133

11 Security 137 11.1 Introduction ...... 137 11.2 General Aspects ...... 138 11.3 The ...... 138 11.4 Apps and the Operating System ...... 139 11.4.1 Deployment and Installation ...... 139 11.4.2 App Separation ...... 139

vi 11.4.3 IPC-Channels ...... 139 11.4.4 The ...... 139 11.4.5 Biometric Sensor ...... 140 11.4.6 Machine Learning and Privacy ...... 140 11.5 Networking and Cloud ...... 140 11.5.1 Networking ...... 140 11.5.2 Cloud Services ...... 140 11.5.3 Secure the Infrastructure ...... 141 11.6 Secure Coding ...... 141 11.6.1 Web Application Security ...... 141 11.6.2 Testing ...... 142 11.7 Forensics ...... 142 11.8 Analysis Tools ...... 142 11.9 Jailbreaks ...... 145 11.10Selected Issues ...... 146 11.13Books on Security ...... 147

12 System Services 151 12.1 Introduction ...... 151 12.2 System Services ...... 151 12.2.1 Review of the iOS Layers ...... 152 12.2.2 Access to Photos and ...... 152 12.2.3 Sharing ...... 154 12.2.4 Inter Process Communication ...... 155 12.2.5 watchOS ...... 157

13 Cloud Services 161 13.1 Introduction ...... 161 13.2 Cloud ...... 161 13.2.1 iCloud – Ideas and Concepts ...... 162 13.2.2 iCloud Development with CloudKit ...... 163 13.2.3 CloudKit JS ...... 166

14 Local and Push Notifications 171 14.1 Introduction ...... 171 14.2 Notifications ...... 171 14.2.1 Local Notifications ...... 173 14.2.2 Push Notifications ...... 174

vii 15 Machine Learning 183 15.1 Introduction ...... 183 15.2 Data Acquisition ...... 184 15.2.1 Data Augmentation ...... 184 15.2.2 Training ...... 184 15.2.3 Limitations ...... 184 15.2.4 Validation ...... 184 15.3 Image Classification ...... 184 15.3.1 Working with Image Classifiers ...... 185 15.4 Natural Language Classification ...... 186

16 Selected topics 189 16.1 Introduction ...... 189 16.1.1 Extensions – Today Widgets ...... 190 16.1.2 App Search with ...... 191 16.1.3 Speech Recognition and Search ...... 193 16.1.4 Gamecenter ...... 194 16.1.5 Mobile Payment ...... 194 16.1.6 Testing and Optimisation ...... 195 16.1.7 Evolving Swift ...... 196 16.1.8 ...... 196 16.1.9 iOS Accessories ...... 198 16.1.10 iOS in Enterprises ...... 201 16.1.11 Scientific Research and iOS ...... 202 16.1.12 Schools and Educational Institutions ...... 202 16.1.13 App Store and iTunes Connect ...... 202 16.2 Non-technical Remarks – Ethics ...... 203

A Appendix 231 A.1 iOS Version Changes ...... 231 A.2 watchOS Version Changes ...... 233 A.3 UX Design Demos ...... 234 A.4 UX Design Wireframe Templates ...... 235

viii 1 About Programming for iOS and watchOS

“Most good programmers do programming not because they expect to get paid or get adulation by the public, but because it is fun to program.”

[Linus Torvalds]

Summary: This book1 should give students a concise summary of impor- tant aspects of iOS programming.

1.1 Introduction

The author designed this book as a visual, but technical, guide for learning iOS and watchOS programming by focusing on the concepts and present- ing only the most interesting aspects of the selected topics. Source snippets support a hands-on approach to picking up new concepts.

1.2 WIIFM – What is In It For Me

To learn fast and focus on implementation, the book is structured as shown in Table 1.1. Each section contains the basic concepts with short demo code and real world code. Requirements: readers are expected to know the basis of User Centred Design (UCD), they should be fluent with the unix command line, and should have experience with at least one or two object- oriented languages. Furthermore, to stimulate discussion among students, one or more relevant scientific papers are added to each section.

1 All Materials form open mobile development are distributed under the Creative Commons http://creativecommons.org/ Share Alike Attribution licence.

1 Section Topic 1 Concepts General concepts and development setup 2 Apple Watch watchOS design patterns 3 Swift I Basics of Swift programming 4 Swift II Advanced concepts in Swift 5 iPhone and iPad The iOS GUI layout, navigation and interaction 6 Concurrency Background tasks and web services 7 Persistency ORM with core data 8 Location GPS, maps and location-based services 9 Animation Graphics 2D 10 Sensors Proximity, ambient light, gyro and other sensors 11 Security Operating system, network and app security 12 System Services Address book and social framework integration 13 Cloud iCloud services 14 Notifications Local and push notifications 15 Machine Learning Classifying text and images 16 Selected Topics Further frameworks and technologies

Table 1.1: Section overview.

2 1.3 Overall Concepts

To understand the different approaches when developing for Apple, it is useful to take a quick look at the overall philosophy of the design and implementation as well as to glance at the technical concepts incorporated.

1.3.1 Philosophy User experience (UX) counts! Pay attention to details; for example, the de- sign, size and usage of Apple app badges is specified in detailed paragraphs on the official development pages2. Know your customer. In general, Ap- ple users are more likely to spend money on software and services and accept a more restricted environment than Android users.

1.3.2 App Design on Storyboards Storyboards support the design and implementation of a GUI as it provides interactive ways of connecting single views to set up the flow through an application.

1.3.3 Selected Frameworks Developers program in a given set of documented frameworks with an of- ficial API. Use of so-called private API calls will expel any app from the App Store. Find frameworks relevant for writing games in Section 9 Animation and important frameworks for security and privacy in Section 11 Security.

1.4 Development Tools

The main integrated development environment (IDE) is Xcode3, see Fig- ure 1.1, which incorporates graphical design of user interfaces, coding and debugging. It is supplemented with further tools such as Instruments for profiling to detect performance issues or Application Loader to publish

2 Find App Store marketing resources and identity guidelines at https://developer.apple.com/app-store/marketing/guidelines/#downloadOnAppstore. 3 https://en.wikibooks.org/wiki/Swift learning#Xcode

3 apps to the store via iTunes Connect. In Section 16.1.6 Outlook, Code Op- timisation hints and pointers to code optimisation and performance tweak- ing are given. by mathias knoll

Figure 1.1: The different sections of the Xcode IDE: Toolbar with the Assistant Editor, Navigation Area, the Main Editor pane, Debug Area, and a Utility Area.

1.4.1 Finding Help Documentation and the API reference can be downloaded and integrated into Xcode, or viewed online at the Library https:// developer.apple.com/library/.

1.4.2 Xcode An single iPhone or iWatch user interface (UI) view is called a scene. Sev- eral scenes are visualised on the Xcode Storyboard and can be connected

4 interactively. Without writing any code the views designed in the story- board already can be run on a real device for simple click prototypes sim- ulating the flow through an app. Xcode makes relatively sparse use of wizards and code generation – they are used, for example, in initial project setup. Beware of the following aspects of Xcode IDE: Firstly, it is difficult or sometimes impossible to refactor in Swift, and secondly, the files listed in the left Project Navigator view do NOT correspond to the actual file system. Be careful! • Views: Use the Assistant View to see the GUI on the storyboard and associated code. • Editors: In the Project Navigator, you can view files and assets. Be- sides *.swift and *.storyboard files, there are xib Interface Builder files (internally stored in XML format). The plists are property lists. xcassets are Xcode Assets, a package of resources such as images in different sizes4. • Editors: The Document Outline is helpful to view the structure of GUI elements. • Compile and Run: select a proper Scheme that defines the build set- tings before running the app in the simulator(s) or on real devices. • Breakpoints and Debugging: If breakpoints are set, a debug build (as defined within a scheme) is chosen automatically and the program pauses at the specified breakpoint. Use the debugger to view values.

1.4.3 Deployment on Real Devices A (free) Apple ID is necessary to test and deploy applications on real iOS devices.

Debugging on Real Devices Starting with iOS 11 debugging over WLAN is supported. This gives more flexibility when testing sensors or performing test with real users.

4 Apple suggests not letting an app scale images down (memory spikes!) or up (ugly artefacts!) automatically at runtime. It is preferred to use Automator to prepare images in proper formats in advance.

5 1.4.4 App Store Deployment

In a section called Member Center, developers can log in5 and manage de- velopment teams – including the Certificates and Profiles – online. Before upload every App has to be run on a real device. Only then is the gen- eration of an Archive possible. Such archives can be managed with the Xcode Organiser. Uploading apps for (beta testing with TestFlight or) fi- nal publishing is done via the iTunes Connect web site and the local Xcode Organiser.

1.4.5 Non-Apple Services – A Swift Package Manager

CocoaPods6 provides a package management system for Swift libraries which can take care of dependencies between libraries. It is important to use an Xcode workspace to allow CocoaPods to manages libraries required by you project as additional sub projects (modules). An Apple supported package manager7 is available for macOS, but still under development for the iOS platform.

1.5 Suggested Reading

There is a great free course (see Figure 1.2) by Paul Hegarty for Stanford University on iTunesU https://itunes.apple.com/de/course/developing-ios- 11-apps-with-swift/id1309275316.

1.6 Further Reading

Find selected research papers and pointers to the tutorials in the corre- sponding sections of this book. The following books are highly recom- mended:

5 https://developer.apple.com/membercenter 6 https://cocoapods.org 7 https://swift.org/package-manager/ and https://github.com/apple/swift-package-manager.

6 einn psfrSmartphones for Apps Designing Hegarty. Paul by versity 1.2: Figure h orefriSdvlpetwt wf tSafr Uni- Stanford at Swift with development iOS for course The eineyu products. your (Hurff, perience Love People 2015 Products Designing and wrong interfaces. going user improve is vi- to how many what gives of – examples old sual rather being book despite The – apps. designing when users on cus (Clark, Tapworthy hfsyu ou nhwpol ex- people how on focus your shifts ) 7 2010 xlishwt fo- to how explains )

screenshot by john feiner 8 Swift Programming Language and iOS Apps

Swift for Programmers (P. Deitel and H. Dei- tel, 2015) helps you learn the tricky parts of Swift, such as failable initialisers, the com- parable protocol, generics, mutating func- tion and operator overloading. It also shows how to bridge Swift and Objective-C. Com- pare also Learning Swift: Building Apps for macOS, iOS, and Beyond by Manning, Buttfield-Addison, and Nugent (2018). Swift 4 protocol-oriented program- ming (Hoffman, 2017) stresses the ad- vantages of designing with protocols over creation of class hierarchies. Furthermore it gives many source code examples of how to implement design patterns appropriately. Slim book – great content.

iOS 8 for Programmers: An App-Driven Ap- proach with Swift (P. Deitel, H. Deitel, and A. Deitel, 2014) covers seven fully coded iOS apps.

Swift 4 Programming Cookbook: 50 Task- Oriented Recipes to Make You Productive With Swift 4 (Moon, 2017) An extensive description of datatypes and classes (with generics) for beginners. It includes a short section for server-side swift on Ubuntu with the Vapor framework. iOS 9 Swift Programming Cookbook – So- lutions & Examples for iOS Apps (Naha- vandipoor, 2016) covers topics such as maps and location, multitasking, web and multi- media and even watch apps and security. 9 WatchOS Applications

Learning WatchKit Programming – A Hands-On Guide to Creating Apple Watch (Lee, 2015) is an excellent, albeit slightly outdated introduction to watchOS, as watchOS 2 handles things differently and the API has undergone multiple changes.

10 Security

*OS Internals: Volume III – Security & Insecurity (Levin, 2018) a very advanced book (updated frequently, printed on de- mand) with detailed information about the operating systems, including security as- pects of file system and kernel, with a col- lection of sample code (including exploits). Check out the updates for 2018 at http:// newosxbook.com/. You may need Bitcoins to buy this book. iPhone and iOS Forensic (Hoog and Strzempka, 2011) helps you perform inves- tigation, and analysis on mobile security for Apple iPhone, iPad and iOS devices.

Third edition of Practical Mobile Foren- sics: A Hands-On Guide to Mastering Mo- bile Forensics for the iOS, Android, and the Windows Phone Platforms by Mahalik et al. (2018).

The Mobile Application Hacker’s Handbook (Chell et al., 2015) encourages you to see your app through the hacker’s eyes.

11 1.7 Review Questions

A good approach to answering following question is to try to define and explain a given term in your own words, to state typical usage, domains (what problem is this a solution for?), and alternatives. Furthermore, ex- plain the advantages AND disadvantages and optionally give source code examples.

1.7.1 Some Basic Questions • Can you explain and compare the development processes for iOS and Android applications?

• Can you explain terms such as assets, schemes or storyboard?

1.7.2 Some More Advanced Questions • Can you name the main development, debugging, testing and distri- bution tools used by an iOS development team including their rele- vant features?

• Can you explain the App Store distribution process including the requirements concerning app id, bundle id, entitlements, certificates and profiles. What are the differences to beta testing with TestFlight?

12 Bibliography

Chell, Dominic, Tyrone Erasmus, Shaun Colley, and Ollie Whitehouse (Feb. 2015). The Mobile Application Hacker’s Handbook. Wiley. ISBN: 9781118958513. Clark, Josh (June 2010). Tapworthy: Designing Great iPhone Apps. O’Reilly Media, Inc. ISBN: 9781449381653. Deitel, Paul and Harvey Deitel (Jan. 2015). Swift for Programmers. Deitel Developer Series. Pearson Education. ISBN: 9780134021560. Deitel, Paul, Harvey Deitel, and Abbey Deitel (2014). iOS 8 for Program- mers: An App-Driven Approach with Swift. Deitel Developer Series. Pearson Education. ISBN: 9780133965414. Hoffman, Jon (2017). Swift 4 Protocol-Oriented Programming: Bring Pre- dictability, Performance, and Productivity to Your Swift Applications. 3rd ed. Packt Publishing, p. 210. ISBN: 1788473825. Hoog, Andrew and Katie Strzempka (June 16, 2011). iPhone and iOS Forensics: Investigation, Analysis and Mobile Security for Apple iPhone, iPad and iOS Devices. Syngress, p. 336. ISBN: 9781597496599. Hurff, Scott (Jan. 3, 2015). Designing Products People Love: How Great Designers Create Successful Products. O’Reilly Media, Inc. ISBN: 9781491923672. Lee, Wei-Meng (2015). Learning WatchKit Programming: A Hands-On Guide to Creating Apple Watch Applications. Pearson Education. ISBN: 9780134195544. Levin, Jonathan (Aug. 2018). *OS Internals: Security & insecurity / Jonathan Levin. 2nd ed. Vol. 3. Technologeeks.com. ISBN: 991055531. Mahalik, Heather, Satish Bommisetty, Oleg Skulkin, and Rohit Tamma (Jan. 23, 2018). Practical Mobile Forensics: A Hands-On Guide to Mastering Mobile Forensics for the iOS, Android, and the Windows

13 Phone Platforms. 3rd ed. Packt Publishing, p. 402. ISBN: 1788839196. URL: https://books.google.at/books?id=3tRJDwAAQBAJ. Manning, J., P. Buttfield-Addison, and T. Nugent (Mar. 2018). Learning Swift: Building Apps for macOS, iOS, and Beyond. O’Reilly Media. ISBN: 9781491987520. URL: http://search.ebscohost.com/login.aspx? direct=true&scope=site&db=nlebk&db=nlabk&AN=1743325. Moon, Keith (2017). Swift 4 Programming Cookbook: 50 Task-Oriented Recipes to Make You Productive With Swift 4. Packt Publishing. ISBN: 9781786466631. URL: https : / / books . . at / books ? id = 7JlGDwAAQBAJ. Nahavandipoor, Vandad (Jan. 2016). IOS 9 Swift Programming Cook- book: Solutions & Examples for IOS Apps. O’Reilly Media, Inc. ISBN: 9781491936696.

14 2 Coding for the Apple Watch

“Simplicity is the ultimate sophistication.”

[Leonardo da Vinci]

Summary: This chapter deals with the creation of applications for the Apple Watch. Designing for very limited space and coding for watchOS involves applying many well-known iOS concepts, but also some very dif- ferent and special approaches.

2.1 watchOS

This section on watchOS is structured as follows: first we explain the basic concepts, then we present some short demo code which gives insights into suggested ways of coding. The section concludes by mentioning some papers and pointers for tutorials.

2.2 The Apple Watch

In September 2014 from Apple announced a smartwatch called Apple Watch during the keynote1 at the Apple Special Event in Cupertino with the words Apple Watch is the most personal device we’ve ever cre- ated., so it is perceived2 by many as an (Internet of Things (IoT) gadget. Note for setting up and using the Apple Watch an iPhone with iOS 8.2 or greater is required!

1 Watch announced at Apple Event http://www.apple.com/live/2014-sept-event/. 2 http://www.zdnet.com/article/apple-watch-its-an-internet-of-things-play/

15 2.3 Apples Philosophy

The ecosystem is closed, but tightly integrated, as can be seen for the watch streaming via AirPlay to remote computers in Figure 2.1. iwatch screenshot by john feiner

Figure 2.1: The philosophy of Apple is to provide a tight integrated user experience throughout their – very often closed – ecosystem.

Before starting to code it is suggested to work through the Ap- ple Human Interface Guidelines, which can be found at https:// developer.apple.com/watch/human-interface-guidelines/. For example, watch out for new Dock concept for watchOS 3 which allows to open recently used apps in a fast way. Read through the hints for choosing the proper layout and global tint color for branding and legibility in the Section Visual Design.

2.4 Overall Architecture

The application consists of the iOS App (the iPhone App, responsible for main tasks such as fetching data from the Internet), the WatchKit Extension (watch programme logic coded in Swift) and the WatchKit App(the story- board and resources for the graphical user interface of the watch). Since watchOS 2 the WatchKit Extensions (see Figure 2.2 Architecture and Fig- ure 2.3 File System) run on the watch3 and allows standalone applications.

3 The current watchOS is called watchOS 4; version 2.0 of watchOS was introduced at the WWDC in 2015. Early versions run the extensions on the iPhone and running apps without an iPhone nearby was not possible.

16 Watch-App iPhone-App Extension omnigraffle drawing by john feiner

Figure 2.2: Since watchOS 2 the extensions reside in the Apple Watch. pathfinder screenshot by john feiner

Figure 2.3: In the file system find the iOS App, the WatchKit App (hold- ing the storyboard with the graphical interface for the watch app) and the WatchKit Extension (with the logic for the watch app).

17 2.5 Basic Concepts

To getting started with programming, it is important to note the differences from classical iOS development for and . The Apple Watch has very limited screen size and very limited interaction possibilities. The concept of connection GUI-elements with code: Use IBActions and IBOutlets as can be seen in Figure 2.4. xcode screenshot by john feiner

Figure 2.4: Same concept of code–GUI connections: Outlets and Actions.

2.5.1 Views and Navigation

The Dock allows fast navigation to favourite applications. Starting up an application for the first time could take a while. That is why notifications present a static preview first, then a dynamic preview and finally the loaded app. Another way to present information about the status of an application to the user is to include the selected information directly (on fixed places) in the watch face. Those small info-elements, shown in Figure 2.5 are called complications. As the space for the graphical user interface of an app is very limited (Figure 2.7), design is hard, but development is easy. Only few widgets (for example Button, Label, Image, Switch, Slider, Map, Timer ) and lay- outing options (e.g. horizontal/vertical Group, Table) exists and the ways of interaction (with gestures such as tap, swipe, long press, pan) are limited also.

18 ain,wihrpeetsalifraineeet ietyicue in included directly elements faces. watch information the small represent which cations, 2.5: Figure pteapp. the up starting without presented immediately is data static dynamic with or preview a tion(s) 2.6: Figure osbeveso h pl ac nld ocle compli- called so include Watch Apple the on views Possible nnotifica- On

iwatch screenshot by john feiner 19 prt pr utn n text. and buttons spare devel- to oper forces which GUI the 2.7: Figure

iwatch screenshot by john feiner iie pc for space Limited

iwatch screenshot by john feiner 2.5.2 The Navigation Flow Two main ways of navigation between separate views are possible. The Hierarchical and the Page-based navigation concepts are possible. For example a button triggers an push (hierarchical with Back button at top left corner) or modal (page-based with Cancel button at top left) Action Segue. No source code is required to perform those navigations. Only, if one wants to transfer data from one view to the other, code4 is necessary.

2.5.3 Watch Kit Interface Elements Selected GUI-Elements5: For images one selects the image from an asset catalog, picker, table.

2.5.4 Interaction A digital crown allows scrolling and zooming. Standard input via a but- ton, a digital crown and tap/swipe gestures on a touch screen are extended with voice input (, Internet connection via iPhone requried) and some sensors (measure heart rate via infrared and visible-light LEDs and photo- diodes). The output options are vibration, sound and force feedback screen (Taptic Engine). photo by john feiner

Figure 2.8: A digital crown, a button and a touchscreen supporting normal and force touch.

4 Overriding method contextForSegue(withIdentifier:) to send data and get data in awakeWithContext is explained later in this section. 5 All the possible elements are WKInterfaceGroup, WKInterfaceTable, WKInterfaceImage, WKInterfaceSeparator, WKInterfaceButton, WKInterfaceSwitch, WKInterfaceSlider, WKInterfacePicker, WKInterfaceLabel, WKInterfaceDate, WKInterfaceTimer, WKInterfaceMap, WKInterfaceMovie, WKInterfaceMenu and WKInterfaceMenuItem.

20 2.5.5 Views and Controllers A UI is called a Scene. If should hold some custom coded logic, it needs a controller assigned. Therefore a swift source code file – based on the the WKInterfaceController Watch Kit Interface Controller class – must be created (via Menu/File/New/watchOS/Source/WatchKit Class/Subclass of WKInterfaceController). After drag and drop of a new interface controller on the storyboard, the previous created class needs to be set accordingly (inside the new Scene select the WKInterfaceController and in the Enti- tyEditor view set the dropdown to your newly created class as shown in Figure 2.9). xcode screenshot by john feiner

Figure 2.9: Set the base class of your Interface Controller in the Story- board by dropdown-selection of the newly created new Swift class.

On application start, the main view is selected by the main, the initial, interface controller. This startup view is indicated on the storyboard with the gray arrow-left. The startup view can be changed by assigning the attribute is initial controller at a different interface controller.

Short Demo Code – Connecting the Code and GUI Draw connections with CTRL-Drag-and-Drop from the target – e.g.. a button – to a proper location within your code. See Listing 2.1 on page 22 for an example of an IBOutlet6 and an IBAction. The variable titleBtn is of type WKInterfaceButton, which represents the type of an GUI-widget. The annotated function sayHello() is an action triggered by a button click. One gotcha is the invisibility of the connection between UI elements and code. If connections are superfluous or missing, it might result in a crash at runtime. 6 IB is short for Interface Builder, which refers to a former separate application to build graphical user interfaces for macOS.

21 1@IBOutletvar titleBtn : WKInterfaceButton! 2 3@IBActionfunc sayHello () { 4titleBtn.setTitle(” Navigation Demo” ) 5 } Listing 2.1: IBOutlet and IBAction define the GUI to code connection. screenshot by john feiner screenshot by john feiner

Figure 2.10: An empty cir- Figure 2.11: A yellow warn- cle indicates a missing con- ing sign indicates a miss- nections to an UI element. ing connection to the source code.

Development hint: Inspect the connection from the source code to the UI by looking for empty circles (instead line ) left of the actual code as shown in Figure 2.10 and check for warning signs shown in Fig- ure 2.11 on the UI which indicate missing connections to the code.

Real World Code Find an example of a SIMPLE Apple Watch Application written in Swift at https://github.com/KevinPy/WatchCalc. A nice little animation called SpaceWatchRunner be found at https://github.com/charlieelliott/ SpaceWatchRunner. An rather large document based app showing in- tegration of iOS and watchOS – called Lister – is available at https: //github.com/bishalg/Lister-for-watchOS-iOS-and-OS-X-.

2.5.6 The Life Cycle of an Interface Controller Any interface controller class has besides the init method for initialisation several methods which are invoked by the system at special times: awake- WithContext, willActivate and didDeactivate shown in Listing 2.2.

Short Demo Code – Interface Controller Life Cycle

22 1 class NewsSourcesController : WKInterfaceController { 2 let news = [ ” Die Zeit” , ”NY Times” ] 3 let imgs = [” zeit” , ” nyt” ] // check ( names in ) ” Assets ”! 4 // Connection to a table on the watch 5 // ( note : tables need a custom row controller class ) 6@IBOutletvar newsTable : WKInterfaceTable ! 7 8 override func awake ( withContext context : Any?) { 9 super .awake(withContext: context) 10 // Configure objects . Here : simply two rows ... 11 newsTable.setNumberOfRows(2, 12 withRowType : ” newsContr” ) 13 for (idx, title ) in news . enumerated () { 14 if let r=newsTable.rowController(at:idx) 15 as?NewsRowController { 16 r.newsDescription.setText(title) 17 r.newsImage.setImageNamed(imgs[idx]) 18 } 19 } 20 self .becomeCurrentPage() // we open this page first 21 } 22 override func willActivate () { 23 // controller is about to be visible 24 super .willActivate() 25 } 26 override func didDeactivate () { 27 // view controller is no longer visible 28 super .didDeactivate() 29 } 30 // Optionally , we might handle table row selection 31 override func table ( table : WKInterfaceTable , 32 didSelectRowAtrowIndex:Int) { 33 p r i n t ( ” Your selection : ( news [ rowIndex ])” ) \ 34 35 openAnotherScene(withData:news[rowIndex]) 36 }

Listing 2.2: The interface controller life cycle method awake might set up data for a table shown on the UI.

23 2.5.7 Programming Navigation with/without Segues The navigation as explained so far can be accomplished without any code. This allows designers to create clickable prototype without any program- ming skills.

Passing Data From One View to the Next using Segues After segues are named by setting the Identifier in the Attribute Inspec- tor, data can be passed from one view (see Listing 2.3 on page 24) to the (see Listing 2.4). By using the overriding method con- textForSegue(withIdentifier:) in the sending controller and by getting data in the method awakeWithContext in the receiving controller. 1 override func contextForSegue ( 2withIdentifiersegueIdentifier:String)> Any? { 3 if segueIdentifier == ” hierarchicalpush” { 4print(” Segue : ( segueIdentifier ) ” ) \ 5 return [ ” Message” : ” Hierachical Navi” ] 6 else if segueIdentifier == ” pagebasedshowmodal” } { 7print(” Segue : ( segueIdentifier ) ” ) \ 8 return [ ”SoupNo” :19] 9 // now check in modal controller : 10 // context : Optional( SoupNo = 19; ) { } 11 else // Storyboard Segue ID not set / known } { 12 p r i n t ( ” unknown segue ( segueIdentifier ) ” ) \ 13 return nil 14 } 15 } Listing 2.3: Passing data via Segue from the first interface controller.

1 override func awake ( withContext context : Any?) { 2 super .awake(withContext:context) 3 let msg = context ?? ”Warn : we got no data” 4print(”we got context : ( msg ) ” ) \ 5 if let dict = context as?NSDictionary { 6feedbackLabel.setText( 7 ”You pressed the ’ modal ’ button . ” ) 8 if let val = dict [”SoupNo” ] { 9 switch val { 10 case is Int : 11 p r i n t ( ” Soup no : ( val ) ” ) \ 12 default :

24 13 p r i n t ( ” did not expect ( val ) ” ) \ 14 } 15 // push sends us Message ’ Fail fast !’ 16 else if let msg = d i c t [ ” Message” ] } { 17 feedbackLabel.setText( 18 ”You ’ pushed ’ message ( msg ) ” ) \ 19 else if let msg = d i c t [ ”News” ] as?String } { 20 feedbackLabel.setText(msg) 21 else } { 22 p r i n t ( ”No idea what to do with ’ ( dict ) ’ ” ) \ 23 } 24 } 25 } Listing 2.4: Retrieve the data in another interface controller.

Navigate And Pass Data From One View to the Next View Without Segues Navigation can be triggered in code with the methods pushCon- troller(withName:context:) for hierarchical, push views or presentCon- troller(withName:context:), see Listing 2.5, for page-based, modal views. This way the navigation has no connection (segue) on the storyboard and is invisible to designers. The method dismiss, see Listing 2.6, will close the scene and navigates back from the target (second) to the calling (first) interface controller. 1 func openAnotherScene ( withData data : String ) { 2 // open (”push” or ” present ”) other Scene ( s ) 3 // ( without segue on UI ) 4 // pass any data using parameter ” context :” 5presentController( 6withName:” NavigationDemoPushModal” , 7context:[”News” :data]) 8 } Listing 2.5: Passing data without using a segue.

2.5.8 Controls and Views How to configure the UI elements (controls and views) is explained within the Apple Documentation. Find, for example, explanations about the usage

25 1@IBActionfunc navigateBackTheManualWay () { 2 // remove current modal scene ( controller ) 3dismiss()// CANCEL 4 5 // remove the pushed view ( controller ) 6 // self . pop () // < BACK Listing 2.6: Navigate back to the calling interface controller.

of Tables at the Apple Developer Documentation7 or the use of Pickers at Apple8 or in Tutorials9.

2.5.9 Complications Complications, as already presented in Figure 2.5, allow the integration of limited information directly into the watch face. Users configure the ap- pearance by customising the main face showing the time, by adding very short, short or long info chunks at given positions at the watch face. In the source code programmers code the callback logic into controllers based on the Clock Kit class CLKComplicationDataSource. Inside methods such as getPlaceholderTemplateForComplication or getCurrentTimelineEntryFor- Complication the developer sets up the templates for headerTextProvider, body1TextProvider and the body2TextProvider.

2.5.10 Notifications Notifications are delivered from a remote or local source. When remote (push notifications) or local notifications events arrive on an iPhone the iOS decides which are forwarded to the watch. On the phone the user is informed about the notification. If the user wants to view the notification the system displays first a short (a short look interface provided by a system

7 Tables how-to: https://developer.apple.com/library/watchos/documentation/General/ Conceptual/WatchKitProgrammingGuide/Tables.html. 8 Picker API: https://developer.apple.com/reference/watchkit/wkinterfacepicker. 9 Tutorials such as Sneaky Crab http://www.sneakycrab.com/blog/2015/6/12/ wkinterfacepicker-in-watchkit-20-using-the-digital-crown, http://www.raywenderlich.com/117249/watchos-2-tutorial-part-2-tables or http://www.appcoda.com/selectable-table-watchkit/ include links to GitHub to get the complete source code.

26 template with app name, app icon and notification message string), then – if the user keeps looking at the screen – a longer version (static or dynamic long look interface) of the information. Finally, the user might dismiss the notification, trigger an action via tapping on buttons presented, or even launch the corresponding app10 itself.

2.5.11 Features

Bluetooth Networking

Remote control, share data and to other Apple devices. For exam- ple, the remote control for works on bluetooth headsets or streams to . iPhone and watchOS apps work together

Apps share settings and privacy issues (see Figure 2.12) for example Siri (see Figure 2.13) speech recognition.

Configuration via iPhone

The watch has to be set up and configured with an iPhone as shown in Figure 2.14.

2.5.12 The Life Cycle of a Watch Application

The extension delegate holds the logic for the watchOS App. Any exten- sion delegate class has besides the applicationDidFinishLaunching method for initialisation several methods which are invoked by the system at spe- cial events: applicationDidBecomeActive, i.e. a paused tasks will be re- sumed, so you might refresh data on the user interface with your code and applicationWillResignActive when a call comes in or the users quits the app so logic to pause tasks is put in here.

10 Find the concept and details of short and long notifications at https://developer.apple.com/library/watchos/documentation/General/Conceptual/ WatchKitProgrammingGuide/BasicSupport.html.

27 h iPhone. the 2.14: Figure privacy. and behaviour system as such 2.12: Figure h ealdstig o h ac aet ecngrdon configured be to have watch the for settings detailed The hrn settings Sharing

screenshot by john feiner 28 e tonce. at are set watch and iPhone for tion 2.13: Figure

iphone screenshot by john feiner pehrecogni- Speech

iphone screenshot by john feiner Short Demo Code – Extension Delegate Life Cycle Find in Listing 2.7 the life cycle methods for an extension delegate. 1 class ExtensionDelegate : NSObject , WKExtensionDelegate { 2 3 func applicationDidFinishLaunching () { 4 // Perform any final initialization 5 // of your application . 6 } 7 8 func applicationDidBecomeActive () { 9 // Restart any tasks that were paused ... 10 // ... optionally refresh the user interface . 11 } 12 13 func applicationWillResignActive () { 14 // ... the application ... 15 // move from active to inactive state . ... 16 // ...pause ongoing tasks , disable timers , etc . 17 } Listing 2.7: The life cycle methods for an extension delegate.

2.5.13 Communication between WatchOS and iOS Apps The framework WatchConnectivity11 provides means to transfer data be- tween an iPhone and the watch. Those data transfers might happen only when both the iOS and the watchOS apps are active, or delayed in the background and the app is notified when it wakes up. The different ways of communication (update context after wake up, send live, send infos delayed) depend on the life cycle12 (paired? active?) of the watch app. 1 import WatchConnectivity 2 class InterfaceController : WKInterfaceController , WCSessionDelegate { 11 The WatchConnectivity framework with WCSession is presented in the video of WWDC 2015 https://developer.apple.com/videos/play/wwdc2015/713/ and discussed at https: //developer.apple.com/library/watchos/documentation/WatchConnectivity/Reference/ WatchConnectivity framework/index.html#//apple ref/doc/uid/TP40015269. 12 Find WCSession explained at https://developer.apple.com/reference/watchconnectivity/wcsession.

29 3 override func willActivate () { 4 super .willActivate() 5session=WCSession.default 6session?.delegate=self 7session?.activate() 8 } Listing 2.8: On the watch: set up a session between the watch app and the app.

1@IBActionfunc fetchCurrentSongName () { 2session?.sendMessage( 3[” fetch” : ” currentSong” ], 4replyHandler:(response) in { 5 if let idx = response . index (forKey : ” CurrentSong” ) { 6 if let songname = response [ idx ]. value as?String { 7 self .songs.append(songname) 8 else } { 9NSLog(” UnexpectedType : ( response [ idx ]. value ) ” ) \ 10 } 11 else } { 12 self .songs.append(”Unknown msg ( response ) ” ) \ 13 } 14 , } 15 e r r o r H a n d l e r : (err) in { 16 NSLog ( ” Error sending ( err ) ” ) \ 17 } 18 ) 19 } Listing 2.9: On the watch: send a message to the iPhone app.

1 func session ( session : WCSession, 2didReceiveMessagemessage:[String:Any], 3replyHandler:@escaping([String:Any])> Void ) { 4NSLog(” watchOS : We received from ios : ( message ) ” ) \ 5 for msg in message { 6 if let songname = msg. value as?String { 7songs.append(songname) 8OperationQueue.main.addOperation { 9 self .updateSongTable() 10 } 11 else } { 12 NSLog ( ”No idea how to add : ( msg ) ” ) \ 13 }

30 14 } 15 r e p l y H a n d l e r ( [ ” info” : ” Thanks” ]) 16 } Listing 2.10: On the watch: receive a message from the iPhone app. To code this way of communication13 the provided API has to be used, whereby, a WCSession – WCSession.default() –, a WCSessionDelegate (see Listing 2.8), the sendMessage (see Listing 2.9) method as well as the didReceiveMessage (see Listing 2.10) callback are the main ingredients. 1 import WatchConnectivity 2 3 // Will be instantiated in the app delegate : 4 class SongConnectivityHandler : NSObject , 5WCSessionDelegate { 6 var session = WCSession. default 7 8 // dynamic variables can be observed 9 // using key value observers 10 @objc dynamic var messagesFromWatch = [ String ]() 11 12 override init () { 13 super . init () 14 s e s s i o n . d e l e g a t e = self 15 s e s s i o n . a c t i v a t e ( ) 16 NSLog ( ” Info : ” ) 17 NSLog ( ” watch paired : ( session . isPaired ) ” ) \ 18 NSLog ( ” app installed : ( session . isWatchAppInstalled ) ” ) \ 19 } 20 21 func session ( session : WCSession, 22 didReceiveMessage message: [String : Any], 23 replyHandler: @escaping ([String : Any]) > Void ) { 24 if message [” fetch” ] as?String==” currentSong” { 25 let song = ” Bright Side” 26 let msg = ” ( Date () ) : fetching ( song ) ” \ \ 27 messagesFromWatch.append(msg) 28 r e p l y H a n d l e r ( [ ” CurrentSong” :song]) 29 else } { 30 NSLog ( ” Sorry , ( message ) with unknown command . ” ) \ 31 } 32 } 13 See tutorial by Ralf Ebert at https://www.ralfebert.de/tutorials/watchos2-watchkit-connectivity/.

31 Listing 2.11: On the phone: setup a handler class to manage the connection to the watch.

1 import WatchConnectivity 2@UIApplicationMain 3 class AppDelegate : UIResponder , UIApplicationDelegate { 4 var connectivityHdl : SongConnectivityHandler? 5 6 func application ( application : UIApplication , 7didFinishLaunchingWithOptions 8launchOptions: 9[UIApplication.LaunchOptionsKey:Any]?)> Bool { 10 if WCSession . isSupported () { 11 self .connectivityHdl = SongConnectivityHandler() 12 else // e . g . on iPad : } { 13 NSLog ( ” ios : sorry , WC Session not supported” ) 14 } 15 return true 16 } Listing 2.12: On the phone: one single instance of the custom session handler is initiated. In Listing 2.11 a custom handler on the iphone is presented and in Listing 2.12 one single instance of the handler is created on app startup. 1 var connectivityHdler : SongConnectivityHandler? 2 3 override func viewDidLoad () { 4 super .viewDidLoad() 5 self .connectivityHdler = 6(UIApplication.shared.delegate 7 as?AppDelegate)?.connectivityHdl 8 9 // we want to know , if messages 10 // from the watch modified 11 // data in SongConnectivityHandler 12 self .connectivityHdler?.addObserver(self , 13 forKeyPath : ” messagesFromWatch” , 14 o p t i o n s : [ ] , 15 c o n t e x t : nil ) 16 } 17 deinit { 18 self .connectivityHdler?.removeObserver(self ,

32 19 forKeyPath : ” messagesFromWatch” ) 20 } 21 22 // callback , the observed value has chanded 23 override func observeValue ( 24 forKeyPath keyPath: String?, 25 of o b j e c t : Any ? , 26 change: [NSKeyValueChangeKey : Any]?, 27 context: UnsafeMutableRawPointer?) { 28 if let handler = self .connectivityHdler { 29 if keyPath == ” messagesFromWatch” { 30 OperationQueue.main.addOperation { 31 let msgs = handler 32 . messagesFromWatch 33 . j o i n e d ( s e p a r a t o r : ” n” ) \ 34 self .logMessages.text = msgs 35 } 36 } 37 } 38 } Listing 2.13: On the phone: observing an internal value of the custom connection handler to get informed when data arrived from the watch.

1@IBActionfunc sendAnotherSong ( sender : Any) { 2 if let sess = connectivityHdler?.session { 3sess.sendMessage( 4[” newSong” :newSongTextField.text!], 5replyHandler:(response) in { 6NSLog(” ios : we got ( response ) ” ) , \ } 7errorHandler:(error) in { 8NSLog(” ios send err : ( error ) ” ) \ } 9) 10 } 11 } Listing 2.14: On the phone: sending data to the watch.

The watch might send data. The custom connection handler on the iPhone changes its internal values (Note the dynamic keyword). List- ing 2.13 explains how to notice changes in the connection handler using key-value observing. Listing 2.14 shows the code for sending messages to the watch using sendMessage. In this example the data from the watch modifies a variable (data structure) in the session handler. The variable

33 ftewth aet ecngrda hw nFigure in devices. several shown sensor on Figure as rate together work (see heart configured unlocking the be to like to apps aspects have iPhone watch, for the access of as such settings, Privacy 2.5.14 helpful. are the methods if even – the later and placed) – read started and not transmitted send To is be active. app to already background be the must in application watchOS messages the and iOS the both might GUI. which the ViewController on data a modified in the observer) display key-value (using observed is ihteaoeue ehd snMsaeaddidReceiveMessage) and (sendMessage methods used above the With ouetesno ftewatch. the of sensor the use to app an disallow or allow tings 2.15: Figure euiyadPrivacy and Security transferUserInfo euiyset- Security updateApplicationContext (osbydlyd u l esgsaekept) are messages all but delayed, ((possibly iphone screenshot by john feiner 34 2.17 u n a raealong rity. secu- a enhanced create for passcode can one But watch. the unlocking for stan- dard are passcode short a for 2.16: Figure n Figure and odrmsae r re- are messages (older 2.16 ornumbers Four h ac can watch the ) 2.15 Security .

iphone screenshot by john feiner 14 Table in mary at Find specifications notifications. to technical respond to the interactive users In dynamic allow and which notifications Siri added. (grouped) including 5 were watchOS runs capabilities It cellular leased. 4 the watchOS 2018 2017 September 2016 September in In 3.0, 2015. watchOS of April in market to the put was Watch Apple The 2.6 iue2.17: Figure iphones-to-prolong see technology, previously https://ihsmarkit (AMOLED) the Diode to Light-emitting comparison Organic in Active-Matrix consumption used power reduces display Light-emitting (OLED) Organic (LTPO) Diode Oxide Polycrystalline Low-temperature new The pl ac eis2 Series Watch Apple ehia Data Technical 2.6 nokn a oko ac n hn ttesm time. same the at phone and watch on work can Unlocking . com/research-analysis/apple-may-introduce-ltpo-tft-backplanes-for- . . html pl ac eis4 Series Watch Apple . aepofwth a nrdcdaogwith along introduced was watch, waterproof a , https://www 35 ihalre display larger a with . apple

iphone screenshot by john feiner . com/watch n sum- a and 14 a re- was Facet Details 40mm Display LTPO OLED capacitive touchscreen 324x394px 44mm Display LTPO OLED capacitive touchscreen 368x448px Network LTE and UMTS, Bluetooth 5.0 (BTLE), NFC, WiFi 802.11b/g/n 2.4 GHz, GPS Battery up to 18h, Inductive Charging Memory 512 Main Memory, 16 GB Storage Chip System on Chip (SOC) Dual Core, (64 bit) Actuators Speaker, Vibration, Digital Crown with Haptic Feed- back Sensors Force Touch Display, Electrical and Optical Heart Sen- sor, Microphone, Ambient Light, Barometric Altime- ter, Accelerometer, Gyroscope; Fall Detection

2.7 Further Reading

Current research (papers) and some pointers to tutorials can be found in this section.

2.7.1 Introduction Tutorials Building an example WatchKitApp is explained at http://www.techotopia.com/ index.php/Building an Example WatchKit App. See how to use a picker control – and thereby the digital crown – explained in the ar- ticle http://www.sneakycrab.com/blog/2015/6/12/wkinterfacepicker-in- watchkit-20-using-the-digital-crown. Interface navigation in Apple Watch apps: http://www.informit.com/articles/article.aspx?p=2272657

2.7.2 Books Find extensive examples in the book Learning WatchKit Programming – A Hands-On Guide to Creating Apple Watch Applications (Lee, 2015).

2.7.3 Related Research Papers To improve experimenting with Internet of Things (IoT) devices, the Sen- Stick – equipped with more than eight sensors, wireless charging and weighing only 8g – was presented by Nakamura et al. (2016).

36 2.8 Review Questions

A good approach to answering the following questions. Furthermore, ex- plain the advantages AND disadvantages and optionally give source code examples.

2.8.1 Some Basic Questions • Can you explain the structure and the life cycle of watchOS applica- tions?

• What is special about Glances and Complications?

• Which kinds of feedback can be provided by local and remote noti- fications?

2.8.2 Some More Advanced Questions • How do extensions relate to iOS and watchOS applications?

Bibliography

Lee, Wei-Meng (2015). Learning WatchKit Programming: A Hands-On Guide to Creating Apple Watch Applications. Pearson Education. ISBN: 9780134195544. Nakamura, Yugo, Yutaka Arakawa, Takuya Kanehira, and Keiichi Ya- sumoto (Sept. 12, 2016). “SenStick 2: Ultra Tiny All-in-one Sensor with Wireless Charging”. In: Proc. ACM International Joint Confer- ence on Pervasive and Ubiquitous Computing: Adjunct (UbiComp 2016). ACM, pp. 337–340. ISBN: 9781450344623. DOI: 10 . 1145 / 2968219 . 2971399. URL: http : / / doi . acm . org / 10 . 1145 / 2968219 . 2971399.

37 38 3 The Swift Programming Language

“Most of you are familiar with the virtues of a programmer. There are three, of course: laziness, impatience, and hubris.”

[Larry Wall]

Summary: The programming language for iOS was Objective-C, but has changed to a new designed programming language named Swift. Ba- sics concepts and language features, such as functional programming and object-oriented programming, are discussed in this section. Further, more complex aspects such as lazy initialisation, operator overloading, guard statements, generics and function pointers, are discussed in Section 4.

3.1 Introduction

This section on the basics of Swift programming is structured as follows: first the general concepts are explained. Then many examples with short demo code help to build understanding. Finally, we provide some papers and pointers to tutorials.

3.1.1 Programming in Swift We focus on special aspects of the Swift programming language. It is a modern1, open source2, language without tracing garbage collection, but with ARC style garbage collection. Many developers recognise useful

1 The predecessor Objective-C is 10 years older than Java, relies on message passing and has a rather hard-to-read syntax. Objective-C incorporated many features from Smalltalk. 2 Visit https://swift.org to learn how to download, install and run Swift (including the Playground) on Linux.

39 concepts from Python (tuples), Lisp(closures), JavaScript (dictionaries), Haskel (inferred data types, functional), Ruby (closures), C#/Java (gener- ics) and more.

3.1.2 Running Swift Scripts Executable3 Swift scripts can be executed in the terminal ./info.swift as shown in Listing 3.1. The command swift without any arguments opens an interactive Read-eval-print-loop (REPL) in the terminal, see Figure 3.1. 1#!/usr/bin/envswift 2 // 1.) Set execute permissions : chmod +x * . swift 3 // 2.) Run the script in the terminal : ./ info . swift 4 5 import Foundation 6 7 let pi = ProcessInfo () 8print(” ” ” 9 This Swift script ( CommandLine . arguments . first ?? ” ” ) \ 10 is running on a machine 11 with ( pi . activeProcessorCount ) processors .... \ 12 ” ” ” ) Listing 3.1: Run script as command line scripts.

3.1.3 REPL in the Playground The Playground allows live execution of code snippets in a restricted en- vironment. Access to user interface (UI) libraries is possible, as shown in Listing 3.2. With import Foundation one may use the core libraries – basic utility classes and operating system services – for apps, with import UIKit (includes Foundation) users have access to the libraries for iOS develop- ment.

3.1.4 Documentation in Playgrounds Nicely formatted source code with even nicer documentation can be achieved by using Markdown4. In the playground you can work with mul- 3 Execute Swift on macOS or on Linux. For example, use the Docker image https://github.com/apple/swift-docker. 4 For examples of how to add playground documentation in markdown see https://developer.apple.com/library/content/documentation/Xcode/Reference/

40 0print( t ) n i r p medium ) . d t e 20 r l o y h t S : s . . from 19 r e ( e l t y t g t a n S m i r o . r F 18 t e r s t e a t D t . a m r df o 17 F = e t a D ( e t l n 16 y i t = r S e p t a e d l 15 y . t S e df m i t 14 . df 13 12 11 10 9 8 7 6 5 4print(3 2 1 il lsadwt eodt tx n image and (text data demo Listing with and files tiple var var var import // // // // // // iue3.1: Figure xcode use Import string ... s = msg d=Date() UIKit such f=DtFratr() DateFormatter = df ” ” markup 3.2 to The ” Value UIKit View . as find ” UIKit includes interpolation wf a erna ELi h iu rmcSterminal. macOS or Linux the in REPL as run be can Swift good Hello formatting Foundation / Show of other old East for World navigator other ref/ iOS welcome code : . \ ! ” ( demo development Directions fragments libraries ” message // 41 in Oct Xcode . , ” 3, : : east more \ menu 2016, ( msg Resources . rawValue . swift ) ... 11:08 ... ” ” ,a hw in shown as ), ) ) ” PM )

screenshot on macOS by john feiner 21 22 // ... to find resource ” config . json” 23 let url = Bundle.main. url ( 24 f o r R e s o u r c e : ” config” , 25 w i t h E x t e n s i o n : ” json” )! 26 do { 27 let config = try String ( contentsOf : url ) 28 p r i n t ( c o n f i g ) // version : 3 { } 29 catch } { 30 p r i n t ( ” Resource not found” ) 31 } Listing 3.2: Live compiling and execution of swift code in the Playground.

3.2 The Swift Language

When taking a look (see Listing 3.3) at the available keywords5, you will find some unconventional ones, such as deinit, protocol, willSet or un- owned. 1 // Declarations : 2 class deinit enum extension func import init let 3 protocol static struct subscript typealias var 4 5 // Statements : 6 break case continue default do else fallthrough 7 if in for return switch where while 8 9 // Expressions and Types : 10 as dynamicType is new super self 11 12 // debug symbols / identifiers 13 #column #file #function #line 14 15 // Reserved in particular contexts 16 // ( e . g . for operator overloading , observers , 17 // handling references ) : 18 precedence associativity didSet willSet infix inout 19 mutating nonmutating none operator left right 20 override postfix prefix unowned unowned ( safe )

5 Summary of the grammar https://developer.apple.com/library/ios/documentation/ Swift/Conceptual/Swift Programming Language/zzSummaryOfTheGrammar.html#/ /apple ref/doc/uid/TP40014097-CH38-ID458.

42 21 unowned ( unsafe ) set get 22 23 // Access Level : 24 private public internal Listing 3.3: A list of Swift keywords.

3.2.1 Coding Style Guides and Documentation It is essential that other developers can read your code easily. Please follow the coding conventions and style guides6. Learn from others and read their code! In Xcode you can get help and information about classes and meth- ods by using ALT-click on variables. Even better, with CMD-click you can jump directly to the source code. The code documentation in Mark- down works for your own code, as described in http://nshipster.com/swift- documentation/, for example with -param or -returns: annotations.

3.2.2 Type Inference Firstly, Swift is a modern static typed7 language, in which the type of a variable is defined by type inference; see Listing 3.4. Types with a question mark ? attached are optional types, meaning they can be nil. A very special type called AnyObject? can contain a reference to any type or even nil. 1 // Type inference : 2 // vars have type of ” detected” data type : 3 var age=42, height =0.0 ; let haircolor=UIColor. blue 4 let pi=3.1415926 5 let (x,y) = (1030,320) 6 let largeFloatingPointNo :Double = 4 533.0000000 2 7 8 // type casting 9 let descriptionText = ” Your age is ” +String(age) 10 11 // optional types can be nil 12 var dbConnectionUrlOrNil : String? = nil

6 Find the Swift style guide at GitHub https://github.com/github/swift-style-guide and the Apple guidelines for API design at https://swift.org/documentation/api-design-guidelines/. 7 The strict typing system makes the language safer than many other programming languages, as it allows the compiler to already detect errors at compile time preventing several categories of bugs.

43 13 14 // dictionaries 15 var quotes = [ ” Richard III” : ” Off with his head ! ” , 16 ” Timon of Athens” : ”We have seen better days” ] Listing 3.4: Type inference means auto-detection of the type at first assignment during variable definition.

3.2.3 Optionals With optionals it is possible to do optional chaining, which means that chained methods are only called if the variable is not nil. Find an example in Listing 3.5. 1 // optional chaining 2 class Book var titlePage :TitlePage? { } 3 class TitlePage var text=”Don ’ t Panic ! ” { } 4 let guideToTheGalaxy = Book() ; 5guideToTheGalaxy.titlePage=TitlePage() 6 if let txt = guideToTheGalaxy. titlePage ?. text { 7print(”On the tile is written ’ ( txt ) ’.” ) \ 8 else } { 9print(” There is no title page for this book . ” ) 10 } Listing 3.5: Optional chaining spares the check if an object is nil before working with this object.

3.2.4 Functions and Parameters Functions (see Listings 3.6, 3.7 and 3.8) can handle default and named pa- rameters. As parameters have an internal name and optionally, a different external name (the latter often used for assigning speaking names to pa- rameters), the underscore simplifies usage by not requiring the use of the name for a parameter when calling the function. Values are passed by value8 by default, but by marking parameters as inout, references can be passed along. With inout a value is copied into the

8 Only objects created out of classes are reference types (structs are value types), which means that the reference is the value passed, so the contents of an object within a function can still be modified.

44 function or method and copied back before leaving the function or method (i.e.: copy in – copy out). Variable argument lists are possible as well as multiple return values. Even return values can have named parameters. 1 func power ( b: Int , // named 2exp:Int=2,// named and default 3msg:String=” Result : ” ) 4 > (powRes:Int , 5powMsg:String) { 6 var r=1 7 // unused variable , so better use underscore 8 for in 1...exp { 9r*=b 10 } 11 // returning a tuple 12 return (r ,” ( msg ) ( r ) ” ) \ \ 13 } 14 // accept multiple return values from a function 15 var (result , prettyMessage) = power(b: 4) 16 // usage of named return values . Here : ”powMsg” 17 let res = power(b:6 ,exp:3 ,msg:” The final result is” ) 18 p r i n t ( r e s . powMsg ) Listing 3.6: Functions allow default and named parameters even for return values. Note the order of the parameter must not be changed.

1 func swapTwoFloats ( a: inout Float , 2 b: inout Float ) { 3 let tmpA = a 4a=b 5b=tmpA 6 } 7 var notPi : Float = 3.141599 8 var anotherFloat = Float (107) 9swapTwoFloats(¬Pi,&anotherFloat) 10 p r i n t ( ”now notPi is ( notPi ) ” ) \ Listing 3.7: Using inout parameters allows side effects, as the reference is passed. Note that the underscore allows us to leave away the otherwise required argument labels for parameters when calling a function.

1 func sumUp ( numbers : Double . . . ) >Double { 2 var sum=0.0 3 for e in numbers sum+=e { } 4 return sum

45 5 } 6print(sumUp(3.4,5.3,2.9,1, 4) ) Listing 3.8: Functions might have variable number of parameters.

3.2.5 Closures Functions without a name (function pointers) are called closures (they are stored including the current environment/scope and executed later) and are defined in place, for example to modify each element of a list in a short and convenient way (see Listing 3.9). Instead of naming arguments, $0, $1 and so on can be used. 1 // closures : 2 // ( params ) > returnType in { 3 // statements 4 // } 5 let authors = [” Jelinek” , ” Joyce” , ” Haas” ] 6 let welcomeTexts = authors .map( { 7(author:String)> String in 8 ”Welcome , ( author ) ! ” \ 9 ) } 10 print(welcomeTexts) // [”Welcome , Jelinek !” , ... 11 // closure with inferred types 12 [ 5 , 6 ] . s o r t e d ( by : s1 , s2 in return s1>s2 ) // [6 ,5] { } 13 // closure / function with implicit parameter $0 14 let c= $0 3 { * } 15 c ( 4 ) // 12 Listing 3.9: Closures are used as a short way to define a function.

3.3 Data Types: Tuples, Dictionaries, Structs and Enums

Typical data types such as Int are implemented as structs9. In addition to tuples and dictionaries, advanced structures and enumerations can be defined; see Listing 3.10.

9 When navigating with CMD-click on Int to its implementation, the original data type and implemented protocols are revealed. For who are interested, see also http://ankit.im/swift/2016/01/12/swift-mysterious-builtin-module/. To go deeper into this, transparent annotations for performance optimisation can be found in, for example, the source of Bool https://github.com/apple/swift/blob/master/stdlib/public/core/Bool.swift.

46 1 // Advanced Datatypes : Dictionaries 2 var shakespeare = [ 3 ” Richard III” : ” Off with his head ! ” , 4 ” Timon of Athens” : ”We have seen better days” ] 5 // V1 ) providing an optional value with operator : ?? 6print(shakespeare[” Richard III” ] 7??” Sorry , given key not found” ) 8 // ” Off with his head !” 9 10 s h a k e s p e a r e [ ” Richard VII” ]=” Did he exist? ” 11 // V2 ) force unwrap / cast optional value with ”!” 12 p r i n t ( s h a k e s p e a r e [ ” Richard VII” ]! ) 13 // ” Did he exist?” 14 15 // Custom Datatypes : enumerations 16 enum Emoticons { 17 case Smile , Twinkle , Sad 18 func getSmily () > String { 19 switch self { 20 case .Smile: return ” :)” 21 case .Twinkle: return ” ;)” 22 case .Sad: return ” :(” 23 } 24 } 25 } 26 print( Emoticons.Smile.getSmily() ) // :) Listing 3.10: Examples of definition and usage of lists tuples dictionaries structs and Enums.

3.3.1 Map and Reduce The functional style of programming is supported by filter, map, reduce and chaining functionality. Listing 3.11 provides an example of modifying list elements. 1 // Map and Reduce 2 let numbers = [101,102,111,112,121,122,131,132] 3 4 let sum = numbers.reduce(0) $0 + $1 { } 5print(sum) 6 7 let evenNumbers = numbers . f i l t e r $0 % 2 == 0 { }

47 8print(evenNumbers) 9 10 var msges = evenNumbers .map ” ( $0 ) is even” { \ } 11 print(msges.joined(separator: ” ; ” )) // 102 is even;112 ... Listing 3.11: Map and reduce support a functional programming style.

3.4 Object-Oriented Programming (OOP)

The visibility of classes can be set with the different protection levels of private, fileprivate, internal (default visibility to the module10), public or open. Classes hold properties and set the visibility, the privacy, via public (access for everyone), internal access (default, within defining module) or private (access restricted to defining source file) modifiers. The class modifier declares a method as a class method. Properties have getters and setters11, as can presented in Listing 3.1212. 1 // OOP Classes and Instances 2 public class BaseMark { 3 // child classes have to OVERRIDE init () !!!! 4 5 var exam : String = ” Algorithms” 6 7 var description : String { 8 return ” The mark for ( exam ) is ( value ) . ” \ \ 9 } 10 class func about () > String { 11 return ” This is class BaseMark” 12 } 13 // Property with a public getter , 14 // but a private (= within source file ) setter : 15 public private ( set ) var value = 2 16 17 // read only computed property 18 public var prettyValue : String { 19 return ” Your mark in ( exam ) is ( value ) . ” \ \ 20 } 10 The module is a single unit of code distribution (built target), i.e. the current application or framework 11 Note, that setters are not allowed to throw errors. Compare Swift proposal at https://forums.swift.org/t/proposal-allow-getters-and-setters-to-throw/191. 12 The given example uses the shorthand newValue in the setter.

48 21 // setter with implicit variable ” newValue” 22 public var examination : String { 23 get return ”Exam in ( exam ) ” { \ } 24 set exam=newValue { } 25 } 26 } 27 print(BaseMark.about()) // This is class BaseMark 28 var m= B a s e M a r k ( ) 29 p r i n t (m. exam ) // Algorithms 30 m. e x a m i n a t i o n = ” Linux” 31 print(m.prettyValue) // Your mark in Linux is 2. Listing 3.12: Object-oriented programming in Swift.

3.4.1 Inheritance, Polymorphism Classes may define some designated initialisers and optionally many con- venience initialisers13. The convenience initialisers will call a designated initialiser with self.init(...). Single inheritance is supported (see List- ing 3.13) and redefined methods have to be marked with the override key- word. Some and differences from other programming languages: a) there is no global base class that all other classes inherit from; b) a class you define will be a base class and will get a default initialiser; c) for any class you derive from a base class, a designated initialiser of the base class must be called14. 1 // OOP inheritance 2 // with designate and convenience initialisers 3 public class Mark : BaseMark { 4 override public init () { 5 super . init () 6 self .exam=” German Course” 7 } 8 // convenience initialiser 9 public convenience init (nameOfTheExame:String) { 10 self . init ()

13 Find a description of the exact use of init, super and setting variables in initialisation at http://ashfurrow.com/blog/swift-initializers/. A special case is failable initializers which return possible nil. 14 Find different ways of calling initialisers at https://developer.apple.com/library/ content/documentation/Swift/Conceptual/Swift Programming Language/ Initialization.html#//apple ref/doc/uid/TP40014097-CH18-ID216.

49 11 self .exam=e 12 } 13 override var description : String { 14 return ” This is the mark for exam ( self . exam ) . ” \ 15 } 16 } 17 var m2=Mark ( ) 18 print(m2.description) // The mark for Algorithms is 2. Listing 3.13: Single inheritance in Swift.

3.4.2 Extensions With Extensions – see Listing 3.14 – you can add functionality to existing classes. It is not possible to override existing functionality. 1 // OOP Extending Classes 2 extension Mark { 3 func tryAgain () > String { 4 return ” Try again to pass exam ( self . exam ) ” \ 5 } 6 } 7 var m3=Mark ( ) 8print(m3.tryAgain()) Listing 3.14: Extensions to any – even built-in – class.

3.4.3 Protocols By defining a Protocol as shown in Listing 3.15, the obligatory methods for a class are specified. 1 // OOP Protocols 2 protocol Translation { 3 var getGermanDescriptionIn : String get { } 4 } 5 public class InternationalMark :BaseMark, Translation { 6 // ... 7 var getGermanDescriptionIn : String { 8 return ”Unknown german description for ’ ( exam ) ’.” \ 9 } 10 } 11 var m4=InternationalMark ()

50 Listing 3.15: Protocols guaranteed that classes provide the expected functionality.

3.4.4 Protocol Extensions Similar to mix-ins in other languages, it is possible to add functionality to classes by extending the protocol. See Listing 3.16. 1 extension Translation { 2 func printLanguages () { 3print(”We provide in D: ” ) 4print(self .getGermanDescriptionIn ) 5 } 6 } 7m4.printLanguages() // Prints : We provide ... Listing 3.16: Protocol extensions allow to add functionality to protocols.

3.5 Further Hints

Basic string interpolation might be not sufficient for advanced formatting. Listing 3.17 gives tips for sting formatting according the IEEE printf spec- ification15. 1 var piStr = NSString( 2format:” pi approx %.4 f ” ,Double.pi) 3 // outputs : pi approx 3.1416 Listing 3.17: Selected tips for coding in Swift: Formatted strings.

3.6 Completely Unnecessary Knowledge

Parts of Swift code are generated by the Python tool Generate Your Boil- erplate (GYB16).

15 http://pubs.opengroup.org/onlinepubs/009695399/functions/printf.html 16 https://github.com/apple/swift/blob/master/utils/gyb.py

51 3.7 Required Reading

The official Apple documentation for The Swift Programming Lan- guage can be found at https://developer.apple.com/library/content/ documentation/Swift/Conceptual/Swift Programming Language/index.html.

3.8 Further Reading

This section contains some current research (papers) as well as pointers to tutorials. For example, find upcoming features at the Swift Program- ming Language Evolution at https://github.com/apple/swift-evolution and various Swift projects (Compiler, Standard Library, Package Manager) at https://swift.org/about/.

3.8.1 Introduction Tutorials A Swift learning tutorial can be found at Wikibooks: https:// en.wikibooks.org/wiki/Swift learning. A beginner’s tutorial is available at http://www.raywenderlich.com/115253/swift-2-tutorial-a-quick-start.A functional way of coding is introduced at http://www.raywenderlich.com/ 82599/swift-functional-programming-tutorial.

3.8.2 Research Papers The strengths and weaknesses of teaching Swift at the university are dis- cussed in Rogers and Siever (2015). For a possible approach to breaking the login see Maiti et al. (2015).

3.8.3 About Programming Language Swift Chris Lattner explains the advantages of the LLVM over GNU Compiler Collection (GCC) (fast compilation, expressivity of compiler error mes- sages) and the Swift programming language over various other Languages (C#, Groovy, Rust) in an Slashdot interviewLattner (2018). He mentions also the missing (async/await) and upcoming (co-routines, dynamic reflec- tion) Swift 5 features.

52 3.9 Tasks

Try to solve the following demo tasks to gather experience in coding Swift:

• Take some available open data (e.g. http://data.wien.gv.at/daten/ geo?service=WFS&request=GetFeature&version=1.1.0&typeName= ogdwien:HYDRANTOGD&srsName=EPSG:4326&outputFormat= json) and map the information into a collection of objects. Consider using classes with extensions.

• Filter the data set to find out selected data of interest (e.g. how many hydrants are available in Vienna; which one is the most northern one). Consider using functional programming with map, filter, re- duce.

• Do a calculation using the data (e.g: find the loneliest hydrant in Vienna).

3.10 Review Questions

A good approach to answering the following questions. Furthermore, ex- plain the advantages AND disadvantages and optionally give source code examples.

3.10.1 Some Basic Questions • What do we mean by type inference?

• What is special about optionals and optional chaining?

3.10.2 Some More Advanced Questions • Can you explain and give examples of weak and strong references?

53 Bibliography

Lattner, Chris (Jan. 10, 2018). Slashdot’s Interview With Swift Creator Chris Lattner. (Visited on 01/23/2017). Maiti, Anindya, Murtuza Jadliwala, Jibo He, and Igor Bilogrevic (2015). “(Smart)Watch Your Taps: Side-channel Keystroke Inference Attacks Using Smartwatches”. In: Proc. 2015 ACM International Sympo- sium on Wearable Computers (ISWC 2015). ACM, pp. 27–30. ISBN: 9781450335782. DOI: 10.1145/2802083.2808397. Rogers, Michael P. and Bill Siever (May 2015). “Switching to Swift: In- structional Issues and Student Sentiment”. In: J. Comput. Sci. Coll. 30.5, pp. 144–150.

54 4 Advanced Topics for the Swift Programming Language

“Beware of bugs in the above code; I have only proved it cor- rect, not tried it.”

[Donald E. Knuth]

Summary: Swift contains many features which are inspired by or are adopted from other programming languages. Basic language features have been covered in Section 3. Advanced concepts and language features are discussed in this section.

4.1 Introduction

This section on advanced Swift programming topics is structured as fol- lows: first we explain the advanced topics (ARC/LLVM, strong/weak ref- erences) and language concepts such as lazy variables, computed proper- ties, property observers, operator overloading, guard statements, function pointers, nested functions, mutating functions and generics of Swift. Then we will look at many examples with short demo code to develop your un- derstanding. Finally, we list some papers and pointers to tutorials. Note: if you would like to run Swift code on Linux, get started quickly with Docker1 containers or take a look at web-based2 compile services.

1 Find a Docker image for Swift 4 on Ubuntu https://github.com/swiftdocker/docker-swift. 2 You might consider trying out the IBM Swift Sandbox online at https://swiftlang.ng.bluemix.net/#/repl.

55 4.2 Advanced Programming in Swift

This section focuses on the tricky details of special programming tech- niques.

4.2.1 ARC – Automatic Reference Counting Swift – in the same way as Objective-C – uses automatic garbage collec- tion. This is possible due to an automatically linked runtime, which is therefore included in every app. The way memory is reserved and freed is called ARC, where during definition of a variable, memory is acquired and a counter is set to one. For every additional strong reference to these memory cells, a retain (an increase of the counter) takes place. On release of a variable the counter is reduced. When reaching zero, no reference to this part of the memory is necessary, so the objects can be disposed of. Note that the com- piler automatically adds the necessary retain and release statements to the code. Some attributes might be expensive to create for objects; for example, an attribute might be a new further complex object as shown in Listing 4.1. In this case, it might make sense to tag the attribute lazy3 for deferred creation on demand. Note the use of [unowned self] to mark the reference inside the block as not owned4 by the block. 1 // Lazy properties 2 // Further note : 3 // by using self the closure would capture 4 // a strong reference to its own object 5 // so we need [ unowned self ] 6 7 class Book { 8 9 var author : String 10 11 lazy var titlePage : String = { 12 [ unowned self ] in

3 Find extensive examples at http://mikebuss.com/2014/06/22/lazy-initialization-swift/. 4 In this case we definitely do not want a strong reference which would possibly cause a memory leak by creating a strong reference cycle inhibiting the deallocation. We could have used weak instead of unowned, but the difference between unowned and weak is that weak is used for optionals and could therefore be nil.

56 13 return ” This is a book by ( self . author ) ! ” \ 14 () } 15 16 init (author: String) { 17 self .author = author 18 } 19 } 20 21 let p=Book( author : ” Maximilia” ) 22 p r i n t ( p . a u t h o r ) // Maximilia 23 24 // only when using the attribute the codeblock 25 // after ’ lazy var xxx = ’ ... gets executed { } 26 p . t i t l e P a g e // This is a book by Maximilia ! Listing 4.1: Lazy initialisation defers the creation of properties to it’s first usage.

4.2.2 LLVM and LLDB – Compiler and Debugger The Xcode Integrated Development Environment (IDE) uses the Low Level Virtual Machine (LLVM, see also 4.10) to compile the sources and to build the application. The debugger inserts meta-information into the code to allow live debugging in the simulators and on real devices.

4.3 OOP – Advanced Object-Oriented Programming

Swift provides some features, which are not available in many other lan- guages, such as lazy properties, property observers, operator overloading, guards, lazy properties or generics.

4.3.1 Stored and Computed Properties, Property Observers Different to normal stored properties the computed properties are just get- ter and setters to and do not store values themselves. the hooks which are used in classes to get informed if special object methods are invoked are called property observers. For example, the property observers for getters and setters report the access or change of attributes, as shown in Listing 4.2. 1 // Property Observers 2 public class Window {

57 3 static let maxVal : Int =99 4 // Attributes 5 public var xPos : Int=0 { 6 willSet (newValue) { 7print(”We try to set attribute ’ xPos ’ ” , 8 ” from current value ( self . xPos ) ” , \ 9 ” to new value ( newValue ) ” ) \ 10 } 11 didSet (oldValue) { 12 p r i n t ( ”We set attribute xPos to ( xPos ) ” , \ 13 ” ( from old value : ( oldValue ) ) ” ) \ 14 if xPos > Window . maxVal { 15 xPos = Window . maxVal 16 } 17 } 18 } 19 } 20 let helpAndDocu=Window() 21 helpAndDocu.xPos = 114 // watch the log : LOG : we try to set .. 22 p r i n t ( ”We get capped value of : ( helpAndDocu . xPos ) ” ) // 99 \ Listing 4.2: Property observers trigger a callback when properties are read or modified.

4.3.2 Operator-Overloading An advanced feature of a programming language is the possibility of (re- )defining the use of prefix (for example –i), postfix (for example i++) and infix (for example: a+b ) operators with so-called operator func- tions. Additionally, the precedence and the associativity can be specified. When accessing elements of an object via an index (for example rgbCol- orCodes[”red”]) the subscript functionality has to be defined. some ex- amples are given in Listing 4.3. 1 class Quote : CustomStringConvertible { 2 var txt = ” ” 3 var by = ” ” 4 var description : String return ” ( txt ) by ( by ) ” { \ \ } 5 6 init ( txt :String , byAuthor : String ) { 7 self .txt=txt 8by=byAuthor

58 9 } 10 // infix operator overloading 11 static func +(left :Quote,right :Quote) > Quote { 12 return Quote ( left .txt + ” ” + right .txt, 13 left .by+” and ” + right .by) 14 } 15 } 16 extension Quote { 17 // compound assignment operator += 18 static func += ( left : inout Quote , right :Quote) { 19 left .txt += ” ” + right .txt 20 left .by += ” and ” + right .by 21 } 22 } 23 let pablo = Quote( 24 [ ” Every child is an artist . ” , 25 ” The problem is how to remain” , 26 ” an artist once he grows up . ” ]. joined(separator: ” ” ), 27 ” Pablo Picasso” ) 28 let edsger = 29 Quote ( ” Simplicity is prerequisite for reliability . ” , 30 ” Edsger W. Dijkstra” ) 31 32 let combiquote = pablo + edsger 33 p r i n t ( combiquote ) // Every child is an artist ... Listing 4.3: Operator overloading assigns new meaning to given operators.

4.4 Asserts, Exceptions and Preconditions

For higher quality in software projects, it is recommended that the input of functions Listing 4.7 and methods be checked at runtime. Errors are caught with the do, try and catch keywords as can be seen in Listing 4.5 and Listing 4.6. During development, you might consider using assert statements to crash the app; for example, if the incoming values for the parameters are not in the expected range (see Listing 4.4). 1 // FOR DEVELOPMENT ONLY, app might CRASH 2 class Game { 3 var name = ” unknown” 4 init ( n:String) self .name=n { } 5 } 6 class GameCollection {

59 7 var games : [Game] 8 init ( someGames :Game . . . ) { 9 self .games=someGames 10 } 11 subscript (index: Int) > Game { 12 // use assertion to check pre conditions 13 // if false => app is terminated ( crash ) 14 a s s e r t ( in d e x >=0,”No negative indices ! ” ) 15 return self .games[index] 16 } 17 } 18 let classicGames = 19 GameCollection( Game( ” Chess” ), Game(”Go” )) 20 21 // a bad way to code loop over elements : 22 for idx in 2...2 { 23 // CRASH : assertion failed : No negative indices ! 24 print( classicGames[idx] ) 25 } Listing 4.4: Asserting the input value forces the app to crash rather than to go on in an undefined state.

1 let urlString = ” http :// perl . com” 2 if let url = URL(string: urlString) { 3 do // compare : try catch { 4 let html = try 5String(contentsOf:url) 6print(html)// ”

1 enum Person : Error { 2 case DoesNotExist , IsAbroad 3 } 4 let team = [”Mia” , ” Leo” , ” Ina” ] 5 func positionOf (person p: String ) throws > Int { 6 if !team.contains(p) {

60 7throwPerson.DoesNotExist 8 } 9 return team . index ( of :p) ! 10 } 11 do { 12 let idxOfIna = try positionOf ( person :” Ina” ) 13 p r i n t ( i d x O f I n a ) // 2 14 let idxOfEva = try positionOf ( person : ” Eva” ) 15 p r i n t ( idxOfEva ) // exception 16 catch let e } { 17 p r i n t ( e ) // Error : Does not Exist 18 } Listing 4.6: Custom exception conform to the Error protocol.

1 // ensure some prerequisites are fulfilled 2 // before executing the logic 3 // in the body of a function 4 func outputScaleable (imageName img: String ) { 5guardimg.hasSuffix(” . svg” ) else { 6 return 7 } 8print(” ( img ) ’.” ) // flower . svg \ 9 } 10 outputScaleable(imageName: ” flower . jpg” ) 11 outputScaleable(imageName: ” flower . svg” ) 12 outputScaleable(imageName: ” flower . png” ) Listing 4.7: To check the value of arguments, the guard statement is used.

4.5 Nested Functions and Function Pointers

In Listing 4.8 you can see nested functions and the use of function pointers. Working with references to functions is similar to working with normal variables. 1 import CoreFoundation 2 // Nested functions and function pointers : 3 func getMeTheQuotesFunc (whichOne : String ) 4 > (() > (String) ) { 5 6 func randomQuoteFunction () > String { 7 let quotes=[” Quote A” , ” Quote B” , ” Quote C” ] 8 let randomNum : UInt32

61 9=arc4randomuniform ( UInt32( quotes . count ) ) 10 let someInt = Int (randomNum) 11 return quotes [someInt] 12 } 13 14 switch whichOne { 15 case ” random” : 16 return randomQuoteFunction 17 default : 18 return { 19 ( ) >String in 20 return ” ’ ( whichOne ) ’ not implemented yet” \ } 21 } 22 } 23 24 var getRandomQuote = getMeTheQuotesFunc (whichOne : ” random” ) 25 p r i n t ( ” The result of the getRandomQuote () : ” ) 26 print( getRandomQuote() ) // e . g .: Quote C Listing 4.8: Return the reference to a function and invoke the function via this function pointer.

4.6 Generics

To allow different types of parameters and return values, it is a well- established approach not to duplicate code, but rather use generic types. In Listing 4.9 you can see examples of functions accepting several differ- ent – even custom – types of parameters. 1 func swappingAnything( left : inout T, 2 right : inout T) { 3 let tmp = left 4 left= right 5 right=tmp 6 } 7 8 var a=”we ” ,b=” try ” ,c=” generics” 9 10 p r i n t ( a+b+c + ” ! ” ) 11 swappingAnything(&a, &b) // we try generics ! 12 p r i n t ( a+b+c + ” successfully? ” ) // try we generics ... Listing 4.9: With generics code duplication is avoided and strong typing is still enforced.

62 4.7 Structs and Mutating Functions

Structures are not dynamic and even the internal variables cannot be changed. Only by declaring a function as mutating are changes to inter- nals allowed. 1 // Struct need mutating functions : 2 // without mutating the struct would not be 3 // allowed to change its internals 4 5 struct IntStack { 6 var items = [ Int ]() 7 mutating func push ( item : Int ) { 8items.append(item) 9 } 10 mutating func pop () > Int { 11 return items . removeLast () 12 } 13 func delLastItem () { 14 // Note : changing inner values with 15 // items . pop () would NOT BE ALLOWED 16 } 17 } 18 var s=IntStack() 19 s . push ( 4 ) 20 s . push ( 4 4 ) 21 p r i n t ( s . i t e m s ) // [4 , 44] Listing 4.10: Changing the internals of a struct needs mutating functions.

4.8 Design Patterns in Swift

Design patterns are categorised into the creational (such as builder, factory method, singleton, see Listing 4.11) to create objects, structural (such as bridge, facade, proxy) for object composition, and behavioural (such as strategy, command, observer) for communication between types. 1 // Singletons : 2 class SoundPlayer : CustomStringConvertible { 3 private init () {} 4 static let sharedInstance = SoundPlayer () 5 var description : String { 6 return ” The SINGLE Sound Player” 7 }

63 8 } 9 let sp = SoundPlayer.sharedInstance 10 p r i n t ( sp ) // prints : The SINGLE Sound Player Listing 4.11: The – for too often misused – singleton design pattern. Better avoid global objects.

4.9 Limitations of Swift

Reflection is limited in Swift. A Mirror object, see Listing 4.12, repre- sents the structure of instances of any type and can report properties at runtime. Computed properties are not supported. No modifications to ex- isting classes can be done at runtime. 1 class Book { 2 var noOfPages=7 3 } 4 var b=Book () 5print(” Type : ( type ( of : b ) ) ” ) // Type : Book \ 6 7 // Mirror provides read only 8 // Introspection / Reflection : 9 let m= Mirror(reflecting: b) 10 p r i n t ( ” Type : (m. subjectType ) ” ) // Type : Book \ 11 12 for case let (label?, value) in m. children { 13 p r i n t ( l a b e l , v a l u e ) // noOfPages 7 14 } Listing 4.12: Reflection (introspection) using a Mirror structure.

4.10 The Future of Swift

A main focus of the next version 5 of Swift is Application Binary In- terface (ABI) stability5, i.e. a compiled version can run – and commu- nicate with other programs or libraries – across several version of the iOS (whereby Swift gets shipped with the operating system). Further- more Apple improves projects such as the Swift Package Manager6 or the

5 https://swift.org/abi-stability/. 6 Currently, Swift Package Manager is macOS only https://swift.org/package-manager/.

64 Language Server Protocol 7 which provides support during development, e.g. code completion, of Swift applications in other editors, for example, VS Code8.

4.11 Required Reading

Check out the section about ARC in the documentation by Apple at https://developer.apple.com/library/ios/documentation/Swift/Conceptual/ Swift Programming Language/AutomaticReferenceCounting.html. It ex- plains the differences between strong, weak, and unowned references visually and gives further examples of references to unowned self and weak delegate variables used inside closures.

4.12 Further Reading

Find further tutorials, background information and some current research papers in this section.

4.12.1 Tutorials An extensive demo using generics for a linked list can be found at http: //waynewbishop.com/swift/generics/.

4.12.2 Background Information Find aspects of Swift compiling with the LLVM infrastructure at https:// www.accelebrate.com/blog/thinking-swift-part-ii/. Optionally, you might take a look at the LLVM presentation http://llvm.org/pubs/2008-10-04- ACAT-LLVM-Intro.pdf (including a comparison of gcc and clang) or at the official LLVM home page http://llvm.org.

4.12.3 Research Papers Several aspects of sandboxing are discussed in Bucicoiu et al. (2015). Ways to optimise the performance of the Swift programming language by

7 https://github.com/apple/sourcekit-lsp. 8 https://github.com/RLovelett/langserver-swift.

65 reducing the atomic operation of reference counting is explained in Ungar, Grove, and Franke, 2017.

4.13 Tasks

Try to solve the following demo tasks to gain experience with the advanced concepts of the Swift programming language:

• Perform operations on objects (e.g. countries) holding large data (a 300 MB image each) where lazy evaluation is necessary for perfor- mance (consider using lazy vars).

• Define (small set of) Application Programming Interfaces () (e.g. getMostBeautifulLocationOf, distanceToCapitalOf (consider using protocols).

• Invent useful operations on the data (e.g.: distance between coun- tries, GPS location of a city in the country). Consider operator over- loading or subscripts.

4.14 Review Questions

A good approach answering the following questions is to try to define and explain a given term in your own words, to state typical usage and do- mains (this is a solution for which problem). Then try to state possible alternatives for the technology and/or for this approach. Furthermore, ex- plain the advantages AND disadvantages and optionally give source code examples.

4.14.1 Some Basic Questions

• Can you explain the use, advantages and disadvantages of advanced OOP with protocols and extensions?

• What is special about property observers?

66 4.14.2 Some More Advanced Questions • Can you explain the structure of the Low Level Virtual Machine In- frastructure (LLVM) and the tasks done by the single components?

• What are the (practical) differences of classic vs. ARC garbage col- lection implementations?

Bibliography

Bucicoiu, Mihai, Lucas Davi, Razvan Deaconescu, and Ahmad-Reza Sadeghi (2015). “XiOS: Extended Application Sandboxing on iOS”. In: Proc. 10th ACM Symposium on Information, Computer and Com- munications Security (ASIA CCS 2015). ACM, pp. 43–54. ISBN: 9781450332453. DOI: 10.1145/2714576.2714629. Ungar, David, David Grove, and Hubertus Franke (Oct. 24, 2017). “Dy- namic Atomicity: Optimizing Swift Memory Management”. In: Proc. 13th ACM SIGPLAN International Symposium on on Dynamic Lan- guages (DLS 2017). ACM, pp. 15–26. ISBN: 9781450355261. DOI: 10.1145/3133841.3133843.

67 68 5 Coding iOS for iPhone and iPad

“Every child is an artist. The problem is how to remain an artist once he grows up.”

[Pablo Picasso]

Summary: For applications running on iPhone and iPad developers need to know how user interaction and navigation flow work. Both, coding in Swift and understanding how to design and layout a graphical user in- terface (GUI) are necessary. The development relies on usage of many iOS Frameworks and, of course, the sound handling of the development tools provided.

5.1 Introduction

This is the iOS introduction section for iPhone and iPad programming. The topic includes basics about designing for flexible layout and the dif- ferent options for navigation flow and interaction(-design). The connec- tion between and interaction of user interfaces and code are necessary to achieve programmatic reactions to user input that respond with output on the graphical interface. Many screenshots and short code examples help with understanding. Finally, we point to some papers and tutorials.

5.2 Coding for iOS

Development for iOS is similar for iPhone and iPad, so the logic should and will be the same for both. The main difference is the UI layout for different screen sizes, which can either be done separately – by means of separate storyboards – for iPhone and iPad, or in a unified way, called universal app.

69 5.3 General App Design

The same basic concept for watchOS design holds for iOS design; always design for and with the target users. User Centred Design (UCD) helps to solve problems for people, in contrast to just implementing what might be interesting or fun for the developer! After modelling personas, doing paper prototyping and initial evaluations by real users, developers can optimise the overall flow and interaction early on. When finally starting to imple- ment the app, the Apple Human Interface Guides1 are the first materials to work through.

5.3.1 Performance Users expect a lot from mobile applications. One important point is that smartphone apps are used in a burst-like manner, which means short time spans of usage. For example, a user plays a game and gets interrupted by an incoming call. This means that every app must be able to save state at any time, so as not to loose information, and must stick to content first in order to recover fast and continue to work seamlessly. Note that you will rarely find a save button.

5.3.2 Energy Consumption If an app uses too many resources, it will drain the battery fast; especially GPS is known to consume a great deal of energy. Developers must save power whenever possible!

5.4 App Life Cycle

Before an application starts up, it is in the state not running. If an app is (very briefly) in an inactive state, it is in the foreground, but not re- ceiving events. Finally, the app changes into the normal state active.A special state is the background state – where the app is executing code – for a rather short fixed period of time before the app is in suspended mode, where the app is still in memory, but not executing code. The transitions

1 Learn about clarity, deference and depth at https://developer.apple.com/ios/human-interface-guidelines/.

70 – see Figure 5.1 (figure based on app life cycle explanations by Apple2)– between those states trigger events to life cycle methods3 such as willFin- ishLaunchingWithOptions or applicationWillResignActive method of class AppDelegate in file AppDelegate.swift. Note that the class AppDelegate is annotated with @UIApplicationMain to be the main entry point after application launch.

Not running

Suspended

Inactive

Background

Active

Background

Foreground omnigraffle drawing by john feiner

Figure 5.1: Transitions trigger events and custom code in the delegate can be executed for those app life cycle methods.

5.5 Unit Tests

Adding unit tests before writing code is good practice in the spirit of Test-driven Development (TDD). In Listing 5.1, the Swift file in the iOS Unit Testing Bundle – which can be added through the File/New/Tar- get template-wizard – includes the class under test with the annotation @testable. Specifying selected blocks as measureBlock allows detailed performance analysis. Asynchronous4 testing can be done, as shown in Listing 5.2, with so-called expectations: inside the test case the code blocks

2 Compare app life cycle at https://developer.apple.com/library/ios/documentation/iPhone/Conceptual/ iPhoneOSProgrammingGuide/TheAppLifeCycle/TheAppLifeCycle.html. 3 Find a detailed description at https://developer.apple.com/library/ios/documentation/ iPhone/Conceptual/iPhoneOSProgrammingGuide/TheAppLifeCycle/ TheAppLifeCycle.html#//apple ref/doc/uid/TP40007072-CH2-SW3. 4 and the subject of queues and priorities of queues will be discussed in detail in Section 6.3, Dispatch Queues.

71 at the statement waitForExpectations until the async code fulfils the expecta- tion. 1 import XCTest 2 // Note : use underscores ( ) for dash ( ) in module name 3@testable import omd ios devel chapter 05 Unit Tests 4 5 var backTask : BackgroundFunctionality? = nil 6 7 class omd ios devel chapter 05 iOS UnitTestingTests : 8XCTestCase { 9 let urlStr = ” http :// . tiles . mapbox . com” + 10 ” / v3 / examples . map zr0njcqy / geocode” + 11 ” / vienna . json” ; 12 override func setUp () { 13 super .setUp() 14 // Functionality must be implemented somewhere : 15 backTask = BackgroundFunctionality() 16 } 17 func testFetchingData () { 18 self .measure { 19 let result = backTask?.fetchData(fromURL: 20 URL( s t r i n g : self .urlStr) ) 21 a s s e r t ( r e s u l t != ” ” , {} 22 ”We expect ’ non empty ’ json” ) 23 } 24 } Listing 5.1: Adding an iOS Unit Testing Bundle separates the test cases from the rest of the source code..

1 func testAsyncTasks () { 2 let ex = expectation(description: ” MyAsyncTask” ) 3 let lowPrioQueue = DispatchQueue . global ( 4qos:.background) 5 6lowPrioQueue.async { 7 // we simulate a long running background task 8sleep(3) 9ex.fulfill() 10 } 11 waitForExpectations(timeout:5, handler: (err) in { 12 if let e=err { 13 p r i n t ( ” sorry timeout with err= ( e ) ” ) \ 14 else } {

72 15 p r i n t ( ” everything ok :)” ) 16 } 17 ) } 18 } Listing 5.2: Asynchronous test cases waiting for an expectation to be fulfilled..

See Section 16.1.6 about test automation with Continuous Integration (CI) on macOS servers.

5.6 Widgets

UI controls, sometimes called widgets or UI views, are accessed from the code via their IBOutlets, in a similar way as in GUIs for watchOS; see Section 2 Programming watchOS.

Table Some controls, such as a table (view), need information to render them- selves and display given data. To fill a table with data, you have to set the datasource, see Figure 5.2, and implement methods (callbacks) in the associated delegate and datasource class such as numberOfRowsInSec- tion, cellForRowAtIndexPath and optionally titleForHeaderInSection in a source code class according to the UITableViewDataSource protocol as shown in Listing 5.3. Interaction with the table UI are forwarded (dele- gated) to the class which implements methods of the UITableViewDelegate protocol such as didSelectRowAt. 1 class ViewControllerWithTableEmbedded : UIViewController , 2UITableViewDataSource,// how many sections , rows , ..? 3UITableViewDelegate// trigger actions { 4 5 // some methods of UITableViewDataSource protocol 6 func numberOfSections ( in tableView : UITableView) 7 > Int return 2 { } 8 9 func tableView ( tableView : UITableView , 10 numberOfRowsInSection section: Int) 11 > Int return 3 { } 12 13 func tableView ( tableView : UITableView ,

73 2ddeetoA ) h t a P x e d n I : h t a P x e d n i ) t n I : didSelectRowAt n o i t c e s n o i t c e ( S n t I n r i e r d p a ( e H t r n o i F r e p l t i t : r e i f i 36 t n e d I h 35 t i w 34 33 32 31 = 30 t x e t . 29 ? ) l h e t b a a P L x t e 28 d x n e I t . l 27 : l e h t c a P x e 26 d n i 25 cellForRowAt 24 23 22 21 20 19 18 17 16 15 14 5 row. given a at entry single a for data and example, rows for of report, number to methods the implement must controller The 5.3: Listing } xet ih d le n erhfntoaiyt als ssonin shown as tables, to functionality search https://www and filter add might Experts idashort a Find } } func func // // // // a return let iue5.2: Figure > > ” method and and alVe ( tableView ( tableView configure . \ raywenderlich UITableViewCell ( String? indexPath el=tableView.dequeueReusableCell( = cell set set ” ” 5 in Selected cell of uoilat tutorial the style section Set { UITableViewDelegate the . com/472-uisearchcontroller-tutorial-getting-started alVe UITableView , : tableView alVe UITableView , : tableView id return . dataSource section to table row to \ basic http://www { ” ( ” \ indexPath LabelCell for 74 view +1) ( Section indexPath ” :indexPath) and Elem in with delegate . row codingexplorer ” \ for . ( section a protocol section . row \ ( prototype section o table. a for indexPath +1) ”

+1) screenshot by john feiner LabelCell +1) ” ) . . com/getting- ” ” ” ) + } . cells row ” . +1) , { ” started-uitableview-swift/.

Custom Designed Views Programmers can create their custom views, which are rendered live in the storyboard at design time.

5.7 Accessibility

UI elements have hidden features, such as storing metadata for support people with disabilities. The programmer can enhance the user experience for these users with adjusted, custom spoken text, read to the end user by the built-in system screenreader.

5.8 UI Tests

For testing, it is important to provide a unique identifier for every widget in the user interface (can be set within the widgets Accessibility properties). Only widgets with accessibility enabled can be reached during UI testing. In Listing 5.4 the Swift file in the iOS UI Testing Bundle – which can be added through the File/New/Target template-wizard – is shown. After adding a test function and positioning the cursor inside the function body, you can start recording a UI test case. 1 import XCTest 2 class omd ios devel chapter 05 iOS UITestingUITests : 3XCTestCase { 4 5 override func setUp () { 6 super .setUp() 7 // called before the invocation of each test method 8continueAfterFailure=false 9XCUIApplication().launch() 10 } 11 12 func testChangingTemperature () { 13 let app = XCUIApplication () 14 // Note : Storyboard / Identity inspector : 15 // accessibility identifier must be set 16 let sl=app. sliders [” temperatureSlider” ] 17 // slide to

75 18 sl.adjust(toNormalizedSliderPosition: 0.5) 19 // check : 20 XCTAssert ( s l . v a l u e as?String==”50%” ) 21 } Listing 5.4: Xcode supports recording UI test statements (by pressing the red record button at the bottom of the code window). Developers need to add code for assertions. Finally, the test cases can replayed..

Watch the video UI Testing in Xcode 76 from the WWDC 2015, where the main concepts around XCUIApplication, XCUIElement, and XCUIEle- mentQuery are explained in detail. If an UI tests fails, check out the test report in the Report Navigator where you will find screenshots taken auto- matically during tests. Starting with version 8.1 of Xcode, the single layers of the UI – the UI Hierarchy – can be inspected during debugging as 3D stack.

5.9 Auto Layout

With different iPhone sizes the formerly fixed number of pixels is now vari- able and the UI controls are arranged7 in a dynamic way. The rendering engine Auto Layout uses constraints to decide where and how elements ap- pear onscreen. The general idea of constraints is explained in Figure 5.38. In the Auto Layout Tutorial in iOS 11: Getting Started9 many examples about constraints and, for example, ways of setting the leading and trailing spaces, the width and height dynamically – with help of the weights (the priorities) – can be found.

6 WWDC2015: UI Testing https://developer.apple.com/videos/play/wwdc2015-406. 7 UI controls are aligned (e.g. centred) by the auto layout engine according the edges of their alignment rectangle and not by the possibly larger frame rectangle which might include ornamentations or invisible space such as borders around a button, see https://developer.apple.com/library/archive/documentation/UserExperience/ Conceptual/AutolayoutPG/DebuggingTricksandTips.html. 8 Image from https://en.wikibooks.org/w/index.php?title=Swift learning&stable=1#Auto Layout by Mathias Knoll 9 Auto Layout Tutorial in iOS 11 https://www.raywenderlich.com/160527/auto-layout-tutorial-ios-11-getting-started.

76 nse)U elements. UI (nested) 5.3: Figure uoLyu losteseicto fmn osrit for constraints many of specification the allows Layout Auto 77

by mathias knoll Development Tips When working with Xcode and Auto Layout, grouping UI elements into stack views (containers) and working with layout constraints (priorities, especially hugging and compression priorities) might need a bit of train- ing. Select one item (possible more items while pressing shift) and then Add New Constraints to the element (this action of generating new con- straints is also referred to as pinning). The pinning dialogue allows you to attach constraints such as setting the relation – the distance – to the top or the left border (or the relation of one element to the other). A preview can be activated in Xcode via Show the Assistant Editor to preview the arrangement in different screen layouts. Avoid complex layout, e.g. by grouping and nesting elements into (horizontal and/or vertical) stack views and use as few constraints as possible. Try visual UI debugging by snap- shotting a view and carrying out 3D analysis of the layers (click Debug View Hierarchy).

5.10 Navigation Flow

The navigation from one Storyboard element (UI view or scene) to an- other is modelled by special paths called segues. Those segues represent the move to another screen action and can be modal (page-based) or hi- erarchical (push). In Figure 5.4 you can see the CTRL-drag connection set up for push navigation from a button in the first view to a second view. Furthermore, in Figure 5.4, the view controller for the first view is equipped with a callback method @IBAction func unwindBackToMyMainView( sender: UIStoryboardSegue) (any method name is ok) which allows you to un- wind back from any other view. By specifying CRTL-drag connections in the second (detail) view – for example, connections from a button in the Interface Builder to the Exit icon on top – the unwind segue is triggered on exit. Before a storyboard segue is executed, data can be written to the des- tination view controller by overriding prepare(for:sender:). An alternative way is to trigger a segue programmatically in the code by calling methods such as performSegue(withIdentifier:sender:). 1 override func prepare ( for segue : UIStoryboardSegue , 2sender:Any?) { 3 if let transData = inputA . text {

78 4print(”We prepare ’ ( transData ) ’..” ) \ 5 if segue . destination is DetailViewController { 6 if let target = segue. destination 7 as?DetailViewController { 8print(” .. to set at the target view controller” ) 9 // Our DetailViewController 10 // has a public property ( ) ” data” 11 t a r g e t . d a t a = [ ” data” :transData] 12 } 13 } 14 } 15 } Listing 5.5: To set data onto the destination view controller you can override function prepare(:for segue :sender)..

See also Section 2.5.7 WatchOS – Passing Data for code snippets about passing data from one scene to the next.

Exit (a) Push the Detail-View on the view stack (b) Unwind back to the Main View. Detail-View

push (on stack) (b) CTRL-DRAG Exit Done Main-View (a) CTRL-DRAG // (a) no code required // (b) no code required Show Settings

// (a) no code required unwind // (b) for unwinding: (remove from stack) @IBAction func unwindBackToMainView( sender: UIStoryboardSegue{ // optional: // code when back in main } omnigraffle drawing by john feiner

Figure 5.4: Push views onto the view stack and unwind back.

Applications need to be handled in very different ways depending on the overall navigation concept. Hence, at the start of every project in Xcode you have to choose on either the master detail (e.g. for iPad with list navi- gation on the left and main view in the centre), tabbed (app with some tabs on the bottom), page-based (single to swipe through), single view, or

79 game (SceneKit or similar frameworks for 2D or 3D game support) modes for the application.

Development Hint When developers need stack navigation later on, this can be achieved by embedding a view into a navigation controller (after selection of the view in the storyboard, select in the menu bar items Editor / Embed In / Navigation Controller). This way the navigation stack including the top bar with back functionality is available. Check out the demo application SegueCatalog from Apple via the Xcode documentation and inspect the views to learn about Initial Con- troller which determines the starting scene and Navigation Controller with the relationship to the Root View Controller. View controllers can push with Show or Show Detail (i.e. replace) other scenes into view hiding the current one, but could also present modal scenes, or popover (overlays). An outer Split View Controller allows to show two scenes side-by-side, e.g. master-details scenes on a tablet or single (but related) master-detail scenes on phones. A Storyboard can be split up into multiple storyboards connected via Storyboard References. Also find out how to unwind, i.e. navigating back to any other scene (any scene, which specifies in the code a line similar (You can choose an arbitrary method name) to following: @IBAction func unwindToCurrent( segue: UIStoryboardSegue) ). The unwinding {} can than be configured in the storyboard, as already explained and shown in Figure 5.4.

5.11 Interaction Design for Multitouch Input

On the user interaction is more powerful – relative to classi- cal desktop applications – in the sense that additional input through touch screens is possible. In the code it is possible to react to events from ges- tures such as swipe (e.g. show next image for the slideshow) or pinch (e.g.: zooming into a map). Beginning with iOS9 there is a technology called 3D Touch, see Get- ting Started with 3D Touch10, which – depending how hard the user presses

10 https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/ Adopting3DTouchOniPhone/index.html#//apple ref/doc/uid/TP40016543.

80 on an item – presents a kind of context menu, a peek preview of an item, or opens the item in a different window. This is faster and more powerful than the classic touch and hold procedure.

5.12 Internationalisation

To distribute apps in international stores, the interface has to be localised in multiple languages. Internationalisation (I18N) can be supported by getting text from language-aware resource bundles.

5.13 Further Reading

In this section we point you to some current research (papers) and tutorials.

5.13.1 Beginner and Advanced Tutorials On http://www.techotopia.com/index.php/IOS 9 App Development Essentials you find the online version of an eBook iOS 9 App Development Essentials by Smyth (2015). The use of Model Controllers as extension to simple MVC and how to share state (with singletons, the app delegate and depen- dency injection) between Swift view controllers is discussed in https: //code.tutsplus.com/tutorials/the-right-way-to-share-state-between-swift- view-controllers--cms-28474. Find ways to create simple tables at https://www.codingexplorer.com/ getting-started-uitableview-swift/ and powerful tables with search and fil- ter functionality at https://www.raywenderlich.com/472-uisearchcontroller- tutorial-getting-started.

5.13.2 Research Papers – Education Aspects of using storyboards for educating developers are discussed in Rogers and Shoemake (2014).

5.13.3 Research Papers – Usability In Tan et al. (2014) there is a discussion of the effect of developer-specified explanations for permission requests on smartphone user behaviour.

81 5.13.4 Research Papers – Security Pewny and Holz (2013) show that Control Flow Integrity (CFI) can avoid certain iOS vulnerabilities by extending the LLVM compiler.

5.14 Review Questions

A good approach to answering following questions is to try to define and explain a given term in your own words, to state typical usage and do- mains (this is a solution for which problem). Then try to state possible alternatives for the technology and/or for this approach. Furthermore, ex- plain the advantages AND disadvantages and optionally give source code examples.

5.14.1 Some Basic Questions • Can you explain the most important life cycle events?

• What are the most important contrasts between using segues in code vs. in the storyboard?

• Can you name the main similarities and differences in the architec- tures of iOS and Android Applications?

5.14.2 Some More Advanced Questions • Can you give examples of Delegate and Datasource usage with UIV- iews and their respective controllers?

• What possibilities are available for pinning and which priorities can be set for constraints in Auto Layout?

82 Bibliography

Pewny, Jannik and Thorsten Holz (2013). “Control-Flow Restrictor: Compiler-Based CFI for iOS”. In: Proc. 29th Annual Computer Se- curity Applications Conference (ACSAC 2013). ACM, pp. 309–318. ISBN: 9781450320153. DOI: 10.1145/2523649.2523674. Rogers, Michael P. and Andrew O. Shoemake (May 2014). “Storyboards: A Graphical Development Tool for the Rest of Us”. In: J. Comput. Sci. Coll. 29.5, pp. 171–178. Smyth, Neil (Oct. 17, 2015). iOS 9 App Development Essentials. CreateS- pace. ISBN: 1518651550. URL: https://www.createspace.com/5806391. Tan, Joshua, Khanh Nguyen, Michael Theodorides, Heidi Negron-Arroyo,´ Christopher Thompson, Serge Egelman, and David Wagner (2014). “The Effect of Developer-Specified Explanations for Permission Re- quests on Smartphone User Behavior”. In: Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI 2014). ACM, pp. 91– 100. ISBN: 9781450324731. DOI: 10.1145/2556288.2557400.

83 84 6 Background Tasks in iOS

“Software is hard. It’s harder than anything else I’ve ever had to do.”

[Donald Knuth]

Summary: Ensuring a responsive UI is critical for all iOS apps. This means, that longer lasting tasks have to be done in the background. Con- currency in iOS requires knowledge of execution queues. Basics concepts and language features are discussed in this section.

6.1 Introduction

This section about background tasks in iOS is structured as follows: first we explain the general concepts of (dispatch queues). Then, many exam- ples for fetching data from web services follow. Serialisation and deserial- isation of (JSON) data is discussed. Finally, we point to some papers and tutorials about concurrency with queues.

6.2 Concurrency

For easier ways to write concurrent code Grand Central Dispatch and dis- patch queues are provided. The alternative – i.e. to code threads directly – is discouraged1.

1 Arguments for Grand Central Dispatch and against coding threads (single task threads, worker threads or thread pools) including the tricky part of synchronisation: https://developer.apple.com/library/ios/documentation/General/Conceptual/ ConcurrencyProgrammingGuide/ThreadMigration/ThreadMigration.html#//apple ref/ doc/uid/TP40008091-CH105-SW1.

85 6.3 Dispatch Queues

Dispatch queues may execute tasks synchronously as FIFO (first in first out). Serial dispatch queues (e.g. the main queue) wait for one task to finish before starting the next one (and are therefore a great alternative to locks for synchronisation). Asynchronous queues, i.e. concurrent dispatch queues see Listing 6.1, on the other hand will not wait for a task to finish before starting the next tasks. The pool of concurrent tasks is managed automatically. Priority2 of execution can be set by specifying Quality of Service. 1 import Dispatch 2 let concurrentQueue = DispatchQueue( // custom queue 3label:” at . fhj . ims . worker” , 4qos:.background)// very low priority 5 6concurrentQueue.async { 7print(” Working in the background” ) 8 } Listing 6.1: Concurrency with Grand Central Dispatch and Dispatch Queues.

Optionally, for reusing code blocks, you can set up and later execute so called DispatchWorkItems. Special flags allow you to set up these Dis- patchWorkItems in different ways, for example as barrier tasks. A barrier task waits for currently executing (synchronous and) asynchronous tasks in the queue to be completed before starting the next task in an asynchronous queue.

6.4 Fetching Data

To fetch data from remote servers, an https (http is considered insecure) connection is used to retrieve binary or textual data in the background. Simple scenarios could involve fetching textual data into a string or binary data into an image. After data arrives, we have to switch back to the main queue to update the user interface.

2 Use Quality Of Service (QoS) class User-interactive for high or Background for low priority queues. See https://developer.apple.com/library/content/documentation/ Performance/Conceptual/EnergyGuide-iOS/PrioritizeWorkWithQoS.html.

86 For accessing web services configuring a completion handler with the dataTask (using the shared singleton object of URLSession) and starting the download with task.resume() is recommended3. Another approach of loading images – using explicit queues – is shown in Listing 6.2 – note the (very basic) exception handling.

1 import UIKit 2 class ViewController : UIViewController { 3@IBOutletweak var theImg : UIImageView! 4 let imageURLs = [ // https required ! 5 ” https :// st . pimg . net / perlweb / images / camel head . png” , 6 ” https ://www . apache . org / img / asf logo . png” , 7 ” https :// rubyonrails . org / images / rails logo . svg” ] 8 override func viewDidLoad () { 9 super .viewDidLoad() 10 p l a y S l i d e s h o w ( ) 11 } 12 func playSlideshow () { 13 theImg.image = UIImage(named: ”dummy” ) // local assets 14 DispatchQueue.global().async { 15 for urlStr in self .imageURLs { 16 p r i n t ( ” Loading image ( urlStr ) ...” ) \ 17 if let url = URL(string:urlStr) { 18 do { 19 let data = try Data( contentsOf : url ) 20 let image = UIImage( data : data ) 21 DispatchQueue.main.async { 22 p r i n t ( ” Update UI in main thread” ) 23 self .theImg.image=image 24 } 25 catch let e // e . g . insecure http : } { 26 p r i n t ( ”E : ( e ) ” ) // could not be opened \ 27 } 28 s l e e p ( 3 ) 29 else } { 30 p r i n t ( ”W: ( urlStr ) is not valid ! ” ) \ 31 } 32 } 33 } 34 }

3 URLSessionConfiguration allows to use caching, setting timeouts or even allow downloading while the app is suspended and wake the app on completion.

87 Listing 6.2: Fetching data in the background and updating the UI in the main queue. A typical scenario, where you need asynchronous queues for perfor- mance (see Figure 6.1) and synchronous queues to prevent race conditions (see Figure 6.2) would be the following: • Create a concurrent (async) queue.

• Create a dispatch group (to group together and time synchronise a parallel download of the next step)

• Start two concurrent (async) downloading tasks (e.g. downloading multiple images from two different servers in parallel) and store the images in a custom data store you created before.

• Use group.notify to get notification when downloading of both back- ground tasks is completed.

• In the group notify code block you can now update the GUI (back in the main thread with DispatchQueue.main.async ... update ui code { ... ) by for example reloading a table with the data of your custom } data store. Unfortunately, this code introduces a concurrency bug, i.e. a race con- dition, two threads executing the same code at the same time. In this case, the two threads are writing at the same time – without synchronisation – to the custom data store. To solve the concurrent bug, a sync queue can be used:

• Create a sync queue inside your data store.

• Move the code for every operation (append, count,..) accessing data into a block executed by the serial queue to prevent unsynchronised, i.e. concurrent, access to the data store.

This way a) the async queue drastically improves download perfor- mance, b) the dispatch group simplifies synchronisation to know when download has finished and c) the synch queue prevents concurrent access of data which would otherwise corrupt the data because of race conditions.

88 (async) Thread-Group All downloads Download from server A finished => Update UI Load from B

group.notify { Load from C code block

}

time

Start (concurrent) threads A,B,C Threads A, B, C done omnigraffle drawing by john feiner

Figure 6.1: Typical usage of async queues to speed up performance with concurrent downloads.

time

async A: Download from server A Download from server A (1) (2) async B: Load from B Load from B (1) (1) (1) (2) (2) X X (1) (1) (2) Sync Queue code-block A code-block B

Unprotected (1,2) Protect access with (concurrent) data access sync queues and causes Data Race. (1,2) avoid Data Race.

DATA: find (1) image and update (2) url/thumbnail omnigraffle drawing by john feiner

Figure 6.2: Prevent Data Race with sync queues. Concurrent access to in-memory data (structures) is prohibited and corruption is not possible anymore.

89 6.5 Consuming Java Script Object Notation (JSON) Web Services

To fetch serialised object data from remote servers (e.g. from a ReST web service) it is necessary to parse the incoming JSON as shown in Listing 6.3.

1 import Foundation 2 let urlStr = 3 ” https :// . apple . com / search? term= christoph+ lollo” 4 class Track { 5 var song = ” ” 6 var artist = ” ” 7 var price = Float (0) 8 var currency = ”EUR” 9 } 10 if let url = URL(string: urlStr) { 11 if let d=try?Data(contentsOf:url) { 12 do { 13 let parsedData = 14 try JSONSerialization.jsonObject(with: d as Data ) 15 // resultCount = 3; { 16 // results = ( artistName = ” Christoph”, .... { 17 if let dict = parsedData as?[String:Any] { 18 if let results = dict[” results” ] as?[Any] { 19 for case let el as [String:Any] in results { 20 let a=Track() 21 if case let song as String = el[” trackName” ] { 22 a . song = song } 23 if case let artist as String = el[” artistName” ] { 24 a . a r t i s t = a r t i s t } 25 if case let price as Float = el [” trackPrice” ] { 26 a . p r i c e = p r i c e } 27 if case let curr as String = el[” currency” ] { 28 a . c u r r e n c y = c u r r } 29 let info = ”Buy ( a . song ) from ( a . artist ) ” + \ \ 30 ” for ( a . price ) ( a . currency ) . ” \ \ 31 p r i n t ( i n f o ) 32 } 33 } 34 } 35 catch let err } { 36 p r i n t ( ”E : ( err ) ” ) \ 37 } 38 else } { 39 p r i n t ( ”W: invalid ( url ) ” ) \ 40 }

90 41 } Listing 6.3: JSON data must be serialised into your own objects.

6.6 JSON Deserialisation

To convert JSON into Swift data types, create specialised initialisers for your classes as shown in Listing 6.4 and Listing 6.5. This simplifies the code for the actual parsing (see Listing 6.6). 1 struct Song { 2 let name : String 3 let artist : String 4 let price : Float 5 let currency : String 6 var cost : String { 7 get return ” ( self . price ) ( self . currency ) ” { \ \ }} 8 } 9 extension Song { 10 init ?( json : [ String : Any] ) { 11 guard let name = json [ ” trackName” ] as?String, 12 let artist = json[” artistName” ] as?String, 13 let price = json[” trackPrice” ] as?Float, 14 let currency = json [” currency” ] as?String 15 else { 16 return nil 17 } 18 self .name = name 19 self .artist = artist 20 self .price = price 21 self .currency = currency 22 } 23 24 } 25 extension Song : CustomStringConvertible { 26 var description : String { 27 return ” ( self . name ) by ( self . artist ) ” \ \ 28 } 29 } Listing 6.4: Prepare a class for objects to be initialised with JSON.

1 struct Playlist { 2 var songs : [Song] = []

91 3 } 4 extension Playlist { 5 init ?( json : [ String : Any]) { 6 // resultCount = 3; { 7 // results = ( artistId = 130218208; { 8 // artistName= .... , .. ) } { } } 9guard 10 let count = json [” resultCount” ] as?Int, 11 let jSongs = json [” results” ] as?[[String:Any]] 12 else { 13 return nil 14 } 15 p r i n t ( ”JSON API : ( count ) songs loaded ! ” ) \ 16 for jsonSong in jSongs { 17 if let newSong = Song ( json : jsonSong ) { 18 self .songs.append(newSong) 19 } 20 } 21 } 22 } Listing 6.5: Prepare another class for a collection to be initialised with JSON data.

1 var tracklist : Playlist? 2 let songURL = ” https :// itunes . apple . com / search” + 3 ” ? term=Van+ Morrison&limit=7” 4 // resultCount = 3; { 5 // results = ( artistId = 130218208;... , { } 6 // ... ,... ) { } } 7 if let url = URL(string : songURL) { 8 if let data = try? Data( contentsOf : url ) { 9 do { 10 let parsedData = try JSONSerialization . jsonObject ( 11 with : d a t a as Data , 12 options:.allowFragments) 13 if let dict = parsedData as?[String:Any] { 14 tracklist = Playlist(json:dict) 15 } 16 catch let err } { 17 p r i n t ( ”E : ( err ) ” ) \ 18 } 19 else } { 20 p r i n t ( ”W: invalid ( url ) ” ) \ 21 }

92 22 } 23 if let mysongs = t r a c k l i s t ?. songs { 24 for (idx,s) in mysongs . enumerated () { 25 p r i n t ( ” ( ( idx+1) ) : Buy ( s ) for ( s . cost ) . ” ) \ \ \ 26 } 27 } Listing 6.6: Parsing JSON into special custom classes.

1 struct SongDC : Codable { 2 let artistName : String 3 let trackName : String 4 } 5 struct PlaylistDC : Codable { 6 var resultCount : Int 7 var results : [SongDC] 8 } 9 10 if let songUrl = URL( string : 11 ” https :// itunes . apple . com / search? term= christoph+ lollo” ) { 12 13 // optional configuration of cache and timeout 14 let request = URLRequest( url : songUrl , 15 cachePolicy: URLRequest.CachePolicy. reloadIgnoringLocalCacheData , 16 timeoutInterval: 60.0) 17 let task = URLSession. shared . dataTask(with: request ) { 18 data, response, error in 19 if let err = error { 20 p r i n t ( ” Error ( err ) ” ) \ 21 return 22 } 23 guard let resp = response as?HTTPURLResponse, 24 (200...299).contains(resp.statusCode) else { 25 p r i n t ( ” Error : Status code <200 or >= 300.” ) 26 p r i n t ( ” Check the response ! ” ) 27 return 28 } 29 if let d=data { 30 let decoder = JSONDecoder() 31 if let sl = try? 32 decoder.decode(PlaylistDC. self ,from:d) { 33 // we are in NSOperationQueue => 34 // to update GUI switch to main queue !! 35 p r i n t ( ”We got ( sl . resultCount ) songs : ” ) \

93 36 for song in sl . results { 37 p r i n t ( 38 ” ( song . trackName ) by ( song . artistName ) ” ) \ \ 39 } 40 } 41 } 42 } 43 t a s k . resume ( ) 44 } Listing 6.7: Parsing JSON into special custom classes. In Listing 6.7 an other possibility to load data with a dataTask from the URLSession is used. The parsing of JSON is done transparently by the Decoder protocol for each data class or struct with the JSONDecoder extracting the keys. For details on this approach to JSON parsing, visit the Apple tutorial Working with JSON in Swift at https://developer.apple.com/ swift/blog/?id=37. Visit Awesome Swift4 to find a range of Extensible Markup Language (XML) and JSON parsing libraries, for example Gloss5, or SwiftyJSON6.

6.7 Required Reading

Read the sections in the official Apple documentation for Con- currency Programming Guide at https://developer.apple.com/library/ ios/documentation/General/Conceptual/ConcurrencyProgrammingGuide/ Introduction/Introduction.html#//apple ref/doc/uid/TP40008091 – note that programming threads is discouraged for general use and Grand Central Dispatch and operation objects should be your first choice!

6.8 Further Reading

You might watch the video and view the slides of applevideoWWDC: https://developer.apple.com/videos/play/wwdc2016/720/ Concurrent Pro- 4 A list of Swift frameworks can be found at https://github.com/Wolg/awesome-swift#jsonxml-manipulation. 5 A tutorial for the JSON parsing library Gloss https://github.com/hkellaway/Gloss can be found at https://www.raywenderlich.com/120442/swift-json-tutorial. 6 SwiftyJSON https://github.com/SwiftyJSON/SwiftyJSON and optionally for parsing data you could use the SwiftyJSON extension Alamofire-SwiftyJSON from https://github.com/SwiftyJSON/Alamofire-SwiftyJSON.

94 gramming With Grand Central Dispatch in Swift 3 from the WWDC 2016. The official Apple documentation for watchOS concurrency when working with CoreData can be found at https://developer.apple.com/ library/watchos/documentation/Cocoa/Conceptual/CoreData/Concurrency.html and for iOS core data concurrency at https://developer.apple.com/library/ ios/documentation/Cocoa/Conceptual/CoreData/Concurrency.html.

6.8.1 Introduction Tutorials A tutorial from April 2015 – unfortunately still not updated to Swift 3 – is available at http://www.raywenderlich.com/79149/grand-central- dispatch-tutorial-swift-part-1 and with more details – such as dispatch groups – in http://www.raywenderlich.com/79150/grand-central-dispatch- tutorial-swift-part-2.

6.8.2 Research Papers Smartphone application delay optimisation is discussed in Nguyen (2014). Optionally, an in-depth analysis of Queue-Dispatch Asynchronous Systems including worst-case complexity for special problems (Parikh coverability problem and termination problem) can be found in Geeraerts, Heußner, and Raskin (2015).

6.9 Review Questions

A good approach to answering the following questions. Furthermore, ex- plain the advantages AND disadvantages and optionally give source code examples.

6.9.1 Some Basic Questions • What are the differences (including priority) between available cus- tom and default queues?

• What is meant by Quality of Service?

• What are the requirements for GUI updates?

95 6.9.2 Some More Advanced Questions • How and why are ReSTful web services called asynchronously in concurrent threads?

• Explain ways of XML or JSON serialisation and then elaborate on the problem of type checking?

96 Bibliography

Geeraerts, Gilles, Alexander Heußner, and Jean-Franc¸ois Raskin (Apr. 2015). “On the Verification of Concurrent, Asynchronous Programs with Waiting Queues”. In: ACM Trans. Embed. Comput. Syst. 14.3, 58:1–58:26. DOI: 10.1145/2700072. Nguyen, David T. (2014). “Smartphone Application Delay Optimizations”. In: Proc. 2014 Workshop on PhD Forum (PhD forum 2014). ACM, pp. 3–4. ISBN: 9781450329408. DOI: 10.1145/2611166.2611168.

97 98 7 Persistency

“There’s a good reason why nobody studies history, it just teaches you too much.”

[by Noam Chomsky]

Summary: Several options exists for saving data within iOS apps. Not only the automatic object relational mapping (O/R-M) with Core Data but several basics concepts and language features are discussed in this section.

7.1 Introduction

This section about persistency for iOS apps is structured as follows. First we explain the general concepts user defaults as shared local or cloud- based object stores; data models implemented using protocols and classes with a delegate; object relational mapping using core data. Then we present many examples with short demo code help to build understanding. Finally, we point to some papers and tutorials.

7.2 Persistency

Persistency is necessary to store any data for preserving the state of an app even when the app is terminated and restarted. A simple way to save the app state is to use shared preferences.

7.2.1 Default Configurations To avoid any hard coded values in the source code configurations can be stored in property lists, such as Info.plist, shown in Listing 7.1.

99 1 // Read configs from Info . plist 2 if let url = 3Bundle.main.infoDictionary?[” WebServiceURL” ] as?String { 4print(url)// https ://my . web . service . com 5 else } { 6print(” Specify key ’ WebServiceURL ’ in Info . plist” ) 7 } Listing 7.1: Loading configurations from property files.

7.2.2 Persistency with Shared Preferences

A default key-value store called UserDefaults enables you to persist data1 to preserve and restore it later (see Listing 7.2). You can set any object with setValue:forKey and read an object with value:forKey by specifying a key. Simple objects like bool, integer, float or string can be set and retrieved in a type-safe manner with, for example, set:value:forKey and integer:forKey.

1 func loadCurrentHitCount () >Int { 2 return UserDefaults . standard . integer ( 3forKey:” at . fhj . statistics . hitcount” ) 4 } 5 6 func incHitCount () >Int { 7 let userDefaults = UserDefaults . standard 8 let currHitCount = loadCurrentHitCount () + 1 9userDefaults.set (currHitCount, 10 forKey : ” at . fhj . statistics . hitcount” ) 11 userDefaults.synchronize() 12 return currHitCount 13 } Listing 7.2: Storing key-values (values might be any object) into shared prefs using the UserDefaults API.

Never store passwords in simple key value stores such as UserDefaults. See Section 11 for the usage of the (cryptographic) secure Keychain.

1 The data will be saved to the disk periodically by the system, hence the deprecated call synchronize might only be necessary before the app exits.

100 7.2.3 Cloud-Based KeyValue Store Persisting data to the cloud can be done transparently with NSUbiqui- tousKeyValueStore2. This way key value pairs are stored for users with an iCloud account. Any changes to the iCloud data will trigger a push event and apps should register NSUbiquitousKeyValueStoreDidChangeEx- ternallyNotification to get the notification. The synchronize method will sync the local cache with the online iCloud data.

7.2.4 Object/Relational Mapping (O/R-M) Before implementing the actual save and read operations of complex data to a database, the cloud or to local file system, the developer has to focus on the data model. With modern languages the implementation of the model will often mean designing and implementing protocols first and classes afterwards. Or, as with Core Data, you can just design them visually (and have the code auto-generated).

7.2.5 Persistency with Core Data When creating new projects, the O/R-Mapping technology Core Data can be chosen. Some code for implementing persistency (persistentContainer) with Core Data is added to the AppDelegate as shown in Listing 7.3. The data model is inside the *.xcdatamodeld file and can be interactively used to set up entities and their relationships. 1 import CoreData 2 3@UIApplicationMain 4 class AppDelegate : UIResponder , UIApplicationDelegate { 5 var window : UIWindow? 6 // MARK : Core Data stack ( THIS IS AUTO GENERATED CODE ) 7 lazy var persistentContainer : NSPersistentContainer = { 8 let container = NSPersistentContainer( 9name:” omd ios devel chapter 07 iOS Persistency” ) 10 container.loadPersistentStores(completionHandler: { 11 (storeDescription, error) in 12 if let error = error as NSError? // during devel { 2 Read the introduction to the API at https://developer.apple.com/library/mac/documentation/Foundation/Reference/ NSUbiquitousKeyValueStore class/#//apple ref/doc/uid/TP40010934-CH1-SW27.

101 13 f a t a l E r r o r ( ” Unresolved error ( error ) , ” + \ 14 ” ( error . userInfo ) ” ) \ 15 } 16 ) } 17 return container 18 () } 19 // MARK : Core Data Saving ( THIS IS AUTO GENERATED CODE ) 20 func saveContext () { 21 let context = persistentContainer . viewContext 22 if context .hasChanges { 23 do { 24 t r y c o n t e x t . save ( ) 25 catch // during devel } { 26 let nserror = error as NSError 27 f a t a l E r r o r ( ” Unresolved error” + 28 ” ( nserror ) , ( nserror . userInfo ) ” ) \ \ 29 } 30 } 31 } Listing 7.3: Core data code in the app delegate. After selecting the *.xcdatamodeld file in Xcode3, one can add entities (e.g. Slideshow Entity with non-optional attribute title and Photo Entity with sequenceNo and imageUrl) with the relation 1:n for allowing a work slideshow and a holiday slideshow to each hold many photos). It might be better to switch off the automatic code generation (switch off the corresponding setting for the entities in data model designer) if you want to modify the (mapping) classes. Find in Figure 7.2 the necessary layers for working with the O/R-M. Xcode then offers manual generation of source code via Editor Create NSManagedObject Subclass and creates files such ! as Photo+CoreDataClass.swift, Photo+CoreDataProperties.swift, SlideshowPresentation+CoreDataClass.swift, and SlideshowPresenta- tion+CoreDataProperties.swift. The classes can be augmented with your custom methods afterwards, as shown in Listing 7.4 and Listing 7.5. 1 import CoreData 2@objc(Photo) 3 public class Photo : NSManagedObject { 3 Find step-by-step instructions at https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ CoreData/KeyConcepts.html#//apple ref/doc/uid/TP40001075-CH30-SW1.

102 code. 7.1: Figure O/R-Mapping Memory (Objects) Harddisk /Filessystem(RelationalDB) Harddisk Store Coordinator Store iue7.2: Figure Extension toClass NSManagedObjectModel Model (= Managed Objects DB: e.g. Table Class SQLite ) withAttributes Object … … … niyRltosi einralw o ognrt source generate to you allows Designer Relationship Entity (s) /- betRltoa apn R with ORM Mapping Relational Object O/R-M NSEntity Descriptor API (= Protocol ) MSEntityDescriptor NSManagedObjectModel o) Codegeneration: forEntityRelationships design o) Visual InitialisationCode Create Wizard o) CoreData Xcode IDEallows: Your Code Your Generated Code o) update,delete o) find,create withObjects Work Model (addmethods) Extension Model & Setup O/R-Mapping o) Persist: store 103 () … try … letsearchResults=tryreq.execute() do{ let req:NSFetchRequest=Photo.fetchRequest() … into:managedObjectContext)as!Photo forEntityName:"Photo", letnewPhoto=NSEntityDescription.insertNewObject( managedObjectContext.performAndWait { funcincreaseViewCount(){… public classPhoto:NSManagedObject{ @objc(Photo) @NSManagedpublicvarimageUrl:String extension Photo{ persistentContainer.persistentStoreCoordinator managedObjectContext.persistentStoreCoordinator = concurrencyType:.mainQueueConcurrencyType) NSManagedObjectContext.init( let {… lazy varpersistentContainer:NSPersistentContainer= managedObjectContext.save managedObjectContext

xcode screenshot by john feiner = CoreData () .

omnigraffle drawing by john feiner 4 func increaseViewCount () { 5 // Core Data written in Objective C 6 // => NSDecimalNumber 7 self .viewCount?.adding(1) 8 } 9 } Listing 7.4: After triggering the generation of classes (via menu Create NSManagedObject Subclass), additional custom methods might be added.

1 import CoreData 2 extension Photo { 3@nonobjcpublic class func fetchRequest () 4 > NSFetchRequest { 5 return NSFetchRequest(entityName : ” Photo” ); 6 } 7 // e . g : change from String? to String 8 // if imageUrl should be not optional 9@NSManagedpublic var imageUrl : String 10 @NSManaged public var viewCount : NSDecimalNumber? 11 12 } Listing 7.5: The extension to the class manages the mapping from Swift objects to the (Objective-C) Core Data persistence layer. In your custom code, the managedObjectContext (created with help of a persistentStoreCoordinator, the local XML, Binary, SQLite or In- memory database logic) is prepared as shown in Listing 7.6. Furthermore, it is necessary to save the context after performing changes as shown in Listing 7.74. The Xcode IDE allows the creation of stub-classes of sub- type NSManagedObject out of the entities created interactively, as shown in Figure 7.1. Saving data to the store can be done via NSEntityDescrip- tion.insertNewObjectForEntityForName. To batch-retrieve many objects you create a NSFetchRequest and use the managed object context to exe- cute the request (managedObjectContext!.executeFetchRequest). 1 // (1) container ( get ” singleton” instance from the app ) 2 let persistentContainer = (UIApplication . shared . delegate 3 as !AppDelegate).persistentContainer 4 // (2) managedcontext ( coordintor from container )

4 A long, detailed explanation for the necessary code can be found at http://jamesonquave.com/blog/core-data-in-swift-tutorial-part-1/.

104 5 let managedObjectContext = 6NSManagedObjectContext.init ( 7concurrencyType:.mainQueueConcurrencyType) 8managedObjectContext.persistentStoreCoordinator= 9persistentContainer.persistentStoreCoordinator

Listing 7.6: When working with data, a managedObjectContext is needed.

1 // (4) search and ( a ) create or ( b ) modify and ( c ) store 2managedObjectContext.performAndWait { 3 let req : NSFetchRequest =Photo.fetchRequest() 4 do // execute search { 5 let searchResults = try req . execute () 6 if searchResults .isEmpty // (4a ) create new item { 7 let newPhoto = NSEntityDescription . insertNewObject ( 8forEntityName:” Photo” , 9into:managedObjectContext)as !Photo 10 newPhoto . imageUrl =” ftp :// pic . org / img ( Date () ) . png” \ 11 newPhoto.viewCount=0 12 else // (4b ) modify item : } { 13 if let currPhoto = searchResults . first { 14 // call method we defined in class : 15 currPhoto.increaseViewCount() 16 p r i n t ( ” ( currPhoto . imageUrl ) ” ) \ 17 else } { 18 p r i n t ( ” Error , we expect at least one photo ! ” ) 19 } 20 } 21 do // (4c ) finally : save ! { 22 try managedObjectContext.save() 23 catch } { 24 p r i n t ( ” Error on save ( error ) ” ) \ 25 } 26 catch } { 27 p r i n t ( ” Error when searching ( error ) ” ) \ 28 } 29 } Listing 7.7: Search requests help to find objects. Then we might create/modify and finally save (persist) them.

105 7.2.6 Alternatives to Core Data The realm5 open source object database is an alternative solution to man- age your data.

7.2.7 Secure Persistency For secrets, such as passwords, it is recommended to use a keystore to store data. Find an Apple video about Keychain (including the foundations of TouchID and Secure Enclave) at https://developer.apple.com/videos/play/ wwdc2014-711/ and a tutorial at http://www.raywenderlich.com/92667/ securing-ios-data-keychain-touch-id-1password.

7.3 Required Reading

Read the source code of the core data demo from https://github.com/ iascchen/SwiftCoreDataSimpleDemo.

7.4 Further Reading

To get the idea of CoreData, check out the project CoreDataBooks from the official Apple documentation (but – unfortunately – note this project is still written in Objective-C).

7.4.1 Introduction Tutorials see the article What is Core Data? by Apple for an overview of the features, at https://developer.apple.com/library/watchos/documentation/ Cocoa/Conceptual/CoreData/index.html. Select the Section Integrating Core Data with iOS.

7.4.2 Research Papers Chang et al. (2015) present MobiPluto, a File System Friendly Deniable Storage for Mobile Devices for (Linux-based) Android Phones. End-to- end encryption with Crypto Phones is explained in Shirvanian and Saxena (2015), which also describes the evaluation of a range of crypto phones through usability testing. 5 https://realm.io

106 7.5 Review Questions

A good approach to answering the following questions. Furthermore, ex- plain the advantages AND disadvantages and optionally give source code examples.

7.5.1 Some Basic Questions • Where and how do you define O/R mapping?

• What is the difference between eager and lazy loading?

7.5.2 Some More Advanced Questions • What are the security consequences you would expect when storing to Core Data, UserDefaults and Keystore?

• Can you lay out use-cases for cloud documents and their synchroni- sation needs?

Bibliography

Chang, Bing, Zhan Wang, Bo Chen, and Fengwei Zhang (2015). “Mo- biPluto: File System Friendly Deniable Storage for Mobile Devices”. In: Proc. 31st Annual Computer Security Applications Conference (AC- SAC 2015). ACM, pp. 381–390. ISBN: 9781450336826. DOI: 10.1145/ 2818000.2818046. Shirvanian, Maliheh and Nitesh Saxena (Dec. 7, 2015). “On the Security and Usability of Crypto Phones”. In: Proc. 31st Annual Computer Secu- rity Applications Conference (ACSAC 2015) (Los Angeles, California, USA). ACM, pp. 21–30. DOI: 10.1145/2818000.2818007.

107 108 8 Location-Based Services

“We must design for the way people behave, not for how we would wish them to behave.”

[Donald A. Norman]

Summary: Smartphone users expect any app to take advantage of the current context. For example, the current time and location could be rel- evant for the running application to provide hints about the language to use or the day of the day. For optimal user experience (UX), avoid to get information from users, which could be retrieved automatically. Basics concepts and features such as displaying maps and getting routing infor- mation are discussed in this section.

8.1 Introduction

This section about location and maps for iOS apps is structured as fol- lows: First we explain the general concepts such as Location-based Ser- vices (LBS) including GPS, Point of Interests (POIs), Maps, Overlays, or Geofencing, (reverse) Geocoding. Then we present several practical source code snippets – for example CLLocationManager and the CLLoca- tionManagerDelegate – to illustrate the implementation in Swift in more detail. Finally, we point to some papers and tutorials.

8.2 Location-Based Services

In LBS, a web service provides information dependent on the current spa- tial position of the user. For example, an app sends a search request to the server to find the nearest gas station along with the current location

109 detected via the GPS. The service then finds relevant POIs, sends the posi- tions back and the app renders them as overlays on a map.

8.2.1 Users Context In a more general, broader view, the context of the end user with the smart- phone should be taken into consideration. Available data from different sensors and networks can be gathered and relevant information such as date and time of day, location (in longitude, latitude and altitude), but also the current speed when moving, darkness and current network informa- tion. That information – and possibly much more – can provide a good guess about a user’s current situation (For example, one might calculate the situation as ’The user left home a few minutes ago and is on the way to the office in a train which is delayed by 7 minutes’).

8.2.2 GPS and GLONASS Location To detect the current location (shown in Listing 8.2 and Listing 8.3) the user has to accept permission for the app to use the current user location re- ported by GPS and/or Globalnaya Navigazionnaya Sputnikovaya Sistema, or Global Navigation Satellite System (GLONASS). This MUST be done in the info.plist (see Listing 8.1) file with the addition of the key NSLoca- tionWhenInUseUsageDescription and entry for the message displayed to the user. If the location should be retrieved while the app is in the back- ground, the alternative key NSLocationAlwaysUsageDescription has to be used. In the code a callback for the updates of a user’s location is pro- vided and after asking for permission with requestWhenInUseAuthoriza- tion() you can startUpdatingLocation(). 1 CFBundleName 2 $(PRODUCT_NAME) 3 CFBundlePackageType 4 APPL 5 CFBundleShortVersionString 6 1.0 Listing 8.1: The XML source view of the file Info.plist shows the required message string presented to the user for requesting permissions to detect GPS.

110 1 import CoreLocation 2 public class WhereAmI : NSObject { 3 private var locMgr : CLLocationManager? = nil 4 public init (delegate:CLLocationManagerDelegate) { 5 super . init () 6 self .locMgr = CLLocationManager() 7 self .locMgr?.delegate = delegate 8 } 9 public func startup () { 10 // for background usage add following key 11 // to info . plist : NSLocationAlwaysUsageDescription 12 self .locMgr?.requestAlwaysAuthorization() 13 self .locMgr?.desiredAccuracy = kCLLocationAccuracyBest 14 self .locMgr?.startUpdatingLocation() 15 } 16 public func stop () { 17 self .locMgr?.stopUpdatingLocation() 18 } 19 } Listing 8.2: The get the current location the Core Location Manager is configured to inform a delegate about location changes.

1 import CoreLocation 2 class ViewController : UIViewController , 3CLLocationManagerDelegate { 4@IBOutletweak var longitude : UILabel! 5@IBOutletweak var latitude : UILabel! 6@IBActionfunc startUpdateLocation ( sender : Any) { 7getLocations(forSeconds:60) 8 } 9 func getLocations (forSeconds secs : Int=15) { 10 let whereAmI = WhereAmI( delegate : self ) 11 whereAmI . s t a r t u p ( ) 12 DispatchQueue.main.asyncAfter( 13 deadline: .now() + .seconds(secs)) { 14 whereAmI . s t o p ( ) 15 } 16 } 17 // callback : tells the delegate (= this class ) 18 // that location has changed : 19 func locationManager ( 20 manager : CLLocationManager , 21 didUpdateLocations locations: [CLLocation]) { 22 if let ll = locations . last {

111 23 let pos = ll . coordinate 24 l o n g i t u d e . t e x t = ” long : ( pos . latitude ) ” \ 25 l a t i t u d e . t e x t = ” lat : ( pos . longitude ) ” \ 26 } 27 } Listing 8.3: This examples runs the location detection for one minute and updates the gui with the latest location every time the manager detects GPS updates.

For simulating changing user location, one can create and deliver GPS Exchange Format in XML (GPX)1 files to the simulator by adopting the configuration in the scheme.

8.2.3 Geofencing and Location-Based Services Geofencing limits the scope of interest in some specified way. This ap- proach helps to find things nearby. For location-based services the idea of covering an area is important. The simplest case of Geofencing uses a radius to select a circle of interest inside the map. Elements moving into this circle from outside can be monitored via the callback didEnterRegion.

8.2.4 Maps When a UIView Map is presented, the selection of the region to show is set programmatically by setRegion. The user can interact with maps by zooming and panning. Callback methods are provided to react to this ac- tions e.g. with regionDidChangeAnimated. For the MapKit Framework has to be added to the project (Project/Targets/General/Linked Frameworks and Libraries). Listing reflst:apple-map shows how to focus on a specified location/region on the map (Figure 8.1). 1 // Required : MapKit Framework added to the project 2 import MapKit 3 class MapViewController : UIViewController { 4@IBOutletweak var mapWithPhotos : MKMapView! 5 override func viewDidLoad () { 6 super .viewDidLoad() 7 let home = CLLocation ( 8latitude:47.434,

1 GPS Exchange Format in XML.

112 9longitude:15.32) 10 // after a while ,... 11 DispatchQueue.main.asyncAfter( 12 deadline: .now() + .seconds(3)) { 13 weak var m= self .mapWithPhotos 14 // ... zoom in and ... 15 let dist : CLLocationDistance = 9988 // m 16 let homeRegion = 17 MKCoordinateRegionMakeWithDistance( 18 home.coordinate, dist, dist) 19 m?.setRegion(homeRegion, animated: true ) 20 // ... add a pin . 21 let myPosLabel = MKPointAnnotation () ; 22 myPosLabel.coordinate=home.coordinate; 23 myPosLabel . t i t l e =” Sweet home” 24 m?.addAnnotation(myPosLabel) 25 } 26 } Listing 8.4: Add the MapKit Framework to the project and a Map onto the view. iphone emulator screenshot by john feiner

Figure 8.1: Zooming in on a map after a few seconds, then adding an annotation with a message attached.

When a developer chooses Google Maps over the built-in Apple Maps,

113 a slightly trickier2 setup is necessary, as the Google API key is necessary and same libraries via pods3 have to be installed.

8.2.5 Overlays on Maps To show landmarks – points of interests (POIs) – one can use addAnno- tation on the mapView. Such annotations are small custom drawn labels (optionally with subtitle and image) attached to the current map view. By implementing a class that conforms to the MKAnnotation protocol, anno- tations are presented when the user taps on a pin. The so called callout is of type MKMapItem. Another possibility is to use overlays, which add a more complex layer of arbitrary information to the map view with render- ForOverlay.

8.2.6 Geocoding and Reverse Geocoding The class CLGeocoder enables translation of city names to geo locations and vice versa.

8.2.7 Remarks for Enterprises When setting up customer search for (locations of) enterprises using maps, it is important to make the exact location and some meta data, i.e. business information such as the url to the homepage, available on the official Apple Maps Service. Small enterprises should therefore register their location – via claim your location – online4. For restaurants providing a map exten- sion to display booking tables directly on the map might be of interest. See also the Section 16.1.3 about booking rides with Siri.

8.3 Further Reading

Necessary preparations for using Google Maps instead of the built-in Apple Maps are described in http://www.appcoda.com/google-maps-api- tutorial/.

2 Find, for example, a tutorial at http://www.appcoda.com/google-maps-api-tutorial/. 3 The pods dependency manager (Cocoa Pods http://cocoapods.org) will help to download the required libs and set up the project structure. 4 https://mapsconnect.apple.com.

114 Stores might use geo-fencing technology to offer special and customised services for users entering the building, see https:// www.bloomberg.com/news/articles/2018-11-15/inside-nike-s-store-of- the-future.

8.3.1 Introduction Tutorials Just showing the current position on the map can be achieved by following the instructions of https://www.veasoftware.com/posts/map-view-current- location-in-swift-xcode-7-ios-9-tutorial. How to get started with MapKit: http://www.raywenderlich.com/ 90971/introduction-mapkit-swift-tutorial and how to add overlays: http: //www.raywenderlich.com/87008/overlay-views-mapkit-swift-tutorial. A geofencing tutorial by Ray Wenderlich can be found at http:// www.raywenderlich.com/95014/geofencing-ios-swift.

8.3.2 Research Papers The question, of whether and how users can turn off tracking service set- tings is discussed in Moonsamy, Batten, and Shore (2013). The paper Tan et al. (2014) shows how users are more likely to accept permission requests if the developer gives textual explanations.

8.4 Review Questions

A good approach to answering the following questions. Furthermore ex- plain the advantages AND disadvantages and optionally give source code examples.

8.4.1 Some Basic Questions • Can you explain the use of location-based services and their advan- tages and disadvantages in terms of privacy?

• What are the differences between overlays, pins, placemarks and map items?

• What level of accuracy can we expect from different ways of deter- mining the location of a user?

115 8.4.2 Some More Advanced Questions • Can you state the idea geofencing and describe its application?

• What are the (practical) limitations of (reverse) geocoding?

116 Bibliography

Moonsamy, Veelasha, Lynn Batten, and Malcolm Shore (2013). “Can Smartphone Users Turn Off Tracking Service Settings?” In: Proc. Con- ference on Advances in Mobile Computing & Multimedia (MoMM 2013). ACM, 455:455–455:463. ISBN: 9781450321068. DOI: 10.1145/ 2536853.2536864. Tan, Joshua, Khanh Nguyen, Michael Theodorides, Heidi Negron-Arroyo,´ Christopher Thompson, Serge Egelman, and David Wagner (2014). “The Effect of Developer-Specified Explanations for Permission Re- quests on Smartphone User Behavior”. In: Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI 2014). ACM, pp. 91– 100. ISBN: 9781450324731. DOI: 10.1145/2556288.2557400.

117 118 9 Animations

“Well, luckily with animation, fantasy is our friend”

[Steven Spielberg]

Summary: To support gaming, one can select from several frameworks (i.e.: CoreAnimation, SpriteKit, SceneKit, Unity, or Metal) on the iOS plat- form. To draw custom 2D images (pixel oriented painting on a canvas) CoreAnimation can be used. Basics concepts such as differences between the frameworks, 2D vs. 3D frameworks, the game loop and finally selected features for 2D drawing and animation are discussed in this section.

9.1 Introduction

This section is about rendering and animation on a 2D canvas visualisation for iOS apps. First we explain the general concepts such as animation using a game loop and double buffering. Then we present some examples for drawing custom widgets (UIViews). Finally, we refer to some papers and tutorials.

9.2 Animation

In principle, animation means to draw changes onscreen in such a way that a human brain perceives objects moving. Therefore the redrawing – the refresh rate – must be fast enough to trick the eye. Animations can be done at a low level, by recalculating and redrawing a bunch of pixels again and again. At a higher level of abstraction, we use sprites, where objects are assembled in advance (or images are loaded) and then whole objects are steered across a screen.

119 9.2.1 Game Loop For playing games the refresh rate should be high enough (at least 30 Frame per Second (FPS), ideally more), but the CPU should not be over- loaded with work. Therefore, a so-called game loop consists of long pauses, a little bit of calculations (for defining a new position of pixels) and finally a short redrawing phase. If drawing an image takes a long time, optimisation can be archived through double buffering, i.e. by using two buffers: a hidden one for preparing (many possible slow drawing opera- tions) the image in the background and a visible one in the foreground. The prepared images can then be copied to the foreground very rapidly.

9.2.2 2D Drawing In the following paragraph the abbreviation CG stands for CoreGraph- ics (based on the graphics drawing engine 2D), which is the base framework for custom pixel-oriented drawing. This low-level way of paint- ing on a UIView handles pixels (e.g. for simple shapes) directly. This drawing and finally some animation can be done with the Core Anima- tion framework. The base idea is to create one’s own custom UIView and define the background, size, border or other attributes. Methods such as animate(withDuration:) allow to customise the movement by setting the duration and the (x/y coordinates of the) target position and/or changes in attributes such as alpha or the stretching1. In Listing 9.1 a class for a cus- tom element is prepared by subclassing UIView, by providing a custom draw method (see Listing 9.2 and in Listing 9.4 this element is created and animated2. 1 import UIKit 2 // Live Preview in Interface Builder ( IB ) 3@IBDesignable 4 class Spinner : UIView { 5 var name = ”No Name” 6 7 convenience init (name: String) { 8 let (posX, posY, w, h) = (20,20, 50,50)

1 Find various basic animation examples at http://mathewsanders.com/prototyping-iOS-iPhone-iPad-animations-in-swift/. 2 Many more examples can be found at http://www.techotopia.com/index.php/ An iOS 8 Swift Graphics Tutorial using Core Graphics and .

120 9 self . init ( 10 frame: CGRect(x:posX, y:posY, 11 width : w, h e i g h t : h ) ) 12 commonInit ( ) 13 self .name = name 14 } 15 required init ?( coder c:NSCoder) { 16 super . init (coder:c) 17 } 18 override init (frame: CGRect) // for IB { 19 super . init (frame: frame) 20 commonInit ( ) 21 } 22 private func commonInit () { 23 self .backgroundColor = UIColor.blue 24 self .layer.cornerRadius = 10.0 25 self .layer.masksToBounds=true 26 self .layer.borderColor = UIColor.red.cgColor 27 self .layer.borderWidth = 2.0 28 s t a r t A n i m a t i o n ( 3 ) 29 } Listing 9.1: The definition of a custom view requires a base class of type UIView.

1 // Custom drawing code : 2 var degree = 0 // current degree of ” spinning” line 3 override func draw ( rect : CGRect) { 4 let halfWidth = self .bounds.midX 5 let halfHeight = self .bounds.midY 6 let center = CGPoint(x: halfWidth , y: halfHeight) 7 8 let radians :Double = Double(degree) 9 * (Double.pi/180) 10 let toX = center . x + 11 CGFloat(cos(radians)) * halfWidth 12 let toY = center . y + 13 CGFloat(sin(radians)) * halfHeight 14 15 let context = UIGraphicsGetCurrentContext () 16 context?.setStrokeColor(red: 0.5, green: 0.5, 17 blue:0.5,alpha:1.0) 18 context?.setLineWidth(2.75) 19 context?.move(to: center) 20 context?.addLine(to: CGPoint(x: toX, y: toY))

121 21 context?.strokePath() 22 } Listing 9.2: The definition of a custom view requires a draw function, which is called from time to time to update the UI.

1 var speed = 1 // slow / fast line rotation 2 let refreshRate = 0.1 // in s => 10 fps 3 var myTimer : Timer? = nil 4 5 func startAnimation ( speed : Int ) { 6 self .speed=speed 7myTimer=Timer.scheduledTimer( 8withTimeInterval:refreshRate, 9repeats:true ) [ weak self ] in { 10 self ?.updateTimer () 11 } 12 } 13 func updateTimer () { 14 // recalc degrees 15 // e . g .: for one rotation per sec : 16 // 6 degrees per refresh 17 d e g r e e += 6 * speed 18 if degree >=360 degree = 0 { } 19 // ask to update Ui 20 self .setNeedsDisplay() 21 } 22 func stopAnimation () { 23 myTimer?.invalidate() 24 } Listing 9.3: To refresh the custom view a timer triggers the update of the UI.

1 class ViewController : UIViewController { 2 let speedy = Spinner (name: ” Speedy” ) 3 override func viewDidLoad () { 4 super .viewDidLoad() 5 self .view.addSubview(speedy) 6speedy.startAnimation(3) Listing 9.4: The class Spinner represents a custom UIView and can draw itself.

122 For periodic update, you can set a timer (see an example3 for Timer.scheduledTimer(timeInterval:) in Listing 9.3) which calls setNeeds- Display to tell the system that the custom view should be redrawn, i.e. the drawRect method should be called as soon as possible. A timer can be stopped with invalidate. Listing 9.5 shows how an animation is triggered to move a view during a given time frame. 1UIView.animate(withDuration:7.0,delay:2, 2options:[.autoreverse,// . repeat 3.allowUserInteraction], 4animations: { 5 self .speedy.alpha = CGFloat( Double(0.3) ) 6 self .speedy.frame = CGRect(x: 150, y: 120, 7width:self .speedy.frame.size.width * 2, 8height:self .speedy.frame.size.height 3) , * } 9completion:(value: Bool) in { 10 p r i n t ( ” Everything ok : ( value ) . Let ’ s hide . ” ) \ 11 self .speedy.alpha = CGFloat(Double(0.1)) } 12 ) Listing 9.5: Animate any view across the UI.

9.2.3 Advanced Games with SpriteKit in 2D and SceneKit in 3D Working with predefined Actors, so called Sprites, is possible with the SpriteKit4. By specifying , position and scaling, first the objects are created. Then they are added to a scene and are animated. The 3D high-performance rendering framework is called SceneKit and can work with imported 3D artefacts. As with it uses the lower layer of Open Graphics Library (OpenGL). For cross-platform games you can choose the Unity framework, which allows the writing of portable code. A rather new way for maximal graphics performance is provided by the Metal framework. As Metal is vendor-specific and GPU-accelerated, it

3 Note the usage of an @objc attribute indicating the dynamic method call by name (compare Objective-C selectors). 4 Find a starting point in the space invader tutorial at http://code.tutsplus.com/tutorials/ create-space-invaders-with-swift-and-sprite-kit-implementing-classes--cms-23355. Furthermore, simple demo code is available at https://ebook.n-studio.fr/chapter7-extract.

123 is an even faster 3D graphics framework than OpenGL ES.

9.3 Required Reading

Animations are discussed in the Apple documentation at https:// developer.apple.com/library/ios/documentation/WindowsViews/Conceptual/ ViewPG iPhoneOS/AnimatingViews/AnimatingViews.html#//apple ref/ doc/uid/TP40009503-CH6-SW2.

9.3.1 Introduction Tutorials Several tutorials for the SpriteKit can be found at http://www.raywenderlich.com/ category/sprite-kit, whereas the Unity framework can be found at http: //www.raywenderlich.com/category/unity.

9.3.2 Research Papers Spider, by Andrus et al. (2014) enables the native execution of iOS apps on Android. Optional research paper reading: a discussion of privacy concerns for data of multiple users Hu et al. (2014) (not about games specifically).

9.4 Review Questions

A good approach to answering the following questions. Furthermore, ex- plain the advantages AND disadvantages and optionally give source code examples.

9.4.1 Some Basic Questions • What is meant by the term game loop?

• What are the conceptual differences between Core Animation and SpriteKit?

9.4.2 Some More Advanced Questions • What are the conceptual differences between OpenGL ES and MetalM?

124 • What are the conceptual differences between SceneKit and Unity?

• Explain in detail the optimal frame rate and the ideas of double buffering.

Bibliography

Andrus, Jeremy, Alexander Van’t Hof, Naser AlDuaij, Christoffer Dall, Nicolas Viennot, and Jason Nieh (Feb. 2014). “Cider: Native Execution of iOS Apps on Android”. In: SIGPLAN Not. 49.4, pp. 367–382. DOI: 10.1145/2541940.2541972. Hu, Hongxin, Gail-Joon Ahn, Ziming Zhao, and Dejun Yang (2014). “Game Theoretic Analysis of Multiparty Access Control in Online Social Networks”. In: Proc. 19th ACM Symposium on Access Control Models and Technologies (SACMAT 2014). ACM, pp. 93–102. ISBN: 9781450329392. DOI: 10.1145/2613087.2613097.

125 126 10 Sensors

“Intelligence is the ability to adapt to change.”

[Stephen Hawking]

Summary: Modern smartphones have many sensors built in. The Gyroscopic-, Magnetometer-, Accelerometer- or, GPS-sensor are well known to the users, but further sensors such as Proximity-, Barometer- and Ambient Light-sensor might provide even more context about the user.

10.1 Introduction

This section is about a range of sensors (location, motion, radio1) and actu- ators (vibration, force-feedback2) available in high-end smartphones. First the available sensors and their general usages are explained. Hints for un- conventional use in apps are presented. Then some examples with short demo code should help foster understanding. Finally, find some papers and pointers to tutorials.

10.2 Sensors and Actuators

The sensors measure the surroundings and the current state of the device. In most cases the raw sensor data is not available to the developers. The

1 Many smartphones have a High-Frequency (HF) radio, which is disabled by default as discussed in http://www.npr.org/sections/alltechconsidered/2015/04/16/400178385/ the-hidden-fm-radio-inside-your-pocket-and-why-you-cant-use-it and http://blog.nablabs.org/tag/fm-chips/. 2 See a discussion about Haptics with the inertial haptic eccentric rotating mass (ERM) actuators, and high-definition haptics with the piezo actuator, at http://www.ti.com/lit/an/slyt483/slyt483.pdf.

127 Actuator Possible Use Case Vibration Twist, by vibrating, the smartphone on the table for a 360 degree panorama. Sound Send a hidden audio password to the listening desktop com- puter. Camera Place virtual objects on detected surfaces for augmented re- ality games.

Table 10.1: Possible use cases for different actuators. developer can register for notifications (e.g. proximityMonitoringEnabled) and can respond to events, such as UIDeviceOrientationDidChangeNotifi- cation3, emitted by the corresponding sub systems. Actuators are rare, but the main feature of vibration allows the device to give haptic feedback to the user.

Development Notes Note that during development, the following sensors are not supported by the iOS simulators4 and have to be tested on real devices: accelerome- ter, gyrometer, camera, microphone, proximity sensor, barometer and the ambient light sensor.

10.2.1 Actuators In Table 10.1 some actuators are listed with examples of unconventional usage.

10.2.2 Sensors Sensors provide input for the software on the device. For example the multitouch screen can distinguish between the gestures tap, double tap, touch and hold (long press), drag, swipe, flick, and pinch. The tree-axis

3 https: //developer.apple.com/library/ios/documentation/UIKit/Reference/UIDevice Class/. 4 https://developer.apple.com/library/content/documentation/IDEs/Conceptual/ iOS Simulator Guide/TestingontheiOSSimulator/TestingontheiOSSimulator.html.

128 gyroscope, combined with the accelerometer, allows the device to monitor rotation and acceleration along the x-, y- and z- axis. For Motion Aware- ness, the raw data of the sensors is combined into a so called Motion API, which allows identification of different kinds of Activity. For example, it can detect current user states of motion such as walking, running, cycling, automotive or stationary. It includes a Pedometer for step counting and telling the current pace. Phones with a barometer (measuring the pressure and supporting the altimeter by calculating relative altitudes) can use it to count floors ascended or descended. Raw accelerator inputs are often used for game controllers or to detect a shake event5. The compass works via the magnetometer-sensor and measures the earth’s magnetic field, which then requires a correction for the direction to the geographic north pole depending on the current geographic location6. The ambient light sensors serve to preserve battery charge by reducing the screen brightness in low ambient light or in the dark. The proximity sensor measures the distance to the user (again to save battery by switching off the display during phone calls, when the smartphone is near the user’s head), but unfortunately de- livers only boolean values via the API call proximityState. The GPS sensor generates data about the global position via satellites and is heavily used for location-based services (see Section 8). The biometric sensor Touch ID allows the user to unlock the device (and the keychain) with fingerprint authentication7. NFC for payment is available with high-value phones, but – as too often with Apple technologies – is limited to the use with Apple Pay8 terminals (see Section 16.1.5). A barometer allows the phone to get basic data and more accurate information about the (change in) height, because the GPS-derived altitude values are often very imprecise. Audio input is not for calls only, but for detecting the song name of the music playing or to give commands via the iOS speech assistant system Siri. The cameras on the phone provide image and video data that can be

5 Sometimes shake events are used to undo an action, which might be a questionable behaviour for some users. 6 Find a sample project Teslameter on the developer library, which reports the raw x, y, and z magnetometer values at https: //developer.apple.com/library/ios/samplecode/Teslameter/Introduction/Intro.html. 7 See video https://developer.apple.com/videos/play/wwdc2015-706/ about Security and Your Apps. From minute 35 on, the TouchID and the Secure Enclave including code samples to unlock the keychain are explained. 8 Find documentation at https://developer.apple.com/apple-pay/.

129 Sensor Possible State Battery level Already on the road for a longer time? Proximity Smartphone at the ear? Orientation10 Watching videos laying on the couch? Accelerometer On the move? Gyroscope Looking down? Barometer Bad mood, because of rainy weather? (Multi-)touch Can you press with your nose?

Table 10.2: Possible user state as detected by different sensors.

processed to extract information such as quick response QR codes or face detection9. In Table 10.2 some possible states are listed, based on different input from single sensors. Via external equipment – connected wireless, via Bluetooth (nowadays BTLE) or via cables11 – many more input and output options can be imag- ined12. You can find many more commercial sensors such as iBeacon13 on- line (see also Section 16.1.9), iCelsius14 or iHealth Pulse Oximeter, Zepp Golf Sensor, miCoach Smart Ball15.

9 Core Image provides face detection, as explained in the Core Image Programming Guide at https://developer.apple.com/library/ios/documentation/GraphicsImaging/ Conceptual/CoreImaging/ci detect faces/ci detect faces.html. 11 Find the External Accessory Framework Reference for , wireless and Bluetooth at https://developer.apple.com/library/ios/documentation/ ExternalAccessory/Reference/ExternalAccessoryFrameworkReference/index.html. 12 The site http://programmingiphonesensors.com/redpark/cable.html discusses the creation of your own sensors on Arduino boards and the connection them to the smartphone. 13 You can find indoor location technologies with iBeacons at http://estimote.com. 14 Bluetooth sensors for temperature measurement are available at http://www.icelsius.com. 15 The Accessories section in the Apple Online Store could be a first start http://www.apple.com/shop/iphone/iphone-accessories/health-fitness for finding this type of sensors.

130 10.2.3 Sample Code

When using sensors (such as camera, gyro) the hardware availability has to be checked first. Then the update frequency for motion sensors (gyro, accelerator, compass) can be set and a callback is registered. Finally, the sensor is activated. One should never forget to deactivate a sensor after usage for saving battery life. Listing 10.1 shows a simplified version (as alternative to a callback a timer is used to request motion data periodically) of how motion detection can be set up16 and Listing 10.2 shows an example of using the image sen- sor to take a photo. Many more sample code examples can be found at https://developer.apple.com/sample-code/wwdc/2015/?q=. The simplest way of providing haptic feedback is explained in Listing 10.3.

1 import CoreMotion 2 class ShakingViewController : UIViewController { 3 var trigger : Timer! 4 var cmmgr = CMMotionManager ( ) 5 6@IBOutletweak var lblCoreMotionInfo : UILabel! 7 override func viewDidLoad () { 8 super .viewDidLoad() 9cmmgr.startAccelerometerUpdates() 10 11 trigger = Timer.scheduledTimer( 12 withTimeInterval: 0.1, 13 r e p e a t s : true ) { 14 in 15 if let accDta = self .cmmgr.accelerometerData { 16 let v=accDta. acceleration 17 self .lblCoreMotionInfo.text = 18 ”x = ( v . x ) ny= ( v . y ) nz= ( v . z ) ” \ \ \ \ \ 19 } 20 } 21 } 22 @IBAction func stopUpdating ( sender : UIButton) { 23 trigger.invalidate() 24 }

16 Find a full-featured demo in the iOS Developer Library: MusicMotion https://developer.apple.com/library/ios/samplecode/MusicMotion/Listings/ MusicMotion MotionManager swift.html#//apple ref/doc/uid/TP40016160- MusicMotion MotionManager swift-DontLinkElementID 7.

131 Listing 10.1: Use one of core motion framework to add motion awareness to the app.

1 // Privacy Note : REQUIRED in Info . plist : 2 // ’ NSCameraUsageDescription ’ 3 class TakePhotoUIViewController : UIViewController , 4UIImagePickerControllerDelegate, 5UINavigationControllerDelegate { 6 var imagePicker : UIImagePickerController ! 7@IBActionfunc takeAPhotoNow ( sender : Any) { 8 self .imagePicker = UIImagePickerController() 9 self .imagePicker.delegate = self 10 if UIImagePickerController 11 .isSourceTypeAvailable(.camera) { 12 imagePicker.sourceType = .camera 13 else // cam not available in Simulator } { 14 imagePicker.sourceType = .photoLibrary 15 } 16 present(imagePicker, animated: true ,completion:nil ) 17 } 18 @IBOutlet weak var theImage : UIImageView! 19 func imagePickerController ( 20 picker: UIImagePickerController , 21 didFinishPickingMediaWithInfo info: [String : Any]) { 22 imagePicker.dismiss(animated: true , 23 c o m p l e t i o n : nil ) 24 let imgInfo = info[UIImagePickerControllerOriginalImage] 25 theImage.image = imgInfo as?UIImage 26 } 27 func imagePickerControllerDidCancel ( 28 picker : UIImagePickerController) { 29 d i s m i s s ( animated : true ,completion:nil ) 30 } Listing 10.2: Use one of the image sensors (cameras) to get an image.

1 import AudioToolbox 2 class ViewController : UIViewController { 3 func vibrate () { 4AudioServicesPlayAlertSound(kSystemSoundIDVibrate ) ; 5 } Listing 10.3: A simple, but very effective way of notifying users is to provide vibration feedback.

132 1#if targetEnvironment ( simulator ) 2print(”We are running on Simulator” ) 3#else 4print(”We are running on iPhone ( real device ) ” ) 5#endif Listing 10.4: Check, whether the application code is executed on the simulator or on a real device. Note that some sensors are not supported by the simulator. In List- ing 10.4 the way of checking the compilation target is shown.

10.2.4 More Frameworks Through a variety of network connections, many other devices can com- municate with smartphones. Whole frameworks exist for streaming audio (Inter-app Audio) and video data to other devices, or to play games via the central . Data can be exchanged directly or via cloud services using CloudKit or Handoff. information is managed us- ing the HealthKit framework (see Section 16.1.9) and HomeKit (see Sec- tion 16.1.9) allows users to control17 their home.

10.3 Required Reading

See the Apple documentation for the APIs of iOS devices https:// developer.apple.com/library/ios/documentation/UIKit/Reference/UIDevice Class/ for orientation, battery and proximity state (-notifications). In the Core Motion framework the events from the accelerometer, gyroscope and pedometer are presented at https://developer.apple.com/documentation/ coremotion.

10.4 Further Reading

The video18 https://developer.apple.com/videos/play/wwdc2015-706/ from the World Wide Developer Conference (WWDC) 2015 about Se-

17 The idea of HomeKit is to provide a unified API, instead of using vendor-specific home automation systems. 18 Hint: Find beneath the video the transcript and select the interesting sections, such as secure enclave, in the text.

133 curity and Your Apps is a first introduction into network security (TLS traffic), sandboxing and, above all, about the backgrounds and the usage of the Touch-ID technology.

10.4.1 Research Papers In Raja et al. (2014), the possible use of iPhones for biometric identifica- tion is discussed.

10.5 Review Questions

A good approach to answering following questions. Furthermore, explain the advantages AND disadvantages and optionally give source code ex- amples.

10.5.1 Some Basic Questions • Could you explain the usage, the advantages and disadvantages as well as the proper technical terminology for at least two actuators and five sensors?

• What is special in terms of the performance, power, connection-time, kind of connection, and security about BTLE?

• Which kinds of user feedback are supported by which actuators?

• Can you name types of user feedback that are not supported by ac- tuators?

10.5.2 Some More Advanced Questions • How does a haptic engine work internally? (For experts: How do ERM19/LRAs20 work, in comparison to HighDefinition-Haptics?)

• What are the (practical) limitations of the proximity sensor?

19 ERM Eccentric Rotating Mass Actuator 20 LRA Linear Resonant Actuator

134 Bibliography

Raja, Kiran B., R. Raghavendra, Christoph Busch, and Soumik Mondal (2014). “An Empirical Study of Smartphone Based Iris Recognition in Visible Spectrum”. In: Proc. 7th International Conference on Secu- rity of Information and Networks (SIN 2014). ACM, 239:239–239:246. ISBN: 9781450330336. DOI: 10.1145/2659651.2659704.

135 136 11 Security

“We must plan for freedom, and not only for security, if for no other reason than that only freedom can make security secure.”

[Karl Popper]

Summary: Security aspects, such as the use1/misuse2 of several cod- ing/cracking techniques on iOS are discussed and compared to Android where appropriate. Furthermore some tools for code analysis, for decrypt- ing and disassembling iOS apps are presented.

11.1 Introduction

Security aspects are related to every part of the system3, from the basic infrastructure up to the code in the app. Here we will focus on a limited number of selected security topics, without trying to cover the complete field. Typical defensive techniques with iOS include security mechanism such as authentication, auditing, authorisation/Kernel Authorisation (KAuth), Mandatory Access Control Framework (MACF), code signing, Apple Mobile File Integrity for sandboxing as well as privacy and data

1 We recommend reading the Section Secure Coding of the Apple Developer Library at https://developer.apple.com/library/mac/documentation/Security/Conceptual/ SecureCodingGuide/Introduction.html#//apple ref/doc/uid/TP40002415. 2 Selected Inter-process Communication (IPC) security problems are discussed in Xing et al., 2015. 3 For example, in September 2015 the XcodeGhost malware was successful, by infecting the Xcode IDEs downloaded by developers, and compromised many high-profile apps. See http://researchcenter.paloaltonetworks.com/author/claud-xiao/ for XcodeGhost and many posts about current iOS malware.

137 protection aspects. A few techniques – such as system integrity protection – are only available with macOS and not with iOS. The covered aspects of security are structured as follows: first we ex- plain the general iOS security concepts and – where appropriate – their differences to other mobile and desktop operating systems such as macOS, Linux, Windows, Android, then we present some examples for secure cod- ing with short demo code that should help to create more secure apps. Then we will look at a selection of tools for analysing code and app bundles. We will discuss a few (maybe still existing) iOS security issues and approaches to fix them. We point to some relevant papers and tutorials.

11.2 General Aspects of iOS Security

The iOS operation system and the available apps are and will never be 100% secure4. Creative attackers will always find ways to hack into sys- tems and crack applications. The important aspect is to be aware of this fact and make cracking the system as difficult as possible, by designing software with security in mind and improving code steadily. The tradeoff might be high cost or decreased usability and user acceptance, hence some- times pragmatic approach (to a degree) is inevitable. Different kinds of attacks can be named, such as memory-based, stack-smashing, and return- oriented programming. The defence (stack canaries, non-executable data, address space layout randomisation, control flow integrity) has to be con- sidered. If we are using a web view inside an app we have to consider Cross-Site Scripting (XSS), Cross-Site Request Forgery (XSRF), session hijacking or simple Structured Query Language (SQL) injections. Part of the security considerations is the testing of system components or the whole system (e.g. penetration testing).

11.3 The Operating System

The iOS Darwin operating system includes the XNU hybrid kernel5 which is based on the Mach kernel, components from FreeBSD and IOKit.

4 In 2016 the New York Times reported from Fake Retail Apps at http://www.nytimes.com/2016/11/07/technology/more-iphone-fake-retail-apps- before-holidays.html? r=2. 5 Find the source of XNU at https://github.com/opensource-apple/xnu or with https://opensource.apple.com.

138 11.4 Apps and the Operating System

Of main concern is the environment the smartphone apps are deployed to. The permissions enforced by the environment, compare security enhanced linux, can limit the interaction with the operating system and with other apps.

11.4.1 Deployment and Installation App distribution can only be done by authenticated persons via the official app store. With unique App IDs and related Bundle IDs for developers – including certificates assembled into profiles for development and deploy- ment – the author of each app is traceable.

11.4.2 App Separation Any custom app, and many of the system services, are sandboxed. That means a strong separation between the apps is achieved. This is based on permissions in iOS and differs from Android, where each app gets is assigned a Linux/Unix user ID. Bundled apps and extensions within bundles (all created by one devel- oper) can communicate via special IPC APIs provided.

11.4.3 IPC-Channels The possibilities for IPC are rather limited even within an App Group. Shared preferences and shared files are allowed. More sophisticated ap- proaches such as Mach Ports are sometimes only available on macOS and not for iOS, or they are available as private APIs only. As Berkley Stan- dard Distribution (BSD) sockets6 (Unix domain sockets) are in the end a kind of files, it might be possible to use them for IPC.

11.4.4 The Keychain Passwords can be stored in a central keychain and furthermore the keychain service can synchronise the keychains using cloud services.

6 At http://ddeville.me/2015/02/interprocess-communication-on-ios-with-berkeley-sockets Damien DeVille discusses the use of Berkeley sockets on iOS.

139 11.4.5 Biometric Sensor

Accessing credentials from the keychain by help of TouchID and FaceID7 is used for commodity reasons. The use of this biometric sensor is addi- tional, i.e. from time to time entering the passcode is still required.

11.4.6 Machine Learning and Privacy

To optimise usability in terms of keyboard experience, such as suggesting popular Emojis and new words, Apple asks for permission to monitor de- vice usage. With Local Differential Privacy, Apple (2017), the data will be randomised, collected and anonymised to protect personal and sensitive data.

11.5 Networking and Cloud

The network communication has to be secured by using proper protocols supporting secure channels and – if required – encrypted data storage on the (cloud) servers.

11.5.1 Networking

Networking should always be secured by using Transport Layer Security (TLS), especially when authentication is used and credentials being sent. High-level APIs are proposed and it is better to refrain from coding on the protocol level (unless required).

11.5.2 Cloud Services

Security issues affect thousands of people at the same time, so patches need to be applied fast! See for example the iPhone exploit8.

7 A camera creates an 3D depth map of the face and combines it with information from the infrared camera to a match user’s face, see https://support.apple.com/en-us/HT208108. 8 ZDNet reports on Find My iPhone exploit http://www.zdnet.com/article/apple-patches-find-my-iphone-exploit/.

140 11.5.3 Secure the Infrastructure

Besides networks, secure storage is highly relevant. See Chang et al. (2015) for a description of the MobiPluto approach, where Plausibly De- niable Encryption (PDE) (discussed already in Section 7) for the Linux- based Android Phones is discussed. For end-to-end encryption with Crypto Phones read Shirvanian and Saxena (2015).

11.6 Secure Coding

Developers on the iOS platform should take a look at the hints given in the Secure Coding Section9 of the developer documentation. The checklist included in these materials should be especially useful. Secure coding10 begins with a secure design (for example thread mod- elling), involves (automated) code reviews (static and/or dynamic code analysis) and many way of testing (unit testing, integration tests, white box fuzz testing and more). As Swift is a statically strong typed lan- guage, many errors are detected at compile time, as discussed with the Low Level Virtual Machine Infrastructure (LLVM). Nevertheless the use of additional, external tools – for checking code statically or even dynam- ically at runtime – is always of advantage. Besides the usability requirement to save state for the sake of users’ commodity, it is necessary to save (user related) data in a secure way. In various scenarios backups in the cloud, documents in the cloud (see Sec- tion 13) and automatic sync between devices are a must-have.

11.6.1 Web Application Security

Many apps include a browser (based on ). For these applications, the hints provided at the Open Web Application Security Project (OWASP) http://www.owasp.org/ are highly relevant.

9 https://developer.apple.com/library/mac/documentation/Security/Conceptual/ SecureCodingGuide/Introduction.html#//apple ref/doc/uid/TP40002415. 10 Find at the infosec institute many articles related to iOS security, for example Secure Coding Practices for iOS Development at http://resources.infosecinstitute.com/ios- application-security-part-25-secure-coding-practices-ios-development/.

141 11.6.2 Testing Unit testing on iOS devices is discussed along with UI testing in Sec- tion 5. Other test techniques such as fuzzy testing, external audits and others should be taken into consideration for high-security apps.

11.7 Forensics

A suggested way (Mahalik et al., 2018, p14 ff) to preserve as much infor- mation as possible when inspecting a device is to proceed with following steps:

• Step 1 Evidence Intake to document ownership, type of involved de- vices, but also the goals, the type of data searched for. • Step 2 Identification of, for example, the legal authority and details about devices. • Step 3 Preparation phase to do research and prepare tools. • Step 4 Isolation to disconnect from networks. • Step 5 Processing data by extraction from the phone. • Step 6 Verification is necessary to ensure accuracy and to be sure the data has not been modified. • Step 7 Documenting the tools and dates of the examination and in- formation about the collected data. • Step 8 Presentation by reporting (to the court). • Step 6 Archiving to preserve the data extracted.

11.8 Tools for Analysis

Encrypted apps (every binary in the app store is encrypted by the devel- oper and Apple with their signatures) have to be decrypted for analysis. See iOS application reverse engineering11 by Julien Bachmann from SCRT

11 Find iOS application reverse engineering at http://www.gsdays.fr/wp-content/uploads/ 2011/09/jbachmann iOS apps reverse engineering-1.0-gsdays1.pdf

142 or in Security Analysis12 by Ray Wenderlich. The application (ipa file) is a zip bundled directory (see Listing 11.113 and Figure 11.1). It contains the code signature (use codesign to check the signature as shown in List- ing 11.2), provisioning information, many dynamic libraries, configuration files, other resources such as translation and launch images and finally, the binary itself. The binaries can be disassembled/decompiled with several tools14. With standard Unix tools such as strings (see Listing 11.5) or lipo (see Listing 11.3), otool15 (see Listing 11.4) help to display information of object libraries. Debugging (see Listing 11.6) might – depending on the architecture of the binary – only work on hacked devices. 1unzip d ” opmnini” ˜/Downloads 2/OperaMini 10.1.1. ipa \ \ 3ls al opmini/ 4ls al opmnini/Payload/Opera Mini . app / \ 5# treeopmini 6# ... 7# languages: * .lproj/ 8# localizable strings 9# resources/assets: * .png 10 # c o n f i g s : * .plist 11 # CodeSignatue / CodeResources 12 # . . . Listing 11.1: Extract the *.ipa to view the file structure.

1codesign vd opmnini/Payload/Opera Mini . app / \ 2#Executable=.../opmnini/Payload/OperaMini.app/OperaMini 3#Identifier=com.opera.OperaMini 4#Format=bundlewithMachOuniversal(armv7arm64) 5# CodeDirectoryv=20200size=44923flags=0x0(none ) 6# hashes=2237+5location=embedded 7#Signaturesize=3487 8#Info.plistentries=37

12 iOS App Security and Analysis Part 2 at http://www.raywenderlich.com/46223/ios-app-security-analysis-part-2. 13 Until iTunes 12.7 the *.ipa files habe been located at /Music/iTunes/iTunes˜ Media/Mobile Applications/. Currently, one can save the ipa files using the cache of the Apple Configurator 2 tool as shown in https://ios.gadgethacks.com/how-to/ download-ipa-files-for-ios-apps-your-iphone-0184056/. 14 Find the commercial IDA Pro at Hex-Rays https://www.hex-rays.com/products/ida/, Hopper at http://www.hopperapp.com . 15 Object file displaying tool http://www.manpagez.com/man/1/otool/.

143 6#mgccpt ecpusubtypecp . . . caps e p y t b u s u p c pe puty c magic r e d a : e h armv7 # . Mach i n i m a # r e p o Mach 16 # : 15 armv7 . i n i m l a 14 r o e o p t o o armv7 . 13 # i n i m a r e p 12 o e 11 l i f 10 =836 =100 s e z e i l s i f =1 t n =6 ou s c e l u r s t n e =2 m e n r o i i u s q r e e r v l a n r e t Resources n I d e # l a e S 11 # 10 9#TeamIdentifier= U L 7 F 3 4 E 7 D N 9 8 7lipo6 5#O4#opmnini/Payload/OperaMini.app/ p3#Architectures2lipo e1c do p r m n i n i/P aM a y l o a d/O i p e r a n ia r e:a r m v 7a r m 6 4 etre odn aydfeetrsucsadcngrto lsbesides files configuration binary. actual and the resources different many holding rectories 11.1: Figure uptoeaii.armv7 operamini output noOpera info hnam7Opera armv7 thin hoperamini.armv7 h p niea P sadrcoysrcue nosubdi- into structured directory a is – IPA an inside – app The itn 12 hc ftecd ssigned. is code the if Check 11.2: Listing \ Mini in \ h a ie: file fat the \ Mini ii.ap/ app . Mini Oexecutablearm 144

pathfinder screenshot by john feiner 17 # 0xfeedface 12 9 0x00 ... Listing 11.3: Extract a specific binary out of a fat binary.

1otool loperamini.armv7 grep ”ENC” B1 A4 | 2#Loadcommand12 3# cmdLCENCRYPTION INFO 4# cmdsize20 5# cryptoff16384 6#cryptsize7864320 7# cryptid1 Listing 11.4: Find out if binary is encrypted. Check if cryptid=1.

1stringsoperamini.armv7 grep @ ” CLLocation | \ 2 # maybe no string / class Corelocation found , 3 # if binary is encrypted Listing 11.5: Strings inside unencrypted or decrypted binaries might be inspected with the strings tool.

1lldboperamini.armv7#worksonly if decrypted 2(lldb)l 27 327#import 428 529intmain(intargc,char* argv []) 630 { 731@autoreleasepool { 832intretVal=UIApplicationMain(argc,argv, 9 nil ,@” AppDelegate” ); 10 33 return retVal ; 11 34 } 12 35 } Listing 11.6: The debugger might reveal readable source code.

11.9 Jailbreaks

In *OS Internals: Security & insecurity / Jonathan Levin Levin (2018) several jailbreaks are analysed and gives deep insight of how to use com- binations of weaknesses and vulnerabilities to attack an iOS device.

145 iOS Version Jailbreak Explanations 6.x evasi0n Overcome KASLR. 7.1.x Pangu Axe Bypass the codesigning. 9.3.4 Pegasus A remote, private, hidden jailbreak mal- Trident ware. 10.2 Yalu Bypass (hardware) KPP. 10.3 Triple Fetch User mode sandbox escape. Supports XPC remote code execution. 11.1.2 QiLin Toolkit A pure data-only jailbreak.

Table 11.1: A list of selected Jailbreaks .

A timeline with the releases of iOS jailbreaks is available at the iPhone Wiki16. The value of zero day attacks might about a million of dollars17.

11.10 Selected Issues

Some final hints may help to qualify and quantify the level of security on current iOS devices: try to learn about known issues18 and attacks, in- cluding the history of detection and mitigation. For example, XARA the cross-app resource access attacks are discussed in Xing et al. (2015).

11.11 Required Reading

The approximately 60-page iOS Security iOS11 by Apple (2018) discusses System Security (booting, secure enclave, TouchID), Encryption (Pass- codes, Keychain), App Security (signing, extensions and app groups, ac- cessories), Network (TLS, Virtual Private Network (VPN), AirDrop), Ap- ple Pay (authorisation, NFC), Internet Services (AppleID, iCloud, Siri),

16 https://www.theiphonewiki.com/wiki/Timeline 17 Check out the prices for remote jailbreak with persistence for iOS at https://www.zerodium.com/program.html. 18 The Common Vulnerabilities and Exposures CVE lists many iOS security vulnerabilities at https://www.cvedetails.com/vulnerability-list/vendor id-49/ product id-15556/Apple-Iphone-Os.html, for example, more than 150 in each of the categories DoS, Code Execution and Memory Corruption in 2015.

146 Device Controls (mobile device management, remote wipe), Privacy Con- trols (location services, privacy policy). This is a good starting point for coding in respect of the suggested must-have security for each smartphone application.

11.12 Further Reading

For privacy information by Apple in Austria visit https://www.apple.com/ legal/privacy/de-ww/ the Apple Datenschutzrichtlinie.

11.12.1 Introduction Tutorials

A Security Analysis Tutorial by Ray Wenderlich can be found at http: //www.raywenderlich.com/46223/ios-app-security-analysis-part-2. Some governments have created dedicated security guides, such as the iOS Hard- ening Configuration Guide, which can be accessed at http://asd.gov.au/ publications/iOS8 Hardening Guide.pdf. This guide provides details on aspects such as risks and mitigations (jailbroken devices) or setup of fire- wall rules (listing apple host names and ports).

11.13 Selected Books on iOS Security

Hoog and Strzempka (2011) provide a sound introduction to iOS Foren- sics. Learn about Android and iOS forensics analysis in Mahalik et al. (2018) and iOS jailbreaks in Levin (2018).

11.13.1 Security and Insecurity

Advanced discussions of iOS security can be found at http:// technologeeks.com/. For example, following topics of security and insecurity are listed for their workshops: iOS architecture, boot se- quence, Mandatory Access Control (MAC) framework, Apple Mobile File Integrity, sandboxing, data protection, kernel and Kernel Extensionss (KEXTs), IPC and XPC, vulnerabilities and exploits.

147 11.13.2 Research Papers The paper Raghavendra, Raja, and Busch (2016) discusses biometric ad- vances with light field cameras for face and iris recognition. Read again the work of Xing et al. (2015) (already listed for Section 12 System Services) and focus on the one hand on the iOS weaknesses through Schemes and on the other hand on the various analysis tools used to inves- tigate them. Specifics of app sandboxing are discussed in Bucicoiu et al., 2015 (as already listed in Section 7 Persistency). Research has shown, that security is related to software quality. The paper of Habchi et al., 2017 explains the most relevant code smells and compares them to Android code smells.

11.14 Review Questions

A good approach to answering following questions. Furthermore, explain the advantages AND disadvantages and optionally give source code ex- amples.

11.14.1 Some Basic Questions • Can you explain the deployment certificates needed to put an app to the store?

• Which kind of security for which use-cases could you provide by using a keystore?

• Can you draw the architecture and flow during usage of the TouchID biometric sensor?

11.14.2 Some More Advanced Questions • Can you name the modes of IPC and known issues?

• Can you explain the weaknesses of Uniform Resource Locator (URL) schemes on iOS in comparison to Android?

• Can you explain the different approaches of separating apps on An- droid and on iOS and their consequences?

148 • Can you explain the terms and specific tasks of Secure Enclave?

• What do you know about the use and status of PDE?

Bibliography

Apple (Dec. 2017). “Learning with Privacy at Scale”. In: Apple Machine Learning Journal 1.8. URL: https://machinelearning.apple.com/2017/ 12/06/learning-with-privacy-at-scale.html. — (Jan. 2018). iOS Security iOS11. URL: https : / / www . apple . com / business/docs/iOS Security Guide.pdf (visited on 07/26/2018). Bucicoiu, Mihai, Lucas Davi, Razvan Deaconescu, and Ahmad-Reza Sadeghi (2015). “XiOS: Extended Application Sandboxing on iOS”. In: Proc. 10th ACM Symposium on Information, Computer and Com- munications Security (ASIA CCS 2015). ACM, pp. 43–54. ISBN: 9781450332453. DOI: 10.1145/2714576.2714629. Chang, Bing, Zhan Wang, Bo Chen, and Fengwei Zhang (2015). “Mo- biPluto: File System Friendly Deniable Storage for Mobile Devices”. In: Proc. 31st Annual Computer Security Applications Conference (AC- SAC 2015). ACM, pp. 381–390. ISBN: 9781450336826. DOI: 10.1145/ 2818000.2818046. Habchi, Sarra, Geoffrey Hecht, Romain Rouvoy, and Naouel Moha (May 20, 2017). “Code Smells in iOS Apps: How Do They Compare to Android?” In: Proc. 4th International Conference on Mobile Software Engineering and Systems (MOBILESoft 2017). IEEE Press, pp. 110– 121. ISBN: 9781538626696. DOI: 10.1109/MOBILESoft.2017.11. Hoog, Andrew and Katie Strzempka (June 16, 2011). iPhone and iOS Forensics: Investigation, Analysis and Mobile Security for Apple iPhone, iPad and iOS Devices. Syngress, p. 336. ISBN: 9781597496599. Levin, Jonathan (Aug. 2018). *OS Internals: Security & insecurity / Jonathan Levin. 2nd ed. Vol. 3. Technologeeks.com. ISBN: 991055531.

149 Mahalik, Heather, Satish Bommisetty, Oleg Skulkin, and Rohit Tamma (Jan. 23, 2018). Practical Mobile Forensics: A Hands-On Guide to Mastering Mobile Forensics for the iOS, Android, and the Windows Phone Platforms. 3rd ed. Packt Publishing, p. 402. ISBN: 1788839196. URL: https://books.google.at/books?id=3tRJDwAAQBAJ. Raghavendra, R., Kiran Bylappa Raja, and Christoph Busch (May 2016). “Exploring the Usefulness of Light Field Cameras for Biometrics: An Empirical Study on Face and Iris Recognition”. In: Trans. Info. For. Sec. 11.5, pp. 922–936. DOI: 10.1109/TIFS.2015.2512559. Shirvanian, Maliheh and Nitesh Saxena (Dec. 7, 2015). “On the Security and Usability of Crypto Phones”. In: Proc. 31st Annual Computer Secu- rity Applications Conference (ACSAC 2015) (Los Angeles, California, USA). ACM, pp. 21–30. DOI: 10.1145/2818000.2818007. Xing, Luyi, Xiaolong Bai, Tongxin Li, XiaoFeng Wang, Kai Chen, Xi- aojing Liao, Shi-Min Hu, and Xinhui Han (Oct. 2015). “Cracking App Isolation on Apple: Unauthorized Cross-App Resource Access on MAC OS X and iOS”. In: Proc. 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS 2015). ACM, pp. 31– 43. ISBN: 9781450338325. DOI: 10.1145/2810103.2813609.

150 12 System Services

“But they are useless. They can only give you answers.”

[Pablo Picasso on calculating machines]

Summary: The apps on iOS can and should make use of the built-in sys- tem services such as Address Book to retrieve the email addresses for con- tacting friends. With the Sharing API1 it is possible to communicate with other apps or with system services such as the media library. It is possible to post messages on social media networks. Ways of communication be- tween several iOS apps and between an iOS app and a watchOS app are discussed.

12.1 Introduction

This section is about functionality provided by system services, which can be accessed by a single application to enhance its functionality. The inter- process communication between apps and also the inter-device communi- cation to the external watchOS apps are shown. In doing this, we first sur- vey the available options and their general usage. Then we look at some examples with short demo code to develop your understanding. Finally, we point to some papers and tutorials.

12.2 System Services

System services include simple access to users’ contacts, () events and images via special framework APIs, but also lower level Inter-App

1 By using the built-in iOS API of UIActivity, one can share web pages, photos, location infos, contacts, or game high scores.

151 Communication (for example via AirDrop or special URL schemes) or Cloud services. Very special frameworks are, for example, the Multipeer Connectivity framework for communication with nearby devices and the Core Telephony framework which would allow implementation of VoIP apps.

12.2.1 Review of the iOS Layers System services are accessed through special frameworks, which are part of layers in a layered architecture. The main layers of the iOS operating systems are (top down): The layer, the Media layer, the layer and, finally, the Core OS layer. • Cocoa Touch Layer: includes for example high-level features for the user (touch) interaction • Media Layer: includes for example Graphics, Audio, Video, AirPlay • Core Services Layer: includes for example high-level service fea- tures • Core OS Layer: includes for example Bluetooth, External Acces- sories or Networking

12.2.2 Access to Photos and Contacts The typical use of standard and simple system services is explained with the example of accessing the user’s Address Book. As contact data is pos- sibly private and hence regarded as sensitive data, access must be granted explicitly.

Access to the Address Book You can search for contacts using the Contacts framework2 in the local address book, but also in the cloud (iCloud contacts). In Listings 12.1, 12.2 and 12.3 the user is asked for permission to access the contact store – if access is not already granted – and then the search is done by setting search predicates for a query.

2 Beginning with iOS 9 the Contact Framework has replaced the AddressBook framework used up to iOS version 8.

152 1 // Privacy : set NSContactsUsageDescription in Info . plist 2 import Contacts 3 class ViewController : UIViewController { 4 var contactStore : CNContactStore! 5 6 override func viewDidLoad () { 7 super .viewDidLoad() 8 self .contactStore = CNContactStore() 9 } Listing 12.1: The Contacts framework allows to access the address book of the user.

1@IBActionfunc searchForNames ( sender : AnyObject) { 2 let authorizationStatus = 3CNContactStore.authorizationStatus(for :.contacts) 4 switch authorizationStatus { 5 case .authorized: 6complHdlForContacts(accessOK:true ) 7 case .denied, .notDetermined: 8 self .contactStore.requestAccess(for :.contacts) { 9(gotAccess,accessError)> Void in 10 if gotAccess { 11 self .complHdlForContacts(accessOK: true ) 12 else } { 13 p r i n t ( ” Did not get access to contacts” ) 14 } 15 } 16 default : 17 self .complHdlForContacts(accessOK: true ) 18 } 19 } Listing 12.2: For privacy reasons, users have to be asked for permission to access the contacts, unless access is already granted.

1@IBOutletweak var searchField : UITextField! 2@IBOutletweak var results : UILabel! 3 func complHdlForContacts (accessOK:Bool) { 4 // back from ( possible ) background thread 5DispatchQueue.main.async { 6 if let terms = self .searchField.text { 7 let contacts = self .getContacts(searchTerm: terms) 8 var msgs = ” ”

153 9 for contact in contacts { 10 var msg = ” ( contact . givenName ) ” * \ 11 for addr in contact . emailAddresses { 12 msg += ” n t ( addr . value . description ) ” \ \ \ 13 } 14 msgs += msg+” n” \ 15 } 16 self .results.text = !msgs.isEmpty ? msgs 17 : ”No contacts matching ’ ( terms ) ’...” \ 18 } 19 } 20 } 21 func getContacts (searchTerm: String ) > [CNContact] { 22 var contacts :[CNContact] = [] 23 let predicate = 24 CNContact.predicateForContacts( 25 matchingName: searchTerm) 26 let keysToFetch = [CNContactGivenNameKey , 27 CNContactFamilyNameKey, 28 CNContactEmailAddressesKey] 29 do { 30 let resConts = try contactStore . unifiedContacts ( 31 matching: predicate, 32 keysToFetch: keysToFetch as [CNKeyDescriptor] ) 33 c o n t a c t s = r e s C o n t s 34 catch } { 35 p r i n t ( ” ( error ) ” ) \ 36 } 37 return contacts 38 } Listing 12.3: The actual search. The same privacy and security issues apply when accessing other data. Media access for sharing or editing photos or videos is handled in a similar way, i.e. the user is asked to give (grant) permission.

12.2.3 Sharing Text and images3 might be shared between services, apps and social frame- works using Activities. In Listing 12.4 and Listing 12.5 you can see ways of sharing textual and image data.

3 An additional way of sharing your data is to create Custom Activities as explained at http://nshipster.com/uiactivityviewcontroller/.

154 1@IBAction func shareSomeText ( sender : UIButton) { 2 let shareMyNote = ” Note : mr x requests help on task y . ” 3 let activityViewController = UIActivityViewController( 4activityItems:[shareMyNote], 5applicationActivities:nil ) 6activityViewController.popoverPresentationController? 7.sourceView=self .view 8 self .present(activityViewController, 9animated:true ,completion:nil ) 10 } Listing 12.4: Share some text with other apps.

1 // Note : add ” what . png” and ” flower . png” to your assets ! 2@IBAction func shareImages ( sender : UIButton) { 3 if let shareImage = UIImage(named: ” what” ), 4 let shareImage2 = UIImage(named: ” flower” ) { 5 let activityViewController = UIActivityViewController( 6activityItems:[shareImage,shareImage2], 7applicationActivities:nil ) 8activityViewController.excludedActivityTypes=[ 9UIActivityType.postToTwitter]// don ’ t share on TW 10 // UIActivityType . postToFacebook ] 11 activityViewController.popoverPresentationController? 12 . sourceView = self .view 13 self .present(activityViewController, 14 animated : true ,completion: { 15 p r i n t ( ” Back from image sharing screen” ) 16 ) } 17 else } { 18 p r i n t ( ” Sharing error : could not find” + 19 ” specified images in assets . ” + 20 ” Please add the images , then try again ! ” ) 21 } 22 } Listing 12.5: Save images to the photo library or share images with other applications or services.

12.2.4 Inter Process Communication To share data between apps, several approaches are viable. IPC enables processes to send each other data and messages. What is allowed (URL

155 schemes, audio data streaming4) and what is forbidden (pipes, fifos, sock- ets, shared mem) – and to what extent – can be looked up in the Apple Developer Documentation5. Normal applications are separated very strictly at the operation system level and cannot access each other’s resources. To circumvent this restric- tion, Application Groups are allowed to share files and directories as well as preferences (UserDefaults) and communicate via Mach Ports6. Those App Group Containers have to set up online at the Apple Developer Mem- ber Center and the corresponding apps must be configured via Capabilities to use the same container id.

URL Schemes If an application registers7 a new arbitrary URL Scheme in the info.plist file, the user is directed to this app from other apps. 1 <-- to open urls such as 2 slideshow: //show/me/sun.svg --> 3 4 ... 5 CFBundleURLTypes 6 7 8 CFBundleURLName 9 at.fhj.slideshowapp 10 CFBundleURLSchemes 11 12 slideshow 13

4 Find a demo application for audio data streaming by Apple at https://developer.apple.com/library/ios/samplecode/InterAppAudioSuite/Introduction/ Intro.html. 5 Inter-App Communication explained by Apple at https://developer.apple.com/library/ ios/documentation/iPhone/Conceptual/iPhoneOSProgrammingGuide/Inter- AppCommunication/Inter-AppCommunication.html. 6 Since iOS 8 apps within a Shared App Group are allowed to use XPC for Inter Process Communication IPC using the Mach Ports. Find further explanations at http: //ddeville.me/2015/02/interprocess-communication-on-ios-with-mach-messages/. 7 Find possible info entries, especially look for the URL schemes at https://developer.apple.com/library/ios/documentation/General/Reference/ InfoPlistKeyReference/Articles/CoreFoundationKeys.html#//apple ref/doc/uid/ 20001431-102207.

156 14 15 16 ... Listing 12.6: Register a custom URL Scheme slideshow:// by configuring a unique pattern for your app bundle id in the info.plist.

See Listing 12.6 for the XML entries necessary to register a cus- tom scheme in the iOS system. The application stated will be invoked, for example, when a user clicks on a made up custom link such as slideshow://show/me/sun.svg inside a html page in the browser.

12.2.5 Communication between watchOS and iOS Apps As already described in Section 2.2 The Apple Watch Communication with iOS, the framework WatchConnectivity enables data transfer between an iPhone and the Apple Watch.

12.3 Required Reading

Check out the Section Kernel and Device Drivers Layer at https: //developer.apple.com/library/mac/documentation/MacOSX/Conceptual/ OSX Technology Overview/SystemTechnology/SystemTechnology.html which is not specifically about iOS, but to lookup and review the meaning of following – some Mac specific, some general – terms: XPC, Mach, Darwin, BSD, Apple events (Mac specific), and file system events, kernel queues, kernel events, IPC, sockets, ports, streams, pipes, shared memory (general).

12.4 Further Reading

In this section we list some current research (papers) and tutorials.

12.4.1 Introduction Tutorials Find a tutorial for using the Contact Store at http://www.appcoda.com/ ios-contacts-framework/. An introduction tutorial to social me- dia sharing can be found at https://www.hackingwithswift.com/ example-code/uikit/how-to-share-content-with-the-social-framework-

157 and-slcomposeviewcontroller and https://www.hackingwithswift.com/ read/3/overview. How to set up the communication between iOS and watchOS is explained in https://www.ralfebert.de/tutorials/watchos2-watchkit- connectivity/.

12.4.2 Hooks to the Apple Developer Documentation Find the Core Services Layer of iOS (and a short description of a variety of frameworks) explained at https://developer.apple.com/library/prerelease/ ios/documentation/Miscellaneous/Conceptual/iPhoneOSTechOverview/ CoreServicesLayer/CoreServicesLayer.html#//apple ref/doc/uid/TP40007898- CH10-SW5.

12.4.3 Research Papers The operating system (iOS) should keep the apps’ resources separated, but Xing et al. (2015) have shown many shortcomings and inter app com- munication issues (IPC Interception and other cross-app resource attacks, XARA). Phua, Jin, and Kim (2017) discuss Gratifications using Facebook, Twit- ter, Instagram, or Snapchat to follow brands. Different platforms are suit- able for example for sharing problems, for showing affection, for commu- nity engagement and commitment.

12.5 Review Questions

A good approach to answering the following questions is to try to define and explain a given term in your own words, to state typical usage and domains (this is a solution for which problem). Then try to state possible alternatives for the technology and/or for this approach. Furthermore ex- plain the advantages AND disadvantages and optionally give source code examples.

12.5.1 Some Basic Questions • What are viable ways to communicate between apps? Which meth- ods are limited in which way and how are restrictions enforced?

158 • Can you sketch the steps for accessing contacts?

• How can communication from iOS to watchOS be implemented?

12.5.2 Some More Advanced Questions • Explain the basic concepts of selected IPC methods, such as Kernel queues, Signals, Sockets/Ports, Streams, Pipes, or Shared memory.

Bibliography

Phua, Joe, Seunga Venus Jin, and Jihoon (Jay) Kim (Feb. 2017). “Gratifi- cations of Using Facebook, Twitter, Instagram, or Snapchat to Follow Brands”. In: Telemat. Inf. 34.1, pp. 412–424. DOI: 10.1016/j.tele.2016. 06.004. Xing, Luyi, Xiaolong Bai, Tongxin Li, XiaoFeng Wang, Kai Chen, Xi- aojing Liao, Shi-Min Hu, and Xinhui Han (Oct. 2015). “Cracking App Isolation on Apple: Unauthorized Cross-App Resource Access on MAC OS X and iOS”. In: Proc. 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS 2015). ACM, pp. 31– 43. ISBN: 9781450338325. DOI: 10.1145/2810103.2813609.

159 160 13 Cloud Services

“We are all now connected by the Internet, like neurones in a giant brain.”

[Stephen Hawking]

Summary: Users love cloud services for commodity: they like to sync their data automatically between several devices. Furthermore, the cloud enables users to access their documents Through The Web (TTW). Addi- tional online service provide functionality such as video editing or backup and restore. A few years ago such services required local computers with proper software configured. This section gives an introduction of iCloud services from the developer’s view.

13.1 Introduction

This section about iOS cloud services is structured as following. First we explain the general concepts of selected cloud-based services. Then we present some examples, including short demo code, to help to create cloud- enabled apps. Finally, we refer to some relevant papers and tutorials.

13.2 Cloud

The term Cloud is used for multiple different aspects of cloud computing: Software as a Service (SaaS), Platform as a Service (PaaS) Infrastructure as a Service (IaaS). The iOS Security Guide states, that the iCloud services are hosted by Amazon (S3) and Google Cloud1.

1 Back in 2014 Apple acknowledged to have services running on Amazon Web Services (AWS) and Microsoft Azure.

161 h lu yoedvc spse oalteohrdvcs hs aacan data Those devices. other the all to 2 pushed is device one by cloud the other and synchronisation devices, ground taking multiple over after bookmarks immediately synchronising images photo, an uploading data, and settings smartphone the as such GBs), bers some example to For up (including files apps online. local available become hand, features documents) one iCloud the and On data user needed. related book- when syn- (contacts, and data cloud artefacts this the store in chronise to transparently users is smartphone iCloud of documents) Apple marks, the of idea overall The 13.2.1 Apple number for the infrastructure of customers. service cloud its idea the and rough of a importance gives the services and services of of list This displayed. is iCloud services. 13.1: Figure https://www nFigure In , Keynote CodIe n Concepts and Idea iCloud . keychain apple ), 13.1 ahor hwn h urn ytmsau fApple of status system current the showing A idFriends Find Mail . com/support/systemstatus/ h current the , evc osoepasswords, store to service Contacts ak eoepsil,to ntigsoe into stored Anything too. possible, become tasks Notes and ytmStatus System , , Calendar idiPhone Find 162 . h fc ut ( suite office the , , 2 Photos fApesrie,soe and stores services, Apple of nteohrhn,services hand, other the On . , akpadrestore and backup Documents Pages (arbitrary

, safari screenshot by john feiner Num- back- of add up to a considerable amount of space usage in the smartphones (see Figure 13.2). iphone screenshot by john feiner

Figure 13.2: Syncing data using cloud services might consume consider- able space on the devices.

Mind the vendor lock-in when using iCloud, because the service is for Apple Platform only. Other cloud sync services, for example DropBox3 or the use of private cloud services such as ownCloud4 could be the software of choice in a more open cross platform5 environment.

13.2.2 iCloud Development with CloudKit From the developer’s view it is necessary to use iCloud APIs to store (trans- parently) in the cloud. Cloud syncing is not trivial even for large compa- nies as Apple. Some reasons for many developers’ headaches when work- ing with the first versions of iCloud – introduced 2011 – are explained by theverge6 about performance issues, corrupted files and even data loss. During app development7 you have to enable Cloud Kit and setup a database schema to be able to subsequently fetch records from the cloud

3 https://www.dropbox.com 4 https://owncloud.org 5 In the Apple documentation the term cross platform means just iOS and macOS. 6 http://www.theverge.com/2013/3/26/4148628/why-doesnt-icloud-just-work 7 Find the CloudKit Quick start instructions at

163 and subscribe to changes of data. In Listing 13.1, Listing 13.2 and Listing 13.3 a CKRecord – a set of key value pairs representing an object – is stored to a cloud database8. The cloud database can be private or public, i.e. visible to other users on a read-only basis. 1 import CloudKit 2 import CoreLocation 3 4 class CloudPOIs : NSObject { 5 let container : CKContainer 6 let publicDB : CKDatabase 7 let privateDB : CKDatabase 8 9 override init () { 10 // check ( online ) entitlements / capabilities 11 // for current App ID 12 self .container = CKContainer.init ( 13 i d e n t i f i e r : ” iCloud . at . fh joanneum . drive2” ) 14 self .publicDB = container.publicCloudDatabase 15 self .privateDB = container.privateCloudDatabase 16 } Listing 13.1: A cloud kit container holds a public and a private cloud database and allows reading of custom defined records.

1 func fetchAllData ( nearLoc city : CLLocation , 2completion:@escaping(result : String ) > Void ) { 3 let locationPredicate = 4NSPredicate(format: 5 ” distanceToLocation : fromLocation:(%K,%@) < %f ” , 6 ” Location” ,city,10)// within 10 km 7 let query = CKQuery( 8recordType:” OpenMobileDevelopment POIs” , 9predicate:locationPredicate) 10 publicDB.perform(query , inZoneWith: nil ) { 11 resultingPOIs, error in 12 if let err = error { 13 p r i n t ( ”ERROR . Logged in with Apple ID? ( err ) ” ); \ 14 return 15 } https://developer.apple.com/library/ios/documentation/DataManagement/Conceptual/ CloudKitQuickStart/Introduction/Introduction.html. 8 Source taken from http://shrikar.com/ios8-cloudkit-tutorial-part-1/.

164 16 var results : [String] = [] 17 resultingPOIs?.forEach record in { 18 if let title = record.value( 19 forKey : ” Title” ) as?String { 20 let notes = record . object ( 21 forKey : ” Remarks” )?.description ?? ” ” 22 let loc = record . object ( 23 forKey : ” Location” )?.description ?? ” ” 24 r e s u l t s . append ( ” ( title ) ( notes ) ( loc ) ” ) \ \ \ 25 } 26 self .modify(poi: record, updatedTitle: ” ( title ! ” ) \ } 27 28 } 29 DispatchQueue.main.async { 30 completion( results.joined(separator: ” n” )) \ 31 } 32 } 33 } Listing 13.2: Fetching data from a cloud kit container database using a flexible query with a search predicate.

1 func modify ( poi record : CKRecord , 2updatedTitletitle:String) { 3record.setValue(title,forKey:” Title” ) 4 // CloudKit Dashboard : check write permissions 5 self .publicDB.save( record ) { 6(record,error)in 7 if let err = error { 8print(” Could not save modified entry : ( err ) ) ” ) \ 9 else } { 10 p r i n t ( ” Debug : record saved ! ” ) 11 } 12 } 13 } Listing 13.3: Writing modified data back into a cloud kit container needs the proper permissions set online in the CloudKit dashboard.

165 Key-Value Data in iCloud Storing key values in UserDefaults can be extended to cloud storage by using NSUbiquitousKeyValueStore9. See also Section 7.2.3 Shared Pref- erences.

Documents in iCloud With UIDocument10 syncing, arbitrary files can be synced via iCloud11. Find explanatory images at the iCloud Design Guide12. Note that every document is inside an iCloud Container, which holds additional meta info (Name, Modification Date, Size, File Type) necessary for synchronisation. The meta info is updated first, then the device automatically pulls changed bits from the cloud at appropriate times (e.g. when the app comes to the foreground).

Core Data in iCloud The databases (SQLite) behind the core data O/R-Mapping can be synced (transmitting incremental changes) to the cloud. Any changes trigger events, which can be observed if required. See Designing For Core Data in iCloud13.

13.2.3 CloudKit JS A version of the CloudKit API for JavaScript14 is available to be used by web applications. This way web apps can access data inside documents (with a custom schema) stored in the cloud.

9 https://developer.apple.com/library/ios/documentation/Foundation/Reference/ NSUbiquitousKeyValueStore class/index.html. 10 https://developer.apple.com/library/ios/documentation/DataManagement/Conceptual/ DocumentBasedAppPGiOS/ManageDocumentLifeCycle/ ManageDocumentLifeCycle.html 11 https://developer.apple.com/library/ios/documentation/General/Conceptual/ iCloudDesignGuide/Chapters/Introduction.html#//apple ref/doc/uid/TP40012094 12 See same URL Chapter 2: Designing for Documents in iCloud. 13 Again same URL Chapter 3: Designing for Core Data in iCloud. 14 https://cdn.apple-cloudkit.com/cloudkit-catalog/.

166 13.3 Required Reading

Read the iCloud Fundamentals at https://developer.apple.com/library/ ios/documentation/General/Conceptual/iCloudDesignGuide/Chapters/ iCloudFundametals.html#//apple ref/doc/uid/TP40012094-CH6-SW1, es- pecially about the structure of iCloud containers.

13.4 Further Reading

Syncing cloud services between many computers and the involved meta- data artefacts are discussed in Oestreicher (2014).

13.4.1 Introduction Tutorials Find a CloudKit tutorial at http://shrikar.com/ios8-cloudkit-tutorial-part- 1/.

13.4.2 Research Papers See the poster, Su et al. (2016), about the Edge Computing Paradigm mov- ing the computation from the cloud to the edge of the network. An experi- ment putting semantic reasoners on edge nodes in a smart city scenario for analysing latency and scalability.

13.5 Review Questions

A good approach to answering the following questions is to try to define and explain a given term in your own words, to state typical usage and domains (this is a solution for which problem). Then try to state possible alternatives for the technology and/or for this approach. Furthermore ex- plain the advantages AND disadvantages and optionally give source code examples.

13.5.1 Some Basic Questions • Can you explain at least three ways of syncing to iCloud?

167 • Which drawbacks in terms of performance, security, footprint and cross platform usage are to be expected with different cloud service providers?

• How can a cloud based password store be secure?

168 Bibliography

Oestreicher, Kurt (2014). “A Forensically Robust Method for Acquisition of iCloud Data”. In: Digital Investigation 11, Supplement 2. Fourteenth Annual DFRWS Conference, S106–S113. DOI: 10.1016/j.diin.2014. 05.006. URL: http://www.dfrws.org/2014/proceedings/presentations/ DFRWS2014-p12.pdf. Su, Xiang, Pingjiang Li, Yuhong Li, Huber Flores, Jukka Riekki, and Christian Prehofer (2016). “Towards Semantic Reasoning on the Edge of IoT Systems”. In: Proc. 6th International Conference on the Internet of Things (IoT 2016). ACM, pp. 171–172. ISBN: 9781450348140. DOI: 10.1145/2991561.2998469.

169 170 14 Notifications

“Message for you, Sir”

[Monty Python’s Holy Grail]

Summary: With local and remote Push Notifications users are informed about events such as new chat messages or current weather conditions without having the corresponding app running. Apple, Google and Mi- crosoft provide specialised cloud services to push messages in real time to the Android, iOS or Windows Phone devices. This section compares and explains the architectures and functionalities of those push services.

14.1 Introduction

This section about iOS notifications services is structured as follows: First we explain the general concepts of local and remote notifications (the latter also called cloud or push notifications). Then we compare the APNs to Firebase Cloud Messaging (FCM). Finally, we list some relevant papers and tutorials.

14.2 Notifications

Different types of notifications, such as alerts, sounds, or badge icons in- form the user of events even when a specific app is in the background or not running. Local Notifications are created on the device and can be scheduled in advanced. They are triggered by the operation system directly1. Remote

1 The similarities and differences between local and remote notification are explained at https://developer.apple.com/library/content/documentation/NetworkingInternet/ Conceptual/RemoteNotificationsPG/.

171 Notifications are triggered by specialised cloud services and are transmit- ted over the Internet to the smartphone. © Mike Ulm

Figure 14.1: Badge notifications indicate updates for apps, even if they run in the background or are not running at all.

We distinguish between alerts (or toast notifications) used, for exam- ple, when receiving a new text message or a new calendar appointment. Badge notifications are attached to the app icons, see Figure 14.1, and dis- play the number of new events2. Raw notifications – if available – are consumed in the background by the app for updating data, but they are not intended for user interaction and are not displayed on the UI. On Windows there are live tiles which can be modified to display tile notifications. Users may disable all, or specific notifications for any app on their devices to reduce distraction, or to save battery life.

2 On the Microsoft platforms, in addition to the number of new events glyphs can be added for display.

172 14.2.1 Local Notifications The advantage of local notifications is the independence from network and online services. The operating system – called Notification Center in iOS – engages different queues for storing relevant events in a central place and presenting them in a unified, consistent way. If users interact with a notification they can jump directly to the app responsible for handling the notification. In Listing 14.1 the user has to allow the app to send notifications by accepting a corresponding popup asking for permission. Then – as shown in Listing 14.2 – a local notification might be scheduled for delivery. 1 // see API for UNUserNotificationCenter : 2 // to request authorisation , declare types 3 // schedule local notifications , ... 4 import UserNotifications 5 6@UIApplicationMain 7 class AppDelegate : UIResponder , UIApplicationDelegate { 8 var window : UIWindow? 9 10 func application ( application : UIApplication , 11 didFinishLaunchingWithOptions launchOptions: 12 [UIApplicationLaunchOptionsKey: Any]?) > Bool { 13 let center = UNUserNotificationCenter . current () 14 // ask user for permissions 15 center.requestAuthorization(options: [.alert , .sound]) { 16 ( g r a n t e d , e r r o r ) in 17 if let err = error { 18 p r i n t ( ” Local Notifications not granted : ( err ) ” ) \ 19 else } { 20 p r i n t ( ” User granted local Notifications :)” ) 21 } 22 } 23 return true 24 } Listing 14.1: Registering for notification types to be fired by the app.

1 class ViewController : UIViewController { 2@IBActionfunc scheduleLocalNotif ( sender : Any) { 3print(”We prepare event to appear in 7 secs ...” ) 4 self .debugInfo.text = ” Adding Notification” 5 let content = UNMutableNotificationContent ()

173 6content.title=” Upcoming Event ! ” 7content.body=” Swift Coding Session” 8content.sound=UNNotificationSound.default () 9 10 // Deliver the notification in seven seconds . 11 let trigger = UNTimeIntervalNotificationTrigger( 12 timeInterval: 7, repeats: false ) 13 // Schedule the notification . 14 let request = UNNotificationRequest( 15 i d e n t i f i e r : ”UID077” , // unique notif id 16 c o n t e n t : c o n t e n t , 17 t r i g g e r : t r i g g e r ) 18 // check iOS in a few secs 19 let center = UNUserNotificationCenter . current () 20 center.add(request) 21 } Listing 14.2: Schedule a local notification to be send in a few seconds. Optionally, swipe down on the home screen to view notifications.

A local notification can be updated by scheduling a notification with the same notification identifier you specified when creating the original notification. To remove one or more notifications you can use removePend- ingNotificationRequests(withIdentifiers:).

14.2.2 Push Notifications Push notifications are received3 by the operating system from specialised cloud services to inform users of news even when the relevant app is not running. The app can be in the background or not started, so no polling (of several apps) is necessary to receive real time updates. For power consumption and usability reasons, developers should con- sider the lowest frequency for sending events without reducing the user experience. For example, a personalised message might be send immedi- ately to a single user, whereas more generic notifications to multiple users, such as updates to the weather conditions, work perfectly well if they are delivered at 30-minute intervals.

3 For iOS users need ports 5223 (443 for feedback on WiFi-only networks) open to get push notifications as listed at https://support.apple.com/en-us/HT203609. Additionally, for sending the port 2195 and for the feedback service the port 2196 have to be open. The target Apple servers are located at the 17.0.0.0/8 address block.

174 As mentioned above, background services could poll for information, but this is strongly discouraged by Apple because it would drain the bat- tery. With iOS 7, the background refresh functionality was added. With background refresh iOS schedules data fetch cycles for each app that suit the user’s app usage. From time to time, the app is launched by the sys- tem to synchronise with a background service. When the user opens the app, updated data is already available. The drawback from the developer’s point of view is, that you will never know when or how often the server is contacted and data gets synchronised. Only a setMinimumBackground- FetchInterval (and an Info.plist entry fetch for UIBackgroundModes) can be specified for the background fetch. For security reasons, sensitive or confidential data should never be in- cluded as payload for notifications. As for reliability, delivery is not guar- anteed by the cloud providers such as Apple, Google or Microsoft. A burden developers should consider is the overhead for enabling push notifications compared to background services. Many steps are required to set these up, including authentication tokens, certificates and deploy- ment profiles. Additionally, an infrastructure for the app server4 has to be provided to manage the registration of clients (devices), their tokens (e.g. channel URIs) and logic to trigger events. As sending events via the APNs could fail, the app server (or Provider in Apple’s terminology) can ask for confirmation of successful delivery5.

APS – Apple Push Notifications, A System Overview Apple with its APNs is one of several push notification providers. For a comparison of Apple, Google and Microsoft push notification services see Li, Du, and Chen (2013). The life cycle for the different push notification providers is similar. On a closer look, there are differences in functionality, in the way the service

4 Libraries such as PushSharp https://github.com/Redth/PushSharp/ ease the burden to send notifications to multiple push notification providers (Android, Amazon, Blackberry, Windows 8, iOS). 5 Up to iOS 8 the provider sends off an event and checks for success later at the APNs Feedback service. With iOS 9 the API is now based on HTTP/2, which supports the request/response model. Multiplexed requests (streams) and binary protocol as discussed in http://odecee.com.au/wwdc-2015-big-changes-to-apple-push-notifications/.

175 is set up and in the flexibility to interact with other systems. The flow of a notification at APNs (see Figure 14.2) is as follows:

• 1: An app registers for push notifications

• 1a: The app fetches a device token from the APNs server

• 1b: The app receives the token to store locally

• 2: The app sends the token to the app server (called provider) which can add the unique and specific token to its database and manage the notification targets.

• 3: An event occurs and the app server (the provider) selects device tokens which should be notified. Those tokens and the messages (max. 4 KB) are sent6 to the APNs server.

• 4 The APNs server pushes the notification to the app, or – if the device is offline – stores (the latest) one to be sent later.

APN 4

1a

1b 3 App 2

App-Server omnigraffle drawing by john feiner

Figure 14.2: Apple push notifications: retrieve token (1), register at app server (2), trigger an event on the app server and emit the notification (3) to APNs for delivery (4).

6 For each HTTP/2 POST request, the response tells about success, or error because of – for example – a no longer active device tokens.

176 The program logic for registering and for the callbacks to handle the device-token received and handling user notifications is documented in the Apple developer documentation7. The basic setup requires the app to register for remote (badge, sound, alert) notifications using registerForRemoteNotifications. With didRegis- terForRemoteNotificationsWithDeviceToken and didFailToRegisterForRe- moteNotificationsWithError you can react to success (Device Token re- trieved) or failure. Find a short, but very useful explanation with NW- Pusher8, a tool to simulate the app-server locally for test reasons.

FCM – Firebase Cloud Messaging, A Comparison FCM9 is the new incorporation of Google Cloud Messaging (GCM). It is very powerful in that it allows distribution of push notifications to sub- scribed topics and even Upstream Messaging. Upstream Messaging allows apps to react to notifications and send messages back to the FCM server. Furthermore, push notifications can be consumed by Chrome, Firefox and Opera10. The life cycle flow is as follows: • 1: Registering a client app whereby a unique number, the project number, for an API project is required (identifies the app server). • 1a: The client app obtains a registration token using the Instance ID API11. • 1b: The client app sends the registration token to app server. • 1c: The app server saves the registration token. • 2: The app server sends the message to the FCM server. An API key for authorisation is required. This API key has been generated in advance during project setup.

7 Find a push notification system overview at https://developer.apple.com/library/ content/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/ APNSOverview.html#//apple ref/doc/uid/TP40008194-CH8-SW1. 8 NWPusher is a macOS or iOS app which can connect to the APNs to send push notifications https://github.com/noodlewerk/NWPusher. 9 https://firebase.google.com 10 https://firebase.google.com/docs/cloud-messaging/js/client. 11 The instance ID represents a unique ID per instance of an Android or iOS app.

177 • 3: The FCM queues the message if the device is offline.

• 4: The FCM server sends the message to the device.

• 5: The app on the client device receives the message.

For upstream messages with the Extensible Messaging and Presence Proto- col (XMPP) connection server see https://firebase.google.com/docs/cloud- messaging/xmpp-server-ref. FCM and iOS clients limit the size of notifi- cation messages to 2KB (data messages up to 4KB). This limit can be circumvented by adding functionality that recontacts the FCM connection server after the client app is brought to the foreground and then fetches larger data blocks12.

Web Push Notification

For web pages and web apps the standardised Web Notifications API13 can be engaged.

Third-party Push Notification Services

In Section 14.2.2 you can find selected third-party push notification provider which simplify use of real time messaging.

14.3 Required Reading

The introduction of the official Apple iOS Developer Library at https://developer.apple.com/library/prerelease/content/documentation/ NetworkingInternet/Conceptual/RemoteNotificationsPG/APNSOverview.html#/ /apple ref/doc/uid/TP40008194-CH8-SW1 includes many figures to visu- alise the flow of data for delivering Push Notifications.

12 A section on the Google developer documentation handles the integration of Google libraries for iOS apps https://firebase.google.com/docs/cloud-messaging/ios/client. 13 Find the W3C API at http://www.w3.org/TR/notifications/ and an intro at Mozilla https://developer.mozilla.org/en-US/docs/Web/API/Notifications API.

178 14.4 Further Reading

A comparison of many third-party push notification providers can be found at: http://www.buzinga.com.au/buzz/mobile-push-notification- services/. Some popular ones are Urban Airship Engage https:// www.urbanairship.com/products/engage, Mobile Push Notification with PubNub https://www.pubnub.com/products/mobile-push/, Pushover https: //pushover.net, PushWoosh https://www.pushwoosh.com or PushSharp which can be found at Codeproject: http://www.codeproject.com/Articles/ 868000/Push-Notification-For-Windows-iOS-Android-Quick-Ea. Ama- zon Simple Notification Service (SNS) Mobile Push Notifications http: //docs.aws.amazon.com/sns/latest/dg/SNSMobilePush.html allows to send messages to Amazon Device Messaging (ADM), Apple Push Notification Service (APNs), Baidu Cloud Push (Baidu), Google FCM (previously known as GCM), Microsoft Push Notification Service for Windows Phone (Microsoft Push Notification Service (MPNS)), Windows Push Notifica- tion Service (Windows Push Notification Service (WNS)).

14.4.1 Introduction Tutorials For local notifications the tutorial iOS-8 Notifications in Swift14 is a good starter. The tutorial iOS-9 Background Execution15 discusses ways to refresh data by system-triggered background fetches. Check out the Apple Push Notification Tutorials at tutsplus16 still in Objective-C, or intertech17 in Swift. Find online (https://pusher.com) at the Pusher18 website a tutorial us- ing the pusher-library for sending remote notifications.

14.4.2 Research Papers Alghamdi and Seeling (2015) discuss push services that are aware of user’s activities in order to send notification to the right device at the right time.

14 http://shrikar.com/ios-8-notifications-in-swift/ 15 https://blog.newrelic.com/2016/01/13/ios9-background-execution/ 16 http://code.tutsplus.com/tutorials/setting-up-push-notifications-on-ios--cms-21925 17 http://www.intertech.com/Blog/push-notifications-tutorial-for-ios-9/ 18 https: //blog.pusher.com/how-to-send-ios-10-notifications-using-the-push-notifications-api/

179 If you are interested in formal specification and verification to improve security and reliability of push notification systems, we suggest the paper of Ding, Song, and Zhang (2014). A high-performance push notification implementation for mobile Internet in combination with IoT is presented by Pan et al. (2015).

14.5 Review Questions

A good approach to answering the following questions. Furthermore ex- plain the advantages AND disadvantages and optionally give source code examples.

14.5.1 Some Basic Questions • Can local and/or remote push notifications mitigate the problem of battery-draining background processes?

• Can you explain how Background Fetch works with iOS?

• Can you draft first the registration process and then the flow of a notification?

14.5.2 Some More Advanced Questions • Could you name the main differences between the Apple, Google and Microsoft Push Services?

• How and where can push notification services be attacked? Consider the security implications for developers, especially the app server with custom setup and custom logic.

180 Bibliography

Alghamdi, Yousef and Patrick Seeling (Jan. 2015). “Activity-Based Cloud Sending: Push Services for User Device Multiplicity”. In: Proc. 12th Annual IEEE Consumer Communications and Networking Conference (CCNC 2015), pp. 556–561. DOI: 10.1109/CCNC.2015.7158034. Ding, Junhua, Wei Song, and Dongmei Zhang (June 2014). “An Approach for Modeling and Analyzing Mobile Push Notification Services”. In: Services Computing (SCC), 2014 IEEE International Conference on, pp. 725–732. DOI: 10.1109/SCC.2014.99. Li, Na, Yanhui Du, and Guangxuan Chen (Dec. 2013). “Survey of Cloud Messaging Push Notification Service”. In: Proc. International Confer- ence on Information Science and Cloud Computing Companion (ISCC- C 2013), pp. 273–279. DOI: 10.1109/ISCC-C.2013.132. Pan, Zhaotai, Xiaoxing Liang, Yu Chen Zhou, Yi Ge, and Guo Tao Zhao (June 2015). “Intelligent Push Notification for Converged Mo- bile Computing and Internet of Things”. In: Web Services (ICWS), 2015 IEEE International Conference on, pp. 655–662. DOI: 10.1109/ICWS. 2015.92.

181 182 15 Machine Learning

“I am in the camp that is concerned about super intelligence.”

[Bill Gates]

Summary: Machine learning can be utilised by iOS apps. Models are created and provided by the developer and integrated into an app. A model trained with many data sets (for example trained with many images of fruits and a set of categories such as apple, pear and banana) can report for a new data (for example a golden delicious) the probabilities of belonging to a specific category (85% being an apple).

15.1 Introduction

The basic terms used in machine learning are discussed in Lewis and Den- ning (2018). Machine learning models – classifiers – can be created with various tools and frameworks such as Caffee1. For usage in iOS apps they need to be converted with Core ML Tools2 to a special internal format. To create models for classification in a first step training data sets are necessary. Raw image or text data for each category must be provided. Then, the training with image or textual test data is followed by a vali- dation step where the success rate of distinction between categories are calculated. For each data used in the training and validation phase the ex- pected category is known (or manually assigned) in advance. Finally, a trained model can be used in an app and report the probabilities of new input data belonging to (one of) the given categories.

1 Deep machine learning framework Caffee http://caffe.berkeleyvision.org. 2 Use Python package Core ML Tools https: //developer.apple.com/documentation/coreml/converting trained models to core ml for conversion of trained models.

183 15.2 Data Acquisition

Trainings data sets have to be generated to train a model. For later classify- ing new data (find out if an image of a fruit depicts an apple), the training data provides has to tell which category – out of a set of possible categories (for example the alternatives apple, pear, banana) – each data point of the test data belongs to.

15.2.1 Data Augmentation Optionally, the amount of input training data can be extended by augmen- tation of existing data. To increase the input training data set without pro- viding new data small variation to the input data can be applied to create further input data. For example an existing image can be resized, flipped, cropped or slightly distorted to create several new images out of a given one.

15.2.2 Training A model is trained by input of the training data set where for each data the category is specified.

15.2.3 Limitations This static approach of training the model once on macOS is limited. At the moment it is not possible to improve the model dynamically on the phone. Thus, reinforcement learning is not available.

15.2.4 Validation To find out the quality of a classifier, to validate a model, some of the generated data is set aside and not used for training. Instead, those part of the data is used for validation by reporting the percentage of properly classified data.

15.3 Image Classification

Image classification models can be created in an interactive way using the Assistant Editor in the Xcode Playground see Listing 15.1.

184 1 import CreateMLUI 2 import Foundation 3 import CreateML 4 5 let builder = MLImageClassifierBuilder () 6builder.showInLiveView() Listing 15.1: A model can be created interactively in the Xcode Playground showing an image classifier.

15.3.1 Working with Image Classifiers Any trained machine learning model added to an app can be used to deliver the probability of belonging to a specific category. Listing 15.2 shows the Swift code for classifying images with a given model.

1 let model = try ! VNCoreMLModel( 2 for :ImageClassifier().model) 3 let request = VNCoreMLRequest(model: model) { 4(request,error)in 5guardlet results = request . results 6 as?[VNClassificationObservation]else { 7fatalError(” Results Error” ) 8 } 9 var result = ” ” 10 for classification in results { 11 r e s u l t += ” ( classification . identifier ) ” \ 12 r e s u l t += ” ( classification . confidence 100)% n” \ * \ 13 } 14 // show on UI each of the propabilities 15 // of given image belonging to available categories : 16 self .resultLabel.text = result 17 } 18 let handler = VNImageRequestHandler( 19 cgImage:image.cgImage!) 20 guard (try? handler.perform([request])) != nil else { 21 f a t a l E r r o r ( ” Error on model” ) 22 } Listing 15.2: The probabilities of a given input image belonging to the different categories are reported by the machine learning model.

185 The dedicated hardware support of the neural engine (within the Bionic chip3) for machine learning frameworks within iPhones allows high per- formance classification even within real time video streams of augmented reality apps. The size of models (a simple one should not use more than 10 or 20 MB) might be a limiting factor for using machine learning classifiers in an app.

15.4 Natural Language Classification

Textual classifiers can be set up by using training data sets available in ta- bles. Find a simple spam detection example at https://www.appcoda.com/ create-ml/.

15.5 Required Reading

In a short article with question and answers the basic terms used in machine learning are discussed in Lewis and Denning (2018). The limitations of neuronal networks – (multi-layered) perceptron – are mentioned.

15.6 Further Reading

Find the open source machine learning framework TensorFlow at https: //www.tensorflow.org and the open source deep machine learning frame- work Caffee at http://caffe.berkeleyvision.org. Further alternatives are the Microsoft Cognitive Toolkit (CNTK) https://github.com/Microsoft/cntk, Pytorch https://pytorch.org (with AllenNLP https://allennlp.org) or Keras https://keras.io (which runs on top of TensorFlow, CNTK or Theano http: //www.deeplearning.net/software/theano/).

15.6.1 Introduction Tutorials The tutorial at AppCoda https://www.appcoda.com/create-ml/ explains the creation and usage of image classifiers with fruit data to tell images of bananas and apples apart. Furthermore, an example of spam detection using textual classification with tabular data is shown.

3 Find out the details about power consumption and performance of the A12 Bionic chip at https://www.apple.com/lae/iphone-xs/a12-bionic/.

186 15.7 Review Questions

A good approach to answering the following questions is to try to define and explain a given term in your own words, to state typical usage and domains (this is a solution for which problem). Then try to state possible alternatives for the technology and/or for this approach. Furthermore ex- plain the advantages AND disadvantages and optionally give source code examples.

15.7.1 Some Basic Questions • Explain the terms machine learning, classifications, features and model.

• Which data will you use for the validation phase of a model?

• Name and explain limitations (performance, size) of using models in smartphone apps.

15.7.2 Some More Advanced Questions • Can you tell the differences between neural networks, deep neural networks and recurrent neural networks?

• Explain the differences of supervised and unsupervised learning as well as reinforcement learning.

Bibliography

Lewis, Ted G. and Peter J. Denning (Dec. 2018). “Learning Machine Learning”. In: Communications of the ACM 61.12, pp. 24–27. DOI: 10.1145/3286868.

187 188 16 Selected Topics

“Learning and innovation go hand in hand. The arrogance of success is to think that what you did yesterday will be sufficient for tomorrow.”

[William Pollard]

Summary: This section contains pointers to further topics of the Apple ecosystem. Several technologies and techniques are available to extend iOS applications and/or to interact with other parts of the Apple infras- tructure. Finally, a few words are spend on possibilities for research, for companies and for marketing activities.

16.1 Introduction

This section provides an outlook on further iOS topics and is structured as follows: First we will look at some additional frameworks for iOS such as Extensions (action, audio-unit, content-blocker, custom-keyboard, document-provide, photo-editing, share, shared-links, Spotlight, today), search with Siri, the Game Center and Mobile Marketing and Payment Technologies (Wallets, Passbook, In-App Purchase, Newsstand1 ). Then we survey other parts of the Apple Ecosystem (For iOS and macOS and tvOS: networking with AirDrop, Cloud-Sync, Handoff2, Continuity, Air- Play, AirPrint, Inter-App Audio Streaming). A subsection covers Acces- sories (BTLE and WiFi accessories and their configuration, HealthKit, HomeKit, CarPlay and iBeacon). Especially for companies, the section iOS in Enterprises might be interesting, where we discuss Mobile Device

1 https://developer.apple.com/newsstand/. 2 https://developer.apple.com/handoff/

189 Management (MDM), in-house provisioning and per app VPN configura- tions. Finally, we look briefly at Research the ResearchKit and Marketing for the possibilities of App Store Analytics.

16.1.1 Extensions – Today Widgets With iOS App Extensions, service apps can be provided for other apps. These helper functionalities can be, for example, custom keyboards, photo- editing tools, or ways to share documents from any existing app. Extension points can be implemented with iOS even for the network and for .

Today View in Notification Center In the notification center, we find a section Today, which enables previews of live data of an app. These app extensions (of type Today) in the To- day view are called widgets.AToday Extension is technically similar to a normal iOS app. As it might hold rather complex logic it may consume considerable network and CPU resources. Users can switch off the widgets if they wish. 1 import NotificationCenter 2 3 class TodayViewController : UIViewController , 4NCWidgetProviding { 5 var cachedLabel = ” loading” 6@IBOutletweak var lblExam : UILabel ! 7 override func viewDidLoad () { 8 super .viewDidLoad() 9 self .lblExam.text = cachedLabel // 10 } 11 override func viewDidAppear ( animated : Bool) { 12 super .viewDidAppear(animated) 13 self .lblExam.text = cachedLabel 14 } 15 func widgetPerformUpdate ( completionHandler : 16 (@escaping (NCUpdateResult) > Void ) ) { 17 // e . g . fetch fresh data . if ok : 18 self .cachedLabel=” Math 7. Dec . ” 19 completionHandler(NCUpdateResult.newData) 20 // else : completionHandler ( NCUpdateResult . noData ) 21 }

190 Listing 16.1: This simple widget should be extended to provide updated information for the Today live preview. In Xcode you can add a new target Today Extension to an existing project. To extend the simple application logic shown in Listing 16.1 you might want to display data from web services. Therefore, you need to communicate data between app and app extension. For security reasons (permissions) it is required that you set up an App Group for the app and the app extension (the Today widget), because both are separate processes and processes have only a limited number of options for IPC. See the com- plete tutorial at AppCoda3. A technical note: For performance reasons, the system occasionally captures a snapshot of the contents of the widget’s view. The snapshot can be displayed fast. Later the snapshot is replaced with the widget’s live view.

16.1.2 App Search with Spotlight Search for iOS (and macOS) is implemented by a central ser- vice called Spotlight4. Smartphone apps can subclass NSUserAc- tivity5 to model searchable records and configure them (by setting isEligibleForSearch) to be indexed by the Spotlight search engine to present custom app data in the search results. A more flexible way of putting application data into the index is (see List- ing 16.2) to create CSSEarchableItems and add them with CSSearchableIndex.default().indexSearchableItems(). With deep links, users can jump from the search hits right to the most ap- propriate view of an app. 1 import CoreSpotlight 2 import MobileCoreServices 3 class ViewController : UIViewController { 4@IBOutletweak var searchTermTxtFld : UITextField ! 5@IBOutletweak var searchDescriptionTxtFld : UITextField! 6@IBActionfunc addSearchTerm ( sender : Any) { 7addAnotherSearchTerm(

3 http://www.appcoda.com/app-extension-programming-today/. 4 https://developer.apple.com/reference/corespotlight 5 https://developer.apple.com/reference/foundation/nsuseractivity

191 8term:searchTermTxtFld.text!, 9description:searchDescriptionTxtFld.text!) 10 } 11 12 func addAnotherSearchTerm ( term trm : String , 13 descriptiondes:String) { 14 let attributeSet = 15 CSSearchableItemAttributeSet( 16 itemContentType: kUTTypeText as String ) 17 attributeSet.title = trm // e . g .: Musil 18 attributeSet.contentDescription = des // e . g : Der Mann 19 20 let item = CSSearchableItem ( 21 u n i q u e I d e n t i f i e r : ” ims ( trm ) ” , // ims Musil \ 22 d o m a i n I d e n t i f i e r : ” at . fhj . ims” , 23 attributeSet: attributeSet) 24 25 CSSearchableIndex 26 . default () 27 .indexSearchableItems([item]) { 28 ( e r r o r : E r r o r ? ) > Void in 29 if let err = error { 30 p r i n t ( ” Indexing Err : ( err . localizedDescription ) ” ) \ 31 else } { 32 p r i n t ( ” ( item ) successfully indexed ! ” ) \ 33 } 34 } 35 } Listing 16.2: The search API allows to put data into the index for global search.

1 import CoreSpotlight 2@UIApplicationMain 3 class AppDelegate : UIResponder , UIApplicationDelegate { 4 func application ( 5 application : UIApplication , 6 continue userActivity : NSUserActivity , 7restorationHandler:@escaping([Any]?)>Void ) >Bool { 8 if userActivity . activityType 9==CSSearchableItemActionType { 10 if let uniqueIdentifier = userActivity 11 .userInfo?[CSSearchableItemActivityIdentifier] 12 as?String { 13 openDeepLinkForID(id:uniqueIdentifier)

192 14 } 15 } 16 return true 17 } 18 func openDeepLinkForID ( id : String ) { 19 p r i n t ( ” TODO move to page x and” ) 20 p r i n t ( ” display content with id= ( id ) ” ) // ims Musil \ 21 } Listing 16.3: The search results allow to jump into the corresponding app.

How to put an item with an id into the Spotlight search index is shown in Listing 16.2. The callback – the location in the code handling the given deep link holding an unique id – can be found in Listing 16.3.

16.1.3 Speech Recognition and Search

For security and privacy aspects, see the Section Siri and Dictation6 in the Apple documentation.

Siri Integration

The voice recognition function Siri allows users to input search terms and questions via audio. Siri delivers search results dependent on the coun- try7 Siri. For example, information about sports events, restaurants, movie showtimes and dictionary entries. Ethical aspects of artificial systems like Apple Siri, Amazon Echo Alexa and Microsoft Cortana are in need of pub- lic discussion. An example of SiriKit8 is the support for customers booking a ride using a Siri-based search. This location-based information is also available for customers while using maps (see Section 8.2.4 for Map Extensions).

6 http://www.apple.com/privacy/approach-to-privacy/ 7 Find availability at http://www.apple.com/ios/feature-availability/#siri. 8 https://developer.apple.com/library/content/documentation/Intents/Conceptual/ SiriIntegrationGuide/index.html.

193 16.1.4 Game Center If your gaming app is made for Game Center9, gamers are not only able to track their scores but they can also invite friends to a game and compete against each other. The infrastructure consists of the online portal, a Game Center app and a framework to be included into a game.

16.1.5 Mobile Payment For mobile payment – see Apple Pay10 – Apple created its own closed infrastructure. From the technical point of view it relies on the Secure Ele- ment chip running the Java Card platform. The communication is handled by the NFC protocols. Payment is authorised with the TouchID fingerprint sensor or with confirmation on a connected Apple Watch.

Wallets Wallets store your credit card information and replace the actual plastic . When you check out at a point of sale (Point of Sale (POS)) ac- cepting Apple Pay, users can confirm the purchase using TouchID. As of August 2017 Apple pay is limited to only 21 countries11. These include, for example, the US, Canada, Russia, China and Australia. In Europe Ap- ple Pay is only available in the Ireland, Isle of Man, Italy, United Kingdom, France, Spain, Switzerland and Vatican City.

In-App Purchase A common business model for developers is to make the app free and let the customers buy items within the app. Apple keeps 30% of the purchase price. How to code for the store within your app is described in the In-App Purchase Programming Guide12 where the StoreKit framework architec- ture is laid out. 9 The Game Center Programming Guide https://developer.apple.com/library/ios/documentation/NetworkingInternet/ Conceptual/GameKit Guide/Introduction/Introduction.html. 10 http://www.apple.com/apple-pay/ 11 Check the availability for Apple Pay at http://www.apple.com/ios/feature-availability/#apple-pay. 12 In-App Purchase with StoreKit framework https://developer.apple.com/in-app-purchase/.

194 Companies can place their advertisements inside your smartphone appl and pay for promotion on a per-view basis. Developers who integrate the iAd Framework13 can monetise their work by allowing ad banners to pop up inside their apps. Apple keeps 30% of the money companies pay for pre- senting ads. See also Section 16.1.13 for iTunes Connect of how to set up iAds for your apps.

16.1.6 Testing and Optimisation Unit testing can be automated with bots using macOS server and CI to build and run apps. As with Android alpha and beta testing, Apple has set up TestFlight to test apps with internal and external beta testers before final deployment to the store. Performance issues with the apps could be approached by code optimisation. The app store provides ways to optimise apps by means of App Thinning.

TestFlight The TestFlight app is available to download and install beta apps. Test- Flight apps will not be checked by Apple in-depth and furthermore they expire after 60 days14.

Code Optimisation Find many performance tips at the App Programming Guide15, mostly fo- cusing on reducing the app’s power consumption, memory footprint and network traffic. For concurrent programming see also the Concurrency Programming Guide16.

13 iAd Guide https://developer.apple.com/library/ios/documentation/UserExperience/ Conceptual/iAd Guide/Introduction/Introduction.html. 14 Details of TestFlight https://developer.apple.com/testflight/. 15 https://developer.apple.com/library/ios/documentation/iPhone/Conceptual/ iPhoneOSProgrammingGuide/PerformanceTips/PerformanceTips.html. 16 Grand central dispatch Grand Central Dispatch for concurrency https://developer.apple.com/library/ios/documentation/General/Conceptual/ ConcurrencyProgrammingGuide/Introduction/Introduction.html

195 App Thinning

App Thinning17 should reduce the network load by reducing the data to download. The Artwork in the Asset Catalog is tagged for each platform by the developer and sliced by the App Store. Delivery of selected re- sources to the apps can be deferred. Also, submission to the store might be done with archived apps including the Bitcode. Bitcode is an LLVM IR18 intermediate representation of the app. These apps can be optimised later on the App Store by Apple, i.e. compiled on demand before delivery.

16.1.7 Evolving Swift

Swift is published as Open Source Software under the Apache License 2.0. To spread Know-How in Swift, the programming language has already been ported to Linux. To make Swift (-frameworks) run on servers, the Server APIs Project19 was founded with a strong focus on HTTP/2 and WebSocket support.

16.1.8 Apple Ecosystem

To enable several Apple devices and operation systems to work together, Apple provides techniques for transparent iOS, macOS and tvOS (Apple TV) interaction.

AirDrop Network Connection

With macOS AirDrop, devices can – after they find each other via Blue- tooth using Bonjour20– set up an additional WiFi ad-hoc network to ex- change data, i.e. to drop a file from one computer to the other.

17 Find App Thinning on iOS, tvOS and watchOS at https://developer.apple.com/library/ ios/documentation/IDEs/Conceptual/AppDistributionGuide/AppThinning/ AppThinning.html#//apple ref/doc/uid/TP40012582-CH35. 18 LLVM Design and Overview at http://llvm.org/docs/. 19 https://swift.org/server-apis/ announced in October 2016 by Chris Bailey on https://swift.org/blog/server-api-workgroup/. 20 is Apple’s implementation of the zero-configuration networking protocol to discover, publish and resolve services.

196 Cloud-Sync

As already discussed in the Section 13 Cloud Services, applications can share settings, credentials and data over iCloud.

Continuity and Handoff

With Continuity users can auto-unlock their devices, can access a universal clipboard, create instant WiFi hotspots, accept phone calls on the desktop, send SMS from the desktop, or continue their work on documents with Handoff. Handoff – which can be included by programmers21 into any app – synchronises current work state of a document, for example a mail draft, over iCloud to another device nearby.

AirPlay and AirPrint

To mirror the current screen content or for streaming videos and photos to different devices such as AppleTV a technology called AirPlay22 is used. It works via peer-to-peer Wi-Fi connections. For wireless printing, AirPrint- enabled printers23 are announced, discovered and auto-configured via the AirPrint protocol, hence no installation or driver download is required.

Inter-App Audio Streaming

Audio app that are instruments, audio filters (effects) or audio generators are enabled to use each other’s functionality. An app (node) publishes an audio object to be used by other apps (hosts). Find the details of inter-app audio streaming explained at the Audio session Programming Guide24.

21 Handoff Programming Guide https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/ Handoff/HandoffFundamentals/HandoffFundamentals.html. 22 https://support.apple.com/en-us/HT204289 23 List of AirPrint enabled printers can be found at https://support.apple.com/en-us/HT201311. 24 Use the AVAudioSession API for streaminghttps://developer.apple.com/library/ios/documentation/Audio/Conceptual/ AudioSessionProgrammingGuide/Introduction/Introduction.html

197 16.1.9 iOS Accessories and MFi Accessories (i.e. custom hardware connected via cable or Bluetooth, for example, external speakers, keyboards, headphones, chargers, or HomeK- it/AirPlay/CarPlay devices) have to be certified for use with Apple devices. Hardware manufacturers register at Apple classes for their devices and de- velopers use this information and the Accessory-Framework25 to set up an EASession first and then transfer data back and forth via streams (NSInput- Stream / NSOutputStream).

Made For iPhone – MFi Apple is very strict with external accessories. Only hardware vendors en- rolled26 in the MFi licensing program are allowed to create AirPlay audio accessories for iOS devices.

CarPlay A specialised accessory – a car – can be accessed via the CarPlay pro- tocol. Automobile manufacturers27 allow multimedia and routing infor- mation from iOS devices to be displayed in displays built into a car. Car manufacturers, unlike normal developers, can also provide special apps to access dedicated car features.

HomeKit The HomeKit28 enables remote control of accessories at users’ homes, e.g. switching off the lights or closing the garage door. Optionally, the Ap- ple TV box can serve as hub and enables remote access via the Internet. The protocol is Apple-specific and the hardware vendors have to enrol in the MFi program to sell compliant devices. Developers use the Home- Kit framework29 and allow the users to set up Homes, Rooms (optionally

25 External accessories https://developer.apple.com/library/ios/featuredarticles/ ExternalAccessoryPT/Introduction/Introduction.html. 26 Enrolling to MFi program at https://mfi.apple.com/MFiWeb/enroll.action. 27 List of models supporting CarPlay http://www.apple.com/ios/carplay/available-models/. 28 HomeKit is introduced at https://developer.apple.com/homekit/. 29 HomeKit Framework at https://developer.apple.com/library/ios/documentation/ HomeKit/Reference/HomeKit Framework/index.html.

198 grouped into Zones) with Services provided by each accessory. If Blue- tooth or Wi-Fi is not available, HomeKit communicates via Push Notifica- tions and iCloud.

HealthKit Using the HealthKit30 Framework, users can record and share their health information. Sharing personal health data with your doctor could be ben- eficial, but users should think carefully about this as privacy violations are possible. Apple claims that health data is never shared between devices (via iCloud) and sensitive data is separated in two databases (health data and operational meta-data) available only after unlocking the phone.

iBeacon Detecting the current location with good accuracy is difficult indoors. With iBeacon31 – BTLE devices powered by a battery and emitting radio signals with their id for more than one year – location-aware apps can be imple- mented. For example, region monitoring or determining the approximate distance to a beacon is possible. The operating system can constantly scan for BTLE in the background (searching continues after app termination, even after rebooting the system) and start an app if the device is in the range of a beacon. 1 import CoreLocation 2 class ViewController : UIViewController , 3CLLocationManagerDelegate { 4 let locationManager = CLLocationManager () 5 // a family of beacons is a ” region” 6 let region = CLBeaconRegion( proximityUUID: UUID( 7uuidString:” B9407F30 F5F8 466E Aff9 25556B57FE6D” )!, 8identifier:” Estimotes” ) 9 var searching=false 10 @IBAction func searchBeacons ( sender : Any) { 11 if !searching { 12 searchBtn.setTitle( ” Stop seaching ...” , for :.normal)

30 HealthKit Framework https://developer.apple.com/library/ios/documentation/HealthKit/Reference/ HealthKit Framework/index.html#//apple ref/doc/uid/TP40014707. 31 Find a document explaining the features and limitations of iBeacons at https://developer.apple.com/ibeacon/Getting-Started-with-iBeacon.pdf.

199 13 if (CLLocationManager.authorizationStatus() 14 != CLAuthorizationStatus.authorizedWhenInUse) { 15 locationManager.requestWhenInUseAuthorization() 16 } 17 resultTextView.text=” Search started ...” 18 locationManager.startRangingBeacons( in :region) 19 else } { 20 searchBtn.setTitle( ” Start seaching for iBeacons ! ” , 21 for :.normal) 22 locationManager.stopRangingBeacons( in :region) 23 resultTextView.text=” Click to search again . ” 24 } 25 searching = !searching 26 } 27 override func viewDidLoad () { 28 super .viewDidLoad() 29 locationManager.delegate = self 30 } 31 func locationManager ( manager : CLLocationManager , 32 didRangeBeacons beacons: [CLBeacon], 33 in region : CLBeaconRegion) { 34 p r i n t ( beacons ) 35 let knownBeacons = beacons . f i l t e r { 36 $0.proximity != CLProximity.unknown } 37 let closestBeacon = knownBeacons[0] as CLBeacon 38 print(closestBeacon.accuracy) // or . proximity 39 resultTextView.text=” ( closestBeacon . description ) ” * \ 40 } Listing 16.4: Ranging known iBeacons by their Universally Unique Identifier (UUID) nearby. Using the major and minor numbers provide increased accuracy, but are optional.

In Listing 16.4 the Core Location Manager starts ranging for beacons in the vicinity and in the callback the proximity and accuracy of an iBea- con can be determined. Note that for privacy reasons, authorisation must be requested. A company that decides to use an iBeacon infrastructure configures them with major and minor (16 bit, i.e. unsigned integer values between 0 and 65535) additionally to the proximity UUID (128bit, often given as 32 hexadecimal digits) each. The first could be used to identify a store, the second the place within the store. Helpful callbacks are in this case the didEnterRegion and didExitRegion callbacks.

200 Development hints For development reasons one might turn an iOS device into an iBeacon32.

16.1.10 iOS in Enterprises Enterprises have special needs to safeguard company data. Some of the options are to use Mobile Device Configuration Profiles, enforce use of VPN, and In-house Provisioning for apps.

Mobile Device Management (MDM) Advanced (pre-)configuration33 of new devices and remote wipe are cru- cial for administrators when hundreds of iPhones and iPad are involved. For mobile MDM XML based profiles34 are generated, signed and sent to the smartphones. This way the phones are set to specific settings such as Wi-Fi, VPN, email, LDAP, Calendaring Extensions to WebDAV (CalDav) and printers or restrictions are applied to various device features. Option- ally, large companies might order pre-configured hardware, with company- specific apps pre-installed. As an alternative to restricting network and apps this way, third-party security containers can be used.

Device Configuration on Mac with Configurator The Apple Configurator 235 provides a command line tool cfgutil for au- tomation.

In-House Provisioning If deployment of apps is not viable via the App Store36, for example for security and privacy reasons, apps can be transferred to smartphones via

32 https://developer.apple.com/library/content/documentation/UserExperience/ Conceptual/LocationAwarenessPG/RegionMonitoring/RegionMonitoring.html. 33 With the Device Enrolment Program (DEP) new devices could be deployed preconfigured. See deploy.apple.com. 34 https://developer.apple.com/library/content/featuredarticles/ iPhoneConfigurationProfileRef/Introduction/Introduction.html. 35 https://itunes.apple.com/us/app/apple-configurator-2/id1037126344?mt=12 36 With the Volume Purchase Program (VPP) educational institutions and companies can buy larger number of apps and use redemption codes.

201 cable. This solution has its limits when employees are not nearby or the installation must be done on many devices. Then, in-house provisioning could be a solution, in which the company sets up an app store of its own.

Personal VPN To secure the network two approaches are common. A virtual private net- work (VPN) can be set up for all network connections to route all traffic through the company’s firewall and malware scanners. The alternative is a per app VPN, where a single app routes all its traffic through a VPN transparently. Optionally, accessing specific domains in Safari a VPN con- nection can be enforced.

16.1.11 Scientific Research and iOS Several research methods require data collection. An example is conduct- ing surveys with many participants. Apple introduced the open-source framework named ResearchKit37 dedicated to scientific data collection. The iOS APIs allows developer to create apps38 for sending tasks to users and collecting the results. For field experiments in the medical field, HealthKit and ResearchKit can work together and exchange data.

16.1.12 Schools and Educational Institutions Apple allows to download apps and content (see VPP), manage devices (see MDM and DEP) and manage accounts (Apple-IDs) for students with School Manager39 and supports content creation as documented in the Ed- ucation Development Guide40.

16.1.13 App Store and iTunes Connect The Apple App Store is the central distribution platform for all apps. Suc- ceeding in selling iOS apps on a larger scale might involve just luck – or proper marketing. Developers, managers and marketing people use iTunes

37 Find some ResearchKit GUI examples at http://researchkit.org/hig/index.html. 38 A good starting point could be the demo ORKCatalog app available at https://github.com/ResearchKit/ResearchKit/tree/master/samples/ORKCatalog. 39 Find details about School Manager at https://school.apple.com/. 40 https://itunes.apple.com/us/book/education-deployment-guide/id1092741957

202 Connect41 to manage users and roles, set up App Store information for every app, track sales, retrieve statistical data of app usage and check fi- nancial reports. Of special interest for developers is the subsection App Analytics, where several metrics, such as the number of views in the Store, the App units sold and the euros earned are listed per deployed app and country. Possibly – if users choose to allow it – usage activity or crash logs can be inspected and analysed.

16.2 Non-technical Remarks – Ethics

Customers rarely care about the effects of choosing the one or the other products. Unfortunately, too often the same holds for programmers. Ques- tions arise about the responsibility of large companies for society. Think of questions such as (not) paying a fair amount of taxes, of (not) paying attention to environmental consequences, of (not) accepting child labor. Common Criticism of Large Companies:

• Large companies pay lower rates of42 taxes (relative to small, local companies).

• Large companies treat their suppliers unfairly, even accepting child labor, to their earnings.

• Large companies exploit the planet43 and fulfil only the minimum ecological standards required by law.

• Mega companies form effective cartels/monopolies, holding patents, suing competitors and are not as innovative as they claim (in com- parison to many small and more agile companies).

41 Check out the online portal iTunes Connect at https://itunesconnect.apple.com. 42 Check out several news items about avoidance of taxes by Apple http://www.forbes.com/sites/kellyphillipserb/2016/08/30/apple-found-to-have- received-illegal-tax-benefits-ordered-to-pay-14-5-billion-in-back-taxes. 43 Greenpeace is ranking companies with metrics such as transparency, policy and efficiency http://www.greenpeace.org/usa/global-warming/click-clean/.

203 What Large Companies (such as Apple) Claim

• Apple claims to take care of the environment44 as, for example, by having about about 93% of their facilities running on renewable en- ergy (2015), minimising devices’ power consumption in sleep mode (0.5 watts for a MacBook), avoiding toxic material in the products (such as PVC) and recycling most of the materials.

• Apple claims to monitor45 fair working conditions for about 1.6 mil- lion employees working at their suppliers.

Who should we trust, and to what extend? How can we check and validate the claims?

16.3 Required Reading

See Search Best Practices46 for integrating iOS Search APIs. Find out what Apple suggest concerning efficiency, speed of a search, which content to index and how to present search results.

16.3.1 Introduction Tutorials

The tutorial iOS 8: Creating a Today Widget47 explains step by step the implementation of a widget in the Today section of Notification Center. Chris Wagner explains in iOS 9 App Search Tutorial: Introduction to App Search48 the code for presenting custom app data in search results.

44 Find the environmental reports for the last years at http://www.apple.com/lae/environment/reports/. 45 Find the supplier responsibility documents at http://www.apple.com/supplier-responsibility/. 46 Find iOS Search API Best Practices and FAQs at https://developer.apple.com/library/ios/technotes/tn2416/ index.html. 47 See http://code.tutsplus.com/tutorials/ios-8-creating-a-today-widget--cms-22379. 48 Find App Search Tutorial at http://www.raywenderlich.com/116608/ios-9-app- search-tutorial-introduction-to-app-search.

204 16.4 Further Reading

16.4.1 Selected Articles on History Randy H. Katz from the University of California, Berkeley reviews the Wireless Information Systems (Katz, 2015) with six challenges from 1994.

16.4.2 Research Papers – Future Gollakota and Kravets (2015) present four papers about even smarter phones hitting the market in the near future. One of those is about a system called Tango by Gordon et al. (2015) which allows sharing of computing power between smartphone and server.

16.4.3 Research Papers – Ethics Walsh (2016) discusses the need for laws to prevent artificial intelligence systems from being mistaken for humans.

16.4.4 Research Papers – Security Liu et al. (2016) discuss security aspects of iBeacon transmitters.

16.4.5 Apple Documentation for Enterprises • Enterprise Blueprint – going mobile with iOS: https://itunes.apple.com/ us/book/enterprise-blueprint/id1080432997

• iOS Deployment Primer: How to manage many iOS devices https: //itunes.apple.com/us/book/ios-deployment-primer/id1114173122.

• iOS Technical Training: skills to perform large-scale deployments of iOS deviceshttps://itunes.apple.com/us/book/ios-technical-training/ id1034669586

16.5 Review Questions

A good approach to answering following question is to try to define and explain a given term in your own words, to state typical usage, domains

205 (what problem is this a solution for?), and alternatives. Furthermore, ex- plain the advantages AND disadvantages and optionally give source code examples.

16.5.1 Some Basic Questions • When and how is the today view of iOS updated with new data?

• Siri can be used for Spotlight search on the smartphone. How does it work offline (in comparison to online) and which data is transmitted, processed and returned.

• Can you draft an architectural overview of game center collabora- tion?

16.5.2 Some More Advanced Questions • Can you name several ways for companies to secure data on the devices when allowing their employees to Bring Your Own Device (BYOD)?

• Describe some arguments why custom keyboard extensions and HealthKit data are considered to be critical in terms of privacy and security.

206 Bibliography

Gollakota, Shyamnath and Robin Kravets (Dec. 2015). “Smartphones: Getting Smarter Every Day”. In: GetMobile: Mobile Comp. and Comm. 19.3, pp. 9–9. DOI: 10.1145/2867070.2867074. Gordon, Mark S., David Ke Hong, Peter M. Chen, Jason Flinn, Scott Mahlke, and Zhuoqing Morley Mao (Dec. 2015). “Tango: Accelerating Mobile Applications Through Flip-Flop Replication”. In: GetMobile: Mobile Comp. and Comm. 19.3, pp. 10–13. DOI: 10.1145/2867070. 2867075. Katz, Randy H. (Dec. 2015). “Adaptation and Mobility in Wireless Infor- mation Systems: After 20 Years”. In: GetMobile: Mobile Comp. and Comm. 19.3, pp. 5–8. DOI: 10.1145/2867070.2867072. Liu, Chenhao, Peng Zhao, Kaigui Bian, Tong Zhao, and Yan Wei (June 2016). “The detection of physical attacks against iBeacon transmit- ters”. In: 2016 IEEE/ACM 24th International Symposium on Quality of Service (IWQoS), pp. 1–10. DOI: 10.1109/IWQoS.2016.7590453. Walsh, Toby (June 2016). “Turing’s Red Flag”. In: Communications of the ACM 59.7, pp. 34–37. DOI: 10.1145/2838729.

207 208 List of Figures

1.1 The different sections of the Xcode IDE: Toolbar with the Assistant Editor, Navigation Area, the Main Editor pane, Debug Area, and a Utility Area...... 4 1.2 The course for iOS development with Swift at Stanford University by Paul Hegarty...... 7

2.1 The philosophy of Apple is to provide a tight integrated user experience throughout their – very often closed – ecosystem...... 16 2.2 Since watchOS 2 the extensions reside in the Apple Watch. 17 2.3 In the file system find the iOS App, the WatchKit App (holding the storyboard with the graphical interface for the watch app) and the WatchKit Extension (with the logic for the watch app)...... 17 2.4 Same concept of code–GUI connections: Outlets and Ac- tions...... 18 2.5 Possible views on the Apple Watch include so called com- plications, which represent small information elements di- rectly included in the watch faces...... 19 2.6 On notification(s) a preview with static or dynamic data is presented immediately without starting up the app. . . . . 19 2.7 Limited space for the GUI which forces developer to spare buttons and text...... 19 2.8 A digital crown, a button and a touchscreen supporting normal and force touch...... 20 2.9 Set the base class of your Interface Controller in the Sto- ryboard by dropdown-selection of the newly created new Swift class...... 21

209 2.10 An empty circle indicates a missing connections to an UI element...... 22 2.11 A yellow warning sign indicates a missing connection to the source code...... 22 2.12 Sharing settings such as system behaviour and privacy. . . 28 2.13 Speech recognition for iPhone and watch are set at once. . 28 2.14 The detailed settings for the watch have to be configured on the iPhone...... 28 2.15 Security settings allow or disallow an app to use the sensor of the watch...... 34 2.16 Four numbers for a short passcode are standard for unlock- ing the watch. But one can create a long passcode for en- hanced security...... 34 2.17 Unlocking can work on watch and phone at the same time. 35

3.1 Swift can be run as REPL in the Linux or macOS terminal. 41

5.1 Transitions trigger events and custom code in the delegate can be executed for those app life cycle methods...... 71 5.2 Set dataSource and delegate for a table...... 74 5.3 Auto Layout allows the specification of many constraints for (nested) UI elements...... 77 5.4 Push views onto the view stack and unwind back...... 79

6.1 Typical usage of async queues to speed up performance with concurrent downloads...... 89 6.2 Prevent Data Race with sync queues. Concurrent access to in-memory data (structures) is prohibited and corruption is not possible anymore...... 89

7.1 Entity Relationship Designer allows you to generate source code...... 103 7.2 O/R-M Object Relational Mapping ORM with CoreData.. 103

8.1 Zooming in on a map after a few seconds, then adding an annotation with a message attached...... 113

210 11.1 The app – inside an IPA – is a directory structured into subdirectories holding many different resources and con- figuration files besides the actual binary...... 144

13.1 A dashboard showing the current system status of Apple services...... 162 13.2 Syncing data using cloud services might consume consid- erable space on the devices...... 163

14.1 Badge notifications indicate updates for apps, even if they run in the background or are not running at all...... 172 14.2 Apple push notifications: retrieve token (1), register at app server (2), trigger an event on the app server and emit the notification (3) to APNs for delivery (4)...... 176

A.1 UX design demos of sketching the (layout, user interac- tion and flow of) screens of an iPhone app. It includes a description of relevant interface elements, above all the actions triggered on tap, shake and swipe events...... 234 A.2 Wireframe UX design templates for sketching phone and watch GUI layouts. Print this page and draw on paper. Pa- per allows fast brainstorming and sketching of many, many different ideas – and it makes it easier to throw away the bad ones...... 235

211 212 List of Tables

1.1 Section overview...... 2

10.1 Possible use cases for different actuators...... 128 10.2 Possible user state as detected by different sensors. . . . . 130

11.1 A list of selected Jailbreaks ...... 146

A.1 Notable changes in different iOS versions...... 232 A.2 Notable changes in different watchOS versions...... 233

213 214 Listings

2.1 IBOutlet and IBAction define the GUI to code connection. 22 2.2 The interface controller life cycle method awake might set up data for a table shown on the UI...... 23 2.3 Passing data via Segue from the first interface controller. . 24 2.4 Retrieve the data in another interface controller...... 24 2.5 Passing data without using a segue...... 25 2.6 Navigate back to the calling interface controller...... 26 2.7 The life cycle methods for an extension delegate...... 29 2.8 On the watch: set up a session between the watch app and the ios app...... 29 2.9 On the watch: send a message to the iPhone app...... 30 2.10 On the watch: receive a message from the iPhone app. . . 30 2.11 On the phone: setup a handler class to manage the connec- tion to the watch...... 31 2.12 On the phone: one single instance of the custom session handler is initiated...... 32 2.13 On the phone: observing an internal value of the custom connection handler to get informed when data arrived from the watch...... 32 2.14 On the phone: sending data to the watch...... 33 3.1 Run script as command line scripts...... 40 3.2 Live compiling and execution of swift code in the Play- ground...... 41 3.3 A list of Swift keywords...... 42 3.4 Type inference means auto-detection of the type at first as- signment during variable definition...... 43 3.5 Optional chaining spares the check if an object is nil before working with this object...... 44

215 3.6 Functions allow default and named parameters even for re- turn values. Note the order of the parameter must not be changed...... 45 3.7 Using inout parameters allows side effects, as the refer- ence is passed. Note that the underscore allows us to leave away the otherwise required argument labels for pa- rameters when calling a function...... 45 3.8 Functions might have variable number of parameters. . . . 45 3.9 Closures are used as a short way to define a function. . . . 46 3.10 Examples of definition and usage of lists tuples dictionar- ies structs and Enums...... 47 3.11 Map and reduce support a functional programming style. . 47 3.12 Object-oriented programming in Swift...... 48 3.13 Single inheritance in Swift...... 49 3.14 Extensions to any – even built-in – class...... 50 3.15 Protocols guaranteed that classes provide the expected functionality...... 50 3.16 Protocol extensions allow to add functionality to protocols. 51 3.17 Selected tips for coding in Swift: Formatted strings. . . . . 51 4.1 Lazy initialisation defers the creation of properties to it’s first usage...... 56 4.2 Property observers trigger a callback when properties are read or modified...... 57 4.3 Operator overloading assigns new meaning to given oper- ators...... 58 4.4 Asserting the input value forces the app to crash rather than to go on in an undefined state...... 59 4.5 With do catch and try statements, exceptions can be caught. 60 4.6 Custom exception conform to the Error protocol...... 60 4.7 To check the value of arguments, the guard statement is used. 61 4.8 Return the reference to a function and invoke the function via this function pointer...... 61 4.9 With generics code duplication is avoided and strong typ- ing is still enforced...... 62 4.10 Changing the internals of a struct needs mutating functions. 63 4.11 The – for too often misused – singleton design pattern. Better avoid global objects...... 63

216 4.12 Reflection (introspection) using a Mirror structure. . . . . 64 5.1 Adding an iOS Unit Testing Bundle separates the test cases from the rest of the source code...... 72 5.2 Asynchronous test cases waiting for an expectation to be fulfilled...... 72 5.3 The controller must implement methods to report, for ex- ample, the number of rows and data for a single entry at a given row...... 73 5.4 Xcode supports recording UI test statements (by pressing the red record button at the bottom of the code window). Developers need to add code for assertions. Finally, the test cases can replayed...... 75 5.5 To set data onto the destination view controller you can override function prepare(:for segue :sender)...... 78 6.1 Concurrency with Grand Central Dispatch and Dispatch Queues...... 86 6.2 Fetching data in the background and updating the UI in the main queue...... 87 6.3 JSON data must be serialised into your own objects. . . . . 90 6.4 Prepare a class for objects to be initialised with JSON. . . 91 6.5 Prepare another class for a collection to be initialised with JSON data...... 91 6.6 Parsing JSON into special custom classes...... 92 6.7 Parsing JSON into special custom classes...... 93 7.1 Loading configurations from property files...... 100 7.2 Storing key-values (values might be any object) into shared prefs using the UserDefaults API...... 100 7.3 Core data code in the app delegate...... 101 7.4 After triggering the generation of classes (via menu Create NSManagedObject Subclass), additional custom methods might be added...... 102 7.5 The extension to the class manages the mapping from Swift objects to the (Objective-C) Core Data persistence layer...... 104 7.6 When working with data, a managedObjectContext is needed...... 104

217 7.7 Search requests help to find objects. Then we might cre- ate/modify and finally save (persist) them...... 105 8.1 The XML source view of the file Info.plist shows the re- quired message string presented to the user for requesting permissions to detect GPS...... 110 8.2 The get the current location the Core Location Manager is configured to inform a delegate about location changes. . . 111 8.3 This examples runs the location detection for one minute and updates the gui with the latest location every time the manager detects GPS updates...... 111 8.4 Add the MapKit Framework to the project and a Map onto the view...... 112 9.1 The definition of a custom view requires a base class of type UIView...... 120 9.2 The definition of a custom view requires a draw function, which is called from time to time to update the UI. . . . . 121 9.3 To refresh the custom view a timer triggers the update of the UI...... 122 9.4 The class Spinner represents a custom UIView and can draw itself...... 122 9.5 Animate any view across the UI...... 123 10.1 Use one of core motion framework to add motion aware- ness to the app...... 131 10.2 Use one of the image sensors (cameras) to get an image. . 132 10.3 A simple, but very effective way of notifying users is to provide vibration feedback...... 132 10.4 Check, whether the application code is executed on the simulator or on a real device...... 133 11.1 Extract the *.ipa to view the file structure...... 143 11.2 Check if the code is signed...... 143 11.3 Extract a specific binary out of a fat binary...... 144 11.4 Find out if binary is encrypted. Check if cryptid=1. . . . . 145 11.5 Strings inside unencrypted or decrypted binaries might be inspected with the strings tool...... 145 11.6 The debugger might reveal readable source code...... 145 12.1 The Contacts framework allows to access the address book of the user...... 153

218 12.2 For privacy reasons, users have to be asked for permission to access the contacts, unless access is already granted. . . 153 12.3 The actual search...... 153 12.4 Share some text with other apps...... 155 12.5 Save images to the photo library or share images with other applications or services...... 155 12.6 Register a custom URL Scheme slideshow:// by configur- ing a unique pattern for your app bundle id in the info.plist. 156 13.1 A cloud kit container holds a public and a private cloud database and allows reading of custom defined records. . . 164 13.2 Fetching data from a cloud kit container database using a flexible query with a search predicate...... 164 13.3 Writing modified data back into a cloud kit container needs the proper permissions set online in the CloudKit dashboard.165 14.1 Registering for notification types to be fired by the app. . . 173 14.2 Schedule a local notification to be send in a few seconds. Optionally, swipe down on the home screen to view notifi- cations...... 173 15.1 A model can be created interactively in the Xcode Play- ground showing an image classifier...... 185 15.2 The probabilities of a given input image belonging to the different categories are reported by the machine learning model...... 185 16.1 This simple widget should be extended to provide updated information for the Today live preview...... 190 16.2 The search API allows to put data into the index for global search...... 191 16.3 The search results allow to jump into the corresponding app. 192 16.4 Ranging known iBeacons by their UUID nearby. Using the major and minor numbers provide increased accuracy, but are optional...... 199

219 220 Glossary

AirDrop API for exchanging documents between Apple computers over WiFi connections. 146, 189, 196

AirPlay API for streaming multimedia content from iOS devices to Ap- pleTV. 16, 152, 189, 197, 198

AirPrint API for wireless printing. 189, 197

APNs Apple Push Notification service sends remote messages to iOS de- vices. Compare to Cloud Messaging with Google Firebase platform. 171, 175–177, 179, 211

App Store Apple’s online store to distribute iOS apps. 3, 12, 202, 224

App Thinning Slicing assets for platform specific delivery. 195

Apple System Optimised file system for Solid State Disk (SSD). Features file-level encryption and wipe out storage fast. 232

Apple Mobile File Integrity Kernel extension to enforce entitlements. 137, 147

Apple Watch Apple’s smartphone. 2, 10, 15, 17–19, 22, 35, 36, 157, 194, 209

ARC Automated Reference Counting. 39, 56, 65, 67

Auto Layout For automatically arranging UI elements on a view accord- ing given constraints. 76

Bonjour Zero configuration protocol to auto-setup network configura- tions such as connections to printers. 196

221 Bundle ID Unique name of any app. 139

CarPlay API for access to car information and providing multimedia data for entertainment. 189, 198

CloudKit This frameworks support storing data in the cloud. For web applications a JS API CloudKit JS is available. 133, 163, 165–167, 219

Continuity API for allowing users to transparent switch from working on iOS devices to working on macOS computers. See Handoff. 189, 197, 232

Core Animation API for graphics rendering using sprites. 124

Core Data Object-relational layer. 101, 107

Core Motion API for accelerometer, gyroscope, and pedometer events. 133

CoreGraphics API for 2D drawing. 120

CoreML API for machine learning enabling vision, natural language or sentiment analysis. For example, embedding a machine learning model for flowers might result in a 81% prediction of a flower in an image being of type rose. 232

Darwin The iOS operating system which includes the hybrid XNU kernel. 138

FaceID Authorisation using the so called TrueDepth camera in combina- tion with an infrared camera to create a depth map to match user’s face. 140

Find My iPhone Service to ping, lock or remote delete lost iPhones. 140

Game Center Social platform supporting gaming. For example central place to manage high score lists. 194

Grand Central Dispatch Managing queues for concurrency. 71, 85, 95, 195

222 Handoff API for synchronising data between iOS and macOS devices to allow user to work on an other device or computer. See Continuity. 133, 189, 197, 222

HealthKit API for working with health data. 133, 189, 199, 202, 206, 232

HomeKit API for home automation. 189, 198, 199 iBeacon Low energy bluetooth devices which emit radio signals allowing smartphones to estimate the distance. 130, 189, 199–201, 205, 219 iCloud Apple Cloud Services. 101, 152, 161–163, 166, 167, 197, 199 In-App Purchase Buying content, digital goods or subscriptions within an app. 189, 194

IOKit An object-oriented framework to access the I/O (device) drivers.. 138 iOS The operating system for Apple smartphones and tablets. 1, 69, 129, 133, 138, 139, 141, 147, 148, 171, 189, 190, 196, 224 iPad Apple tablet. 2, 18, 69, 201, 224 iPhone Apple smartphone. 69, 76, 134, 224, 231 IR A lightweight and low-level intermediate representation of code used by LLVM. Optimisers input LLVM IR assembly language and out- put bitcode. 196 iTunes Connect A web interface to manage apps in the Apple App Store. 4, 6, 195, 202, 203

LDAP Lightweight Directory Access Protocol to query, for example, Windows Active Directory systems. 201

LLVM Compiler infrastructure. Informal: Low level virtual machine in- frastructure. 52, 196, 223 macOS The operating system for Apple desktop and computers (and servers). 41, 64, 139, 195, 196, 210

223 MapKit API for Apple Maps. The library prefix is MK. 115

Metal Fast 3D graphics framework, GPU-accelerated. 123, 124

MFi Made for iOS (iPhone and iPad). 198

Newsstand Auto-renewable subscription for updating app content. 189

Objective-C Programming language prior to Swift. 123

Quality of Service Priorities for tasks in (concurrent) queues can be set according to the level of quality of service. 86, 95

Quartz 2D Drawing engine. 120

ResearchKit The API supporting scientific research activities. 190

SceneKit The API for 2D games. 80, 123, 125

Siri The iOS voice assistant. 129, 193

SiriKit The API for the iOS voice assistant and user requests originating from Maps. 193

Spotlight Full text search engine of iOS and macOS. 189, 191, 193

SpriteKit Graphics engine for animated objects. 124

StoreKit API for In-App purchase, i.e. to programmatically offer addi- tional content from the within an app. 194

TestFlight App Store for alpha and beta testing apps. 195

TouchID Authorisation using the biometric finger print sensor. 140, 146, 148, 194 tvOS The operating system for Apple TV. 196 widget An app extension in the Today view allow users check status fast and frequently. 190, 191

224 WWDC Apple World-wide Developer Conference. 16

Xcode Integrated Development Environment (IDE). 4, 102, 104, 184

XNU A part of the Darwin operating system, the (open sourced) hybrid kernel for iOS (and macOS). XNU comprises Mach and (Free)BSD and IOKit. 138

XPC Cross-process (remote) procedure calls. The high-level API is pro- vided by XPC Services. 146

225 226 Acronyms

ABI Application Binary Interface. 64

AMOLED Active-Matrix Organic Light-emitting Diode. 35

API Application Programming Interface. 66, 139, 204

BSD Berkley Standard Distribution. 139, 157

BTLE Bluetooth Low Energy. 36, 130, 134, 189, 199

BYOD Bring Your Own Device. 206

CalDav Calendaring Extensions to WebDAV. 201

CI Continuous Integration. 73, 195

CNTK Microsoft Cognitive Toolkit. 186

DEP Device Enrolment Program. 201, 202

FCM Firebase Cloud Messaging. 171, 177–179

FPS Frame per Second. 120

GCC GNU Compiler Collection. 52

GCM Google Cloud Messaging. 177, 179

GLONASS Globalnaya Navigazionnaya Sputnikovaya Sistema, or Global Navigation Satellite System. 110

GPS Global Positioning Service. 2, 36, 66, 70, 109, 110, 127, 129

227 GPX GPS Exchange Format in XML. 112

GUI Graphical User Interface. 2, 5

HF High-Frequency. 127

IaaS Infrastructure as a Service. 161

IDE Integrated Development Environment. 57, 104

IoT Internet of Things. 15, 180

IPC Inter-process Communication. 137, 139, 148, 155, 157, 159, 191

JSON Java Script Object Notation. 90–94, 96, 217

KASLR Kernel Address Space Layout Randomisation. 146

KAuth Kernel Authorisation. 137

KEXT Kernel Extensions. 147

KPP Kernel Patch Protection. 146

LBS Location-based Services. 109

LTPO Low-temperature Polycrystalline Oxide. 35, 36

MAC Mandatory Access Control. 147

MACF Mandatory Access Control Framework. 137

MDM Mobile Device Management. 189, 201, 202

MPNS Microsoft Push Notification Service. 179

NFC Near-field Communication. 36, 129, 146, 194

OLED Organic Light-emitting Diode. 35, 36

OOP Object-Oriented Programming. 48, 57, 66

228 OpenGL Open Graphics Library. 123, 124

ORM Object-Relational Mapping. 2, 103, 210

OWASP Open Web Application Security Project. 141

PaaS Platform as a Service. 161

PDE Plausibly Deniable Encryption. 141, 149

POI Point of Interest. 109, 110

POS Point of Sale. 194

REPL Read-eval-print-loop. 40, 41, 210

SaaS Software as a Service. 161

SOC Silicon On Chip. 36

SQL Structured Query Language. 138

SSD Solid State Disk. 221, 232

TDD Test-driven Development. 71

TLS Transport Layer Security. 140, 146

TTW Through The Web. 161

UCD User Centred Design. 70

URL Uniform Resource Locator. 148, 152, 155, 156

UUID Universally Unique Identifier. 200, 219

VPN Virtual Private Network. 146, 190, 201, 202

VPP Volume Purchase Program. 201, 202

WNS Windows Push Notification Service. 179

229 XML Extensible Markup Language. 94, 96, 104, 110, 112, 157, 218

XMPP Extensible Messaging and Presence Protocol. 178

XSRF Cross-Site Request Forgery. 138

XSS Cross-Site Scripting. 138

230 A Appendix

“If you sense there must be more, there is more.”

[by Alan Cohen]

A.1 iOS Version Changes

Find in Table A.1 selected changes over the years. See also the visual history1.

iOS Changes Remarks 1.0 UI Januar 9, 2007. UI interaction with a touchscreen iPhone is a revolutionary new mobile phone that allows you to make a call by simply tapping a name or number in your address book... https://web.archive.org/web/ 20071006005308/http://www.apple.com/ iphone/features/index.html. 2.0 Apps June 9, 2008. iPhone OS software developer kit (SDK). App Store for 62 countries. https://www.apple.com/newsroom/2008/06/ 09Apple-Introduces-the-New-iPhone-3G/. 2.0 SandBox The SandBox introduced named SeatBelt. 3.0 Push June, 2009. Push notifications for 3rd party apps.

1 A visual history of iOS https://www.theverge.com/2011/12/13/2612736/ios-history-iphone-.

231 4.0 Multitasking April 8, 2010. Multitasking for Apps. For example, background audio. https://www.apple.com/newsroom/2010/04/ 08Apple-Previews-iPhone-OS-4/. 5.0 Standalone Activate phone without connecting to a computer. 8.0 Extensible Extensibility and Continuity, HealthKit, HomeKit, Widgets. 10.3 Apple Files SSD optimised file system. System https://developer.apple.com/library/content/ documentation/FileManagement/ Conceptual/APFS Guide/Introduction/ Introduction.html. 11.0 ARKit, Vi- Face tracking with augmented reality, image sion, CoreML classification and natural language processing with machine learning APIs. 11.0 64-bit June 28, 2017: 64-bit Apps on iOS 11; 32-bit apps will not launch anymore. https://developer.apple.com/news/?id= 06282017b. 12.0 NL, OpenGL Natural Language framework; OpenGL ES Deprecation deprecated, use Metal.

Table A.1: Notable changes in different iOS versions.

232 A.2 watchOS Version Changes

Find in Table A.2 selected changes for watchOS.

watchOS Changes Remarks 1.0.1 Apps Third-party apps (on the phone). 2.0 Native Native apps (on watch). 3.2 Siri Januar 24, 2017. SiriKit for 3rd party apps. https://developer.apple.com/news/?id= 01242017b. 4.0 Background Late 2017. Background modes for navigation and audio recording https://developer.apple.com/watchos/. 5.0 eSim and 4 (GPS + Cellular)

Table A.2: Notable changes in different watchOS versions.

233 A.3 UX Design Demos by john feiner

Figure A.1: UX design demos of sketching the (layout, user interaction and flow of) screens of an iPhone app. It includes a description of relevant interface elements, above all the actions triggered on tap, shake and swipe events.

234 A.4 UX Design Wireframe Templates by john feiner

Figure A.2: Wireframe UX design templates for sketching phone and watch GUI layouts. Print this page and draw on paper. Paper allows fast brainstorming and sketching of many, many different ideas – and it makes it easier to throw away the bad ones.

235