COVIDTAS COVID-19 Tracing App Scale—An Evaluation Framework

Home , Aarogya Setu, BlueTrace, Contact tracing, COVIDSafe, COVID Alert, Digital contact tracing

sustainability

Article COVIDTAS COVID-19 Tracing App Scale— An Evaluation Framework

Raghu Raman 1,*, Krishnashree Achuthan 2, Ricardo Vinuesa 3,* and Prema Nedungadi 4

1 Amrita CREATE, Amrita Vishwa Vidyapeetham, Amritapuri 690525, India 2 Amrita Center for Cybersecurity Systems and Networks, Amrita Vishwa Vidyapeetham, Amritapuri 690525, India; [email protected] 3 KTH Engineering Mechanics, SE-100 44 Stockholm, Sweden 4 Amrita School of Engineering, Amrita Vishwa Vidyapeetham, Amritapuri 690525, India; [email protected] * Correspondence: [email protected] (R.R.); [email protected] (R.V.)

Abstract: Mobile apps play an important role in COVID-19 tracing and tracking, with different countries taking different approaches. Our study focuses on 17 government owned COVID-19 Contact Tracing Apps (CTAs) and analyze them using a proposed COVIDTAS framework. User satisfaction is not directly related to the COVIDTAS score or the interaction between users and the app developers. To increase adoption of CTAs, government leadership must offer assurance to its citizens that their identify will be concealed and emphasize the beneﬁts of CTAs as it relates to shared public health. While no country has topped the list on all three major factors (COVIDTAS Score, User Reviews, and User Ratings), the CTA from India seems to have above average performance on all three factors.

Keywords: COVID-19; coronavirus; contact tracing; governance; privacy

Citation: Raman, R.; Achuthan, K.; Vinuesa, R.; Nedungadi, P. 1. Introduction COVIDTAS COVID-19 Tracing App Scale—An Evaluation Framework. The COVID-19 pandemic, through its sheer scale has highlighted and widened the Sustainability 2021, 13, 2912. https:// socioeconomic divides that exist today and made vulnerable populations such as those doi.org/10.3390/su13052912 with chronic illnesses, elderly, minorities, people of lower socioeconomic backgrounds, etc., more susceptible [1]. Mobile phones and digital technology has been in the forefront Academic Editor: Jose Ramon Saura of addressing this crisis in fields varying from education to [2] healthcare [3]. One such approach is contact tracing using mobile applications. The concept of contact tracing in rela- Received: 22 January 2021 tion to infectious diseases in not new. For years, practitioners of community medicine have Accepted: 26 February 2021 isolated those most likely to be infected by a certain infectious disease from any infected Published: 8 March 2021 individual, isolated these potential candidates, and provided treatment as needed [4]. This approach is extremely effective when done at a local level, with a relatively low number of Publisher’s Note: MDPI stays neutral cases [5]. Contact tracing can be done by local health care workers in person, or by use of with regard to jurisdictional claims in technologies ranging from RFID [6] to wearable tracking devices [7]. published maps and institutional affil- In recent times, with the advent of smart phones and the high penetration of mobile iations. technology, cell-phone apps have been widely used for contact tracing. Mobile contact tracing applications can be used to identify the proximity of each user to others, including those infected with a contagious disease. Once an infected person is identified, some mobile applications even warn its users when contact is imminent [8]. Clearly, such applications Copyright: © 2021 by the authors. have a potential to minimize person to person transmission of diseases. In addition, these Licensee MDPI, Basel, Switzerland. applications allow healthcare workers to identify those at risk of infection, allowing for This article is an open access article pre-emptive isolation and treatment to minimize further spread as well as the intensity distributed under the terms and of infestation. conditions of the Creative Commons These applications have become more mainstream in light of the ongoing COVID-19 Attribution (CC BY) license (https:// pandemic. It should be noted that the use of such applications is not without controversy, creativecommons.org/licenses/by/ and it has been argued that extensive data gathering by governments can eventually 4.0/).

Sustainability 2021, 13, 2912. https://doi.org/10.3390/su13052912 https://www.mdpi.com/journal/sustainability Sustainability 2021, 13, 2912 2 of 19

do more harm than good [9]. However, mobile apps have been employed successfully across the world for years to prevent the spread of contagious diseases. Therefore, a methodology to assess the applications currently employed for contact tracing on criteria such as data security, privacy, data retention, etc., is of signiﬁcance. Understanding these applications in the context of various discrete criteria can help assess their risk to citizens and governments should the data be misused. In that light, this study presents a novel COVIDTAS framework for assessing COVID-19 tracing applications. The evaluations rely on public data and reviews on various platforms for analysis. However, before analysis it is essential to gain a basic understanding of contact tracing applications and their use in medicine. In that light, the next section explores a few such examples with a focus on how government mechanisms in the past have employed mobile contact tracing apps. This is followed by a discussion on digital apps used for COVID-19 tracing speciﬁcally. The study then focuses on COVID-19 applications that are initiated by governments and the concerns associated with a government driven approach. The data used in this study is then presented and analyzed before discussing the results of the analysis. In addition, the perceptions surrounding government controlled COVID-19 applications is also discussed in this paper before presenting the conclusion and future research direction.

2. History of Contact Tracing Contact tracing becomes essential when the potential of a disease for person to person spread is high. Naturally, the earlier applications of mobile apps for contact tracing has been for diseases such as Ebola, SARS/MERS, TB, etc. Broadly speaking, these applications fall under m-health or mobile health interventions, where mobile technology is used to aid health care. It should be noted that while this paper focuses on contact tracing, m- health encapsulates applications such as remote diagnosis, data sharing, and information dissemination in addition to contact tracing. Contact tracing apps have been employed extensively in managing the spread of Ebola in African countries over the last decade. For example, in the 2014–15 outbreak of Ebola in West Africa, mobile interventions were used for contact tracing in addition to community level manual tracing. In Sierra Leone, a mobile app termed as Ebola Contact Tracing application (ECT app) was employed for contact tracing [10]. Results show that app-based interventions allowed for more concrete, accurate, and timely updating of the centralized database if done correctly. In addition, the manpower and paperwork required for contact tracing was also drastically reduced [10]. A study of 58 Ebola contact tracing apps by [11] identiﬁes three applications, Community Care, Sense Ebola Follow up, and Surveillance and Outbreak Response Management and Analysis System as capable of carrying out surveillance, contact tracing, management of cases, and data management for laboratory data, thereby making them complete data management systems for Ebola and similar contagious diseases. A more generic software Go.data [12] which can be employed for any disease outbreak with peer to peer transmission has been used successfully in Democratic Republic of Congo for Ebola contact tracing successfully. Similarly, mobile apps have been employed for contact tracing in case of TB, SARS, and other respiratory diseases [13,14].

3. Present Use of Contact Tracing Apps for COVID-19 The potential use of contact tracing in the control of COVID-19 has been well estab- lished by studies, especially in cases where there was less chance of transmission before the onset of symptoms [15]. The use of mobile apps for contact tracing has come to the mainstream with the recent COVID-19 pandemic. Governments and healthcare agencies across the world are employing mobile apps for identifying potential paths of infection and isolating those at risk. The MIT project on COVID-19 tracking apps lists over 25 apps at this stage, which are employed by governments across the world for contact tracing during this pandemic [16]. The project lists a number of applications ranging from those that are compulsory to those that are optional. Many vary widely in terms of transparency, Sustainability 2021, 13, 2912 3 of 19

data destruction (for privacy), and the technology employed. Similarly, FIPRA (2020) lists the applications being used in Europe for contact tracing. A few examples include the COVIDSafe app by Australian government, which faced resistance from the public in general due to concerns about privacy and was not entirely successful in providing contact tracing due to lack of widespread adoption of the application [17]. India’s Aarogya Setu app, which is the most downloaded contact tracing application, effectively collects personal information and constantly adds to location information in real time for its users [18]. On the other hand, the Corona-Warn-App used in Germany posed no major privacy concerns as it was developed with potential privacy infringements in mind. Data is depersonalized and stored locally to avoid any concerns in this regard [19]. While many countries such as India have made contact-tracing apps practically compulsory for activities such as air travel [20], other countries such as the UK have refrained from employing similar procedures due to public concerns over privacy [17,21,22]. There are clearly diverse approaches taken by governments during this pandemic when it comes to mobile contact tracing. However, in general, contact tracing apps have been employed as a novel tool for curtailing the spread of this disease. The concerns raised by many with regards to privacy and data security in relation to these applications are valid and need to be discussed in detail.

4. Government Interventions in Contact Tracing The role of governments in contact tracing has been a matter of much discussion. Multiple surveys and studies find varying degrees of public trust in government contact tracing applications. For example, a survey carried out in the Republic of Ireland with over 8000 participants shows a general acceptability for a contact tracing application. In fact, 54% of the respondents were positive about downloading such an app, while another 30% were moderately likely to do the same [23]. This is in sharp contrast with a survey on over 2000 participants from the UK, where the acceptance rate for a government-controlled application was under 55%. However, there was greater acceptability for an app controlled by the NHS [24], clearly highlighting a lack of trust in the government handling sensitive medical information. Citizens of many other countries are effectively left without a choice in the matter. For example, while data collection for public good needs no individual consent in China, it is possible for the government to provide private parties access to the same data, thereby leading to potential privacy concerns [19]. Similarly, a study on a COVID-19 contact tracing app propagated by the government of France highlights how such applications, though immediately beneficial, may be a serious challenge to data privacy in the future. The study also suggests that the government may be downplaying this adverse effect for the immediate purpose of contact tracing [25]. Similarly, the use of contact tracing in Russia has raised serious concerns about it being used for mass movement surveillance. The application suggested in Moscow may require citizens to generate a unique code each time they exit their living space, and share the code with law enforcement, should they be asked to do so [26]. Some countries such as South Africa and the UAE are yet to have concrete data privacy laws, making the deployment of such applications all the more concerning as they do not have to adhere to existing data laws [27]. Amnesty international (2020) has raised privacy concerns related to contact tracing apps and identifies the applications employed in Bahrain, Kuwait, and Norway to be among the least safe for data privacy. There have also been significant differences in how governments across the world handled these concerns related to surveillance and data security. For example, Singapore chose to publicly acknowledge the potential security concerns with its application and asked its citizens to concentrate on the immediate contact tracing requirement [28]. Hong Kong employed geo fencing instead of GPS tracking to provide some degree of privacy to its citizens while South Korea addressed such concerns by stringently restricting access to the database [29]. In other countries such as India and Russia the concerns regarding misuse of data from tracking apps is still significant [26,30,31]. The limited number of studies available on this issue highlights that mass adoption of these Sustainability 2021, 13, 2912 4 of 19

applications is necessary for them to be successful. This can either be achieved through legislation (making it compulsory to use the application), or through trust building with the public. In addition, with regards to COVID-19, the disease disproportionately affects the elderly, who, despite having smart phones, are often technology averse and unlikely to install and use mobile applications [32]. In general, at least in democratic systems, such mobile applications are unlikely to succeed without adequate public trust as illustrated by cases in the UK [33] and Australia 17].

5. Challenges and Apprehensions Regarding Government Contact Tracing The papers discussed so far clearly illustrate the challenges of balancing individual rights regarding data privacy, with the immediate need for contact tracing [34]. From literature, it is also seen that speciﬁc policy and technology measures can help alleviate and minimize these concerns. This is best illustrated by the approach taken by Germany, where data privacy was built into the contact tracing application while also providing decentralized storage of data, thereby minimizing the loss or misuse of largescale data [35]. Similarly, by employing geo fencing instead of GPS tracking, Hong Kong has managed to address concerns of governments tracking its citizens to a great extent. Existing privacy and data security laws also play a part in protecting citizens of any country against the misuse of this sensitive information. In general, it is safe to conclude that in democratic systems, it is best that the government spend adequate resources on educating the public about the importance of contact tracing applications while simultaneously being transparent in addressing any concerns regarding privacy and data security.

6. Materials and Methods Contact Tracing Apps (CTAs) are designed to automate the process of gathering data of contacts of people with infected individuals, so as to identify the risks based on the context, proximity and duration of contact. CTAs estimate proximity through Bluetooth or Global Positioning System (GPS). GPS is not suitable as the accuracy is low in built-up spaces. Bluetooth is available in smartphones and can support proximity estimation, however the distance estimation may vary based on surrounding objects and the orientation of the phone [30], thus leading to false positive and negative alerts. Contact tracing apps may be designed with a centralized architecture, where all activities revolve around a trusted server that supports storing encrypted information, analyzing risk of contacts and notifying such identiﬁed contacts [36,37]. In this case, the identity of the users is known to the trusted server. India’s Aarogya Setu and Singapore’s TraceTogether are examples of centralized architecture (Table1).

Table 1. Characteristics of COVID-19 CTAs.

Architecture Centralized Decentralized Pre-register Yes No Store personal identiﬁcation Yes No Privacy Generated by server Generated at client device Server can run data analytics None, as server does not Contact risk analysis including risk analysis and maintain data notiﬁcations A malicious attack can Data is stored on device, Data protection compromise the central server minimal data uploaded to to access user information server Data retention Long-term Comparatively lesser Provides data on disease Yes Yes transmission (epidemiology)

In contrast, the decentralized architecture minimizes the information exchange and anonymizes user identiﬁcation with the server [38]. Australia’s COVIDSafe [39] is built Sustainability 2021, 13, x FOR PEER REVIEW 5 of 17

Sustainability 2021, 13, x FOR PEER REVIEW 5 of 17

countries from N. America, three countries from Europe, two from South America, one country from Africa, Australia, and New Zealand. • countries Stage from II involved N. America, comprehensive three countrie researchs thatfrom involved Europe, analyzing two from the Southdesign America,of CTAs, privacy policies and their mapping to the mandatory requirements of the state. one country from Africa, Australia, and New Zealand. • Additionally, policies on transparency were verified through open-source availabil- Stage ityII ofinvolved CTA software comprehensive databases and research adherence thatto GDPR involved requirements. analyzing The adherence the design of CTAs,to privacy various policiesGDPR requirements and their mapping were assessed to the for mandatorynon-European requirements countries from of pub- the state. Additionally,licly available policies authentic on transparency and reputable were sources verified that providedthrough criticalopen-source analysis availabil- of Sustainability 2021, 13, 2912 ity of CTACTA’s software security and databases privacy features. and adhere Cross-validationsnce to GDPR with requirements. news articles and/orThe adherence re- 5 of 19 to varioussearch GDPR publications requirements were carried were out assessed wherever forpossible. non-European countries from pub- • Stage III involved evaluation of publicly available reviews inclusive of those shared licly availableby user, legal authentic firms and and the media.reputable sources that provided critical analysis of CTA’swith security decentralized and privacy architecture. features. Cross-validations There is also a hybrid with model, news articles where onlyand/or the re- personal ID searchTableand 1.publications Characteristics data of COVID-19 were of COVID-19 carried positive CTAs. out wherever users or volunteers possible. are saved at the server [40]. • Stage III ArchitectureinvolvedThis study evaluation is focused of onpublicly contactCentralized av tracingailable applications reviews inclusiveDecentralized used by of governments those shared for COVID- by user,19 mitigation.legalPre-register firms and Data the from media. Contact Yes Tracing Apps of 17 countries No with at least one country Storefrom personal each identification of the seven continents Yes were aggregated as part No of this study (Table2). A Table 1. CharacteristicsPrivacy of COVID-19 CTAs. Generated by server Generated at client device multistage approachServer was can adopted run data to analytics collect includ- the data.None, as server does not Contact risk analysis Architecture ing risk analysisCentralized and notifications maintainDecentralized data Table 2. COVID-19A malicious CTAs attack explored can compromise in the study. Pre-register Yes Data is stored on device, No min- Data protection the central server to access user infor- Store personal identification Yes imal data uploaded No to server Continent Country Appmation Name URL PrivacyData retention Generated Long-term by server ComparativelyGenerated at lesser client device nic.goi.aarogyasetu (accessed on AsiaProvides India data on disease Server can run Aarogya data analytics Setu includ- None, as server does not Contact risk analysis Yes Yes 7 March 2021) transmission (epidemiology)ing risk analysis and notifications maintain data A malicious attack can compromise sg.gov.tech.bluetrace (accessed on Asia Singapore TraceTogether Data is stored on device, min- DataTable protection 2. COVID-19 CTAs exploredthe central in the server study. to access user infor- 7 March 2021) imal data uploaded to server Continent Country Appmation Name com.mic.bluezone URL (accessed on Asia Vietnam Bluezone—Contact Detection DataAsia retention India Long-term Aarogya Setu nic.goi.aarogyasetu Comparatively7 March lesser 2021) Asia Singapore TraceTogether sg.gov.tech.bluetrace Provides data on disease jp.go.mhlw.covid19radar (accessed on AsiaAsia Japan Vietnam Bluezone—Contact COCOA—COVID-19Yes Detection com.mic.bluezoneYes transmission (epidemiology) 7 March 2021) Asia Japan COCOA—COVID-19 jp.go.mhlw.covid19radar com.hamagen (accessed on המגן 2—האפליקציה הלאומית למלחמה בנגיף AsiaAsia Israel Israel com.hamagen (March 2021 7 הקורונה .Table 2. COVID-19 CTAs explored in the study Asia UAE TraceCovid ae.tracecovid.app Continent Country App Name ae.tracecovid.app URL (accessed on AsiaAsia UAE Qatar TraceCovid EHTERAZ com.moi.covid19 Asia North India Aarogya Setu ca.gc.hcsc.canada.stopcov nic.goi.aarogyasetu7 March 2021) Canada COVID Alert Asia AmericaSingapore TraceTogether com.moi.covid19sg.gov.tech.bluetraceid (accessed on AsiaAsia North Qatar Vietnam Bluezone—Contact EHTERAZ Detection com.mic.bluezone Mexico COVID-19MX mx.gob.www7 March 2021) Asia America Japan COCOA—COVID-19 jp.go.mhlw.covid19radar Australia Australia COVIDSafe au.gov.health.covidsafeca.gc.hcsc.canada.stopcovid (accessed המגן 2—האפליקציהCOVID Alert הלאומית למלחמה בנגיף North America Canada Asia IsraelNew nz.govt.health.covidtracecom.hamagenon 7 March 2021) Tracer הקורונהNew Zealand NZ COVID Zealand mx.gob.wwwr (accessed on North AmericaAsia MexicoUAE COVID-19MX TraceCovid ae.tracecovid.app Europe Germany Corona-Warn-App de.rki.coronawarnapp7 March 2021) Asia Qatar EHTERAZ fr.gouv.android.stopcovi com.moi.covid19 Europe France StopCovid au.gov.health.covidsafe (accessed on AustraliaNorth Australia COVIDSafe ca.gc.hcsc.canada.stopcovd Canada COVID Alert 7 March 2021) AmericaEurope Spain Radar COVID es.gob.radarcovidid South nz.govt.health.covidtracer (accessed New ZealandNorth New ZealandBrazil NZCoronavirus—SUS COVID Tracer br.gov.datasus.guardioes AmericaMexico COVID-19MX mx.gob.wwwon 7 March 2021) AmericaSouth Columbia CoronApp co.gov.ins.guardianesde.rki.coronawarnapp (accessed on EuropeAustraliaAmerica GermanyAustralia Corona-Warn-App COVIDSafe au.gov.health.covidsafe (covid.trace.morocconz.govt.health.covidtrace7 March 2021 وﻗﺎﻳﻨﺎ Africa NewMorocco New Zealand NZ COVID Tracer Zealand fr.gouv.android.stopcovidr (accessed Europe France StopCovid Europe Germany Corona-Warn-App de.rki.coronawarnappon 7 March 2021) fr.gouv.android.stopcovies.gob.radarcovid (accessed on EuropeEurope SpainFrance Radar StopCovid COVID 7 Marchd 2021) Europe Spain Radar COVID br.gov.datasus.guardioes es.gob.radarcovid (accessed on South AmericaSouth Brazil Coronavirus—SUS Brazil Coronavirus—SUS br.gov.datasus.guardioes7 March 2021) America co.gov.ins.guardianes (accessed on South AmericaSouth Columbia CoronApp Columbia CoronApp co.gov.ins.guardianes7 March 2021) America covid.trace.morocco (accessed on covid.trace.morocco وﻗﺎﻳﻨﺎ AfricaAfrica MoroccoMorocco 7 March 2021)

The process followed for data selection and analysis is presented below. • Stage I involved selection of countries were primarily based on two factors (1) countries that had proactive involvement of state-owned bodies in the control, coordination and reporting of their national status and (2) additionally, to ensure good geographical spread, countries were chosen from various continents. This resulted in three countries being chosen within Asia, three countries from Middle East, two countries from N. Sustainability 2021, 13, 2912 6 of 19

America, three countries from Europe, two from South America, one country from Africa, Australia, and New Zealand. • Stage II involved comprehensive research that involved analyzing the design of CTAs, privacy policies and their mapping to the mandatory requirements of the state. Additionally, policies on transparency were verified through open-source availability of CTA software databases and adherence to GDPR requirements. The adherence to various GDPR requirements were assessed for non-European countries from publicly available authentic and reputable sources that provided critical analysis of CTA’s security and privacy features. Cross-validations with news articles and/or research publications were carried out wherever possible. • Stage III involved evaluation of publicly available reviews inclusive of those shared by user, legal firms and the media. Certain countries such as USA, Russia, China, S. Africa, etc., were excluded from this study due to lack of adoption of CTAs or reliance on private or multiple entities for contact tracing apps. India, France, Spain, Brazil, Colombia, and Mexico had the highest number of infected people in their respective continents, i.e., Asia, Europe, South America, and North America. Additionally, they were also amongst the Top 10 countries worldwide from the highest number of total cases and total number of deaths perspective. The reason Qatar and Israel were chosen was because of their high density of COVID-19 cases i.e., total cases per million population. The interesting aspect of analyzing data from Singapore was that in spite of having a relatively large number of cases i.e., close to 10,000 per million population, the number of deaths per million is amongst the lowest worldwide i.e., five per million. On the other hand, Vietnam and New Zealand were amongst the countries that had the lowest number of COVID-19 cases per million population and two to three orders of magnitude lower deaths per million in comparison to the highly infected countries. On user ratings and reviews, data was collected primarily from Google app store rather than Apple app store due to the former being a much larger repository of user downloads and usage statistics.

7. Qualitative and Quantitative Parameters in the COVIDTAS Framework A reliable and sound method for analyzing the various applications selected was essential for this study. The framework should be able to assess the effectiveness of the application on various fronts and provide an overall score for the application for comparison with its peers. This study proposes The Covid Tracing App Scale (COVIDTAS) framework as a platform to compare the various dimensions of these apps. This includes elements of human-centric design from usability, technology and privacy perspectives, their effectiveness in tracking and communicating proximity to COVID-19 patients and ultimately the user perception and acceptance based on their direct feedback. This framework has the potential to serve as an effective tool to key stake holders i.e., the public, medical personnel and federal authorities and assist with enhancing adoption, usage and impact from such apps. COVIDTAS framework was adapted from the sociotechnical framework proposed by [41] that pertained to analyzing the design configurations of the app from technology, governance and user perspectives. Our framework extends this even further by including factors that reflect user experience and sentiment that are critical to its success. The data collected included both qualitative and quantitative parameters. The qualitative parameters included: the extent of data security, transparency, ease of use, access to information, architecture, and so on. While synthesizing the data, emphasis was placed on the congruence between the qualitative parameters and the policies governing CTAs. Particular attention was paid to good and suboptimal practices that led to the identification of technological, organizational, cultural, and individual factors that were captured in the final empirical framework. The scoring rubric for all qualitative factors ranged between 0 and 2 with 2 representing the highest rating for the most preferred option, 1 representing moderate or partial compliance, and 0 representing the lowest rating for inadequate coverage or poor compliance (Table3) Sustainability 2021, 13, 2912 7 of 19

Table 3. Factors and scoring rubric for the COVID-19 CTAs.

Factors Score 2 Score 1 Score 0 Data collection should be compliant with the General Data Protection Regulation (GDPR) [6] and respect the privacy of the individual. A Data Protection Noncompliant or disregard Impact Assessment (DPIA) must Full compliance Partial implementation of GDPR for privacy be carried out before the deployment of any contact-tracing system. The purpose of the app and the mechanisms to assess its usage need to be clearly defined Transparency rights: They include the right of users to be notified, to control their own data, transparency regarding which Users are in full control of Partial fields are controlled Users have zero control of personal data are collected, and of their data by users their data explanation of app-produced output. The app should be auditable Accessibility refers to availability All citizens have equal access to CTAs are only available to certain of CTA to everyone without N/A the CTA sections of public any discrimination Availability of multimodal Availability of fewer than two At least 10 documents and videos Availability of less than five educational materials documents that were difficult were available reference materials and/or tutorials to find Type of protocol underlying the Decentralized protocols Hybrid protocols Centralized protocols design of the CTA Data management with respect to Local and temporary retainment Inadequate documentation on Centralized storage and extended amount of data being collected, of data data storage retainment of data retained and saved Data security refers to in-built Security maintained with mechanisms to prevent tampering Partial compliance to No consideration of data pseudorandom identifiers and or altering of data while on the security requirements security needs frequent renewals device, during access or transfer Swift and user-controlled Additional steps involved in Activation and Cumbersome to activate or activation and deactivation deactivation such as calling deactivation process impossible to deactivate process through app of authorities Nature of CTA’s federating body State-owned body N/A N/A Mandatory or voluntary Voluntary Partially voluntary Fully enforced download and use of CTA Open-sourced and CTA is open-sourced or not Open-sourced and unsupported Not open-sourced allows contributions

Some of the other factors such as legislative policies, incidental finding or dual-use policy mentioned in Vinuesa et al. (2020) were difficult to compare due to nonavailability of information to make reasonable comparisons between the 17 CTAs. It is also assumed factors such as status of fundamental rights of individuals and any discrimination or stigmatization are low or fairly similar across countries not influential in this analysis. Several quantitative data points were systematically analyzed from the 17 countries. These included considering their total population, the number of COVID-19 cases in their countries and the number of CTA downloads to derive the percentage of adoptees. Additionally, the total number of cases, total number of deaths and total number of COVID- 19 tests taken per million of the population were tabulated. Viewing this data set helped ensure and confirm our objective of selecting countries that captured a wide spectrum of impact from COVID-19 in their respective countries. The quantitative factors coded into the framework included the user review and star ratings of the respective apps. This involved looking at both reviews and star ratings of 17 CTAs between March and September 2020. Analyzing the total number of reviews and responses to the reviews revealed that there Sustainability 2021, 13, 2912 8 of 19

was a direct correlation between the number of responses and the percentage of adoption. CTAs from six countries (i.e., Vietnam, Japan, France, Spain, Brazil, and Australia) in this cohort had provided no responses to reviews. Previous studies [42] have shown that user acceptability is critical in adoption of CTAs resulting in tangible management of the spread of COVID-19. More specifically the privacy concerns were found to be the largest impediment for acceptance and usage in developed countries such as Germany and US. The model used in their work, i.e., health belief model, anchors on the user perception of both i.e., consequences of a health threat and how effective the proposed ways of its management or protection from it are. Thus, direct experiences and opinions of users are important to accommodate in viewing the impact of CTAs holistically. The user ratings utilized in our framework included their review-based ratings and star ratings. A one- or two-star rating was considered inadequate and earned a score of 0 while a four-star or five-star rating was assigned a score of 2. Additionally, Google’s ratings between 0 and 5 were also captured and linearly extrapolated and mapped to score of 0–2. Comparisons between Google’s app ratings and user’s review ratings showed reasonable congruence between them in most cases. We first provided a characterization of all the CTAs from Google Play Store. Then we analyzed the parameters like number of installs, average star ratings, total reviews, reviews by star ratings from the website applfow.io and androidrank.org. The COVIDTAS score was derived by averaging scores of all 12 factors in the framework. All factors included in the framework were given equal weightage.

8. Results and Discussion The COVIDTAS framework evaluated whether the 17 state-owned CTAs selected for this study satisﬁed parameters such as privacy and data protection, transparency rights, data management, etc. as shown in Table4. Each CTA was thoroughly studied to identify if the apps were compliant with the 12 parameters that were the foundation of the COVIDTAS scale. Overall, the New Zealand app scored the highest (1.82), followed by the apps from Canada and Australia (1.67), and those from Japan and Germany (1.57). The apps that scored less than 1 were from Qatar and Columbia. Though the EHTERAZ tracing app from Qatar was legally enforced on the public, it was not available in the local language and did not support app-related education and tutorials for the public. The app scored 0 in other aspects as well such as transparency rights, ease of accessibility and security. The South American apps also scored low on the COVIDTAS scale (Brazil 0.67, Columbia 0.58) as they scored 0 in all other parameters. It was also unclear if these state-owned apps can be removed or deactivated easily. All these apps (Middle East and South America) and COVID-19MX from Mexico did not comply with data minimization principles and were found to be a security risk to their users. The research involved analyzing the average star rating given to the apps by users. To do so, the 1 to 5 scale of Google Play Store were converted into 0, 1, 2, wherein 1–2 was scored 0, 2.01–3.50 was scored 1, and 3.51–5.0 was scored 2. It was determined that the apps from Asia, North America, Columbia, and Africa received the highest rating of 2, whereas apps from Japan, Israel, UAE, New Zealand, Australia, Europe, and Brazil received a one-star rating. When exploring the review ratings for these apps, only six apps scored a rating of 2, namely, India, Vietnam, Canada, Mexico, Columbia, and Morocco. A higher percentage of users (68% to 89%) gave these six apps a four- or ﬁve-star rating. The other 11 CTAs scored 1 for this parameter. Sustainability 2021, 13, 2912 9 of 19

Table 4. (a) COVIDTAS framework. (b) COVIDTAS framework. (a) Bluezone—Contact Parameters Aarogya Setu TraceTogether COCOA—COVID-19 TraceCovid EHTERAZ COVID Alert COVID-19MX COVIDSafe Detection Country India Singapore Vietnam Japan UAE Qatar Canada Mexico Australia Continent Asia Asia Asia Asia Asia Asia N. America N. America Oceania Privacy and 1 1 1 1 0 0 2 0 2 data protection Transparency rights 2 1 0 2 0 0 1 0 2 Accessibility 1 1 1 1 1 0 1 0 2 Education and tutorials 2 2 1 2 1 0 2 0 2 Decentralized protocol 1 0 2 2 2 0 0 0 0 Data management 1 2 2 2 0 0 2 0 2 Security 1 2 2 1 0 0 2 0 2 App easy to 2 1 1 2 2 1 2 1 2 deactivate/remove Public ownership 2 2 2 2 2 2 2 2 2 Use 1 2 2 2 2 0 2 2 2 Google Play Store ratings 2 2 2 1 1 2 2 2 1 Google Play Store reviews 2 1 2 1 1 1 2 2 1 COVIDATAS score 1.50 1.42 1.50 1.58 1.00 0.50 1.67 0.75 1.67 Sustainability 2021, 13, x FOR PEER REVIEW 8 of 17

belief model, anchors on the user perception of both i.e., consequences of a health threat and how effective the proposed ways of its management or protection from it are. Thus, direct experiences and opinions of users are important to accommodate in viewing the Sustainability 2021, 13, x FOR PEER REVIEW 8 of 17 impact of CTAs holistically. The user ratings utilized in our framework included their review-based ratings and star ratings. A one- or two-star rating was considered inadequate and earned a score of 0 while a four-star or five-star rating was assigned a belief model, anchors on the user perception of both i.e., consequences of a health threat score ofand 2. Additionally, how effective the Google’s proposed ratings ways betweenof its management 0 and 5 were or protection also captured from it andare. Thus,linearly extrapolateddirect experiencesand mapped and to opinions score of of0–2. users Comparisons are important between to accommodate Google’s inapp viewing ratings the and user’s reviewimpact ofratings CTAs showed holistically. reasonable The user congruenceratings utilized between in our themframework in most included cases. their Wereview-based first provided ratings a characterization and star ratings. of all A th one-e CTAs or two-starfrom Google rating Play was Store. considered Then we analyzedinadequate the parameters and earned like a score number of 0 whileof inst a foalls,ur-star average or five-star star ratings,rating was total assigned reviews, a reviewsscore by star of 2. ratings Additionally, from the Google’s website ratings applfow.io between and0 and androidrank.org. 5 were also captured The and COVIDTAS linearly score wasextrapolated derived and by mappedaveraging to score scores of 0–2. of Comparisonsall 12 factors between in the Google’s framework. app ratings All factors and user’s review ratings showed reasonable congruence between them in most cases. included in the framework were given equal weightage. We first provided a characterization of all the CTAs from Google Play Store. Then we analyzed the parameters like number of installs, average star ratings, total reviews, 8. Resultsreviews and byDiscussion star ratings from the website applfow.io and androidrank.org. The COVIDTAS Thescore COVIDTAS was derived framework by averaging evaluated scores whetherof all 12 thefactors 17 state-ownedin the framework. CTAs All selected factors for this studyincluded satisfied in the parameters framework weresuch givenas priv equalacy weightage. and data protection, transparency rights, data management, etc. as shown in Table 4. Each CTA was thoroughly studied to identify 8. Results and Discussion if the apps were compliant with the 12 parameters that were the foundation of the COVIDTASThe scale. COVIDTAS framework evaluated whether the 17 state-owned CTAs selected for this study satisfied parameters such as privacy and data protection, transparency rights, data management, etc. as shown in Table 4. Each CTA was thoroughly studied to identify Table 4. (a) COVIDTAS framework. (b) COVIDTAS framework. if the apps were compliant with the 12 parameters that were the foundation of the COVIDTAS scale. (a)

TableBluezone— 4. (a) COVIDTAS framework. (b) COVIDTAS framework. Aarogya TraceToget COCOA— COVID Parameters Contact TraceCovid EHTERAZ COVID-19MX COVIDSafe Setu her COVID-19 (a) Alert Detection Country India Singapore VietnamBluezone— Japan UAE Qatar Canada Mexico Australia Aarogya TraceToget COCOA— COVID Parameters Contact TraceCovid EHTERAZ COVID-19MX COVIDSafe Continent Asia Setu Asia her Asia AsiaCOVID-19 Asia Asia N. AmericaAlert N. America Oceania Privacy and data Detection 1 1 1 1 0 0 2 0 2 protection Country India Singapore Vietnam Japan UAE Qatar Canada Mexico Australia Continent Asia Asia Asia Asia Asia Asia N. America N. America Oceania Transparency rights 2 1 0 2 0 0 1 0 2 Privacy and data 1 1 1 1 0 0 2 0 2 Accessibility protection1 1 1 1 1 0 1 0 2 Education and Transparency 2rights 2 2 1 1 0 2 2 1 0 0 0 2 1 0 0 2 2 tutorials Accessibility 1 1 1 1 1 0 1 0 2 DecentralizedEducation and 1 2 0 2 2 1 2 2 2 1 0 0 0 2 0 0 2 0 protocol tutorials Decentralized Data management 1 1 2 0 2 2 2 2 0 2 0 0 2 0 0 0 0 2 protocol Security 1 2 2 1 0 0 2 0 2 Data management 1 2 2 2 0 0 2 0 2 App easy to Security 2 1 1 2 1 2 2 1 2 0 1 0 2 2 0 1 2 2 deactivate/remove App easy to 2 1 1 2 2 1 2 1 2 Public ownershipdeactivate/remove 2 2 2 2 2 2 2 2 2 Use Public ownership1 2 2 2 2 2 2 2 2 2 0 2 2 2 22 2 2 Google Play Store Use 1 2 2 2 2 0 2 2 2 2 2 2 1 1 2 2 2 1 Google Play Store Sustainability 2021, 13, 2912 ratings 2 2 2 1 1 2 2 2 10 1 of 19 Google Play Storeratings Google Play Store2 1 2 1 1 1 2 2 1 reviews 2 1 2 1 1 1 2 2 1 reviews COVIDATAS score 1.50 1.42 1.50 1.58 1.00 0.50 1.67 0.75 1.67 COVIDATAS score 1.50 1.42 1.50 1.58 1.00 0.50 1.67 0.75 1.67 Table 4. Cont. (b) (b) (b) NZ Corona-NZ Corona- Radar Coronavirus—Radar Coronavirus— המגן 2 המגן- 2 - האפליקציה האפליקציה הלאומית למלחמה למלחמה בנגיף בנגיף הקורונה הקורונה وﻗﺎﻳﻨﺎ وﻗﺎﻳﻨﺎ Parameters NZ COVID Tracer Coro-na-Warn-App Stop-CovidParameters Radar Parameters COVIDCOVID COVIDWarn- Corona-virus—SUS StopCovidWarn- StopCovid CoronAppCoronAppCoronApp COVIDCOVIDSUS SUS Tracer TracerApp App Country New Zea-land Germany France SpainNew Brazil Columbia Mo-rocco Israel CountryNew Germany France Spain Brazil Columbia Morocco Israel Country GermanyZealand France Spain Brazil Columbia Morocco Israel Zealand Continent Oceania Europe Europe EuropeContinent Oceania S.Europe America Europe Europe S. America S. America S. America Africa Africa Asia Asia Continent Oceania Europe Europe Europe S. America S. America Africa Asia Privacy and 2 2 2 2 0 0 1 1 data protection Transparency rights 2 2 2 1 0 0 0 2 Accessibility 1 1 2 1 1 0 1 1 Education and tutorials 2 1 1 1 0 0 0 1 Decentralized pro-tocol 2 2 0 2 0 0 1 1 Data management 2 2 2 2 0 0 2 2 Security 2 2 2 2 0 0 1 2 App easy to 2 2 2 1 1 1 1 1 deac-tivate/remove Public ownership 2 2 2 2 2 2 2 2 Use 2 2 2 2 2 0 2 0 Google Play Store ratings 1 1 1 1 1 2 2 1 Google Play Store reviews 1 1 1 1 1 2 2 1 COVIDATAS score 1.75 1.67 1.58 1.50 0.67 0.58 1.25 1.25 Sustainability 2021, 13, 2912 11 of 19

Some interesting observations can be made regarding the performance of these COVID- 19 apps by employing the COVIDTAS framework. The stress placed on privacy, transparency, and data security seems to be dependent on the state of development of the country involved. Each CTA was examined under the “Privacy and data protection” COVIDTAS parameter to assess if data collection was in agreement with the General Data Protection Regulation (GDPR), user’s privacy was respected, data protection impact assessment was done, and other such variables given in the Table3. It was seen that CTAs from Europe, Australia, New Zealand, and Canada completely fulfilled these requirements, whereas countries from the Middle East (UAE, Qatar) were not adequate, which, as mentioned above, was reflected in their low COVIDTAS score (1, 0.5, respectively). To achieve complete protection of users’ data, CTAs should ideally allow interoper- ability, use Bluetooth technology and follow a complete decentralized protocol. Those that followed a centralized approach were from Singapore (TraceTogether), Qatar (EHTERAZ), Canada (COVID Alert), Mexico (COVID-19MX), Australia (COVIDSafe), France (Stop- Covid), and South America (Coronavirus—SUS, CoronApp). Israel, Canada, Australia, New Zealand, and European countries score high in parameters related to privacy, transparency, data management, and security. New Zealand, Germany, Australia, and France score a maximum possible score of 8 (2 in each category) while Israel, Canada, and Spain score 7. It is also interesting to note that these ‘Western’ countries score higher than their Asian counterparts of similar socioeconomic development. Singapore and Japan, the two developed Asian economies considered in this study score 6 points in this regard. Vietnam and India, the two developing Asian economies also score 6 each in this regard. Countries in the Middle East (with the exception of Israel, which is a democracy) and South America (with relatively unstable governments) seem to fare the lowest in this regard, with Brazil, Columbia, UAE, and Qatar scoring zero across these parameters. It is safe to assume that healthy democracies fare better in terms of data integrity and security due to higher levels of accountability to the public. In addition, economic development usually allows for greater data protection and citizens of developing countries face greater challenges in this regard [43]. This trend continues with COVID-19 applications as well. When factors that determine the effectiveness of the application such as decentralization, accessibility, and awareness raising (through education) are considered, New Zealand and Japan score the highest points, while India, Vietnam, Australia, Germany, and Spain are close behind. Large scale advertisements and interventions regarding COVID-19 and CTAs in general have been shown to improve acceptability of the application [44]. This characteristic does not seem to be dependent on geography, socioeconomic development, or type of government. Ten apps were observed to be educative and user friendly as they provided reading materials and tutorials within the app and external access to websites to improve awareness on the disease as well as information on using the app correctly. However, the apps from Qatar, Mexico, Europe (Germany provided website access), South America, and Africa did not provide any public health awareness or app information. In countries such as India, use of CTAs was mandated in certain locations such as airports, government offices while it was not mandated at other public places such as malls and recreational sites. The ease of removing the app from the device and whether the installation is forced legally in any way are both measures of the degree of control citizens have over the application. In this regard, much like with data privacy, developed ‘Western’ economies (Australia, New Zealand, Canada, Germany, and France) score the highest values along with Japan, and UAE. It should be noted that Singapore, a developed economy from Asia does not score as highly in this regard, much like with data security. The only serious deviation in the trend between data security and the degree of control citizens have over the application itself is in UAE, where the data security and privacy itself is very low, but citizens are not legally required to install the application and can deactivate the application Sustainability 2021, 13, 2912 12 of 19

at will. This may be because the UAE government relied more on ground teams for testing rather than contact tracing for managing the pandemic [38,45]. The signiﬁcant trends to be observed with regards to privacy is that both the form of government and state of socioeconomic development seems to have a relation to the importance placed on data privacy. Stable democracies with better development indicators seem to fare better in this regard while developing countries and unstable democracies and monarchies are less so. Within developed democracies, Western countries fare better than their eastern and Middle Eastern counterparts. With exceptions, these same countries also give a greater degree of control to its citizens regarding the installation and utilization of the application. The effectiveness of the application in terms accessibility, awareness generation and decentralization does not seem to have any dependence on the developmental status or governance style of the country.

9. Public Perception of CTAs At the beginning, there was a lot of uncertainty and inexperience in managing the COVID-19 pandemic. Countries around the world tackled it in different ways. Vietnam and Singapore developed strategies to eradicate the virus, Australia tried to arrest and control the transmission, whereas Sweden tried to build herd immunity. Another point to note was the response time taken by the authorities to suppress virus transmission, Italy had a comparatively slower response in contrast to South Korea which had a faster response in preventing the infection. The study reviewed 17 CTAs that targeted the COVID-19 disease from all over the world. The customer ratings of these applications are provided in Table5. CTAs that were centrally, federally and government sponsored were included in the study. Based on average customer ratings (>4) on Google Play Store, it was observed that certain country-specific apps had scored the higher ratings, namely Bluezone—Contact detection (Vietnam; 4.62), Aarogya Setu (India, 4.38), Wiqaytna (Morocco; 4.36;), and CoronApp (Columbia; 4.03). Bluezone—Contact detection app was launched in April and has the second highest ratings on Google Play (total ratings: 67,422) after India. Overall average ratings demon- strated that a high proportion of users were satisfied with the quality of the app for both Vietnamese (84%) and English (61%) versions, with the Vietnamese app scoring higher. In total, 12% gave one or two stars for the Vietnamese version, and for the English version 27%. Total reviews received from users was 19,196 with 1600 reviews/month. In total, 2914 reviews received replies with 243 replies coming every month. Aarogya Setu app from India is one of the most popular CTAs with more than 100 million downloads and the highest number of ratings from Play Store (1,335,785). The app also received the most reviews (54,043; 4504 reviews/month) and replies to reviews (33,385; 2782 replies/month). A total of 79% of users were satisfied with the performance of this app with 16% giving it a one- or two-star rating. Compared to all other CTAs, the average star rating (4/5) for Wiqaytna CTA (Morocco) was the highest (89%) indicating that users were satisfied with the quality and performance of this app, whereas only a small percentage (9%) disliked or hated the app. In the Play Store, 12,300 total ratings, and 4723 total reviews with 394 reviews/month was observed. The app received 309 overall replies to reviews and 26 replies/month. CoronApp from Columbia was launched in March, making it one of the earliest CTA launches. After India, this app received the most replies to reviews. About 64,511 users rated this app, which received 35,968 reviews and 2997 reviews/month. There were 31,202 replies to reviews and 2600 replies/month. Users gave a four- or five-star rating (68%), a few gave one- or two-star ratings (23%). Tracing apps that received the lowest user ratings (<3) were NZ COVID Tracer (2.48; New Zealand), COVIDSafe (2.75; Australia), StopCovid (2.88; France), and COCOA— COVID-19 (2.99; Japan). Sustainability 2021, 13, x FOR PEER REVIEW 11 of 18

Avg. Star Rat- Table 5. GoogleAvg. Rat- Play Store COVID-19 CTA ratings. Coun ing (0 1 2) 1–2, Avg. Total Rat- Avg. Rating by Total Re- CTA Name URL ings by Re- Total Replies try 2.01–3.50; 3.51– Ratings Avg.ings Star Rating (0 1 2) 1–2,Reviews (0 1 2) views Avg. Ratings Avg. Rating by CTA Name Country URL views Avg. Ratings Total Ratings Total Reviews Total Replies 5.00 2.01–3.50; 3.51–5.00 by Reviews Reviews (0 1 2) Aarogya Setu India nic.goi.aarogyasetu nic.goi.aarogyasetu 2 4.376 1,335,785 4.28 2 54,043 33,385 Aarogya Setu India 2 4.376 1,335,785 4.28 2 54,043 33,385 Sin- (accessed on 7 March 2021) sg.gov.tech.bluetrace TraceTogetherTraceTogether gapor sg.gov.tech.bluetrace Singapore 2 4.024 7383 23.112 1 4.024 2707 7383893 3.112 1 2707 893 e (accessed on 7 March 2021) Vi- com.mic.bluezone (accessed Bluezone—ContactBluezone—Contact detection Vietnam 2 4.623 67,422 4.373 2 19,196 2914 etna com.mic.bluezone on 7 March2 2021) 4.623 67,422 4.373 2 19,196 2914 detection jp.go.mhlw.covid19radar COCOA—COVID-19m Japan 1 2.989 6295 2.871 1 3157 0 COCOA—COVID-19 Japan jp.go.mhlw.covid19radar(accessed on1 7 March 2021) 2.989 6295 2.871 1 3157 0 המגן 2 - האפליקציה com.hamagen (accessed on Israel com.hamagenIsrael 1 3.418 4629 13.278 1 3.4181739 4629889 3.278 1 1739 889 הלאומית למלחמה בנגיף 7 March 2021) הקורונה TraceCovid UAE ae.tracecovid.app 1 3.373 314 2.904 1 167 24 ae.tracecovid.app (accessed TraceCovid UAE 1 3.373 314 2.904 1 167 24 EHTERAZ Qatar com.moi.covid19 on 7 March 2 2021) 3.926 26,194 3.053 1 9685 434 Can- ca.gc.hcsc.canada.stop-com.moi.covid19 (accessed COVIDEHTERAZ Alert Qatar 2 3.872 3033 23.966 2 3.926 1383 26,194 205 3.053 1 9685 434 ada covid on 7 March 2021) Mex- ca.gc.hcsc.canada.stopcovid COVID-19MXCOVID Alert Canadamx.gob.www 2 3.602 3116 24.021 2 3.8721810 303338 3.966 2 1383 205 ico (accessed on 7 March 2021) Aus- mx.gob.www (accessed on COVID-19MXCOVIDSafe au.gov.health.covidsafe Mexico 1 2.753 12,909 23.147 1 3.602 7353 3116 0 4.021 2 1810 38 tralia 7 March 2021) New au.gov.health.covidsafe COVIDSafenz.govt.health.covid- Australia 1 2.753 12,909 3.147 1 7353 0 NZ COVID Tracer Zea- (accessed on1 7 March 2021)2.478 2816 2.115 1 2200 1038 tracer land nz.govt.health.covidtracer NZ COVID Tracer New Zealand 1 2.478 2816 2.115 1 2200 1038 Ger- (accessed on 7 March 2021) Corona-Warn-App de.rki.coronawarnapp 1 3.38 72,971 3.438 1 27,247 5679 many de.rki.coronawarnapp Corona-Warn-App Germany 1 3.38 72,971 3.438 1 27,247 5679 Franc fr.gouv.android.stop-(accessed on 7 March 2021) StopCovid 1 2.88 9121 2.864 1 4741 0 e covid fr.gouv.android.stopcovid StopCovid France 1 2.88 9121 2.864 1 4741 0 Radar COVID Spain es.gob.radarcovid(accessed on1 7 March 2021) 3.181 1778 2.902 1 942 0 Coronavírus—SUS Brazil br.gov.datasus.guardioeses.gob.radarcovid 1 (accessed 3.238 14,080 2.829 1 6658 0 Radar COVID Spain 1 3.181 1778 2.902 1 942 0 Co- on 7 March 2021) CoronApp lum- co.gov.ins.guardianesbr.gov.datasus.guardioes 2 4.033 64,511 3.601 2 35,968 31,202 Coronavírus—SUS Brazil 1 3.238 14,080 2.829 1 6658 0 bia (accessed on 7 March 2021) Mo- co.gov.ins.guardianes CoronApp (wiqaytna covid.trace.morocco Columbia 2 4.359 12,300 24.566 2 4.033 4723 64,511309 3.601 2 35,968 31,202 rocco (accessed on 7 March 2021) covid.trace.morocco wiqaytna) Morocco 2 4.359 12,300 4.566 2 4723 309 CTAs(accessed that were on 7 centrally, March 2021) federally and government sponsored were included in the study. Based on average customer ratings (>4) on Google Play Store, it was observed that certain country-specific apps had scored the higher ratings, namely Bluezone—Contact detection (Vietnam; 4.62), Aarogya Setu (India, 4.38), Wiqaytna (Morocco; 4.36;), and CoronApp (Columbia; 4.03). Bluezone—Contact detection app was launched in April and has the second highest ratings on Google Play (total ratings: 67,422) after India. Overall average ratings demon- strated that a high proportion of users were satisfied with the quality of the app for both Vietnamese (84%) and English (61%) versions, with the Vietnamese app scoring higher. Sustainability 2021, 13, 2912 14 of 19

Implementing strict lockdown strategies, improved rapid testing facilities, and case management has enabled New Zealand to contain the COVID-19 outbreak twice. For the purpose of tracing and isolating COVID-19 contacts, New Zealand health authorities developed the NZ COVID Tracer CTA. The app received 2816 ratings on Google. The number of total reviews received was 2200 and 183 reviews/month. In total, this app received 1038 replies to reviews and 87 replies/month. User satisfaction rating (four or five stars) was observed to be 23% in contrast to 68% who gave one- or two-star ratings (hated or disliked the app). COVIDSafe from Australia received a total of 12,909 ratings, with 7353 reviews in total and 613 reviews/month. This app received no replies to reviews. Half of the users gave a rating of four or five stars and 44% gave it one star or two stars. In June, the StopCovid app from France was released to the public in English and received 9121 ratings. Out of 4741 reviews, this app received 395 reviews/month. An interesting observation noted with this app was that there were no replies to reviews. In total, 53% of the public gave it four or five stars and 38% gave it a one- or two-star rating. In Japan, COCOA—COVID-19 was launched in June in the local language. It received 6295 overall ratings. This app received 3157 reviews with 263 reviews/month. Like the StopCovid app from France, this app also received no replies to reviews at all. Users who gave it a four- or five-star rating (38%) were lower than those who rated it one or two stars (46%). It is evident that the public perception of the application has a correlation to the utility or data privacy measures taken by the application. Countries like India, Vietnam, and Canada which scored above 1.5 also scored high in customer reviews. Morocco, which scored 1.25 in the framework also has the highest overall review score. However, it is evident that the COVIDTAS framework cannot directly measure customer experience. Countries like Australia, New Zealand, Germany, France, and Spain which had COVIDTAS scores above 1.5 have relatively low average ratings, with the application in New Zealand scoring a low 2.12 average score. Surprisingly, the application from Mexico which had a low COVIDTAS score of 0.75 has a remarkably high average customer review score of 4.02. Therefore, it is best to view customer experience and the effectiveness of the application as viewed by the COVIDTAS framework as two separate metrics for assessing the application. A more detailed analysis of the CTAs from the perspective of customer experience is provided in Table6. India, Vietnam, Canada, Mexico, and Morocco have a higher percentage of four- and five-star reviews, indicating greater customer satisfaction. Similarly, the number of times the application has responded to its customers is also a key indicator. When normalized by the number of reviews, it is seen that application from Colombia has responded to 83% of the reviews. Similarly, India and Israel also have high response rates of 61% and 54%, respectively. However, no direct correlation to customer satisfaction could be observed. In fact, highly rated applications from Morocco and Canada have relatively low response rates. This could partly be because customer complaints are more likely to warrant a response from the developer, leading to a larger number of responses to applications. In addition, the approach taken by the nation towards promoting the application could also play a part in the response rate. For example, in a country like India with relatively low technology literacy, the application was made necessary for various travel processes. Therefore, the application had to engage with its users and ensure that any difficulties in using the application be sorted out. Such interventions may not have been necessary in Morocco or Canada, where the application was entirely voluntary. Sustainability 2021, 13, 2912 15 of 19

SustainabilitySustainability 2021, 202113, x ,FOR 13, PEERx FOR REVIEW PEER REVIEW 14 of 18 13 of 17

Sustainability 2021, 13, x FOR PEER REVIEW 14 of 18 Table 6. Google Play Store user reviews for CTAs. Sustainability 2021, 13, x FOR PEER REVIEW 14 of 18

Total Total Four, Five One, Two Country CTA Name One-Star % Two-Star % Three-Star % Four-Star % Five-Star % Table 6. Google Play Store user reviews for CTAs. Reviews Replies Star Reviews Star Reviews India Aarogya Setu 5658 13%Table 1297 6. Google 3% Play Store 1864 user reviews 4% for 3413 CTAs. 8% 30,258 71% 54,043 33,385 1% 0% Table 6. Google Play Store user reviews for CTAs. Four, Five One, Two One- Two- Three- Four- Total Re- Total Re- CountrySingapore TraceTogetherCTA Name 746% 32% 188% 8%% 285% 12% Five-Star 216 % 9% 912 39%Star Re- 2707Star Re- 893Four, 0% Five One, 0%Two StarTable 6. GoogleStar Play Store userStar reviews for CTAs.Star views plies Four, Five One, Two Four, Five One, Two One- Two-One- Three-Two- Four-Three- Four- Total Re- Total Re- viewsTotal views Total CountryVietnamCountry Bluezone—ContactCTA Name DetectionCTA Name (English) 115% 21%One- % 32 % Two- 6%% % 61Three-% 11%Five-Star% 51Four-% 9%% Five-Star 281 52%% Star Re-Total 540Star Re- Total 39 Star 1% Star 0% Country CTA Name Star StarStar % StarStar %Star Star % Star views% Five-Starplies % Four, FiveReviews One, TwoReplies Star Star India Aarogya Setu One- 5658 13% Two-Star1297 3% Three-1864Star 4% Four-3413 Star8% 30,258 Star71% Total 54,043 Re- Total 33,385 Re- 1%Reviews 0% Replies Vietnam Bluezone—Contact Detection (Vietnamese) 1595 9% 470 3% 865 5% 1517 8% 13,736 76%views 18,183views 2875Reviews 1% Reviews 0% SingaporeCountry TraceTogether CTA Name 746 32%% 188 8%% 285 12%% 216 9%% Five-Star 912 39%% 2707 893 Star 0% Re- Star 0% Re- Reviews Reviews IndiaIndia Aarogya Aarogya Setu Setu 5658Star 13% 1297 5658Star 3%13% 1864Star1297 4% 3%3413Star 18648% 30,2584% 341371% 54,0438%views 30,258 33,385plies 71% 1% 54,043 0% 33,385 1% 0% India AarogyaCOCOA—COVID-19 Setu Contact App 5658 13% 1297 3% 1864 4% 3413 8% 30,258 71% views 54,043 views 33,385 1% 0% SingaporeVietnamJapan TraceTogetherBluezone—Contact Detection (English) 746 1151000 32%21% 33%18832 8%6% 387 28561 13%12%11% 50321651 9%9% 16% 912281 348 39%52% 11% 2707 540 838 893 39 27% 0% 1% 3157 0% 0% 0 0% 0% IndiaVietnamSingapore Aarogya Bluezone—Contact TraceTogether Setu(Japanese) Detection (Vietnamese) 5658 1595 13%9% 1297 746470 3% 3%32% 1864 865188 4% 5% 8%34131517 2858%8% 12%30,25813,736 21671%76% 9%54,043 18,183 912 33,385 2875 39% 1% 1% 2707 0% 0% 893 0% 0% VietnamSingapore Bluezone—Contact TraceTogether Detection (English) 115746 21%32% 18832 6% 8% 28561 11%12% 21651 9% 281 912 52% 39% 2707 540 893 39 1%0% 0% JapanVietnam COCOA—COVID-19Bluezone—Contact(English) Contact Detection App (Japanese) (English) 1000 33% 115387 13%21% 503 32 16% 6% 348 6111% 11% 838 27%51 9% 3157 281 0 52% 0% 540 0% 39 1% 0% VietnamIsrael Bluezone—Contact Detection (Vietnamese) 1595121 9% 35% 470 3% 57 865 16% 5% 1517 40 8% 12% 13,736 3476% 10% 18,183 94 2875 27% 1% 346 0% 121 0% 1% 0% 1% 2875 1% 0% 18,183 0% 1% 76% 12139 13,736 346 540 8% 27% 151752% 94 5%281 86510%9% 5134 3% 11%12% 470 6140 16%6%9% 15953257 21%35% 115121 המגן Vietnamese) - 2) האפליקציה (Detection(Englishהלאומית למלחמה Detection בנגיף הקורונה (VietnamIsraelVietnam Bluezone—Contact(English Bluezone—Contact JapanVietnamVietnam COCOA—COVID-19Bluezone—Contact Bluezone—Contact Detection Contact DetectionApp(Vietnamese) (Japanese) (Vietnamese) 10001595 33%9% 1595387 470 13% 3%9% 503 865 470 16% 5% 3%1517348 86511%8% 13,736 5% 838 1517 27%76% 8% 18,183 3157 13,736 2875 0 76% 0%1% 18,183 0% 2875 1% 0% 0% 0% 0 0% 3157 1% 27% 673 838 1054 11% 34849% 16%517 5037% 78 13% 9% 387 99 7%33% 100076 27% 284 המגן המגןApp - -2(Japanese) 2 האפליקציה האפליקציה Contact הלאומית הלאומית למלחמה למלחמה בנגיף בנגיף הקורונה הקורונה( (IsraelJapan ((Hebrew COCOA—COVID-19 IsraelJapanJapan COCOA—COVID-19English COCOA—COVID-19(Hebrew) Contact App Contact (Japanese) App (Japanese) 1000 121 35%33% 100038757 16%13%33% 50340 387 12%16% 13%34834 50310%11% 16% 838 94 348 27% 11% 3157 346 838 121 0 27% 0% 3157 1% 0% 0 0% 0% 0% 1% 673 0% 1054 0% 49% 95 517 224 7% 33% 78 73 9% 8% 9918 7%13% 30 76 10% 27%23 36% 80284 המגןהמגן 22 -- האפליקציההאפליקציה הלאומיתהלאומית למלחמהלמלחמה בנגיףבנגיף הקורונההקורונה ((Israel Israel ((Russian 1% 0% 121 0%1% 346 1%0% 27% 673121 94 346 1054 10% 27%34 49% 94 12%517 4010%7% 16%7834 12%9% 57 9940 16%7%35% 1217657 27%35%המגן 2 - 284121האפליקציההמגן 2 - הלאומית האפליקציה למלחמה הלאומית בנגיף למלחמה הקורונה בנגיף( הקורונה (IsraelIsraelIsrael (HebrewEnglish(English 0% 0% 1% 0% 24 95 167 224 33%43% 58 73 8%7% 1810 13%10% 3014 10%9% 2312 36%30% 40 80 המגן 2 - האפליקציה הלאומית למלחמה בנגיף הקורונה (IsraelUAE ( TraceCovidRussian 0% 1% 673 0% 1054 1% 49% 673 517 1054 7% 49%78 9%517 997% 78 7% 9% 76 99 7%27% 28476 27%המגן 2 - 284האפליקציההמגן 2 - הלאומית האפליקציה למלחמה הלאומית בנגיף למלחמה הקורונה בנגיף( הקורונה(IsraelIsrael (Hebrew(Hebrew) (Russian 0% 1% 673 0% 1054 0% 49% 434 517 9685 7% 41%78 9%3323 995% 436 7% 6% 76 493 7%27% 284530 41%המגן 2 - 3303 האפליקציה הלאומית למלחמה בנגיף הקורונה ( QatarIsrael EHTERAZ(Hebrew 0% 0% 95 0% 224 1% 33% 24 73 167 8% 1843% 58 13% 7% 3010 10%10% 14 23 9% 36%12 30% 4080 המגן 2 - האפליקציה הלאומית למלחמה בנגיף הקורונה ( )UAE Israel TraceCovid 0% 0% 95 0% 224 0% 33% 95 73 224 8% 33%18 73 13% 8% 30 10%18 13% 23 30 10%36% 2380 36%המגן 2 - 80האפליקציה הלאומית למלחמה בנגיף הקורונה (IsraelIsrael Russian(Russian QatarCanada EHTERAZ COVID Alert 3303 212 41%17% 53064 7%5% 49369 6%5% 43699 5%8% 3323836 41%65% 9685 1383 434 205 0% 1% 0% 0% UAEUAE TraceCovid TraceCovid 40 30% 12 40 9%30% 14 12 10% 9% 10 147% 10% 58 43%10 7% 167 58 24 43% 1% 167 0% 24 1% 0% MexicoUAEUAE COVID-19MX TraceCovid TraceCovid (Spanish) 293 40 17% 30% 4046 3%30% 12 94 9%12 5% 9% 14 162 149% 10% 10%1133 1066%10 7% 1810 58 38 43% 43% 1% 167 0% 24 24 1% 1% 0% 0% CanadaQatar COVIDEHTERAZ Alert 3303 212 17%41% 53064 5%7% 49369 5%6% 43699 8%5% 3323836 65%41% 13839685 205434 1%0% 0% AustraliaQatarQatar COVIDSafe EHTERAZ EHTERAZ 2597 3303 36% 41% 3303568 8% 53041% 498530 7% 7% 7% 493 439 4936% 6% 6%3189 43643644% 5% 7353 3323 3323 0 41% 41% 1% 9685 0% 434 434 0% 0% 0% 0% MexicoCanadaQatar COVID-19MXCOVID EHTERAZ Alert (Spanish) 293212 17% 33034664 3%5%41% 9469 530 5% 7%162 99 4939%8% 6%1133836 66%43665% 5% 18101383 3323 205 38 41% 1% 9685 0% 434 0% 0% NewCanada Zealand NZ COVID COVID Tracer Alert 1269 59% 212202 9%17% 165 64 8% 5% 157 69 7% 5% 343 16%99 8% 2200 836 1038 65% 0% 1383 1% 205 1% 0% AustraliaMexicoCanadaCanada COVIDSafeCOVID-19MX COVID COVID (Spanish) Alert Alert 2597 293 212 36%17% 17%568 21246 8%3%17% 64 49894 5%64 7%5% 5% 69439162 696%9% 5%5%31891133 9944%66%99 8% 73531810 836 836 38 0 65% 65% 1% 1383 0% 205 205 1% 1% 0% 0% Germany Corona-Warn-App (English) 250 21% 73 6% 82 7% 96 8% 687 58% 1188 402 1% 0% NewAustraliaMexico ZealandMexico NZCOVIDSafe COVID COVID-19MX COVID-19MX Tracer (Spanish) (Spanish) 12692597 293 59%36% 17% 202568 293 9%8%17% 46 165498 3%46 8%7% 3% 94157439 94 7%6% 5%5%3189 343 162162 16%44% 9% 22007353 1133 1133 1038 0 66% 66% 0%1% 1810 1%0% 38 38 1% 1% 0% 0% GermanyGermany Corona-Warn-App Corona-Warn-App (English) (German) 2506692 21%26% 193173 6%8% 230782 7%9% 227996 8%9% 12,153687 58%48% 25,3621188 4025277 1% 1% 0% 0% NewAustralia ZealandAustralia NZ COVID COVIDSafe COVIDSafe Tracer 1269 2597 59% 36% 2597 202 9% 56836% 165568 8% 8% 8% 498 157 498 7% 7%7% 343 439439 16% 6% 2200 3189 3189 1038 44% 44% 0% 7353 1% 0 0 1% 1% 0% 0% GermanyFrance Corona-Warn-App StopCovid France (English)(German) 6692 62 26%30% 193117 8%8% 230718 9%9% 227923 9%11% 12,15386 48%42% 25,362 206 5277 0 1% 1% 0% 0% GermanyNewNew Zealand Zealand Corona-Warn-App NZ NZ COVID COVID (English) Tracer Tracer 250 1269 21% 59% 126973 6% 20259% 82 202 9% 7% 9% 165 96 1658% 8% 8%687 157 15758% 7% 1188 343 343 402 16% 16% 1% 2200 0% 1038 1038 0% 0% 1% 1% FranceNew Zealand StopCovid NZ COVIDFrance (French) Tracer 1795 40% 1269 418 9% 59% 410 202 9% 9% 367 165 8% 8% 1475 157 33% 7% 4465 343 0 16% 0% 2200 0% 1038 0% 1% FranceGermany StopCovidCorona-Warn-App France (English) (German) 6692 62 30%26% 193117 8% 230718 9% 227923 11%9% 12,15386 42%48% 25,362 206 5277 0 1% 0% GermanyGermany Corona-Warn-App Corona-Warn-App (English) 250 21% 250 21% 73 6%73 6% 8282 7%7% 9696 8% 687 687 58% 58% 1188 402 402 1% 1% 0% 0% SpainGermany Radar Corona-Warn-AppCOVID (English) (English) 19 48% 2504 10%21% 1 73 3% 6% 5 8213% 7%11 28%96 8% 40 687 0 58% 0% 1188 1% 402 1% 0% France StopCovid France (French)(English) 1795 62 40%30% 41817 9%8% 41018 9% 36723 11% 8% 147586 33%42% 4465 206 0 0%1% 0% SpainGermany Germany Radar Corona-Warn-AppCOVID Corona-Warn-App (Spanish) (German) 276 6692 35% 26% 669282 10% 193126% 891931 8% 11% 8% 2307 61 2307 8% 9% 9% 284 2279227936% 9% 792 12,153 12,153 48% 48% 0% 25,362 25,362 0% 5277 5277 1% 1% 0% 0% SpainFranceGermany RadarStopCovid COVID Corona-Warn-App France (English) (French) (German) 1795 19 48% 40% 6692 4184 10% 9%26% 4101 1931 3%9% 8% 367 5 230713% 8% 9% 147511 227928% 33% 9% 4465 40 12,153 0 48% 0% 25,362 1%0% 5277 1% 0% BrazilFrance France Coronavirus—SUS StopCovid StopCovid France(English) France (English) 131 62 38% 30% 6232 9%30% 17 20 8%17 6% 8% 18 12 184% 9% 9%147 2343%23 11% 342 86 86 0 42% 42% 0% 206 0% 0 0 1% 1% 0% 0% SpainFrance Radar COVID StopCovid (Spanish)(English) France (English) 276 19 35%48% 82 624 10%30% 891 17 11% 3% 8% 61 5 1813% 8% 9% 28411 36%28%23 11% 792 40 86 0 42% 0% 206 0%1% 0 1% 0% Brazil Coronavirus—SUS (Portuguese) 2550 41% 592 10% 539 9% 407 7% 2081 34% 6169 0 0% 1% BrazilSpainFrance France Coronavirus—SUSRadar StopCovidCOVID StopCovid (Spanish) (English)France France (French) 131276 1795 38%35% 40% 17953282 10%9% 418 40% 2089 418 9% 11%6% 9% 410 1261 4104% 8% 9% 9%147 284 36743% 36736% 8% 342792 1475 1475 0 33% 33% 0% 4465 0% 0 0 0% 0% 0% 0% BrazilColumbia Coronavirus—SUS CoronApp—Colombia (Portuguese) 2550 220 41%51% 59234 10%8% 53946 9%11% 40743 7%10% 208192 34%21% 35,698 6169 31,202 0 0%68% 1%23% BrazilSpain Spain Coronavirus—SUS Radar Radar COVID COVID (English) (English) (English) 131 19 38% 48%32 19 9%48% 4 20 10%4 6% 10% 112 14% 3% 3%147 543% 5 13% 342 11 11 0 28% 28% 0% 40 0% 0 0 0% 0% 1% 1% %0 %1 19 420 %77 322 %7 30 %4 17 %3 12 %9 39 وﻗﺎﻳﻨﺎ (Morocco (Wiqaytna) (English ColumbiaBrazilSpain Spain CoronApp—ColombiaCoronavirus—SUS Radar Radar COVID COVID (Portuguese) (Spanish) (Spanish) 2550 220 276 51%41% 35%592 27634 10%8%35% 82 53946 10%82 11% 9% 10% 89 40743 8910% 7% 11% 11% 208192 6121%34%61 35,698 8% 6169 284 284 31,202 0 36% 36% 68% 0% 792 23% 1% 0% 0% 0% 0% Spanish) 220 8% 27636 1%35% 51 82 2% 10%169 896% 11%2401 83%61 8% 2877 284 290 36% 1% 792 0% 0% 0%)وﻗﺎ ﻳﻨﺎ MoroccoSpain (Wiqaytna) Radar (French) COVID %0 %23 %1 %68 19 31,202 420 35,698 %21%77 32292 %7%10 3043 %4%11 1746 %8%3 1234 %9%51 39 220 وﻗﺎﻳﻨﺎ (MoroccoColumbia (Wiqaytna) CoronApp—Colombia (English BrazilBrazil Coronavirus—SUS Coronavirus—SUS (English) 131 38% 131 38% 32 9%32 9% 20 20 6%6% 1212 4% 147 147 43% 43% 342 0 0 0% 0% 0% 0% %0 %1 19 290 420 2877 %77%83 2401322 %7%6 16930 %4%2 5117 %3%1 3612 %9%8 39 220 وﻗﺎوﻗﺎﻳﻨﺎ ﻳﻨﺎ (Morocco (Wiqaytna)(Wiqaytna) (French)(English BrazilBrazil Coronavirus—SUS Coronavirus—SUS (Portuguese) 2550 41% 2550 59241% 592 10% 10% 539 539 9% 9% 407 407 7% 2081 2081 34% 34% 6169 0 0 0% 0% 1% 1% Portuguese) 220 8% 255036 1%41% 51 592 2% 10%169 5396% 9%2401 40783% 7% 2877 2081 290 34% 1% 6169 0% 0 0% 1%) وﻗﺎﻳﻨﺎ (MoroccoBrazil (Wiqaytna) Coronavirus—SUS (French Columbia CoronApp—Colombia 220 51% 34 8% 46 11% 43 10% 92 21% 35,698 31,202 68% 23% Columbia CoronApp—Colombia 220 51% 34 8% 46 11% 43 10% 92 21% 35,698 31,202 68% 23% %0 %0 %1 %1 19 19 420 %77 %77 322 322 %7 3030 %4%4 17 17 %3 12%3 %129 39 %9 39 وﻗﺎﻳﻨﺎ (MoroccoMorocco (Wiqaytna)(Wiqaytna) (English) (English %0 %1 290 2877 %83 2401 %6 169 %2 51 %1 36 %8 220 وﻗﺎﻳﻨﺎ (Morocco (Wiqaytna) (French %0 %0 %1 %1 290 290 2877 %83 %83 2401 2401 %6 169169 %2%2 51 51 %1 36%1 %368 220 %8 220 وﻗﺎﻳﻨﺎ (MoroccoMorocco (Wiqaytna)(Wiqaytna) (French) (French Sustainability 2021, 13, 2912 16 of 19

Some of the challenges in our study were the lack of organized and comprehensive information of CTAs and the inability to download several of them due to lack of accessibility outside of their respective countries. Hence our reliance on collection of data had depended on publicly available information. The rapidity in launch and usage of CTAs was the need of the hour and ability to inculcate the right processes or approaches to CTAs in some countries could have been unintentionally compromised. Awareness and adoption of CTAs is no silver bullet in the battle against COVID- 19 pandemic. In the last few weeks, announcements about COVID-19 vaccines is very encouraging but the citizens should not become complacent about the use of CTAs. In this regard, Govt. leadership must also offer assurance to its citizens that their identify will be concealed and emphasize the beneﬁts of CTAs as it relates to shared public health.

10. Conclusions and Outlook The current COVID-19 pandemic has shown the importance of expert-based decision making from governments, and highlighted the importance of developing a transparent and inclusive communication campaign. An excellent example of this is the development of COVID-19 smartphone apps, which are aimed at performing contact tracing at a massive scale. As discussed in this study, there are a number of technical solutions to deploy COVID- 19 apps which are privacy-preserving, as well as effective when it comes to providing relevant epidemiological information to fight the pandemic. In many cases, the governments have not succeeded in conveying the message that decentralized digital contact tracing is not only able to fulfill all the ethical requirements, but it also provides invaluable information to contain the spread of the infection. This is particularly relevant in countries where less strict measures have been taken. For instance, it is surprising that Sweden has not developed a CTA, which could have significantly helped the lenient approach to the pandemic adopted by their Public Health Agency. Clearly, the current situation confronts the interests of governments, which want to gather as much information as possible to effectively tackle the COVID-19 spread, and individuals, who are concerned about their privacy and other essential ethical issues such as discrimination and stigmatization. In this work we proposed an evaluation framework, the COVID Tracing App Scale (COVIDTAS), to assess the compliance of CTAs with all these principles. This framework builds on other ethical studies [41], which were based on the 17 Sustainable Development Goals (SDGs) of the United Nations (UN) [46], adding other aspects such as user review and ratings. This study features one of the largest sociotechnical assessments of CTAs, where 17 government-owned technical solutions are thoroughly evaluated. Our analysis indicates that the CTA by New Zealand reached the highest score (1.82), followed by the CTAs by Canada and Australia (1.67), and those from Japan and Germany (1.57). Some important comments regarding the CTAs that scored less than 1 are given next, i.e., the ones from Qatar, Brazil and Colombia. For example, the EHTERAZ tracing app from Qatar was legally enforced to the public, but it was not available in the local language. Interestingly, the CTAs that obtained low scores are generally located in areas with significant data gaps and digital divides [47], which are critical not only terms of handling the pandemic but also regarding a more sustainable global future. The present framework can be used to analyze newly developed CTAs, so that the public can assess them and make informed decisions based on the compliance with the COVIDTAS methodology. Furthermore, we expect that a wide range of applications will rely on smartphone apps in the near future, ranging from economic issues to measurements of pollution. These apps may constitute an integral part of our lives, and we want to ensure that they will fulfill all the ethical requirements in terms of privacy and equality. Therefore, this framework can form the basis for the study of the future apps that will disrupt our societies. Future research in this area should also aim at gaining a more comprehensive understanding of the role of CTAs in managing this pandemic. In addition, as data stabilizes, more significant statistical analysis on each CTA can be carried out, identifying the best approach for data privacy, user trust, and disease tracking. Sustainability 2021, 13, 2912 17 of 19

Author Contributions: Conceptualization, R.R. and K.A.; methodology, R.R. and R.V.; validation, R.R. and K.A.; writing—R.R. and K.A and P.N.; writing—review and editing, P.N. and R.V.; All authors have read and agreed to the published version of the manuscript. Funding: This research received no external funding. Institutional Review Board Statement: Not Applicable. Informed Consent Statement: Not Applicable. Data Availability Statement: Not Applicable. Acknowledgments: This work derives direction and ideas from the Chancellor of Amrita Vishwa Vidyapeetham, Mata Amritanandamayi Devi. The authors would like to thank Adarsh, Romita S for their help in the study. Conﬂicts of Interest: The authors declare no conﬂict of interest.

References 1. Bhaskar, S.; Rastogi, A.; Menon, K.V.; Kunheri, B.; Balakrishnan, S.; Howick, J. Call for Action to Address Equity and Justice Divide During COVID-19. Front. Psychiatry 2020, 11, 559905. [CrossRef] 2. Ghate, S.; Parekh, B.J.; Thapar, R.K.; Nadkarni, P.R.; Sen, S.; Bansal, U.; Sambhariya, C.H.; Popat, S.; Bhattacharya, P.; Kirtani, S.; et al. Indian Academy of Pediatrics Guidelines on School Reopening, Remote Learning and Curriculum in and After the COVID-19 Pandemic. Indian Pediatr. 2020, 57, 1153–1165. [CrossRef] 3. Braithwaite, I.; Callender, T.; Bullock, M.; Aldridge, R.W. Automated and partly automated contact tracing: A systematic review to inform the control of COVID-19. Lancet Digit. Health 2020, 2, e607–e621. [CrossRef] 4. Klinkenberg, D.; Fraser, C.; Heesterbeek, H. The Effectiveness of Contact Tracing in Emerging Epidemics. PLoS ONE 2006, 1, e12. [CrossRef] 5. Eames, K.T.D.; Keeling, M.J. Contact tracing and disease control. Proc. R. Soc. B Boil. Sci. 2003, 270, 2565–2571. [CrossRef][PubMed] 6. Bahri, S. Enhancing quality of data through automated SARS contact tracing method using RFID technology. Int. J. Netw. Virtual Organ. 2007, 4, 145. Available online: https://ideas.repec.org/a/ids/ijnvor/v4y2007i2p145-162.html (accessed on 26 December 2020). [CrossRef] 7. Seshadri, S.; Gockowski, L.F.; Lee, J.; Sroda, M.; Helgeson, M.E.; De Alaniz, J.R.; Valentine, M.T. Self-regulating photochemical Rayleigh-Bénard convection using a highly-absorbing organic photoswitch. Nat. Commun. 2020, 11, 1–8. [CrossRef][PubMed] 8. Cullen, Z.; Farronato, C. Outsourcing Tasks Online: Matching Supply and Demand on Peer-to-Peer Internet Platforms. Manag. Sci. 2020.[CrossRef] 9. Bharadwaj, A. Vulnerable Truths: Research in the Climate of Political Control. Am. Anthr. 2020, 122, 386–387. [CrossRef] 10. Danquah, B.A.; Chirove, F.; Banasiak, J. Effective and ineffective treatment in a malaria model for humans in an endemic region. Afr. Mat. 2019, 30, 1181–1204. [CrossRef] 11. Tom-Aba, D.; Nguku, P.M.; Arinze, C.C.; Krause, G. Assessing the Concepts and Designs of 58 Mobile Apps for the Management of the 2014–2015 West Africa Ebola Outbreak: Systematic Review. JMIR Public Health Surveill. 2018, 4, e68. [CrossRef][PubMed] 12. World Health Organization. Go.Data. 2020. Available online: https://www.who.int/godata (accessed on 26 December 2020). 13. Ha, Y.P.; Tesfalul, M.A.; Littman-Quinn, R.; Antwi, C.; Green, R.S.; Mapila, T.O.; Bellamy, S.L.; Ncube, R.T.; Mugisha, K.; Ho-Foster, A.R.; et al. Evaluation of a Mobile Health Approach to Tuberculosis Contact Tracing in Botswana. J. Health Commun. 2016, 21, 1115–1121. [CrossRef][PubMed] 14. Ohannessian, R. Telemedicine: Potential applications in epidemic situations. Eur. Res. Telemed. 2015, 4, 95–98. [CrossRef] 15. Hellewell, J.; Abbott, S.; Gimma, A.; Bosse, N.I.; Jarvis, C.I.; Russell, T.W.; Munday, J.D.; Kucharski, A.J.; Edmunds, W.J.; Sun, F. Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts. Lancet Glob. Health 2020, 8, e488–e496. [CrossRef] 16. O’Neill, P.H.; Mosley, T.R.; Johnson, B. A ﬂood of coronavirus apps are tracking us. Now it’s time to keep track of them. MIT Technology Review, 7 May 2020. Available online: https://www.technologyreview.com/2020/05/07/1000961/launching-mittr- covid-tracing-tracker/ (accessed on 26 December 2020). 17. Thomas, R.; Michaleff, Z.; Greenwood, H.; Abukmail, E.; Glasziou, P. More than privacy: Australians’ concerns and mis- conceptions about the COVIDSafe App. medRxiv 2020.[CrossRef] 18. Dhar, T. Aarogya Setu- Carrying Your Privacy in Your Hands? SSRN Electron. J. 2020.[CrossRef] 19. Fulbright, N.R. Contact Tracing Apps: A New World for Data Privacy. Available online: https://www.nortonrosefulbright. com/en-fr/knowledge/publications; https://www.nortonrosefulbright.com/en-fr/knowledge/publications/d7a9a296/contact- tracing-apps-a-new-world-for-data-privacy (accessed on 26 December 2020). 20. Morley, G.; Grady, C.; McCarthy, J.; Ulrich, C.M. Covid-19: Ethical Challenges for Nurses. Hast. Cent. Rep. 2020, 50, 35–39. [CrossRef] Sustainability 2021, 13, 2912 18 of 19

21. Lomas, N. UK Reported to Be Ditching Coronavirus Contacts Tracing in Favor of ‘Risk Rating’ App|TechCrunch. 6 August 2020. Available online: https://techcrunch.com/2020/08/06/uk-reported-to-be-ditching-coronavirus-contacts-tracing-in-favor-of- risk-rating-app/ (accessed on 26 December 2020). 22. Pagliari, C. The ethics and value of contact tracing apps: International insights and implications for Scotland’s COVID-19 response. J. Glob. Health 2020, 10, 020103. [CrossRef][PubMed] 23. O’Callaghan, T.F.; O’Donovan, M.; Murphy, J.P.; Sugrue, K.; Tobin, J.T.; McNamara, A.E.; Yin, X.; Sundaramoorthy, G.; Brennan, L. The bovine colostrum and milk metabolome at the onset of lactation as determined by 1H-NMR. Int. Dairy J. 2021, 113, 104881. [CrossRef] 24. Wiertz, C.; Banerjee, A.; Acar, O.A.; Ghosh, A. Predicted Adoption Rates of Contact Tracing App Conﬁgurations—In-sights from a choice-based conjoint study with a representative sample of the UK population. April 2020. Available online: https: //ssrn.com/abstract=3589199 (accessed on 26 December 2020). 25. Rowe, F. Contact tracing apps and values dilemmas: A privacy paradox in a neo-liberal world. Int. J. Inf. Manag. 2020, 55, 102178. [CrossRef] 26. Kelion, L. Coronavirus: Moscow rolls out patient-tracking app. BBC News, 1 April 2020. Available online: https://www.bbc. com/news/technology-52121264 (accessed on 26 December 2020). 27. Viljoen, I.M.; Castelyn, C.D.V.; Pope, A.; Botes, M.; Pepper, M.S. Contact tracing during the COVID-19 pandemic: Pro-tection of personal information in South Africa. S. Afr. J. Bioeth. Law 2020, 13, 15–20. [CrossRef] 28. Gov.sg. PM Lee’s Address on the COVID-19 Situation in Singapore (21 April 2020). Available online: https://www.gov.sg/ article/pm-lees-address-on-the-covid-19-situation-in-singapore-21-april-2020 (accessed on 26 December 2020). 29. Utzerath, J.; Bird, R.; Cheng, G. Contact tracing apps in China, Hong Kong, Singapore and South Korea-Lexology. 24 April 2020. Available online: https://www.lexology.com/library/detail.aspx?g=99dca469-455d-4f7a-b025-00bf1d10ff6b (accessed on 26 December 2020). 30. Khurshudyan, I. Coronavirus Tests Limits of Russia’s Surveillance State. The Washington Post, 5 April 2020. Available online: https://www.washingtonpost.com/world/europe/coronavirus-russia-surveillance-tracking/2020/04/04/0798f4dc-75 19-11ea-ad9b-254ec99993bc_story.html (accessed on 26 December 2020). 31. Clarance, A. Aarogya Setu: Why India’s Covid-19 Contact Tracing App Is Controversial. BBC News, 4 May 2020. Available online: https://www.bbc.com/news/world-asia-india-52659520 (accessed on 26 December 2020). 32. D’Amore, R. Many Seniors Don’t Own a Smartphone. Experts Say It Might Be a ‘Gap’ in Tracking Apps. National|Global-News.ca, 9 July 2020. Available online: https://globalnews.ca/news/7152179/coronavirus-canada-contact-tracing-app-seniors/ (accessed on 26 December 2020). 33. Guinchard, A. Our digital footprint under Covid-19: Should we fear the UK digital contact tracing app? Int. Rev. Law Comput. Technol. 2021, 35, 84–97. [CrossRef] 34. Bengio, Y.; Ippolito, D.; Janda, R.; Jarvie, M.; Prud’Homme, B.; Rousseau, J.-F.; Sharma, A.; Yu, Y.W. Inherent privacy limitations of decentralized contact tracing apps. J. Am. Med. Inform. Assoc. 2021, 28, 193–195. [CrossRef][PubMed] 35. Jacob, S.; Lawarée, J. The adoption of contact tracing applications of COVID-19 by European governments. Policy Des. Pract. 2020, 1–15. [CrossRef] 36. Li, G.; Geng, E.; Ye, Z.; Xu, Y.; Lin, J.; Pang, Y. Indoor Positioning Algorithm Based on the Improved RSSI Distance Model. Sensors 2018, 18, 2820. [CrossRef] 37. Bay, J.; Kek, J.; Tan, A.; Hau, C.S.; Yongquan, L.; Tan, J.; Quy, T.A. BlueTrace: A Privacy-Preserving Protocol for Community-Driven Contact Tracing across Borders; Government Technology Agency: Singapore, 2020. 38. Rivest, S.; Forrest, J.R.K. Defence compounds in pollen: Why do they occur and how do they affect the ecology and evolution of bees? New Phytol. 2019, 225, 1053–1064. [CrossRef] 39. Ahmed, N.; Michelin, R.A.; Xue, W.; Ruj, S.; Malaney, R.; Kanhere, S.S.; Seneviratne, A.; Hu, W.; Janicke, H.; Jha, S.K. A Survey of COVID-19 Contact Tracing Apps. IEEE Access 2020, 8, 134577–134601. [CrossRef] 40. Castelluccia, C.; Bielova, N.; Boutet, A.; Cunche, M.; Lauradoux, C.; Métayer, D.L.; Roca, V. DESIRE: A Third Way for a European Exposure Notiﬁcation System Leveraging the best of centralized and decentralized systems. arXiv 2020, arXiv:2008.01621. 41. Vinuesa, R.; Theodorou, A.; Battaglini, M.; Dignum, V. A socio-technical framework for digital contact tracing. Results Eng. 2020, 8, 100163. [CrossRef] 42. Walrave, M.; Waeterloos, C.; Ponnet, K. Tracing the COVID-19 Virus: A Health Belief Model Approach to the Adoption of a Contact Tracing App. JMIR Public Health Surveill. 2020, 6, e20572. [CrossRef][PubMed] 43. Tovi, M.D.; Muthama, M.N. Addressing the Challenges of Data Protection in Developing Countries. 2013. Available online: www.ea-journals.org (accessed on 26 December 2020). 44. Haziq, S. UAE Takes Covid-19 Tests to New Heights with Millions in Record time. News|Khaleej Times, 11 July 2020. Avail- able online: https://www.khaleejtimes.com/coronavirus-pandemic/uae-takes-covid-19-tests-to-new-heights-with-millions- in-record-time (accessed on 26 December 2020). 45. Munzert, S.; Selb, P.; Gohdes, A.; Stoetzer, L.F.; Lowe, W. Tracking and promoting the usage of a COVID-19 contact tracing app. Nat. Hum. Behav. 2021, 5, 247–255. [CrossRef][PubMed] Sustainability 2021, 13, 2912 19 of 19

46. Vinuesa, R.; Azizpour, H.; Leite, I.; Balaam, M.; Dignum, V.; Domisch, S.; Felländer, A.; Langhans, S.D.; Tegmark, M.; Nerini, F.F. The role of artiﬁcial intelligence in achieving the Sustainable Development Goals. Nat. Commun. 2020, 11, 1–10. [CrossRef][PubMed] 47. Naudé, W.; Vinuesa, R. Data, Global Development, and COVID-19: Lessons and Consequences; WIDER Working Paper Series Wp-2020-109; World Institute for Development Economic Research (UNU-WIDER): Helsinki, Finland, 2020.