Feasibility of a web-based stress management intervention for university students in

Dilfa Juniar (  [email protected] ) Vrije Universiteit Amsterdam https://orcid.org/0000-0003-4977-7614 Wouter van Ballegooijen Vrije Universiteit Amsterdam Mieke H.J. Schulte Vrije Universiteit Amsterdam Anneke van Schaik Amsterdam UMC - Locatie AMC: Amsterdam UMC Locatie AMC Jan Passchier Vrije Universiteit Amsterdam Elena Heber Get.On Institute for Online Help Training Hamburg Dirk Lehr Leuphana Universität Lüneburg: Leuphana Universitat Luneburg Sawitri Supardi Sadarjoen Yarsi University: Universitas Yarsi Heleen Riper Vrije Universiteit Amsterdam

Research

Keywords: Indonesia, cultural adaptation, feasibility study, internet intervention, low and middle income countries (LMICs), stress management, university student

Posted Date: September 3rd, 2021

DOI: https://doi.org/10.21203/rs.3.rs-796995/v1

License:   This work is licensed under a Creative Commons Attribution 4.0 International License. Read Full License Feasibility of a web-based stress management intervention for university students 1 2 3 in Indonesia 4 5 6 7 Dilfa Juniar1,2, M.Psi; Wouter van Ballegooijen1,3,4, PhD; Mieke H.J. Schulte1, PhD; Anneke 8 9 3,4 1 5 6 10 van Schaik , MD PhD; Jan Passchier , PhD; Elena Heber , PhD; Dirk Lehr , PhD; Sawitri 11 12 Supardi Sadarjoen2, Dr; Heleen Riper1, PhD 13 14 15 16 17 1 Section Clinical Psychology, Vrije Universiteit Amsterdam, The Netherlands 18 19 2 20 Faculty of Psychology, YARSI University, , Indonesia 21 22 3 Department of Psychiatry, Amsterdam UMC, Amsterdam, The Netherlands 23 24 4 GGZ inGeest Specialised Mental Health Care, Amsterdam, The Netherlands 25 26 27 5 GET.ON Institute for Online Help Training, Hamburg, Germany 28 29 6 Leuphana University, Lüneburg, Germany 30 31 32 33 34 35 36 37 38 39 Abstract 40 41 42 43 44 Background 45 46 University students are susceptible to excessive stress. A web-based stress management 47 48 49 intervention holds promise to improve stress but is still at a novel stage in Indonesia. This 50 51 present paper reports the feasibility of the intervention we developed (‘Rileks’) among 52 53 54 university students in Indonesia in terms of acceptability and usability, and to propose 55 56 recommendations for future improvements. 57 58 59 60 61 62 63 64 1 65 Methods 1 2 3 A single-group pre-test and post-test design was used. Participants with DASS-42 Stress 4 5 scale score ≥ 15 were given access to the intervention (N= 68). The main outcome measures 6 7 were the Client Satisfaction Questionnaire-8 (CSQ-8), the System Usability Scale (SUS), and 8 9 10 intervention uptake. Participants’ experience in each session was evaluated using closed and 11 12 open-ended questions for future improvements. Descriptive statistics were used to examine 13 14 15 primary outcome and qualitative session evaluations. Participants’ responses to each topic of 16 17 the open questions were summarized. 18 19 20 21 22 Results 23 24 The intervention was evaluated as being satisfactory (CSQ-8 average score 21.89 (SD 8.72), 25 26 27 range 8-32). However, the intervention’s usability was still below expectation (SUS average 28 29 score 62.8 (SD 14.74), range 0-100). The core modules were completed by 10 participants 30 31 32 (14.7%) and the study drop out rate was 63.23% at post-assessment. The module content was 33 34 rated generally positively with some notes for improvement covering content and technical 35 36 37 aspects. 38 39 40 41 42 Conclusions 43 44 The study indicates that Rileks is potentially feasible for Indonesian university students. In 45 46 order to be optimally applied in such context and before scaling up web-based intervention in 47 48 49 general in Indonesia a further development and refinement are needed. 50 51 52 53 54 Keywords: Indonesia; cultural adaptation; feasibility study; internet intervention; low and 55 56 middle income countries (LMICs); stress management; university student. 57 58 59 60 61 62 63 64 2 65 Study protocol registration: International registered report identifier (IRRID) number 1 2 3 DERR1-10.2196/11493 4 5 6 7 8 Key messages regarding feasibility 9 10 11 1) What uncertainties existed regarding the feasibility?; 12 13 14 15 Web-based interventions are very novel in Indonesia. Rileks is the first web-based stress 16 17 management intervention adapted specificly for university students in Indonesia, but it is 18 19 20 unknown to what extent it is feasible to implement this intervention. . The present study 21 22 aimed to evaluate the feasibility of Rileks among university students in Indonesia in terms of 23 24 25 acceptability, usability and intervention uptake. 26 27 28 29 2) What are the key feasibility findings?; 30 31 32 33 The findings indicate adequate participant satisfaction and that Rileks’ usability and uptake 34 35 can be improved. 36 37 38 39 3) What are the implications of the feasibility findings for the design of the main study? 40 41 42 43 The present study provides directions for further development of Rileks, e.g. regarding 44 45 46 content and technical aspects, in order to optimize the experience for Indonesian university 47 48 students. 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 3 65

1 2 3 4 Background 5 6 Globally, an increasing number of university students experience stress [1–3]. To a certain 7 8 9 extent, stress can be advantageous in stimulating human thriving [4]. However, ongoing high 10 11 levels of stress may lead to negative outcomes, such as psychological distress, anxiety, 12 13 14 depression, physical illness, substance abuse, and impaired work-related and academic 15 16 performance [4–6]. The prevalence varies across studies and among countries, but overall, 17 18 19 studies suggest that 20 to 25 percent of university students around the globe suffer from 20 21 psychological distress [3] coming from various study fields [7]. However, most of them do 22 23 not receive support in reducing their high stress levels [5,8]. This is due to various reasons, 24 25 26 including fear of stigma for seeking help for mental health problems [5] and limited 27 28 availability of skilled mental health professionals within universities [5,9]. This has also been 29 30 31 reported by university students in Indonesia [10], despite considerable need for psychological 32 33 support e.g. two-thirds of Indonesian nursing and medical students experience moderate to 34 35 36 severe levels of stress) [11,12]. 37 38 39 40 Web-based interventions may overcome some of the issues related to this treatment gap. 41 42 43 They may provide an accessible and potentially less stigmatising alternative compared with 44 45 face-to-face treatment, because clients can use web-based interventions privately [13,14]. 46 47 48 Web-based interventions could be especially suitable for a university student population [15], 49 50 since younger, well-educated individuals already tend to seek information and help for 51 52 53 emotional and mental health problems through the internet [16–18]. A meta-analysis reported 54 55 that web-based and computer-delivered stress-management interventions can effectively 56 57 58 diminish university students’ stress with an effect size of 0.73 (95% CI – 1.27 to – 0.19, 59 60 P=0.008) [19]. A web-based stress-management intervention culturally adapted for 61 62 63 64 4 65 Indonesian university students might be feasible, because of the increasing availability of the 1 2 3 internet in Indonesia [20], and consequently easier internet access at their universities and to 4 5 a lesser extent, at home. 6 7 8 9 10 Several web-based stress-management interventions for university students have been 11 12 developed in high-income countries [19]. GET.ON Stress, has been investigated in several 13 14 15 RCTs [21–23] and is based on the Lazarus transactional model and consists of problem- 16 17 focused and emotion-focused coping strategies [24]. It was originally developed for German 18 19 20 employees and has been adapted for German speaking university students and re-named 21 22 StudiCare Stress [25]. As part of the current project, the GET.ON Stress intervention was 23 24 culturally adapted to the Indonesian context 25 , using Barrera’s integrative cultural adaptation 26 27 model as a guideline [26]. The first two steps of the adaptation process have led to the 28 29 ‘Rileks’ (which means relax; calm in ) intervention and have been 30 31 32 previously reported [27]. 33 34 35 36 37 Due to the novelty of this kind of intervention in Indonesia, instead of a pilot study, a 38 39 feasibility study was conducted as the third step with an emphasis on the process of 40 41 42 developing, implementing, and assessing preliminary responses of participants to the new 43 44 intervention [28,29]. Furthermore, a feasibility study helps to evaluate components 45 46 (participant recruitment, accuracy of the intervention protocol, and ability to execute the new 47 48 49 intervention) necessary for the next large-scale study [30,31]. The present paper reports on 50 51 the third step of Barrera’s model, which is the feasibility study of the preliminary version of 52 53 54 Rileks. 55 56 57 58 59 60 61 62 63 64 5 65 This study aimed to investigate the feasibility of Rileks among university students in 1 2 3 Indonesia in terms of acceptability, usability and intervention uptake. The secondary aim was 4 5 to investigate stress, anxiety, and depression reduction, improvement of quality of life, and 6 7 additionally to generate feedback for further refinement of Rileks. 8 9 10 11 12 13 14 15 16 Methods 17 18 19 Participants and sample size 20 21 The study targeted university students with a minimum age of 19 years in Indonesia. 22 23 24 Information about our study and our website was disseminated through social media 25 26 platforms such as Facebook, Instagram, and WhatsApp groups; and through presentations at 27 28 29 events at two universities in Indonesia (i.e. Universitas YARSI and STAN (Indonesian State 30 31 College of Accountancy)) by the principal investigator (DJ). Inclusion criteria were having a 32 33 34 score of 15 (low stress level) or higher on the Depression Anxiety Stress Scales-42 (DASS- 35 36 42; [32]), studying at a university in Indonesia, having access to the internet, and being able 37 38 39 to speak Bahasa Indonesia fluently. Participants were recruited between 10 and 19 October 40 41 2018. 42 43 44 45 46 A formal calculation of sample size used for effectiveness trials is not suitable for a 47 48 feasibility study [33]. In our study, we intended to include at least 50 participants with a 49 50 51 saturation of 75 participants to ensure sufficiently reliable estimates of our main study 52 53 parameters. We based this estimate on a systematic study analysing sample sizes in pilot and 54 55 56 feasibility studies in the United Kingdom reporting a median of 36 participants for a 57 58 feasibility study sample size [33]. 59 60 61 62 63 64 6 65 Study design and procedure 1 2 A single-group pre-test (t0) and post-test (t1) design was used. Interested university students 3 4 5 who signed up on our website subsequently received a link to the screening measurement 6 7 (DASS-42 Stress Scale). Eligible university students then had to submit their signed 8 9 10 electronic informed consent form before completing the baseline measurements (t0). All 11 12 included participants received login credentials to the intervention and only those who logged 13 14 15 in received post-treatment measurements (t1) ten weeks after t0. Ten weeks was chosen as 16 17 the post-measurement time point, as we considered this to be sufficient time for participants 18 19 to complete the intervention. All measures were self-reported and administered online. 20 21 22 23 24 25 Intervention 26 27 Rileks consists of six sessions and an optional booster session provided four weeks after 28 29 30 intervention completion. The first session comprises psycho-educational information about 31 32 stress, based on the Lazarus’ emotion-focused and problem-focused coping strategies [24]. 33 34 35 The second session comprises the six-step problem-solving method based on problem- 36 37 solving therapy [34,35]. In the second session, participants work on their problem-solving 38 39 40 skills by applying the method to their individual problem. In sessions three to five, 41 42 participants are introduced to emotional regulation techniques based on the Affect Regulation 43 44 45 Training (ART) [35,36]. The techniques include muscle and breathing relaxation, acceptance 46 47 and tolerance of emotions, and self-support in difficult situations. These techniques are 48 49 explained one-by-one in each session respectively. In the last session, participants are asked 50 51 52 to reassess their goals for the training and to identify their personal warning signs for stress. 53 54 Furthermore, participants are asked to write a letter to themselves about how they imagined 55 56 57 their life would be after applying the methods and techniques they had been taught. 58 59 Furthermore, a booster session may be given as an option to evaluate the letter they had 60 61 62 63 64 7 65 written to themselves in the last session, reassess their goals, and make plans to continue 1 2 3 applying what they had learnt from Rileks. 4 5 6 7 Each session contains general information, examples related to the exercises, exercises 8 9 10 guided by eCoaches, quizzes, slide shows encompassing explanations related to stress- 11 12 management methods and techniques, audio files, and downloadable material which were all 13 14 15 presented on a secure platform. Access to the platform was given to the participants based on 16 17 their email-addresses and self-designated passwords. Participants were advised to log in once 18 19 20 or twice per week. A reminder was sent to the participants if they did not log in within seven 21 22 days. Within 48 hours after completion of each session, four trained psychologists acting as 23 24 eCoaches gave personalized written feedback on the exercises. The eCoaches followed 25 26 27 guidelines about the feedback process that are defined according to the standardized manual 28 29 on feedback writing for the intervention [37]. 30 31 32 33 Primary outcome measures 34 35 Acceptability was measured by assessing client’s satisfaction with Rileks using the translated 36 37 version of the Client Satisfaction Questionnaire-8 (CSQ-8) [38–40]. The scale consists of 8 38 39 40 questions on four-point Likert scales (scored 1-4) with total score ranges from 8 (great 41 42 dissatisfaction) to 32 (great satisfaction). We set an average score of above 20 (20 is the 43 44 45 median total score) as criterion to acceptable satisfaction. The CSQ-8 has good reliability as 46 47 it has been reported to have a Cronbach’s coefficient of 0.92 [41]. 48 49 50 51 52 The Indonesian version of the System Usability Scale (SUS) [42] was used to assess the 53 54 usability of Rileks in terms of user-friendliness [43,44]. The scale comprises ten statements 55 56 57 scored on a 5-point Likert scale, ranging from 'strongly disagree' to 'strongly agree'. The total 58 59 scores were then transformed into a scale score ranging from 0 to 100, with higher scores 60 61 62 63 64 8 65 representing higher usability. A score of 70 or more was considered adequate as a feasibility 1 2 3 criterion [44]. Cronbach’s coefficient of the Indonesian version has been reported as 0.84 4 5 [42] which indicates good reliability. Both the CSQ-8 and the SUS were only administered at 6 7 post-intervention. 8 9 10 11 12 Intervention uptake was measured by assessing the number of participants who completed the 13 14 15 core online sessions (sessions 1 to 5) where participants learn the basic principles of problem- 16 17 solving and emotion regulation. The criterion for acceptable adherence was set at 60% or 18 19 20 more participants who completed the core sessions [27]. The 60% threshold was based on 21 22 previous meta-analyses on adherence to Internet-based Cognitive Behavioral Therapy (iCBT) 23 24 for Depression which found that 65.1% of participants completed the entire sessions of iCBT 25 26 27 [45]. 28 29 30 31 32 33 Secondary outcome measures 34 35 The severity of stress, anxiety, and depression was measured using the Indonesian version of 36 37 the Depression Anxiety Stress Scale-42 (DASS-42) [46]. The scale consists of 42 items, 38 39 40 divided into three subscales: depression, anxiety, and stress (each scale containing 14 items; 41 42 range 0-3 on each item) with a higher score indicating a greater degree of severity [32]. The 43 44 45 Indonesian version of DASS-42 shows excellent overall reliability with a Cronbach's 46 47 coefficient (α) value of 0.95, and high internal consistency in the separate depression, 48 49 50 anxiety, and stress subscales (α being 0.91, 0.85, and 0.88) [46]. 51 52 53 54 Quality of life was measured by the Indonesian version of the World Health Organization- 55 56 57 Quality Of Life brief version (WHOQOL-BREF) [47]. It consists of two items which 58 59 measure overall quality of life and general health and 24 items which measure how the 60 61 62 63 64 9 65 respondent felt in the last 2 weeks, across 4 domains; physical health, psychological health, 1 2 3 social relationships, and environmental health [47,48]. Among the Indonesian population, the 4 5 WHOQOL-BREF has been reported as having internal consistency of 0.41-0.77 in the four 6 7 domains [49], and reliability with ICCs 0.70-0.79 in the four domains [47]. The DASS-42 8 9 10 and the WHOQOL-BREF were assessed both at t0 and t1. 11 12 13 14 15 16 Other measurements 17 18 Demographic variables were collected to describe characteristics of the study population (see 19 20 table 1). Cultural related aspects evaluation of Rileks modules were measured within the 21 22 23 aspects of language, case examples and visual presentation. With regard to language use, we 24 25 asked whether the participants could understand the words in each session. To evaluate case 26 27 28 examples and visual presentation, the participants were asked whether the case examples and 29 30 pictures used in the sessions represented Indonesian university students’ context. 31 32 33 Participants’ experience with eCoaches was evaluated using six questions concerning 34 35 participants’ satisfaction with the contact with their eCoach. Some of the questions were 36 37 adapted from another study which assessed participants’ satisfaction with the online 38 39 40 therapeutic contact [50]. A selection of responses was provided for each question. An open 41 42 question asking participants’ recommendations for future improvement of the eCoaches was 43 44 45 also asked. 46 47 48 49 50 Furthermore, by the end of each session on the intervention platform, participants gave an 51 52 evaluation about their experience in each session. The participants assessed general aspects of 53 54 each session (e.g. usefulness, easiness, time needed to complete the module) using a Likert 55 56 57 scale from 1 (e.g. very useful, very easy) to 5 (e.g. not useful at all, very difficult) and 58 59 specific aspects of the sessions (e.g. the structure of the module) using a Likert scale from 1 60 61 62 63 64 10 65 (positive judgement) to 7 (negative judgement) and some open questions e.g. “What did you 1 2 3 like and not like about this session; how might you have benefitted from it”. 4 5 6 7 8 Statistical analyses 9 10 11 Descriptive statistics were used to examine primary outcomes, demographic data and 12 13 sessions evaluation. Participants' satisfaction (CSQ-8) and system usability (SUS) as 14 15 feasibility parameters were summarized using means and standard deviations. Intervention 16 17 18 uptake was summarized with frequency of participants who logged in or completed each 19 20 session. Differences in demographic characteristics between participants who logged in and 21 22 23 those who did not log in were tested using a chi-square test in terms of sex, level of 24 25 education, field of study, and university location. Independent sample t-tests were used to 26 27 28 assess differences in the mean of age, the DASS-42 and WHOQOL-BREF scores. 29 30 31 32 33 Secondary outcomes were analysed using two-tailed paired-samples t-tests with a level of 34 35 significance of α = 0.05. A normality test at each time point was conducted beforehand, using 36 37 the Shapiro–Wilk tests. As the distributions significantly differed from normal (p values < 38 39 40 0.05), a sensitivity analysis using non-parametric tests e.g. Wilcoxon was conducted. 41 42 Furthermore, within-group effect sizes (Cohen’s d) were calculated. The Statistical Package 43 44 45 for the Social Sciences (SPSS) version 26 was used. 46 47 48 49 50 In order to understand the participants’ experience and provide recommendations relevant to 51 52 future refinement, participants’ responses to each topic of the open questions were 53 54 summarized by categorizing responses with similar themes. Categorization was done by the 55 56 57 principal investigator (DJ) and one of the Indonesian team members as a second rater. The 58 59 summary was finalized by consensus between the two. 60 61 62 63 64 11 65 Results 1 2 3 4 Enrollment and participant characteristics 5 6 A total of 191 university students registered and completed the screening on the study 7 8 website. Most of the participants (n=169; 88.5%) learned about the study through social 9 10 11 media (e.g. Facebook, Instagram, WhatsApp groups). A small proportion (n=22; 11.5%) were 12 13 informed through presentations by the principal investigator (DJ) in two universities. Of the 14 15 16 191 registered university students, 40 (21%) scored lower than 15 on the DASS-42 stress 17 18 scale, which can be considered as normal stress, and 151 (79%) were eligible for inclusion. 19 20 21 See Figure 1 for study flow. 22 23 24 25 Baseline questionnaires were completed by 121 participants. The majority were female 26 27 28 (n=103; 85.1%), had a bachelor’s degree (n= 94; 77.7%) in social and behavioral sciences 29 30 (n= 83; 68,6%), and studied on the island of (n=79; 65.3%). The average DASS-42 31 32 33 stress score at baseline was 23.11 (SD=5.8), indicating moderate stress. A third of the 34 35 participants (n=41; 33.9%) fell into the severe stress category (i.e. ≥ 26 -33). Those who 36 37 38 logged in mostly study in Java and scored lower on psychological quality of life compared to 39 40 those who did not logged in. See Table 1 for all participants’ characteristics at baseline. 41 42 43 44 45 46 Primary outcomes 47 48 Ten weeks after t0, post-treatment questionnaires were sent to the 68 participants who had 49 50 51 logged in, which were completed by 25 participants thus the study drop out rate was 63.23%. 52 53 The CSQ-8 mean was 21.89 (SD 8.72) which meets our acceptable satisfaction criterion of a 54 55 mean score of 20. Table 2 provide information about mean item scores on the Client 56 57 58 Satisfaction Questionnaire-8. However, the SUS mean was 62.80 (SD 14.74), with lowest 59 60 score in the learnability item (See table 3). With regard to intervention uptake, all 121 61 62 63 64 12 65 enrolled participants received login credentials for Rileks by email. Of those who enrolled, 68 1 2 3 participants (56.2%) logged in. These 68 did not differ from those who never logged in 4 5 (n=53; 43.8%) on any baseline characteristics with the exception of university location (2 6 7 8 21.17; p<.05) and psychological quality of life (t(113)=2.14; p<.05) (see Table 1). The core 9 10 sessions were completed by 10 participants (14.71%). This number is below our acceptable 11 12 uptake criterion of 60%. Reasons for non-adherence were mostly unknown because those 13 14 15 participants could not be reached. Several known reasons for non-adherence or withdrawal 16 17 included time management problems, rare use of email so they missed notifications, 18 19 20 unexpected events, (e.g. internship to a remote village with limited internet coverage), and 21 22 technical problems such as unfamiliarity with the login system. Table 4 outlines the number 23 24 25 of completed sessions in Rileks. 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 Table 1. Characteristics of participants at baseline and p-value of the difference between 48 49 those who did not log in and those who did 50 51 Baseline Did not log in Logged in 52 53 (n = 121) (n = 53) (n = 68) 54 Age range 19 - 42 19 - 39 19 - 42 55 56 Mean age (years, SD) 24.03 (4.61) 24.7 (4.41) 23.56 (3.55) 57 Sex (n, %) 58 Female 103 (85.1) 44 (83.0) 59 (86.76) 59 60 Male 17 (14.0) 9 (17.0) 8 (11.76) 61 62 63 64 13 65 Did not fill out 1 (0.8) - 1 (1.47) 1 2 Level of education (n, %) 3 Bachelor and equivalent 94 (77.7) 46 (86.8) 48 (70.59) 4 Master 18 (14.9) 5 (9.4) 13 (19.12) 5 1 (1.9) 6 Doctoral 3 (2.5) 2 (2.94) 7 Did not fill out 6 (5.0) 1 (1.9) 5 (7.35) 8 Field of study (n, %) 9 10 83 (68.6) 30 (56.6) 53 (77.94) 11 Social and behavioral science 12 Business and administration 12 (9.9) 9 (17.0) 2 (2.94) 13 Languages 6 (5.0) 4 (7.5) - 14 - 3 (5.7) 15 Humanities - 16 Others 20 (16.5) 7(13.2) 13 (19.12) 17 University location (n, %) 18 79 (65.3) 30 (56.6)* 49 (72.06)* 19 Java* 20 20 (16.5) 10 (18.9) 10 (14.70) 21 /Borneo 8 (6.6) 2 (3.7) 6 (8.82) 22 10 (8.3) 8 (15.1) 2 (2.94) 23 24 East Nusa Tenggara 2 (1.7) 2 (3.8) - 25 Others 2 (1.7) 1 (1.9) 1 (1.47) 26 27 DASS-42 score (M, SD) 28 Depression 14.88 (8.70) 13.37 (8.21) 16.16 (8.90) 29 Anxiety 15.54 (6.72) 15.54 (6.92) 15.58 (6.24) 30 31 Stress 23.11 (5.80) 22.89 (5.91) 23.28 (5.53) 32 Stress level (n, %) 33 Mild 36 (29.8) 16 (30.2) 20 (29.41) 34 40 (33.1) 16 (30.2) 24 (35.29) 35 Moderate 36 Severe 41 (33.9) 20 (37.7) 21 (30.88) 37 Extremely severe 4 (3.3) 1 (1.9) 3 (4.41) 38 39 WHO-QOL-Bref score (M, SD) 40 41 Physical health 41.83 (7.49) 42.15 (7.23) 41.56 (7.82) 42 Psychological health* 47.90 (14.29) 51 (14.07)* 45.34 (13.98)* 43 Social relationship 49.04 (19.29) 48.69 (20.65) 49.33 (18.39) 44 45 Environmental health 53.91 (11.67) 53.71 (11.97) 54.07 (11.61) 46 Overall QOL 2.98 (0.87) 3.08 (0.88) 2.91 (0.86) 47 Overall health 2.90 (0.89) 3.04 (0.79) 2.79 (0.97) 48 49 * p < .05; ** p < .01 50 51 52 Table 2. Mean item scores on the Client Satisfaction Questionnaire-8 for Rileks (4 point 53 54 scale) 55 Item Mean 56 57 1. How would you rate the quality of service you have received from Rileks? 3.20 58 2. Did you get the kind of service you wanted? 2.92 59 3. To what extent has Rileks met your needs? 2.72 60 61 62 63 64 14 65 4. If a friend were in need of similar help, would you recommend Rileks to him or 3.24 1 her? 2 5. How satisfied are you with the amount of messages you have received from 3.16 3 Rileks? 4 5 6. Have Rileks helped you to deal more effectively with your problems? 3.28 6 7. In an overall, general sense, how satisfied are you with the help you have received 2.96 7 from Rileks 8 9 8. If you were to seek help again, would you use Rileks again? 3.04 10 11 12 13 14 15 Table 3. Mean item scores on the System Usability Scale for Rileks (5 point scale) 16 Item Mean 17 18 19 1. I think that I would like to use Rileks frequently. 3.48 20 21 2. I found Rileks unnecessarily complex.1 3.40 22 3. I thought Rileks was easy to use. 3.56 23 24 4. I think that I would need the support of a technical person to be able to use Rileks.1 3.88 25 26 5. I found the various functions in Rileks were well integrated. 3.72 27 6. I thought there was too much inconsistency in Rileks.1 3.56 28 29 7. I would imagine that most people would learn to use Rileks very quickly. 3.88 30 31 8. I found Rilkes very cumbersome to use.1 3.44 32 33 9. I felt very confident using Rileks 3.32 34 10. I needed to learn a lot of things before I could get going with Rileks.1 2.88 35 36 1 Reversed item 37 38 39 40 Table 4. Rileks session completion 41 42 n (%) 43 44 Logged in 68 (100) 45 Completed module 1 40 (58.82) 46 Completed module 2 26 (38.23) 47 Completed module 3 16 (23.53) 48 Completed module 4 12 (17.65) 49 Completed module 5 10 (14.71) 50 Completed all 6 modules 9 (13.23) 51 52 53 54 55 Secondary outcomes 56 57 At t1, the DASS-42 stress scale was completed by 23 participants consist of the core sessions 58 59 60 completers and non-completers. The DASS-42 depression and anxiety scales, as well as the 61 62 63 64 15 65 WHOQOL-BREF, were completed by 20 participants. At t1, participants reported 1 2 3 significantly lower levels of stress (M= -10.04, 95% CI 5.36 to 14.72), depression (M= – 4 5 6.85, 95% CI 1.37 to 12.32), and anxiety (M= -6.45, 95% CI 1.56 to 11.34), with high effect 6 7 size on stress and moderate effect size on anxiety and depression. Participants also reported 8 9 10 significantly improved quality of life in terms of physical health (M= 21.25, 95% CI -29.05 11 12 to -13.44), psychological health (M= 12.50, 95% CI -20.25 to -4.75), overall quality of life 13 14 15 (M= 0.55, 95% CI -0.96 to -0.13), and overall health (M= 0.70, 95% CI -1.31 to -0.09). 16 17 However, we did not find significant differences in the social relationship and environmental 18 19 20 aspect of the quality of life (see Table 5). Wilcoxon signed-rank tests for non-parametric 21 22 distributions yielded the same results. 23 24 Table 5. DASS-42 and WHOQOL-BREF scores at baseline and post-treatment. 25 26 27 Measure Baseline Post assessment Cohen’s d 28 M (SD) M (SD) DASS-42 29 1 30 Stress 24.74 (6.33) 14.70 (11.41)** 0.93 Anxiety2 17.95 (8.35) 11.50 (8.34)* 0.62 31 2 32 Depression 17.10 (10.74) 10.25 (11.39)* 0.58 33 WHO QOL-Bref 34 Physical health2 42.30 (8.28) 63.55 (13.95)** 35 Psychological heatlh2 43.85 (17.74) 56.35 (20.14)* 36 Social relationship2 49.40 (17.99) 54.35 (19.64) 37 Environmental health2 54.55 (13.32) 50.60 (15.14) 38 Overall QOL2 2.70 (1.08) 3.25 (1.07)* 39 Overall health2 2.70 (1.03) 3.40 (0.94)* 40 1 n = 23; 2 n = 20; * p < .05; **p < .001 41 42 43 44 45 46 Feedback for future refinement 47 48 49 Module/session evaluation 50 51 Participants rated the individual modules generally positively (see Table 6). The open 52 53 54 questions response summary (Additional file 1) indicated that in general, participants were in 55 56 favour of the clear examples given in the modules and case examples which they felt they 57 58 59 could relate to. Furthermore, participants suggested that the module content had provided 60 61 62 63 64 16 65 them with new and comprehensive information as well as exercises which can help to 1 2 3 manage their stress in terms of recognition, being more reflective and accepting of oneself, 4 5 learning how to examine problems, and thinking of positive activities which can help to 6 7 manage their stress. Moreover, Rileks was considered to be a medium where participants 8 9 10 were able to express their problems openly without feeling ashamed. 11 12 13 14 15 Participants also mentioned things they did not like mostly technical problems such as 16 17 unfamiliarity with the system, poor audio quality, some files being too large to download, 18 19 20 some repeated questions and confusion about how to do the exercises despite the given 21 22 examples. 23 24 Table 6. Module evaluation 25 26 27 Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 28 n=40 n=26 n=16 n=12 n=10 n=9 29 30 General usefulness Useful Useful Useful Very useful Useful Very useful 31 Easy to complete Easy Easy Easy Easy Easy Very easy 32 33 How long did it take 30 - 60 30 – 60 30 – 60 30 – 60 60 – 90 30 – 60 34 minutes minutes minutes minutes minutes 35 minutes 36 Clarity Undecided Clear Clear Clear Clear Very clear 37 38 Sub-content usefulness Undecided Useful Useful Useful Useful Very useful 39 40 41 Pleasantness Undecided Pleasant Undecided Pleasant Pleasant Very pleasant 42 Comprehensibility Undecided Compre- Compre- Compre- Compre- Very compre- 43 hensible hensible hensible hensible hensible 44 45 46 47 48 Suggestions for improvements were: adding interactive functions for communication with the 49 50 eCoach or other professionals to get direct help such as live chat, especially in parts where 51 52 53 participants need to remember and deal with negative events they had; making the 54 55 intervention simpler and shorter; adding links to music and video guidance, especially for 56 57 relaxation; provide downloadable materials; and a better display for those accessing Rileks 58 59 60 from a mobile phone. 61 62 63 64 17 65 1 2 3 Cultural and technical aspects 4 5 6 All participants (n=23) could understand the language used in the Rileks modules, including 7 8 idioms and metaphors. Most participants (n=20) thought that Rileks used suitable media, such 9 10 11 as audio guidance for relaxation or a slide show to explain theory. Case examples in Rileks 12 13 were considered to represent Indonesian university students’ context (n=22). Furthermore, 14 15 one participant suggested adding a case example that speaks for university students who 16 17 18 come from a family with a low economic background. 19 20 21 22 23 24 eCoach evaluation 25 26 Of the 23 participants, 13 said they missed face-to-face communication and 7 had hoped for 27 28 more support. The quality of support was generally considered positive. The eCoach support 29 30 31 helped participants gain insight in managing their problems and go through the modules with 32 33 confidence and motivation. Moreover, the presence of eCoaches made them feel understood 34 35 36 and less lonely. 37 38 39 40 41 Open questions about suggestions for future refinement were responded to by 22 participants. 42 43 Suggestions for refinement from the open questions can be summarized as follows. Firstly, 44 45 the feedback process could be more interactive and personal (n=3). Participants hoped that 46 47 48 they would be able to ask further questions by replying to the feedback they had received. 49 50 Secondly, some participants suggested the use of a medium other than email to deliver 51 52 53 feedback, e.g. using chat (n=2). Thirdly, they would like the possibility of maintaining a 54 55 future connection with the coaches for further counselling after the intervention had ended 56 57 58 (n=2). Lastly, there was a suggestion to combine offline and online treatment to meet the 59 60 61 62 63 64 18 65 needs of a participant who felt that they could not fully express themselves to the eCoach 1 2 3 through written text (n=1). 4 5 6 7 8 Discussion 9 10 11 The main aim of this study was to evaluate the feasibility of Rileks as part of adaptation 12 13 process of a web-based stress management intervention among university students in 14 15 16 Indonesia. Rileks was reported as being acceptable even though it’s usability and intervention 17 18 uptake were still below our expected criteria. Study findings showed that participants’ stress 19 20 21 level and quality of life improved at t1. The intervention was appreciated by participants in 22 23 terms of content usefulness, easiness to complete, comprehensibility, suitability for 24 25 26 Indonesian university students’ context, and the quality of eCoach support with more eCoach 27 28 interaction was desired. 29 30 31 Primary outcome 32 33 34 Rileks was rated as generally satisfactory which indicates that the intervention was 35 36 acceptable. The most satisfying aspect of Rileks as perceived by the participants was that the 37 38 39 intervention helps them deal with their problem effectively and that they would recommend it 40 41 to their friends. With regard to usability, Rileks received a lower than expected threshold 42 43 score with the learnability aspect as a main challenge. As reported, participants needed to 44 45 46 learn and become familiar with a number of new technical aspects related to the system 47 48 before they could engage with the intervention. We considered this to be due to the fact that a 49 50 51 web-based intervention was relatively new in Indonesia, and hence participants did not have 52 53 much prior experience of using such an intervention, which may have affected their 54 55 56 perception of its usability [51]. Another possible explanation is that the user interface of the 57 58 system is not user friendly enough. Even though learnability was a challenge, participants felt 59 60 positive about being able to learn it in a relatively short period of time. 61 62 63 64 19 65 1 2 3 The study findings revealed that the uptake of the intervention’s core modules was below our 4 5 criteria. However, the number of participants who completed the core modules in our study 6 7 still falls within the range of the reported number of module completers in a systematic 8 9 10 review on computerized cognitive behavioral therapy (cCBT) (12%-100%) [52]. Moreover, a 11 12 systematic review reported that only 30% of patients adhered to treatment until the third 13 14 15 session of face-to-face interventions [53] making a 23.53% module 3 uptake sufficient, 16 17 considering that a cCBT is more likely to have lower adherence rate [52]. Thus we consider 18 19 20 that Rileks still has potential even though it was not met the uptake criteria yet. 21 22 23 24 A few reported reasons for non-adherence include time management issues, e.g. being too 25 26 27 busy, having competing activities; and technical problems such as unfamiliarity with the 28 29 login system. This is supported by a systematic review which argues that unfamiliarity with 30 31 32 computer or internet, and feeling too busy to complete treatment may contribute to drop-out 33 34 from internet-based treatment [54,55]. Other reasons for intervention drop-out in our study 35 36 37 were that participants missed notifications because they seldom used or checked their email. 38 39 This implies the need for future studies to utilize means other than email to send reminders to 40 41 42 participants, in order to boost intervention uptake such as personal message via preferred 43 44 platform e.g. instant chat message. Furthermore, problems with internet connections in rural 45 46 area were also experienced which indicate that other form of channel such as a mobile phone 47 48 49 application may also be explored as a potential option as the use of mobile application does 50 51 not always require a constant internet connectivity. 52 53 54 55 56 Interestingly it was found that stress severity level was associated with adherence as those 57 58 59 who logged in and engaged tended to have higher stress levels and significantly poorer 60 61 62 63 64 20 65 psychological quality of life compared to those who did not log in. This is supported by other 1 2 3 studies that confirmed that participants with less severe problems and difficulties may be less 4 5 motivated and are subsequently more likely to drop out of internet-based treatment [54,56]. 6 7 8 9 10 11 Secondary outcome 12 13 Participants’ stress levels at the post-treatment assessment were significantly lower than at 14 15 the pre-treatment assessment. In addition, the level of anxiety and depression was also 16 17 18 significantly lower among participants at post-intervention. This finding is in line with a 19 20 previous meta-analysis which reported that face-to face and internet interventions do reduce 21 22 23 stress levels as well as symptoms of anxiety and depression among university students 24 25 [19,57,58]. Furthermore, participants’ overall quality of life as well as health and 26 27 28 psychological quality of life were better at post-treatment compared with pre-treatment. Our 29 30 study findings may give initial implications for the clinical impact of Rileks, but should be 31 32 33 interpreted with caution due to our study design and small sample size. 34 35 36 37 38 Future web-base intervention studies and Rileks refinements 39 40 41 Our findings demonstrate that web-based intervention studies targeting university students in 42 43 Indonesia may be feasible. We reached 191 potential participants from several islands in 44 45 Indonesia in ten days, with 80.13% invited participants giving their consent and filling in 46 47 48 baseline questionnaires. This indicates that university students in Indonesia were interested in 49 50 our study. The use of internet, social media, and particularly WhatsApp groups play an 51 52 53 important role in the process of reaching targeted potential participants. These strategies 54 55 allowed us to reach potential participants from many different cities, islands, and fields of 56 57 58 study with relatively little effort. Furthermore, taking part in activities involving the target 59 60 group was also useful in disseminating information on the study. Such activities allowed 61 62 63 64 21 65 face-to-face interaction between principal investigator (PI) and the target group, where PI 1 2 3 could give information on stress and our study thus increasing university students’ awareness 4 5 of stress and the credibility of our study. However, our study has a relatively high drop-out 6 7 rate (63.23%) which is commonly found in other internet interventions [52]. It is unfortunate 8 9 10 that we do not have sufficient data to explain the reasons for drop-out due to the 11 12 unresponsiveness of our participants. This may be because we approached the dropped-out 13 14 15 participants via email which was not the most convenient medium for them. Thus we 16 17 recommend for future studies to use other ways to approach their participants e.g. sending a 18 19 20 short survey about reasons for drop-out through any of their preferred devices or platforms. 21 22 23 24 Rileks related to both content, e.g. scope 25 Participants’ feedback to improve future versions of 26 27 of case examples and wording, and technical aspects e.g. suggesting a medium other than 28 29 email to send notifications and better display on the mobile phone version. Involvement of 30 31 32 relevant stakeholders in the process of refining the content and technical aspects would be 33 34 valuable for the purpose of achieving an optimal form of Rileks. One highlight of our 35 36 37 findings was the request for the availability of interactive communication with the eCoach. 38 39 This suggested that even though university students in Indonesia are open to the use of 40 41 42 internet-based interventions, face-to-face or interactive communication is still preferred. This 43 44 outcome is in line with other research which suggests that face-to-face interaction is still 45 46 considered an essential and significant element of mental health services [56,59,60]. 47 48 49 According to the latter study, Indonesian people in general would still prefer face-to-face 50 51 contact especially when they can access it nearby. This condition may make the use of a web- 52 53 54 based intervention to overcome mental health gap in Indonesia seem challenging. One 55 56 alternative is to use lay counsellors e.g trained-up psychology students as eCoaches. 57 58 59 Furthermore, reaching out and involving local community health centers as support may be a 60 61 62 63 64 22 65 possible means for future dissemination and implementation of web-based interventions in 1 2 3 Indonesia [61]. Another alternative is using blended strategy by combining module based and 4 5 instant chat or video conference for more interactive communication between client and the 6 7 eCoach. 8 9 10 11 Strengths and limitations 12 13 To the best of our knowledge, Rileks is the first web-based intervention to provide stress 14 15 management in the Indonesian university student context. Furthermore, it is among the first 16 17 18 web-based psychological intervention being culturally adapted from western culture to Asian 19 20 culture. Furthermore, other than quantitative assessment our study also considers qualitative 21 22 23 data from open questions to give more insight into participants’ experience in working with 24 25 the Rileks modules and system, including eCoaches. However, our study has also a number of 26 27 28 limitations. The dropout rates from enrollment stage to post-assessment was relatively high. 29 30 The data obtained in the post-assessment only came from 38% of the participants who were 31 32 33 logged in to the intervention thus did not equally represent all participants. 34 35 36 37 38 Conclusion 39 40 41 Rileks shows potential feasibility for Indonesian university students. However, our findings 42 43 also underscore the need for further development of this kind of intervention in order to 44 45 optimize the experience for Indonesian university students. Further refinements are needed 46 47 48 regarding content and technical aspects. Despite the potential of web-based interventions and 49 50 eMental Health in general to minimize the mental health gap among the Indonesian 51 52 53 population, our study implies that more work needs to be done before we can scale up this 54 55 kind of intervention in Indonesia. In sum, the intervention has potential, but it needs 56 57 58 refinement before it can be optimally applied in this setting. 59 60 61 62 63 64 23 65 Declarations 1 2 Ethics approval 3 The study has been reviewed and approved by the Indonesian ethics committee at YARSI 4 5 6 University (project number: 193/KEP-UY/BIA/VIII/2017). 7 8 9 Consent for Publication 10 Not applicable 11 12 13 14 Data availabitity 15 The study datasets are available on request to the corresponding author. 16 17 18 Competing interests 19 20 None declared 21 22 23 Funding 24 The project was supported by LPDP (The Indonesian Endowment Fund for Education) in 25 26 27 2015, funded as part of principal investigator’s PhD trajectory. 28 29 30 31 32 Author contributions 33 Dilfa Juniar, M.Psi were the principal investigator who takes responsibility for data 34 35 36 collecting, data processing, and drafting the manuscript. Wouter van Ballegooijen, PhD and 37 38 Heleen Riper, PhD supervised the study process and data interpretation, also gave critical 39 40 41 revision to the manuscript. Mieke H.J. Schulte, PhD; Anneke van Schaik, MD PhD; Jan 42 43 Passchier, PhD; Elena Heber, PhD; Dirk Lehr, PhD; Sawitri Supardi Sadarjoen, Dr critically 44 45 revised the manuscript. 46 47 48 49 50 Acknowledgements 51 We thank Desy Mulyanti Subagia, S.Psi for her role as a second rater in summarizing study 52 53 54 participants’ responses to open questions. 55 56 57 58 59 60 61 62 63 64 24 65 1 2 References 3 1. Robotham D. Stress among higher education students: Towards a research agenda. 4 5 High Educ 2008;56(6):735–746. [doi: 10.1007/s10734-008-9137-1] 6 2. Stallman HM. Psychological distress in university students: A comparison with 7 general population data. Aust Psychol 2010;45(4):249–257. [doi: 8 10.1080/00050067.2010.482109] 9 10 3. Kumaraswamy N. Academic stress, anxiety and depression among college students-A 11 brief review. Int Rev Soc Sci Humanit 2013;5(1):135–143. 12 4. Alzahem AM, Van Der Molen HT, Alaujan AH, Schmidt HG, Zamakhshary MH. 13 Stress amongst dental students: A systematic review. Eur J Dent Educ 2011;15(1):8– 14 18. PMID:21226800 15 16 5. Storrie K, Ahern K, Tuckett A. A systematic review: Students with mental health 17 problems-A growing problem. Int J Nurs Pract 2010;16(1):1–6. PMID:20158541 18 6. Beiter R, Nash R, McCrady M, Rhoades D, Linscomb M, Clarahan M, Sammut S. The 19 prevalence and correlates of depression, anxiety, and stress in a sample of college 20 21 students. J Affect Disord [Internet] Elsevier; 2015 Mar 1 [cited 2018 Nov 6];173:90– 22 96. [doi: 10.1016/J.JAD.2014.10.054] 23 7. Hilger-Kolb J, Diehl K, Herr R, Loerbroks A. Effort-reward imbalance among students 24 at German universities: associations with self-rated health and mental health. Int Arch 25 Occup Environ Health [Internet] Springer Berlin Heidelberg; 2018;91(8):1011–1020. 26 27 PMID:30022255 28 8. Hunt J, Eisenberg D. Mental Health Problems and Help-Seeking Behavior Among 29 College Students. J Adolesc Heal [Internet] Elsevier; 2010 Jan 1 [cited 2018 May 30 28];46(1):3–10. [doi: 10.1016/J.JADOHEALTH.2009.08.008] 31 9. Drum DJ, Denmark AB. Campus suicide prevention: Bridging paradigms and forging 32 33 partnerships. Harv Rev Psychiatry 2012;20(4):209–221. PMID:22894730 34 10. Setiawan JL. Willingness to seek counselling, and factors that facilitate and inhibit the 35 seeking of counselling in Indonesian undergraduate students. Br J Guid Couns 36 2006;34(3):403–419. [doi: 10.1080/03069880600769654] 37 38 11. Purwati S. Tingkat Stres Akademik pada Mahasiswa Reguler Angkatan 2010 Fakultas 39 Ilmu Keperawatan Universitas Indonesia. Universitas Indonesia; 2012. 40 12. Augesti G, Lisiswanti R, Saputra O, Nisa K. Differences in Stress Level Between First 41 Year and Last Year Medical Students in Medical Faculty of Lampung. J Major 42 2015;4:50 56. [doi: 10.1007/s00234-017-1955-3] 43 – 44 13. Hedman E, Ljótsson B, Lindefors N. Cognitive behavior therapy via the Internet: A 45 systematic review of applications, clinical efficacy and cost-effectiveness. Expert Rev 46 Pharmacoeconomics Outcomes Res 2012;12(6):745–764. PMID:23252357 47 14. Sobowale K, Nguyen M, Weiss B, Van TTH, Trung LT. Acceptability of internet 48 49 interventions for youth mental health in Vietnam. Glob Ment Heal 2016;3. [doi: 50 10.1017/gmh.2016.18] 51 15. Chan JK, Farrer LM, Gulliver A, Bennett K, Griffiths KM. University Students’ Views 52 on the Perceived Benefits and Drawbacks of Seeking Help for Mental Health Problems 53 on the Internet: A Qualitative Study. JMIR Hum Factors JMIR Publications Inc.; 2016 54 55 Jan 19;3(1):e3. [doi: 10.2196/humanfactors.4765] 56 16. Escoffery C, Miner KR, Adame DD, Butler S, McCormick L, Mendell E. Internet Use 57 for Health Information Among College Students. J Am Coll Heal [Internet] 2005 58 [cited 2018 May 28];53(4):183–188. [doi: 10.3200/JACH.53.4.183-188] 59 17. Burns K. At issue: Community college student success variables: a review of the 60 61 62 63 64 25 65 literature. 16(2). The Community College Enterprise; 2010 [cited 2018 May 28]. p. 33. 1 18. Batterham PJ, Calear AL. Preferences for Internet-Based Mental Health Interventions 2 3 in an Adult Online Sample: Findings From an Online Community Survey. JMIR Ment 4 Heal [Internet] JMIR Publications Inc.; 2017 Jun 30 [cited 2018 Jun 11];4(2):e26. 5 PMID:28666976 6 19. Davies EB, Morriss R, Glazebrook C. Computer-delivered and web-based 7 interventions to improve depression, anxiety, and psychological well-being of 8 9 university students: A systematic review and meta-analysis. J Med Internet Res 10 [Internet] JMIR Publications Inc.; 2014 May 16 [cited 2018 May 22];16(5):e130. 11 PMID:24836465 12 20. APJII. Penetrasi & Profil Perilaku Pengguna Internet Indonesia. Apjii [Internet] 13 14 2018;51. Available from: www.apjii.or.id 15 21. Ebert DD, Lehr D, Heber E, Riper H, Cuijpers P, Berking M. Internet- and mobile- 16 based stress management for employees with adherence-focused guidance: efficacy 17 and mechanism of change. Scand J Work Environ Health [Internet] 2016 18 Sep;42(5):382–394. [doi: 10.5271/sjweh.3573] 19 20 22. Heber E, Lehr D, Ebert DD, Berking M, Riper H. Web-Based and Mobile Stress 21 Management Intervention for Employees: A Randomized Controlled Trial. J Med 22 Internet Res [Internet] JMIR Publications Inc.; 2016 Jan 27 [cited 2018 May 23 28];18(1):e21. PMID:26818683 24 23. Ebert DD, Heber E, Berking M, Riper H, Cuijpers P, Funk B, Lehr D. Self-guided 25 26 internet-based and mobile-based stress management for employees: results of a 27 randomised controlled trial. 2016 [cited 2018 May 28]; [doi: 10.1136/oemed-2015- 28 103269] 29 24. Lazarus RS, Folkman S. Stress, Appraisal, and CopingSpringer Publishing Company 30 31 [Internet]. Springer ; 1984 [cited 2018 May 29]. Available from: 32 http://www.springerpub.com/stress-appraisal-and-coping.html 33 25. Fleischmann RJ, Harrer M, Zarski A-C, Baumeister H, Lehr D, Ebert DD. Patients’ 34 experiences in a guided Internet- and App-based stress intervention for college 35 students: A qualitative study. Internet Interv [Internet] Elsevier; 2018 Jun 1 [cited 2019 36 37 Jan 21];12:130–140. [doi: 10.1016/J.INVENT.2017.12.001] 38 26. Barrera M, Castro FG, Strycker LA, Toobert DJ, Toobert DJ. Cultural adaptations of 39 behavioral health interventions: a progress report. J Consult Clin Psychol [Internet] 40 NIH Public Access; 2013 Apr [cited 2018 May 28];81(2):196–205. PMID:22289132 41 42 27. Juniar D, van Ballegooijen W, Karyotaki E, van Schaik A, Passchier J, Heber E, Lehr 43 D, Sadarjoen SS, Riper H. Web-Based Stress Management Program for University 44 Students in Indonesia: Systematic Cultural Adaptation and Protocol for a Feasibility 45 Study. JMIR Res Protoc [Internet] 2019 Jan 25 [cited 2019 Feb 14];8(1):e11493. [doi: 46 10.2196/11493] 47 48 28. Gitlin LN. Introducing a new intervention: An overview of research phases and 49 common challenges. Am J Occup Ther 2013;67(2):177–184. [doi: 50 10.5014/ajot.2013.006742] 51 29. Orsmond GI, Cohn ES. The distinctive features of a feasibility study: Objectives and 52 53 guiding questions. OTJR Occup Particip Heal 2015;35(3):169–177. [doi: 54 10.1177/1539449215578649] 55 30. Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility 56 study? A Rev Curr Pract Editor policy BMC Med Res Methodol [Internet] 2010;10:1– 57 7. [doi: 10.1186/1471-2288-10-67] 58 59 31. Tickle-Degnen L. Nuts and Bolts of Conducting Feasibility Studies Linda Tickle- 60 Degnen MeSH TERMS feasibility studies occupational therapy randomized controlled 61 62 63 64 26 65 trials as topic treatment outcome. Am J Occup Ther [Internet] 2013;67:171–176. [doi: 1 10.5014/ajot.2013.006270] 2 3 32. Lovibond PF, Lovibond SH. The Structure Of Negative Emotional States: Comparison 4 Of The Depression Anxiety Stress Scales (DASS) With The Beck Depression And 5 Anxiety Inventories. Res Ther [Internet] 1995 [cited 2018 May 28];33(3):335–343. 6 33. Billingham SA, Whitehead AL, Julious SA. An audit of sample sizes for pilot and 7 feasibility trials being undertaken in the United Kingdom registered in the United 8 9 Kingdom Clinical Research Network database. BMC Med Res Methodol [Internet] 10 BioMed Central; 2013 Dec 20 [cited 2018 Jun 7];13(1):104. [doi: 10.1186/1471-2288- 11 13-104] 12 34. Nezu AM, Nezu CM, D’Zurilla TJ. Problem-Solving Therapy. Probl Ther 2012; [doi: 13 14 10.1891/9780826109415] 15 35. Heber E, Ebert DD, Lehr D, Nobis S, Berking M, Riper H. Efficacy and cost- 16 effectiveness of a web-based and mobile stress-management intervention for 17 employees: design of a randomized controlled trial. BMC Public Health [Internet] 18 BioMed Central; 2013 Dec 15 [cited 2018 Apr 14];13(1):655. [doi: 10.1186/1471- 19 20 2458-13-655] 21 36. Berking M, Ebert D, Cuijpers P, Hofmann SG. Emotion regulation skills training 22 enhances the efficacy of inpatient cognitive behavioral therapy for major depressive 23 disorder: A randomized controlled trial. Psychother Psychosom 2013;82(4):234–245. 24 [doi: 10.1159/000348448] 25 26 37. Heber E. Feedback Guide für GET ON Stress-neue Version. 2013. 27 38. Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of client/patient 28 satisfaction: Development of a general scale. Eval Program Plann [Internet] 29 1979;2(3):197–207. [doi: https://doi.org/10.1016/0149-7189(79)90094-6] 30 31 39. Attkisson CC, Zwick R. The Client Satisfaction Questionnaire Psychometric 32 Properties and Correlations with Service Utilization and Psychotherapy Outcome. Eval 33 Program Plann 1982;5:233–237. 34 40. Utoyo DB, Jaya ES, Arjadi R, Hanum L, Astri K, Putri MDD. Preliminary Study on 35 the Effectiveness of Short Group Cognitive Behavioral Therapy (GCBT) on 36 37 Indonesian Older Adults. PLoS One 2013;8(2). PMID:23437339 38 41. De Wilde EF, Hendriks VM. The Client Satisfaction Questionnaire: psychometric 39 properties in a Dutch addict population. Eur Addict Res [Internet] Karger Publishers; 40 2005 [cited 2018 May 22];11(4):157–62. PMID:16110221 41 42 42. Sharfina Z, Santoso HB. An Indonesian adaptation of the System Usability Scale 43 (SUS). 2016 Int Conf Adv Comput Sci Inf Syst ICACSIS 2016 2017;(December):145– 44 148. [doi: 10.1109/ICACSIS.2016.7872776] 45 43. Brooke J. SUS - A quick and dirty usability scale. Usability Eval Ind 1996; 46 PMID:19172650 47 48 44. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability 49 scale. Int J Hum Comput Interact 2008;24(6):574–594. PMID:33402155 50 45. van Ballegooijen W, Cuijpers P, van Straten A, Karyotaki E, Andersson G, Smit JH, 51 Riper H. Adherence to Internet-Based and Face-to-Face Cognitive Behavioural 52 53 Therapy for Depression: A Meta-Analysis. García AV, editor. PLoS One [Internet] 54 Public Library of Science; 2014 Jul 16 [cited 2018 May 28];9(7):e100674. [doi: 55 10.1371/journal.pone.0100674] 56 46. Damanik ED. Pengujian reliabilitas , validitas , analisis item dan pembuatan norma 57 Depression Anxiety Stress Scale ( DASS ): Berdasarkan penelitian pada kelompok 58 59 sampel Yogyakarta dan Bantul yang mengalami gempa bumi dan kelompok sampel 60 Jakarta dan sekitarnya. 1997 61 62 63 64 27 65 47. Purba FD, Hunfeld JAM, Iskandarsyah A, Fitriana TS, Sadarjoen SS, Passchier J, 1 Busschbach JJ V. Quality of life of the Indonesian general population: Test-retest 2 3 reliability and population norms of the EQ-5D-5L and WHOQOL-BREF. Luo N, 4 editor. PLoS One [Internet] Public Library of Science; 2018 May 11 [cited 2018 Jun 5 18];13(5):e0197098. [doi: 10.1371/journal.pone.0197098] 6 48. Skevington SM, Lotfy M, O ’connell KA. The World Health Organization’s 7 WHOQOL-BREF quality of life assessment: Psychometric properties and results of 8 9 the international field trial A Report from the WHOQOL Group. Qual Life Res 10 [Internet] 2004 [cited 2018 May 28];13:299–310. Available from: https://search- 11 proquest-com.vu- 12 nl.idm.oclc.org/docview/883821351/fulltextPDF/7C6B2E10AA940AFPQ/2?accountid 13 14 =10978 15 49. Ch Salim O, Sudharma NI, Kusumaratna RK, Hidayat A. Validity and reliability of 16 World Health Organization Quality of Life-BREF to assess the quality of life in the 17 elderly. Universa Med [Internet] 2007 [cited 2018 Jun 18];26:27–38. 18 50. Knaevelsrud C, Maercker A. Internet-based treatment for PTSD reduces distress and 19 20 facilitates the development of a strong therapeutic alliance: a randomized controlled 21 clinical trial. BMC Psychiatry [Internet] BioMed Central; 2007 Dec 19 [cited 2018 22 Apr 19];7(1):13. [doi: 10.1186/1471-244X-7-13] 23 51. Muddimer A, Peres SC, McLellan S. The Effect of Experience on System Usability 24 Scale Ratings. J Usability Stud 2012;7(2):56 67. 25 – 26 52. Waller R, Gilbody S. Barriers to the uptake of computerized cognitive behavioural 27 therapy: A systematic review of the quantitative and qualitative evidence. Psychol 28 Med 2009;39(5):705–712. [doi: 10.1017/S0033291708004224] 29 53. Christensen H, Griffiths KM, Farrer L. Adherence in Internet Interventions for 30 31 Anxiety and Depression: Systematic Review. J Med Internet Res [Internet] 2009 [cited 32 2020 Jun 29];11(2):e13. PMID:19403466 33 54. Melville KM, Casey LM, Kavanagh DJ. Dropout from internet-based treatment for 34 psychological disorders. Br J Clin Psychol 2010;49(4):455–471. [doi: 35 10.1348/014466509X472138] 36 37 55. Beatty L, Binnion C. A Systematic Review of Predictors of, and Reasons for, 38 Adherence to Online Psychological Interventions. Int J Behav Med 2016;23(6):776– 39 794. [doi: 10.1007/s12529-016-9556-9] 40 56. Arjadi R, Nauta MH, Bockting CLH. Acceptability of internet-based interventions for 41 42 depression in Indonesia. Internet Interv [Internet] 2018 Sep [cited 2018 May 14];13:8– 43 15. [doi: 10.1016/j.invent.2018.04.004] 44 57. Yusufov M, Nicoloro-Santabarbara J, Grey NE, Moyer A, Lobel M. International 45 Journal of Stress Management Meta-Analytic Evaluation of Stress Reduction 46 Interventions for Undergraduate and Graduate Students Meta-Analytic Evaluation of 47 48 Stress Reduction Interventions for Undergraduate and Graduate Students. Int J Stress 49 Manag [Internet] 2018; [doi: 10.1037/str0000099] 50 58. Harrer M, Adam SH, Baumeister H, Cuijpers P, Karyotaki E, Auerbach RP, Kessler 51 RC, Bruffaerts R, Berking M, Ebert DD. Internet interventions for mental health in 52 53 university students: A systematic review and meta-analysis. Int J Methods Psychiatr 54 Res 2019;28(2):1–18. [doi: 10.1002/mpr.1759] 55 59. Sillence E, Briggs P, Harris PR, Fishwick L. How do patients evaluate and make use 56 of online health information? Soc Sci Med 2007; [doi: 57 10.1016/j.socscimed.2007.01.012] 58 59 60. Harper Shehadeh MJ, Abi Ramia J, Cuijpers P, El Chammay R, Heim E, Kheir W, 60 Saeed K, van Ommeren M, van’t Hof E, Watts S, Wenger A, Zoghbi E, Carswell K. 61 62 63 64 28 65 Step-by-Step, an E-Mental Health Intervention for Depression: A Mixed Methods Pilot 1 Study From Lebanon. Front Psychiatry 2020;10(February):1–11. [doi: 2 3 10.3389/fpsyt.2019.00986] 4 61. Kohrt BA, Asher L, Bhardwaj A, Fazel M, Jordans MJD, Mutamba BB, Nadkarni A, 5 Pedersen GA, Singla DR, Patel V. The role of communities in mental health care in 6 low-and middle-income countries: A meta-review of components and competencies. 7 Int J Environ Res Public Health 2018;15(6). [doi: 10.3390/ijerph15061279] 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 29 65 Additional file 1 2 3 File name : Additional file 1 4 5 6 7 File format : Word document (.doc) 8 9 10 Title : Response summary of open questions on module evaluation 11 12 Description of data : Questions in Rileks’ module evaluation, quotes from participants’ 13 14 15 responses to each question, and points of summary. 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 30 65 Figure EQUATOR Network Reporting Checklist (see Reporting Standards section for this article type via the Information For Authors button above)

STROBE Statement—checklist of items that should be included in reports of observational studies

Item Page Relevant text from No. Recommendation No. manuscript Title and abstract 1 (a) Indicate the study’s design with a commonly used term in the title or the abstract 2 A single-group pre-test and post-test design was used (b) Provide in the abstract an informative and balanced summary of what was done and what was 1 - 2 The whole abstract section found Introduction Background/rationale 2 Explain the scientific background and rationale for the investigation being reported 4-5 The whole background section Objectives 3 State specific objectives, including any prespecified hypotheses 6 This study aimed to investigate the feasibility of Rileks among university students in Indonesia in terms of acceptability, usability and intervention uptake. The secondary aim was to investigate stress, anxiety, and depression reduction, improvement of quality of life, and additionally to generate feedback for further refinement of Rileks. Methods Study design 4 Present key elements of study design early in the paper 7 Study design and procedure: A single-group pre-test (t0) and post-test (t1) design was used. Setting 5 Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, 6 - 7 The study targeted university follow-up, and data collection students with a minimum age of 19 years in Indonesia….. Information about our study and our website was disseminated

1 through social media platforms such as Facebook, Instagram, and WhatsApp groups; and through presentations at events at two universities in Indonesia….. Participants were recruited between 10 and 19 October 2018.

Interested university students who signed up on our website subsequently received a link to the screening measurement (DASS-42 Stress Scale)…. All included participants received login credentials to the intervention and only those who logged in received post- treatment measurements (t1) ten weeks after t0….. All measures were self-reported and administered online. Participants 6 (a) Cohort study—Give the eligibility criteria, and the sources and methods of selection of 6 Participants and sample size. participants. Describe methods of follow-up Inclusion criteria: a minimum Case-control study—Give the eligibility criteria, and the sources and methods of case age of 19 years, having a score ascertainment and control selection. Give the rationale for the choice of cases and controls of 15 (low stress level) or higher Cross-sectional study—Give the eligibility criteria, and the sources and methods of selection of on the Depression Anxiety participants Stress Scales-42 (DASS-42; [32]), studying at a university in Indonesia, having access to the internet, and being able to speak Bahasa Indonesia fluently.

2 (b) Cohort study—For matched studies, give matching criteria and number of exposed and Not applicable NA unexposed (NA) Case-control study—For matched studies, give matching criteria and the number of controls per case Variables 7 Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers. 8 - 10 Primary outcome: Acceptability, Give diagnostic criteria, if applicable usability, intervention uptake.

Secondary outcome: Severity of stress, anxiety, and depression; Quality of life

Data sources/ 8* For each variable of interest, give sources of data and details of methods of assessment 8 - 10 Acceptability was measured by measurement (measurement). Describe comparability of assessment methods if there is more than one group assessing client’s satisfaction with Rileks using the translated version of the Client Satisfaction Questionnaire-8 (CSQ-8) [38–40].

The Indonesian version of the System Usability Scale (SUS) [42] was used to assess the usability of Rileks in terms of user-friendliness [43,44].

Both the CSQ-8 and the SUS were only administered at post- intervention.

3 Intervention uptake was measured by assessing the number of participants who completed the core online sessions (sessions 1 to 5) where participants learn the basic principles of problem-solving and emotion regulation.

The severity of stress, anxiety, and depression was measured using the Indonesian version of the Depression Anxiety Stress Scale-42 (DASS-42) [46].

Quality of life was measured by the Indonesian version of the World Health Organization- Quality Of Life brief version (WHOQOL-BREF) [47].

The DASS-42 and the WHOQOL-BREF were assessed both at t0 and t1. Bias 9 Describe any efforts to address potential sources of bias 13 …Of those who enrolled, 68 participants (56.2%) logged in. These 68 did not differ from those who never logged in….

Note: we addressed attrition bias by looking at differences between groups who started the

4 intervention and those who did not started the intervention. Study size 10 Explain how the study size was arrived at 6 A formal calculation of sample size used for effectiveness trials is not suitable for a feasibility study [33]. In our study, we intended to include at least 50 participants with a saturation of 75 participants to ensure sufficiently reliable estimates of our main study parameters. We based this estimate on a systematic study analysing sample sizes in pilot and feasibility studies in the United Kingdom reporting a median of 36 participants for a feasibility study sample size [33].

Continued on next page

5 Quantitative 11 Explain how quantitative variables were handled in the analyses. If applicable, describe which 11 Descriptive statistics were used to variables groupings were chosen and why examine primary outcomes, demographic data and sessions evaluation….. Differences in demographic characteristics between participants who logged in and those who did not log in were tested using a chi-square test in terms of sex, level of education, field of study, and university location. Independent sample t-tests were used to assess differences in the mean of age, the DASS-42 and WHOQOL-BREF scores. Statistical 12 (a) Describe all statistical methods, including those used to control for confounding 11 Participants' satisfaction (CSQ-8) methods and system usability (SUS) as feasibility parameters were summarized using means and standard deviations. Intervention uptake was summarized with frequency of participants who logged in or completed each session.

Secondary outcomes were analysed using two-tailed paired-samples t- tests with a level of significance of α = 0.05. (b) Describe any methods used to examine subgroups and interactions NA NA (c) Explain how missing data were addressed NA NA (d) Cohort study—If applicable, explain how loss to follow-up was addressed NA NA

6 Case-control study—If applicable, explain how matching of cases and controls was addressed Cross-sectional study—If applicable, describe analytical methods taking account of sampling strategy (e) Describe any sensitivity analyses 11 A normality test at each time point was conducted beforehand, using the Shapiro–Wilk tests. As the distributions significantly differed from normal (p values < 0.05), a sensitivity analysis using non- parametric tests e.g. Wilcoxon was conducted. Results Participants 13* (a) Report numbers of individuals at each stage of study—eg numbers potentially eligible, examined 12 A total of 191 university students for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed registered and completed the screening on the study website.

Of the 191 registered university students, 40 (21%) scored lower than 15 on the DASS-42 stress scale, which can be considered as normal stress, and 151 (79%) were eligible for inclusion.

Baseline questionnaires were completed by 121 participants.

Ten weeks after t0, post-treatment questionnaires were sent to the 68 participants who had logged in, which were completed by 25 participants

7 At t1, the DASS-42 stress scale was 16 completed by 23 participants consist of the core sessions completers and non-completers. The DASS-42 depression and anxiety scales, as well as the WHOQOL-BREF, were completed by 20 participants. (b) Give reasons for non-participation at each stage 13 Reasons for non-adherence were mostly unknown because those participants could not be reached. Several known reasons for non- adherence or withdrawal included time management problems, rare use of email so they missed notifications, unexpected events, (e.g. internship to a remote village with limited internet coverage), and technical problems such as unfamiliarity with the login system. (c) Consider use of a flow diagram Figure 1 Figure 1. Study flow Descriptive data 14* (a) Give characteristics of study participants (eg demographic, clinical, social) and information on 12; 14 … Baseline questionnaires were exposures and potential confounders completed by 121 participants. The majority were female (n=103; 85.1%), had a bachelor’s degree (n= 94; 77.7%) in social and behavioral sciences (n= 83; 68,6%), and studied on the island of Java (n=79; 65.3%). The average DASS- 42 stress score at baseline was

8 23.11 (SD=5.8), indicating moderate stress. A third of the participants (n=41; 33.9%) fell into the severe stress category (i.e. ≥ 26 -33). Those who logged in mostly study in Java and scored lower on psychological quality of life compared to those who did not logged in. See Table 1 for all participants’ characteristics at baseline.

Table 1. (b) Indicate number of participants with missing data for each variable of interest 12 Primary outcome: Ten weeks after t0, post-treatment questionnaires were sent to the 68 participants who had logged in, which were completed by 25 participants…

At t1, the DASS-42 stress scale was 16 completed by 23 participants consist of the core sessions completers and non-completers. The DASS-42 depression and anxiety scales, as well as the WHOQOL-BREF, were completed by 20 participants. (c) Cohort study—Summarise follow-up time (eg, average and total amount) NA NA Outcome data 15* Cohort study—Report numbers of outcome events or summary measures over time 15; 16 Table 4; Table 5 Case-control study—Report numbers in each exposure category, or summary measures of exposure NA NA Cross-sectional study—Report numbers of outcome events or summary measures NA NA

9 Main results 16 (a) Give unadjusted estimates and, if applicable, confounder-adjusted estimates and their precision 11 Secondary outcomes were analysed (eg, 95% confidence interval). Make clear which confounders were adjusted for and why they were using two-tailed paired-samples t- included tests with a level of significance of α = 0.05. (b) Report category boundaries when continuous variables were categorized 8 - 9 We set an average score of above 20 (20 is the median total score) as criterion to acceptable satisfaction.

A score of 70 or more was considered adequate as a feasibility criterion [44].

The criterion for acceptable adherence was set at 60% or more participants who completed the core sessions [27]. (c) If relevant, consider translating estimates of relative risk into absolute risk for a meaningful time NA NA period Continued on next page

10 Other analyses 17 Report other analyses done—eg analyses of subgroups and interactions, and sensitivity analyses 11 A normality test at each time point was conducted beforehand, using the Shapiro–Wilk tests. As the distributions significantly differed from normal (p values < 0.05), a sensitivity analysis using non- parametric tests e.g. Wilcoxon was conducted. Discussion Key results 18 Summarise key results with reference to study objectives 19 The main aim of this study was to evaluate the feasibility of Rileks as part of adaptation process of a web- based stress management intervention among university students in Indonesia. Rileks was reported as being acceptable even though it’s usability and intervention uptake were still below our expected criteria. Study findings showed that participants’ stress level and quality of life improved at t1. The intervention was appreciated by participants in terms of content usefulness, easiness to complete, comprehensibility, suitability for Indonesian university students’ context, and the quality of eCoach support with more eCoach interaction was desired.

Limitations 19 Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss 23 However, our study has also a both direction and magnitude of any potential bias number of limitations. The dropout rates from enrollment stage to post- assessment was relatively high. The

11 data obtained in the post-assessment only came from 38% of the participants who were logged in to the intervention thus did not equally represent all participants. Interpretation 20 Give a cautious overall interpretation of results considering objectives, limitations, multiplicity of 23 - 24 Rileks shows potential feasibility for analyses, results from similar studies, and other relevant evidence Indonesian university students. However, our findings also underscore the need for further development of this kind of intervention in order to optimize the experience for Indonesian university students. Further refinements are needed regarding content and technical aspects. Despite the potential of web-based interventions and eMental Health in general to minimize the mental health gap among the Indonesian population, our study implies that more work needs to be done before we can scale up this kind of intervention in Indonesia. In sum, the intervention has potential, but it needs refinement before it can be optimally applied in this setting.

Generalisability 21 Discuss the generalisability (external validity) of the study results 23 …However, our study has also a number of limitations. The dropout rates from enrollment stage to post- assessment was relatively high. The data obtained in the post-assessment only came from 38% of the participants who were logged in to the intervention thus did not equally represent all participants.

12 Other information Funding 22 Give the source of funding and the role of the funders for the present study and, if applicable, for the 24 Financial support: The project was original study on which the present article is based supported by LPDP (The Indonesian Endowment Fund for Education) in 2015, funded as part of principal investigator’s PhD trajectory.

*Give information separately for cases and controls in case-control studies and, if applicable, for exposed and unexposed groups in cohort and cross-sectional studies.

Note: An Explanation and Elaboration article discusses each checklist item and gives methodological background and published examples of transparent reporting. The STROBE checklist is best used in conjunction with this article (freely available on the Web sites of PLoS Medicine at http://www.plosmedicine.org/, Annals of Internal Medicine at http://www.annals.org/, and Epidemiology at http://www.epidem.com/). Information on the STROBE Initiative is available at www.strobe-statement.org.

13 Supplementary Files

This is a list of supplementary les associated with this preprint. Click to download.

STROBEchecklistv4combinedPlosMedicine.docx Additionalle1.docx