Exploring Interaction Patterns in Job Search

Alfan Farizki Wicaksono Alistair Moffat The University of Melbourne The University of Melbourne Melbourne, Australia Melbourne, Australia

ABSTRACT for the query (the SERP, which itself might be broken into pages by We analyze interaction logs from Seek.com, a well-known Aus- the interface protocol, or might be presented as a single listing via tralasian employment site, with the goal of better understanding the continuous scrolling) is clicked so as to arrive at the corresponding ways in which users pursue their search goals following the issue of job details page; and an application action is recorded when the each query. Of particular interest are the patterns of job summary job seeker clicks the “apply” button in that job details page, with viewing and click-through behaviors that arise, and the differences the presumption being that they will then enter more information, in activity between mobile/tablet-based users (Android/iOS) and and at some time in the future, lodge their application. We also computer-based users. know the absolute time/date associated with the first action in each sequence. KEYWORDS As an example, consider the action sequence:

Interaction logs; user behavior; job search A0 = ⟨(“I”, 1, 0), (“I”, 3, 1), (“C”, 3, 3), (“I”, 4, 9), ACM Reference Format: (“I”, 5, 10), (“C”, 5, 11), (“I”, 6, 20), (“I”, 4, 22), Alfan Farizki Wicaksono and Alistair Moffat. 2018. Exploring Interaction (“C”, 4, 25), (“I”, 6, 40), (“I”, 7, 41)⟩ . Patterns in Job Search. In 23rd Australasian Document Computing Symposium (ADCS ’18), December 11–12, 2018, Dunedin, New Zealand. ACM, New York, In A0 there are three clicks, at ranks 3, 5, and 4; the user has started NY, USA, 8 pages. https://doi.org/10.1145/3291992.3292004 by viewing the position summary at rank 1 and ends by viewing the one at rank 7; and it takes 20 seconds until the summary at rank 1 INTRODUCTION 6 is viewed for the first time. Interaction logs are valuable resources for improving the quality Many authors have studied the interaction logs from web search and effectiveness of the systems they are drawn from. In the area of services [3, 4, 9–11, 20, 24, 25, 30]. However relatively little at- web search, interaction logs can be leveraged to improve document tention has been paid to the interaction logs from job search en- ranking algorithms [1, 14]; the usability of the system either via gines. Kudlyak and Faberman [16] analyze the transaction logs query suggestion [5] or query auto-completion [13]; and the pre- from SnagAJob, an online job search service. They observe the re- sentation of search results [27]. Interaction logs are also important lationship between the number of job applications per week and for developing search evaluation methods [2, 8, 21, 28]. the duration of job search process, measured in weeks. Spina et al. Here we examine an English-language interaction log derived [23] report statistics from the query and click-through logs of an from the well-known Australasian job search service Seek.com, to Australasian job search site Seek.com, and compare with general characterize the behaviors of users when engaged in post-query web search patterns in regard to the frequency distribution of click reading and evaluating activities. Our emphasis is on chronologically- ranks, and the number of distinct queries per user. Most recently, ordered action sequences, defined as Mansouri et al. [18] examine the job-related search queries issued to a general-purpose Persian-language search service. A = ⟨(a1,r1,m1), (a2,r2,m2),... ⟩ , Our work here explores a more comprehensive job search in- where at is an action type, rt is the position in the search ranking teraction log than any of these, in that not only does it include that is the site of the action, and mt is the start time of that action click-through actions, but also impression data and explicit appli- in seconds, normalized so that m1 = 0. Three types of action occur: cation actions. In addition, the log differentiates between browser- impressions, when at = “I”; mouse clicks, when at = “C”; and job based computer users (that is, those interacting with a web-based applications, when at = “A”. An impression occurs when any job service using browsers running on desktop and laptop computers) summary is completely visible on screen for at least 500 millisec- and Android/iOS-based users using the “Seek.com app” on mobile onds, and, in the absence of eye-fixation information, provides at phone and tablet-type devices. We use that distinction as part of least prima facie evidence that the user was reading that job sum- our analysis. mary. A click action occurs when a job summary in the result page Table 1 summarizes some characteristics of the data used in Permission to make digital or hard copies of all or part of this work for personal or this investigation. For privacy and commercial reasons this data classroom use is granted without fee provided that copies are not made or distributed cannot be made public. The terms of service and privacy policies for profit or commercial advantage and that copies bear this notice and the full citation of Seek.com were complied with at all times during the collection on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or and analysis of this data. republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. 2 BACKGROUND ADCS ’18, December 11–12, 2018, Dunedin, New Zealand © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-6549-9/18/12...$15.00 Interaction log study. Silverstein et al. [20] analyze query logs https://doi.org/10.1145/3291992.3292004 containing approximately one billion search requests, collected Android/iOS Browser and Moffat confirmed findings from the previous query andterm- level log studies, and also reported click-through-based findings, Users 4,705 3,894 including that the distribution of click-throughs across ranks is Queries/SERPs 39,545 20,129 top-weighted, and that the elapsed time between any two consecu- SERP size unlimited paginated, 20 tive click-throughs is typically short. The latter suggests that the Table 1: Job search interaction logs used in this study. The data is majority of users only need a few seconds to assess the usefulness from Seek.com, a well-known job search engine, and consists of or otherwise of most clicked documents. representative samples drawn from the period 30 July 2018 (Mon- day) to 26 August 2018 (Sunday) for both types of users. To prevent ambiguity, phone/tablet-originated queries via web browser soft- Job search. One of the more comprehensive studies in the job ware are excluded from both data sets. search domain was conducted by Jansen et al. [12], using an Excite dataset that had been filtered to identify job-related queries. Their findings suggest that users seeking jobs usually pose one query before ending the session; and that the majority use three terms from AltaVista, one of the first commercial web search engines. when constructing job-related queries. Recently, Kudlyak and Faber- Their main focus was on query-specific patterns such as the num- man [16] used data from the SnagAJob online job search service ber of queries per session, the number of terms and operators per to study the correlation between job search effort, measured in query, the frequency distribution of popular queries, and the char- the number of weekly applications, and job search duration. They acteristics of query revisions through the course of each session. found evidence for an income effect, whereby job seekers with poor Silverstein et al. also carried out a colocation analysis to identify job-finding prospects tend to make more job applications and have associations between terms across all queries. Their findings in- a longer search duration, suggesting that job seeker behavior is cluded that web search users typically posed short queries, and that affected by the strength of labor market they stand in. the majority of users only examine the first page of the paginated Spina et al. [23] studied user behavior in job and talent search, ranked-list of documents. Prior to that work, Jansen et al. [11] had and compared those behaviors with those of typical web search conducted similar studies of query logs, but at a smaller scale, cov- users in terms of query popularity and click depth. The datasets ering 51,473 queries from the Excite service, with similar findings used for this comparative evaluation were generated by Seek.com in terms of web search query length and the number of result pages (an Australasian job search company) and Yandex (a Russian web that were viewed. search company) for job/talent search and web search respectively. Spink et al. [24] extended the study of Jansen et al. [11], analyzing Finally, Mansouri et al. [18] analyze approximately 500,000 job- a log of approximately one million Excite queries, documenting related web queries selected from parsijoo.ir, a Persian general- very similar phenomena. Spink et al. also employed a more prin- purpose web search engine. They found that job search activities cipled approach in classifying a random sample of web queries, tend to be more intense near the beginning of each week, and to building insights in terms of common search topics. Lempel and then slowly decline through the balance of the week. Moran [17] used a different sample of seven million AltaVista queries to study the popularity of query topics and viewed pages, primarily as an argument for and evaluation of a new query caching Browser versus mobile users. There have been extensive obser- policy. One of their notable findings is that the frequency distribu- vational studies based on either lab-setting user studies or interac- tion of query topics can be modeled as a power law distribution. tion logs for both mobile and laptop/desktop (web browser) users. Other studies explored logs of web queries from BWIE search engine Joachims et al. [15] studied the behavior of participants while they Google [4]; from AlltheWeb.com [25]; and from Fireball [9]. Jansen and were using the search engine (via a desktop computer) to Spink [10] provide a perspective across all of those search engines, complete ten tasks, where half of them were informational and including a geographical evaluation of the differences between US the remainder were navigational. Click-throughs and eye-fixations and Europe-based services. They report query and session length, occurring during the search process were recorded, providing evi- query terms, query operators, and viewed pages in a comparative dence that on average web searchers had a tendency to scan down study that provides a clear reference point as to the situation that the ranking from top to bottom; that prior to a click action at rank pertained at the time. i the user had typically viewed the snippets before rank i in prefer- Most of the early query log studies did not examine temporal ence to those deeper than rank i; and that user clicking behavior aspects of web queries. As an exception, Beitzel et al. [3] explore varied according to their trust in the search engine being used as both topical and temporal dimensions from a log of billions of well as the quality of the SERPs. web queries from the AOL search service, examining how particular Cutrell and Guan [7] reported the results from an eye-tracking query topics vary over time. For example, they found that queries study to observe user behavior and performance of desktop users related to “personal finance” are popular between 7–10am. Zhang across informational and navigational tasks. They confirmed many and Moffat [30] conducted an analysis using interaction records of the findings of Joachims et al. [15], such as mean arrival time at that contained click-through information as well as queries. That each rank position and the distribution of viewed positions before dataset comprises approximately fifteen millions queries with their clicking at a particular rank, and also developed further results corresponding chronologically-ordered click-throughs, sampled regarding the relationship between task complexity and snippet from the MSN live web search engine during May 2006. Zhang length. By employing the click rate on relevant results (that is, 0.6 Android/iOS Browser 1 1 2 2 0.5 Click-throughs 85.1% 78.4% 3 3 Job applications 86.1% 77.5% 4 4 0.4 5 5 Table 2: Proportion of click-through actions and job applications 6 6 0.3 in which the user viewed one or more items beyond rank rt before

click rank 7 click rank 7 clicking-through or applying for the job listed at rank rt . The dif- 8 8 0.2 ference between Android/iOS and browser users is significant in 9 9 p < . z 10 10 0.1 both bases, with 0 05 using a two-sided -test. 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 # distinct docs clicked # distinct docs clicked 0.0 ′ max{rt ′ | t < t} − rt Android/iOS Browser Figure 1: Distribution of click ranks as a function of number 0 14.9% 21.6% of click-throughs in the action sequence. The left plot is for An- 1 43.6% 30.0% droid/iOS queries, the right one browser-initiated queries. 2 27.5% 21.3% 3 6.9% 9.5% 4 2.1% 5.7% the click accuracy) to measure the performance of users, they re- 5 1.0% 2.6% ported that increasing the length of snippets improved user per- ≥ 6 4.0% 9.3% formance for informational tasks, but degraded it for navigational ′ tasks. Thomas et al. [26] also carried out eye-tracking work, and Table 3: Distribution of the set of values max{rt ′ | t < t} − rt , noted that while users typically first arrived at documents in the given that rt = “C”. All Android/iOS versus browser relationships ranking in order, it tended to be via a “two steps forward, one step are significantp ( < 0.05, two-sided z-test). backward” progression. In recent work Ong et al. [19] compare the behavior of mobile and desktop users when interacting with web-based SERPs and one or more documents beyond rank rt . For example, in the action completing identical search tasks. Differences in behavior were sequence A0 given above, before clicking at rank 4, the user viewed observed as two information scent-based factors on the first page documents at rank 5 and 6. In the same example, that did not were varied: the volume of relevance and positional distribution occur with the click-throughs at ranks 3 and 5. The second row of relevant items, following prior work by Wu et al. [29]. Their in Table 2 shows the proportion of all click actions for which one findings include that compared to mobile users, desktop users take or more documents beyond some rank rt were registered as an longer to complete search tasks, viewed deeper ranking positions, impression at some step t ′ < t, prior to a click-through taking and clicked on more documents at deeper ranks. However, mobile place at rank rt at step t. As can be seen from the table, users users more accurately assess useful documents as being relevant, inspect summaries beyond rank rt more than two-thirds of the compared to desktop users. time, a higher fraction than the corresponding 50% result observed by Joachims et al. [15]. In addition, the difference between the two 3 IMPRESSIONS, CLICKS, APPLICATIONS access modes is statistically significantp ( < 0.05, z-test), with the We now consider a number of facets of the job search query inter- difference perhaps a consequence of the scrolling actions of browser action logs for Seek.com, summarized already in Table 1. users being more precise than those of Android/iOS users. The next results measure the depth the users reached relative Positional distribution of click-throughs. Clicks were observed to the rank r associated with the t th operation in the action se- in connection with fewer than 40% of the SERPs generated in re- t quence, given that they viewed one or more documents beyond sponse to queries, for both the browser- and mobile-sourced queries. rank rt before clicking at rank rt . Table 3 summarizes the frequency For the subset of queries for which click-throughs were noted, we ′ distribution (proportion) of max{r ′ | t < t} − r values over all categorized the ranks at which clicks occurred as a function of the t t action sequence indices t for which a = “C”. In the majority of total number of clicks in that SERP. Figure 1 shows those distribu- t cases, users had viewed the documents one or two further down tions. Each cell in Figure 1 represents the observed probability that the ranking than rank r within the first t − 1 elements of the action the user clicked at rank i (vertical axis), given that they clicked on t sequence, before then deciding at step t to click the document at a total of n distinct job summaries (horizontal axis). As expected, rank r . The differences in proportion between the two types of the more clicks there are in a SERP, the further down the SERP the t users are statistically significant across all rowsp ( < 0.05 using a clicks reach. But even so, the clicks tend to be top-heavy, and the two-sided z-test), meaning that browser users are more likely to re- closer to the top of the ranking a job summary appears, the more turn back past multiple job summaries to a “mentally bookmarked” likely it is to be clicked, regardless of how many clicks there will potential click point than are Android/iOS users. eventually be. That is, users tend to click on summaries early in the Figure 2 confirms the results summarized in Table 3, showing the ranking, even if they end up clicking multiple items. distribution of impression ranks as a function of click rank rt when Impressions deeper than clicks. Prior to actually clicking at at = “C”. In Figure 2, each cell represents the observed frequency ′ some rank rt at step t in the action sequence, users typically view of user views (impressions) on documents at rank rt ′ where t < t, 1.0 Android/iOS Browser 1 1 Clicks 2 2 freq. mean τ τ > 0 freq. mean τ τ > 0 3 3 0.8 4 4 2 48.5% 0.78 89.2% 53.1% 0.79 89.4% 5 5 0.6 3 23.3% 0.81 88.0% 20.4% 0.81 89.1% 6 6 4 10.7% 0.86 94.1% 13.3% 0.85 94.7% 7 7 0.4 5 5.9% 0.87 94.8% 4.3% 0.86 97.7% 8 8 impression ranks impression ranks 9 9 Table 5: Mean value of Kendall’s τ for extracted click sequences, 10 10 0.2 stratified by number of clicks, and the percentage of positive τ 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 values in each group. click rank (rt) click rank (rt) 0.0

Figure 2: Distribution of impression ranks preceding click actions at rank r . The left plot is for Android/iOS users, the right one for 0.5000 Android/iOS Users t Browser Users browser-initiated queries. 0.2500

0.1250

Android/iOS Browser 0.0625 proportion Clicks on 2 distinct summaries 87.0% 88.1% 0.0312 Clicks on 3 distinct summaries 87.9% 87.4% 0.0156 Clicks on 4 distinct summaries 88.3% 91.1% 0.0078 Table 4: Proportion of queries for which the last click action (at -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 step lc in the action sequence) is also the deepest click-through difference in click-through rank rank in the sequence. Figure 3: Distribution of click-through jumps. Note the logarithmic scale on the vertical axis. amalgamated over all of the available action sequences for which a = “C”. For example, consider the three action sequences: t τ score for each action sequence based on the sequence of click A1 = ⟨(“I”, 1), (“I”, 2), (“I”, 3), (“I”, 4), (“I”, 3), (“C”, 3)⟩ ranks when ordered by the step number. That is, the sequences ⟨(t,r ) | (a ,r ,m ) ∈ A⟩ were formed, and τ coefficients used A2 = ⟨(“I”, 1), (“I”, 3), (“I”, 5), (“I”, 3), (“C”, 3)⟩ t t t t to determine the extent to which t and rt correlate. Contiguous A = ⟨(“I”, 1), (“I”, 4), (“C”, 3), (“I”, 3), (“C”, 3), (“I”, 4), (“C”, 4)⟩ . 3 duplicate clicks to the same rank position were removed as part of where for simplicity the timestamp information mt is omitted. the filtering process. Looking again at the example action sequence Across these three sequences the observed “impression before click” A0, that derived sequence would be ⟨(3, 3), (6, 5), (9, 4)⟩, with a τ likelihood that a user views a summary at rank 1 before clicking score of 1/3 = 0.33. at rank 3 is 4/4 = 1, and of viewing the summary at rank 2 before Table 5 shows that most users click on summaries in generally clicking at rank 3 is 1/4 = 0.25. The even shading in the upper- increasing order. To further reinforce that observation, Figure 3 right triangular regions in Figure 2 clearly indicates that the user depicts the distribution of click-through jumps for the two types viewed almost all documents above rank rt , before it was clicked, of users, where a positive jump arises if two consecutive click- and had also viewed one or two documents (three in the case of throughs correspond to concords when computing τ coefficients, browser-originated SERPs) documents beyond rank r. As already and negative jumps correspond to discords. The fact that positive noted, this phenomenon is stronger for browser-based queries than click-through jumps dominate the distribution is strong evidence for Android/iOS-based queries. that job search users in general click from the top towards the bottom. Note that Android/iOS-based job search users exhibit a Click ordering. It is also interesting to examine the extent to slightly stronger tendency to do this. which the click actions are ordered in terms of rank position. We define the last click point, lc, as the index in the action sequence of After the last click. The bars in Figure 4 show the distribution of the final click-through, lc = max{t | at = “C”}. For example, in the each of: the deepest impression before the last click-through in the earlier action sequence A0 we have lc = 9, since the action (“C”, 4) action sequence; the deepest impression following the last click- at step 9 is the last click action. Given that the users clicked on through; and the last impression following the last click action. The two to four distinct documents, Table 4 shows that for 87%–91% of overall pattern is clear: even after their last click-through, users queries the last click is the deepest, suggesting that users do indeed continue to inspect further summaries in the SERP. For browser- proceed through the ranked list mostly sequentially. based queries, it is also interesting to see that the lower side of We also examined the extent of the ordering in the full set of the third element in each group of three (last impression after last clicks occurring in each action sequence, stratifying the queries click) extends to the shallowest rank position of the page, that is, based on number of click-throughs, and then computing a Kendall’s rank 1 for the first page, and rank 21 for the second page. This 70 70

60 60

50 50

40 40

30 30

impression ranks 20 impression ranks 20

10 10

0 0 1 3 5 7 9 11 13 15 17 19 21 23 25 1 3 5 7 9 11 13 15 17 19 21 23 25 Android/iOS - depth of last click Browser - depth of last click

Figure 4: Distribution of the deepest impression prior to the last click action (left element in each group of three); the deepest impression following the last click (middle element in each group); and the last impression following the last click (right element). The green triangle represents the mean value of each box-whisker element. Only odd-depth last click-throughs are shown, with Android/iOS queries in the left-hand pane, and browser-based queries in the right-hand pane; and only action sequences in which impressions occurred both before and after the last click are included, so that all three elements in each group are over the same set of action sequences.

Android/iOS Browser Android/iOS Users 5.0×10−1 The last click in A at index lc is... Browser Users −1 ...the deepest click in A 95.4% 96.9% 2.5×10 ...the last action in A 16.5% 40.0% 1.2×10−1

...the deepest action in A 14.0% 18.8% 6.2×10−2

The deepest click in A is... proportion 3.1×10−2 A ...the deepest action in 15.5% 19.6% 1.6×10−2

Table 6: Statistics in regard to the last click and the deepest click 7.8×10−3 across the set of action sequences. All paired differences are signifi- 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cant (p < 0.05, two-sided z-test). # distinct new docs

Figure 5: Fraction of action sequences, categorized by the number suggests that browser users tend to scroll up the page to reach the of previously-unseen items viewed beyond the last click-through. top position before they pose a new query or stop searching. Note the logarithmic vertical scale. Two assumptions arise when considering click sequences: (1) If a user clicks at some rank rt , they have already examined the documents at ranks 1 to rt . A further question that then arises is whether the impressions (2) The last click-through action is the last element in the action that occur following the last click-through are to items that have sequence. been previously viewed, or to new items being looked at for the first Zhang et al. [31] develop their observation model based on the time. Figure 5 shows the frequency distribution (proportion) of the first assumption, but also use the average interval between the number of distinct “first impression” items viewed following the clicks within that zone to extrapolate further impressions beyond last click action, not counting impressions for job summaries that the last click-through. The second assumption is used by Smucker had previously generated one or more impressions in the action se- and Clarke [21] to approximate the time spent by user to examine quence. For example, in the example provided by A0 (see Section 1), the SERP, by computing the time duration between the search the number of new items viewed beyond the last click is one (the commencement and the last click. Carterette [6] and Azzopardi item at rank 7). Figure 5 shows that only around 40–50% of last click et al. [2] also employ the second assumption as a basis to meta- actions (for both Android/iOS and browser users) were followed evaluate their proposed user models (and hence dual effectiveness by impressions that exposed hitherto unviewed summaries. metrics) by measuring how well their proposed models predict user stopping behavior. Job applications. The second row of Table 2 shows the proportion The first of these two assumptions seems to be supported by of job application actions for which the user viewed one or more the results depicted in Figure 2. However, the second assumption documents beyond some rank rt before initiating the application is at odds with our results, which clearly record users inspecting process for a job whose summary was at rank rt . Table 7 expands on multiple job summaries beyond the last click. Table 6 provides that summary information, listing how deep the users go relative to further information about the relationship between the last click the application ranks rt at which at = “A”, showing the frequency ′ and the deepest (highest rank) click. distribution of the differences max{rt ′ | t < t} − rt . 1.0 1 1 Android/iOS Users 2 2 Browser Users 3 3 0.8 4 4 5 5 0.6 6 6 7 7 0.4 8 8 impression ranks impression ranks 9 9 10 10 0.2 application rate (apps per query) 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 0.0 0 1 2 3 4 5 6 application rank (rt) application rank (rt) 0.0 # distinct docs clicked

Figure 6: Distribution of impression ranks observed before a job Figure 8: Job application rate as a function of click-throughs per application action took place at rank rt . The left plot is for An- query. In general, the likelihood of making an application increases droid/iOS users, the right one for browser-initiated queries. as the number of distinct summaries clicked increases. The vertical scale is linear; but for commercial reasons is not labeled. ′ max{rt ′ | t < t} − rt Android/iOS Browser 1 56.0% 36.8% actions took place, stratified by rank rt . Each cell in Figure 7 repre- 2 29.5% 24.2% sents the observed likelihood of the user clicking the job summary ′ 3 7.5% 7.6% at that rank rt ′ , where t < t, before they apply for job advertised at 4 1.4% 4.6% rank rt , the “click before application” ratio. The most notable – and ≥ 5 5.6% 26.8% predictable – effect here is that an application at rank rt is always ′ preceded at some point in the action sequence by a click-through Table 7: Distribution of the set of values max{r ′ | t < t} − r , t t at rank r . But click-throughs also occur at other ranks, including given that a = “A”. All Android/iOS versus browser relationships t t greater than r . are significantp ( < 0.05, two-sided z-test), except in row “3”. t We also stratified the action sequences based on the number of click-throughs they contained, as described earlier. Within each stratum the application rate was computed, the total number of job 1.0 1 1 applications in those action sequences divided by the number of 2 2 SERPs served. Figure 8 plots the resultant data, and shows that the 3 3 0.8 application rate broadly increases as the number of click-throughs 4 4 increases, especially for Android/iOS-sourced queries. (For com- 5 5 0.6 mercial reasons we cannot provide labels on the vertical scale.) In 6 6 the case of the Android/iOS-based users, there is a clear relationship

click ranks 7 click ranks 7 0.4 8 8 between number of clicks and application rate. The relationship is 9 9 less apparent for browser-based users, with the reasons behind the 0.2 10 10 difference in behavior not currently known. 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 application rank (rt) application rank (rt) 0.0 4 TIME-BASED TRENDS This section explores the timestamps that are associated with each Figure 7: Distribution of click-through ranks observed before a item in the action sequences. job application action took place at rank rt . The left plot is for Android/iOS users, the right one for browser-initiated queries. Time of day. Figure 9 shows the times during the day at which click-throughs (top) and job applications (bottom) occur. Not unex- pectedly, job search intensity is low during the night (sleeping time) Figure 6 elaborates on Table 7, showing the distribution of im- and highest during the day (“working” time) and into the evening pression ranks that occurred before application actions took place (leisure time). Others – see, for example, Zhang and Moffat [30] – at each rank r . Each cell in Figure 6 represents the observed frac- t have also observed the same patterns in terms of click-throughs tion of cases in which user viewed (via an impression in the action and queries for web search. It is also interesting to note that the sequence at some step t ′ < t) a job summary at that rank before proportion of click-throughs of desktop-based browser users is eventually applying for the job appearing at rank r , the “impres- t higher than that of Android/iOS users during working hours (10am sion before application” distribution. Table 2 and Table 7 show that to 6pm), but the opposite happens in the early morning and in the browser-based SERPS generate more impressions beyond rank r t evening. It seems possible that users are using office computers in before applications at rank r than do Android/iOS-based SERPs. t one job to search for new positions elsewhere. This finding is also Clicks and applications. Figure 7 shows the corresponding dis- in agreement with the observational studies on Bing search logs tribution of click-through ranks observed before job application conducted by Song et al. [22]. ehv nlzditrcinlg rvddby provided logs interaction analyzed have We oietbe-ae nri/O p-rgntdqeis and queries, app-originated covering Android/iOS mobile/tablet-based service, search job line CONCLUSION 5 as used is median the if tendency. faster central are the but triangles), on action green users the (the app-based in average that appears longer sort) take users any Browser-based (of sequence. action recorded next is the action click-through until as each through when and measured from time position, rank elapsed the by click stratified times, reading Time. Reading midnight. is before there evening weekend the the in at occurring whereas activity morning, considerable the also in tend active users more weekdays be On to users. Android/iOS for weekdays and days definitive. than rather indicative being as regarded be should at differences significant showed comparisons day-versus-day four-element the the with al. agreement et in Mansouri week, by the observed trends of end the slowly to then through and Wednesday, decreases appli- and and Monday click-throughs between peaks of cations, number the Job search via as period. measured week, collection data intensity, the the four-week of days during seven identified the across applications and throughs week. of Day and data. weekdays sampled to all restricted over periods, aggregated two-hour in pane) (lower tions 9: Figure

iue1 oprsteciktruhitniybtenweekend between intensity click-through the compares 11 Figure proportion proportion 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.000 0.025 0.050 0.075 0.100 0.125 0.150 ieo lc-hogs(pe ae n o applica- job and pane) (upper click-throughs of Time 0:00 - 1:59 0:00 - 1:59 Browser Users Android/iOS Users Browser Users Android/iOS Users p iue1 iial hw h itiuino click- of distribution the shows similarly 10 Figure 0 =

iue1 lt h itiuino o description job of distribution the plots 12 Figure 2:00 - 3:59 2:00 - 3:59 .

05 4:00 - 5:59 4:00 - 5:59 n ec httepteni iue10 Figure in pattern the that hence and , 6:00 - 7:59 6:00 - 7:59

8:00 - 9:59 8:00 - 9:59

[18] 10:00 - 11:59 10:00 - 11:59 39 , 545 oe oee,ta oeof none that however, Note, . 12:00 - 13:59 12:00 - 13:59

ure n EP for SERPs and queries 14:00 - 15:59 14:00 - 15:59

16:00 - 17:59 16:00 - 17:59 Seek.com 18:00 - 19:59 18:00 - 19:59

20:00 - 21:59 20:00 - 21:59

non- an , 22:00 - 23:59 22:00 - 23:59 20 , 129 hc eearaykoni h oegnrlwbsac context. search web general more the in known already were which ae oehr hs bevtospoiea mrvdunder- improved an provide observations these together, Taken that: include These ain lwrpn)ars h aso h week. the of days the across pane) (lower cations 10: Figure h nlsso cinsqecsi h a htw aedn so done have we that way the in sequences that action believe of nevertheless analysis observed we the the explained, which and motivated in are overlay actions modeling a without study, purchase. tional car and home as such decisions, including major perhaps other marketplaces, other for online to those to only also not but transferable tools, partic- be search this job may with and interact engine, seekers search job job which ular in ways the of standing of some observations, of number a in resulted has queries. evaluation browser-originated Our computer desktop/laptop for SERPs • • • • • • hti,wieti urn okhsbe rmrl nobserva- an primarily been has work current this while is, That sipt” nodrfo o onadi h EP n sim- and SERP, the in downward top from order in “snippets”) adte ehp pl o)tejbstae trank at situated job the for) apply perhaps then (and erhatvt ihohratvte nawyta sls achiev- less is that devices. way mobile a on in able activities other job with interleave activity search users computer-based queries, that app-based suggesting for times perhaps than several queries browser-based are for durations longer reading SERP also elapsed application Measured job a making of the increases. in chance clicked the documents increases, distinct of SERP number the as SERP. general, each In with the associated after action summaries click-through job last further several examine to tend Users SERP. sum- description job rank given of any number beyond small maries a examine to tend Users early at summaries job positions. click-through rank to likely more are Users top from actions click-through downward. their undertake to tend ilarly on (impressions summaries description job view to tend Users proportion proportion 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 0.00 0.05 0.10 0.15 0.20 itiuino lc-hogs(pe ae n appli- and pane) (upper click-throughs of Distribution Mon Mon Tue Tue Wed Wed r t Thu Thu eoedcdn oclick-through to deciding before Fri Fri Browser Users Android/iOS Users Browser Users Android/iOS Users Sat Sat Sun Sun r t nthe in ac niain fwa eairlpten ih eregarded be might patterns behavioral what of indications vance eea epu discussions. helpful several by and num- LP150100252) (project scheme ber funding Projects Linkage Council’s Research Acknowledgment. of as kinds such different sites the that represent work, activity that future “personas” in construct behavior to job-search of and aspect that We explore lodgment. to asapplication plan such decisions significant prefacing ad- as providing include would streams so data doing similar of other goals for used Possible be future. can in that tool general a is here for one right the queries, Android/iOS queries. for browser-based is pair each in element 12: Figure and pane) pane). (upper (lower users users Android/iOS browser for days weekend and days 11: Figure

proportion proportion time (seconds) 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.000 0.025 0.050 0.075 0.100 0.125 0.150 2 2 2 2 2 2 2 − − 1 1 1 5 9 3 7 7 3 lc-hogsi w-orpros oprn week comparing periods, two-hour in Click-throughs 1 ouetraigtm,srtfe yrn.Theleft by rank. stratified time, reading Document 0:00 - 1:59 0:00 - 1:59 Weekends Weekdays Weekends Weekdays

2:00 - 3:59 2:00 - 3:59 2 hswr a upre ne h Australian the under supported was work This 4:00 - 5:59 4:00 - 5:59

6:00 - 7:59 6:00 - 7:59 Seek.com Seek.com

8:00 - 9:59 8:00 - 9:59 3 click rank 10:00 - 11:59 10:00 - 11:59 support. utnZblpriiae in participated Zobel Justin . 12:00 - 13:59 12:00 - 13:59 4

14:00 - 15:59 14:00 - 15:59

16:00 - 17:59 16:00 - 17:59 5 18:00 - 19:59 18:00 - 19:59

20:00 - 21:59 20:00 - 21:59 6 22:00 - 23:59 22:00 - 23:59 [12] REFERENCES [31] [21] [19] [14] [11] [10] [30] [29] [28] [27] [26] [25] [24] [23] [22] [20] [18] [17] [16] [15] [13] [1] [9] [8] [7] query Context-aware Li. H. [6] and Chen, E. Liao, Z. He, Q. Pei, J. Jiang, D. Cao, H. [5] [4] [3] [2] e erhegn ur log. query engine search web td fue ure nteweb. the on queries user of study A .Mfa,M adro,F coe,adJ oe.Udrtniguserbehavior Zobel. Understanding J. and F.Scholer, M. Sanderson, Moffat, A. 1518,2018. 1185–1188, .J asn .J asn n .Sik sn h e olo o ok Implications work: for look to web the Using Spink. A. and Jansen, J. retrieval: K. Jansen, information J. life B. Real Saracevic. T. and Bateman, J. Spink, A. Jansen, com- J. A B. web? wide world logs. the transaction searching 2006. engine we search are nine How of Spink. parison A. and newbies. Jansen and J. experts B. internet of behavior search Networks Web Computer Strube. G. and Hölscher comparison In C. interleaved data. Estimating click Rijke. historical of de from study M. outcomes and eye-tracking Whiteson, An S. for? Hofmann, looking In K. you search. are web What in usage Guan. information Z. and Cutrell Aconceptual E. user utility: In and investigation. for models, framework user effectiveness, System Carterette. B. In data. 2008. session 883, and click-through mining by suggestion 2001. 319–325, pages In e-Business. for Vi analysis statistical A. and Cacheda F. log. query web Temporal categorized Grossman. topically D. Tech. large and Frieder, very O. a Chowdhury, of A. analysis Jensen, C. E. engine Beitzel, search M. of S. utility the In Measuring Craswell. pages. N. result and Thomas, incorpo- P. by Azzopardi, ranking L. search In web information. Improving behavior Dumais. user S. rating and Brill, E. Agichtein, E. .Zag .A .Pr,adA oft lc-ae vdnefrdcyn weight decaying for evidence Click-based metrics. Moffat. effectiveness A. search and in Park, distributions F. In A. behavior. L. Zhang, user search Y. on Symp. observations Comp. Some Doc. Aust. Moffat. cognition A. for and need and Zhang In Y. scent information behavior. Using search online Sud. understand A. effectivenessto and Kelly, for search D. Wu, evidence C. W. Empirical Moffat. In A. models. Beyond and Wicaksono Mei. F. Q. A. and Chang, Y. Yamada, M. 2016. In Wang, presentation. P. whole-page Jie, Optimizing ranking: L. it.In Yin, have D. eyes The Wang, do: Y. Conf. users Soc. What Retri. Info. Moffat. web Asia A. European and versus Scholer, U.S. F. Thomas, Jansen. P. J. B. and Ozmutlu, The trends. C. web: searching H. the Ozmutlu, Searching S. Spink, Saracevic. A. T. and queries. Jansen, their and J. public B. Wolfram, D. Spink, A. In Cavedon, investigation. L. initial Baldwin, An 2017. T. search: talent Wong, and W. job Sadeghi, in S. Ren, Y. In Maistro, M. relevance. Spina, search D. search improve user to exploiting devices and tablet Exploring WWW and Wang. mobile K. effectiveness and on Wang, of behavior H. calibration Ma, Time-based H. Song, Y. Clarke. A. L. In C. measures. and large very Smucker a D. of M. Analysis Moricz. M. and Henzinger, M. Marais, scent H. information Silverstein, C. Using In behavior. Scholer. search 2017. web F. 295–304, desktop and and Sanderson, mobile understand M. to Järvelin, K. Ong, K. In Study logs. search: query job engine Online search using Farhoodi. M. behavior and search Campos, users’ R. of Zahedi, S. in M. results Mansouri, query B. of prefetching and In caching engines. Predictive search Moran. S. and Lempel R. https://papers. Richmond, duration, of search Bank and Researve search ssrn.com/sol3/papers.cfm?abstract_id=2442910. Federal job 14-12, of Paper intensity Working The 2014. Faberman. J. and Kudlyak M. feedback. interpreting implicit Accurately Gay. as G. data and Hembrooke, clickthrough H. Pan, B. Granka, L. Joachims, T. In data. 2002. clickthrough 133–142, using pages engines for search behavior Optimizing reformulation Joachims. user T. Learning Cheng. In P. and auto-completion. Chien, query P. Ke, Y. Jiang, J. recruiting. and seeking job online for 82:6–7,2007. 58(2):166–178, , ae 2111,2013. 1201–1212, pages , rc CIKM Proc. rc SIGIR Proc. rc SIGIR Proc. II Forum SIGIR rc WWW Proc. 31:3 4,2000. 346, – 33(1):337 , ae 5117,2018. 1571–1574, pages , a nesadn o epeuesac nie:A engines: search use people how Understanding na. ¯ ae –,2006. 1–8, pages , ae 514 2012. 95–104, pages , .Ae.Sc n.S.Tech. Sc. Inf. Soc. Amer. J. ae 1–2,2013. 416–427, pages , ae 0–1,2018. 605–614, pages , rc SIGIR Proc. II Forum SIGIR 62:23,2002. 36(2):32–38, , ae 92,2003. 19–28, pages , rc SIGIR Proc. II Forum SIGIR rc of n xb -uiesade-Work and e-Business Exhb. and Conf. Proc. rc CHI Proc. rc SIGIR Proc. ae 4–5,2014. 445–454, pages , nentResearch Internet rc CIKM Proc. II Forum SIGIR rc SIGIR Proc. 31:–2 1999. 33(1):6–12, , ae 0–1,2011. 903–912, pages , n.Retr. Inf. ae 0–1,2007. 407–416, pages , n.Po.&Man. & Proc. Inf. ae 92,2006. 19–26, pages , 21:–7 1998. 32(1):5–17, , 23:2–3,2001. 52(3):226–234, , ae 7918,2012. 1779–1783, pages , rc WSDM Proc. 11:–1 2017. 51(1):4–11, , ae 5–6,2014. 557–566, pages , II rsp eCommerce Wrkshp. SIGIR 31:66,2010. 13(1):46–69, , 54–6 2005. 15:49–66, , rc KDD Proc. .Ae.Sc n.Sc. Inf. Soc. Amer. J. rc SIGIR Proc. rc SIGIR Proc. ae 103–112, pages , 42(1):248–263, , ae 875– pages , rc KDD Proc. pages , pages , Proc. Proc. Proc. , , ,