INFO310 0 Advanced Topics in Model-Based Information Systems Kandidat 102

Oppgaver Oppgavetype Vurdering Status

Introduction Dokument Automatisk poengsum Levert

Plagiarism and Declaration Dokument Automatisk poengsum Levert

1 Essay Filopplasting Manuell poengsum Levert

INFO310 0 Advanced Topics in Model-Based Information Systems

Emnekode INFO310 PDF opprettet 16.11.2016 16:37 Vurderingsform INFO310 Opprettet av Andreas Lothe Opdahl Starttidspunkt: 03.11.2016 14:00 Antall sider 18 Sluttidspunkt: 09.11.2016 14:00 Oppgaver inkludert Nei Sensurfrist Ikke satt Skriv ut automatisk rettede Nei 1 Kandidat 102 Seksjon 1

1 OPPGAVE

Essay

Upload your file here. Maximum one file.

BESVARELSE

Filopplasting Filnavn 9066477_cand-9347761_9157556 Filtype pdf Filstørrelse 1482.886 KB Opplastingstid 09.11.2016 12:42:35

Neste side Besvarelse vedlagt

INFO310 0 Advanced Topics in Model-Based Information Systems Page 2 av 18 Kandidat 102

SEMANTIC TECHNOLOGIES IN SEARCH ENGINES: GOOGLE AND COMPETITORS

MARIO MARTINEZ REQUENA [email protected] Student number: 248948

INFO310 0 Advanced Topics in Model-Based Information Systems Page 3 av 18 Kandidat 102

Index 1. Introduction ...... 2 2. in Google ...... 2 1. How Google works ...... 3 2. ...... 3 3. Knowledge Vault ...... 4 4. Google Hummingbird ...... 5 5. Minor semantic patents ...... 7 1. Identification of semantic units from within a search query ...... 7 2. Inferring User Interests ...... 7 3. Competitors ...... 7 1. Kngine ...... 7 2. Wolfram Alpha ...... 8 3. Comparative Study ...... 9 4. Conclusion and future directions ...... 11 5. Personal opinion and difficulties throw this work ...... 13 6. Referencies ...... 14

INFO310 0 Advanced Topics in Model-Based Information Systems Page 4 av 18 Kandidat 102

1. Introduction

We are all living on the information age. We have digital components all over the place, from our cars to health trackers. We live in a world where we use our smartphones as a part of us. Smartphones has become the first way that the humans have to interact with the digital word as the number “traditional computers” was surpassed by the intelligent phones in 2011 [1]. This is the first thing to understand this new era of the human-to-machine interaction´. Smartphones can be interpreted now as a part of the “new human being”. Now the relation needs to be more user friendly, more organic. Following with this concept and applied to the search engines, that nowadays are a kind of access to the collective memory, they need to get a “questions and answers” dynamic, an “human touch”, and this is in part achieved by introducing in the traditional search engines parts of semantic search. Semantic search, according to the definition provided by [2], wants to improve the search accuracy by analysing the context and intent of the user. Both of this concepts are really important because they can change radically the correct answer to the same search question. In a normal search engine, it would not even be noticed. This is why the world leaders search companies are introducing it on them powerful engines, and in this paper is going to be discussed why and how.

2. Semantic search in Google

According to this rank [3], Google is by far, the first search engine on the internet, so this makes it the first subject of analysis. The engine has been upgraded during the years. During its 18 years, the algorithm behind has been changed many times, and big changes are announced publicly by Google. One of the first semantic big changes that Google has introduced was the Knowledge Graph, on May 16 2012, that aim to give to the users a more “environmental information” and entity recognition about the search that the user performs perform [4]. Apart from the Knowledge Graph, Google perform some changes to the engine itself. The latest ones have been Google Caffeine, designed to return results faster changing the way the crawlers index the pages, Google Panda, that aimed to display the higher quality sites first, Google Penguin, that corrects the errors from the Panda update and penalises the sites that are artificially increasing the rank of their pages and, finally, in September 26, 2013, they announced the biggest in the algorithm change since 2001, Google Hummingbird. This upgrade aims to, apart from the already included synonyms, to understand the context and intent of the user. In other words, they introduced semantic search on their algorithm. Even if Google Hummingbird probably one of the biggest semantic changes to the search engine, have been on Google for a long period of time. They are not as big as Hummingbird, but they all help to create a more semantic Google

INFO310 0 Advanced Topics in Model-Based Information Systems Page 5 av 18 Kandidat 102

1. How Google search engine works

The whole process between a Google search cannot be displayed as a line because half of the process is being realized constantly. This is the crawling and indexing process: Google send crawlers, called Googlebots, surf the internet. They got throw the web by following links from page to page. Apart from traditional links, they also crawl through books, maps, Wikipedia, CIA world factbook, etc. It is a continuous process and because of that, the sites that are frequently updated will get more crawled. A copy of each page is stored on a gigantic Index as well as some data about it. This index also contains images. From the search perspective, a user performs a Search Query. Google analyse, correct and will try to understand the string of characters/voice command/image. This is the part Hummingbird upgraded. Then, based on this analysis, it will pull pages from its index, and Google will rank them based on more than 200 internal parameters. These parameters are almost secret. On this set of filters are included the quality, freshness and number of users that enter on this pages, between others. This is the part where the SEO experts works, trying to perfect details that makes that a page is considered as “good quality” for Google in order to put it up on higher the list. After this ranking, Google will pick relevant pieces to show from the page according to the search and will elaborate the search page itself.

2. Knowledge Graph

One of the main Google statements is the following: “Google’s mission is to organize the world’s information to make it universally accessible and useful.” The introduction of the Knowledge Graph is behind this statement. It is not a remarkable change on the search algorithm, but it is one of the first big approaches that Google has taken to the semantics technologies in its search engine. The knowledge graph is a that contains information about entities and relationships between entities. Knowledge extracts information out of text from Wikipedia, and the CIA World Factbook. Basically, it is not processing you subject of search as a string of characters that need be found on a database, it is treating your query as an entity, a real world object or character, and as an actual object will be related to other entities. The entities can be classified as the way that they are obtained:  Explicit entities: These entities are extracted directly with technologies from the structured mark-up of a webpage.  Implicit entities: These entities are referred or derived from a text on page. In order to get the entities out of the text algorithms for processing the natural language are used. This type of knowledge graph has been used by others companies in different fields:

 Bing is the second search engine and it works really similar to Google, so, in 2013 announced Satori Knowledge Base, with near to 0 information about how it works.

INFO310 0 Advanced Topics in Model-Based Information Systems Page 6 av 18 Kandidat 102

 Another popular search engine such as Yahoo! and Baidu also uses these technologies.  Perhaps one of the most important datasets of this type is the one from LinkedIn. Every user of this social work network is an entity, as well as the jobs, companies, skills, fields of study, degrees, certificates, countries and miscellaneous. The main difference with the one with Google is that is totally based on human contribution, instead of the automatic extraction of information done by Google and Bing. LinkedIn is created structuring and standardizing existing data and it is update every time the information is expanded or updated.

3. Knowledge Vault

After the approach from Google to the large-scale knowledge base with Knowledge Graph, the company realises that they need to extend the horizons behind this idea. The method here to extract the information is more complex than the one used on the knowledge graph. The knowledge graph gets the entities and the relationships focusing on a text-based extraction. This method is really noisy and far from complete. The knowledge graph extract information out of a set of limited sites as Wikipedia, wikidata and so. Basically, if an entity does not exist on this limited environment, it does not exist for the knowledge graph even though it could be on the web. So the knowledge vault, as an expansion of the knowledge graph, is a web-scale probabilistic knowledge base that obtain the entities out of a more diverse set of sources. The knowledge graph (KV) has three principal components:  Extractors: They are self-explanatory. These systems pull information out of the web and classify it depending on the reliability that it could have with a number from 0 to 1. The extraction methods are exposed below: o Text documents: Google uses Neuro-Linguistic Programing (NLP) tools over the files that they want to analyse. Named entity recognition (NER), part of speech tagging (POS), dependency parsing, co-reference resolution (for each document), and entity linkage. After that, they use relation extractors to get pairs of entities out of interesting predicates. With the entities extracted, they search for more sentences in which this pair is mentioned to extract patterns or structures, so in that way they train their system. Then, they search for more sentences with the same structure to extract more entities of the correct type. o HTML trees (DOM): Another way to extract information is to analyse the DOM trees of the normal text pages or other more complex sources where the data can be stored on databases and queried throw HTML forms. Then, to extract the entities they train the system as on the text documents case but for the DOM trees. o HTML tables (TBL): These tables, apart that being used for visual formatting, contain relational information. The difference here compared to the last examples is that the information about relationships is contained on the column header. In order to obtain information out of these tables, they need first to perform named entity linkage. After that, they try to identify the relationship that are being expressed in each column.

INFO310 0 Advanced Topics in Model-Based Information Systems Page 7 av 18 Kandidat 102

o Human annotated pages (ANO): These webpages are the ones where the webmaster has manually added annotations following ontologies from a set of organizations and webpages (schema.org principally). They focused on small set of predicates related to people so they need to map schema.org to the schema for each of these predicates.  Graph-based priors: These systems compute the prior probability of each possible bit of information based on the information already stored information.  Knowledge fusion: This system computes the true probability of the information, based on the relationship between extractors and priors. With these three factors the process of creating the knowledge vault could be explained easily. First, the information is extracted from the whole web by diverse methods and sources. It is a far more complicated process than the one on the knowledge graph, and it is not substituting it, it is expanding it. After the extraction of information based on these four methods, they need to combine the results using some mathematical formulas to balance the information extracted out of each one based on the reliability number obtained on the extraction. This knowledge vault is analysing the information it gets in order to determine how reliable is, as this way of information extraction is not as reliable as the one from the knowledge graph, that obtain the facts just from trusted sites as Wikipedia or the CIA.

4. Google Hummingbird

Google Hummingbird was introduced with two aspects in mind: Improve the speed and accuracy of the Google algorithm and introduce the semantics. One of the founders of Google, Larry Page, said once “The ultimate search engine would understand exactly what you mean and give back exactly what you want”. This update aims to achieve this statement. Hummingbird was released to give to the new Google conversational search a background. Even though Google does not explain Hummingbird deeply, users and experts have done extensive research about what is behind this update and how does it changes results of Google search, specifically SEO. Hummingbird is an evolution for Google in the field of natural language recognition. They have improved their ability to disambiguate entities and concepts based on the research done by the Metaweb Technologies company. This company created Freebase, an online knowledge database. The whole enterprise was purchased by Google in 2010.

INFO310 0 Advanced Topics in Model-Based Information Systems Page 8 av 18 Kandidat 102

The schema above shows how the first important part of the Google Hummingbird works, the query transformation. In order to translate the query on a simpler one, it needs to be rewritten correctly. This is done applying a statistical language approach. To make it, Google will search throw a synonym database, finding substitutes to the important words. It will also search the on the knowledge graph entity to determine if they are entities. The ones that does not have substitutes or own entities are considered as not important words and are skipped. For example, for the query “when was Obama elected?” Google will probably understand just the concepts “When” “Obama” “Elected”, skipping was because it is not relevant for the query. This ability to clarify user intentions and displaying better information was also improved with the use of Knowledge Graph and Knowledge Vault, as with these two tools it can disambiguate a search based on previous ones. This whole update was observed with horror from the SEO experts because they were thinking that was going to change again drastically as with Panda and Penguin the way Google has to organize results. This fact can be easily checked performing a search about hummingbird on Google, and half of the results will be about how Google will change the rank of my webpage. The question is simple to answer: it depends. With Hummingbird Google changed parts of the ranking system software, making it more favourable to the user and less to the “cheating SEO manager”. The ranking is now more determined by the quality and freshness in addition to the usefulness to the user and less by keywords and spam links that traditional SEO strategy have been using. Google Hummingbird was more focused on the intent and context of the query than the actual words used on the query. That means that you should be able to get the same results if you perform two different queries but with the same semantic meaning (for example: “grey cats” and “cats with grey hair”). According to this study [10], the change was noticeable, experiencing an increase of 20% more similar results for different queries with same meaning.

INFO310 0 Advanced Topics in Model-Based Information Systems Page 9 av 18 Kandidat 102

5. Minor semantic patents

1. Identification of semantic units from within a search query

This patent was made by the year 2000 but it is not clear when it was introduced on the engine, showing the long relationship between the Google search engine with semantics. This upgrade on the algorithm is more noticeable on the search engine results page (SERP) for long search queries with more than one subject. When a search query is made, the string of characters is divided into substrings. Each substring is searched and a values is calculated based on the number of documents that contain that substring. Then, according to this values, the semantics units are chosen.

2. Inferring User Interests

The interest of a particular user can be really useful to disambiguate the intentions of a query or provide more precise ads for this user. The way that Google patented to clarify this field is the following: It is a social network approach. The user to analyse has a set of relationships with a group of other users, so a set of interest is determined for our subject. The system will determine a value for every single interest based on the relationships that the subject has with other users and the values for these interest of these “environmental users”. This is a similar approach as the one that Spotify has for recommending music. A simple way to understand why this works is explaining this with songs. If user A and user B like 9 songs, and user B also likes a 10th song, there are high probabilities that the user A is going to like this song.

3. Competitors

Even though is not determined yet how semantic is Google, one of the main goals of this paper, I will compare show here two very different “search engines”. The main competitor in terms of number of user is Bing but the way they work is really similar, so it will not be displayed. The selection of search engines that will be explained are Kngine, a semantic one and Wolfram Alpha, a complex computational. These two are not specifically “search engines”, as both of them tries to provide a single answer to a given query or question, they are “answer engines”, but inside, they work totally different.

1. Kngine

Kngine is a semantic answer engine that, instead of showing a list of links to pages where the answer to your question could be, it interprets the content of the web pages and organizes

INFO310 0 Advanced Topics in Model-Based Information Systems Page 10 av 18 Kandidat 102

the information in knowledge databases. It displays the result in an organized and prepared way, also providing the link to the page where your result has been found and, if it could not elaborate an appropriate answer, it will provide a link to the page where it thinks that your answer could be. Kngine uses a knowledge base called “Knigne live Objects”. This knowledge base has more than 7 million concepts and is used to determine synonyms, relations between concepts, meaning of concepts, document classification, analysis and context-based search. It also has interesting abilities as it could manage ambiguity on the search and, apart of results for traditional queries, it gives answers to questions. In addition, when the user performs a search, can rate the answer in order to make it more precise.

1 Grammatical error handling and answer to question on Kngine 2 Result for questions on Kngine

2. Wolfram Alpha

Wolfram Alpha is a computational knowledge engine or answer engine. The first difference between Wolfram Alpha and Google is that you can perform complex computational operations directly on the search bar: the user can write, for example, an equation, and the site will compute it and show the result or, if the user wants it, a step by step solution. It can handle a wide range of scientific (mathematical, physics, chemist…) operations. Apart from all this calculations, Wolfram has lots of differences in the way that it performs the searches and displays the results with Google: when the user performs a search on WA, it computes answers from a knowledge base of data that has been processed already. This data comes from other sites and books. This is perhaps the most important difference with Google: Wolfram works with curated data, Google works with the web itself, so when a question is made, for Wolfram it already exists, and Google auto generates it. As well as Kngine, this answer engine tries to provide a single answer, a set of data (knowledge) that has been processed instead of a long index of webpages where you can find

INFO310 0 Advanced Topics in Model-Based Information Systems Page 11 av 18 Kandidat 102

the answer. In addition, Wolfram can provide a set of the most important information if a non- question search is made:

3. Comparative Study

In order to show how effective are these engines, and to see how they compare to each other, I will include a study made on 2012. I will also indicate that this study was made before the implementation of Google Hummingbird (2013) but it is the last study that compares these three interesting search engines. The study was made with 77 participants that represent a wide spectrum of internet and search engine users, from frequent users to hardly ever users. Apart from Google, Kngine and Wolfram Alpha, they also evaluated other engines as Hakia, AltaVista and MetaGer. The participants classified 770 semantic and 770 non-semantic search results except for Wolfram Alpha, that will consist just in 77 search results. This tables are the raw data of the result of this study:

INFO310 0 Advanced Topics in Model-Based Information Systems Page 12 av 18 Kandidat 102

Appart from these two tables, is also important to include these four graphics that help to clarify the results:

We are going to focus just on the results concerning to Google, Kngine and WA. On the top left corner, we have the Relevant Results. As it could be easily seen, Google has the higher number of relevant results, followed by Wolfram Alpha (it is not the real number, as it only provides 77 search results. It is scaled to match the rest). Following, in the third position is the Kngine search engine, with a surprisingly higher amount of relevant results on the non-semantic part. The links to content table has little to be commented. Wolfram Alpha does not provide links to webs because the search is performed on a close database instead of the open web as the other two search engines. As it was planned, Google provides the highest amount of links to content. The third table is also interesting because it shows a clear inclination to show not relevant results for the semantic searches even on the semantic engines. That could be caused because a semantic search includes more non-relevant words on the query. Even though, Google again performs the best. The last table is also a clear win for Google.

INFO310 0 Advanced Topics in Model-Based Information Systems Page 13 av 18 Kandidat 102

This last table is shown as a conclusion for this study. It displays the precision of the different search engines, and, as expected, Google is the clear winner again, followed by Kngine and then, Wolfram Alpha, that was evaluated independently.

4. Conclusion and future directions

One of the main goals was to clarify how semantic is Google and it is probably impossible to answer. Google provides very little information about their internal structure and algorithms, so it is really difficult to clarify how much of the result of one search could be caused about semantics or traditional algorithms. Anyway, Google itself says that 90 % of the search results since the implementation of Hummingbird have changed. Talking about the “semantic” parts of the search engine, with the Knowledge Graph and Knowledge Vault Google has given one of the most important steps into the future: understanding the web as a knowledge source, not just like a set of strings of characters. It is really important that the biggest internet enterprise goes in that direction because it is showing an exciting future. The fact that Google started to classify the internet to make wisdom out of it is half of the path. The other half is the way humans have to interact with Google and access to the all the information that it can provide, and, even though this paper is very focused on a more traditional written search query improvement as Hummingbird, it is the base of all the conversational search that Google has been developing these last years. understands easily long queries, questions and even orders.

INFO310 0 Advanced Topics in Model-Based Information Systems Page 14 av 18 Kandidat 102

In the last Google conference, Google has announced a set of products that can prove the direction that Google has taken: Google Home and Google . Google Home is a little product that is aimed to do conversational searches via and execute voice commands and Google Assistant is an already available intelligent personal assistant. Apart from the voice search, it has a chat based side integrated in the instant message system from Google, Google Allo. This variant of the Google Assistant is also interesting because is like chatting throw a normal conversation.

As we can see in the screenshot, it answers to questions with a single answer as we have seen on the Kngine and Wolfram Alpha, but with a huge difference: it shows it in a conversational way, close to how a friend will do it if you ask the same question. It could also memorise what are you talking about, distinguish where the user is, scroll throw its contacts, trace the user geological movements and anterior searches and interest. All of that is to give the awareness to Google of the intent and context of the user.

INFO310 0 Advanced Topics in Model-Based Information Systems Page 15 av 18 Kandidat 102

Talking now about the comparison with the other theoretically more complex search engines, and in general, the whole search engine industry, it is clear that Google is more than one step ahead of the rest. The higher amount of users, the variety of Google services (Youtube, Google +, Gmail, Maps…), the vastness webpage index and crawling, the velocity and reliability of the algorithms and results (90% of users select results from the first page) and the ultimate facts that in this year Google Chrome has become the most used web browser and Google’s Android operative systems has 84% of the smartphone market gives to Google an unfair advantage in the search engine field. It seems nearly unfair to compare the search engine of Google with other minor engines because they have note the capability to go as far on the web as Google, and, if the comparison is done, the results are a clear win to Google even in semantic fields before implementing the biggest semantic update. Long story short, Google has no real competitors on the search engine field.

5. Personal opinion and difficulties throw this work

Now I will explain first the difficulties that I have found working on this work and why I consider it as far from complete but further to be completed. When I decided to write about this topic it seemed like I will be able to find lot of information because how big and how important is Google. I was obviously not right. Google makes a great job as a company to keep really complex stuff simple towards the public, and the information that they give about its internal parts and functioning is one example. On the web you can find many documents talking about the updates from Google and what they say about their changes, but all this information is really superficial. It does not explain what algorithms changes, how are new algorithms and the most that you can find in the best case scenario (with certain exceptions) is an external very limited study analysing what impacts they have on the surface. If you want to obtain technical information you need to surf through the 30.000 patents that Google has, and they are too specific, so you need to explain a ton of them or too technical, including complex mathematical algorithms. I started this work thinking about going deep on the Google engine and clarifying the semantic parts and I think that I partially make it, but not as deep or technical as I thought I would be able to. This article has not become a technical article because the information on which is based is not technical. With the last paragraph on mind, I will give my final answer to the question “Is Google a semantic engine?” It is more than that. That is the complete answer. It is clear that Google has

INFO310 0 Advanced Topics in Model-Based Information Systems Page 16 av 18 Kandidat 102

made a huge effort to include semantics technologies on its search engine, but not with the objective on mind to make Google semantic. Instead, with a more pragmatic point of view, has been picking parts from different technologies as semantics, machine learning, deep learning and many more to make Google more complete than the rest of the competitors. The last part of this paper is my personal opinion. Google is used because is the best and more complete search engine and because of tradition. The first part of this statement has been explained throw this article and the second one is the one that I will explain now. The human being is a creature of habit, it feels comfortable when they know the environment and the repetition of task make it feel better, safer. This is the sentimental part that explains why once we get use to one thing, it is really complicated to change your mind about moving to other similar but yet different service to do the same thing as you have been doing. Google have made it really well creating a “closed environment” where you can stay comfortable. The google search engine drives you throw the web with near to 0 difficulties, you can expend hours entertained in Youtube, maps can guide you back to your home, you can receive emails with Gmail. It is a cloud, and Google seems to be improving this with the addition of many future components as the assistant, Google Home, Google Assistant and Google Daydream. It seems like as the internet is penetrating more and more on our lives, Google is making it as well, expanding their domain even to areas where we have not expected ever. You cannot understand the existence and importance of the internet without Google, and you cannot visualize the future of the web and in general, the relationships of the human with the machine without Google, and the best part is that semantics technologies are the base of the bridge between both.

6. Referencies

1. Canalys. “Smart phones overtake client PCs in 2011.” 2011. https://www.canalys.com/newsroom/smart-phones-overtake-client-pcs-2011 2. Wikipedia. “Semantic Search” https://en.wikipedia.org/wiki/Semantic_search 3. eBizMBA. “Top 15 Most Popular Search Engines”. November 2016 http://www.ebizmba.com/articles/search-engines 4. Heiko Paulheim. “Knowledge Graph Refinement: A Survey of Approaches and Evaluation Methods”. http://www.semantic-web-journal.net/system/files/swj1167.pdf 5. Wikipedia, “Google search” https://en.wikipedia.org/wiki/Google_Search 6. “Google Algorithm Changes history” https://moz.com/google-algorithm-change 7. Sergio Redondo “SEO 101: What is Semantic Search and Why Should I Care?” November 2014, https://www.searchenginejournal.com/seo-101-semantic-search-care/119760/ 8. David Amerland, “Google Semantic Search” Book 9. Danny Sullivan “FAQ: All About the New Google “Hummingbird” Algorithm” September 2013 http://searchengineland.com/google-hummingbird-172816 10. Searchmetrics, “Google‘s Hummingbird Algorithm: The Entity Search Revolution” http://pages.searchmetrics.com/rs/searchmetricsgmbh/images/Searchmetrics_Hummingbir d_Study_2014.pdf 11. Sanjana Singh, Surbhi Bhardwaj, Shivansh Mudgil “Humming bird search algorithm” http://www.ijrdo.org/International-Journal-of-Research-&-Development-Organisation- pdf/International-Journal-Of-Computer-Science-Engineering/Journal-Of-Computer- Science-Engg-March-15/18.pdf

INFO310 0 Advanced Topics in Model-Based Information Systems Page 17 av 18 Kandidat 102

12. Xin Luna Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy , Thomas Strohmann, Shaohua Sun, Wei Zhang “Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion”. https://www.cs.ubc.ca/~murphyk/Papers/kv-kdd14.pdf 13. Shumeet Baluja, Yushi Jing, Dandapani Sivakumar and Jay Yagnik. “Inferring User Interests” (2008). US11742995. Current Assignee: Google Inc. https://patents.google.com/patent/US20080275861A1/en?q=graph&assignee=Google+Inc &page=1 14. Krishna Bharat Sanjay Ghemawat, Urs Hoelzle “Identification of semantic units from within a search query” US7249121B1. Current Assignee: Google Inc. https://patents.google.com/patent/US7249121B1/en?q=semantic+search&assignee=Googl e+Inc 15. Sander Dieleman. “Recommending music on Spotify with deep learning” http://benanne.github.io/2014/08/05/spotify-cnns.html 16. Hasan Girit, Robert Eberhard, Bernd Michelberger, and Bela Mutschler “On the Precision of Search Engines: Results from a Controlled Experiment” http://dbis.eprints.uni- ulm.de/798/1/GiEbMiMu12.pdf 17. Theodore Gray, “The Secret behind the Computational Engine in Wolfram|Alpha” 2009 http://blog.wolframalpha.com/2009/05/01/the-secret-behind-the-computational-engine- in-/ 18. Mike Murphy, “If you need a smart assistant in your life, get a Google Home” http://qz.com/827100/should-you-get-a-google-goog-home-over-an--amzn-echo/ 19. Elyse Betters, “What is Google Assistant, how does it work, and when can you use it?” http://www.pocket-lint.com/news/137722-what-is-google-assistant-how-does-it-work- and-when-can-you-use-it 20. Eric Sharp, “The First Page of Google, by the Numbers” http://www.protofuse.com/blog/first-page-of-google-by-the-numbers/ 21. IDC Research, Inc. “Smartphone OS Market Share, 2016 Q2” http://www.idc.com/prodserv/smartphone-os-market-share.jsp 22. Darren Orf, “Google Chrome Is Now the Most Popular Web Browser”. May 2016 http://gizmodo.com/google-chrome-is-now-the-most-popular-web-browser- 1774266161

INFO310 0 Advanced Topics in Model-Based Information Systems Page 18 av 18