Decentralised Web Search

Web Search Engine Software For Everyone: We can remove dependency from (a) large search engine provider

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine For Everyone http://yacy.net Topics Tech Demo Dev

•Search Engine Technology how large-scale search engines are made available for everyone using peer-to-peer technology

•Demonstration: what you can do in just ﬁve minutes: installation, crawling, searching, monitoring, scheduling

•System Components and Development: Details about a search appliance components like scheduler, document parser, administration and visualization. Easy integration into a web page. APIs for external index queries and external index feeding components.

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Search Engine Components Tech Demo Dev

Retrieval, Indexing, Storage and Search Components

ﬁltering,

Depth = 0 Start-URL parsing Text Analysis Indexing

links words Depth = 1

Double Link Stop words

Crawler Check Check Depth = 2 URL Reverse Crawl Stack Word Index

Word URL References

YaCy has an Database integrated NoSQL ranking, Database. The veriﬁcation, database stores a visualisation Reverse Word Index, Metadata

Search and the source

Interface documents.

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Large Search Cluster: Model Tech Demo Dev

Efﬁcient search engines are constructed using a matrix of many small search engines

Search Engine Cluster

Search Search Search Search Search Engine Engine Engine Engine Engine

vertical scaling:vertical per second more queries horizontal scaling: more documents

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Large Search Cluster in Data Center Tech Demo Dev

Usually such search engine clusters are hosted by one organization in a data center

Search Search Search Search Search Engine Engine Engine Engine Engine

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Large Search Cluster: Decentralised Tech Demo Dev

Imagine you can take the software outside and connect peers decentralised

Search Search Search Search Search Engine Engine Engine Engine Engine

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Decentralised Search with YaCy 1/3 Tech Demo Dev

YaCy is a search engine appliance that can be used either in a data center or as a decentralised network of private peers

Peer Peer Peer Peer Peer

How can a search matrix be distributed? The peers are ordered using an ordering on peer hashes. The hash-ordering is closed at the end and the resulting network can be drawn as a circle...

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Decentralised Search with YaCy 2/3 Tech Demo Dev

A ,Folded‘ Search Matrix

Peer Peer Peer

Peer Peer This peer (as an A peer which searches example) fetches some Peer Peer information can access Web pages and directly peers holding distributes index Peer Peer the corresponding fragments to other index peers. Peer Peer

Peer Peer

Peer Peer DHT-Store Peer DHT-Read

YaCy peers store index fragments according to a ‘folded‘ ordering on word-hashes and url-hashes in a distributed hash table (DHT). The index is distributed redundantly to save the index when some peers are not available. The redundancy also helps to increase search performance.

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Decentralised Search with YaCy 3/3 Tech Demo Dev

The ,default‘ YaCy Search Engine Network

Peer Types: Junior Senior Principal DHT-Store behind ﬁrewall or router has open server port publishes seed-lists DHT-Read

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net YaCy Search Cluster in a Data Center Tech Demo Dev

http:// sciencenet.fzk.de 300 million documents ,Sciencenet‘: Search Engine for scientiﬁc content in the Karlsruhe Institute of Technology: 34 computers running YaCy in it‘s own network

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Decentralised Search for Everyone Tech Demo Dev

Search Engine @Home > 1 Billion Documents

People run they own YaCy search peer at home and create independent search for everyone

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Beneﬁts Tech Demo Dev

Impact of running your own search engine:

become independent from large search engine operators keep company secrets search tracks can reveal industrial research targets your personal relevance you can create a ranking method for your personal needs same rights for all people everyone can run a search engine

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Demo: Users Tech Demo Dev

linuxtag.org

linux-club.de

geoclub.de

fsfe.org

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Use Cases Tech Demo Dev

•Decentralised Peer-to-Peer Web Search search engines for everyone •High-Performance Search Clusters generic search portals for any need

•Internet Search Portal for a project combining wikis, blogs, forums and portal pages •Alert-Service for News using RSS create a News-Feed using recent search results for a speciﬁc topic

•Intranet Search Appliance search in local web servers and ﬁle shares

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Search Interface Tech Demo Dev

API for search results is RSS (Opensearch) and JSON SRU

Facets: Domains, Authors

every link is veriﬁed before it is displayed: the content is loaded, parsed and used for a search snippet generation

APIs Opensearch (search results with RSS), JSON, AJAX tools Standards Tools search widget, ready-to-use code snippets to embed search everywhere

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Document Retrieval and Parser Tech Demo Dev

A Search Engine should support people in the search for documents in unstructured formats: this needs a kind of ‘understanding‘ of content

Connection Parsing Interpretation load and crawl from: read document formats: find metadata (headline, author, date, locations) HTTP, HTTPS, FTP, filesystem, HTML, XHTML, RSS, RDF, SMB-shares XHTML+RDFa, FOAF, vCard, find links of different kind Flash, PDF, PS, Word, Excel, (text, images, movies etc.) Import from: Visio, Powerpoint, Dublin Core / XML files, OpenOffice, RTF, csv, gzip, zip, store statistical data for OAI-PMH, wikimedia dumps, tar, rar, bzip2, 7zip, images search suggestions SQL databases (EXIF), torrent files

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Search Result Ranking Tech Demo Dev

a prototype discussion do you use the same about ranking no. PR is difﬁcult ranking as G**gle? and sometimes useless (i.e. in intranets) then you cannot be better? that‘s what we have many ranking criteria lucene has and users can mix them. but is this better? what is ‘better‘? G**gle deﬁnes ‘better‘ as: ‘most people like it‘ I have an idea: .... suddenly people think about their personal relevance requirements..... then what is the similar to best ranking? G**gle PR do experiments! If you run your own search engine, then you may need your own ranking. Different contents may need different rankings. every peer? when doing a remote in YaCy, you can search, the remote peer uses your own combine many ranking too! weighted attributes FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Parts of a Search Appliance Tech Demo Dev

Search Engine Data Visualisation

retrieval, indexing, storage and search components index creation process, system load, link structure, p2p net conﬁguration Scheduler and Steering Database Administration

automatic scheduled re-indexing and crawl queues, robots.txt, rss feeds, scheduler back-up of search appliance set-up data, p2p connections, network messages

FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net Search Interface Integration Tech Demo Dev

Code Snippet Example #1: a search window in an iframe How to integrate a </p><p>Code Snippet Example #2: a search box (points to new page) Code Snippet #2 looks like: <form method="get" accept-charset="UTF-8" action="http://141.52.175.43:8080/yacysearch.html"> <div> <div>MySearch</div> <input type="text" name="query" value="" maxlength="80" /> <input type="hidden" name="verify" value="true" /> <input type="hidden" name="maximumRecords" value="10" /> The YaCy administration interface <input type="hidden" name="meanCount" value="5" /> offers more code snippets. An <input type="hidden" name="resource" value="local" /> <input type="hidden" name="urlmaskfilter" value=".*" /> example from <input type="hidden" name="prefermaskfilter" value="" /> /ConfigSearchBox.html <input type="hidden" name="display" value="2" /> <input type="hidden" name="nav" value="all" /> looks like: <input type="submit" name="Enter" value="Search" /> </div> </form></p><p> your YaCy peer provides help pages with code snippets for an easy integration!</p><p>FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net External Index Retrieval Tech Demo Dev</p><p>> curl http://localhost:8080/yacysearch.rss?query=foaf&maximumRecords=10 <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type='text/xsl' href='/yacysearch.xsl' version='1.0'?> <rss version="2.0" xmlns:yacy="http://www.yacy.net/" How to get Opensearch/ xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" JSON Search Results:  - do a normal web search in YaCy <item> - replace the ‘html‘ extension of <title>Friend of a Friend (FOAF) project</title> <link>http://www.foaf-project.org/</link> the result page URL with ‘rss‘ <pubDate>Fri, 23 May 2008 02:00:00 +0200</pubDate> - for json, replace the ‘html‘ </item> extension with ‘json‘ <item> <title>FOAF - Wikipedia</title> <link>http://de.wikipedia.org/wiki/FOAF</link> Standards: <pubDate>Tue, 08 Jan 2008 01:00:00 +0100</pubDate> The YaCy-internal Dublin Core </item> Metadata Format fits into the RSS <item> format for search result data in <link>http://microformats.org/wiki/xfn-to-foaf</link> Opensearch standard very well. <pubDate>Fri, 09 May 2008 02:00:00 +0200</pubDate> </item> </rss> If wanted, also JSON can be used as export format. Opensearch Standard: http://www.opensearch.org SRU Standard for Queries: http://www.loc.gov/standards/sru/specs/search-retrieve.html</p><p>FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net External Index Feeding Tech Demo Dev</p><p>Standards: <?xml version="1.0" encoding="utf-8"?> YaCy can import standard  Dublin Core Metadata <surrogates xmlns:dc="http://purl.org/dc/elements/1.1/"> XML files as input for indexing <record> <dc:title><![CDATA[Alan Smithee]]></dc:title> <dc:identifier>http://de.wikipedia.org/wiki/Alan_Smithee</dc:identifier> <dc:description> <![CDATA['''Alan Smithee''' ist ein Anagramm von „The Alias Men“.]]> </dc:description> <dc:language>de</dc:language> <dc:date>2009-04-14T00:00:00Z</dc:date>  </record> </surrogates></p><p>How to import Dublin Core Files: just place the xml files into a hand-over directory at DATA/SURROGATES/in/</p><p>The Dublin Core XML File Standard: http://dublincore.org/documents/dc-xml-guidelines/</p><p>FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net License: GPL Installation <a href="/tags/Free_software/" rel="tag">Free Software</a> Tech Demo Dev</p><p>•Download from http://yacy.net</p><p>YaCy for Windows YaCy for Mac YaCy for <a href="/tags/Debian/" rel="tag">Debian</a> YaCy for Linux / generic (tar.gz)</p><p>•Just Extract the Package, then Start the Start-Script There are simple installers for Windows, Mac and a debian release, but it is easy to just install the generic release because it contains everything that is needed. •Administration using the Web Interface YaCy is a Web Application. The administration can be done completely using the built-in web interface with your web browser. Just open http://localhost:8080 The main configuration is done when you select your use case (Distributed P2P Web Search, Portal Search, Intranet Search) after just two clicks. •Support We have a web forum: http://forum.yacy.de Some information can be found at the wiki: http://wiki.yacy.de ...or contact me: mc@yacy.net</p><p>FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine for everyone http://yacy.net what you can do</p><p>• learn about search engine technology and teach other people • create your own search portal • be creative! -- we listen to your ideas • help -- make a translation of the administration interface!</p><p>FOSS ASIA - Ho Chi Minh City, Vietnam 2010 Michael Christen Web Search Engine For Everyone http://yacy.net</p> </div> </article> </div> </div> </div> <script type="text/javascript" async crossorigin="anonymous" src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-8519364510543070"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.6.1/jquery.min.js" crossorigin="anonymous" referrerpolicy="no-referrer"></script> <script> var docId = 'f88f5ac3dfd7c3349d67eaca97f885ab'; var endPage = 1; var totalPage = 23; var pfLoading = false; window.addEventListener('scroll', function () { if (pfLoading) return; var $now = $('.article-imgview .pf').eq(endPage - 1); if (document.documentElement.scrollTop + $(window).height() > $now.offset().top) { pfLoading = true; endPage++; if (endPage > totalPage) return; var imgEle = new Image(); var imgsrc = "//data.docslib.org/img/f88f5ac3dfd7c3349d67eaca97f885ab-" + endPage + (endPage > 3 ? ".jpg" : ".webp"); imgEle.src = imgsrc; var $imgLoad = $('<div class="pf" id="pf' + endPage + '"><img src="/loading.gif"></div>'); $('.article-imgview').append($imgLoad); imgEle.addEventListener('load', function () { $imgLoad.find('img').attr('src', imgsrc); pfLoading = false }); if (endPage < 7) { adcall('pf' + endPage); } } }, { passive: true }); </script> <script> var sc_project = 11552861; var sc_invisible = 1; var sc_security = "b956b151"; </script> <script src="https://www.statcounter.com/counter/counter.js" async></script> </html>