DataparkSearch Engine 4.50 Reference manual DataparkSearch Engine 4.50: Reference manual Copyright © 2003-2008 by OOO DataPark Copyright © 2001-2003 by Lavtech.com corp. This project is dedicated to Noémie. Table of Contents 1. Introduction............................................................................................................................................1 1.1. DataparkSearch Features.............................................................................................................1 1.2. Where to get DataparkSearch. ....................................................................................................2 1.3. Disclaimer ...................................................................................................................................2 1.4. Authors........................................................................................................................................2 1.4.1. Contributors....................................................................................................................3 2. Installation..............................................................................................................................................4 2.1. SQL database requirements ........................................................................................................4 2.2. Supported operating systems ......................................................................................................4 2.3. Tools required for installation .....................................................................................................5 2.4. Installing DataparkSearch ...........................................................................................................5 2.5. Possible installation problems.....................................................................................................8 2.6. Installation registration................................................................................................................8 3. Indexing ................................................................................................................................................10 3.1. Indexing in general....................................................................................................................10 3.1.1. Configuration................................................................................................................10 3.1.2. Running indexer ..........................................................................................................10 3.1.3. How to create SQL table structure ...............................................................................10 3.1.4. How to drop SQL table structure..................................................................................10 3.1.5. Subsection control ........................................................................................................11 3.1.6. How to clear database...................................................................................................11 3.1.7. Database Statistics........................................................................................................11 3.1.8. Link validation..............................................................................................................12 3.1.9. Parallel indexing...........................................................................................................12 3.2. Supported HTTP response codes ..............................................................................................12 3.3. Content-Encoding support ........................................................................................................14 3.4. Stopwords..................................................................................................................................14 3.4.1. StopwordFile command ..............................................................................................14 3.5. Clones........................................................................................................................................14 3.5.1. DetectClones command...............................................................................................15 3.6. Specifying WEB space to be indexed .......................................................................................15 3.6.1. Server command ..........................................................................................................18 3.6.2. Realm command ..........................................................................................................18 3.6.3. Subnet command .........................................................................................................18 3.6.4. Using different parameter for server and it’s subsections ............................................19 3.6.5. Default indexer behavior .............................................................................................19 3.6.6. Using indexer -f <filename> .........................................................................20 3.6.7. ServerDB, RealmDB, SubnetDB and URLDB commands ......................................20 3.6.8. URL command.............................................................................................................20 3.7. Aliases.......................................................................................................................................20 3.7.1. Alias indexer.conf command..................................................................................20 3.7.2. Different aliases for server parts...................................................................................21 3.7.3. Using aliases in Server commands ..............................................................................21 3.7.4. Using aliases in Realm commands ..............................................................................22 3.7.5. AliasProg command.....................................................................................................22 iv 3.7.6. ReverseAlias command ...............................................................................................23 3.7.7. Alias command in search.htm search template........................................................24 3.8. Servers Table.............................................................................................................................24 3.8.1. Loading servers table....................................................................................................24 3.8.2. Servers table structure ..................................................................................................25 3.8.3. Flushing Servers Table .................................................................................................25 3.9. External parsers.........................................................................................................................25 3.9.1. Supported parser types .................................................................................................25 3.9.2. Setting up parsers .........................................................................................................25 3.9.3. Avoid indexer hang on parser execution.......................................................................26 3.9.4. Pipes in parser’s command line ....................................................................................27 3.9.5. Charsets and parsers .....................................................................................................27 3.9.6. DPS_URL environment variable..................................................................................27 3.9.7. Some third-party parsers...............................................................................................27 3.10. Other commands uses in indexer.conf ..............................................................................30 3.10.1. Include command ......................................................................................................30 3.10.2. DBAddr command.....................................................................................................30 3.10.3. VarDir command .......................................................................................................32 3.10.4. NewsExtensions command........................................................................................32 3.10.5. SyslogFacility command............................................................................................32 3.10.6. Word length commands..............................................................................................32 3.10.7. MaxDocSize command..............................................................................................32 3.10.8. MinDocSize command...............................................................................................33 3.10.9. IndexDocSizeLimit command ..................................................................................33 3.10.10. URLSelectCacheSize command .............................................................................33 3.10.11. URLDumpCacheSize command.............................................................................33 3.10.12. UseCRC32URLId command ..................................................................................33