DOCSLIB.ORG

Instantly Identify and Triage Many Languages

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Instantly Identify and Triage Many Languages Rosette® BIG TEXT ANALYTICS Language Identifier RLI RLI ROSETTE Identify languages and encodings Language Identifier Sortedwww.basistech.com Languages [email protected] +1 617-386-2090 Base Linguistics RBL RBL ROSETTE Search many languages with high accuracy InstantlyBase Linguistics identify and triageBetter Search Entity Extractor REX REX ROSETTE Tag names of people, places, and organizations manyEntity languages Extractor within largeTagged Entities English Primary Language Entity Resolver 8% RES voRESlumes ROSETTE of text. French Make real-world connections in your data Chinese Entity Resolver Chinese RealPrimary Scrip Identitiest 即时识别和处理大量多语言文本。 22% Arabic 39% Latin Identifiez et triez instantanément plusieurs French French Name Indexer English RNI languesRNI à travers ROSETTE de nombreux textes. Match names between many variations Name Indexer Matched Names %31 اﻟﺘﺤﺪﻳﺪ واﻟﺘﺼﻨﻴﻒ اﻟﻔﻮري ﻟﻠﻌﺪﻳﺪ ﻣﻦ اﻟﻠﻐﺎت ﺿﻤﻦ ﻛﻤﻴﺎت ﻛﺒﻴﺮة ﻣﻦ اﻟﻨﺼﻮص. Arabic Name Translator RNT RNT ROSETTE Translate foreign names into English Name Translator Translated Names Identify languages and Supported Categorizer Languages transform ROSETTE encodings 55 RCA Categorize Everything In Sight RCA Rosette® LanguageCategorizer Identifier (RLI) analyzes text from a few words to whole KEY FEATURES Sorted Content documents, to detect the languages and character encoding with speed and very high accuracy. Automatic language identification is the necessary first - Simple API Sentiment Analyzer step for applications that categorize, search, process, and store text in many - Fast and scalable ROSETTE - Industrial-strength support RSA languages.RSA Individual documents may be routed to language specialists, or sent Detect The Sentiments Of Your Text - Easy installation into language-specificSentiment analysis pipelines Analyzer (such as Rosette Base Linguistics) to Actionable Insights - Flexible and customizable improve the quality of search results. - Java or C++ - Unix, Linux, Mac, or Windows For applications that analyze tweets, search keywords, and other short text, - Component of the Rosette SDK RLI offers market-leading accuracy for language detection given 1-3 words (<20 bytes) up to a full sentence. RLI achieves its incredible accuracy through the use of proprietary algorithms with information-rich language profiles derived from statistical analysis, in addition to language-specific methods for short text language detection. Basis Technology continually improves the Rosette product family with language additions, feature updates, and the latest innovations from the academic world. Select Customers Start using RLI today StumbleUpon Try our free product evaluation www.basistech.com Rosette® BIG TEXT ANALYTICS Language Identifier RLI RLI ROSETTE Identify languages and encodings Language Identifier Sorted Languages Rosette® Base Linguistics BIG TEXT ANALYTICS RBL RBL ROSETTE Search many languages with high accuracy IDENTIFICATIONBase LinguisticsFEATURES LANGUAGELanguage BOUNDARY Identifier LOCATORBetter SearchENCODING CONVERSION RLI RLI ROSETTE - Identifies the primary or dominant language Identify languages and encodings Entity Extractor J'ai été surprise par cette surprise. Vice President Language Identifier Sorted Languages REX REXof a document ROSETTE Tag names of people, places, and organizations Biden spoke about this in Munich. El carpintero Although modern text encoding standards, - Identifies Entitythe language Extractor scripts within the Tagged Entities prensa los bordes de la placa decorativa. Proper such as XML, mandate the use of Unicode, document, such as Latin and Cyrillic wound careBase management Linguistics prevents die Geige gibt many existing applications, documents, ROSETTE Entity Resolver - Determines the languages and their RBL websites, and data streams use “legacyRBL einen schoenen Search Laut von many sich. languages with high accuracy RES RESpercentagesROSETTE within multilingual documents encodings,” such as ASCII, ISO 8859-1, Shift-JIS, Better Search Make real-world connections in your data Base Linguistics Entity Resolver Real Identitiesand many others. - Works with texts that have been ENGLISHFRENCH GERMAN SPANISH transliterated, such as Arabic chat that is Rosette accurately converts large collections written in the Latin script Digital text is often composed of multiple Name Indexer Entity Extractor of text with these legacy encodings into a ROSETTE RNI RNI languages within the same document, ROSETTE Match names between many variations - Accurate with short strings—from 1-3 wordsREX single, uniform format in the UnicodeREX standard. presenting Tag a challenge names to ofcomputers people,Matched and places, Names and organizations (<20 bytes)Name to a full sentence Indexer to enable full humans alike. RLI enriches the text with start This converted text can then be used in any Entity Extractor Tagged Entities analysis of search queries, tweets, image and end markers for each language placed language, which eliminates data corruption and captions, metadata, news headlines, email Name Translator within multilingual documents—even if all the other problems due to incompatible code. subject lines, ROSETTE and more. RNT RNT languages are written in the same script— Translate foreign names into English Entity Resolver Name Translator Translated Names RESsuch as English, French, German, or Italian. RES ROSETTE Boundaries Make of each real-world writing system connections are also in your data Entity Resolver Real Identities Categorizer detected, such as Latin, Cyrillic, Japanese kana, RCA RCA ROSETTE or Chinese hanzi. Categorize Everything In Sight Categorizer Sorted Content Name Indexer LANGUAGE AND ENCODING COMPATIBILITYRNI RNI ROSETTE Sentiment Analyzer Match names between many variations Name Indexer Matched Names ROSETTE RSA Detect The Sentiments Of Your Text RSA Albanian — ISO-8859-1, Windows-1252 Lithuanian — ISO-8859-13, Windows-1257 SentimentLanguage/ Analyzer Arabic — ISO-8859-6, Windows-720,Actionable Macedonian Insights — ISO-8859-5, Windows-1251 Windows-1256 Malay — ISO-8859-1, Windows-1252 Encoding Pairs Arabic (transliterated)Name —Translator ISO-8859-1, Malayalam — ISCII-Malayalam 188 ROSETTE RNT Windows-1252, Windows-1256 Norwegian — ISO-8859-1, Windows-1252RNT Translate foreign names into English Bengali — ISCII-Bengali Pashto — ISO-8859-6, Windows-1256 Name Translator Translated Names Bulgarian — ISO-8859-5, Windows-1251, KOI8-R Pashto (transliterated) — ISO-8859-1, Languages Catalan — ISO-8859-1, Windows-1252 Windows-1252 Chinese, Simplified— GB-2312, GB-18030, Persian — ISO-8859-6, Windows-1256 with Unicode HZ-GB-2312,Categorizer ISO-2022-CN Persian (transliterated) — ISO-8859-1, 55 ROSETTE RCAChinese, TraditionalCategorize — Big5, Everything Big5-HKSCS In Sight Windows-1252, Windows-1256 RCA Croatian — Windows-1250 Polish — ISO-8859-2, Windows-1250 Categorizer Sorted Content Czech — ISO-8859-2, Windows-1250 Portuguese — ISO-8859-1, Windows-1252 Latin Script Danish — ISO-8859-1, Windows-1252 Romanian — ISO-8859-2, Windows-1250 Dutch — ISO-8859-1,Sentiment Windows-1252 Analyzer Russian — ISO-8859-5, Windows-1251, KOI8-R, Variants English — ISO-8859-1, Windows-1252 IBM-866, Mac Cyrillic ROSETTE 7 RSAEstonian —Detect ISO-8859-13, The Windows-1257Sentiments Of YourSerbian Text — ISO-8859-5, Windows-1251RSA (Transliterations) Finnish — ISO-8859-1, Windows-1252 Serbian (transliterated) — ISO-8859-2, Sentiment Analyzer Actionable Insights French — ISO-8859-1, Windows-1252 Windows-1250 German — ISO-8859-1, Windows-1252 Slovak — Windows-1250 Legacy Greek — ISO-8859-7, Windows-1253 Slovenian — Windows-1250 Gujarati — ISCII-Gujarati Somali — ISO-8859-1, Windows-1252 44 Encodings Hebrew — ISO-8859-8, Windows-1255 Spanish — ISO-8859-1, Windows-1252 Hindi — ISCII-Hindi Swedish — ISO-8859-1, Windows-1252 Hungarian — ISO-8859-2, Windows-1250 Tagalog — ISO-8859-1, Windows-1252 Icelandic — ISO-8859-1, Windows-1252 Tamil — ISCII-Tamil Indonesian — ISO-8859-1, Windows-1252 Telugu — ISCII-Telugu Compatibility Italian — ISO-8859-1, Windows-1252 Thai — Windows-874 Japanese — EUC-JP, ISO-2022-JP, Shift-JIS, Turkish — ISO-8859-9, Windows-1254 Code Base Platform Support Shift-JIS-2004 (JIS X 0213) Ukrainian — ISO-8859-5, Windows-1251, KOI8-R Kannada — ISCII-Kannada Urdu — ISO-8859-6, Windows-1256 Korean — EUC-KR, ISO-2022-KR Urdu (transliterated) — ISO-8859-1, Kurdish — Windows-1256 Windows-1252 Kurdish (transliterated) — ISO-8859-1, Uzbek — ISO-8859-5, Windows-1251, KOI8-R Windows-1252, Windows-1256 Uzbek (transliterated) — Windows-1251 Latvian — ISO-8859-13, Windows-1257 Vietnamese — TCVN, VIQR, VISCII, VNI, VPS © 2015 Basis Technology Corporation. “Basis Technology HEADQUARTERS FEDERAL WEST COAST EUROPE ASIA Corporation” , “Rosette”, and “Highlight” are registered trademarks One Alewife Center 2553 Dulles View Dr. 1700 Montgomery St. Furzeground Way 9-6 Nibancho, of Basis Technology Corporation. “Big Text Analytics” is a trademark Cambridge, MA Suite 450 San Francisco, CA Middlesex UB11 1BD, Chiyoda-ku of Basis Technology Corporation. All other trademarks, service marks, and logos used in this document are the property of their respective 02140 Herndon, VA 94111 UK Tokyo 102-0084, owners. (2015-06-29-RLI) 20171 Japan.
Load more

Recommended publications
  • Consonant Characters and Inherent Vowels
    Global Design: Characters, Language, and More Richard Ishida W3C Internationalization Activity Lead Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 1 Getting more information W3C Internationalization Activity http://www.w3.org/International/ Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 2 Outline Character encoding: What's that all about? Characters: What do I need to do? Characters: Using escapes Language: Two types of declaration Language: The new language tag values Text size Navigating to localized pages Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 3 Character encoding Character encoding: What's that all about? Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 4 Character encoding The Enigma Photo by David Blaikie Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 5 Character encoding Berber 4,000 BC Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 6 Character encoding Tifinagh http://www.dailymotion.com/video/x1rh6m_tifinagh_creation Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 7 Character encoding Character set Character set ⴰ ⴱ ⴲ ⴳ ⴴ ⴵ ⴶ ⴷ ⴸ ⴹ ⴺ ⴻ ⴼ ⴽ ⴾ ⴿ ⵀ ⵁ ⵂ ⵃ ⵄ ⵅ ⵆ ⵇ ⵈ ⵉ ⵊ ⵋ ⵌ ⵍ ⵎ ⵏ ⵐ ⵑ ⵒ ⵓ ⵔ ⵕ ⵖ ⵗ ⵘ ⵙ ⵚ ⵛ ⵜ ⵝ ⵞ ⵟ ⵠ ⵢ ⵣ ⵤ ⵥ ⵯ Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 8 Character encoding Coded character set 0 1 2 3 0 1 Coded character set 2 3 4 5 6 7 8 9 33 (hexadecimal) A B 52 (decimal) C D E F Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 9 Character encoding Code pages ASCII Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 10 Character encoding Code pages ISO 8859-1 (Latin 1) Western Europe ç (E7) Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 11 Character encoding Code pages ISO 8859-7 Greek η (E7) Copyright © 2005 W3C (MIT, ERCIM, Keio) slide 12 Character encoding Double-byte characters Standard Country No.
    [Show full text]
  • HAIL: an Algorithm for the Hardware Accelerated Identification of Languages, Master's Thesis, May 2006
    Washington University in St. Louis Washington University Open Scholarship All Computer Science and Engineering Research Computer Science and Engineering Report Number: WUCSE-2006-36 2006-01-01 HAIL: An Algorithm for the Hardware Accelerated Identification of Languages, Master's Thesis, May 2006 Charles M. Kastner This thesis examines in detail the Hardware-Accelerated Identification of Languages (HAIL) project. The goal of HAIL is to provide an accurate means to identify the language and encoding used in streaming content, such as documents passed over a high-speed network. HAIL has been implemented on the Field-programmable Port eXtender (FPX), an open hardware platform developed at Washington University in St. Louis. HAIL can accurately identify the primary languages and encodings used in text at rates much higher than what can be achieved by software algorithms running on microprocessors. Follow this and additional works at: https://openscholarship.wustl.edu/cse_research Part of the Computer Engineering Commons, and the Computer Sciences Commons Recommended Citation Kastner, Charles M., " HAIL: An Algorithm for the Hardware Accelerated Identification of Languages, Master's Thesis, May 2006" Report Number: WUCSE-2006-36 (2006). All Computer Science and Engineering Research. https://openscholarship.wustl.edu/cse_research/187 Department of Computer Science & Engineering - Washington University in St. Louis Campus Box 1045 - St. Louis, MO - 63130 - ph: (314) 935-6160. Department of Computer Science & Engineering 2006-36 HAIL: An Algorithm for the Hardware Accelerated Identification of Languages, Master's Thesis, May 2006 Authors: Charles M. Kastner Corresponding Author: [email protected] Web Page: http://www.arl.wustl.edu/projects/fpx/reconfig.htm Abstract: This thesis examines in detail the Hardware-Accelerated Identification of Languages (HAIL) project.
    [Show full text]
  • Vntex — Typesetting Vietnamese Hàn Thế Thành Reinhard Kotucha
    VnTEX — Typesetting Vietnamese Hàn Thế Thành Reinhard Kotucha Abstract VnTEX is an extension to Donald Knuth’s TEX typesetting system which provides support for typesetting Vietnamese. The primary site of VnTEX is http://vntex.sf.net. 1 Where to get Help The current maintainers of VnTEX are: I Hàn Thế Thành [email protected] I Reinhard Kotucha [email protected] I Werner Lemberg [email protected] There is a mailing list (very low traffic) for questions about VnTEX and typesetting Vietnamese. To subscribe to the list, visit: http://lists.sourceforge.net/lists/listinfo/vntex-users There is also a Wiki: http://vntex.info 2 Related Documents The following files are part of the VnTEX distribution I Hàn Thế Thành, Hỗ trợ tiếng Việt cho TEX I Hàn Thế Thành, Minimal steps to typeset Vietnamese I Hàn Thế Thành và Thái Phú Khánh Hòa, Dùng font với VnTEX The following files are not part of VnTEX but might be part of the TEX distribution you are using. I The American Mathematical Society, Hướng dẫn sử dụng gói amsmath, http://ctan.org/tex-archive/info/amslatex/vietnamese/amsldoc-vi.pdf http://ctan.org/tex-archive/info/amslatex/vietnamese/amsldoc-print-vi.pdf I H. Partl, E. Schlegl, I. Hyna, T. Oetiker, Một tài liệu ngắn gọn giới thiệu về LATEX 2", Translated by Nguyễn Tân Khoa. http://ctan.org/tex-archive/info/lshort/vietnamese/lshort-vi.pdf I Wolfgang May, Andreas Schlechte, Mở rộng môi trường định lý. Translated by Huỳnh Kỳ Anh. http://ctan.org/tex-archive/info/translations/vn/ntheorem-doc-vn.pdf 1 3 Typesetting Vietnamese In order to typeset Vietnamese, you need a text editor which supports Vietnamese.
    [Show full text]
  • Legacy Character Sets & Encodings
    Legacy & Not-So-Legacy Character Sets & Encodings Ken Lunde CJKV Type Development Adobe Systems Incorporated bc ftp://ftp.oreilly.com/pub/examples/nutshell/cjkv/unicode/iuc15-tb1-slides.pdf Tutorial Overview dc • What is a character set? What is an encoding? • How are character sets and encodings different? • Legacy character sets. • Non-legacy character sets. • Legacy encodings. • How does Unicode fit it? • Code conversion issues. • Disclaimer: The focus of this tutorial is primarily on Asian (CJKV) issues, which tend to be complex from a character set and encoding standpoint. 15th International Unicode Conference Copyright © 1999 Adobe Systems Incorporated Terminology & Abbreviations dc • GB (China) — Stands for “Guo Biao” (国标 guóbiâo ). — Short for “Guojia Biaozhun” (国家标准 guójiâ biâozhün). — Means “National Standard.” • GB/T (China) — “T” stands for “Tui” (推 tuî ). — Short for “Tuijian” (推荐 tuîjiàn ). — “T” means “Recommended.” • CNS (Taiwan) — 中國國家標準 ( zhôngguó guójiâ biâozhün) in Chinese. — Abbreviation for “Chinese National Standard.” 15th International Unicode Conference Copyright © 1999 Adobe Systems Incorporated Terminology & Abbreviations (Cont’d) dc • GCCS (Hong Kong) — Abbreviation for “Government Chinese Character Set.” • JIS (Japan) — 日本工業規格 ( nihon kôgyô kikaku) in Japanese. — Abbreviation for “Japanese Industrial Standard.” — 〄 • KS (Korea) — 한국 공업 규격 (韓國工業規格 hangug gongeob gyugyeog) in Korean. — Abbreviation for “Korean Standard.” — ㉿ — Designation change from “C” to “X” on August 20, 1997. 15th International Unicode Conference Copyright © 1999 Adobe Systems Incorporated Terminology & Abbreviations (Cont’d) dc • TCVN (Vietnam) — Tiu Chun Vit Nam in Vietnamese. — Means “Vietnamese Standard.” • CJKV — Chinese, Japanese, Korean, and Vietnamese. 15th International Unicode Conference Copyright © 1999 Adobe Systems Incorporated What Is A Character Set? dc • A collection of characters that are intended to be used together to create meaningful text.
    [Show full text]
  • Basis Technology Unicode対応ライブラリ スペックシート 文字コード その他の名称 Adobe-Standard-Encoding A
    Basis Technology Unicode対応ライブラリ スペックシート 文字コード その他の名称 Adobe-Standard-Encoding Adobe-Symbol-Encoding csHPPSMath Adobe-Zapf-Dingbats-Encoding csZapfDingbats Arabic ISO-8859-6, csISOLatinArabic, iso-ir-127, ECMA-114, ASMO-708 ASCII US-ASCII, ANSI_X3.4-1968, iso-ir-6, ANSI_X3.4-1986, ISO646-US, us, IBM367, csASCI big-endian ISO-10646-UCS-2, BigEndian, 68k, PowerPC, Mac, Macintosh Big5 csBig5, cn-big5, x-x-big5 Big5Plus Big5+, csBig5Plus BMP ISO-10646-UCS-2, BMPstring CCSID-1027 csCCSID1027, IBM1027 CCSID-1047 csCCSID1047, IBM1047 CCSID-290 csCCSID290, CCSID290, IBM290 CCSID-300 csCCSID300, CCSID300, IBM300 CCSID-930 csCCSID930, CCSID930, IBM930 CCSID-935 csCCSID935, CCSID935, IBM935 CCSID-937 csCCSID937, CCSID937, IBM937 CCSID-939 csCCSID939, CCSID939, IBM939 CCSID-942 csCCSID942, CCSID942, IBM942 ChineseAutoDetect csChineseAutoDetect: Candidate encodings: GB2312, Big5, GB18030, UTF32:UTF8, UCS2, UTF32 EUC-H, csCNS11643EUC, EUC-TW, TW-EUC, H-EUC, CNS-11643-1992, EUC-H-1992, csCNS11643-1992-EUC, EUC-TW-1992, CNS-11643 TW-EUC-1992, H-EUC-1992 CNS-11643-1986 EUC-H-1986, csCNS11643_1986_EUC, EUC-TW-1986, TW-EUC-1986, H-EUC-1986 CP10000 csCP10000, windows-10000 CP10001 csCP10001, windows-10001 CP10002 csCP10002, windows-10002 CP10003 csCP10003, windows-10003 CP10004 csCP10004, windows-10004 CP10005 csCP10005, windows-10005 CP10006 csCP10006, windows-10006 CP10007 csCP10007, windows-10007 CP10008 csCP10008, windows-10008 CP10010 csCP10010, windows-10010 CP10017 csCP10017, windows-10017 CP10029 csCP10029, windows-10029 CP10079 csCP10079, windows-10079
    [Show full text]
  • Implementing Cross-Locale CJKV Code Conversion
    Implementing Cross-Locale CJKV Code Conversion Ken Lunde CJKV Type Development Adobe Systems Incorporated bc ftp://ftp.oreilly.com/pub/examples/nutshell/ujip/unicode/iuc13-c2-paper.pdf ftp://ftp.oreilly.com/pub/examples/nutshell/ujip/unicode/iuc13-c2-slides.pdf Code Conversion Basics dc • Algorithmic code conversion — Within a single locale: Shift-JIS, EUC-JP, and ISO-2022-JP — A purely mathematical process • Table-driven code conversion — Required across locales: Chinese ↔ Japanese — Required when dealing with Unicode — Mapping tables are required — Can sometimes be faster than algorithmic code conversion— depends on the implementation September 10, 1998 Copyright © 1998 Adobe Systems Incorporated Code Conversion Basics (Cont’d) dc • CJKV character set differences — Different number of characters — Different ordering of characters — Different characters September 10, 1998 Copyright © 1998 Adobe Systems Incorporated Character Sets Versus Encodings dc • Common CJKV character set standards — China: GB 1988-89, GB 2312-80; GB 1988-89, GBK — Taiwan: ASCII, Big Five; CNS 5205-1989, CNS 11643-1992 — Hong Kong: ASCII, Big Five with Hong Kong extension — Japan: JIS X 0201-1997, JIS X 0208:1997, JIS X 0212-1990 — South Korea: KS X 1003:1993, KS X 1001:1992, KS X 1002:1991 — North Korea: ASCII (?), KPS 9566-97 — Vietnam: TCVN 5712:1993, TCVN 5773:1993, TCVN 6056:1995 • Common CJKV encodings — Locale-independent: EUC-*, ISO-2022-* — Locale-specific: GBK, Big Five, Big Five Plus, Shift-JIS, Johab, Unified Hangul Code — Other: UCS-2, UCS-4, UTF-7, UTF-8,
    [Show full text]
  • San José, October 2, 2000 Feel Free to Distribute This Text
    San José, October 2, 2000 Feel free to distribute this text (version 1.2) including the author’s email address ([email protected]) and to contact him for corrections and additions. Please do not take this text as a literal translation, but as a help to understand the standard GB 18030-2000. Insertions in brackets [] are used throughout the text to indicate corresponding sections of the published Chinese standard. Thanks to Markus Scherer (IBM) and Ken Lunde (Adobe Systems) for initial critical reviews of the text. SUMMARY, EXPLANATIONS, AND REMARKS: CHINESE NATIONAL STANDARD GB 18030-2000: INFORMATION TECHNOLOGY – CHINESE IDEOGRAMS CODED CHARACTER SET FOR INFORMATION INTERCHANGE – EXTENSION FOR THE BASIC SET (信息技术－信息交换用汉字编码字符集 Xinxi Jishu – Xinxi Jiaohuan Yong Hanzi Bianma Zifuji – Jibenji De Kuochong) March 17, 2000, was the publishing date of the Chinese national standard (国家标准 guojia biaozhun) GB 18030-2000 (hereafter: GBK2K). This standard tries to resolve issues resulting from the advent of Unicode, version 3.0. More specific, it attempts the combination of Uni- code's extended character repertoire, namely the Unihan Extension A, with the character cov- erage of earlier Chinese national standards. HISTORY The People’s Republic of China had already expressed her fundamental consent to support the combined efforts of the ISO/IEC and the Unicode Consortium through publishing a Chinese National Standard that was code- and character-compatible with ISO 10646-1/ Unicode 2.1. This standard was named GB 13000.1. Whenever the ISO and the Unicode Consortium changed or revised their “common” standard, GB 13000.1 adopted these changes subsequently. In order to remain compatible with GB 2312, however, which at the time of publishing Unicode/GB 13000.1 was an already existing national standard widely used to represent the Chinese “simplified” characters, the “specification” GBK was created.
    [Show full text]
  • Unicode Compression: Does Size Really Matter? TR CS-2002-11
    Unicode Compression: Does Size Really Matter? TR CS-2002-11 Steve Atkin IBM Globalization Center of Competency International Business Machines Austin, Texas USA 78758 [email protected] Ryan Stansifer Department of Computer Sciences Florida Institute of Technology Melbourne, Florida USA 32901 [email protected] July 2003 Abstract The Unicode standard provides several algorithms, techniques, and strategies for assigning, transmitting, and compressing Unicode characters. These techniques allow Unicode data to be represented in a concise format in several contexts. In this paper we examine several techniques and strategies for compressing Unicode data using the programs gzip and bzip. Unicode compression algorithms known as SCSU and BOCU are also examined. As far as size is concerned, algorithms designed specifically for Unicode may not be necessary. 1 Introduction Characters these days are more than one 8-bit byte. Hence, many are concerned about the space text files use, even in an age of cheap storage. Will storing and transmitting Unicode [18] take a lot more space? In this paper we ask how compression affects Unicode and how Unicode affects compression. 1 Unicode is used to encode natural-language text as opposed to programs or binary data. Just what is natural-language text? The question seems simple, yet there are complications. In the information age we are accustomed to discretization of all kinds: music with, for instance, MP3; and pictures with, for instance, JPG. Also, a vast amount of text is stored and transmitted digitally. Yet discretizing text is not generally considered much of a problem. This may be because the En- glish language, western society, and computer technology all evolved relatively smoothly together.
    [Show full text]
  • Inews V3.4.2 Readme • 9390-65038-00 Rev
    iNEWS® Newsroom Computer System Version 3.4.2 ReadMe Date Revised Changes Made July 28, 2011 Added information related to v3.4.2 release July 5, 2011 Added information related to v3.4.1 release 12 May 2011 Added information related to v3.4.0 release 7 March 2011 Added information related to v3.3.1 release 30 November 2010 Added information related to v3.3.0 release 23 November 2010 Added information related to v3.2.2 release 14 October 2010 Added information related to v3.2.1 release 24 May 2010 Initial version (based off of v3.1.1 doc.) Important Information Avid recommends that you thoroughly read all of the information in this ReadMe file before installing or using any new software release. Note: Search the Avid Knowledge Base (http://www.avid.com/onlinesupport) for the most up-to-date ReadMe file, which contains the latest information that might have become available after the documentation was published. This document describes compatibility issues with previous releases, hardware and software requirements, software installation instructions, and summary information on system and memory requirements, when applicable. This document also lists any hardware and/or software limitations. Note: Since this release of iNEWS will be more widely distributed than previous versions, some notes on important features and changes have been brought forward from previous ReadMe documents. Notes that are less important were not brought forward, and users should reference older iNEWS ReadMe files for additional change information not contained in this ReadMe. Version 3.4.2 ReadMe Contents Important Information .............................................................................................................................................. 1 Compatibility Notes and Issues ................................................................................................................................
    [Show full text]
  • Package 'Fontmplus'
    Package ‘fontMPlus’ February 27, 2017 Title Additional 'ggplot2' Themes Using 'M+' Fonts Version 0.1.1 Description Provides 'ggplot2' themes based on the 'M+' fonts. The 'M+' fonts are a font family under a free license. The font family provides multilingual glyphs. The fonts provide 'Kana', over 5,000 'Kanji', Basic Latin, Latin-1 Supplement, Latin Extended-A, and 'IPA' Extensions glyphs. Most of the Greek, Cyrillic, Vietnamese, and extended glyphs and symbols are included too. So the fonts are in conformity with ISO-8859-1, 2, 3, 4, 5, 7, 9, 10, 13, 14, 15, 16, Windows-1252, T1, and VISCII encoding. More information about the fonts can be found at <http://mplus-fonts.osdn.jp/about-en.html>. Depends R (>= 3.0.0) License MIT + file LICENSE Encoding UTF-8 LazyData true RoxygenNote 6.0.1 Imports hrbrthemes, extrafont, ggplot2 Suggests stringr, knitr, rmarkdown URL https://github.com/bhaskarvk/fontMPlus BugReports https://github.com/bhaskarvk/fontMPlus/issues VignetteBuilder knitr NeedsCompilation no Author Bhaskar Karambelkar [aut, cre], MPlus [cph] Maintainer Bhaskar Karambelkar <[email protected]> Repository CRAN Date/Publication 2017-02-27 08:15:30 1 2 fontMPlus R topics documented: fontMPlus . .2 import_mplus . .3 mplus.fontfamilies . .3 mplus.fonttable . .4 theme_ipsum_mplus_c1 . .4 theme_ipsum_mplus_c2 . .6 theme_ipsum_mplus_m1 . .7 theme_ipsum_mplus_m2 . .9 theme_ipsum_mplus_mn1 . 10 theme_ipsum_mplus_p1 . 12 theme_ipsum_mplus_p2 . 14 Index 16 fontMPlus Additional ggplot2 themes using M+ fonts. Description This is an add-on pacakge for hrbrthemes pacakge. It provides seven ggplot2 themes based on M+ fonts. M+ FONTS The M+ FONTS are a font family under the Free license. You can use, copy, and distribute them, with or without modification, either commercially or noncommercially.
    [Show full text]
  • DVB); Specification for Service Information (SI) in DVB Systems
    Final draft ETSI EN 300 468 V1.5.1 (2003-01) European Standard (Telecommunications series) Digital Video Broadcasting (DVB); Specification for Service Information (SI) in DVB systems European Broadcasting Union Union Européenne de Radio-Télévision EBU·UER 2 Final draft ETSI EN 300 468 V1.5.1 (2003-01) Reference REN/JTC-DVB-128 Keywords broadcasting, digital, DVB, MPEG, service, TV, video ETSI 650 Route des Lucioles F-06921 Sophia Antipolis Cedex - FRANCE Tel.: +33 4 92 94 42 00 Fax: +33 4 93 65 47 16 Siret N° 348 623 562 00017 - NAF 742 C Association à but non lucratif enregistrée à la Sous-Préfecture de Grasse (06) N° 7803/88 Important notice Individual copies of the present document can be downloaded from: http://www.etsi.org The present document may be made available in more than one electronic version or in print. In any case of existing or perceived difference in contents between such versions, the reference version is the Portable Document Format (PDF). In case of dispute, the reference shall be the printing on ETSI printers of the PDF version kept on a specific network drive within ETSI Secretariat. Users of the present document should be aware that the document may be subject to revision or change of status. Information on the current status of this and other ETSI documents is available at http://portal.etsi.org/tb/status/status.asp If you find errors in the present document, send your comment to: [email protected] Copyright Notification No part may be reproduced except as authorized by written permission.
    [Show full text]
  • Implementing Cross-Locale CJKV Code Conversion
    Implementing Cross-locale CJKV Code Conversion Ken Lunde, Adobe Systems Incorporated [email protected] http://www.oreilly.com/~lunde/ 1. Introduction Most operating systems today deal with single locales. Within a single CJKV locale, different operating sys- tems often use different encodings for the same character set. Consider Shift-JIS and EUC-JP encodings for Japanese—Shift-JIS is historically used on MacOS and Windows, but EUC-JP is used on Unix. This makes code conversion a necessity. Code conversion within a single locale is, by and large, a trivial operation that typically involves a mathematical algorithm. In the past, a lot of code conversion was performed by users through dedicated software tools. Many of today’s applications include built-in code conversion routines, but these routines deal with only multiple encodings of a single locale (such as EUC-KR, ISO-2022-KR, Johab, and Unified hangul Code encodings for Korean). Code conversion across CJKV locales, such as between Chinese and Japanese, is more problematic. While Unicode serves as an excellent basis for implementing cross-locale code conversion, there are still problems to be addressed, such as unmappable characters. 2. Code Conversion Basics Converting between different encodings of a single locale, which represents a trivial effort that involves well- established code conversion algorithms (or mapping tables), is a well-understood process these days. How- ever, as soon as code conversion extends beyond a single locale, there are additional complexities that arise, such as the following: • Code conversion algorithms almost always must be replaced by mapping tables because the ordering of characters in different CJKV character sets are different.
    [Show full text]