DOCSLIB.ORG

Cisco Live 2018 Cap by Completing the Overall Event Evaluation and 5 Session Evaluations

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Cisco Live 2018 Cap by Completing the Overall Event Evaluation and 5 Session Evaluations Don’t Send – Deliver! Hrvoje (Harry) Dogan, Technical Marketing Engineer BRKSEC-2337 Cisco Spark Questions? Use Cisco Spark to communicate with the speaker after the session How 1. Find this session in the Cisco Live Mobile App 2. Click “Join the Discussion” 3. Install Spark or go directly to the space 4. Enter messages/questions in the space © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public Abstract In the past years, we've seen new and ingenious ways to exploit email as a threat vector. This, in turn, has brought in many new mechanisms to evaluate authenticity, content and sources of email. More than ever, senders - especially large volume senders - are struggling with getting all of their emails delivered, and keeping all of their email sources in good standing with blacklist and security intelligence providers. This session will show, through established best practices and industry examples, how to ensure that your email is deliverable. Along the way, it will make your email infrastructure, and your email traffic, more resilient, secure and Internet-friendly. We will discuss different techniques used by large email providers, and how to make sure they always like you. Basic knowledge of email operation and SMTP is required. Acquaintance with email authentication will be beneficial to understand the material. Agenda • To Send ≠ To Deliver • Different Faces of Reputation • Message Properties • Concurrency • Delivery Infrastructure • Be A Good Citizen About The Author Hrvoje (Harry) Dogan • Joined Cisco through IronPort acquisition in 2007 • SE, then CSE for Central and Eastern Europe, Europe Emerging... • Instructor Trainer for ESA/WSA, author of multiple whitepapers and Cisco Live sessions on Email and Web security • In 2011, moved to Security Business Group to join the product team • In 2015, relocated to Singapore to cover APJ+GC regions • Cisco Live Distinguished Speaker 2014, 2016, 2017 • Avid sailor, aspiring rock climber and retiring SCUBA diver • Ingress Enlightened agent and operator Presentation Theme Source: https://en.wikipedia.org/wiki/Singlish, retrieved on 12th of December, 2017 What’s Going On? You ask me I ask who? I am not the right person to answer this question Don't worry, sure can one. Don't worry; it'll work. A Piece of Mail Was Sent BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 A Piece of Mail Was Delivered BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 …And In-between? BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12 “There’s a big difference between sending email and delivering email” Joe Uhl, VP of Operations, MailChimp, in https://www.wired.com/2016/07/mailchimp-sends-billion-emails-day-thats-easy-part/ BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 13 The Problem Is Three-Fold User experience and email delays Additional stress on your delivery infrastructure Additional stress on 3rd parties’ receiving infrastructure BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 The Internet Delivers Swift Justice To Mishandled Delivery BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15 The Internet Delivers Swift Justice To Mishandled Delivery EHLO domain.com 554 IP address is blacklisted EHLO domain.com 554 Rejected due to policy restrictions EHLO domain.com 550 [RBL] Sender blocked domain.com 550 mail.server.com ESMTP Connection rejected. Your IP 10.18.3.47 is in RBL EHLO domain.com 550 OU-002 Mail rejected by Windows Live Hotmail for policy reasons. Reasons for rejection may be related to content with spam-like characteristics or IP/domain reputation problems. If you are not an email/network admin please contact your E-mail/Internet Service Provider for help. Email/network admins, please visit http://postmaster.live.com for email delivery information and support BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16 There Is No Carrot! IP Blacklisting AS Blacklisting Monetary charges for delisting BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 17 Different Faces of Reputation Everyone Looks For Different Things In Others • Sending host reputation • Sender Domain Reputation • Reputation by Network Proximity • AS Reputation • Network Owner Reputation • … BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18 What Are We To Do?!? That why asking how? The reason I'm asking the question is, how do I go about doing it? You want do how you do how. You can do it any way you like. Message Properties Overall considerations Correct Encoding Correct MIME Message Format Correct Time What Are You Sending?? BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21 Text Can Be Encoded In Many Ways! Sender’s Computer Encoding: BIG5 Receiver’s Computer Encoding: BIG5 BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 Text Can Be Encoded In Many Ways! Exchange Encoding: GB2312 Sender’s Computer Encoding:-- BIG5Boundary_(ID_nI+mPHhd2wki+0PBonlpeA) Receiver’s Computer Encoding: BIG5 Content-type: text/plain; charset=gb2312 Content-transfer-encoding: 8BIT FYR ________________________________ èƒ: Íê÷¡…∆ ºƒº˛»’∆⁄: 2014ƒÍ4‘¬23»’ œ¬ŒÁ 04:29 ÷¡: ≈̺∞ªÕ; Ñ¢∂°À…; Íê÷¡…∆; ÕıÔwòÂ; óÓçã∫¿; ¡÷è©–¢ ∏±±æ: ¡÷±˛›x; «Òø°Œƒ ÷˜÷º: RE: LCS1-üo∑®å¶ë™7/15ñ|‘£≤øΩY≤ø∆∑÷≥ˆ Ω®◊h: ¨F뙑쓗flM»ÎAPQP-3ÎA∂Œ BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 Setting a Default Encoding on the ESA esa1> scanconfig Configure encoding to use when none is There are currently 5 attachment type specified for plain body text oranything mappings configured to be SKIPPED. with MIME type plain/text or plain/html. 1. US-ASCII Choose the operation you want to perform: 2. Unicode (UTF-8) - NEW - Add a new entry. 3. Unicode (UTF-16) - DELETE - Remove an entry. 4. Western European/Latin-1 (ISO 8859-1) - SETUP - Configure scanning behavior. 5. Western European/Latin-1 (Windows CP1252) - IMPORT - Load mappings from a file. 6. Traditional Chinese (Big 5) - EXPORT - Save mappings to a file. 7. Simplified Chinese (GB 2312) - PRINT - Display the list. 8. Simplified Chinese (HZ GB 2312) - CLEAR - Remove all entries. 9. Korean (ISO 2022-KR) - SMIME - Configure S/MIME unpacking. 10. Korean (KS-C-5601/EUC-KR) - CLUSTERSET - Set how scanconfig is 11. Japanese (Shift-JIS (X0123)) configured in a cluster. 12. Japanese (ISO-2022-JP)13. Japanese - CLUSTERSHOW - Display how scanconfig is (EUC)[1]> 2 configured in a cluster. []> setup Setting default encoding will NOT help if encoding is specified in the message, but is incorrect! BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 Encoding Makes A Difference! • Text can be encoded in many ways • Headers have encoding, too! • RFC2047: Message Header Extensions for Non-ASCII Text Subject: [MARKETING] Merry Christmas BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25 Encoding Makes A Difference! • Text can be encoded in many ways • Headers have encoding, too! • RFC2047: Message Header Extensions for Non-ASCII Text Subject: [MARKETING] Merry=?utf -8?Q?Merry=20Christmas=C2=A0=F0=9F=8E=84?=Christmas BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 MIME-Version: 1.0 Content-type: multipart/mixed; boundary="B_3597992599_406827548" > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3597992599_406827548 Content-type: multipart/alternative; boundary="B_3597992599_1713911382" --B_3597992599_1713911382 Content-type: text/plain; charset="UTF-8” Content-transfer-encoding: 7bit […] --B_3597992599_1713911382 Content-type: text/html; charset="UTF-8” Content-transfer-encoding: quoted-printable <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org=/TR/xhtml1/DTD/xhtml1- strict.dtd"> <html xmlns=3D"http://www.w3.org/1999/xhtml"> […] </html> --B_3597992599_1713911382-- --B_3597992599_406827548 Content-type: application/pdf; name="4854861.pdf"; x-mac-creator="4F50494D"; x-mac-type="50444620” Content-ID: [email protected] Content-disposition: attachment; filename="4854861.pdf” Content-transfer-encoding: base64 […] --B_3597992599_406827548-- BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 27 A MIME Is A Terrible Thing To Waste! • Make sure that ALL of your MIME is done right! • All enclosures and containers properly formatted and closed • All attachments properly named, encoded and with proper disposition • All text parts properly encoded • No duplicates! BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28 Message Format RFC5322 Anyone? Proper PRA Headers Proper Message-ID (From/Sender/Resent-From…) Proper Return-Path Obey Line Length Fold, Fold, Fold!!! BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29 Determining PRA RFC4407 0 Select first non-empty Resent-Sender 1 Found MULTIPLE ? header Select all Message Found Y From ? PRA Malformed headers N 0 Select first non-empty 1 Found MULTIPLE Resent-From ? header Select all Impossible to Found Y Sender determine PRA ? headers BRKSEC-2337 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30 Message Format RFC5322 Anyone? From:Proper
Load more

Recommended publications
  • Basis Technology Unicode対応ライブラリ スペックシート 文字コード その他の名称 Adobe-Standard-Encoding A
    Basis Technology Unicode対応ライブラリ スペックシート 文字コード その他の名称 Adobe-Standard-Encoding Adobe-Symbol-Encoding csHPPSMath Adobe-Zapf-Dingbats-Encoding csZapfDingbats Arabic ISO-8859-6, csISOLatinArabic, iso-ir-127, ECMA-114, ASMO-708 ASCII US-ASCII, ANSI_X3.4-1968, iso-ir-6, ANSI_X3.4-1986, ISO646-US, us, IBM367, csASCI big-endian ISO-10646-UCS-2, BigEndian, 68k, PowerPC, Mac, Macintosh Big5 csBig5, cn-big5, x-x-big5 Big5Plus Big5+, csBig5Plus BMP ISO-10646-UCS-2, BMPstring CCSID-1027 csCCSID1027, IBM1027 CCSID-1047 csCCSID1047, IBM1047 CCSID-290 csCCSID290, CCSID290, IBM290 CCSID-300 csCCSID300, CCSID300, IBM300 CCSID-930 csCCSID930, CCSID930, IBM930 CCSID-935 csCCSID935, CCSID935, IBM935 CCSID-937 csCCSID937, CCSID937, IBM937 CCSID-939 csCCSID939, CCSID939, IBM939 CCSID-942 csCCSID942, CCSID942, IBM942 ChineseAutoDetect csChineseAutoDetect: Candidate encodings: GB2312, Big5, GB18030, UTF32:UTF8, UCS2, UTF32 EUC-H, csCNS11643EUC, EUC-TW, TW-EUC, H-EUC, CNS-11643-1992, EUC-H-1992, csCNS11643-1992-EUC, EUC-TW-1992, CNS-11643 TW-EUC-1992, H-EUC-1992 CNS-11643-1986 EUC-H-1986, csCNS11643_1986_EUC, EUC-TW-1986, TW-EUC-1986, H-EUC-1986 CP10000 csCP10000, windows-10000 CP10001 csCP10001, windows-10001 CP10002 csCP10002, windows-10002 CP10003 csCP10003, windows-10003 CP10004 csCP10004, windows-10004 CP10005 csCP10005, windows-10005 CP10006 csCP10006, windows-10006 CP10007 csCP10007, windows-10007 CP10008 csCP10008, windows-10008 CP10010 csCP10010, windows-10010 CP10017 csCP10017, windows-10017 CP10029 csCP10029, windows-10029 CP10079 csCP10079, windows-10079
    [Show full text]
  • DICOM PS3.5 2021C
    PS3.5​ DICOM PS3.5 2021d - Data Structures and Encoding​ Page 2​ PS3.5: DICOM PS3.5 2021d - Data Structures and Encoding​ Copyright © 2021 NEMA​ A DICOM® publication​ - Standard -​ DICOM PS3.5 2021d - Data Structures and Encoding​ Page 3​ Table of Contents​ Notice and Disclaimer ........................................................................................................................................... 13​ Foreword ............................................................................................................................................................ 15​ 1. Scope and Field of Application ............................................................................................................................. 17​ 2. Normative References ....................................................................................................................................... 19​ 3. Definitions ....................................................................................................................................................... 23​ 4. Symbols and Abbreviations ................................................................................................................................. 27​ 5. Conventions ..................................................................................................................................................... 29​ 6. Value Encoding ...............................................................................................................................................
    [Show full text]
  • ITCAM Extended Agent for Oracle Database Installation and Configuration Guide Figures
    IBM Tivoli Composite Application Manager Extended Agent for Oracle Database Version 6.3.1 Fix Pack 2 Installation and Configuration Guide SC27-5669-00 IBM Tivoli Composite Application Manager Extended Agent for Oracle Database Version 6.3.1 Fix Pack 2 Installation and Configuration Guide SC27-5669-00 Note Before using this information and the product it supports, read the information in “Notices” on page 65. This edition applies to version 6.3.1 Fix Pack 2 of IBM Tivoli Composite Application Manager Extended Agent for Oracle Database (product number 5724-I45) and to all subsequent releases and modifications until otherwise indicated in new editions. © Copyright IBM Corporation 2009, 2013. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Figures ...............v Setting environment variables .......33 Customizing Oracle alert log monitoring . 37 Tables ...............vii Customizing Listener and Net Service monitoring 37 Defining and running customized SQL statements ..............38 Chapter 1. Overview of the agent ....1 Configuring Top SQL monitoring ......40 IBM Tivoli Monitoring ...........1 Configuring agent event monitoring .....42 Features of the Oracle Database Extended agent . 1 Sample Database connection configuration . 43 Functions of the monitoring agent .......2 Starting or stopping the agent .......46 New in this release ............3 Configuring for clustering and positioning in the Components of the IBM Tivoli Monitoring portal navigator
    [Show full text]
  • Automatic Detection of Character Encoding and Language
    Automatic Detection of Character Encoding and Language Seungbeom Kim, Jongsoo Park {sbkim,jongsoo}@stanford.edu CS 229, Machine Learning Autumn 2007 Stanford University 1 Introduction their methods, we aimed at a simple algorithm which can be uniformly applied to every charset, and the algorithm The Internet is full of textual contents in various lan- is based on well-established, standard machine learning guages and character encodings, and their communica- techniques. We also studied the relationship between lan- tion across the linguistic borders is ever increasing. Since guage and charset detection, and compared byte-based al- the different encodings are not compatible with one an- gorithms and character-based algorithms. We used Na¨ıve other, communication protocols such as HTTP, HTML, Bayes (NB) and Support Vector Machine (SVM). and MIME are equipped with mechanisms to tag the char- Using the documents downloaded from Wikipedia [6], acter encoding (a.k.a. charset) used in the delivered con- we evaluated different combinations of algorithms and tent. However, as native speakers of a language whose compared them with the universal charset detector in character set does not fit in US-ASCII, we have encoun- Mozilla. We found two promising algorithms. The first tered a lot of web pages and e-mail messages that are one is a simple SVM whose feature is the frequency of encoded in or labeled with a wrong character encoding, byte values. The algorithm is uniform and very easy to which is often annoying or frustrating. Many authors, implement. It also only needs maximum 256 table entries especially of e-mail messages, are not aware of the en- per each charset and the detection time is much shorter coding issues, and their carelessness in choosing correct than other algorithms.
    [Show full text]
  • HZ – CN-GB – EUC-CN – CP936 • Traditional – EUC-TW – Big Five Et Al
    The Hitchhiker’s Guide to Chinese Encodings Tom Emerson (譨聢·蒨翳芖) Senior Computational Linguist 20th International Unicode Conference Washington, D.C., USA strategy • process • technology • results www.basistech.com Overview • Who am I and why am I here? • What do we mean by “Chinese?” – Simplified vs. Traditional •ChineseCharacter Sets • Introduce Chinese Encodings • Driving Forces • Reality vs. Idealism • Transcoding Issues Who Am I? • “Sinostringologst” at Basis Technology • Lead developer for our Chinese Morphological Analyzer and our Chinese Script Converter • Background in both Computer Science and Linguistics Who Are You? Don’t Panic! The Blowfish Book is the reference for anyone working with CJK character sets. Get it. What is “Chinese?” • For our purposes we are interested in the written language. • In general this means Mandarin. • Topolects (ᮍ㿔) sometimes define their own hanzi for local words, usually for names. • Hence “Written Cantonese” doesn't make a lot of sense. Simplified vs. Traditional • “Simple” and “Full” Form • Mainland China and Singapore use “Simplified Chinese” • Hong Kong, Taiwan, and Macao use “Traditional Chinese” Simplification • Fewer strokes – Easier to learn – Easier to remember – Easier to write • Compare: ৄ vs. 繟 • Simplification is not recent – Some simplified characters in current use date to the pre-Qin period (pre 246 B.C.E.) Simplification 1956: Scheme for Simplifying Chinese Characters 1964: The Complete List of Simplified Characters (2236 characters) 1986: The Complete List of Simplified Characters, 2nd Simplification ऱഏᎾᆖᛎ᝟ႨΕല۟֟׌سഏᎾՕ᝜ࠓࢬข ৫ၲࡨऱ᝜ڣװᆖᛎऱ࿇୶ΖൕڣࠐԼآᖄ ࠓᑪխΕ׌ߡຟਢ۩ᄐհଈΖ ೑䰙໻䌁ᑊ᠔ѻ⫳ⱘ೑䰙㒣⌢䍟࢓ǃᇚ㟇ᇥЏ ᇐ᳾ᴹकᑈ㒣⌢ⱘথሩDŽҢএᑈᑺᓔྟⱘ䌁 ᑊ╂ЁǃЏ㾦䛑ᰃ㸠ϮП佪DŽ Character Sets vs. Encodings • Non-Coded Character Sets – A non-coded character set represents a list of characters that are defined by an organization as the standard set that one is expected to know.
    [Show full text]
  • No U+FFFD Generation for Zero-Length Content Between ISO-2022-JP Escape Sequences
    No U+FFFD Generation for Zero-Length Content between ISO-2022-JP Escape Sequences Henri Sivonen ([email protected]) 2020-08-06 UTR #36: Unicode Security Considerations says1, in part: Character encoding conversion must also not simply skip an illegal input byte sequence. Instead, it must stop with an error or substitute a replace- ment character (such as U+FFFD ( � ) REPLACEMENT CHARACTER) or an es- cape sequence in the output. (See also Section 3.5 Deletion of Code Points.) It is important to do this not only for byte sequences that encode characters, but also for unrecognized or "empty" state-change sequences. For example: […] • ISO-2022 shift sequences without text characters before the next shift sequence. The formal syntaxes for HZ and most CJK ISO-2022 variants require at least one character in a text segment between shift se - quences. Security software written to the formal specification may not detect malicious text (for example, "delete" with a shift-to-double-byte then an immediate shift-to-ASCII in the middle). Background Previously, Gecko, the engine of Firefox and Thunderbird, used a character encod- ing conversion library called uconv, which did not implement the above advice and, like Internet Explorer, did not generate an U+FFFD when two ISO-2022-JP es- cape sequences occurred without any content between them. In Firefox 56, Gecko switched form uconv to encoding_rs, which is a newly-written character encoding conversion library that generates U+FFFD when no content oc- cur between ISO-2022-JP escape sequences, because the above advice from UTR #36 is embodied in the algorithm2 specified in the WHATWG Encoding Standard.
    [Show full text]
  • Representing Information in English Until 1598
    In the twenty-first century, digital computers are everywhere. You al- most certainly have one with you now, although you may call it a “phone.” You are more likely to be reading this book on a digital device than on paper. We do so many things with our digital devices that we may forget the original purpose: computation. When Dr. Howard Figure 1-1 Aiken was in charge of the huge, Howard Aiken, center, and Grace Murray Hopper, mechanical Harvard Mark I calcu- on Aiken’s left, with other members of the Bureau of Ordnance Computation Project, in front of the lator during World War II, he Harvard Mark I computer at Harvard University, wasn’t happy unless the machine 1944. was “making numbers,” that is, U.S. Department of Defense completing the numerical calculations needed for the war effort. We still use computers for purely computational tasks, to make numbers. It may be less ob- vious, but the other things for which we use computers today are, underneath, also computational tasks. To make that phone call, your phone, really a com- puter, converts an electrical signal that’s an analog of your voice to numbers, encodes and compresses them, and passes them onward to a cellular radio which it itself is controlled by numbers. Numbers are fundamental to our digital devices, and so to understand those digital devices, we have to understand the numbers that flow through them. 1.1 Numbers as Abstractions It is easy to suppose that as soon as humans began to accumulate possessions, they wanted to count them and make a record of the count.
    [Show full text]
  • Argument Encoding in Japanese Conversation Pdf Free Download
    ARGUMENT ENCODING IN JAPANESE CONVERSATION PDF, EPUB, EBOOK M. Shimojo | 286 pages | 01 May 2005 | Palgrave USA | 9781403937056 | English | Gordonsville, United States Argument Encoding in Japanese Conversation PDF Book R hints at this behavior in the Encoding section of? We have a dedicated site for Germany. Instead, lean forward, make eye contact and look interested. Conclusions Pages Shimojo, Mitsuaki. Encode version 2. These modes are all actually set via a bitmask. Both of the above tests pass on Cygwin , so the default line terminator is :DOS. Beginning with Perl 5. Its line terminator mode is ignored. To that end you should avoid writing to files by name, and instead explicitly open and close the connection yourself. They have bytes. Thanks James and Paul During recent history, data is moved around a computer in 8-bit chunks, often called "bytes" but also known as "octets" in standards documents. Introduction It causes many mojibake unreadable character problems for all computer users. More reasons may have been given that I do not remember anymore. To find out in detail which encodings are supported by this package, see Encode::Supported. Sequences of zeroes and ones. This is a stateful 7-bit encoding. And here is a Wikipedia article on the Han unification issue: en. Fortunately, we can avoid this altogether by using unicode code points in the string literal explicitly:. Default encodings Perl 5. Do not treat the return value as indicative of success or failure, because that isn't what it means: it is only the previous setting. What if we want to represent non-English characters? It indicates whether a string is internally encoded as "utf8", also without a hyphen.
    [Show full text]
  • DICOM PS3.5 2013 - Data Structures and Encoding Page 2
    PS3.5 DICOM PS3.5 2013 - Data Structures and Encoding Page 2 PS3.5: DICOM PS3.5 2013 - Data Structures and Encoding Copyright © 2013 NEMA - Standard - DICOM PS3.5 2013 - Data Structures and Encoding Page 3 Table of Contents Notice and Disclaimer ........................................................................................................................................... 11 Foreword ............................................................................................................................................................ 13 1. Scope and Field of Application ............................................................................................................................. 15 2. Normative References ....................................................................................................................................... 17 3. Definitions ....................................................................................................................................................... 19 3.1. Reference Model Definitions ......................................................................................................................... 19 3.2. ACSE Service Definitions ............................................................................................................................. 19 3.3. Presentation Service Definitions .................................................................................................................... 19 3.4. Object Identification Definitions
    [Show full text]
  • Naming Text Encodings to Demystify Them
    Naming Text Encodings to Demystify Them Document #: P1885R6 Date: 2021-08-14 Programming Language C++ Audience: LEWG Reply-to: Corentin Jabot <[email protected]> Peter Brett <[email protected]> If you can’t name it, you probably don’t know what it is If you don’t know what it is, you don’t know what it isn’t Tony Van Eerd Target C++23 Abstract For historical reasons, all text encodings mentioned in the standard are derived from a locale object, which does not necessarily match the reality of how programs and systems interact. This model works poorly with modern understanding of text, ie the Unicode model separates encoding from locales which are purely rules for formatting and text transformations but do not affect which characters are represented by a sequence of code units. Moreover, the standard does not provide a way to query which encodings are expected or used by the environment, leading to guesswork and unavoidable UB. This paper introduces the notions of literal encoding, environment encoding, and a way to query them. Examples Listing the encoding #include <text_encoding> #include <iostream> void print(const std::text_encoding & c) { std::cout << c.name() << " (iana mib: " << c.mib() << ")\n" << "Aliases:\n"; 1 for(auto && a : c.aliases()) { std::cout << '\t' << a << '\n'; } } int main() { std::cout << "Literal Encoding: "; print(std::text_encoding::literal()); std::cout << "Wide Literal Encoding: "; print(std::text_encoding::wide_literal()); std::cout << "environment Encoding: "; print(std::text_encoding::environment());
    [Show full text]
  • Naming Text Encodings to Demystify Them
    Naming Text Encodings to Demystify Them Document #: P1885R3 Date: 2020-09-14 Project: Programming Language C++ Audience: SG-16, LEWG Reply-to: Corentin Jabot <[email protected]> If you can’t name it, you probably don’t know what it is If you don’t know what it is, you don’t know what it isn’t Tony Van Eerd Target C++23 Abstract For historical reasons, all text encodings mentioned in the standard are derived from a locale object, which does not necessarily match the reality of how programs and systems interact. This model works poorly with modern understanding of text, ie the Unicode model separates encoding from locales which are purely rules for formatting and text transformations but do not affect which characters are represented by a sequence of code units. Moreover, the standard does not provide a way to query which encodings are expected or used by the system, leading to guesswork and unavoidable UB. This paper introduces the notions of literal encoding, system encoding and a way to query them. Examples Listing the encoding #include <text_encoding> #include <iostream> void print(const std::text_encoding & c) { std::cout << c.name() << " (iana mib: " << c.mib() << ")\n" << "Aliases:\n"; for(auto && a : c.aliases()) { 1 std::cout << '\t' << a << '\n'; } } int main() { std::cout << "Literal Encoding: "; print(std::text_encoding::literal()); std::cout << "Wide Literal Encoding: "; print(std::text_encoding::wide_literal()); std::cout << "System Encoding: "; print(std::text_encoding::system()); std::cout << "Wide system Encoding:
    [Show full text]
  • Programming with Unicode Documentation Release 2011
    Programming with Unicode Documentation Release 2011 Victor Stinner Oct 01, 2019 Contents 1 About this book 1 1.1 License..................................................1 1.2 Thanks to.................................................1 1.3 Notations.................................................1 2 Unicode nightmare 3 3 Definitions 5 3.1 Character.................................................5 3.2 Glyph...................................................5 3.3 Code point................................................5 3.4 Character set (charset)..........................................5 3.5 Character string.............................................6 3.6 Byte string................................................6 3.7 UTF-8 encoded strings and UTF-16 character strings..........................7 3.8 Encoding.................................................7 3.9 Encode a character string.........................................7 3.10 Decode a byte string...........................................8 3.11 Mojibake.................................................8 3.12 Unicode: an Universal Character Set (UCS)...............................9 4 Unicode 11 4.1 Unicode Character Set.......................................... 11 4.2 Categories................................................ 11 4.3 Statistics................................................. 12 4.4 Normalization.............................................. 12 5 Charsets and encodings 15 5.1 Encodings................................................ 15 5.2 Popularity...............................................
    [Show full text]