News
Usenet、NetNews
InterNet News InterNet
–
Network News Transfer Protocol TransferNews Network
– Reader: tin Reader: INN Server: History of netnews of History newsgroups of History operations News NNTP
• • • • • • News softwares Transfer protocolTransfer Introduction
News outlineNews
Computer Center, CS, NCTU 2 Introduction to News
mail and web and web forums mail -
mail messages. mail - FAQ, spam. FAQ, The initial Internet community Theinitial (BBS) board systemsbulletin Like betweene Hybrid dedicatedAbsence of administratora centralserver and
• • • • •
world networked the in importance cultural has significant Usenet The format and transmission of Usenet articles is similar to that of that to similar is articles Usenet of andtransmission The format e Internet Usenet is a worldwide distributed Internet discussion system. discussion Internet distributed a worldwide is Usenet
Introduction to news (1)
Computer Center, CS, NCTU 4
492 7.52 2011 2003
257 5.42 2002 2010
181 4.65 2001 2009
82 3.07 2000 2008
26 2.95 1999 2007
12 2.27 1998 2006
9 1.52 1997 2005
have” resource? have”
-
1 4.5 1996 2004
(TB) (GB)
http://groups.google.com
Year Year • allows newsgroup posting. newsgroup allows a “must is Usenet blogs, and mailing lists. mailing and blogs, to attempt to newsgroups Usenet of archivebuy to efforts made Google also and interface search webhas a Googleone. complete a create Spamming: In 1994, advertisers began broadcasting messags to all all to messags broadcasting began advertisers In1994, Spamming: once.at newsgroups forums, Internet to respect with importance in diminished has Usenet
Introduction to news (2) Daily Vol. Daily Daily Vol. Daily
Computer Center, CS, NCTU 5
Introduction to news (3)
Computer Center, CS, NCTU 6
Tom Truscott, SteveDanielTom Truscott,
Better performance for the grown news grown news trafficperformance for Betterthe Rewritten in C byRewritten Glickman Mark and in 1981 Horton MattWrittenby in 1983 ~ Development on to RickAdams 1989 was passed Internet mail with compatibleformat a message With Tom Truscott and Jim Ellis conceived the idea in conceivedtheidea1979 and Ellis Tom TruscottJim scripts shell Bourne writing with assisted Steve Bellovin Thefirst “netnews” program as transportprotocolits UUCP release“netnews” Thepublic in 1980 of
• • • • • • • • • • B News A News Origins
History History (1)
Computer Center, CS, NCTU 7
up connections up -
forward batch - Unix CopyProtocol Unix - and to - -
Duplicate articlesDuplicate Unix dial over used commonly Mostly news for and mail for NNTP theTCP/IP protocolsSMTP Replacedby nodes networkuse through torecordarewithintheBang paththenews still has which a message passed Store
• • • • • • UUCP
History History (2)
Computer Center, CS, NCTU 8
Network News Network News Transfer Protocol network TCP/IP over articles netnews distribute to in 1985 Introduced in 1986 and BrianKantorPhil Lapsley is createdSpecificationby port for andfor portTCP 563 NNTPS TCP NNTP 119 News Protocol)NNRP(Network Reader Sending requiredarticles
• • • • • • NNTP
History History (3)
Computer Center, CS, NCTU 9
Released by Rich Salz in by1991 RichSalzReleased functionalityintegrateNNTP Thefirst news serverto fully package andUUCP NNTP support used commonlynews servernow Themost package Consortium) Systems byISC(Internet developed activelyStill Developed by Henry Spencer and byHenryGeoff Collyerin 1987 DevelopedSpencer faster program in a cleaner, to result News of B code the Rewrite a news indexThe functioncreationof in 1992 XOVER command NNTP and threadingnews readersinformationheader rapidlyretrieveAllow to
• • • • • • • • • • INN (InterNet News) INN (InterNet The CNEWS version The CNEWS
History History (4)
Computer Center, CS, NCTU 10
Like the public message boards on thepublic BBS. boardsLike message Over 100000 Usenet newsgroups Types: text or Types:binary.text
• • • Newsgroups group. a discussion a repository, is newsgroup A usenet The articles are organized into topical categories called newsgroups. called categories topical into organized are The articles it making theoretically hierarchies, into organized are Newsgroups groups. related find to simpler
newsgroups (1)
Computer Center, CS, NCTU 11
European networks refuse to pay forsome of the highvolume and low content groups such as religion and racism. The suggested category talk.* was fr.* forgroups gatewayed from ARPANET mod.* formoderated discussions net.* forunmoderated groups groups. theof existinga listall of maintaining Primaryreason: thedifficulty explanation:Alternative July 1986 March~ 1987 July Before this
• • • • Great Renaming Great
newsgroups (2)
Computer Center, CS, NCTU 12
rec.* rec.* fordiscussions of recreational activities sci.* for science related discussions soc.* for socializing and discussions of social issues talk.* for discussions about various controversial topics misc.* for miscellaneous discussions humanities.* fordiscussions of the humanities comp.* comp.* for computer related discussions news.* for discussions and announcements about netnews Big 8 in 1995 in8 Big Initiated byAdams Rick Initiated Cabal” “Backbone the organizedSpafford Gene Big 7 in1987
• • • • Great Renaming Great
newsgroups (3)
Computer Center, CS, NCTU 13
rec.food.recipies
8 management board will vote on whether to create the group. voteon board createmanagement whether the8 to will - Drugs, and sex were Drugs, and sex not allowed. mod.gourmand with Dissatisfied in May1987. Created Post an RFD (Request for Discussion) into news.announce.newgroups for Discussion) (Request an RFD Post in news.groups.proposalsDiscuss charter.formalizeddescription,The a name, is with proposal Big
• • • • • • • Brain Reid suggested the alt.* hierarchy as an alternative to talk.* to an alternative as hierarchy alt.* the suggested Reid Brain There was a great controversy over what newsgroups should be allowed. be should newsgroups what over controversy a great was There After the Great Renaming, the creation and organization of new of organization and creation the Renaming, Great the After process. voting automated an became newsgroups
newsgroups (4)
Computer Center, CS, NCTU 14
7. -
jokingly referred to ALT standing for “Anarchists, Lunatics and Terrorists” Lunatics for “Anarchists, standing ALT referred to jokingly distributed.widelynews mayfrom fearingbe preventedthe alt.sex.*:major large size. of extremelyoften alt.binaries.*: It is „alternative‟ to the mainstream Big mainstream It is „alternative‟to the alt.drugsand alt.gourmand, alt.config, were newsgroups alt.*The first an informal involvingfollow a publicprocedurenewsgroups Most before created.in alt.configdiscussion subjecttype.confinedof specificany or Not to newsgroups naturetheof groups sprung up,with which some the Becauseanarchistic
• • • • • • • Two major sections of alt.* of sections Twomajor
alt.* hierarchy alt.*
newsgroups (5)
Computer Center, CS, NCTU 15 8 hierarchies, hierarchies, 8 -
de.*, fr.*, … Messages posted in there “private” newsgroups are generally not passed arenot passed generallyto newsgroups in there“private” Messages posted in theInternettheother general.or providers Other language hierarchies have later been created in parallel to the existing to the in parallelcreated been laterhave hierarchieslanguage Other ones. English tw.bbs.* (innbbsd) tw.*,
• • • Some users prefer to use “Usenet” to refer only to the Big the to only refer to “Usenet” use to prefer users Some medium. entire the incorporate to “netnews” use to and the prefixing versions, own their create also providers access Some way. similar a in name own their with names newsgroups The language of preference in the “original” Usenet hierarchies, hierarchies, Usenet “original” the in preference of The language English. is alt.*, including
newsgroups (6)
Computer Center, CS, NCTU 16
Printable MIME, or MIME, yEnc.Printable -
A moderated newsgroup has one or more individuals as moderators. as one or A moderatedindividualsmore newsgroup has emailedbut are not distributedareby readers to Usenet,Messages submitted instead.moderatorsapproval,to the for approved and them, articles,injectreceivesubmitted reviewModerators line. header “Approved:” thebear must Such articlesarticles. Binaries areposted Binariesin alt.binaries.* in ASCII. encoded text content to distribute created originally was Usenet Quoted binariesbyBase64,Encodeduuencode, PAR. and by of RAR means might be split into piecesThebinaries
• • • • • • • newsgroups Moderated Binary newsgroups Binary
newsgroups (7)
Computer Center, CS, NCTU 17
reply article is called a thread. -
Articles which can be traced to one single non exchanges articleswith them. exchanges The overview is a list of the most frequently used headers, and additional headers, and used frequentlyof the list most a is The overview sizes.as informationsuch article user‟s news on thatavailable onlyis initially posts anitWhen a user article, news serverto one or more server.otherEach talks and servers(peers) Binary content: split among multiple articles, reassembled into a singleunitintoreassembled multiple articles,splitamong content:Binary readernews the by other thearticlesare article.responses to some of majorityPlaintext:
• • • • newsfeeds Headers andoverviews Headers Articles and posts and Articles
News operationsNews (1)
Computer Center, CS, NCTU 18
news/tin) ends have become more have more become ends 、 -
based news reader:tin(chinese/tin based
-
However, these integrated clients are lowof quality: incorrectly implementing Usenet protocols, standards and conventions. A user subscribes to a newsgroup, the news readertrack newsgroup, A user subscribes to a keeps of the software that has which articlesread. user therise of Web, the With World front Wide web common. textCommon Read and post articlesRead and post single readerwere a and news server news implementations: news earlyIn suite.program have integratedan news “communication commonlysuites” or Mailclients reader.
• • • • • • News Reader News
News operationsNews (2)
Computer Center, CS, NCTU 19
carry carry the heaviest CPU load Transit Transit server: handle the basic articles exchange. Numbering server: insert the “Xref:” header line into each article Back end server: data storage Front end server: a user would actually connect to. Only store overviews, and intoSplit Continually increasing storage increasingContinually requirements capacity Network trafficallreceiveof theto successfullyabilitya server Completion: to readersare made availablethetime amount articlesof Retention: systemOverallperformance
• • • • • • News ServerNews
News operationsNews (3)
Computer Center, CS, NCTU 20
suited to analyze the files -
Since 1990s Articles are appended serially to large indexed container files When the end of the file is reached, new articles are written at the beginningthe of file, overwriting the oldest entries. Traditional text manipulation tools are less well Oldest storage scheme Performance relates to OS‟s ability to create, remove and locate files within a dictionary – – – – – –
One file per article Cyclical storage
Flexibility to and spaceused. retention Flexibility A disk storage for the body of storagefor articlean theA diskbody Types:
• • • Spools
News operationsNews (4)
Computer Center, CS, NCTU 21
The maximum The maximum article size varies from site to site Howthe long server keeps articles Text and binary
How user an articleto thequicklycana serverdeliver How Article sizesArticle Retention
• • • Speed Storage management Storage
News operationsNews (5)
Computer Center, CS, NCTU 22
ID -
Delete a article specified by Message In cases where unsuitable content has been posted, Usenet has support for automated removal a of posting from the whole network by creating a cancel message. Create Create newsgroupa new Remove a newsgroup Lists all groups of a hierarchy cancel messages cancel newgroup messages messages rmgroup messages checkgroup
• • • • Control messages (RFC5537) messages Control
News operationsNews (6)
Computer Center, CS, NCTU 23
MMF 、 Challenge - F/s/l(From/subject/lines) 、 Cancel
、 ID) - 20% useful information (mime) (mime) html post host/lines) - 、 、
Ph/l(Posting 、
40% cancel messages
、
Key Key = hash(password, Message -
Specific strings such as binary post No see „em Cancel All notices must be public key signed to be accepted using PGP innbbsdpatch by leeym 40% spams There is authenticationno in cancel messages Written perl by Count by md5(body) NoCeM Cancel message Cancel Cleanfeed
• • • Spam filter Spam
News operationsNews (7)
Computer Center, CS, NCTU 24
nonsense
–
Group.NCTU.edu.tw tw.bbs.campus.nctu ↔ nctu.talk tw.bbs.campus.nctu tw.bbs.comp.linux ↔ ptt tw.bbs.comp.linux NCTU_Talk
Take a break
Computer Center, CS, NCTU 25 NNTP
RFC 3977 Interact with INN news server
standard extensions) -
Reserved forprivate use (non Connection, setup, and miscellaneous messages Newsgroup selection Article selection Distribution functions Posting Reserved forauthentication and privacy extensions Informative Informative message Command completed OK Command soOK far; send the rest of it Command was syntactically correct but failed for some reason Command unknown,unsupported, unavailable, or syntax error
------x9x x0x x1x x2x x3x x4x x8x 1xx 2xx 3xx 4xx 5xx Response codesResponse
• Command and response Command
NNTP (1) NNTP
Computer Center, CS, NCTU 27
). posting ok posting
INN 2.4.6 ready ( ready 2.4.6 INN
INN 2.4.6 ready 2.4.6 INN
NNRP server NNRP server
telnet csnews2 433 csnews2 telnet InterNetNews $ telnet csnews2 119 csnews2 telnet InterNetNews telnet csnews2 433 csnews2 telnet
readersFor your For your peersFor your
liuyh@bsd5$ liuyh@bsd5$ 140.113.235.121... Trying csnews2. to Connected '^]'. is character Escape 200 csnews2 502 Permission denied 502 Permission liuyh@nsd5$ liuyh@nsd5$ 140.113.235.121... Trying csnews2. to Connected '^]'. is character Escape • • Trying 140.113.235.121... Trying csnews2.cs.nctu.edu.tw. to Connected '^]'. is character Escape 200 csnews2 liuyh@csnews NNTP vs. NNRP NNTPvs.
NNTP (2) NNTP
Computer Center, CS, NCTU 28
班 A
物件導向程式設計
] ".
". 課程 [ 0000000284 0000000108 y
. ] (Moderated) ] 計算機網路管理 ]
programming.a programming.a group description group group high low flags low high group - - 系計中公告 課程 [
0000000505 0000000505 0000000131 y
oriented oriented - -
center [ center center 0000000509 0000000509 0000000071 m center - -
AVTIVE NEWSGROUPS
cs.computer cs.course.netadmin cs.course.object list newsgroups cs.* newsgroups list " form in 215 Descriptions 215 Newsgroups in form " form in 215 Newsgroups cs.computer cs.course.netadmin cs.course.object list active cs.* active list • • LIST (find the newsgroups) the LIST (find
NNTP (3) NNTP
Computer Center, CS, NCTU 29
ID: -
OVERVIEW.FMT
. Message References: Bytes: Lines: Xref:full 215 Order of fields in overview database. overview in fields of 215 Order Subject: From: Date: list overview.fmt list • LIST (overview format) LIST (overview
NNTP (4) NNTP
Computer Center, CS, NCTU 30
cs.course.netadmin cs.course.netadmin
cs.course.netadmin
cs.course.netadmin 359 131 131 505 359 505 131 359
…
. 141 504 505 211 131 132 134 138 listgroup group group 211 a newsgroup) (select LISTGROUP GROUP (select a newsgroup) GROUP(select
NNTP (5) NNTP
Computer Center, CS, NCTU 31
\ \
[email protected]> status [email protected]> retrieved; Article [email protected]> retrieved; Article [email protected]> - - -
131
request text separately. text request request text separately. text request
stat 123 stat 423 stat 223
LAST NEXT STAT [ MessageID | Number ] | Number [ MessageID STAT
NNTP (6) NNTP last 223
Computer Center, CS, NCTU 32
\ \
ID: -
Subject: From: Date: Message References: Bytes: Lines: Xref:full
)
art )
NOV
sjwu
135 -
135 : csnews.cs.nctu.edu.tw cs.course.netadmin:131 csnews.cs.nctu.edu.tw : - Sendmail
Sendmail Xref Sendmail 135 - 135 fields follow 135 fields
- 12:25:12 +0000 (UTC)
header) specific the (retrieve ] | MessageID [ range XHDR header
XOVER [ range ] (retrieve the overview) the (retrieve ] XOVERrange [ newsgroups 131 newsgroups subject 131 subject 131
cs.course.netadmin newsgroups matches follow ( follow matches newsgroups cs.course.netadmin cs.course.netadmin subject matches follow ( follow matches subject
NNTP (7) NNTP
. 131 [HW3] 131 … 132 … 134 xover 224 . 132 134 221 131 134 134 … . xhdr 131 131 [HW3] [HW3] 132 Re: xhdr 221
Computer Center, CS, NCTU 33
mail - for -
\
Tung University Tung
Chiao
") … ")
Scotasay
20070201 (" release -
[email protected]> [email protected]> - Host: bsd5.cs.nctu.edu.tw Host: (UTC)+0000 2010 10:49:51 8 Apr Thu, Date: To: [email protected] To: - - cs.course.netadmin - : College of Computer Science, National National Science, Computer of College : 上課講解手
ID:
: Posting Posting
- - csnews.cs.nctu.edu.tw cs.course.netadmin:499 csnews.cs.nctu.edu.tw Agent: tin/1.8.3 Agent: ARTICLE [ MessageID | Number ] | Number MessageID [ARTICLE - : csnews2!csnews.cs.nctu.edu.tw!news.cs.nctu.edu.tw!not : 499
Trace: csnews2.cs.nctu.edu.tw 1270723791 1270723791 … 73853 140.113.235.135 csnews2.cs.nctu.edu.tw Trace: Complaints - - NNTP (8) NNTP
上課講解手檔案已放到網頁上。 . NNTP User Xref References: Computer Center, CS, NCTU 34 mail - for - Tung University Tung Chiao release cs.course.netadmin 上課講解手 Hsiang Liu - liuyh@NCTUCSCC HEAD [ MessageID | Number ] | Number HEADMessageID [ ] Number | MessageID BODY [ 499 : csnews.cs.nctu.edu.tw cs.course.netadmin:499 csnews.cs.nctu.edu.tw : … NNTP (9) NNTP body 499 body 222 上課講解手檔案已放到網頁上。 . Sender: Message Xref . Subject: Re: Subject: 2010 10:49:51 (UTC)+0000 8 Apr Thu, Date: National Science, Computer of College Organization: 1 Lines: head 499 head 221 csnews2!csnews.cs.nctu.edu.tw!news.cs.nctu.edu.tw!not Path: Yung From: Newsgroups: Computer Center, CS, NCTU 35 cs.test Command Command completed OK Command OK so far; send the rest of it Posting - - - 2xx 3xx x4x RFC 5536 RFC Ok, recommended ID recommended Ok, • POST (posting an article used by news clients) newsby used an article (posting POST . 240 From: [email protected] From: Newsgroups: post NNTP telnet test Subject: test ha~ post 340 NNTP (10) NNTP Computer Center, CS, NCTU 36 ihave cs.test Distribution functions - : : 10 May 2010 19:28:01 GMT 2010 19:28:01 May 10 : RFC 5536 RFC : bsd5.cs.nctu.edu.tw!liuyh : • IHAVE (exchange articles with peers) with articles (exchangeIHAVE test . 235 Newsgroups: Newsgroups: from post Subject: Date Message ihave 335 Path [email protected] From: ihave 435 Duplicate NNTP (11) NNTP Computer Center, CS, NCTU 37 25 1 y 25 0 1 y 0 1 100222 100222 000000 47 1 y 47 9 1 y 9 1 2 2 y 2 2 100401 000000 100401 cs.course.netadmin [email protected]> [email protected]> [email protected]> [email protected]> New news follows news New New newsgroups follow. newsgroups New - - - - … hhmmss [YY]YYMMDD newsgroupsNEWNEWS NEWGROUPS [YY]YYMMDD hhmmss [YY]YYMMDDNEWGROUPS Computer Center, CS, NCTU 38 tin A Usenet news reader 8 - 8 - ASCII ASCII UTF or - ASCII [ANSI1986] [ANSI1986] ASCII to UTF - 8 SHOULD 8 SHOULD NOT be used - ASCII) - bit encodings other than UTF bitencodingsother than - The character set of article bodies SHOULD articlein the set be indicatedarticleSHOULD Thecharacterbodiesof headers The US newsgroup descriptionSHOULD be in 8 But… IMHO, what is the hope? news/tin (default: US news/tin(default: Big5) (default: chinese/tin from extendsNNTP US specificationThis [RFC3629] NOT encoding.MUST use anyImplementationsother • • • • • • • • • In RFC 3977RFC In Two versions in portsin Twoversions Install (1) Computer Center, CS, NCTU 40 r‟ - NNTPSERVER=csnews2.cs.nctu.edu.tw „make install‟ is ok install‟ „make is TIN_EDITOR VISUAL EDITOR TINRC=„ • • • • • • Environment Variables Environment Back to install tin install to Back Install (2) Computer Center, CS, NCTU 41 Type) header in postings. header Type) - Big5 = 8 environment - used for MIME (Content MIME for used Screen: encoding big5 utf8 env LC_ALL=zh_TW.Big5 tin (This is for post) If youuse UTF Generate ~/.tin/ and ~/.newsrc~/.tin/Generate Charset # mm_network_charset • • Second execution Second ~/.tin/tinrc Edit First execution First Connect Start Computer Center, CS, NCTU 42 u: unsubscribe from chosen group chosen from unsubscribe u: y (yank): list all newsgroups all list y (yank): name a group for forwards/backwards search /?: group chosento subscribe s: Show some default subscribed newsgroups subscribed default some Show Search Search and subscribe newsgroups Computer Center, CS, NCTU 43 mode allmode (including headers) - r: reply through mail to through mail authorr: reply article^H: show in raw w: post an article to current group post an articlew: to current thissearchforwards/backwards article/?: within to currentf: post followup article current article (overwrite) or supersede (delete)cancel D: h: help forwards/backwardsSubjectsearch for linearticlesby/?: byauthorfor articlesaA: search forwards/backwards of all/unreadarticlesr: toggle display articlenextunreaddisplay • • • • • • • • • • • Read an article an Read Just like what you do in BBS in you do what like Just newsgroup a in Browse Read post and Computer Center, CS, NCTU 44 - - 669,687 - - - 1758,1766 - 664,668 - 914,923,930 - 1145,1164,1186 - 658,661 - 4441 - 4466 675699 1 - - - ! 1 1 ! ! 646 - 505 644,648,652,656 - - 1 newsrc 1 1 ! : : 230198 - 1 ! cat ~/. cat 546806,546808,546823,546885,546908 $ - 1 ! newsgroups subscribed/unsubscribed you articles you have read ~/.newsrc keep will • 1187,1190,1192,1252,1323,1337,1433,1444,1451,1475,1489,1512,152 0,1523,1536,1559,1572,1614,1619,1647,1687,1757 1768,1772,1781,1795,1818 1819,1906,1918,1988,1997,2012,2028,2057,2059,2142 cs.course.netadmin cs.course.sysadmin 688,694,698,703,707,799,831,868,903,913 932,968,983,1023,1054,1115,1131,1142 news.newusers.questions misc.test misc.test.moderated news.announce.newgroups news.answers liuyh@NASA news.announce.newusers After you quit tin or save configuration save or tin quit you After newsrc Computer Center, CS, NCTU 45 - mail - for - center:486 center:486 - example cs.talk To: - 徵資工系系務自動化系統開發（大學部＋研究生）助教 資工系 csnews.cs.nctu.edu.tw!news.cs.nctu.edu.tw!not : csnews.cs.nctu.edu.tw cs.u95:2369 cs.u96:1625 cs.u97:3571 cs.u97:3571 cs.u96:1625 cs.u95:2369 csnews.cs.nctu.edu.tw : cs.talk:2518 nctu.talk:1472787 cs.talk:2518 Subject: Subject: Followup Xref cs.computer cs.g98:85 cs.g97:317 cs.u98:1490 cs.phd:816 cs.course.sysadmin:2142 cs.course.netadmin:491 Path: Path: From: cs.u95,cs.u96,cs.u97,cs.u98,cs.g97,cs.g98,cs.computer Newsgroups: center,cs.course.netadmin,cs.course.sysadmin,cs.phd,cs.talk,nctu.talk crosspost posting After Before posting Before A A Computer Center, CS, NCTU 46