DOCSLIB.ORG

Forecasting Disk Resource Requirements for a Usenet Server

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Forecasting Disk Resource Requirements for a Usenet Server The following paper was originally presented at the Seventh System Administration Conference (LISA ’93) Monterey, California, November, 1993 Forecasting Disk Resource Requirements for a Usenet Server Karl L. Swartz Standford Linear Accelerator Center For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. Email: [email protected] 4. WWW URL: http://www.usenix.org Forecasting Disk Resource Requirements for a Usenet Server† Karl L. Swartz – Stanford Linear Accelerator Center ABSTRACT Three years ago the Stanford Linear Accelerator Center (SLAC) decided to embrace netnews as a site-wide, multi-platform communications tool for the laboratory’s diverse user community. The Usenet newsgroups as well as other world-wide newsgroup hierarchies were appealing for their unique ability to tap a broad pool of information, while the availability of the software on a number of platforms provided a way to communicate to and amongst the computing community. The previous way of doing this ran only on the VM mainframe system and had become increasingly ineffective as users migrated to other platforms. The increasing dependence on netnews brought with it the requirement that the service be reliable. This was dramatically demonstrated when the long-neglected netnews service collapsed under the load of the traditional fall surge in Usenet traffic and the site was without news service for a week while an upgraded system was installed. One result of that painful event was that efforts were made to forecast growth and the accompanying hardware requirements so that equipment could be acquired and installed before problems became visible to the users. This paper describes the major on-disk databases associated with news software, then presents an analysis of the storage requirements for these databases based on data collected at SLAC. A model is developed from this data which permits forecasting of disk resource requirements for a full feed as a function of time and local policies. Suggestions are also made as to how to modify this model for sites which do not carry a full feed. Why NetNews? Why Usenet? SLAC.) BITNET mailing lists provided the main contact with researchers at other sites. The Stanford Linear Accelerator Center (SLAC) is a medium-sized national research labora- There was also a smaller, though still sizable, tory. The lab’s primary missions are research in ele- group of users who used VAX/VMS systems most of mentary particle physics and development of new the time, if not exclusively. These users tended to techniques in particle accelerators. These are large be excluded from VM News and from CONSPIRE, projects that often involve international collabora- though they did make use of BITNET. DECnet mail tions and a diverse user community. This can with other physics sites was also available. This present a formidable problem for communication communications block posed some problems but was with and amongst users, from discussions on the tolerated at the time. design of new experiments to progress reports on Over the past few years the computing environ- current experiments, as well as the more mundane ment became far more diversified. Unix worksta- but equally necessary announcements of network or tions began to appear, while the PCs, Macintoshes, server outages. There is also a tremendous need to and Amigas grew powerful enough that their users stay in touch with what other researchers are doing. had dwindling need for VM. The number of users In the eighties, the majority of computer users who did not use VM on a regular basis, if at all, at SLAC logged onto an IBM mainframe running increased until the VM-based communication model VM. The default user profile on VM brought up a began to fail completely. A new model was needed VM News session upon login, where announcements that would permit users to read announcements and could be made. Most users would see them since participate in discussions from whatever platform they logged into VM regularly. Discussions were they were accustomed to using. The multitude of handled by mailing lists maintained by LISTSERV, platforms made another home-grown solution plus a home-grown VM conferencing system named undesirable. The software which supports Usenet CONSPIRE. (VM News was also developed at (referred to hereafter as netnews software to distin- guish it from the network itself), with ready availa- bility of NNTP-based readers for a variety of plat- †This work supported by the United States Department forms, seemed to solve the problem except for VM. of Energy under contract number DE-AC03-76SF00515, The discovery of the PennState VM NetNews system and simultaneously published as SLAC PUB-6353. completed the solution. 1993 LISA – November 1-5, 1993 – Monterey, CA 195 Forecasting Disk Resource Requirements for a Usenet Server Swartz B News and NNTP software was built and require a substantial amount of disk space and which installed on a Sun fileserver which had some spare fluctuate in size as a function of news activity. The disk space, and the PennState NetNews software, three primary databases are the articles themselves, with NNTP software from Queen’s University, was the history file, and the active file. Ancillary struc- installed on VM.1 Once the bugs were worked out of tures include thread databases, incoming and outgo- this system, NNTP-based readers were acquired for ing spool areas, and log files. While the following other platforms, often with help from interested description is based on a Unix system running the C users. Meanwhile, a local slac newsgroup hierarchy News software, most structures will likely be similar was being populated with new groups and users were on other systems. being introduced to the new system. The article database is by far the largest portion There was also a great deal of interest in non- of a news system. Unix’s hierarchical directory local groups, of course, i.e., Usenet. After a great structure is a handy analog to the newsgroup hierar- deal of debate over proper use of government-funded chy, so the software uses a simple lexical mapping equipment, the appropriateness of censorship in an of newsgroup names into directory names (‘‘.’’ is academic/research community, and the feasibility of mapped to ‘‘/’’) with each article stored in a determining just what was and was not appropriate, separate file. For example, article 42 in group it was decided to carry all groups and assume a news.announce.important is stored as mature and responsible user community. news/announce/important/42 within the Netnews flourished and mostly solved the com- spool directory (often /usr/spool/news). munication problem, but created a new problem in Cross-posted articles are handled by links. These that users came to expect it to be reliable. By early are normally hard links, though there is some sup- 1992 the ever-increasing growth of Usenet traffic port for falling back to symbolic links if a hard link 2 had begun to severely strain the resources of the Sun fails. which was trying to run netnews while also handling Storing each article in a separate file implies a file service and a variety of other tasks. Spool areas tremendous number of relatively small files. If one would overflow and the now-obsolete B News eschews the apparently lightly tested symlink sup- software would collapse, causing substantial delays port – an option not likely to be available for large and user ire. The expiration noose would be netnews servers for much longer – the use of links tightened another notch, staving off disaster for a bit for cross-posted articles further implies that this longer but also aggravating already irate users as entire database must reside within a single disk par- articles expired before they could be read. tition. Unfortunately, the combination of cross- Work was begun to determine what equipment posts, cancellations, varying expiration times for dif- was needed to support netnews for the next few ferent newsgroups, and the common desire to be able years, and to define requirements in the form of to use grep and other Unix tools on the article data- minimum expiration times. Eventually a base make an alternative implementation difficult SPARCserver 2 with 4.5 GB of disk space was and thus unlikely in the near future. (This is a ordered, arriving mere days too late to avert the col- recurrent thread in the news.software.* newsgroups.) lapse of SLAC’s netnews service from the traditional The history file, typically located in September surge in Usenet traffic. While painful at /usr/lib/news/history, records the article ID the time, the degree of pain made it clear just how of each article seen recently by the news system critical netnews had become to SLAC. along with the time it was received, any explicit Popular reference books on managing Usenet expiration time, and, if the article has not yet are notably silent on the matter of forecasting expired, a list enumerating the newsgroups the arti- resources. [1, 2] Therefore, a study of Usenet growth cle appears in and its sequence number in each was begun and has continued, so that future growth group. All of this is stored in a simple, albeit large, needs can be anticipated and handled before another file with one article mentioned per line. In order to crisis occurs. This paper documents the current state speed lookups by article ID, an index is maintained, of this ongoing study. typically using dbz. The last of the primary news databases is the Organization of a Netnews Server’s Data active file, typically /usr/lib/news/active, A netnews server is composed of a number of which lists each newsgroup known to the news sys- databases, several of which have the potential to tem and, for each group, the range of article 1VM NetNews is a full news system, not an NNTP- 2The documentation for the February 1993 C News based reader.
Load more

Recommended publications
  • The Freedom of Speech at Risk in Cyberspace: Obscenity Doctrine and a Frightened University's Censorship of Sex on the Internet
    NOTES THE FREEDOM OF SPEECH AT RISK IN CYBERSPACE: OBSCENITY DOCTRINE AND A FRIGHTENED UNIVERSITY'S CENSORSHIP OF SEX ON THE INTERNET JEFFREY E. FAUCETrE INTRODUcTION On November 8, 1994, Carnegie Mellon University (CMU) removed' a small handful of topics from among the thousands of Usenet newsgroups2 subscribed to by the university computer sys- tem and available on the Internet' .because they contained en- coded sexually explicit images Under the new policy announced by Erwin Steinberg, university vice provost for education, none of the university's 9,000 computers would list the approximately forty newsgroups.5 The newsgroups that were censored are all known as "binaries,"6 and they contain, among other things, encoded imag- 1. Despite the University's restriction on the sexually explicit newsgioups, techno- logically adept students can circumvent the policy by accessing the groups through other file servers. However, for the purposes of this Note, the efforts of these enterprising students are not important when compared with the symbolic effect of the censorship. 2. These newsgroups are "bulletin board-style discussion groups" that can be read from, responded to, and downloaded to an individual's computer. See David Landis, Ex- ploring the Online Universe, USA TODAY, Oct. 7, 1993, at 4D. The Usenet newsgroups are available worldwide via Internet. See id. 3. The Internet is the most commonly known "wide area network" (WAN). It "evolved from networks established by the Department of Defense and the National Science Foundation. [The] Internet connects various government, university, and corporate entities, spans 137 nations, and has at least fifteen million users." Eric Schlachter, Cyberspace, the Free Market and the Free Marketplace of Ideas: Recognizing Legal Differ- ences in Computer Bulletin Board Functions, 16 HASTINGS COMM.
    [Show full text]
  • Cyber-Synchronicity: the Concurrence of the Virtual
    Cyber-Synchronicity: The Concurrence of the Virtual and the Material via Text-Based Virtual Reality A dissertation presented to the faculty of the Scripps College of Communication of Ohio University In partial fulfillment of the requirements for the degree Doctor of Philosophy Jeffrey S. Smith March 2010 © 2010 Jeffrey S. Smith. All Rights Reserved. This dissertation titled Cyber-Synchronicity: The Concurrence of the Virtual and the Material Via Text-Based Virtual Reality by JEFFREY S. SMITH has been approved for the School of Media Arts and Studies and the Scripps College of Communication by Joseph W. Slade III Professor of Media Arts and Studies Gregory J. Shepherd Dean, Scripps College of Communication ii ABSTRACT SMITH, JEFFREY S., Ph.D., March 2010, Mass Communication Cyber-Synchronicity: The Concurrence of the Virtual and the Material Via Text-Based Virtual Reality (384 pp.) Director of Dissertation: Joseph W. Slade III This dissertation investigates the experiences of participants in a text-based virtual reality known as a Multi-User Domain, or MUD. Through in-depth electronic interviews, staff members and players of Aurealan Realms MUD were queried regarding the impact of their participation in the MUD on their perceived sense of self, community, and culture. Second, the interviews were subjected to a qualitative thematic analysis through which the nature of the participant’s phenomenological lived experience is explored with a specific eye toward any significant over or interconnection between each participant’s virtual and material experiences. An extended analysis of the experiences of respondents, combined with supporting material from other academic investigators, provides a map with which to chart the synchronous and synonymous relationship between a participant’s perceived sense of material identity, community, and culture, and her perceived sense of virtual identity, community, and culture.
    [Show full text]
  • Against Cyberanarchy"
    AGAINST "AGAINST CYBERANARCHY" By David G. Posit TABLE OF CONTENTS I. IN TRO DUCTION .....................................................................................................1365 II. UNEXCEPTIONALISM IN CYBERSPACE ................................................................... 1366 III. SETTLED PRINCIPLES ............................................................................................ 1371 IV . FUNCTIONAL IDENTITY ......................................................................................... 1373 V . SC A LE ...................................................................................................................1376 V I. E FFEC TS ................................................................................................................ 138 1 V II. CON SEN T ..............................................................................................................1384 V III. CONCLUDING THOUGHTS .....................................................................................1386 I. INTRODUCTION It makes me indignant when I hear a work Blamed not because it's crude or graceless but Only because it's new... Had the Greeks hated the new the way we do, Whatever would have been able to grow to be old?' Professor Jack Goldsmith's Against Cyberanarchy2 has become one of the most influential articles in the cyberspace law canon. The position he sets forth-what I call "Unexceptionalism"-rests on two main premises. The first is that activity in cyberspace is "functionally identical to
    [Show full text]
  • Liminality and Communitas in Social Media: the Case of Twitter Jana Herwig, M.A., [email protected] Dept
    Liminality and Communitas in Social Media: The Case of Twitter Jana Herwig, M.A., [email protected] Dept. of Theatre, Film and Media Studies, University of Vienna 1. Introduction “What’s the most amazing thing you’ve ever found?” Mac (Peter Riegert) asks Ben, the beachcomber (Fulton Mackay), in the 1983 film Local Hero. ”Impossible to say,” Ben replies. “There’s something amazing every two or three weeks.” Substitute minutes for weeks, and you have Twitter. On a good day, something amazing washes up every two or three minutes. On a bad one, you irritably wonder why all these idiots are wasting your time with their stupid babble and wish they would go somewhere else. Then you remember: there’s a simple solution, and it’s to go offline. Never works. The second part of this quote has been altered: Instead of “Twitter”, the original text referred to “the Net“, and instead of suggesting to “go offline”, author Wendy M. Grossman in her 1997 (free and online) publication net.wars suggested to “unplug your modem”. Rereading net.wars, I noticed a striking similarity between Grossman’s description of the social experience of information sharing in the late Usenet era and the form of exchanges that can currently be observed on micro-blogging platform Twitter. Furthermore, many challenges such as Twitter Spam or ‘Gain more followers’ re-tweets (see section 5) that have emerged in the context of Twitter’s gradual ‘going mainstream’, commonly held to have begun in fall 2008, are reminiscent of the phenomenon of ‘Eternal September’ first witnessed on Usenet which Grossman reports.
    [Show full text]
  • Usenet News HOWTO
    Usenet News HOWTO Shuvam Misra (usenet at starcomsoftware dot com) Revision History Revision 2.1 2002−08−20 Revised by: sm New sections on Security and Software History, lots of other small additions and cleanup Revision 2.0 2002−07−30 Revised by: sm Rewritten by new authors at Starcom Software Revision 1.4 1995−11−29 Revised by: vs Original document; authored by Vince Skahan. Usenet News HOWTO Table of Contents 1. What is the Usenet?........................................................................................................................................1 1.1. Discussion groups.............................................................................................................................1 1.2. How it works, loosely speaking........................................................................................................1 1.3. About sizes, volumes, and so on.......................................................................................................2 2. Principles of Operation...................................................................................................................................4 2.1. Newsgroups and articles...................................................................................................................4 2.2. Of readers and servers.......................................................................................................................6 2.3. Newsfeeds.........................................................................................................................................6
    [Show full text]
  • The Internet Is a Semicommons
    GRIMMELMANN_10_04_29_APPROVED_PAGINATED 4/29/2010 11:26 PM THE INTERNET IS A SEMICOMMONS James Grimmelmann* I. INTRODUCTION As my contribution to this Symposium on David Post’s In Search of Jefferson’s Moose1 and Jonathan Zittrain’s The Future of the Internet,2 I’d like to take up a question with which both books are obsessed: what makes the Internet work? Post’s answer is that the Internet is uniquely Jeffersonian; it embodies a civic ideal of bottom-up democracy3 and an intellectual ideal of generous curiosity.4 Zittrain’s answer is that the Internet is uniquely generative; it enables its users to experiment with new uses and then share their innovations with each other.5 Both books tell a story about how the combination of individual freedom and a cooperative ethos have driven the Internet’s astonishing growth. In that spirit, I’d like to suggest a third reason that the Internet works: it gets the property boundaries right. Specifically, I see the Internet as a particularly striking example of what property theorist Henry Smith has named a semicommons.6 It mixes private property in individual computers and network links with a commons in the communications that flow * Associate Professor, New York Law School. My thanks for their comments to Jack Balkin, Shyam Balganesh, Aislinn Black, Anne Chen, Matt Haughey, Amy Kapczynski, David Krinsky, Jonathon Penney, Chris Riley, Henry Smith, Jessamyn West, and Steven Wu. I presented earlier versions of this essay at the Commons Theory Workshop for Young Scholars (Max Planck Institute for the Study of Collective Goods), the 2007 IP Scholars conference, the 2007 Telecommunications Policy Research Conference, and the December 2009 Symposium at Fordham Law School on David Post’s and Jonathan Zittrain’s books.
    [Show full text]
  • Sending Newsgroup and Instant Messages
    7.1 LESSON 7 Sending Newsgroup and Instant Messages After completing this lesson, you will be able to: Send and receive newsgroup messages Create and send instant messages Microsoft Outlook offers some alternatives to communicating via e-mail. With instant messages, you can connect with a client who is also using instant messaging to ask a simple question and get the answer immediately. On the other hand, Newsgroups allow you to communicate with a larger audience. When considering the purchase of a new graphics program, for example, you can solicit advice and recommendations by posting a message to a newsgroup for people interested in graphics. You don’t need any practice files to work through the exercises in this lesson. Sending and Receiving Newsgroup Messages A newsgroup is a collection of messages related to a particular topic that are posted by individuals to a news server . Newsgroups are generally focused on discussion of a particular topic, like a sports team, a hobby, or a type of software. Anyone with access to the group can post to the group and read all posted messages. Some newsgroups have a moderator who monitors their use, but most newsgroups are not overseen in this way. A newsgroup can be private, such as an internal company newsgroup, but there are newsgroups open to the public for practically every topic. You gain access to newsgroups through a news server maintained either on your organization’s network or by your Internet service provider (ISP). You use a newsreader program to download, read, and reply to news messages.
    [Show full text]
  • Feed Your ‘Need to Read’ – an Explanation of RSS
    by Jen Sharp, jensharp.com Feed your ‘Need To Read’ – an explanation of RSS here exists a useful informational tool so powerful that provides feeds. For example, USA Today has separate that the government of China has completely feeds in its different sections. You know that a page has the banned its use since 2007. Yet many people have capabilities for RSS subscriptions by looking for the orange Tnot even heard about it, despite its juxtaposition in chicklet in the toolbar of your browser. Clicking on the logo front of our attention during our experiences on the web. will start you in the process of seeing what the feed for that Recently I was visiting with my dad, who is well-informed page looks like. Then you can choose how to subscribe. and somewhat technically savvy. I asked him if he had ever Choosing a reader can seem overwhelming, as there are heard of RSS feeds. He hadn’t, so when I pointed out the thousands of readers available, most of them for free. To ever-present orange “chicklet,” he remarked, “Oh! I thought start, decide how you what to view the information. RSS that was the logo for Home Depot!” feeds can be displayed in many ways, including: Very simply, RSS feeds are akin to picking up your ■ sent to your email inbox favorite newspaper and scanning the headlines for a story ■ as a personal browser start page like igoogle.com or you are interested in. Items are displayed with a synopsis and my.yahoo.com links to the full page of information.
    [Show full text]
  • RE-SONANCE---Inov-Em---Be-R-1-9
    GENERAL I ARTICLE Web Search Engines How to Get What You Want from the World Wide Web T B Rajashekar The world wide web is emerging as an all-in-one infonnation source. Tools for searching web-based information includes search engines, subject directories and meta search tools. We take a look at the key features of these tools and suggest-practical hints for effective web searching. Introduction T B Rajashekar is with the National Centre for We have seen phenomenal growth in the quantity of published Science Information information (books, journals, conferences, etc.) since the (NCSI), Indian Institute of Science, Bangalore. industrial revolution. Varieties of print and electronicindices His interests include text have been developed to retain control over this growth and to retrieval, networked facilitate quick identification of relevant information. These information systems and include library catalogues (manual and computerised), services, digital collection development abstracting and indexing journals, directories and bibliographic and managment and databases. Bibliographic databases emerged in the 1970's as an electronic publishing. outcome ofcomputerised production of abstracting and indexing journals like Chemical Abstracts, Biological Abstracts and Physics Abstracts. They grew very rapidly, both in numbers and coverage (e.g. news, commercial, business, financial and technology information) and have been available for online access from database host companies like Dialog and STN International, and also delivered on electronic media like tapes, diskettes and CD-ROMs. These databases range in volume from a few thousand to several million records. Generally, content of these databases is of high quality, authoritative and edited. By employing specially developed search and retrieval programs, these databases could be rapidly scanned and relevant records extracted.
    [Show full text]
  • Congressional Record—Senate S9017
    June 26, 1995 CONGRESSIONAL RECORD — SENATE S9017 the problems of decolonization. It has After this history, the image of puter networks, I believe Congress outlived its purpose. Rather than President Nelson Mandela—a man im- must act and do so in a constitutional search for a new purpose for this Coun- prisoned for 27 years in his fight manner to help parents who are under cil, we should ask whether it should against apartheid—handing the World assault in this day and age. There is a exist at all. Cup trophy to the white captain of the flood of vile pornography, and we must Mr. President, the other major area rugby team is indeed a powerful sym- act to stem this growing tide, because, for reform is in our thinking about bol of the dramatic changes in South in the words of Judge Robert Bork, it what the United Nations is and what Africa. Throughout the country, whites incites perverted minds. I refer to its role should be in American foreign and blacks alike celebrated the victory Judge Bork from the Spectator article policy. We cannot expect the United of the Springboks, the mascot of the that I have permission to insert in the Nations to be clearer in purpose than is national team. RECORD. its most powerful member state. Mr. President, I join with the inter- My bill, again, is S. 892, and provides At its core, the United Nations is a national community in congratulating just this sort of constitutional, nar- collection of sovereign states and is be- the people of South Africa on winning rowly focused assistance in protecting holden to them for guidance, funding, the rugby World Cup.
    [Show full text]
  • Newsreaders.Com: Getting Started: Netiquette
    Usenet Netiquette NewsReaders.com: Getting Started: Netiquette I hope you will try Giganews or UsenetServer for your news subscription -- especially as a holiday present for your friends, family, or yourself! Getting Started (Netiquette) Consult various Netiquette links The Seven Don'ts of Usenet Google Groups Posting Style Guide Introduction to Usenet and Netiquette from NewsWatcher manual Quoting Style by Richard Kettlewell Other languages: o Voila! Francois. o Usenet News German o Netiquette auf Deutsch! Some notes from me: Read a newsgroup for a while before posting o By reading the group without posting (known as "lurking") for a while, or by reading the group archives on Google, you get a sense of the scope and tone of the group. Consult the FAQ before posting. o See Newsgroups page to find the FAQ for a particular group o FAQ stands for Frequently Asked Questions o a FAQ has not only the questions, but the answers! o Thus by reading the FAQ you don't post questions that have been asked (and answered) ad infinitum See the configuring software section regarding test posts (only post tests to groups with name ending in ".test") and not posting a duplicate post in the mistaken belief that your initial post did not go through. Don't do "drive through" posts. o Frequently someone who has an urgent question will post to a group he/she does not usually read, and then add something like "Please reply to me directly since I don't normally read this group." o Newsgroups are meant to be a shared experience.
    [Show full text]
  • Back to the Future: Tracing the Roots and Learning Affordances of Social Software
    1 Chapter 1 Back to the Future: Tracing the Roots and Learning Affordances of Social Software Nada Dabbagh George Mason University, USA Rick Reo George Mason University, USA ABSTRACT This chapter provides a developmental perspective on Web 2.0 and social software by tracing the historical, theoretical, and technological events of the last century that led to the emergence—or re- emergence, rather—of these powerful and transformative tools in a big way. The specific goals of the chapter are firstly, to describe the evolution ofsocial software and related pedagogical constructs from pre- and early Internet networked learning environments to current Web 2.0 applications, and secondly, to discuss the theoretical underpinnings of social learning environments and the pedagogical implica- tions and affordances of social software in e-learning contexts. The chapter ends with a social software use framework that can be used to facilitate the application of customized and personalized e-learning experiences in higher education. INTRODUCTION with their use to augment computational and com- munication capabilities and foster collaboration Is social software merely a continuation of a broad and social interaction, and progressing to their class of older computer-mediated communica- Web 2.0-enabled information aggregation capa- tion (CMC) and collaboration tools, or does it bilities. Throughout this chronological depiction, represent a significant transformation of social we emphasize the socio-pedagogic affordances interaction capabilities? In this chapter, we trace and implications of social software. We conclude the evolution of social software tools, beginning the chapter with a social software use continuum to guide the design of e-learning experiences in academic contexts.
    [Show full text]