Coding with Lisp Coding with Lisp

‘A hip, burgeoning free software community is currently pushing Lisp, arguably the oldest case for Lisp than coder and writer Paul Graham (see Hackers and complete theory of computation within Lisp, is a Painters, LU&D 42), whose own work with Viaweb is good testament rich seam to mine, with demanding papers on programming language in common use, to the fore as a next generation coding platform. Self- both to the power of Lisp as an expressive language and Lisp’s this subject furnished by the likes of Graham, relevance within enterprise level, large-scale projects. Graham’s essays Sussman and Guy Steele, another major player in confessed Lisp newbie, Martin Howse, presents the fi rst part of an accessible guide to the and books provide a singular practical and conceptual resource, the early life of Lisp, and co-inventor of the elegantly tying real world arguments within a neat conceptual intriguing Connection Machine. culture of this fl exible language, and practical implementations under GNU/Linux’ bundle, which is well wrapped up by his strong theoretical grounding and knowledge of the language. He speaks from experience, and MORE TO LISP THAN LISTS those who paraphrase Graham without his depth of understanding Extendibility here means both extending the Lisp will always sound a bit hollow. Nevertheless, Lisp’s features and language itself in a completely transparent advantages must be outlined, and these can readily be pinned down manner, thus building a domain specifi c both to a decent level of abstraction and to highly usable abstractions language for the application, and providing the themselves, such as closures and macros. It’s worth remembering that facility for others to readily extend and customise Lambda The Ultimate Lisp belongs within the family of functional languages, which implies an application. In the fi rst instance, the fi eld of equivalence between software and data allows The art of Lisp, or coding as philosophy for coding custom languages with powerful abstractions, and in the latter case, this form of what Graham calls bottom-up programming naturally results in extensible software, with GNU Emacs as prime example here. From this perspective, Lisp isn’t about writing applications, it’s about writing languages. And, given the growing complexity of both hardware and contemporary software systems, this heavily modular, high level approach to programming and systems architecture is seriously compelling. Indeed, pioneers Sussman and Abelson preface their essential SICP volume with clear indications Coder’s dream or of how to control complexity, through user’s nightmare? establishing conventional interfaces, and Symbolics Genera establishing new languages. environment was a And, alongside those who claim that Lisp is Lisp hacker’s unusable due to an arcane syntax which paradise, allowing multiplies parentheses, some would still maintain coders to dig deep that Lisp is slow, and argue thus that the need into Lisp hardware for speed has lost out in the battle with complexity. But, given that nearly all modern

good points. To hijack Python guru, Tim Peter’s, words, “Good modularity, an essential requirement for Good languages aren’t random collections of languages aren’t random collections of interchangeable features: they large-scale, complex projects. Such have a philosophy and internal coherence that’s never profi tably abstractions, and features such as automatic interchangeable features: they have a philosophy isp has come in from the cold. After years enduring the icy confused with their surface features.” Understanding Lisp is a gradual memory management, courtesy of built-in Artifi cial Intelligence (AI) winter, a period in the late eighties process, and involves some degree of immersion in the history, garbage collection, readily enable new ways and internal coherence that’s never profi tably Lwhen Lisp was dumped by many thanks to associations with philosophy and culture of Lisp as well as a gentle involvement with of programming, and this is Lisp’s great confused with their surface features over-hyped AI research, and a frosty decade in academic obscurity, coding to further elucidate core concepts such as recursion and advantage. the open source community has dusted down this unique language, macros. Yet, Lisp, with an array of dialects, libraries and Sure, the semi-interpreted nature of Lisp, with functions able to be implementations compile during interpretation, and freshened her up with lemon odour. Or rather Lemonodor.com, a implementations is far from the easiest language to get up and tried and tested at the interactive REPL (Read Eval Print Loop) prompt, following a just-in-time model, or allow high profi le Web site which actively promotes and describes Lisp use running with, in contrast to Python, which obviously presents a one or so called top level, assists in rapid development, and, Graham specifi cally for interactive compilation, speed is and can readily be considered the Slashdot of the Lisp world, has stop shop. And although a clear conceptual understanding can be proffers amusing examples of improving server software on the fl y, rarely an issue. And the speed of prototyping or performed this service for the community. readily sought in printed works such as The Little Lisper or in Abelson but Lisp’s real advantage remains in its extensibility which can readily coding is a further issue which should also enter Like a proud mother, Lisp, probably the oldest language still in use and Sussman’s seminal Structure and Interpretation of Computer be seen as stemming from a core feature of the language; the fact the equation. With Lisp, at least one has greater today, can view her children and remark on how their best features all Programs (SICP), practical advice which would further support this that Lisp programs are expressed as Lisp data structures. Indeed, John options for exploration and prototyping, and, if stem from her genes or rather memes. Flexibility and speed of knowledge is sorely lacking within the community. As we’ll see, McCarthy, a key fi gure within AI and inventor and primary needs be, optimisations can be furnished later in development are her watchwords, and key enabling features of the certain initiatives are under way to resolve these issues, but aside from implementor of Lisp, remarks in an essay on the early history of Lisp, the day. Such notions regarding Lisp as sluggish language, such as run-time typing and an interactive approach to evangelising Lisp, an effort which is close to achieving its goals, more that “One can even conjecture that Lisp owes its survival specifi cally belong within an old-fashioned view of Lisp interpretation or compilation have been adopted by newer languages work does need to be done in this direction, and this short article to the fact that its programs are lists, which everyone, including me, which focuses on syntax and on lists, arguing such as Python. Yet, Python misses the killer core or inner beauty only aims to provide an overview for the intelligent and necessarily has regarded as a disadvantage.” Hacker, Pascal Constanza, argues that LISP stands for List Processing Language, which so many celebrate in Lisp. intrigued beginner. that this core feature makes of Lisp much more than just another and which corrals this powerful language within To approach this core takes more than simply enumerating Lisp’s No-one has been more instrumental in persuasively arguing the Turing complete language, and this notion, which embeds a an academic prison. The new view of Lisp is that,

50 LinuxUser & Developer LinuxUser & Developer 51 Coding with Lisp

given automatic highlighting and indentation, and Sussman’s work. Indeed, in their heyday, these pioneers were parentheses and other syntactical issues churning out meta-circular evaluators for subsets and dialects of Lisp disappear. Lisp is a far more fl exible language at an alarming rate, and their work forms an important link between than the acronym would suggest. the more exciting aspects of mathematics, philosophy and computer science. Another valuable starting point here would be the common GETTING META assertion that the proof of Godel’s incompleteness theorem, which is A further consequence of the fl attening of essential to an understanding of AI, would have been easier if he had software and data as quite distinct animals, is invented Lisp fi rst, given Lisp’s predilection for the meta-circular. And that Lisp really does live in the land of the meta, just before any unthinking C coders chime in, a C compiler written in and that’s a place where a good few C, which can be used for bootstrapping, does not belong in the realm sophisticated coders and theorists like to hang of the metacircular which further specifi es that precise semantics must out. Douglas Hofstadter, in his seminal mathematical and meta-math work, Godel, Escher, Bach: An Eternal Golden Braid, provides Another consequence of the much vaunted many mind stimulating adventures at the meta- data/software equivalence, is the unusual level, and Lisp makes for a very natural fi t here. And yet another consequence of the much defi ning quality that Lisp can be written in itself

not be defi ned in the evaluator. The common parallel is with looking up a word in the dictionary, and fi nding that the defi nition uses the original word. That is how things work with a Lisp written in Lisp; eval, which quite obviously evaluates expressions, is implemented by calling eval. In contrast, a C compiler must specify detailed and precise semantics for each and every construct, and take care of boring old parsing. The REPL defi nes all that is needed to build a Lisp interpreter; read an expression, evaluate it and then print the results. It has to be admitted that there’s a certain beauty and simplicity at work here, and Lisp is certainly unique in this respect. A good deal of this simplicity stems from Lisp’s roots, with the theoretical work of John McCarthy in the 1950s, which touches on all the rich L-Lisp, running with an OpenGL Lisp library, puts some life back into a code heavy subject, thematics wrapped up by Godel’s with Lindenmayer systems readily simulating plants and fractals work in the sphere of mathematics. Both McCarthy and Graham write well on this early history of the language, and their texts make for essential vaunted data/software equivalence, is the reading. McCarthy did not set out to design and create a Lisp lives in unusual defi ning quality that Lisp can be written programming language to meet specifi c programming needs or in itself. In practice, this can be achieved in very the land of the satisfy a problem domain, rather he was interested in mathematical few lines of code, and such a resulting beast is meta, and notation and expressing theory. This makes Lisp unique in the fi eld of the rather frightening metacircular interpreter or that’s a place programming, and quite distinct from the functionality associated metacircular evaluator. This creature lies at the with C or C++. Lisp is a fl exible, theoretical language which is where a very heart of an understanding of the history and primarily expressive. conceptual underpinnings of Lisp, and writing good few Rather than jogging through the history of Lisp, which is well such an interpreter forms a useful exercise for the sophisticated rehearsed elsewhere by the likes of McCarthy and Steele, Graham’s novice. Again this is another rich seam well coders and Roots of Lisp paper presents a conceptual walk-through of the birth of worthy of further investigation, and the intrigued Lisp, with McCarthy’s notation translated into Common Lisp code, theorists like could start with Graham’s excellent paper, The and along the way he provides a good description of the primitive Roots of Lisp, or plough straight into Abelson to hang out Lisp forms, which are function calls or macros, before arriving at the

52 LinuxUser & Developer Coding with Lisp Coding with Lisp

The Lemonodor effect detailed explication of an of the huge On Lisp volume to macros, which he considers of great than Common Lisp, more elegant and crystalline, as opposed to the can fi re up, say, CLISP, a pleasant, simple The Common Lisp community has certainly eval function written in importance within the paradigm of bottom-up programming, and baroque, full features of Common Lisp. But there are great Common Lisp interpreter, straight from the gone from strength to strength in the last Lisp. Graham describes macros are certainly an essential, if hard to learn, feature which allows similarities between the languages, and core features of Scheme command line and throw a few Lisp expressions two years, with blogs and wikis as primary Lisp’s elegant syntax and for writing programs that write programs. Macros are quite simply such as continuations, which freeze the state of a computation for at it, but to get some decent work done you’ll medium for the sharing of enthusiasm, notation, and key terms functions which transform expressions and they can themselves call later use, can readily be achieved with macros under Common Lisp. need a more powerful implementation which inspiration and information. There are blogs by and for newbies, and blogs from old such as expression, form, other functions and make use of other macros; a heady brew indeed Once again, it seems that languages cannot so readily be boiled integrates well with an editor such as GNU hands such as Rainer Joswig, and sometime list and atom. Alongside whose power of transformation is unheard of elsewhere. To clear up down to a feature set. Emacs to form an effi cient IDE. In part LU&D contributor, Daniel Barlow. However, six primitive operators, any confusion, macros under Lisp have little to do with their C-based The relative size of two, we’ll pit SBCL against CMUCL, the the Slashdot of all Lisp weblogs, if such a thing can be imagined, is surely Lemonodor, the quote operator, namesakes, which perform purely string substitutions. Macros allow Scheme is also an Lushly produced, with two favoured free implementations, clearly signalling that Lisp is the new rock which has obvious the language to play with its own readily accessible internals as data, issue. Given that Lisp integrate these tightly with GNU Emacs and roll with a mix of hardcore news from parallels with quotation and a good many Common Lisp functions are implemented as doesn’t really bother intriguing illustrations often using a touch of SLIME, throw the Lisp community, John Wiseman’s artistic in the English language, macros themselves. Understanding macros is one thing, and making about the difference packaging and packages into the mix LA lifestyle reports, and a marked emphasis loosely associated with the on Lisp within the fi eld of robotics. is well described as use of them perhaps even more complex, but given the defi nition between built-in and touch on embedding and Lushly produced, with intriguing functioning to distinguish that macros are simply operators that are implemented by functions and user- topic at hand, Lemonodor is a extending with Lisps such as librep and illustrations often loosely associated with the data from code. The transformation, and noting a few example expansions which can defi ned functions, it’s functionalities such as FFI (Foreign topic at hand, Lemonodor is a legend in the lambda notation for readily be tested with the macroexpand-1 function should set the a tough call to decide legend in the land of Lisp Function Interface). land of Lisp. And a mention by Wiseman of a site, surely results in record traffi c, if not of denoting functions is beginner on the right track. Slashdotting proportions. Even one of the best free Common Lisp IDEs, SLIME, makes Scheme can be viewed as a more minimal mention of achieving Lemonodor fame in the startup routines. Inspired by Planet GNOME, Planet Lisp acts as meta blog, collecting language than Common Lisp, more elegant and essential content from a huge number of Lisp- related weblogs. Despite sometimes straying crystalline, as opposed to the baroque, full from the Lisp path into territory which indeed may rival the poor signal-to-noise ratios of features of Common Lisp Slashdot, Planet Lisp does make for a decent daily immersion in Lisp culture. clearly elaborated, and SCHEMING With plentiful RSS feeds, and much cross with great elegance, Though Lisp’s history post-McCarthy does make for interesting linking the Lisp world is well connected, and the greatest resource of them all, cliki.net Graham whips out a reading, with colourful anecdotes peppering the story of computer collects much of these links and blogs and surprise Lisp eval, written science’s most philosophical language and furnishing a classic provides further essential resources. Cliki.net, using functions built from narrative of riches to rags and back again, there is little here that is powered by Daniel Barlow’s CLiki engine, a only seven primitives. totally relevant to the contemporary Lisper, aside perhaps from Common Lisp wiki, is well worth checking out changelog-wise on a daily basis. The practical Further functions can be intriguing material covering the hardware implemented Lisp lisp page is a useful starting point for newbies, elaborated and evaluated Machines and their associated development environments such as if somewhat dated in respect to SLIME use, using this eval, which can Genera, which few contemporary IDEs can even dream of but cliki.net does provide links or information readily be transformed competing with. It’s also worth bearing in mind that given Lisp’s Debugging is a fi ne art under Common Lisp, and along with packaging and macros can on practically every aspect of Lisp culture and practice. Parodying or mirroring the towards a contemporary fl exibility and extendibility which make it easy to create quite readily confuse the beginner SLIME on GNU Emacs makes for a contemporary IDE which matches up to the power and evangelicism of a religious Web site, another Lisp, and thence bended radically different dialects of Lisp, Lisp should really be considered as sheer fl exibility of Lisp languages CLiki, the ALU (Association of Lisp Users) wiki, towards implementations a family of languages rather than a language in its own right. And is packed with so called “Road to Lisp” pages which can easily furnish until the early to mid 80s the Lisp world was seriously splintered which demonstrate the fervent admiration which Lisp provokes. Unfortunately, the ALU abstractions such as with competing dialects and implementations proliferating. To where the core language ends and library functions begin. Under wiki has recently been the subject of object-oriented address these issues, hardcore Lisp hackers gathered to standardise a ANSI Common Lisp, the piece of string is certainly seen as being a Key Links prolonged spam attacks, and its active life programming (OOP). new language, Common Lisp, which is the main Lisp in use today, good deal longer than under Scheme, but to do any useful work Lemonodor *hyper-cliki* does appear to be in danger. Other CLikis of Indeed, in his excellent alongside Scheme, an unusual, elegant dialect created by Sussman Schemers may have to take on board some supplementary libraries. www.lemonodor.com lisp.tech.coop/index note include the *hyper-cliki*, an annotatable Common Lisp reference, and the wonderfully ANSI Common Lisp, and Steele in the late 70s. Common Lisp is well specifi ed in It is perhaps more fi tting to investigate specifi c implementations, SICP ALU wiki funky TUNES project CLiki which outlines a Graham shows as an Common Lisp the Language, or CLtL for those in the know, with PLT Scheme as a hot favourite on that side of the Lisp fence. mitpress.mit.edu/sicp alu.cliki.net/index range of projects and resources towards the exercise how a minimal authored by Guy Steele. ANSI standardisation for Common Lisp Scheme does have a lot of things going for it, and contrary to the creation of a free refl ective computing system, or Lisp-based OS. Other essential online OOP can be followed a few years later. argument that Scheme’s simplicity and beauty don’t play well with Paul Graham CLiki resources linked from cliki.net, include the implemented in Thus, one of the fi rst choices facing the novice Lisp coder, before real-world issues, it is totally possible to produce enterprise level www.paulgraham.com www.cliki.net/index Common Lisp Cookbook, an attempt to create Common Lisp, without even considering free implementations, is whether to ride with apps under Scheme. That said, the free software Common Lisp John McCarthy TUNES project a community resource parallelling the Perl using CLOS (Common Common Lisp or Scheme. There can be no easy answer and the community, grouped around hearty resources such as cliki.net and Cookbook approach and both Successful Lisp www-formal.stanford.edu/jmc tunes.org and Practical Common Lisp, two essential Lisp Object System) question has probably fed more fl ame wars in both communities Lemonodor, just seems much more active, though there is a good online works. features. His preliminary than any other issue. Researching the culture of both dialects can deal of crossover, with seasoned hackers turning their hands to both Guy Steele Successful Lisp language is implemented throw interesting light on theoretical issues under both languages, dialects. For the sake of simplicity, Scheme will remain outside the library.readscheme.org/page1.html www.psg.com/~dlamkins/sl/contents.html in just eight lines of code. and it’s relatively easy to grasp the fundamental differences in feel scope of these articles, with the caveat that an understanding of the PLT Scheme Practical Common Lisp Under the mini OOP embedded language, when it comes down to and approach. Scheme does have a particularly interesting history, conceptual underpinnings of Scheme and the lazy evaluation which www.plt-scheme.org www.gigamonkeys.com/book improving the syntax of message calls to make them read more like and its creation is considered as of seminal importance within the continuations facilitate can prove suitably enriching for any hacker. Lisp, the rather more complex meta world of macros is encountered. history of computing, resulting as it does from an attempt to Beyond this essential conceptual background, what the newbie Planet Lisp Common Lisp Cookbook Once again macros come courtesy of the uniform treatment of code understand Actors, Carl Hewitt’s message passing model of sorely needs to know are the specifi cs of respected implementations planet.lisp.org cl-cookbook.sourceforge.net and data as forms for manipulation. Graham has devoted the whole computation. Scheme can be viewed as a more minimal language and how to get up and running with these most effi ciently. Sure you

54 LinuxUser & Developer LinuxUser & Developer 55 Sensor networks Sensor networks

lobal warming has became the new booming nuclear-powered India in the EARLY WARNING SYSTEMS ghoul of modern politics. Since the middle, sudden prospects of economic ECONOMICS Gcold war, global terrorism, Islam and prosperity and even regional dominance. While global data networks are providing various dictators have been made timely warnings of tsunamis in the Pacifi c, to serve as the new GLOBAL NETWORKS = US the presence of the US, Canada, Japan and counterpoints of global NETWORKS? New Zealand is a factor not only valid for political strategy. But The USA developed the global reach, but geographers, but also for budget allocations the frontiers have none of the global economic and social for respective governments. The Indian changed. Global presence so essential to experience divergent Ocean contains few zones of cooperation warming and its political cultures and environmental and and many of confl ict: India and Pakistan, attendant political concerns. Technical and scientifi c Burma and Thailand, Sri Lanka, Indonesia consequences have expertise was available to the US political and Malaysia have all eyed each other across Microkernels and TinyOS suddenly become a elites, but as recent political events have lengthy borders and through decades of more important issue borne out, the very thought of global confl ict. Asking them to cooperate is often a For anyone with any knowledge of recent Unix history, the debate as to whether than the war in Iraq political engagement with other countries on question of political tact as much as global microkernels or so-called monolithic kernels are to be preferred is fairly old hat. Embedded systems do sometimes rely on them even though they might not have much or the catastrophic equal terms is anathema to the current US economic realities. Where should the money more than a rudimentary kernel running only essential services. Sensor systems tend to Indian Ocean tsunami. administration. and the experts come from to fi nance and run real-time operating systems, usually without prejudice towards the kernel The nations of the Indian Ocean and The very size of the scientifi c establishment run an early-warning network that can be architecture. The design decisions for TinyOS are quite interesting in this regard, since Europe have always been slightly more in the US and its rather smaller, but still run by all Indian Ocean countries? many RTOS companies tend to be quite fond of microkernel architectures. attuned to global environmental concerns. infl uential counterparts in the European RTOS developers like to be able to strip down a kernel to its most essential Many reasons can be adduced: Europe’s Union and Japan leads to similar problems. JASON AND THE ARGONAUTS functions so that they only need to add on services whenever they need. global dominance until the end of the 2nd More than 50 per cent of all scientists There are precedents for global Very small RTOS systems like VXworks and a number of Linux-based RTOS fl avours can scale down to 32 k quite easily, but they tend to be quite world war and the spectacular implosion of globally are employed inside the US. If one is networks that already provide unsuitable for sensor nodes. imperial pretensions in the post-war period to add the number of scientists employed by global oceanographic data: Sensor nodes need to be able to process concurrent data bursts. Since most systems left many Europeans with a clear image of EU countries and organizations, the one of them is called are still using a process model that is quite complex, and traditional multithreading global political and economic realities, but predominance becomes simply staggering. the Argo project models need to be able to interrupt threads whenever necessary, it is vital that individual data processing tasks are always running right through to the end. No interruption of little power to infl uence either. The nations Japan is turning into a major source of which data processing tasks is allowed. This does not mean that tasks cannot run concurrently, of the Indian Ocean were mostly subject to scientifi c expertise, although logistical and consists but it does mean that time constraints are not quite as important as in other RTOS the political will of others until well after cultural issues are inhibiting process. applications. 1945. The late 20th century, however, Sensor nodes also need to wake up and go back to sleep without draining the energy reserves needed by the node. These are devices in the millimeter range and energy afforded countries from Iran to density plus energy effi ciency are extremely important. Microkernels are designed for Indonesia, with a modularity, not for energy effi ciency. Although portability is a major concern for TinyOS There might be few interesting data - there are large numbers of “mote” architectures out there - modularity is a condition for the design of TinyOS, not the major design goal. Microkernels come earlier in the one day and the next day, a storm or evolution of kernel design. TinyOS is far more concerned with running the node while avoiding energy waste. an earthquake Finally, microkernels tend to factor networking out of the kernel and provide separate services to deal with networking and routing. TinyOS has to provide simple networking support (“single-hop networking”) inside the kernel, since routing algorithms require might deluge more time and resources than the average node can deliver. 2k of RAM and, say, a 5 MHz CPU do not allow for that. observing scientists with interesting data

Predicting global warming and natural phenomena, volcanic tremors and shifts in the earth beneath our feet, may depend on global networks of tiny sensors around the world’s oceans. The extent to which the fate of this planet is in our hands is overwhelming, but the complexities of our ecology are still defeating the resources and people researching it. Frank Pohlmann investigates Embedding Sensors in Global Politics

56 LinuxUser & Developer LinuxUser & Developer 57 Sensor networks Sensor networks

of more than 1600 robotic sensors resources and people researching it. cope with sudden increases in the data RAM might sound positively excessive. acquisition board combined with a another mote synchronized all data transmissions that were fl oating 1 mile below sea level to collect The Argo project is just one example. stream. The confi guration of networks Java for mobile platforms is not small motherboard with processor and coming from the motes and a stationary seismic detector a data about water salinity and water What we are talking about here is a would have to change quickly if the comparable for architectural reasons: it radio running in standby mode 24 few miles away. The sensor nodes ran TinyOS of course and temperature changes. In a few years, large number of subsurface fl oats taking need arises. If the sensors are mobile, a is not as heavily componentized as hours a day. data processing was accomplished using Java and Perl 3000 sensors will be stationed in the fairly simple measurements. Imagine greater concentration of sensors in one TinyOS, and neither does it include a Wireless sensor networks usually have routines. world’s oceans. This is complemented having to take meaningful area might be regarded as necessary. If concurrent programming language: to be self-organizing, but when it This is just an experiment, although many other by the laser altimeter satellite Jason 1 measurements in biotopes and around some of them were destroyed, the Java includes concurrent libraries comes to networks the size of the Argos applications can be thought of. The technology to collect and the radar altimeter platform small lakes a few hundred yards across. wider mesh of sensor nodes should not expressed in APIs, not the language network, there is another element to be data all over the world exists and advances in the TOPEX/Poseidon, which provides Now imagine having to accomplish create problems for data transmission, itself. It would also need an OS running taken into account: since wireless sensor miniaturization of networking and CPU technology have surface topography data from almost all that very same task in several hundred like increased latency. The operating the Java virtual machine and the code, networks consisting of motes cannot made global environmental monitoring possible. global oceans. Ocean currents change thousand locations globally. Collecting systems and programs running the conditions which the TinyOS transmit data across long distances, the surface height and curvature of the and collating the data locally and sensors should be stable and follow soft architecture does not impose. they have to employ precisely the same RELATIONS BETWEEN NODES AND POLITICIANS sea, which in turn can be measured to reliably and according to accepted real-time standards. TinyOS is written in a C like principles of mesh-networking and ad- There is another problem that the tsunami in December 2004 two-inch accuracy from a height of scientifi c standards is diffi cult and has There are a number of projects using language called nesC that provides hoc routing, but enable the same kind threw into sharp relief: even if the data are present and they 1300 km. Combined with temperature hardly begun. precisely such a technology. Wireless both a structured componentized of coverage for environmental have been evaluated, they have to be presented to decision development model and a monitoring, and a less dense sensor makers in a comprehensible form. The Indian, Sri Lankan and Oceanography and climatology are concurrency paradigm. Since population. Argo nodes send data Indonesian governments were faced with the need to make continuous data collection is the major directly to satellites, which given the quick decisions; Sri Lanka and India had suffi cient time to not yet threatened by patent lawyers, purpose of this operating system, it is coverage and stability of the nodes warn their populations and cut down the number of dead extremely important that there are makes a lot of sense, but if somewhat substantially. Unfortunately, even the best technology, as although cloud patterns or whale facilities for a programmer to debug more local information is needed, software available for free under the GPL or as hardware for his code for race conditions; the nesC requires too much investment. fairly little money are not suffi cient to make those countries songs may become subject to compiler would report them. The kind of coverage and quick data able to respond to national emergencies quickly. In the case Programming in nesC also includes collection and information evaluation of the Indian Ocean tsunami some data was available, but the copyright or patent law in the future primitives that allow controlling data necessary to prevent, e.g. earthquakes respective governments had few systems available to respond collection from sensor nodes. from killing 100s of thousands is not quickly to the emergency in question. Science and technology, owing to its quite available yet, but fi rst attempts are In undergraduate essays it is considered vital to come up institutional history, is still settled NETWORK ZOOLOGY in the works. with a conclusion that draws on various answers given to the The sensor nodes, also known as TinyOS and various types of motes essay question. Embedded networking and indeed the around the Atlantic and the Pacifi c “motes” when just the radio-board were developed largely at UC availability of sensor networks are political issues, not only due underlying the sensor hardware is Berkeley. A small number of to the importance of the data they collect, but also due to their referred to, are connected and companies tried to establish a number potential ubiquity. It is one thing to observe that surveillance gradient data, a fairly accurate picture YES OR NO? IT IS RANDOM, SIR! sensor networks have been developed controlled via an RF network; a network of application areas like biological tools can become almost invisible. It is another to increase our of the “weather” below the surface of One simple observation with regard to for a number of years and it seems that consisting of TinyOS-run sensor nodes habitat monitoring, building expertise concerning geological and biological systems. the see can be gleaned. global climate and local ecology is the micro-sensor technologies and some or “motes” would need to be extremely monitoring in case of seismic events Pervasive computing is slowly becoming reality and sensor Recent events in the Indian Ocean fact that with our methods of related networking products have left effi cient in transmitting data. and patient monitoring in long-term networks are probably the most important part of it. It can save and the brouhaha around global observation we are unlikely to be able the embrace of academia and entered The motes would not be able to store care facilities. By now, other 10s of thousands of lives, but it can also make Orwell’s 1984 warming - the latter being old news to predict the global climate. the industry. In some cases the much information in RAM, given that researchers working with TinyOS have look like political naivitÈ. And we shouldn’t forget that for the which was common knowledge 25 Randomness and unpredictability are reception of sensor networks has been RAM size ranges from 0.5 k to 8 k are tried to simplify and extend TinyOS moment, many countries still do not have the resources to years ago - suddenly brought global probably features inherent in large, colored by some very negative political common and secondary storage is and mote networks to apply them to instrument their coastlines and seabeds with sensors and data early warning systems and the complex systems: there are no hidden perceptions: sensors the size of fl ies extremely limited. The so-called mica2 new applications. Harvard-based centers. Science and technology, owing to its institutional evaluation of scientifi c data collected all variables whose elaboration would (“motes”) have military and civilian motes, reached a size of 7 mm a few researchers succeeded in testing history, is still settled around the Atlantic and the Pacifi c. over the world to the forefront. Luckily, make the weather or ocean currents military applications whose potential years ago. Traditional TCP/IP motes for a network that was oceanography and climatology are not fully transparent. have civil rights campaigners up in networking is not suitable, since wireless monitoring a volcano for seismic yet threatened by patent lawyers, This also means that the collection of arms. The fears are justifi ed and mesh networks have to be routed ad events and possible eruptions. Key Links although cloud patterns or whale songs ecological data on any scale has to legislation is going to be needed. But hoc and the communication overhead Argo may become subject to copyright or respond to rapid, seemingly random there are uses to which sensor networks would tax the limited power budget VOLCANO SURVEILLANCE www.argo.ucsd.edu/ patent law in the future. This is changes in the environment. There can be put that might ensure mankind’s motes have to run on. Traditional When volcanoes are about to erupt, www.primidi.com/2004/12/03.html somewhat less frivolous than it sounds, might be few interesting data one day longevity, rather than stoking the fi res addressing modes like Ipv6 would be they emit seismic tremors as well as TinyOS since, as we all know, songs are being and the next day, a storm or an of global paranoia. suitable, if it wasn’t for the fact that infrasonic pulses. A sensor array tailored www.tinyos.net/ www.xbow.com/General_info/eventdetails.aspx?eid=54&localeid copyrighted and to our knowledge, earthquake might deluge observing connection establishment and reliability to the work of volcanologists would =3&addressid=54 whale songs are unique to the whales Recent scientists with interesting data. HEY TINY are major issues for IP-based include two sensors detecting infrasonic Sensor Networks singing them. TinyOS is a GPLed operating system networking: it means that they energy waves and tremors. In this particular models have www.intel.com/research/exploratory/heterogeneous.htm Let there be no doubt that data SENSOR NETWORKS AND running the nodes of embedded expenditure for Ipv6 stacks would be case, the sensor array consisted of lternet.edu/technology/sensors/arrays.htm relating to the global climate are in the run for 945 SENSOR POLITICS wireless networks. The networking considered excessive and the need for infrasonic detectors, since other stations Medical Monitoring public domain. The extent to which the Obviously, the sensors would have to components are not vital to the OS, routing servers imposes networking were able to pick up seismic tremors www.eecs.harvard.edu/~mdw/proj/vitaldust/ fate of this planet is in our hands is days on 2 AA be hardy enough to survive a battering which is built to run extremely small overhead that is impossible to maintain. already. Three mica2 motes collected Volcano Monitoring overwhelming, but the complexities of by the elements and the networks processors whose ROM might not Recent models have run for 945 days the data, one transmitted the infrasonic www.eecs.harvard.edu/~werner/projects/volcano/ our ecology are still defeating the batteries linking the sensors have to be fl exible to exceed 256K and for whom 128K of on 2 AA batteries. Not bad for a data data over a point-to-point link and

58 LinuxUser & Developer LinuxUser & Developer 59 Advertising Feature

UK Office: Telephone: 01295 756102

UK Office Fax: 01295 276133

Email: [email protected]

http://www.transtec.co.uk/ transtec 2500 Opteron Servers The ideal entry to 64-bit computing

The transtec AG - THE EUROPEAN IT FACTORY The right product, for the right price at the right time and the right place. A simple message. A high demand. A challenge we fulfil. That is our purpose. Our customers do not want off- the-rack products. They expect tailor-made. That is why they need a competent partner. Competence that transtec has built up within two decades. And we work to strengthen it - every day.

The newest software, the fastest processors, the largest memory chips. The technology scouts from transtec track down the latest inventions in the IT industry. In combination with proven technology the engineers in Tuebingen develop faster and more stable IT systems. In this way, the most innovative technology flows directly into the computer networks of the transtec customer

Only in their own production facilities transtec can fulfil customers` requests. From thousands of variations and combination possibilities, companies can select the technology that suits exactly their needs. transtec builds the equipment together precisely as ordered in the built-to-order process. This makes many of transtec´s 45,000 annually delivered computer systems unique.

The AMD Opteron™ is based on the AMD64 technology, which ● Tower, optional 19“ Rackmount, makes it possible to have 64-bit technology on an x86 platform. Other ● 4 U One AMD Opteron™ 242 1.6 GHz processor important innovations in Opteron™ processors include an integrated ● Max. 2x AMD Opteron™ 250 2.4 GHz storage control module to lower the number bottlenecks in memory ● 1 GB registered ECC DDR-SDRAM (max. 16 GB) and the HyperTransport technology. Opteron-based systems are ideal ● Onboard Dual 10/100/1000BASE-T for use in database servers or for complex applications that need ● Hotswap 36 GB Ultra320 SCSI disk more than 4 GB memory per process. Thanks to the complete 32-bit ● max. 10 Hotswap Ultra320 SCSI disks compatibility of Opteron™ processors, these servers are also perfect if an upgrade to 64-bit technology has already been planned, but not £1595.00 all of the applications are available yet in a 64-bit edition.

transtec-solutions in hardware

60 LinuxUser & Developer regular column Swing digital Amazing Offer! Poles apart

Worth elcome to my new column, I’m Jason Kitcat and I’ll be your there are just over 8,400 Linux users £64.95 FREE host over the coming months as we explore Free Libre Open in Poland, a country of nearly 40 WSource Software (FLOSS), e-government and life in the digital million people. While the project’s lane. Enjoy the ride. methodology is hard to verify the Just so you know that I am qualifi ed to take you on this journey, I’ve results still give us an idea of the low been tooling around with the Internet since around 1996. I’ve run a adoption Linux has had. SUSE 9.29.2 dial-in bulletin board system and started several tech companies. Since Before accepting Microsoft’s 2000 I’ve been building online communities at Swing Digital, a overwhelming dominance I Professional Brighton-based company I co-founded. We host all our communities wanted to double check the on custom versions of Linux and Apache. market share fi gures. After much Jason Kitcat So what’s the fi rst stop on our journey? Poland as it happens. I spent digging I found some more With NEW some time in the south-eastern corner of Poland during the Christmas statistics. Ranking.pl (www. period and was curious to know how Linux is doing in this region. ranking.pl) shows that 98.8% of those accessing the Internet in subscriptions to GNU/ Poles are bursting with pride in their country, rightfully so, still it has its Poland are on Windows with only 1.1 per cent on all forms of Unix fair share of challenges. In a country with high unemployment and low including Linux and MacOS X. 64.6 per cent of Polish Internet users wages relative to the rest of the EU I was expecting FLOSS to play a are on Windows XP. Sadly these fi gures are probably, if anything, Or any one of these major part in Polish digital life. underestimating the WindowsWindows share as Linux and MacOS X users are I was very wrong. I didn’t see a single copy of Linux running in any far more likely to go online than the average user, especially those O’Reilly books! of the offi ces or homes I visited, everyone had Windows. Chatting with stuck with Windows 3.1 or 95. Marek Ogrodnik, an employee at a major technology retailer, I learnt Historically a signifi cant portion of the copies of Windows and GET THE AWARD WINNING SUSE LINUX 9.2 PROFESSIONAL INCLUDING THE Never miss an issue! LINUX KERNEL 2.6.8, KDE 3.3 AND The average man in the street, GNOME 2.6 - AVAILABLE ON 5 CDS AND 2 DVDS PLUS EXTENSIVE COVERDISCS Every LinuxUser & Developer CD is or computercomputer shop fforor that matmatter,ter, DOCUMENTATION. NEWCOMERS AND guaranteed to be packed with hundreds of ADVANCED USERS WILL FIND the latest and greatest Free Software PROFESSIONAL ASSISTANCE AND projects, including many exclusive LinuxUser is still ignorant of the benefi ts Linux could offer INSTRUCTIONS FOR ALL KINDS OF & Developer covermounts ISSUES. THE SUSE LINUX 9.2 that he reckoned that Windows has 99% market-share in Poland. associated software have been shared and not purchased legally. PROFESSIONAL PACKAGE NOW PRIORITY Piled up next to Marek as we chatted were boxes and boxes of However, since a very strict government crackdown on piracy three INCLUDES THE PORTED 64-BIT Mandrake Linux on DVD for £15. It wasn’t selling, but OEM versions years ago the majority of software is grudgingly purchased. Coming APPLICATIONS FOR USERS WHO EMPLOY of Windows XP for £65 were. Consider that for those with jobs in from a Communist era where scarcity forced ingenuity, Poland is THE LATEST TECHNOLOGY OF INTEL AND DELIVERY Poland an excellent salary is £300 per month. With Windows XP culturally ripe for the FLOSS approach to digital life. Unfortunately AMD 64-BIT PROCESSORS AS WELL AS Receive your copy direct to your place or Home Upgrade selling for £190, Offi ce going for £360 and the Windows got there fi rst and it’s going to be very tough overcoming 32-BIT BINARIES. work or home before it reaches the newsstands! academic license for Offi ce a cheeky £70 I couldn’t understand why Microsoft’s huge installed base. Linux was more popular. All is not lost though‚ - As in many other countries, the government Stepping over discarded fl yers for Mandrake I asked Marek why can use procurement policies to force competition. The Polish Please send me the next 12 issues of LinuxUser and Developer and my copy of Suse 9.2 Professional for £59.99 (1 year) or £89.99 (2 years) Linux wasn’t more popular. His answer was similar to those I often Ministry of Finance is using Linux for some backend functions. This Please send me the next 12 issues of LinuxUser and Developer (rest of the world subscription offer) hear back in the UK: Linux is very hard to use, he couldn’t get it to move gained considerable publicity and is probably only the fi rst of even install on the showroom computer - when it does work you many gains for Linux in the Polish government. On the other hand MR/MRS/MISS FIRST NAME ...... Send to; can’t play games, watch movies or buy software the way you can government has also been unhelpful, in at least one case commercial Subscriptions Dept with Windows. The network effect is in full force in Poland - Because installations of Debian Linux were regarded by the local tax offi ce as SURNAME...... LinuxUser and Developer everyone else has Windows, you are better off having Windows. ‘gifts’ so a 30 per cent tax was applied on the assumption that 5 Broadhey, Romily ADDRESS ...... Poland has, like most countries, a technical elite who can and do Debian cost as much as Windows. Oops. Stockport, Cheshire SK6 4NL ignore Microsoft-shaped bandwagons, and inevitably they are playing While Windows’ dominance in markets such as Poland can be ...... Or call the subscriptions hotline on; with Linux. PLD is the Polish Linux Distribution (www.pld.org.pl) depressing, it isn’t the whole picture. The key thing to remember is ...... 0161 4303423 made by Poles. Another Polish distribution is Aurox (www.aurox.org). that whether we look to Eastern Europe, Central America or India, There is also a Polish Linux User Group (www.linux.org.pl). It’s a most people don’t have a computer at all. So we don’t need to wean POSTCODE...... TELEPHONE...... small group, but they seem to have infl uence. Poland caused a storm them off Windows, we need to reach out to them and explain why Or subscribe online at and ingratiated themselves to the FLOSS community by stopping the EU FLOSS can help to make our digital lives more fair and open. EMAIL ...... www.linuxuser.co.uk Software Patent Directive in its tracks. This is the sign of a well informed PLEASE BEGIN MY SUBSCRIPTION WITH ISSUE NUMBER...... Outside the UK - If you live outside of the UK, technorati but unfortunately the average man in the street, or computer we regret to say that the free software offer doesn’t apply. However, you can subscribe for shop for that matter, is still ignorant of the benefi ts Linux could offer. Jason Kitcat ([email protected]) is Managing Director of online I ENCLOSE A CHEQUE PAYABLE TO LINUXUSER & DEVELOPER FOR...... £74.99* if you live within Europe and £89.99* if you live elsewhere, including the US and According to the Linux Counter Project (counter.li.org/bycountry/PL.php) community consultancy, Swing Digital Australia. Both prices include P&P. ALTERNATIVELY PLEASE BILL ME FOR...... *Charged at local currency exchange rate.

LinuxUser & Developer 63 Hacking the kernel Hacking the KernelHacking the kernel

complex remote kgdb hacking - we’ll get to that after we have dealt Jon Masters provides his regular look at Linux Kernel development with the basics and the merits for using debuggers at all. Kernel debuggers, like all software debuggers, allow us to peek into the internals of a running system and discern some specifi c information about the way in which it is operating at run time. This helps us to track down diffi cult to locate bugs but also provides a service as a mechanism for understanding the correct normal operation of a real system. In last month’s column, we devoted a small section to using UML but the time is right now to explore the Taking workings of the kernel in more detail using a combination of available Using gdb could tools. The fi rst of these is the venerable GNU Debugger, gdb. gdb represents one of the most popular and widely used GNU tools after form the subject the GNU Compiler Collection (gcc) and GNU C Library (glibc). It is certainly one of the most ubiquitous debuggers in widespread use matter of a rather the reins. today and is well understood by many programmers. Using gdb could form the subject matter of a rather large book (it large book (it does) does), and also deviate us from our chosen topic, so a thorough exploration of the available features is left up to the reader. It is assumed that the reader has a working environment with appropriate kernel, fi re up gdb with the appropriate kernel Tracing development tools installed, and that the gdb documentation has image and core fi le: been consulted where necessary. gdb is typically used to debug a standalone program or running process, but it can also utilise a jcm@perihelion:~$ sudo gdb /lib/ memory image contained within a “core dump” - produced as a side modules/2.6.10/build/vmlinux /proc/kcore effect of eg. a “segmentation fault”, during which this fi le is dumped GNU gdb 6.1-debian the source if the user has not confi gured a system not so to do. The Linux kernel Copyright 2004 Free Software Foundation, provides a special procfs entry in /proc/kcore, which is a dynamically Inc. Debugging the Linux kernel (continued) generated “core” fi le with a difference. By hooking gdb up to use this GDB is free software, covered by the GNU fi le, we can peek at the state of the running system. General Public License, and you are ernel Hacking is a regular feature on the inner workings of ways to make the system crash horribly. Debugging provides a We won’t be able to change anything through this initial interface, Waiting for gdb welcome to change it and/or distribute the Linux kernel. Each article is split into several sections, mechanism for tracing the execution of a running Linux system and but we can look at certain kernel counters and check on its general to connect to a copies of it under certain conditions. Kwhich vary in both technical detail as well as intended watching as each line of kernel source results in a a series of mental wellbeing to a certain extent. To begin debugging a live remote target Type “show copying” to see the conditions. audience, although the goal is to provide an accessible alternative to operations which form the cohesive whole. Let’s have some fun with There is absolutely no warranty for GDB. existing introductory material. In this issue, we continue our gdb, and kgdb through in a little debugging of Linux on Fedora Core Type “show warranty” for details. investigations with various kernel debugging techniques in 3, within a VMware machine. To follow the instructions in this next preparation for building a python based GUI kernel memory browser section, readers will benefi t from having a kernel image built with Core was generated by `auto BOOT_ debug symbols - see the IMAGE=linux ro root=2103’. “Kernel Debugging” #0 0x00000000 in ?? () Debugging isn’t just a mechanism for fi xing bugs in submenu under “Kernel (gdb) kernel drivers and discovering new and spectacular Hacking”, when building a kernel from the source. Note the use of sudo to run the gdb ways to make the system crash horribly command as the root user. Those without sudo appropriately installed and confi gured can use a in the coming issues, investigate some of the more recent changes to USING GDB FOR FUN AND PROFIT root shell instead. Also note that gdb was passed the kernel and check out the latest news from the various mailing In the last issue, readers were briefl y introduced to the concept of the full kernel vmlinux image from the top level lists. Kernel security is increasingly becoming an issue of interest as kernel debugging using the UML (User Mode Linux) as a mechanism of the kernel source directory. This is the full numerous new exploits have recently surfaced, we’ll look at what’s for running test kernels within a virtual machine on a regular desktop kernel binary prior to any stripping and with full being done to tackle these kinds of kinks in the new development PC. This time around, we will look at using the GNU command line debugging symbols already built in. The kernel process. Your feedback is most welcome, especially any feature debugger, gdb to peak at the running kernel on a regular Linux was booted with the parameters shown in the suggestions - drop us a line at [email protected]. machine and then consider the options for interactive real time kernel last output lines above. It is possible to Learning about the kernel is almost as important as writing code debugging. None of these options require a second PC since it’s interrogate the state of various kernel variables itself, sometimes more so. Debugging isn’t just a mechanism for possible to use a virtual machine PC emulator such as VMware (even from this prompt, but it is not possible to change fi xing bugs in kernel drivers and discovering new and spectacular an evaluation version if just for a single test run) for some of the more the kernel state - so calling kernel functions is out

64 LinuxUser & Developer LinuxUser & Developer 65 Hacking the kernel Hacking the KernelHacking the kernel

Debugging the kernel init process Kernel Personalities with the aid of a remote gdb client. The virtual machine running within VMware is completely controlled by a remote Linux wouldn’t be nearly as amounts of wasted code and be an interesting talk on updated for kernels up to gdb, which can issue commands fun and interesting as it is, resources to achieve a the new development and including 2.6.10. This to set breakpoints and modify the were it not for the efforts relatively simple task. Inotify process - which, it has to be edition is published by his behaviour of the target. of a select group of gifted instead uses a new device said, he doesn’t seem all new employer, Novell, and hackers who busily code node, through which too fond of. now includes additional away in the wee hours of programs can communicate chapters on “Getting the morning, powered with the kernel to request As effort is underway to bring Started with the Kernel”, solely by adrenaline and notifi cations of events on a out the 2.6.11 kernel, various “Modules”, and also the unholy amounts of caffeine fi le - especially through a -rc prereleases have started to Kernel Events layer - well some of them, at regular poll or select syscall become available. Kernel “kobjects and sysfs” that least. The LKML (Linux within its main loop. releases can be tracked most Robert has helped work on. Kernel Mailing List) is a hive effectively by maintaining a All in all, this promises to of activity on a daily basis. Perhaps one of the most daily update of Linus’ be a very interesting read Recent discussions have disturbing issues with BitKeeper tree. The graphical and worthy of a place on included a new “inotify” kernel development at the “revtool” allows one to track the reader’s bookshelf - mechanism in 2.6.10, a moment is the handling of a tree of changes from one unfortunately this title is reworking of the internal critical security issues. We release of the kernel to the not yet available in electronic representations of pipes, have recently seen several next - and to see a complete form and has yet to become and a treatise on “Getting remotely exploitable changelog history for widely available in European Started With Kernel defects in the kernel which arbitrary points along the countries (we’ll do a review Hacking” in which Jonathan could be used by malicious time line. Readers would be just as soon as we can get for the moment. Print the value of jiffi es: allows us to remotely debug a kernel at the source level rather Corbet reminded readers crackers to gain well advised to perform a hold of it!). In the words of than interrupt a running machine and issue limited diagnostic that the new edition of unauthorised access or to daily pull of Linus’s tree and the books’ author: “You (gdb) print jiffi es commands. Kgdb was the original creation of SGI hackers but is Linux Device Drivers will be launch a Denial of Service watch as new patches are should buy a copy or two $1 = 51656788 now a sourceforge project in its own right. It comes in the form of out possibly during Boston (DoS) attack. Fortunately, released unto the and carry them with you various patches which must be applied to the kernel sources in LinuxWorld. fi xes for such problems are unsuspecting masses. Consult everywhere you go’”. The kernel maintains an internal counter, known as the jiffy, which documented order. The kernel boot command line must be altered typically a mere few lines of http://linux.bkbits.net/, represents the number of timer ticks (timer interrupts) which have (in /boot/grub/menu.lst or /etc/lilo.conf, depending upon your Amongst the various mailing code and quickly make it in Documentation/BK-usage The 2005 FOSDEM (Free and taken place since the system was booted. It’s what effectively defi nes platform and bootloader) to include a serial port for kgdb to listen list posts came a replacement to vendor updates - but as directory within the kernel Open Source Developers the “uptime”. We can use this mechanism to interrogate any global on for a remote gdb. to “dnotify”. Dnotify has development is taking place source itself, and the European Meeting) takes kernel variables but must be aware of gdb’s internal caching, which In the example case, kernel 2.6.7 has been patched using the historically represented a bit in the mainstream kernel accompanying website to this place on the weekend of may result in a repeated output of the above value of jiffi es if asked kgdb sources and built to run on an AMD Athlon. The author of a hack job solution to the source tree, these patches series for more specifi c usage February 26 and 27. Your for the same value immediately afterwards. gdb supports full tab chose to use a VMware virtual machine rather than a real physical problem of tracking changes can be delayed by several information. author will be there to completion, and in combination with a Linux source code cross PC to prove the point that even a real machine is not required for to directories. Rather than days getting in to the stable witness the Kernel and reference such as LXR (see resources) allows for limited, but entirely our experimentation purposes. VMware is an example of an Intel having a program source. For example, as of Robert Love released a new Embedded Linux fora, as well safe, perusal of the kernel state. We’ll build on this example in the Architecture machine emulator which in this case runs Fedora Core continuously poll a directory this writing, there remains edition of “Linux Kernel as the general talks held at next issue - to create a graphical browser of kernel state information. 3 as if it were standalone. The modifed kernel was placed in /boot/ for changes, it is possible to a critical exploit in the Development”, revised and this event. vmlinuz-2.6.7 and /boot/grub/menu.lst was modifi ed to register to receive a signal current stable kernel which KGDB - TAKING THE REINS accommodate the new kernel image. An entry of the following when such events occur. has not yet resulted in a It’s all very well using gdb to pick at the running kernel on a form was added to the grub boot menu: Unfortunately, dnotify public release of a fi xed desktop machine, but how does that help us in our quest to learn doesn’t adequately solve the stable kernel. At the more about the innards of Linux? The answer comes in the form of title Kernel Development problem of watching FOSDEM in Brussels at the patches to the kernel source which add remote debugging root (hd0,0) individual fi les for changes end of February, Alan Cox capabilities. Using a second PC (or a VMware virtual machine in kernel /boot/vmlinuz-2.6.7 ro root=/dev/hda1 kgdbwait and can result in large will give what promises to this case) one can remotely take control of the execution of each kgdb8250=0,115200 line of the kernel source. There are two primary kernel debuggers used in such cases: kgb and kgdb. We’ll focus on kgdb, since it This instructs kgdb to halt the kernel as early as possible and wait

66 LinuxUser & Developer LinuxUser & Developer 67 Hacking the kernel LAMP LAMP

for a remote connection from a gdb client via kernel source tree without a radical change of heart on the part of One of the major advantages of a relational database is the ability to the fi rst serial port (ttyS0). Now, another Linus Torvalds. Linus believes debuggers do more harm than machine connected to this serial line can take good, altering the characteristic performance of the system under join more than one table when executing queries. In this month’s article David Tansley looks at remote control thusly: test. He’s right, but us mere mortals exercising appropriate care and due attention probably benefi t more than we lose out. joins, using MySQL with a touch of PHP to show how it can work on a web page (gdb) target remote /dev/ttyp0 [VMware no longer supports direct access to the virtual Remote debugging using /dev/ttyp0 machine’s serial port in newer versions. It is now necessary to use a breakpoint () at kernel/kgdb.c:1212 wrapper (e.g. “vmware-p2s”) in combination with the “named 1212 atomic_set(&kgdb_setting_ pipe” option in the VMware confi guration to connect VMware to a breakpoint, 0); spare pseudo terminal (e.g. /dev/ptyp0) on the reader’s computer. This is quite straightforward - see Resources box.] It is now possible to control the kernel as if it were any other program being debugged. For example, in the following sequence we elect to insert a few breakpoints in the boot sequence, PHP and Joins Linus believes debuggers do stopping execution in init/main.c at “prepare_ namespace” (which is near to the end of the more harm than good, hile there are plenty of in-depth resources available elsewhere customers table; this forms the relationship between the two tables boot sequence) and to then view the console on relational database design, the LAMP developer requires (customer.customer_id to orders.customer_id). This allows us to know ringbuffer (the buffer to which printk saves its altering the characteristic Wat least a basic grasp of the subject. A true relational database which customer has ordered what, and the link back to the customers output before it is sent to the console) and a design provides the minimum of repeated information. If changes are name is the customer_id. backtrace of functions called along the way.: performance of the system required to table attributes, then the design should allow for the database to scale gracefully with the additional data load. Breaking Listing 2. Orders Table (gdb) break prepare_namespace under test, but us mere data into many tables allows for more manageable storage of the product_id order_id customer_id Breakpoint 6 at 0xc05511f6: fi le init/do_ information within the database. 1000 Karate GI 1 mounts.c, line 391. mortals probably benefi t To be able to select related data from these tables, we must be able 1012 Brown Belt 3 (gdb) c more than we lose out to select from more than one table in a single SQL statement. In other 1014 Mouth Guard 3 Continuing. words, we must be able to join these tables in order to extract the 1016 Mitts 5 Breakpoint 6, prepare_namespace () at init/ data and turn it into useful information. The actual join of the tables do_mounts.c:391 goes on behind the scenes, but to inform MySQL that we wish to TABLE CREATION 391 if (saved_root_name[0]) { extract from more than one table a particular SQL syntax is required. To create the database and the customer table, the following sql is used: (gdb) p log_buf $3 = 0xc0588aa0 “<4>Linux version 2.6.7 CREATING THE TABLES [dxtans@bumper dxtans]$ mysql -u root (jcm@perihelion) (gcc version 3.3.5 (Debian To demonstrate how joins can be accomplished we fi rst need to mysql> CREATE DATABASE customer_ord; 1:3.3.5-5)) #1 Tue Jan 1 00:00:00 GMT create two tables, where the fi rst table has a relationship to the mysql> USE customer_ord; 1970\n<6>BIOS-provided physical RAM second table. In my very simplifi ed example tables we have a mysql> CREATE TABLE customer ( map:\n<4> BIOS-e820: “, ‘0’ full_name varchar(45) NOT NULL default ‘’, times>, “ - “, ‘0’ , “9f800 kgdb 1) contains a unique number for each customer (customer_id) which -> customer_id int(10) unsigned NOT NULL auto_increment, (usa”... kgdb.sourceforge.net is the primary key for that table. It also contains the customer’s full -> PRIMARY KEY (customer_id)); (gdb) bt The website accompanying this series name. #0 prepare_namespace () at init/do_ www.jonmasters.org/kernel To create the orders table the following sql is provided: mounts.c:391 Kernel newbies Listing 1. Customer Table #1 0xc01006fd in init (unused=0x0) at init/ www.kernelnewbies.org customer_id full_name mysql> CREATE TABLE orders ( wiki.kernelnewbies.org main.c:638 #kernelnewbies IRC discussion channel 1 Peter Jones -> product_id int(10) unsigned NOT NULL default ‘0’, #2 0xc01032b1 in kernel_thread_helper () 2 Lucy Bling -> order_id varchar(45) NOT NULL default ‘’, Tracking the Linux kernel mailing lists at arch/i386/kernel/process.c:229 www.kerneltraffi c.org 3 David Grass -> customer_id int(10) unsigned NOT NULL default ‘0’, (gdb) 4 Pauline Neave -> PRIMARY KEY (product_id)); Archives of the lkml www.lkml.org 5 Louise Christine This SGI tool is both extremely powerful and The next task is to insert some values into the tables. For the customer Linux Weekly News features Kernel Page available for several architectures at this point. www.lwn.net The orders table (see Listing 2), contains a product_id, and this is the table I have taken the long route and inserted them one at a time like so: While it is technically impressive and certainly primary key for the table. A product name is also provided as well as Linux Kernel Cross Reference of wide use, it is unlikely that tools such as www.lxr.linux.no the customer_id. You may have noticed that the customer_id in the mysql> insert into customer (full_name,customer_id) values (‘Peter kgdb will be integrated in to the mainline orders table contains some of the values from the customer_id in the Jones’,’’);

68 LinuxUser & Developer LinuxUser & Developer 69 LAMP LAMP LAMP

By the time I got to the orders table I was bored with entering data In the SQL used here each table name is comma separated. This mysql> select customer.full_name, orders.order_id customer records have been read in from the customers table to manually, and created a fi le (orders_in.txt) to import the orders rows. informs MySQL that we will be extracting from two tables, and this is -> from customer populate the pull down menu. See Listing 4. cust_select.php. The fi le was a separated fi eld for each row, like so: therefore a join query. Notice that there are 20 rows returned; though -> inner join orders this is a join, it does not provide any really useful information. Using a -> on customer.customer_id=orders.customer_id; LISTING 3. CUST_INC.PHP 1000 Karate GI 1 WHERE clause we can gather better information in our decision making 1012 Brown Belt 3 environment. Let’s extract all customers that have ordered a product: Notice that ‘inner join’ is now used to indicate we are performing an select customer.full_name, orders.order_id Two other joins are the left and right join; The left join will extract -> from customer, orders all rows from the customer table, even if there are rows in ‘customer’ $SERVER=”localhost”; Then after making sure we are back at the database prompt, run the -> where customer.customer_id=orders.customer_id; that do not have a match in the orders table. All will become clear $DB=”customer_ord”; following command: with a query example. Notice in the following query we use a ‘left $USER=”root”;

+------+------+ In the above SQL, join’, and more rows are returned this time because we have $CUST_TABLE=”customer”; mysql> load data local infi le “orders_in.txt” into table orders; | full_name | order_id | notice that I have customers ‘Lucy Bling’ and ‘Pauline Neave’ who have not ordered any Query OK, 4 rows affected (0.00 sec) +------+------+ specifi ed the columns items. This is a non-match between the two tables, but because we # holds initial connection and changing to working database | Peter Jones | Karate GI | Records: 4 Deleted: 0 Skipped: 0 Warnings: 0 I want extracted by are using a left join these results will get printed with a NULL value: $connection=mysql_connect($SERVER,$USER); | David Grass | Brown Belt | | David Grass | Mouth Guard | using their tablename. Both tables should now be loaded up and ready to go. Look at the | Louise Christine | Mitts | column_name. First mysql> select customer.full_name, orders.order_id if (!$connection) { tables with these commands: +------+------+ we select the ‘full_ -> from customer echo “Could not connect to MySQL server!”; name’ from the -> left join orders exit; mysql> select * from customer; customer table as well as the ‘order_id’ from the orders table. We -> on customer.customer_id=orders.customer_id; } mysql> select * from orders; then fi lter it through a WHERE condition, which equates to “if the ‘customer_id’ from the customers table matches the ‘customer_id’ +------+------+ The right join behaves $db=mysql_select_db($DB,$connection); JOINING from the orders table, then print the results.” | full_name | order_id | similarly to the left if (!$db) { +------+------+ A join will create a virtual table, multiplying the tables together. We could also see who has ordered a pair of mitts from our orders join except that we echo “Could not change into the database $DB”; | Peter Jones | Karate GI | Then a WHERE clause containing a condition is used to fi lter out the database, by appending AND to the query. This means “only print | Lucy Bling | NULL | compare from the exit; unwanted rows for the query. Please note that we have fi ve rows in results if both sides of the condition are true.” As we only want to | David Grass | Brown Belt | second table (orders) } the customer table and four rows in the orders table. The most basic print the name (full_name) of the customer this time, we only need | David Grass | Mouth Guard | fi rst. ?> | Pauline Neave | NULL | join is the cross join, which is fairly useless in its most basic form: to select customer.full_name from the customers table: Practically any | Louise Christine | Mitts | +------+------+ query made by a user LISTING 4. CUST_SELECT.PHP mysql> select * from customer, orders; mysql> select customer.full_name can be extracted -> from customer, orders using joins. However, I -> where customer.customer_id=orders.customer_id have just covered the most basic joins and queries; there is a lot more +------+------+------+------+------+ -> and orders.order_id=”Mitts”; to it. Extracting data from tables into a client application is not much

Select A Customer To View Their Order(s) | customer_id | full_name | product_id | order_id | customer_id | fun. Since this is a LAMP tutorial it would be better to web enable our +------+------+------+------+------+ ”; | 4 | Pauline Neave | 1016 | Mitts | 5 | So to print all customers that have ordered a the user is the choice of whether they want to select an individual echo “Please select a Customer
”; | 5 | Louise Christine | 1016 | Mitts | 5 | +------+------+------+------+------+ product, as we did earlier, we could use: customer or see all orders for all customers. This is easily echo “”; accomplished by adding another OPTION VALUE called ‘all’, after the while ($row=mysql_fetch_array($mysql_result))

70 LinuxUser & Developer LinuxUser & Developer 71 LAMP regular column Freestyle Strange anomalies of copyright law (Part 2) Clare’s enclosure

he poet John Clare died in the Northampton General Lunatic July, 2000, Whitaker made a { { Asylum in 1864. During his lifetime he was known as ‘the “provisional bargain”, $full_name=$row[“full_name”]; $mysql_result=mysql_query($sql_all,$connection); Tpeasant poet’, though he was not, strictly speaking, a peasant, presumably verbal, with the $customer_id=$row[“customer_id”]; $num_rows=mysql_num_rows($mysql_result); and lived in an era of social upheaval, enclosures and landless labour, dying John Taylor in May 1864, # display results } else { when a landless labourer had even less rights than a peasant. to pay Clare’s widow, who echo “

$full_name”; $mysql_result=mysql_query($sql_single,$connection); Clare was from a rural labouring family, and had little education, could neither read nor write, } $num_rows=mysql_num_rows($mysql_result); but is now recognised as the greatest English poet of nature, as £10 a year for Clare’s echo “All Customers”; } important as his contemporaries, Keats, Byron and Shelley, of whom manuscripts and publication echo “”; everybody has heard. As the critic Geoffrey Grigson wrote as long ago rights. Later the same year he } # end else # apologise if there’s nothing to show as 1949, “Clare has gradually been transformed from ‘peasant poet’ signed an agreement with Richard Hillesley echo “

”; if ( $num_rows == 0 ) { into poet, from cottage rushlight into what indeed he is, a star of Clare’s widow and her children echo “”; echo “Sorry there is no information”; considerable and most unique coruscation.” for the transfer of their rights to echo “

”; } else { Clare ignored the rules of punctuation and spelling, probably as a Clare’s copyrights. This agreement, discovered in the archive of the mysql_close($connection); # we have results consequence of the slightness of his education, and wrote: “I am Whitaker publishing house in 1932, was destroyed with the rest of the ?> echo “”; gennerally understood tho I do not use that awkward squad of archive during the London blitz eight years later. Whitaker’s edition of echo “”; pointings called commas colons semicolons etc and for the very Clare never appeared, and he transferred the bulk of the surviving Once the user has selected their customer to view, or ‘all’ to see reason that altho they are drilled hourly daily and weekly by every manuscripts (and, some presumed, the copyright) to the care of the all customers orders, the submit button will send them onto while ($row=mysql_fetch_array($mysql_result)) boarding school Miss who pretends to gossip in correspondence they PeterboroughPeterborough MMuseumuseum SSocietyociety bbeforeefore hhisis ddeatheath iinn 11895.895. NNoo eeditiondition ooff cust_view.php, see Listing 5. A check is initially carried out to { do not know their proper exercise for they even set grammarians at ClareClare ppublishedublished bbetweenetween WWhitaker’shitaker’s ddeatheath aandnd RRobinson’sobinson’s ppurchaseurchase ooff determine if the user has loaded this page directly without first $full_name=$row[“full_name”]; loggerheads and no one can assign them their proper places for give “all“all rrightsights wwhatsoeverhatsoever ppossessedossessed bbyy tthehe ccompanyompany iinn tthehe ppublishedublished aandnd going to cust_select.php - if this is the case, then a hyperlink $order_id=$row[“order_id”]; each a sentence to point and both shall point it differently.” The unpublished works of John Clare” in 1965, acknowledged any points them back. This check is carried out by determining if the parsed variable from the cust_select.php form, customer_id has a # display results The purgatorial hell and French bastile of English liberty, value. If not then we know no selection has taken place. Figure 1. echo “”; wherewhere harmlessharmless people areare trappedtrapped populated with the customer details. The screen shot below } shows the result of a query. } and tortured until they die

LISTING 5. CUST_VIEW.PHP # end the script fashion for removing punctuation from modern poetry began with copyright holder. mysql_close($connection); the French poets, Apollinaire and Cendrars, in the early decades of Under the 1842 Copyright Act, an author, or after his death his ?> the twentieth century, but Clare had his own untutored, and personal representative, retained perpetual control over his work as strangely modern, take on the rhythms and nuances of poetry and long as it remained unpublished. This clause remained in force until it punctuation. Clare’s punctuation, and his feelings about it, are at the was fi nally replaced in the 1988 Act with a fi nite, 50-year term of # cust_view.php
Back heart of the academic disputes that surround his work. protection (made potentially extendable by a further 25 years in a # did the user load this page directly ? Clare’s fi rst book sold well by the standards of his time, and outsold 1996 Act). if (empty( $_GET[‘customer_id’] )) { Keats by some margin, but his relative commercial success was short- Although Robinson has contributed much to Clare scholarship over echo “You need to make a selection fi rst”; lived, and he depended for his income on patrons, who sometimes the last half-century, his claims to the ownership of Clare’s legacy has echo “ Customer Menu”; over-edited and censored his material. Much of his better poetry was caused much controversy. According to Goodridge: Robinson “has exit; written during the 23 years he spent in the lunatic asylum - “the enforced this claim, demanding acknowledgement and often } purgatorial hell and French bastile of English liberty, where harmless payment from anyone who wishes to publish Clare material. In the people are trapped and tortured until they die”. Most of his work, view of Tim Chilcott, the effect has been ‘the impoverishment of # include fi le some 2700 poems, remained unpublished during his lifetime, and at editorial debate compared with other Romantic writers, the absence include (“cust_inc.php”); the time of his death he was all but forgotten, despite the best efforts of challenging alternative views, the deadening hand of the of his publisher, John Taylor. authorised defi nitive version.’” # this query is used if user selects a customer name But due to a quirk in English copyright law, 140 years after Clare’s A core concern of Clare’s poetry was the disruption to the $sql_single=”select customer.full_name, orders.order_id from customer death, the ownership of the copyright to Clare’s unpublished writings traditional patterns of life caused by the enclosures of the English inner join orders on customer.customer_id=orders.customer_id AND is still claimed as the sole property of one individual, Professor Eric commons. Like Gerard Winstanley, Clare believed the earth to be a customer.customer_id=’{$_GET[‘customer_id’]}’”; Robinson, who purchased the “rights” for £1 in July, 1965. This has “common treasury for all”. It seems unlikely that Clare would approve # this query is used is user selects all customers been a point of contention among Clare scholars and publishers for of a situation where the rights to his work were enclosed and claimed $sql_all=”select customer.full_name, orders.order_id from customer Using joins and PHP provides a powerful means to generate business the last 40 years. as the property of one individual 140 years after his death. inner join orders on customer.customer_id=orders.customer_id”; intelligent information in a user friendly manner. Being web enabled At the time of Clare’s death the ownership of Clare’s unpublished “And me they turned me inside out is the ultimate user interface; since no installation of client work was passed to James Whitaker, best known as the creator of For sand and grit and stones # determine which query to use applications is required, reports can be made available to anyone with Whitaker’s Almanac, with the intention of producing a posthumous And turned my old green hills about if ( $_GET[‘customer_id’] == ‘all’ ) a browser. collection. According to John Goodridge, writing in the Guardian in And pickt my very bones.”

72 LinuxUser & Developer LinuxUser & Developer 73 Regular features

Otherwords “The absolute transformation of everything that we ever thought about music will take place within 10 years, and nothing is going to be able to stop it. I see absolutely no point in pretending that it’s not going to happen. I’m fully confident that copyright, for instance, will no longer exist in 10 years, and authorship and intellectual property is in for such a bashing”

David Bowie

LinuxUser & Developer 75

Full Name	Item(s) Ordered
$full_name	$order_id