Development of TRGnomoradio

Dan Stevens Department of Software Engineering University of Wisconsin, Platteville Platteville, WI, 53818 stevenda@uwplatt.

Abstract

“Gnomoradio is a program that can find, fetch, share, and play music that is freely available for file sharing.” [1] Unfortunately, the official Gnomoradio client can only run in a Linux GUI environment, and I wanted to run it on my Windows machine using Winamp to play songs. Being fairly experienced writing networked applications, and wanting to practice my C# skills, I set out to write my own Gnomoradio client.

This paper will cover my experiences developing my Gnomoradio client, from selecting a platform, to reverse-engineering the Gnomoradio protocols, designing a user interface, and how my chosen programming language would make or break the project. In short, this is a case study showing how all these Software Engineering principles we've been learning about actually get applied (or not) to a small, one-man project.

Reasoning

I heard about the Gnomoradio project via Slashdot [3]. At the time I had several hundred megabytes of my own music lying around various websites, and I was looking for some free way to promote it (music.download.com was nice but only gave me space for about 3 songs). Gnomoradio looked like the perfect opportunity, and so I signed up as an artist and listed all my songs and URLs to their respective files.

But I had no way of using Gnomoradio to find other people's music, or even to test if my music was actually showing up to Gnomoradio users, as the only client was designed for use in a Linux GUI environment. My Linux box acts as a server only (it doesn't even have a sound card), so that wasn't an option for me. What I really wanted was a program that could query the Gnomoradio network for new songs, enqueue them in Winamp, give me an easy way to rate them, and send in my ratings and download new songs for me. I didn't want or need a fancy GUI or a built-in music player, so even if the official Gnomoradio client had run on my platform, I might have wanted to write my own, just so that it would fit well with the way I work (playing songs through Winamp rather than yet another media player).

Selecting a Platform I had recently received a free copy of Visual Studio .NET 2003 from ELMS and had used C# as part of a CS383 assignment. Unlike Ruby, the language I usually use, which is primarily designed to work in a Unix environment, C# (well, the .NET framework, actually) is designed to work smoothly with Windows. Visual Studio makes making pretty GUIs really easy. It seemed to me a good idea to get to know another language, and something more mainstream like C# would likely look good on resumes. So I started work on TOGoS's C# Gnomoradio client, or TCGnomoradio.

TCGnomoradio

Being a small, one-man project, I didn't think it necessary to go through all the Software Engineering formalities that I've been taught in school: requirements analysis, specification, design and architecture, coding, testing, documentation, and maintenance [4]. Aside from a formal specification, all of these things would get done eventually in some way, but since I was the only person working on the project, there was no need to formalize anything, and since it was such a small project and I could already fit most of it in my head, there was not much need for planning, either. I would define the requirements as I got around to implementing each individual module. At least that was the idea.

The first step (after getting the Winamp interface working, but that was a separate project) was to write modules to implement the Gnomoradio protocols. But at this point I didn't yet know what the Gnomoradio protocols were. In SE-speak, interacting with the rest of the Gnomoradio network using Gnomoradio's protocols was one of the informal requirements of TCGnomoradio, and now I needed to more formally define this requirement based on what the protocols actually were. This was fully expected. What wasn't expected was that the protocols hadn't been previously been documented by anyone else.

So I started looking through the source code of the existing Gnomoradio client. From that I managed to figure out the rainbow hub protocol and the recommendation protocol. Since the song information is represented RDF encoded as XML, it was easy to realize what all the entity types and attributes meant simply by looking at the RDF that was generated for me by the 'Rainbow Builder' web application on the Gnomoradio website. I sent off an email to the Gnomoradio mailing list, and the author of the original program gave me some tips and confirmed my ideas about the protocols [5].

It turns out that the Gnomoradio authors did a pretty good job of making Gnomoradio easy to re-implement, as it's mostly based around HTTP and XML encoded RDF. There are HTTP client libraries and XML and XML encoded RDF parsers available for just about any language I'd care to use, so all I need to do to deal with the rating/recommendation system and the importing of meta-information is to import a few libraries and make a few method calls. And the protocol for interfacing with the hub is so simple that it takes even less code. If there's any trickiness to writing this program, it must lie in the user interface. Making a GUI

Now to get things working I needed some sort of framework to plug all these modules into. Since I was planning on having a nice GUI, anyway, I just worked on that. The GUI was actually the first thing created once I had documented the protocols. I put in a tab control so that I could switch between all the different things I would want to look at – eventually this would include a tab for viewing information on the current song (with a 'refresh' button which would try to determine what song Winamp was currently playing and show its information), a tab showing recommendations (and a button allowing the user to download new ones), a tab allowing the user to manually query the Rainbow hub, a tab showing all downloads in progress (this includes downloads of RDF files as well as downloads of audio files), and a tab showing information on all song, artist, and file descriptors that the program knows about. The end user would probably not be very interested in most of this information, but I need it to know it for debugging purposes, and it might be nice to have anyway.

Designing the GUI was fun and easy, thanks to Visual Studio. Making buttons respond to clicks and get data from forms was also easy enough, thanks to Visual Studio. Where it got tricky was updating the GUI according to received data. For instance, after I ask the hub to give me a list of servers to download a file from, I don't want to just sit around waiting until I've read the whole response because it's not guaranteed that I'll get it right away, and I don't want the GUI to become unresponsive. Therefore I need to spawn a new thread to read the response and update the GUI when it's done. I'd also like to indicate in the GUI that something is happening, by graying out the 'search hub' button and changing the cursor to an hourglass when it is over the button.

Although .NET does some things to try to make it easier to call methods asynchronously, namely the BeginInvoke method on delegates which calls the delegate in a new thread and lets you specify a callback to be called when the operation completes, it's still rather painful to bring everything together to make that happen. I need to create one method to be called when the button is clicked, one method to make the query and read the results from the server (in another thread), one method to be called when the search is done to fill a ListBox with the results. In addition to all that, I had to declare a new delegate type:

private delegate string[] RainbowHubSearchDelegate( string resourcename );

C# seems to be a rather 'wide' language. My 'on-click' method looks like so:

private void RainbowHubSearchButton_Click(object sender, System.EventArgs e) { RainbowHubSearchDelegate del = new RainbowHubSearchDelegate( this.RainbowHubSearch ); del.BeginInvoke( this.RainbowHubSearchBox.Text, new AsyncCallback( this.RainbowHubSearchDone ), null ); }

My verbose variable names probably don't help much, but a lot of the built-in class names and method names look like that, too. This all wouldn't be so bad except that each of these methods ended up being a lot of work. The 'on-search-complete' callback needed to take an IAsyncResult as its only parameter. This I had to cast to an AsyncResult, and then get its 'AsyncDelegate' property to get my original RainbowHubSearchDelegate. Then, to get the results of calling my hub querying method, I needed to call 'EndInvoke' on the delegate. This made for a lot of typing. I also wanted the query hub button to change dependinding on whether or not the hub client was already in use, and not just whether or not it was being used by the user clicking on it. To make this happen I wanted the hub client to trigger an event that would indicate to the GUI when it was in use.

Events in C#, and why they are such a pain to work with

C# provides you with an “event” keyword. You can use this keyword in a class definition to declare that objects of this class will raise the event you specify:

event SandwichEatenEvent SandwichEaten;

When your object sees that it has just eaten its sandwich, it should trigger the event, and anyone who wants to know about it every time a your object's sandiwch is eaten can register to have a delegate called whenever that happens. This is very fine and useful. Unfortunately there is still a lot of work that you must do: you must declare the event type, 'SandwichEatenEvent' as a subclass of Event, and you must declare a delegate type and create a new instance of that type to subscribe to the event. Since the object that represents the event handler, 'SandwichEaten' in my example, can be null if nobody has subscribed to it, you must also make sure it is not null before you try to trigger the event:

if( (SandwichEatenEvent see = this.SandwichEaten) != null ) { see( this, new SandwichEatenEventArgs( 1,2,3 ) ); }

This all gets rather tiresome if you have a lot of events to trigger.

I want it to trigger 2 events, Locked and Unlocked, that my GUI can subscribe to and deactivate or activate the query hub button when the hub client is in use or no longer in use, accordingly. I sometimes need to lock the object using C#'s 'lock' keyword:

lock( object ) { /* do stuff with object */ }

Every object can be locked, but unfortunately for me, every object does not have Locked and Unlocked event that you can subscribe to. I'd like to override my object's Lock and Unlock methods to trigger Locked and Unlocked events, but in fact there are no such methods to override. Locking and Unlocking are low-level operations accessible as the System.Threading.Monitor.Enter and Exit methods, for which C#'s 'lock' keyword is only syntactic sugar [7]. Since there seems to be no way to override the way an object behaves when it is locked by that usual method, I simply invented Lock and Unlock methods specifically for the HubClient class which trigger Locked and Unlock events. And now if someone wants to lock my object when they use it, they should not use the lock keyword. Instead, they must be aware that my object is special and write some extra code. This is what my hub search method actually ended up looking like:

private string[] RainbowHubSearch( string resourcename ) { string[] results; try { this.RainbowHubClient.Lock(); results = this.RainbowHubClient.FindResource( resourcename ); } finally { this.RainbowHubClient.Unlock(); } return results; }

Now I was done with the hub querying. Had I known what I was doing, it would have taken me only 5 minutes or so to implement everything I did, including all the event and delegate type declarations, the checking for null, etc. This being my first experience dealing with events in C#, and not knowing exactly what I wanted this program to do, it took at least an hour.

I went on to implement the descriptor manager (an object that downloads meta- information in the background), the download manager, and the recommendation manager, each of which needed to trigger several events, and each of which had a corresponding display in the GUI that would be updated whenever something it contained (information about a song, status of a related download) changed. I was keeping my head so full of event handler code that I was no longer able to keep the structure of the entire program in my head anymore. What events do I need to subscribe to? Do I ever need to unsubscribe? Is anybody going to care about the changes I am making here? Do I already have an event class that I can use, or do I need to define a new one? I was constantly having to look back and forth between different modules to remind myself what was supposed to be happening where. I became hesitant to add features because every little addition was so tedious and frustrating. Development slowed to a crawl, and I never managed to 'finish this thing over winter break', as I told the Gnomoradio mailing list I would.

Insight from a Ruby programmer

A big reason people are all about spending so much time writing up specifications and diagrams, aside from making the system maintainable to future developers, is that they cannot hold the entire program (or the module they are currently working on) in their head at once. These diagrams provide a quick reference to keep them on track. That's obvious enough, and I have experience having to make such high-level diagrams to keep myself on track designing CPUs for my Microprocessor Design class. But the reason these software people can't hold the system in their heads is not only that the systems they are designing are large and complex, but also that they use programming languages that force them to write all sorts of extra code. I'm not used to this. I think: Why should I need to declare a delegate type and a new event type for each event? Why should I have to write my own lock methods just so that I can have handlers for lock events? Why should I have to check for null events before triggering them? Why should I even have to declare the types of all the parameters and return values of my methods? Aren't these all things the HLL should take care of for me? I think so, but apparently the people that designed C# didn't. I will admit that C# does a better job of helping me out than some languages do. Java, for instance, forces you to use 'getter/setter' methods and place 'throws' declarations all over the place. It puts restrictions on what classes can be defined in what files (only one public class per source file), and doesn't even make an effort to make event-driven programming easy. But I come from a very Linux-y background, where so-called 'scripting languages' seem to be much more popular. Where if you suggest using Perl for a project, it'll actually be considered, not jokingly abbreviated “P.” and noted on the far end of the whiteboard as if were obvious that Perl is totally worthless. I have to step out into the Corportate Programming Language environment once in a while just to remind myself of why I don't step into it more often, as languages like Perl, Python, Ruby (and even PHP) have completely spoiled me.

What's so great about Ruby?

Ruby is a hacker's dream come true. It has all the string processing features of Perl (regular expression syntax built into the language – you don't need to go around explicitly creating Regex and Match objects like you would in Java or C++, though Ruby will certainly allow you to do so if that's your thing), the readability and writability of Python (Ruby's syntax is at least as clean as that of Python – no semicolons required at end-of line, no parentheses required for method calls or 'if' statements, and unlike Python, Ruby's blocks are always marked by an 'end' statement (or an end curly brace; you can use '{ ... }' in place of 'do ... end' to mark a block if you want)). Ruby has all the OO features of Java, borrows several from Smalltalk, and adds more. Would you like to add a new method to the base String class? In Java or C#, you'd be told that that's not allowed, but if that's what you want to do, Ruby won't get in your way. How about implementing an iterator? Normally you'd have to define an iterator class with goForward, goBack, getCurrentItem, and isThereMore Methods. Not so in Ruby. Ruby makes every effort to give the programmer just what he wants with minimal fuss.

Remember all that nonsense I had to go through to do asynchronous method calls in C#? Here's how I could've done it in Ruby (note the lack of casts, excessive method definitions, and an extra 18 lines of code):

Thread.new() do result = NIL @hubclient.synchronize { result = @hubclient.findfile( url ) } @hubclientresults.items.replace( result ) end As for events, Ruby has no built-in support for events. However, after writing a 35-line library called 'TOGoS/Event.rb', event-based programming in Ruby is 50 times easier than it ever was in C#. Here's a fully functional example, requiring only TOGoS/Event.rb: require 'TOGoS/Event' class MyClass event :sandwich_eaten def eat_sandwich() @sandwich_eaten.trigger( self,1,2,3 ) end end

myobj = MyClass.new() myobj.sandwich_eaten.subscribe() do |sender,*args| puts “Teh sandwich was eaten. Here are some args: #{args}” end myobj.eat_sandwich() prints

Teh sandwich was eaten. Here are some args: 123

TRGnomoradio

TCGnomoradio obviously wasn't going to be completed any time soon. Progress was going way too slow for me to possibly finish it by the end of winter break. So I fell back to plan B: Write it in Ruby as a web application. This would have a number of advantages over writing a GUI in C#:

1. I'm much more experienced working with Ruby and web interfaces (commonly together) than I am with C# and GUIs. Familiarity alone should speed development significantly. 2. I had been a little hung up on finding an embeddable web server for C# programs. Everything I found wanted me to have IIS installed. In Ruby, I can use TRServ2, an easily embeddable web server I had written in Ruby for previous projects. This web server will serve the UI to the user, and files to other Gnomoradio users. 3. Using a web interface, the user loses the expectation that data fields will be updated on the fly, as data changes. I would no longer have to worry about the majority of the events that were causing me such pain. Rather, TRGnomoradio would generate web pages for you when you asked them, from current data. 4. Showing images and tables is way easier if you can simply write some HTML than if you have to code and keep track of GUI components. You write HTML however it needs to be to accommodate your data and let the browser worry about it. 5. The user can have multiple views of the same data just by opening multiple tabs. We backend developers are freed from worrying about what the screen size is, what text the user has selected, or what other tabs are open. 6. Ruby + Web interface = automatically cross-platform. The only non-cross-platform element of TRGnomoradio is Winamp. But, because the program communicates with Winamp via HTTP, you could easily run TRGnomoradio on one computer and Winamp on another. Or you could write a separate module to use your favorite media player instead of Winamp. Ruby makes writing applications with such modular architectures easy. And with SS4 (see below) it's even easier.

So I started writing TRGnomoradio. As with TCGnomoradio, the first step was to create a framework to plug all the various modules (resource manager, recommendation manager, download manager) into. The difference from TCGnomoradio being that this framework took the form of a TRServ2 web object instead of a System.Windows.Forms object.

SS4 and TRServ2

TRServ2, as I mentioned above, is a web server I developed for a previous project. SS4 is a framework for running daemons written in Ruby. There's a bit of history behind them (hence the numbers), and that's as good a story as any of this, so I'll give an overview, here.

SpookShare was a file-sharing network I invented back in 11th grade because I wanted to do something with my newly acquired Perl sk!11z. It required you to install a bunch of Perl CGI scripts on your web server, and was never really very popular.

SS2 was another, mostly unrelated file-sharing project that I started years later. Unlike the original SpookShare, SS2 was a daemon written in Ruby and included its own web server, TRServ.

TRServ was an embeddable web server I wrote (in Ruby, of course) primarily for use in SS2, but also used in a few other of my projects. It was so easy to stick into and integrate with an existing Ruby program that sometimes I had to fight the urge to embed it into projects completely unrelated to web servers.

TRServ had some shortcomings, though. I realized these shortcomings while I was trying to transfer a large number of small files from an instance of SS2 using wget. Wget took a long time between each file it transferred – usually about 5 seconds – and this was making the process of transferring all the files go very slow. I eventually realized the problem was that wget was trying to transfer files via a persistent connection, and since TRServ didn't support these, the attempts always failed and for some reason this caused wget to sit around doing nothing for an excessive amount of time after each download. Unfortunately, the way I had written TRServ there was little abstraction provided to web objects to interface with the browser, so it would have been tricky to make it support persistent connections.

SS3 was yet another iteration of SpookShare, this time using a sub-program called TRVous, which used a lot of UDP packets to do peer discovery. Now knowing how important it was to support persistent connections, I took this oppurtinuty to rewrite TRServ and produced TRServ2. This time I made sure that my web server would be able to do advanced HTTP stuff (persistent connections, if-modified-since support, ranged responses, etc) without trouble. Instead of making sub-programs print out an HTTP response directly, I now had them return a 'Response' object, containing any information they might want to send to the client. All the work of translating that object to a response stream is handled in a single function. This way even though the code to support HTTP/1.1 may be gross, it is all contained in one place, making it much easier to maintain than the mess of pseudo-stream objects TRServ used. But when I tested the system on my windows machine, it seemed SS3 and TRVous were having some problems. For unknown reasons, opening a UDP socket from a Ruby program running as a Windows service while your network cable is unplugged results in the process hanging and taking up all your CPU time. I couldn't go around putting that crap on people's computers, so I came up with an alternative peer discovery protocol which involved posting information to a livejournal account:

SS3's dramatized livejournal slttygal557: “omfg my ip address is 137.104.57.183 come dl nmy filez” z_snatch: “lolololol check out the cool animadid gif on my comp 137.104.77.151” mb_coolio: “don't you think ivan from roundhouse is hot!?!?!!” slttygal557: “u hore ^-^ what is ur ip addreess” mb_coolio: “oh so sry T-T 137.104.76.207”

I wrote a module to implement this protocol and added it to SS3. As I added features to SS3, it was becoming a hacked together framework for loading and setting options for different modules based on a set of configuration files. My roommate and I were coming up with new ideas for things we'd like to have SS3 do which would work pretty well as new modules, and I decided that if it was just going to be a framework for loading modules, then I ought to do it right. Hence SS4.

The problem with SS3 was that all the option parsing code was in one file. That code had to be able to parse and act upon every option for every module. That was fine if there were only 3 modules that were ever used, but the way I was adding new modules (or thinking about adding new ones), that one file would end up very big and gross. SS4, instead of implementing everything in one place, gives a set of basic commands for loading objects into the SS4 framework, and provides those modules with methods to add their own configuration directives. This way you don't need to change any existing files to add a module – you can simply write your module file and drop it in the library directory.

It turned out that a lot of the programs I was writing used the same kind of configuration files that I was having SS4 deal with. Turning these programs into SS4 modules not only saved me the work of rewriting all the configuration loading code again, but also made it easier to integrate them with other SS4 modules. Since TRServ2 was already an SS4 module and TRGnomoradio was just a couple of TRServ2 web objects, some configuration directives, and a bunch of backing classes, TRGnomoradio also fit well as an SS4 module. Hoorah for simple, easy-to-use frameworks.

TRGnomoradio was almost done, but unfortunately I ran into a bit of a stumbling block. When I tried to run it on my Windows machine it was having timeout errors and never able to connect to the recommendation server. This problem really shouldn't take too long to figure out, but unfortunately I couldn't spend the time to do so as all my classes had started piling homework on me. This illustrates another problems of Software Engineering: not budgeting time correctly. Although I was correct in assuming that TRGnomoradio would be a lot easier than TCGnomoradio, I had underestimated the amount of time I would spend doing EE labs. Comparing to a standard software development model

If I had followed a more standard software development model, development would have gone like this:

· Write detailed requirements. · Buy lots of expensive (but not necessarily helpful) 'CASE' tools like Rational Rose. · Go through requirements documents, pick out nouns and verbs, and put them in a UML diagram. · Spend way too much fighting with design software and writing time logs. · Break up the parts of the program and work on then one at a time. · Realize half way through that the design is not nice to implement. · Redesign. · Realize that I don't have time to do this again because now I have EE labs to do and a paper to write. Try to work with the old design.

If I had a team of people to work with, laying out a design ahead of time would be very useful, and that time would be well spent. It would keep people on track and let them know exactly what their portion of the program needed to do to work correctly with everyone else's. Even for a single-person project, a well thought out design would be good if the program was too big to keep in one's head, or if I planned to go away and come back to it. But for smaller projects, I've found that the problems of keeping myself on track are better solved by writing small, clear, self-documenting code, and that such code is much easier to write in a high-level language than a low level one. The TOGoS software development model therefore goes like this:

· Don't tackle something huge that's never been done before or that I think I'll get bored with before I finish. · Choose a platform based on what kind of application it is. Use Ruby if I can do everything via web interface. Use C or C++ if I'm going to be doing a lot of computationally intensive operations. Mix Ruby and C if I need both. · Get a general idea of what it needs to do, but leave low-level decisions until I'm actually coding. · Code one part at a time, testing and integrating with the rest of the application as I go. · Realize that the whole thing could be much cleaner and easier to work on if I had architected it some other way. · Start coding again, possibly re-using modules created for the old code. · Get bored with it half way through and decide to do play with something else for a while.

The TOGoSian model assumes that most projects will fail, and therefore does not have me spending a lot of effort on diagrams and planning that will inevitably be obsoleted as soon as I start coding. It is therefore cheaper and faster than the standard model, and when I'm done I've learned a lot more than how to fill out time logs. New modules and new programmer skills can be put into new projects faster, and projects that would otherwise be hard to tackle become small quickly as new knowledge is accumulated from other projects. In short, this model treats a project as a learning experience rather than as an end in itself.

Of course this model is not for everyone. If you're a software company and have a client that needs a product right now, you can't take the chance that it won't actually be completed. Your client wouldn't take it too well if you told him “Oh, yeah, we tried to write this program for you but... halfway through we got bored and played Quake for several days.”.

This model in its purest form wouldn't work well for projects with more than one person. At the very least, you'd need to agree on a platform, break it up into clean modules, and define interfaces.

But for non-critical applications this process works great. If you get something working, that's neat, but even if you don't, it doesn't hurt much, as not a lot of resources went into the project. The bottom-up approach ensures that modules can be re-used in future developments of this project or even unrelated ones, making those projects cheaper and giving them a higher chance of success.

References

1. Gnomoradio.org front page, retrieved March 14, 2005 from http://gnomoradio.org/

2. References APA Style, Retrieved March 14, 2005 from http://www.library.uq.edu.au/training/citation/apa.html

3. Gnomoradio: Creative Commons Music Sharing, Retrieved March 17, 2005 from http://slashdot.org/articles/04/09/10/1439250.shtml?tid=141&tid=95&tid=1

4. Software Engineering Wikipedia article, retrieved March 19, 2005 from http://en.wikipedia.org/wiki/Software_engineering

5. TOGoS, Gnomoradio Protocols (and included emails), retrieved March 19, 2005 from http://www.uwplatt.edu/~stevenda/projects/TCGnomoradio/protocols.html

6. Roland Weigelt, Raising C# Events Doesn't Feel Right, retrieved March 20, 2005 from http://weblogs.asp.net/rweigelt/archive/2005/01/15/353333.aspx

7. How is lock keyword of C# implemented?, retrieved March 20, 2005 from http://blogs.msdn.com/junfeng/archive/2004/02/18/75454.aspx

8. The httpQ website, retrieved March 20, 2005 from http://httpq.sourceforge.net/