Media Players Documentation Release 2.5
Isuma
January 28, 2015
Contents
1 Hardware platforms 3 1.1 Shuttle XPC small desktops...... 3 1.2 Logic Supply desktops (v2.5 series)...... 3 1.3 Advantech rugged servers (1.0 series)...... 3
2 Installation 5 2.1 Install Debian...... 5 2.2 Install local server software...... 6 2.3 Configure Local Server...... 6 2.4 SSH upload account configuration...... 6 2.5 SSH tunnels configuration...... 7 2.6 First file sync...... 7
3 Maintenance 9 3.1 External synchronisation drives...... 9 3.2 Syncing a media player...... 10
4 User guides 11 4.1 How to setup a playlist...... 11
5 Troubleshooting 19 5.1 Test procedure...... 19 5.2 Troubleshooting stuck queues...... 19 5.3 Queue is full but media player sees it empty...... 20 5.4 Logging in to media players on the console...... 20 5.5 Password resets...... 20 5.6 Inspecting status from the commandline...... 20 5.7 Remote login to media players...... 21
6 Development 23 6.1 XMLRPC API...... 23 6.2 Debian packages...... 23
7 Design documentation 25 7.1 Terminology...... 25 7.2 Design strategy for 3.0...... 26 7.3 Similar projects...... 27
8 Original specifications 29
i 8.1 Overview...... 29 8.2 Hardware possibilities...... 32 8.3 Auto-mounting Hotplugged Devices...... 33 8.4 Synchronization scripts...... 33 8.5 Test scripts...... 34 8.6 How to set up kiosk mode...... 35
9 Changing this document 37
ii Media Players Documentation, Release 2.5
The Isuma Media Players project is a fully free and open source software project to create a two-way, distributed content distribution network for communities with poor connexions to the internet.
Contents 1 Media Players Documentation, Release 2.5
2 Contents CHAPTER 1
Hardware platforms
The media players project doesn’t require any special hardware to run, although we do expect the usual: • network interface • display and sound card (to play videos) • storage However, since we are storing videos, we are aiming for large storage specifications. As of July 2014, the Isuma.tv store is around 1 TB (terabyte, or 0.9 TiB, tebibyte), so we currently deploy 2TB hard drives. The rest of this section is dedicated to current and previous platform specifications.
1.1 Shuttle XPC small desktops
We have one Shuttle machine in the office, in the XPC series, there isn’t much to say about it other than its peculiar form factor is not very convenient.
1.2 Logic Supply desktops (v2.5 series)
Around 10 machines were built with some Logic Supply Mini-ITX cases, although the original product link is now dead. We also had trouble with shipping and delivery to Canada. Finally, some hard drive sockets were damaged during travel, which makes us doubt of the viability of this platform on the long term.
1.3 Advantech rugged servers (1.0 series)
We have deployed some Advantech UNO-3282 servers on the field.
Two of those servers were provisionned for Isuma and are still running after years of service. They have sealed cases that are very solid. Advantages: • very sturdy • sealed, so it won’t collect dust • power button protected from accidental tripping
3 Media Players Documentation, Release 2.5
Disadvantages: • heavy • power supply is external
4 Chapter 1. Hardware platforms CHAPTER 2
Installation
This describes how to set up a local server from scratch. You may be able to install the components on an existing system, but this is currently not supported.
2.1 Install Debian
To begin, boot into the Debian “stable” installer, either from USB, CD-ROM or from netboot. Note that the boot priority in the BIOS of the UNO-3272 has options for both USB CDROM and normal IDE CDROM so make sure you pick whichever one you are using. Set up keyboard, timezone, and other settings as normal. When you get to the partitioning screen, set up partitions as follows: First hard drive (sda): 1. Primary, filesystem ext3, size 200MB, bootable flag, mount as /boot 2. Primary, LVM, size 1TB Second hard drive (sdb): 1. Primary, LVM, size 1TB Now choose “Write changes to disk and configure LVM.” Set up LVM as follows: Physical volumes: sda2 and sdb1 Volume group: call it main, and set it to use all of both physical volumes. Logical volumes: 1. Name: root, size 80GB 2. Name: swap, size 3GB 3. Name: media, size 2TB Save the changes, and go back to the partitioning screen. Set up the logical volumes to be mounted as follows: 1. root: filesystem ext3, mount point / 2. swap: use as swap 3. media: filesystem ext3, mount point /var/isuma Now choose to continue and the volumes will be formatted. Formatting the media partition takes about a half hour and looks like it’s stuck at 33% completed for that entire time, so go have a coffee. Set up root and normal users.
5 Media Players Documentation, Release 2.5
Next, choose a local Debian mirror. On the software selection screen, select the “standard system” and “Web server” software collections.
2.2 Install local server software
In /etc/apt/sources.list add: deb http://debian.koumbit.net/debian stable main
Then run: apt-get update apt-get install koumbit-archive-keyring
You will see this warning: W: GPG error: http://debian.koumbit.net stable Release: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 0x6DC388D8B7C0A70A
It is normal, you need to bootstrap the trust to the repository. Optionnally, you can also download the key directly from the web of trust and verify the signatures to check its authenticity. Then run: apt-get install isuma-local-servers isuma-autossh
Optionally, if you want to setup kiosk mode, run: apt-get install isuma-kiosk
2.3 Configure Local Server
Create the config.inc file in the /usr/share/isuma-local-servers/sync/. Configure the unique username/password to access cs.isuma.tv in the form: username:[email protected]. Make sure file is properly formed with opening The username refers to a user that is created within the drupal at cs.isuma.tv.
2.4 SSH upload account configuration
There is configuration required to setup the rsync between the media player and the main isuma.tv website. Copy the [email protected] private key to /var/isuma/.id_rsa on the media player. The key can be found on the isuma.tv server at /home/cachingserver/.ssh/id_rsa. Be sure that it is readeable by and only by www-data (chown www-data /var/isuma/.id_rsa; chmod 600 /var/isuma/.id_rsa). Then run: su www-data -c "ssh [email protected] -i /var/isuma/.id_rsa"
NB: I didn’t do that for v25n11 as there is no caching server and couldn’t find the specified files. Important: Save the fingerprint by entering “yes”
6 Chapter 2. Installation Media Players Documentation, Release 2.5
2.5 SSH tunnels configuration
The SSH tunnels allow Isuma administrators to login remotely on the media players, from the central server. They work with a package called “autossh” that tries its best to keep an SSH connection to the central server open. That connection is then used to create a tunnel back to the SSH server running on the media player. The isuma-autossh package should automatically prompt you for the relevant information. You need to provide it with the public key of the isuma central server, which is: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdr5vyuwKikrW0Vg+mlsRTkG1Jp3wUt4JmLopspln5sWsbAsMDtnUe7T0NW6s6W2tYvlav9m987HrNvAXCWbBeOwIuabbtcUK39QRdEUrAcd1yConPxmnsk7+sN2p7NC76lYIXYQM7hT2pWa3rXpdZQT2leSrrWFW0j/PEvwsEvb+4vmm2POX2L9GJDeb/czdT6/3NNRVlDiSh4rORbtXgDq7cj2qKZ0xnzDOcOpSjXig2eLl6759qLV3p76iNmMoHjWwo3bRypkwtiZ3zmwBPtGlctezgHqukaUKwcbDRNoZgv99QYLzAYqjx7ZqwWBCaNixaXgZhNkKhmlVSRXZB root@ip-10-122-35-248
(!) If that file is changed, you can use the dpkg-reconfigure isuma-autossh command to copy and paste the proper key back into place. Then the isuma-autossh package generates a public key in /root/.ssh/id_rsa.pub, which needs to be manually copied at the end of the /home/tunnel/.ssh/authorized_keys file on the central server.
2.6 First file sync
(From #7744.) Here’s the resyncing process for the file queue. Note that we actually inject the files in the download queue to make sure we haven’t missed anything, but the files could also be injected straight into the File queue. On the central server, list the queues for the media player: mysql> SELECT csc.uid, csc.caching_server_queue, ns.title, ns.sqid FROM caching_server_config csc JOIN nodequeue_subqueue ns ON ns.qid = csc.caching_server_queue WHERE csc.uid=91; +-----+------+------+------+ | uid | caching_server_queue | title | sqid | +-----+------+------+------+ | 91 | 86 | download | 307 | | 91 | 86 | upload | 308 | | 91 | 86 | file | 309 | | 91 | 86 | transcode | 310 | +-----+------+------+------+ 4 rows in set (0.00 sec)
This is an easy way to list the subqueue identifiers for the given media player. In this example, this is media player #6 (uid 91). From this we get the queue id (86) and the subqueue id (307) for the download queue. Then we need to do the same for a reference MP, we take the isuma office one even though we know that it is missing some files: mysql> SELECT csc.uid, csc.caching_server_queue, ns.title, ns.sqid FROM caching_server_config csc JOIN nodequeue_subqueue ns ON ns.qid = csc.caching_server_queue WHERE csc.uid=63; +-----+------+------+------+ | uid | caching_server_queue | title | sqid | +-----+------+------+------+ | 63 | 62 | download | 214 | | 63 | 62 | upload | 215 | | 63 | 62 | file | 216 | | 63 | 62 | transcode | 217 | +-----+------+------+------+ 4 rows in set (1.75 sec)
Then we make the crazy request to inject all files from the reference player into the poor little media player:
2.5. SSH tunnels configuration 7 Media Players Documentation, Release 2.5
mysql> INSERT INTO nodequeue_nodes (SELECT 86,307,nid,position,NOW() FROM nodequeue_nodes WHERE sqid = 216); Query OK, 69566 rows affected, 65535 warnings (3.47 sec) Records: 69566 Duplicates: 0 Warnings: 69566
This may unfortunately introduce duplicates in the file table, but that’s better than an empty table! Then do clear up the download queue, you need to hit the central server for every single file in there: su www-data -c "while true; do /usr/share/isuma-local-servers/sync/download.php; done"
8 Chapter 2. Installation CHAPTER 3
Maintenance
3.1 External synchronisation drives
(See #181 for background.)
3.1.1 External drive format
A drive is identified as carrying Isuma content if it has a folder in its root called ISUMA_CONTENT (all caps). This folder is further subdivided by content type and content options. Type is, for example, video or picture, and options are things like low, high and ipod for various resolutions or bitrates. So, one might find a file structure like this on the drive: / ISUMA_CONTENT video high low ipod picture high low
The synchronization script recurses through this directory structure, copying all files from the external drive to the internal drive (with the exception of any files named “raw”, because raw files are not needed by any machine except the local server they were originally uploaded to, and the site server). It then does the reverse, copying all files on the local drive to the external. In this direction, raw files are copied, because if the upload queue is large, it may be possible for a file to arrive at the site server via hard drive before it arrives via rsync.
3.1.2 Creating a new synchronisation drive
This must be done from an existing media player. 1. connect the drive 2. find the drive identifier (dmesg | tail or cat /proc/partitions) 3. format it with an ext3 or ideally ext4 filesystem: mkfs -t ext4 -j /dev/sdb1 4. mount the drive: mkdir /mnt/sdb1 ; mount /dev/sdb1 /mnt/sdb1 5. create the magic directory: mkdir /mnt/sdb1/ISUMA_CONTENT
9 Media Players Documentation, Release 2.5
6. launch the sync script (or reconnect the drive): /usr/share/isuma-local-servers/automount/driveadded.sh sdb1 This should take around 10 hours. See also #9275.
3.2 Syncing a media player
1. connect the drive 2. observe the led start flashing 3. wait for the led to stop flashing 4. disconnect the drive The media player should now be synced with the drive, and the drive should have the latest content from the media player.
10 Chapter 3. Maintenance CHAPTER 4
User guides
4.1 How to setup a playlist
(once the cabletv package is completed this will all be possible to do remotely through a web interface, and parts of it will be automated) 1. Open www.isuma.tv on your browser. 2. Make sure that you are connected to your IsumaTV Media Player:
3. Sign in with your username and password.
11 Media Players Documentation, Release 2.5
4. Go to your playlist by entering this URL: www.isuma.tv/DID/tv/name of your community and click on your playlist at the bottom of the page to open it.
12 Chapter 4. User guides Media Players Documentation, Release 2.5
5. Click on “EDIT THIS” under the DASHBOARD
6. Edit your playlist by adding and removing videos as well as changing their order. Note: Make sure to only select videos that are currently available on the MP and not videos still in the process of being downloaded to the Media Player. Note: Don’t forget to save your playlist regularly, even if you have not yet finished (the save button is at the bottom of the page). Make a note of the node number for your playlist in the address bar (in this example www.isumatv/node/56706)
4.1. How to setup a playlist 13 Media Players Documentation, Release 2.5
7. You can now save this new playlist to the Media Player for local broadcast. Note: The playlist can be modified from any computer but then needs to be saved on the Media Player in order to be broadcasted. In the address bar type www.isuma.tv/playlist/playlist/node number for your playlist In this example http://www.isuma.tv/playlist/playlist/56706
8. Make sure you see the IP address of your local Media Player. In this example: 192.168.0.110 If instead of reading numbers like this one, you see isuma.tv or amazon this means you are not connected to the Media Player. In this case, return to www.isuma.tv and make sure you have connected to the Media Player and can see the message “You are connected to an IsumaTV Media Player”.
9. Right-click in the middle of the page. Select “Save Page as”.
14 Chapter 4. User guides Media Players Documentation, Release 2.5
10. Change the file name to playlist.xspf (Your file can have any name but it is essential to save it as an .xspf file). Select “WebPage, HTML only” option at the bottom right of the window. Then click “Save”.
11. You will then return to the www.isuma.tv/playlist/playlist page. Press the “Ctrl” and “Q” keys on the keyboard to quit this page. The page will close and a VLC Player window will automatically pop up. Click the Play button, then select “Add. . . ”
4.1. How to setup a playlist 15 Media Players Documentation, Release 2.5
12. Select your playlist (playlist.xspf ). Click “Open”.
13. Click the Loop button and the Full Screen button on the VLC Player. Then Click the Play button to begin playing the playlist.
16 Chapter 4. User guides Media Players Documentation, Release 2.5
You are now done! To stop the playlist: 1. Press the “Esc” (escape) key on the keyboard. 2. Under the “Media” menu at the top left of the Player, select “Quit”. 3. The VLC Player will close and the web browser will reopen automatically on www.isuma.tv after a short delay.
4.1. How to setup a playlist 17 Media Players Documentation, Release 2.5
18 Chapter 4. User guides CHAPTER 5
Troubleshooting
5.1 Test procedure
When a new media player is installed, it needs to be thoroughly tested. This procedure can also be used on existing media players to diagnose problems. 1. ping/pong test 2. URL rewriting for recent and old videos 3. upload videos larger than 8MB without errors 4. upload queue immediately updated 5. upload syncs to website 6. upload audio to website
5.2 Troubleshooting stuck queues
Taken from #10232, to be expanded:
1. login to the server (ssh [email protected] ; mp_ssh_into n7) 2. become the proper user (su www-data -s /bin/bash) 3. look at the logfiles (/var/isuma/log/*.php) 4. if nothing comes to mind, run the script by hand, then look at logfiles again (e.g. /usr/share/isuma-local- servers/sync/download.php) 5. look at /etc/cron.d/isuma-local-servers for the location of scripts to tests To run the download script by hand: su www-data -c /usr/share/isuma-local-servers/sync/download.php
It can be useful to bump up the debugging by setting the DEBUG variable in /usr/share/isuma-local-servers/sync/conf.php to 2.
19 Media Players Documentation, Release 2.5
5.3 Queue is full but media player sees it empty
If the queue is full of good stuff to download but the media player sees it as empty, it could be that the schedule is too restrictive. Try to disable the schedule in the central server and try again.
5.4 Logging in to media players on the console
2.5 media players are in “Kiosk” mode by default, which makes it difficult to diagnose or see what is going on. A Linux console should be available if you type control-alt-F2. Then login with the usual password on the root account.
5.5 Password resets
If the password for the media player is lost, it can be recovered by rebooting in a special Linux mode. See Koumbit’s documentation for that purpose. This technique is complicated and should be considered last resort, if other techniques do not work or are unavailable, as it is difficult and prone to errors. This technique is known as “booting into /bin/sh as init(8)”. 1. reboot the machine (by doing control-alt-delete or by holding the power button for 5 seconds and then clicking it again) 2. you will then see the BIOS screen flash by quickly, then the GRUB menu, which should be shown for a few seconds, quickly hit the shift key to disable the timeout. 3. hit the e key to edit the menu item 4. you are now in an editor, with the arrow keys, navigate to the end of the line starting with linux 5. append init=/bin/sh to the line 6. hit control-x to boot the resulting configuration 7. you should end up at a commandline prompt, enter the following command in order (do not type what is after the # symbol): mount -w -n -o remount / # -w read/write, -n don’t write /etc/mtab mount /usr # in case passwd is on /usr/bin /usr/bin/passwd # sync umount /usr sync sync # it cannot hurt umount -a # will mount / read only reboot -nf # -n don’t sync or write anything, -f don’t call shutdown
8. the machine should reboot with the new root password
5.6 Inspecting status from the commandline
The mp_ssh_config script lists all the media players matching the given pattern (or all MPs if no pattern is pro- vided). Example:
20 Chapter 5. Troubleshooting Media Players Documentation, Release 2.5
$ mp_ssh_config mediaplayerv25 status wan_address lan_address ssh_port name Offline 208.114.191.198 192.168.1.117 14944 mediaplayerv25n1 Offline 65.181.41.57 192.168.1.116 12868 mediaplayerv25n2 Online 184.144.193.175 192.168.1.112 17900 mediaplayerv25n3 Online 70.25.31.106 192.168.2.16 27838 mediaplayerv25n4 Online 70.25.31.106 192.168.2.14 1597 mediaplayerv25n5 Online 68.67.33.253 192.168.0.77 15959 mediaplayerv25n6 Online 70.25.31.106 192.168.2.15 26620 mediaplayerv25n7 Online 173.177.189.83 192.168.20.233 31658 mediaplayerv25n8
This provides a quick overview of matching media players.
5.7 Remote login to media players
To login to the media players remotely through SSH, one need first to be logged into to the central server, then use one of the scripts deployed there (see also #7201 for history). The mp_ssh_into script allows you to log into one or multiple media players directly. You can specify a pattern or just call it directly, in which case you will log into working MPs one after the other. Example of running a command on multiple servers: antoine@ip-10-122-35-248:~$ mp_ssh_into v25 ls -l /var/isuma/.id_rsa found media player mediaplayerv25n1 on port 14944, sshing... ssh: connect to host localhost port 14944: Connection refused found media player mediaplayerv25n2 on port 12868, sshing... root@localhost’s password: -rw------1 www-data root 1671 oct 16 17:22 /var/isuma/.id_rsa Connection to localhost closed. found media player mediaplayerv25n3 on port 17900, sshing... -rw------1 www-data root 1672 oct 16 16:04 /var/isuma/.id_rsa Connection to localhost closed. found media player mediaplayerv25n4 on port 27838, sshing... ssh: connect to host localhost port 27838: Connection refused found media player mediaplayerv25n5 on port 1597, sshing... ssh: connect to host localhost port 1597: Connection refused found media player mediaplayerv25n6 on port 15959, sshing... -rw------1 www-data root 1672 oct 18 13:56 /var/isuma/.id_rsa Connection to localhost closed. found media player mediaplayerv25n7 on port 26620, sshing... -rw------1 www-data root 1671 oct 31 18:31 /var/isuma/.id_rsa Connection to localhost closed. found media player mediaplayerv25n8 on port 31658, sshing... root@localhost’s password: -rw------1 www-data www-data 1671 nov 12 16:13 /var/isuma/.id_rsa Connection to localhost closed.
This logs into all “v2.5” media players and perform one command. If the command isn’t specified, it just logs you in and gives you a shell on the given servers. I think this completes this request, if there’s anything else, please detail more specifically the problems.
5.7. Remote login to media players 21 Media Players Documentation, Release 2.5
22 Chapter 5. Troubleshooting CHAPTER 6
Development
6.1 XMLRPC API
(to be documented, see the archive for now.)
6.2 Debian packages
The Isuma Media Players make an extensive use of Debian packaging to deploy software but also configuration and policy. This section describes how the packages are built and maintained.
6.2.1 Automated package build system
Isuma Debian packages are automatically built by Koumbit’s Jenkins server. The complete documentation about this server is available in the Koumbit wiki, this is only a summary applicable to Isuma packages. When a change is pushed to one of the Debian packages git repository, they are automatically rebuilt within an intervall of around 15 minutes. The package is built within a Debian Wheezy environment and then uploaded into the Koumbit Debian archive, which is automatically signed. Packages are uploaded to unstable by default. To migrate them to testing or stable, a manual operation must be performed on the Debian archive, a server only Koumbit personnel currently has the access to.
6.2.2 Automated package upgrades
Since isuma-local-servers 2.5.0, upgrades are automatically performed on all Media Players. This is done through the use of the unattend-upgrades package. Packages from the Koumbit archive and the main Debian archive are automatically updated. To update more packages automatically, create a new file in /etc/apt/apt.conf.d the specify a new Origins-Pattern that is appended to the existing list. See /etc/apt/apt.conf.d/50unattended-upgrades or /usr/share/doc/unattended-upgrades/README for more information about this software.
6.2.3 Manually building a package
To build the current Debian packages by hand:
23 Media Players Documentation, Release 2.5
git clone [email protected]:isuma-local-servers.git cd isuma-local-servers git-buildpackage
To issue a new version, edit files, commit them, then bump the package version and rebuild: edit file/foo.txt git commit -m"update foo" file/foo.txt dch -r -i "updating foo" # increments the version number and inserts a commit in debian/changelog git-buildpackage # or debuild
Make sure you use -D stable, if you want to make a hotfix for stable. Package is now in .. or ../build-area. To upload the package: scp isuma-local-servers_* [email protected]:/var/www/debian/incoming then on the central server: sudo -u www-data reprepro -b /var/www/debian/ processincoming incoming kind of klunky but works.
6.2.4 Manually installing a package
Copy the package to the local server and run: dpkg -i isuma-local-servers_
If it complains about some dependencies not being installed, run: apt-get install to install them. After installing the package, you will need to perform a few additional steps: # get the ssh private key for the site server and place it in ISUMA_ROOT with the name .id_dsa. scp [email protected]:/home/cachingserver/.ssh/id_rsa /var/isuma/.id_rsa # (password in issue #187)
At this point you can check the logs in /var/isuma/log and make sure things are running properly.
6.2.5 Manually upgrading Media Players
Mass upgrades or installs can be performed with our scripts: mp_ssh_config | grep Online for s in mediaplayerv25n3 mediaplayerv25n4 mediaplayerv25n5; do mp_ssh_into $s apt-get update; done for s in mediaplayerv25n3 mediaplayerv25n4 mediaplayerv25n5; do mp_ssh_into $s apt-get install isuma-local-servers; done
This should normally not be necessary as the Media Players are automatically upgraded.
24 Chapter 6. Development CHAPTER 7
Design documentation
This document is a work in progress design document for the next generation of media players. It will eventually be expanded to explain various design decisions made during the creation of the next media players which we dubbed “3.0”. In doing so, we also explain some of the design of the previous versions, mostly for historical reasons. Previous historical specifications are also available in Original specifications. The basis of this document was originally written by John Hodgins in an email to Antoine Beaupré and has since then been repeatedly refactored.
7.1 Terminology
In this document, the following definitions are used: Local servers Physical machines installed in remote communities with low bandwidth, high-latency or no internet connectivity providing read and write access to a large media library. Media players The IsumaTV implementation of the local servers, mainly distributed in Canada’s northern native communities. Central server A Drupal website with a set of custom modules to keep an inventory of which files are on which local server, currently only deployed on cs.isuma.tv. Website The main Drupal site with all the content and user interface for the public, with custom Drupal modules to rewrite URLs to point to the media players if they are detected after a “ping pong test”. Currently isuma.tv. v1.0, first generation The first implementation of media players distributed during the first phase of the project de- signed jointly by Koumbit and Isuma.TV around 2010. v2.0, second generation, “3g” The second generation of media players was developped around 2012 to adress some bugs and issues with the central server, add a remote diagnostic system (isuma-autossh) and other updates. This is sometimes refered to as “3g” in the litteratue because part of the work on the second generation involved working on the design of a third generation. v2.5 The “v2.5” is an incremental update on the second generation to improve stability and fix a lot of bugs to ease deployments. During that phase, Debian packaging, install procedures and documentation were improved significantly. v3.0, third generation This new generation of media players, what is described in this documentation, most likely a complete rewrite of the local servers code. CDN Content Distribution Network, to quote wikipedia: [A CDN is a] large distributed system of servers deployed in multiple [locations]. The goal of a CDN is to serve content to end-users with high availability and high performance.
25 Media Players Documentation, Release 2.5
7.2 Design strategy for 3.0
The 2.5 Media Player code base as a bit of a proof of concept – it needs to be evaluated and imagine what we would build from the ground up. There are a number of things we could focus on for the long term: open sourcing, stability, and scalability. Decisions we make about these things should be applied to the current development work we are doing as much as possible.
7.2.1 Open Sourcing
The current code base is too specific to Isuma’s use-case to be valuable to anyone else. The next generation should be abstracted and generalized, in order to be useful to a wider variety of projects.
7.2.2 Core functionality
The Isuma Media Players project is a geographically distributed filesystem, with the files on local servers and file metadata on a central server. One could also describe the local servers and the central server as a CDN. Core functionality includes tracking of local server locations on the internet, along with files and basic filesystem functions (add, copy, delete, etc.) Code should be modular so that new functionality could be added and use existing functionality. A well defined API for communication between the different entities (local servers, central servers, clients fetching content, other clients fetching metadata). There are a number of use case scenarios for this kind of system, including ours: putting a webserver in front of these files for access on local networks. This is actually a major rift from the current code base, which is based on a paradigm of queues. The new paradigm would be files, keeping track of their locations, storing and making available their metadata, doing things with them and to them, etc.
7.2.3 System Architecture
The previous system architecture is better described in Original specifications. The basic design is a filesystem with a server and multiple dispersed clients. The server tracks the locations of remote clients on networks, locations of files on remote clients, and other metadata for files. The remote clients contact the central server in order to syncronize files and send location data. The central server also publishes information about files and metadata that can be used by other systems (such as a Drupal-based website) to access and control files in the filesystem. This should be constructed using opensource software. Git annex currently seems like the best candidate to provide much of the basic functionality, what would be required is a Drupal module to work with it.
7.2.4 Projects and code base
Projects: • a documented communications API (see below) • local server code base • central server code base • drupal module that communicates with central server
26 Chapter 7. Design documentation Media Players Documentation, Release 2.5
• future modules for other projects (Wordpress, etc...) Code base: • What language and/or framework is best for this project? • Should the local and central server code use the same language? • PHP/Symfony has some advantages: it’s simple and widely known, and we could reuse much of our existing codebase. But are there advantages to other frameworks (node.js, python/Django) that outweigh this? If we move IsumaTV to Drupal8 in the future then Symfony makes sense, but maybe we’ll move to django instead... • should we use cron-activated scripts or create a compiled daemon?
7.2.5 Communication API
The previous communication API was through XMLRPC. XMLRPC was quite a pain to deal with, but it’s RPC and generally works. The new kid in town is JSON and the REST protocol is also elegant and much simpler to use than XMLRPC.
7.3 Similar projects
This section describes various services and software alternatives to this project. There seems to be an opportunity to build a more generic content distribution system. We should, however, consider the existing alternatives and work from there or at the very least study the way they operate to avoid running into similar problems.
7.3.1 Commercial CDNs
The CDN service is currently mostly implemented by closed-source, for-profit corporations, like Akamai, Cloudflare and others. What we are doing is essentially the same problem, although their use case is simpler because they can usually get the content on the fly and don’t necessarily deal with large files and low bandwidth.
7.3.2 Debian mirrors network
There is a free software projects with similar goals and problems: Debian itself. The Debian project operates a large network of mirrors all over the world. Although a lot of the files on the mirror are small files (less than 1MiB), quite a few are much larger (Chromium is currently around 50MiB) and the total size of an archive snapshot is around 1TB, something quite close to the dataset of Isuma right now. They have some software to sync packages between the different archives, called ftpsync. It is, however, not well released or distributed (there isn’t even a debian package!), or abstracted: it is very tied to the Debian mirrors network.
7.3.3 Git annex
Holy shit, GIT-annex does a lot of what we want, in implementing a distributed file system on a variety of possible remote repositories! Integrated with gitolite providing centralized authentication/authorization, much of our required functionality would be available. Git hooks would make it pretty easy to customize and automate. GIT-annex even has a full metadata system implemented, which was a feature I had in mind for the next version of media players. Building on this seems like a good direction!
7.3. Similar projects 27 Media Players Documentation, Release 2.5
Challenges with a git-annex implementation: • metadata is stored in a separate git branch, by synthesizing commits, which requires git wizardry, and some overhead, see the details of the internals • git-annex is still in heavy development and may move under our feet, although the kickstarter phase is finishing now and the product is quite stable for basis • git-annex may not scale well with lots of files and lots of clients, see this complaint for example The git annex author, Joey Hess, may be available for consulting. In this self-run crowdfunding campaign he was proposing 300$/hr consulting fee, but that is now sold out. After discussions in person with Joey, I get the impression it may be possible to hire him for specific improvements to git-annex, but those would need to be useful for the project as a whole, and not juste for our use case. For example, he may be opened to being contracted for improving scalability, but not for storing “behind NAT” metadata that we need. Those are the things missing from git-annex at this point: • bandwidth limiting - although we could use –bwlimit in rsync fairly easily • local uploads: git-annex won’t provide you with a web interface for uploads, but will certainly take care of local files • glue to automatically mount drives and sync: git-annex doesn’t automatically mount drives but can certainly sync • cache of the HTML files - unless the website is redone in static html • a good view of sync progress in the various MPs for the CS • transcoding - if we keep doing that at all • glue for URL rewriting in the website git-annex manages metadata, and can store arbitrary per-file metadata as well. That meta-data can be exchanged through the git annex assistant, through SSH or XMPP. There had been some talks of using Telehash but this somewhat fell through as the telehash.org upstream development is still incomplete. Looking back at the previous development year Joey mentionned that he will keep an eye on the project and consider other alternatives such as MaidSafe. Those backends could be useful to help media players share data with each other and facilitate communication without talking to the central server.
7.3.4 Camlistore
Another storage system that may be interesting, similar to git-annex, is Camlistore.
28 Chapter 7. Design documentation CHAPTER 8
Original specifications
8.1 Overview
8.1.1 Requirements
1. local caching of media files being downloaded by users on LAN (gets downloaded from main site in back- ground). 2. local caching of media being uploaded by users on the LAN (gets sent to main site in background). 3. local transcoding of uploaded media, so local users can view locally uploaded media before it is uploaded to main site. 4. remote control and configuration of local servers (perhaps using drupal?), allowing: • stats • troubleshooting • control over what videos are uploaded/downloaded and in what order, • patches, upgrades, etc. • bandwidth controls 5. server would be installable from a CD 6. standardized solid hardware: maybe we should store video files on a large internal drive instead of an external, might be safer. 7. local cached version of site – slave db sync from master 8. cached content can be moved via hard drive.
8.1.2 General specs
• pluggable CDN as the model for file delivery, with URL re-writing at the server • pluggable back-end to add new file types and new file delivery methods • local server machines are as simple, as plug-and-play as possible. configuration happens on internet-available servers • communication between the components using XML-RPC • use drupal/php for all coding
29 Media Players Documentation, Release 2.5
• code to drupal standards for formatting, documentation, etc.
8.1.3 Components
1. Central Server (CS) - keeps track of all the files and their locations and is admin interface. • Drupal for authentication, XML-RPC and user-interface, minimal footprint • “Caching Server” module with custom code. • Nodes on Central Server uniquely identity every file that can be potentially cached (by a combination of filepath, type and option) and an array of delivery methods: – Local Servers, Qiniq QFile, default amazon CloudFront delivery, etc... – array (‘backend’ => ‘url’)) where backend can be ‘local_server’ or ‘amazon’ (for now) • Users as back-end clients (individual local servers) and Roles as different types of back-ends (local server, Qfile). Also roles/users for human admins. • NodeQueue to track files (as nodes) as they are synchronized with various back-ends (except amazon default, which is uploaded from Site Server and is the base for all other back-ends) and base for user interface to control sync, etc. • provide url information for files to SS for url re-writing. Determines which back-end to use, based on requesting computer IP (maybe other factors? leave open for future). 2. “Caching Client” module to be installed on Site Server (SS) – communicates with Central Server. • requests url information for files • updates CS with latest uploaded files (as MM action) • implements two-stage file uploading (for video, maybe other filetypes in the future) 3. Local Servers (LS) – simple black box, serve files locally and communicates with Central Server • transcodes locally uploaded files • creates static version of site • solid build, small boxes • large as possible internal hard drive (2TB? 4TB?) • multiple ethernet? • e-sata or other high-speed drive connection • Debian stable • Apache, PHP5 • scripts for receiving files, communicating with CS, transcoding, etc. • scripts for static cache of site • automount attached e-sata drive • easily deployable to local servers (could be CD, other)
30 Chapter 8. Original specifications Media Players Documentation, Release 2.5
8.1.4 Functional outline – local servers example
1. When a Local Server is plugged in and has internet access, contacts the Central Server with a token generated for local servers (all such contact is made using XML-RPC): cs.newcache($token, ‘local_server’) The CS creates a user account for the LS, assigns it to the LS role, and returns to the LS its auth credentials (username and password combo). The CS creates a Download node queue, an Upload node queue, and a Files node queue for the LS. The Download node queue tracks all the files to be downloaded to the LS and is filled with all existing active File nodes. The Upload node queue tracks the files being uploaded to the Site Server from the LS. The Files node queue tracks the files that can be streamed from the LS. These queues can all be manipulated by logged-in admins. 2. On a regular cron cycle (ex. 5 minutes) the LS contacts the CS and sends its auth and LAN and WAN IP: cs.location($auth, LAN_IP, WAN_IP) The CS stores (or updates) these with the LS username, a unique ID and the current timestamp in a “Local Servers” table. The CS removes any stale combos. (When a regular user contacts the Site Server, their WAN IP is used to find the LS in their LAN and URLs are rewritten to this LS.) 3. When a file of a particular type – video SD, video HD, images at various sizes, audio – is uploaded to the Site Server, a node of type “File” is created on the CS which stores the unique files table filepath from the SS and its type (eg. video) and option (eg. high or low). This node is added to the bottom of all Download node queues (unless already present in the Files queue, due to LS conversion). cs.announce($auth, $filename, $type, $option) 5. A LS contacts the CS and is given a url for the file at the top of its Download queue. cs.download($auth, ‘get’) When it has downloaded the file it contacts the CS, the node is added to its Files queue. cs.download($auth, ‘put’, $filename, $type, $option) 6. When a computer contacts the SS its WAN IP is searched for in the “Local Servers” table on the CS. cs.setBackend($auth, $WAN_ip, $session) If found, the LAN IP or IPs are returned to the browser with a JavaScript which looks for the Local Server at the IPs. If found, the CS is alerted and the LS IP and unique ID is stored in the browser’s Session. cs.setBackend($auth, $WAN_ip, $session, $local_server) It will be removed from the Session if it is stale-dated in step 2. 7. Download behaviour – when computers on LS networks request files: If requested files (video, audio, images, etc.) are present in the Files node queue for the local server, then the URL to the file is rewritten using the local IP and sent to the SS for page generation. If there is no local server or the file is not in the Files queue, then the URL may be re-written in other ways (as an RTMP stream from CloudFront, for example.) cs.getURL($auth, $filename, $type, $option, $session) 8. Upload behaviour – when computers on LS networks upload (video) files: Uploading (creating content) will be a two-stage process. A node creation form creates the content node on the SS, and then a second step is returned to the user for uploading the file. The file will be uploaded to the LS, and these steps take place: 1. LS contacts SS (or upload script) with name of file and NID (perhaps other form data, like token) and file is created: • dummy file in files directory, to avoid name collisions • entry in content field and files table • somehow Media mover harvesting is inhibited. 2. LS adds file to Upload node queue for LS cs.upload($auth, ‘put’, $filename, ‘video’, ‘high’) 3. when file is uploaded it replaces dummy file and media mover harvesting is enabled. File conversion proceeds as normal. 4. LS converts video file to hi-res formats. LS contacts CS which adds video to Files queue for the LS (will not be downloaded to this LS when converted on SS). cs.download($auth, ’put’, $filename, ’video’, ’high’) This upload process will be implemented for video but may be adapted to other content (audio, images) if necessary. 9. LS periodically crawls SS to create a static version of the site – urls to media files are the locally cached versions.
8.1. Overview 31 Media Players Documentation, Release 2.5
10. Admin accounts can be created on CS allowing admins to alter queue order and remove files from queues. Also control bandwidth allocation for upload and download. 11. Specially configured hard drives can be connected to LS through high-speed (e-sata) connections. LS will auto- matically mount hard drive and process can be monitored from admin interface. New download files can be copied to LS and will be removed from Download queue and added to Files queue. New upload files can be copied from LS to hard drive and be removed Upload queue (and added centrally when hard drive returns to main office).
8.2 Hardware possibilities
8.2.1 Servers
There are many different classes of system to choose from, depending on exactly how bad an environment these things have to withstand. Predictably, there is a direct relationship between price and reliability. Systems listed below are from least to most reliable. This is not an exhaustive list of systems or candidates, but will be altered based on feedback from the client until we know exactly what kind of system they need.
8.2.2 Consumer
The Acer Aspire Easystore H340 is a home storage server that normally comes with Windows Home Server. Linux also runs on it. If we add 2x2TB hard drives, we have a 5TB system for about $710. Pros: Cheap Simple Low power Front panel lights Cons: Requires adapter for keyboard/mouse/video No linux drivers for front panel lights M$ tax Fans may inhale too much dust Processor may not be powerful enough for transcoding Swapping out the server for a HP MediaServer would give us a much more powerful (dual-core pentium) system for $1010 total. This removes the problems with transcoding, but all other pros/cons remain the same.
8.2.3 Server
We could build a custom system for about $1000 around a case like this: http://ncix.com/products/?sku=37520&vpn=CSE-733TQ-645B&manufacture=SuperMicro Pros: Cheap Commodity parts means spares are easy to find Cons: Time-consuming to put together systems from scratch rather than buy pre-made Fans may inhale too much dust We could also custom-order a server from a big-name manufacturer: A Dell PowerEdge 110 with 4TB would cost $2,769. An HP ProLiant 310 g5 with 4TB would cost $2,953.00 Pros: No assembly required Excellent support Cons: More expensive Non-commodity parts Fans may inhale too much dust
8.2.4 Industrial
This ARK-3440 system by Advantech is fanless and extremely rugged, and so would hold up very well. No price is listed, but judging by similar systems, it probably costs around $2000. The slightly older ARK-3420-S and ARK-3420-S1 have Celeron and Core 2 Duo processors and are and are $1080 and $1550, respectively. Pros: Fanless - no danger of dust or smoke damage Robust
32 Chapter 8. Original specifications Media Players Documentation, Release 2.5
Cons: Expensive Limited storage space (2x2.5” laptop drives). This gives us a max of 2TB of HDD or 1TB of flash. Non-commodity parts Edit Milspec God knows how much these cost. Manufacturers don’t publish prices. They also don’t generally offer dust-proof systems with a lot of storage. Most sealed systems have 1 or 2 2.5” bays, like the above industrial system.
8.2.5 Interface
For machines that don’t have a front panel display, or have one that’s not accessible from Linux, we can use something like The CW1602 This is a 16x2 LCD with 6 buttons and is supported on Linux by LCDproc or lcd4linux software.
8.3 Auto-mounting Hotplugged Devices
8.3.1 udev
udev is a user-mode system that triggers on device insertion or removal, and can be used to change device names in /dev or perform other, more complicated tasks via shell scripts. Scripts are triggered by matching rules . The following rule should match any USB drive plugged into the system and run a script to automount it: KER- NEL==”sd*”, DRIVERS==”usb-storage” This rule matches any SCSI drive: KERNEL==”sd*”, DRIVERS==”sd” This matches an IDE drive: KERNEL==”hd*”, DRIVERS==”ide-disk” Annoyingly, these don’t match on remove, only on insertion. Removing the DRIVERS== clause will make them match on both. So, we use these rules to match on insertion and removal of any USB or SCSI disk:
ACTION=="remove", KERNEL=="sd*" RUN+="/home/debian/driveremoved.sh $kernel" ACTION=="add",