Radio-4-Matic - part 3 - software

posted in Electronics by Cargo Cult on Friday January 11 2013

I promised 'Friday' for this third, code-containing article, and according to my radio it's still Friday (Today in Parliament's playing) - so here we are.

Before I even had a chance to fully document everything, this little time-travelling Roberts radio got spotted - first by the official Raspberry Pi news blog, then MediaUK, then by somebody else... Usually I'm more notable in the computer games world!

Edit 2013-01-12: that 'somebody else'? Radio 4!

Edit 2013-01-13: ... and now Hackaday!

It seems Raspberry Pi radios are definitely a thing - I'm glad to join the club.

Polyfuses

Firstly, a cheeky bait-and-switch with some entirely-not-software-related Raspberry Pi surgery, removing the pesky polyfuses which were messing with my WiFi. This procedure is to rectify a hardware problem that has long since been fixed on the Raspberry Pi - it is only necessary if you have an early Pi, wish to eradicate any remaining warranty and generally understand the implications of removing the current-limiting components from the USB subsystem. If you have any questions, don't.

So here goes. Some surgical pliers (no idea as to the correct terminology) proved ideal for grasping each polyfuse lengthwise, leaving just enough space either end to poke the tip of a soldering iron against the end of the fuse. Apply a small amount of angular force through the pliers so that when the solder joint melts, the end of the fuse pops up slightly, away from the solder pad. Otherwise, detaching both solder joints simultaneously is implausible.

Once the polyfuses are gone, trim some bits of wire to cross the joins. I used 'C' shapes of vaguely thick single-core wire, with bits of insulation left in the middles. Clamping the wire by the side of the board in place with a crocodile-clip was really helpful - the inner one was much trickier. Quickly solder each contact in place, taking care to neither cook the board nor use too much solder - as mentioned earlier, if you have any concerns regarding this procedure, please don't do it.

Polyfuse placementPolyfuse removalPolyfuse replacementWire clampingWire attachmentWireless in a wireless

But still. Both my Pis (named gasbuggy and rulison, don't ask) made a rapid recovery and are now stable with Edimax EW-7811Un WiFi adapters. Yay!

Software!

... And after that diversion, now to the heart of the Radio-4-Matic - the dodgy scripting that metaphorically glues it all together. Implementation isn't exactly wonderful (I'm not much of a programmer), but these notes should hopefully help anyone wishing to improve on the concept, or is thinking of something else entirely. I kind of suspect there are UNIX utilities that would make something like this far easier, but I learned a lot while building this thing.

Anyway.

The first bash-script prototype had get_iplayer (and thus rtmpdump) piping the live Radio 4 straight into a file, with mplayer being started on that file after a crude sleep 8h;. The FLV-wrapped AAC audio straight from rtmpdump proved an issue, with mplayer claiming an end-of-file after playing up until the size of the file as it originally saw it, even thought it had been much-extended since then. Attempts to encourage it to do otherwise were futile.

Time to pipe that audio through ffmpeg, and wrap it in something more acceptable to mplayer's demands. Yes! It worked! For a bit.

The capture side, get_iplayer and rtmpdump, had a tendency to cut out - presumably whenever a seagull over the Atlantic farted, or if a grizzly bear in the Midwest ran out of toilet paper. Something to do with the tubes being blocked? Only one way to improve this - stick the capture utilities in a loop, respawning them whenever they fell over. This would introduce gaps in the recorded audio, resulting in an ever-increasing discrepancy against 'actual' time - and rtmpdump demonstrated a new tantrum by occasionally getting stuck, but without exiting. Sigh.

Then, even when things were actually streaming correctly, ffmpeg would occasionally choke with a 'non monotonically increasing dts to muxer in stream' error. Which was a bug which was apparently long-fixed in ffmpeg. Wait a minute...

ffmpeg

It turns out that the 'ffmpeg' packaged by Debian (and thus Raspbian) is actually libav, a forked version of the original project. Which helpfully provided an ffmpeg binary for 'backwards compatibility'.

Only one thing to do - compile a version of proper, full-strength unforked ffmpeg. Free software enthusiasts are no doubt crying out in agony at the thought of me ignoring whatever technical or political issues led to that fork...

Someone's written a guide to compiling ffmpeg on a Raspberry Pi (ignore stuff before 'Build and Install FFMPEG') - the configuration I used was as follows:

ffmpeg version N-45983-ge12cfd0 Copyright (c) 2000-2012 the FFmpeg developers
built on Oct 24 2012 14:28:08 with gcc 4.6 (Debian 4.6.3-8+rpi1)
configuration: --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-postproc --enable-libxvid --enable-pthreads --enable-libvorbis --enable-gpl --enable-nonfree

Edit 2013-01-12: Bob in the comments has compiled some more useful information on, erm, compiling ffmpeg.

Edit 2013-01-15: Bob's been finding issues with getting a working ffmpeg build on a Raspberry Pi, but he's managed to get one through cross-compiling on a larger machine. I shall investigate - my version got built ages ago. If anyone's afraid of this compiling lark, the actually-libav ffmpeg provided with Raspbian will work with the Radio-4-Matic, it'll just have streaming fail then restart more frequently...

So, after baking in the Raspberry Pi's hypothetical code-oven for some time, out popped a piping hot ffmpeg binary. Which rode straight over non-monotonical packets without issues, squashing one of the potential causes of death.

radio.php

Then came the first version of a hacked-together PHP script for monitoring (and potentially respawning) the various utilities. Then came the second. 'Spaghetti code' wasn't apt - no spaghetti would be quite so convoluted and branched. This thing was awful. The non-blocking IO was ... interesting to get my head around. (The central problem is with rtmpdump getting stuck but never exiting - there needs to be some intermediary checking that it's still returning data. And that intermediary also needs to check if any other subsection has fallen over, or if it's time to start mplayer on its next time-delayed buffer...)

Next came de-spaghettification. A couple of cheesy helper functions for dealing with processes were born, all with pitifully little error-checking apart from where it mattered most. Recording and playback were separated into, um, separate functions, each as a simple switch() statement executing a single segment each time the loop was traversed, switching to other segments as necessary. It's almost readable now. But still in PHP. Mwuhahahaha! A super-modular, object-oriented, multiple-streams-possible Python rewrite is pending.

Licensing: it's the good old MIT licence. In terms of open source, it's very open. More so than Linux.

Copyright (C) 2012-2013 Adam Foster - afoster@hylobatidae.org

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Download here:

One day, I may even set up a github account...

Requirements
  • PHP - needs a PHP 5 command-line interpreter. Required package in Raspbian is php5-cli. I don't recall any extra sub-dependencies...
  • get_iplayer - the Radio-4-Matic is running version 2.82, but that or anything more recent should do. Someone's already written a decent guide to getting it running on a Raspberry Pi.
  • rtmpdump - available in Raspbian's repositories. Imaginatively called rtmpdump. See installation instructions for get_iplayer above.
  • ffmpeg - see pre-baked requirements above. It'll work with the one from Raspbian, but it'll break. Monotonically.
  • mplayer - installed by default? If not, install it!
Running

I've yet to add any particular automation to the current workings - I'm assuming you have SSH access enabled on your Pi, have working sound output and know some UNIX basics.

Radio-4-Matic, in action!

Edit the radio.php to point at the appropriate binaries (I've put all the necessary parameters in, but you may want to tweak them or change the paths depending on where you have them installed), set your playback timezone and adjust the recording time ranges to your liking. Start up a screen session and start Radio-4-Matic with php radio.php. Ctrl-A, D to detach, screen -r to reattach to the screen session - this way you can leave the system running without needing an SSH session open. Depending on where you live, wait something like eight hours then and BBC Radio 4 should start playing - adjusted to your location, of course. Peer at what one particular subsection is doing with a tail -f <logfilename>.log.

It'll probably run on a bigger desktop machine running Linux, or even Mac OS X or other UNIX systems, without modification - there's nothing Raspberry Pi-specific in there. Yet!

(Next tasks: have it running automatically on startup, check if WiFi connection is up, potentially twiddle said WiFi or reboot the Pi in an emergency, reuse any existing audio buffers, ignore You and Yours, integrate with Teasmade for the morning cuppa, and so on and so forth...)

Timing during playback isn't perfect, but for me it typically runs within about 20 seconds of 'proper', adjusted time - the pips are a particularly unfair way of judging its timeliness. I'll look into improving this, but for now it more than maintains the illusion. There's a fudge-factor in the code to adjust for streaming delay, but there's almost certainly a better way of doing this. The Pi's audio clock probably isn't a perfect 44.1kHz so there could be drift from that, too...

If streaming has failed, then on playback you'll get a (correctly timed) gap in the audio until recording resumed. A future feature could be some authentic radio noise until the 'signal' is regained.

Have fun - and if you happen to construct your own Radio-4-Matic, do let me know!

Roberts R707, master of the intertubes.
Command explanation

There's lots of arcane magic in the way the various utilities are being invoked, so here's a rough explanation. There are surely many unnecessary parameters in here - stuff that persisted through different iterations - but this is how I've got Radio-4-Matic running fairly reliably.

get_iplayer

The please-use-only-for-good-not-evil get_iplayer is a monster of a Perl script, which parses stuff from BBC iPlayer to create parameters for starting a Flash-streaming program such as rtmpdump or flvstreamer. The words 'dreadful' and 'hack' come to mind, but it can be incredibly useful for tasks such as this.

get_iplayer --liveradiomode flashaaclow --pid radio:bbc_radio_fourfm --stream --rtmp-liveradio-opts --live

Explanation: start get_iplayer, in live radio mode streaming low-bandwidth AAC audio over flash, programme ID 'radio:bbc_radio_fourfm', streaming to stdout, reassure rtmpdump that no-really-this-is-live.

ffmpeg

Audio from the above gets piped through ffmpeg, a Swiss-army-chainsaw audio/video conversion tool. We don't want to change the underlying AAC-compressed audio, instead just repackage it from its mplayer-confusing FLV container into a much-more-standard ADTS container.

ffmpeg -f flv -i pipe:0 -acodec copy -vn -f adts -strict -2 -

Explanation: start ffmpeg, input format FLV, input from pipe 0 (stdin), disable video recording, copy compressed audio without transcoding or recompressing, output format ADTS, standards compliance level -2, send to stdout.

mplayer

At the required moment, the script starts mplayer on one of the recorded audio files and lets it play until it reaches the end.

mplayer -novideo -nocache -ao alsa

Explanation: start mplayer, disable video output (simply to suppress a warning message, disable file cache (forget any idea of the file being of fixed length, it could still be increasing in size), audio output device 'alsa' (suppresses another warning message).

SSH tunnelling

I'm not going to go into much amount of detail on this, given that it's (a) optional, (b) much more involved to set up given the remote server requirements and (c) it's more than a bit dodgy. But anyway. It's possible, and the script above supports it. I found that streaming became a fair bit more reliable, but still not perfect. You need tsocks and an SSH tunnel. So there.

Article comments (now closed)

Bob's gravatar

1. Thank You For The Tutorial!

Posted by Bob at 8:15AM, Saturday January 12 2013

Thank you for this excellent tutorial. This may come in very useful in helping me entice my family to come visit me in Maryland. They might stay longer if they can hear Radio 4? Even grandchildren?

Also, the entire series helps me better see how to interface hardware ad software. This is an extremely good learning exercise for me.

Bob's gravatar

2. Installing ffmpeg dependencies

Posted by Bob at 2:58PM, Saturday January 12 2013

I tried the RasPi.tv instructions for compiling ffmpeg. There are some required dependencies that have to be installed for this. If you are using Raspbian Wheezy, getting the dependencies installed involves listing http://www.deb-multimedia.org/ in /etc/apt/sources.list -- you can find the details on their website. Then you must manually download and install the www.deb-multimedia.org keyring. Then you need to do an 'apt-get update.' None of these steps are obvious to new Debian users like me; I figured things out with help from RasPi.tv and Google.

The screen program is very convenient for detatching from the terminal window without destroying the compile process running in it. I do nearly all my work with the Pi over an ssh login, so running 'screen bash' let me detach from the terminal that was running the compile, disconnect my ssh session, and do other things for a few hours.

My attempt at the ffmpeg compile did indeed work. The only unusual things RasPi.tv advised was doing 'sudo configure' and 'sudo make'. I don't think that is necessary at all. The only time you need to do anything as root is when you want to run 'make install'.

Cargo Cult's gravatar

3. Re: Installing ffmpeg dependencies

Posted by Cargo Cult at 7:18PM, Saturday January 12 2013

Nice one! You might notice from the version information that I compiled my own ffmpeg quite some time ago, and I'd long forgotten the precise procedure I'd used...

I've added a link in the article. Thanks!

Bob's gravatar

4. Update: Rpi Built ffmpeg binary breaks

Posted by Bob at 9:48PM, Saturday January 12 2013

I compiled my ffmpeg from source directly on my Raspberry, and the compile seemed to complete successfully. (It took about 4 hours.) However when I ran `ffmpeg -v' to see if it will print a nice version string the way it did for you, I got a dismal 'illegal instruction' response. Clearly, something was broken. I checked the ffmpeg.org website and they suggest doing a 'make distclean' in the build directory prior to compiling. This seemed too simple, but I tried it. 4 hours later, the second (apparently successful) compile of ffmpeg also issued an 'illegal instruction' message when I ran 'ffmpeg -version'.

I am guessing this might be because I set up my /etc/apt/sources.list incorrectly for www.deb-multimedia.org. I was pointing at two different repositories there and I could be in error. I did some more searching about and stumbled onto this web page: http://ffmpeg.org/trac/ffmpeg/wiki/How%20to%20compile%20FFmpeg%20for%20Raspberry%20Pi%20%28Raspbian%29

Since I did have a faster machine that I can cross-compile on, I decided to give this a try. It took a while to set up everything but I was very encouraged when I was able to cross-compile ffmpeg successfully. I scp'd my install directory, which I called ~/rpi_bin, to the Raspberry, then ssh'd to the Raspberry, changed to the ~/rpi_bin directory, and then tried './ffmpeg -v'. This gave an error stating it could not load the libaacplus.so.2 library. I quickly added a *.conf file pointing to ~/rpi_bin/lib to /etc//ld.so.conf.d, ran `ldconfig', and tried `ffmpeg -v'. Bingo! Out spits a beautiful version string!

I made many mistakes and did a lot of try, try, try again here. It took long hours to get ffmpeg to output a simple version string. That is the developer's story. Because the Raspberry Pi has very limited resources on board, it is probably well worth it to cross-compile for the Raspberry on another machine if you have a really ambitious project.

billatq's gravatar

5. Funny that other people have done this as well

Posted by billatq at 10:19AM, Sunday January 13 2013

I did essentially the same thing with my own stream for BBC Radio 1, though I had a perl script that took the output from get_iplayer and put it in a set of 8 hour-long buffers to do the offset from Seattle. A different script then would stream the feed over to icecast. The reason for doing it that way was to force it to resynchronize on the hour boundary.

I'd be happy to share my scripts if there's any interest.

Cargo Cult's gravatar

6. Re: Funny that other people have done this as well

Posted by Cargo Cult at 10:50AM, Sunday January 13 2013

billatq: I'm really rather glad that other people have thought of the delay thing - there's a web-version of the concept here, for example:

http://www.delayplayer.com/

The icecast thing sounds pretty cool. Do share!

Apparently there's something on the Playstation 3 that'll do the same, but with television. Some friends here in Seattle watched the Russian New Year for the Seattle New Year that way - apparently it was quite surreal.

Cargo Cult's gravatar

7. Re: Update: Rpi Built ffmpeg binary breaks

Posted by Cargo Cult at 12:19AM, Tuesday January 15 2013

Bob: Hyper-useful information!

I'll see if there's a particular revision of ffmpeg which compiles without issues on the Pi - I don't remember going through nearly as much pain as you have. Thanks for doing some sterling guinea-pig work - I'll update the article again...

DocSavage's gravatar

8. Time Travel For Texas Radio

Posted by DocSavage at 6:01PM, Tuesday February 5 2013

I am moving to Zurich for work from the Gods Country (not Portland OR - Austin Texas). You have hit the nail on the head - if your shows arent on at the right time it almost makes the earth wobble on its axis.

Did you ever get the info on the icecast solution? If you did could you share?

I have a number of pi's laying around and this is perfect!!

Article is now closed for commenting.