an altogether higher class of gibbonindex
Radio-4-Matic - part 3 - software
posted in Electronics by Cargo Cult on Friday January 11 2013
I promised 'Friday' for this third, code-containing article, and according to my radio it's still Friday (Today in Parliament's playing) - so here we are.
Before I even had a chance to fully document everything, this little time-travelling Roberts radio got spotted - first by the official Raspberry Pi news blog, then MediaUK, then by somebody else... Usually I'm more notable in the computer games world!
Edit 2013-01-12: that 'somebody else'? Radio 4!
Edit 2013-01-13: ... and now Hackaday!
Firstly, a cheeky bait-and-switch with some entirely-not-software-related Raspberry Pi surgery, removing the pesky polyfuses which were messing with my WiFi. This procedure is to rectify a hardware problem that has long since been fixed on the Raspberry Pi - it is only necessary if you have an early Pi, wish to eradicate any remaining warranty and generally understand the implications of removing the current-limiting components from the USB subsystem. If you have any questions, don't.
So here goes. Some surgical pliers (no idea as to the correct terminology) proved ideal for grasping each polyfuse lengthwise, leaving just enough space either end to poke the tip of a soldering iron against the end of the fuse. Apply a small amount of angular force through the pliers so that when the solder joint melts, the end of the fuse pops up slightly, away from the solder pad. Otherwise, detaching both solder joints simultaneously is implausible.
Once the polyfuses are gone, trim some bits of wire to cross the joins. I used 'C' shapes of vaguely thick single-core wire, with bits of insulation left in the middles. Clamping the wire by the side of the board in place with a crocodile-clip was really helpful - the inner one was much trickier. Quickly solder each contact in place, taking care to neither cook the board nor use too much solder - as mentioned earlier, if you have any concerns regarding this procedure, please don't do it.
But still. Both my Pis (named gasbuggy and rulison, don't ask) made a rapid recovery and are now stable with Edimax EW-7811Un WiFi adapters. Yay!
... And after that diversion, now to the heart of the Radio-4-Matic - the dodgy scripting that metaphorically glues it all together. Implementation isn't exactly wonderful (I'm not much of a programmer), but these notes should hopefully help anyone wishing to improve on the concept, or is thinking of something else entirely. I kind of suspect there are UNIX utilities that would make something like this far easier, but I learned a lot while building this thing.
The first bash-script prototype had get_iplayer (and thus rtmpdump) piping the live Radio 4 straight into a file, with mplayer being started on that file after a crude sleep 8h;. The FLV-wrapped AAC audio straight from rtmpdump proved an issue, with mplayer claiming an end-of-file after playing up until the size of the file as it originally saw it, even thought it had been much-extended since then. Attempts to encourage it to do otherwise were futile.
Time to pipe that audio through ffmpeg, and wrap it in something more acceptable to mplayer's demands. Yes! It worked! For a bit.
The capture side, get_iplayer and rtmpdump, had a tendency to cut out - presumably whenever a seagull over the Atlantic farted, or if a grizzly bear in the Midwest ran out of toilet paper. Something to do with the tubes being blocked? Only one way to improve this - stick the capture utilities in a loop, respawning them whenever they fell over. This would introduce gaps in the recorded audio, resulting in an ever-increasing discrepancy against 'actual' time - and rtmpdump demonstrated a new tantrum by occasionally getting stuck, but without exiting. Sigh.
Then, even when things were actually streaming correctly, ffmpeg would occasionally choke with a 'non monotonically increasing dts to muxer in stream' error. Which was a bug which was apparently long-fixed in ffmpeg. Wait a minute...
It turns out that the 'ffmpeg' packaged by Debian (and thus Raspbian) is actually libav, a forked version of the original project. Which helpfully provided an ffmpeg binary for 'backwards compatibility'.
Only one thing to do - compile a version of proper, full-strength unforked ffmpeg. Free software enthusiasts are no doubt crying out in agony at the thought of me ignoring whatever technical or political issues led to that fork...
Someone's written a guide to compiling ffmpeg on a Raspberry Pi (ignore stuff before 'Build and Install FFMPEG') - the configuration I used was as follows:
ffmpeg version N-45983-ge12cfd0 Copyright (c) 2000-2012 the FFmpeg developers
built on Oct 24 2012 14:28:08 with gcc 4.6 (Debian 4.6.3-8+rpi1)
configuration: --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-postproc --enable-libxvid --enable-pthreads --enable-libvorbis --enable-gpl --enable-nonfree
Edit 2013-01-12: Bob in the comments has compiled some more useful information on, erm, compiling ffmpeg.
Edit 2013-01-15: Bob's been finding issues with getting a working ffmpeg build on a Raspberry Pi, but he's managed to get one through cross-compiling on a larger machine. I shall investigate - my version got built ages ago. If anyone's afraid of this compiling lark, the actually-libav ffmpeg provided with Raspbian will work with the Radio-4-Matic, it'll just have streaming fail then restart more frequently...
So, after baking in the Raspberry Pi's hypothetical code-oven for some time, out popped a piping hot ffmpeg binary. Which rode straight over non-monotonical packets without issues, squashing one of the potential causes of death.
Then came the first version of a hacked-together PHP script for monitoring (and potentially respawning) the various utilities. Then came the second. 'Spaghetti code' wasn't apt - no spaghetti would be quite so convoluted and branched. This thing was awful. The non-blocking IO was ... interesting to get my head around. (The central problem is with rtmpdump getting stuck but never exiting - there needs to be some intermediary checking that it's still returning data. And that intermediary also needs to check if any other subsection has fallen over, or if it's time to start mplayer on its next time-delayed buffer...)
Next came de-spaghettification. A couple of cheesy helper functions for dealing with processes were born, all with pitifully little error-checking apart from where it mattered most. Recording and playback were separated into, um, separate functions, each as a simple switch() statement executing a single segment each time the loop was traversed, switching to other segments as necessary. It's almost readable now. But still in PHP. Mwuhahahaha! A super-modular, object-oriented, multiple-streams-possible Python rewrite is pending.
Licensing: it's the good old MIT licence. In terms of open source, it's very open. More so than Linux.
Copyright (C) 2012-2013 Adam Foster - email@example.com
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
- radio-4-matic-0.3.zip (5KiB ZIP)
One day, I may even set up a github account...
- PHP - needs a PHP 5 command-line interpreter. Required package in Raspbian is php5-cli. I don't recall any extra sub-dependencies...
- get_iplayer - the Radio-4-Matic is running version 2.82, but that or anything more recent should do. Someone's already written a decent guide to getting it running on a Raspberry Pi.
- rtmpdump - available in Raspbian's repositories. Imaginatively called rtmpdump. See installation instructions for get_iplayer above.
- ffmpeg - see pre-baked requirements above. It'll work with the one from Raspbian, but it'll break. Monotonically.
- mplayer - installed by default? If not, install it!
I've yet to add any particular automation to the current workings - I'm assuming you have SSH access enabled on your Pi, have working sound output and know some UNIX basics.
Edit the radio.php to point at the appropriate binaries (I've put all the necessary parameters in, but you may want to tweak them or change the paths depending on where you have them installed), set your playback timezone and adjust the recording time ranges to your liking. Start up a screen session and start Radio-4-Matic with php radio.php. Ctrl-A, D to detach, screen -r to reattach to the screen session - this way you can leave the system running without needing an SSH session open. Depending on where you live, wait something like eight hours then and BBC Radio 4 should start playing - adjusted to your location, of course. Peer at what one particular subsection is doing with a tail -f <logfilename>.log.
It'll probably run on a bigger desktop machine running Linux, or even Mac OS X or other UNIX systems, without modification - there's nothing Raspberry Pi-specific in there. Yet!
(Next tasks: have it running automatically on startup, check if WiFi connection is up, potentially twiddle said WiFi or reboot the Pi in an emergency, reuse any existing audio buffers, ignore You and Yours, integrate with Teasmade for the morning cuppa, and so on and so forth...)
Timing during playback isn't perfect, but for me it typically runs within about 20 seconds of 'proper', adjusted time - the pips are a particularly unfair way of judging its timeliness. I'll look into improving this, but for now it more than maintains the illusion. There's a fudge-factor in the code to adjust for streaming delay, but there's almost certainly a better way of doing this. The Pi's audio clock probably isn't a perfect 44.1kHz so there could be drift from that, too...
If streaming has failed, then on playback you'll get a (correctly timed) gap in the audio until recording resumed. A future feature could be some authentic radio noise until the 'signal' is regained.
Have fun - and if you happen to construct your own Radio-4-Matic, do let me know!
There's lots of arcane magic in the way the various utilities are being invoked, so here's a rough explanation. There are surely many unnecessary parameters in here - stuff that persisted through different iterations - but this is how I've got Radio-4-Matic running fairly reliably.
The please-use-only-for-good-not-evil get_iplayer is a monster of a Perl script, which parses stuff from BBC iPlayer to create parameters for starting a Flash-streaming program such as rtmpdump or flvstreamer. The words 'dreadful' and 'hack' come to mind, but it can be incredibly useful for tasks such as this.
get_iplayer --liveradiomode flashaaclow --pid radio:bbc_radio_fourfm --stream --rtmp-liveradio-opts --live
Explanation: start get_iplayer, in live radio mode streaming low-bandwidth AAC audio over flash, programme ID 'radio:bbc_radio_fourfm', streaming to stdout, reassure rtmpdump that no-really-this-is-live.
Audio from the above gets piped through ffmpeg, a Swiss-army-chainsaw audio/video conversion tool. We don't want to change the underlying AAC-compressed audio, instead just repackage it from its mplayer-confusing FLV container into a much-more-standard ADTS container.
ffmpeg -f flv -i pipe:0 -acodec copy -vn -f adts -strict -2 -
Explanation: start ffmpeg, input format FLV, input from pipe 0 (stdin), disable video recording, copy compressed audio without transcoding or recompressing, output format ADTS, standards compliance level -2, send to stdout.
At the required moment, the script starts mplayer on one of the recorded audio files and lets it play until it reaches the end.
mplayer -novideo -nocache -ao alsa
Explanation: start mplayer, disable video output (simply to suppress a warning message, disable file cache (forget any idea of the file being of fixed length, it could still be increasing in size), audio output device 'alsa' (suppresses another warning message).
I'm not going to go into much amount of detail on this, given that it's (a) optional, (b) much more involved to set up given the remote server requirements and (c) it's more than a bit dodgy. But anyway. It's possible, and the script above supports it. I found that streaming became a fair bit more reliable, but still not perfect. You need tsocks and an SSH tunnel. So there.
Article is now closed for commenting.