Sunday Things - Sunday edition

posted in Links by Cargo Cult on Sunday January 27 2013

Links! Links links links. Also, I've thought of a short-term, very silly Raspberry Pi project. Wahey!

Surprising amounts of wilderness.
Census Dotmap - US and Canadian census data, plotted to make maps - one black dot per person. I'm in there somewhere...

This is exactly how the 1950s operated.

Hello, you either have plugins turned off or an old version of Macromedia's Flash Player. Get the latest Flash player.

Cyriak: Bonobo: Cirrus - many notable film directors started off making music videos. My brain is deeply terrified of any potential feature films made by Mr. Cyriak...
There may actually be some submarine left somewhere amongst the rust.
Submarine graveyard, Kola Peninsula, Russia - this week's Soviet rust, some deliciously decayed, abandoned submarines! Good news: these ones don't look nuclear. Bad news: the actual nuclear stuff is nearby.

He likes to move it, move it.

Hello, you either have plugins turned off or an old version of Macromedia's Flash Player. Get the latest Flash player.

Maker movement starts integrating cockroach components into projects - move over Arduino, the SpikerBox is here!

Bonus If-BLDGBLOG-made-comics corner:

  • Anomalies - at some point this link will correct itself.

Sunday Things - continuing edition

posted in Links by Cargo Cult on Sunday January 20 2013

More things. On Sunday. You get the idea.

Is that bit supposed to fall off like that?
Lana Sator, post-Soviet urban explorer - she of rocket factory infiltration fame, somehow her rust-laden blog wasn't in my RSS reader. A situation now rectified! via rjp

What goes up keeps going up.

Hello, you either have plugins turned off or an old version of Macromedia's Flash Player. Get the latest Flash player.

Tracking a radiosonde (weather balloon) - listening in on various telemetry systems using hacked USB digital television adapters is definitely a thing. I need to investigate. via
Toasty-hot science. :-(
Siding Spring Observatory fires - Australian bushfires come close to burning down a major observatory. (Spoiler: the expensive bits mostly survived. Luckily!) via proogs

MOOOOooooooooooon <splat>

Hello, you either have plugins turned off or an old version of Macromedia's Flash Player. Get the latest Flash player.

Parting Moon Shots from NASA's GRAIL mission - dual-spacecraft gravitometric lunar mission with tiny educational outreach cameras attached, shooting gloriously atmospheric timelapses from orbit, shortly before colliding with the moon itself.

Bonus Calamitous Chemistry corner:

  • Things I Won't Work With - you don't need to know anything about chemistry to enjoy the wonderful descriptions... 'In every case, the solution detonated spontaneously on standing. And by "spontaneously”, they mean "while standing undisturbed in the dark", so there’s really just no way to deal with this stuff. It’s probably a good thing they didn’t get crystals, because someone would have tried to isolate the hideous things. The cesium salt actually did give a few crystals, which they managed to pluck from the top of the solution and get X-ray data on. A few hours later the remaining batch suddenly exploded, though, which certainly must have been food for thought.'

Sunday Things - oh-hello edition

posted in Links by Cargo Cult on Sunday January 13 2013

So it would appear after the Radio-4-Matic escapades earlier this week, I gained some more readers!

Hello.

To you, I shall explain the Sunday Things - essentially, every Sunday I post some Things. Which are interesting. To me, at least. To you? Let us find out.

Eddie Mair: people's hero.

Hello, you either have plugins turned off or an old version of Macromedia's Flash Player. Get the latest Flash player.

Kitchen Atom Bomb - after my own appearance on PM, here's quite possibly the greatest interview in the history of everything - with Richard Handl, a Swedish man who built a nuclear reactor in his kitchen.
Some sort of factory-thing?
Заброшенный сахарный завод - prised from the bookmarks, explorations of some sort of abandoned post-Soviet chemical factory? I could run it through an automated translator, but that's cheating!
Information leaves shadows.
The Basement - somewhere in Portland there's an old building with an old basement, with a room that holds a hyper-modern secret.

Bonus Cold War Transmissions corner:

Radio-4-Matic - part 4 - Radio 4

posted in Electronics by Cargo Cult on Saturday January 12 2013

So, it seems someone at Radio 4 spotted this thing, and one thing led to another and I ended up being interviewed on PM. Yes, me on national radio! On a proper news programme! Blinking heck.

Like a friendly eye of Sauron.

Not sure how long it'll stay up, so listen now. There's a mention at the beginning, then my bit's at the end.

It got edited down a fair bit (a mention of the Raspberry Pi itself got lost, along with a mention of non-Pi alternatives) - but it most definitely brings across the heart of why I did it. A couple of observations:

  • Talking to a Radio 4 presenter is really weird. Perfect diction.
  • Not helped by Skype being wired into the studio microphones. Incredibly clear.
  • It's like talking to a radio. And the radio's talking back. Fantastic!

Radio-4-Matic - part 3 - software

posted in Electronics by Cargo Cult on Friday January 11 2013

I promised 'Friday' for this third, code-containing article, and according to my radio it's still Friday (Today in Parliament's playing) - so here we are.

Before I even had a chance to fully document everything, this little time-travelling Roberts radio got spotted - first by the official Raspberry Pi news blog, then MediaUK, then by somebody else... Usually I'm more notable in the computer games world!

Edit 2013-01-12: that 'somebody else'? Radio 4!

Edit 2013-01-13: ... and now Hackaday!

It seems Raspberry Pi radios are definitely a thing - I'm glad to join the club.

Polyfuses

Firstly, a cheeky bait-and-switch with some entirely-not-software-related Raspberry Pi surgery, removing the pesky polyfuses which were messing with my WiFi. This procedure is to rectify a hardware problem that has long since been fixed on the Raspberry Pi - it is only necessary if you have an early Pi, wish to eradicate any remaining warranty and generally understand the implications of removing the current-limiting components from the USB subsystem. If you have any questions, don't.

So here goes. Some surgical pliers (no idea as to the correct terminology) proved ideal for grasping each polyfuse lengthwise, leaving just enough space either end to poke the tip of a soldering iron against the end of the fuse. Apply a small amount of angular force through the pliers so that when the solder joint melts, the end of the fuse pops up slightly, away from the solder pad. Otherwise, detaching both solder joints simultaneously is implausible.

Once the polyfuses are gone, trim some bits of wire to cross the joins. I used 'C' shapes of vaguely thick single-core wire, with bits of insulation left in the middles. Clamping the wire by the side of the board in place with a crocodile-clip was really helpful - the inner one was much trickier. Quickly solder each contact in place, taking care to neither cook the board nor use too much solder - as mentioned earlier, if you have any concerns regarding this procedure, please don't do it.

Polyfuse placementPolyfuse removalPolyfuse replacementWire clampingWire attachmentWireless in a wireless

But still. Both my Pis (named gasbuggy and rulison, don't ask) made a rapid recovery and are now stable with Edimax EW-7811Un WiFi adapters. Yay!

Software!

... And after that diversion, now to the heart of the Radio-4-Matic - the dodgy scripting that metaphorically glues it all together. Implementation isn't exactly wonderful (I'm not much of a programmer), but these notes should hopefully help anyone wishing to improve on the concept, or is thinking of something else entirely. I kind of suspect there are UNIX utilities that would make something like this far easier, but I learned a lot while building this thing.

Anyway.

The first bash-script prototype had get_iplayer (and thus rtmpdump) piping the live Radio 4 straight into a file, with mplayer being started on that file after a crude sleep 8h;. The FLV-wrapped AAC audio straight from rtmpdump proved an issue, with mplayer claiming an end-of-file after playing up until the size of the file as it originally saw it, even thought it had been much-extended since then. Attempts to encourage it to do otherwise were futile.

Time to pipe that audio through ffmpeg, and wrap it in something more acceptable to mplayer's demands. Yes! It worked! For a bit.

The capture side, get_iplayer and rtmpdump, had a tendency to cut out - presumably whenever a seagull over the Atlantic farted, or if a grizzly bear in the Midwest ran out of toilet paper. Something to do with the tubes being blocked? Only one way to improve this - stick the capture utilities in a loop, respawning them whenever they fell over. This would introduce gaps in the recorded audio, resulting in an ever-increasing discrepancy against 'actual' time - and rtmpdump demonstrated a new tantrum by occasionally getting stuck, but without exiting. Sigh.

Then, even when things were actually streaming correctly, ffmpeg would occasionally choke with a 'non monotonically increasing dts to muxer in stream' error. Which was a bug which was apparently long-fixed in ffmpeg. Wait a minute...

ffmpeg

It turns out that the 'ffmpeg' packaged by Debian (and thus Raspbian) is actually libav, a forked version of the original project. Which helpfully provided an ffmpeg binary for 'backwards compatibility'.

Only one thing to do - compile a version of proper, full-strength unforked ffmpeg. Free software enthusiasts are no doubt crying out in agony at the thought of me ignoring whatever technical or political issues led to that fork...

Someone's written a guide to compiling ffmpeg on a Raspberry Pi (ignore stuff before 'Build and Install FFMPEG') - the configuration I used was as follows:

ffmpeg version N-45983-ge12cfd0 Copyright (c) 2000-2012 the FFmpeg developers
built on Oct 24 2012 14:28:08 with gcc 4.6 (Debian 4.6.3-8+rpi1)
configuration: --enable-libmp3lame --enable-libtheora --enable-libx264 --enable-postproc --enable-libxvid --enable-pthreads --enable-libvorbis --enable-gpl --enable-nonfree

Edit 2013-01-12: Bob in the comments has compiled some more useful information on, erm, compiling ffmpeg.

Edit 2013-01-15: Bob's been finding issues with getting a working ffmpeg build on a Raspberry Pi, but he's managed to get one through cross-compiling on a larger machine. I shall investigate - my version got built ages ago. If anyone's afraid of this compiling lark, the actually-libav ffmpeg provided with Raspbian will work with the Radio-4-Matic, it'll just have streaming fail then restart more frequently...

So, after baking in the Raspberry Pi's hypothetical code-oven for some time, out popped a piping hot ffmpeg binary. Which rode straight over non-monotonical packets without issues, squashing one of the potential causes of death.

radio.php

Then came the first version of a hacked-together PHP script for monitoring (and potentially respawning) the various utilities. Then came the second. 'Spaghetti code' wasn't apt - no spaghetti would be quite so convoluted and branched. This thing was awful. The non-blocking IO was ... interesting to get my head around. (The central problem is with rtmpdump getting stuck but never exiting - there needs to be some intermediary checking that it's still returning data. And that intermediary also needs to check if any other subsection has fallen over, or if it's time to start mplayer on its next time-delayed buffer...)

Next came de-spaghettification. A couple of cheesy helper functions for dealing with processes were born, all with pitifully little error-checking apart from where it mattered most. Recording and playback were separated into, um, separate functions, each as a simple switch() statement executing a single segment each time the loop was traversed, switching to other segments as necessary. It's almost readable now. But still in PHP. Mwuhahahaha! A super-modular, object-oriented, multiple-streams-possible Python rewrite is pending.

Licensing: it's the good old MIT licence. In terms of open source, it's very open. More so than Linux.

Copyright (C) 2012-2013 Adam Foster - afoster@hylobatidae.org

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Download here:

One day, I may even set up a github account...

Requirements
  • PHP - needs a PHP 5 command-line interpreter. Required package in Raspbian is php5-cli. I don't recall any extra sub-dependencies...
  • get_iplayer - the Radio-4-Matic is running version 2.82, but that or anything more recent should do. Someone's already written a decent guide to getting it running on a Raspberry Pi.
  • rtmpdump - available in Raspbian's repositories. Imaginatively called rtmpdump. See installation instructions for get_iplayer above.
  • ffmpeg - see pre-baked requirements above. It'll work with the one from Raspbian, but it'll break. Monotonically.
  • mplayer - installed by default? If not, install it!
Running

I've yet to add any particular automation to the current workings - I'm assuming you have SSH access enabled on your Pi, have working sound output and know some UNIX basics.

Radio-4-Matic, in action!

Edit the radio.php to point at the appropriate binaries (I've put all the necessary parameters in, but you may want to tweak them or change the paths depending on where you have them installed), set your playback timezone and adjust the recording time ranges to your liking. Start up a screen session and start Radio-4-Matic with php radio.php. Ctrl-A, D to detach, screen -r to reattach to the screen session - this way you can leave the system running without needing an SSH session open. Depending on where you live, wait something like eight hours then and BBC Radio 4 should start playing - adjusted to your location, of course. Peer at what one particular subsection is doing with a tail -f <logfilename>.log.

It'll probably run on a bigger desktop machine running Linux, or even Mac OS X or other UNIX systems, without modification - there's nothing Raspberry Pi-specific in there. Yet!

(Next tasks: have it running automatically on startup, check if WiFi connection is up, potentially twiddle said WiFi or reboot the Pi in an emergency, reuse any existing audio buffers, ignore You and Yours, integrate with Teasmade for the morning cuppa, and so on and so forth...)

Timing during playback isn't perfect, but for me it typically runs within about 20 seconds of 'proper', adjusted time - the pips are a particularly unfair way of judging its timeliness. I'll look into improving this, but for now it more than maintains the illusion. There's a fudge-factor in the code to adjust for streaming delay, but there's almost certainly a better way of doing this. The Pi's audio clock probably isn't a perfect 44.1kHz so there could be drift from that, too...

If streaming has failed, then on playback you'll get a (correctly timed) gap in the audio until recording resumed. A future feature could be some authentic radio noise until the 'signal' is regained.

Have fun - and if you happen to construct your own Radio-4-Matic, do let me know!

Roberts R707, master of the intertubes.
Command explanation

There's lots of arcane magic in the way the various utilities are being invoked, so here's a rough explanation. There are surely many unnecessary parameters in here - stuff that persisted through different iterations - but this is how I've got Radio-4-Matic running fairly reliably.

get_iplayer

The please-use-only-for-good-not-evil get_iplayer is a monster of a Perl script, which parses stuff from BBC iPlayer to create parameters for starting a Flash-streaming program such as rtmpdump or flvstreamer. The words 'dreadful' and 'hack' come to mind, but it can be incredibly useful for tasks such as this.

get_iplayer --liveradiomode flashaaclow --pid radio:bbc_radio_fourfm --stream --rtmp-liveradio-opts --live

Explanation: start get_iplayer, in live radio mode streaming low-bandwidth AAC audio over flash, programme ID 'radio:bbc_radio_fourfm', streaming to stdout, reassure rtmpdump that no-really-this-is-live.

ffmpeg

Audio from the above gets piped through ffmpeg, a Swiss-army-chainsaw audio/video conversion tool. We don't want to change the underlying AAC-compressed audio, instead just repackage it from its mplayer-confusing FLV container into a much-more-standard ADTS container.

ffmpeg -f flv -i pipe:0 -acodec copy -vn -f adts -strict -2 -

Explanation: start ffmpeg, input format FLV, input from pipe 0 (stdin), disable video recording, copy compressed audio without transcoding or recompressing, output format ADTS, standards compliance level -2, send to stdout.

mplayer

At the required moment, the script starts mplayer on one of the recorded audio files and lets it play until it reaches the end.

mplayer -novideo -nocache -ao alsa

Explanation: start mplayer, disable video output (simply to suppress a warning message, disable file cache (forget any idea of the file being of fixed length, it could still be increasing in size), audio output device 'alsa' (suppresses another warning message).

SSH tunnelling

I'm not going to go into much amount of detail on this, given that it's (a) optional, (b) much more involved to set up given the remote server requirements and (c) it's more than a bit dodgy. But anyway. It's possible, and the script above supports it. I found that streaming became a fair bit more reliable, but still not perfect. You need tsocks and an SSH tunnel. So there.