Betacantrips/ Bits

Little things go here. There's also an archive.


For work lately I've been studying Amazon Lambda, a service they provide under the AWS umbrella. The basic theory is that you can use it to serve requests, but not pay anything for the compute time that it isn't serving anyone. It also ties nicely in with Step Functions, another Amazon product that lets you describe workflows and have your data obey them.

I was tasked with building a couple Lambda functions, and also with exploring the Apex tool, meant for administering and developing Lambda functions. My manager had already set up a repository with a "hello world" example written in JS. Because JS was the first language that Lambda supported, that seemed like a good target to continue with. But I quickly got frustrated with JS because most libraries, including the AWS library and the libraries I could find for parsing Zip files, were callback based, even though in 2017 most new code is written with async/await or at the very least Promises. I messed around with promisifying those libraries but it seemed too fiddly and I started to resent the developer experience I found myself in. I noticed that Apex supported Rust and I thought I would try that instead.

Suffice it to say that I think there are still quite a few rough edges in using Rust on Apex. In particular, lots of stuff is underdocumented. I finally got a proof-of-concept working and I thought I would write down some of the things I had to struggle with to try to help the next person doing this.

  • In order to build a Rust application that runs on Lambda, you have to target musl, which is an alternate libc. I'm not sure if this is because the Lambda machines only have musl installed and you have to match it, or if it's just that libc isn't installed and the Rust executable has to be statically linked. Either way, this was pretty easy with rustup: just do rustup target add x86_64-unknown-linux-musl.

  • AWS doesn't natively support Rust, but Apex has a "shim" that lets you execute Rust through a NodeJS parent process, which is supported. Unfortunately, AWS recently disabled support for NodeJS 0.10, and the version of Apex I installed hadn't been updated yet. The fix was on master but it wasn't released yet. I spent some time struggling to figure out how to use Go to build and run a local clone of a thing, but eventually I stumbled on using go get [Apex Github repo URL].

  • Although the Apex Rust example doesn't include serde as an explicit dependency, you need it to be available if you are using serde_derive. Otherwise you will get the error can't find crate for _serde.

  • I needed access to AWS services, so I tried to use the rusoto library. This requires the Rust OpenSSL library, which needs headers from OpenSSL, which needs to be build against Musl. I got into a lot of mischief here:

    • Of course, Musl isn't packaged for Fedora, so you have to download it and build from source. Nothing too bad here, it seemed like I had all the headers I needed. I used a --prefix of /home/ethan/.local/musl. (Note that the configure script doesn't recognize ~.) I didn't want it to go into ~/.local because it will copy include files and libraries over whatever else was already there. Then, per the musl wiki, you can compile other projects (like OpenSSL!) using musl-gcc. ~/.local/musl/bin wasn't on my PATH so I did CC="/home/ethan/.local/musl/bin/musl-gcc".

    • The OpenSSL "target" that I used was linux-x86_64.

    • OpenSSL wouldn't build without linux, asm, and asm-generic in include, and Musl doesn't provide them. I ended up just symlinking in the system ones and that seemed to work OK.

    • OpenSSL 1.1 series seemed to rely on setcontext, getcontext, and makecontext, which Musl doesn't provide (see this thread for more information). There might be a configure flag you can pass to make it not require those functions, but I didn't want to spend any time on it. I was able to get the LTS release of 1.0 to build successfully, and for this exercise that was enough (but obviously this would be something worth fixing in a real project).

    • Of course, you need to provide a --prefix to OpenSSL to get it to find the Musl installation.

    • I think I had the wrong compiler flags -- I tried both -static (because that's what it said in the wiki and I didn't know any better), but that gave me a linker error that said I should recompile with -fPIC. But then I did that too and the final product wouldn't run on Lambda (but see below). Eventually I just got rid of both and it seems to have worked out fine?

    • The Rust OpenSSL library will invoke pkg-config to get the compiler flags it needs, and pkg-config will refuse to cross-compile unless you set an PKG_CONFIG_ALLOW_CROSS environment variable. I don't really know why this exists, since I just set it even though I was clearly out of my depth.

    • If you aren't careful when you build the Rust OpenSSL libraries, they can link against the system OpenSSL library instead of the Musl-based one you wanted. I think because the system library is linked dynamically against libc, the final Rust executable you get will be dynamically linked, and won't run on Lambda. (Normally, the executable I got from apex deploy was statically linked -- again, I'm not sure if this is a strict requirement or just a default or what.) Then, when you deploy your artifact, it won't start up and the logs won't tell you why. (I'm guessing the dynamic linker is failing to find something but the exec() call is returning a success to the NodeJS shim.)

    • In order to make sure the Rust OpenSSL library picks up your changes to environment, you can cargo clean (i.e. blow away the target directory) before doing a rebuild. I had been trying to rm target/foo/deps/openssl* and I think that didn't pick up the changes.

    • The OpenSSL library, when built, hard-codes a path to where it expects root certificates to be. That path can be different on your target system. (On Lambda, it's /etc/pki/tls.) You can specify it by passing --openssldir to the OpenSSL configure script.

    • But then, when you do make install, OpenSSL won't install anything because it will fail to write to /etc/pki/tls. This is true even when there's a --prefix argument, and it's even true when you do make install_sw. You can work around it by providing an INSTALL_PREFIX to make install as documented in the INSTALL file. Then it will "install" to INSTALL_PREFIX + PREFIX, which is a little annoying.

    • What I finally ended up with was:

      env CC="/home/ethan/.local/musl/bin/musl-gcc" ./Configure --prefix=/home/ethan/.local/musl --openssldir=/etc/pki/tls linux-x86_64
      make INSTALL_PREFIX=/tmp/openssl-build install
      # Then, in the project directory:
      env PATH=/home/ethan/.local/musl/bin:$PATH PKG_CONFIG_ALLOW_CROSS=1 OPENSSL_DIR=/tmp/openssl-build/home/ethan/.local/musl apex deploy my-function
  • Your Rust program can't print to stdout because of All output has to be through stderr.

  • I also hit which hasn't made it into an official release at the time of this writing. Fortunately, Cargo lets you specify a git repo as the "version" of a dependency, so I just built against git master.

  • I also hit because I wanted to look for one file, and if it wasn't there, look for another file, and I couldn't get the lifetimes to work out right. I ended up just duplicating some code to create two ZipArchives. Yuck.

I really enjoy working in Rust, even when it's difficult, because it feels like a "good" difficulty where I'm learning and growing as a programmer. I probably enjoy it more than I should because I spent way longer on this than I should have to demonstrate how "bleeding edge" this stuff is. My hope is that by documenting this, other people will start to use it a bit more and file off the sharp edges until it starts to be viable for everyone.

Thanks dalias on #musl, @tj on the Apex Slack, and sfackler on #rust for their invaluable help while I muddled through all this!

Posted Wed Mar 15 18:13:04 2017

bridges and powerlines

Bridges and Powerlines is one of my favorite bands, and one that I've been listening to since 2007 or so, when I found them on 3hive. I don't follow 3hive as much any more, but there was a time when I listened to every post as an act of civil disobedience against the recording industry (because in college, a classmate was sued by the RIAA). My "now playing" still has a collection of random tracks from that period, not to mention entire albums from artists like Say Hi (because), Children of the CPU (of), Margot and the Nuclear So and So's (posts), A Weather (like), and Barcelona (these). These days, though, my favorite website to discover music is Bandcamp. I just learned that the entire Bridges and Powerlines oeuvre is on Bandcamp, and that made me so excited I decided I had to write this blog post about them and their discography.

Bridges and Powerlines are four young Newyorkais men who put out ~NYC style power pop~ (in the words of 3hive). Their sound has evolved over time but it has always included a bit of an electronic element, some vocal harmonies, soaring melodies, and an occasional darker edge. Their lyrics hit the sweet spot for me between abstract enough to be profound and clear enough to be understandable.


Their first album is their self-titled EP. The third song, "Mumbles", is a quintessential Bridges and Powerlines song, starting with the vocal harmonies I mentioned above and some typical themes in the lyrics -- flawed relationships, wandering. My favorite track on this album is almost certainly All In, which feels dangerous, deranged, powerful. When their Wikipedia article talks about the "nervous energy that defined their earliest releases", I think of this song. A lot of people also like the first track, "Carmen", starts with the great line "You can't leave and still surround me/ There are rules that you should know". (You might also find a reprint of this album that bears the title "Sunspots", but it's the same album.)

Ghost Types, their first LP, develops the sound of the EP, enriching it and broadening it. The Thieves, They are Everywhere is a good example of what I mean, with a lot of the same ideas of "Mumbles" but with a richer sound, and maybe more complex lyrical themes. I think this is my favorite of their albums, with a lot of solid songs. Middle Child is pure ear candy, and is lyrically perfect -- "Never know which life to lead/ and which to leave behind" resonates very strongly when FOMO infests your life. This Last Equation is probably my favorite of the songs on this album.

On Eve, their next album, highlights include: The Jameson, which got me through my final months of Peace Corps; Mirabell, which starts off with the killer line "I was a soldier of nothing, ascendant"; the title track Eve, which is just a rocking track. Some of the songs on this album abandon the power-pop feel to go for a quieter sound, like "The Roman Leaders" or "The Leaves".

Then there's Better, which is an EP where each track is named after a different neighborhood of Brooklyn (mostly the trendy ones like Williamsburg and Bushwick, but also East New York). In my mind, this is the album where a laptop was added to the Bridges and Powerlines lineup -- there's a "break" in the middle of "Williamsburg", and some background tinkling sounds on "East New York", both of which are unusual in the rest of the canon. Park Slope is probably my favorite of the bunch, with its optimistic refrain that "We'll see mountains/ we'll see forests/ when it's over".

National Fantasy is their latest album. Although Boston Survival Guide says that there's a "positive, upbeat vibe here", I find the album pretty dark, even considering that above I described most of their work as having an edge. If you gave yourself a mixtape to grieve the horrific breakup you had with the year 2016, the title track would fit right in.

But it's still familiar, and the "laptop" sound from Better is reined in a little better here. I'm not sure I can say it's my "favorite" song because it's so melancholy, but I've listened to "National Fantasy" dozens of times. I'm completely in love with the lyrics, especially the opening lines ("Roads end like they never saw it coming/ But a few seem to slowly fade into nothing").


Bridges and Powerlines isn't just a band with great songs or even great albums. Over their five albums, there's a progression, and while it isn't as explicit as in e.g. the work of the Mountain Goats, I believe there are arcs here if you look for them.

The easiest theme to pick out is the theme of flawed relationships. In "The Ghost Types", the singer "once had a young love/ to lie about". In "The Last Equation", "Come down, come down, in the light of my room/ If I want, if I want, I'll just hide this from you". The singer of "Mirabell" is "a soldier of nothing, ascendant/ You were a signpost, simple, dependent", and "I might feel more a lifetime later in an old city". By the time we get to Better, there's clearly an ambivalence about the person who "broke into my body, stole my blood/ Built a tiny boat, and cried out 'flood'", who the singer also exhorts, "Don't cry/ We might see tomorrow if we try" (in "Bushwick"). This theme comes to a point in National Fantasy, where it's the focus of "Hurting Kind" ("To me your kind of love/ Is a hurting kind of love/ And I could never have enough").

There's a theme of leaving, and besides that, knowing when to leave. "The Thieves, They are Everywhere" starts this off, with the wistful "Just one more night", and "Middle Child" struggling with the lives to live and to leave behind. On "The Jameson", the refrain goes, "Why am I still here?", and on "Red Hook", it's "I'm going on my way". Finally, we come to National Fantasy, where the singer admits on "XOXO" that "It's better I am alone", even though in "Lindsay" he says "I just want to stay with you one last time".

The songs flirt with politics and history, though they don't ever quite go so far as to have a position, or maybe they just don't tell you what the position is. "New Mexico" on Eve seems like it could be about the Trinity nuclear test; "The Maine" is definitely about the ship that exploded in Havana. In "Gazes Wide", the singer tells us that "America is grinning, gazes wide", but I think that's more about show business than politics. The cover of National Fantasy has a picture of the sign at the Champlain border crossing, which says "United States of America", but reversed. Maybe the closest is "XOXO" from National Fantasy, which says "The way it was, it wasn't that way", which could be a rebuttal to a program of making America great again.

I think "Lindsay" from National Fantasy, with its refrain of "You're so far away", makes an interesting contrast to "Mumbles", from the first EP ("You're beautiful from far away/ but we don't even have a place where we can be alone"). "Mumbles" has a wistful longing, with a backdrop of struggle ("Tries/ We try to be someone") and wanderlust ("Marching the foreign lands"). In "Lindsay", it's almost like that struggle has turned sour, and the longing is now a loss -- "Your songs, they don't love me like they used to", and "You're so far away".

Of course stuff like this doesn't make or break an artist's appeal to you, but if you do like them, it can add a richness and depth to the songs that you only notice after ten years.


I am super into Bridges and Powerlines, and now that their music is on Bandcamp, it's super easy for you to be super into them too! Get started now.

Posted Sun Jan 8 08:24:10 2017


I just realized I forgot to mention up here a cute little hack I wrote a while ago called rst2wp. It's basically a Wordpress client with a few neat features, customized to my workflow:

  1. Reads source in ReStructuredText, which is the most coherent plaintext-formatting language out there.
  2. Lets you specify tags, not just categories (unlike most WP clients I've found).

3. Searches for image tags in each post, downloads them, and then uploads them to your server (so you don't hotlink images).

It's already successfully running my travelogue.

The docstring in the source is the only documentation so far. Any questions, email me.

Posted Sun Apr 25 23:24:56 2010


I'm OCD about whitespace in source code. Are you? If so, you may find ethan-wspace, the definitive emacs customizations for people who are OCD about whitespace, to be useful. The readme on Github will probably tell you all you need to know.

Posted Sun Apr 25 23:19:07 2010

So Long 2P.P.!

To bid a fond farewell to our friends at 2 Player Productions, I had this cake printed. The graphic was original, but based pretty clearly on the design of the Nintendo Game Boy. Because I'm not an artist, it looks like crap. To spare other people the misfortune of having to create their own hideous Game Boy design, I decided I'd put up my version.

I did this in Inkscape to maximize scalability, and also because the operations for handling geometric primitives are better. Here's the SVG source.

I used the fonts Alpha Beta BRK and Nanosecond Thin from the aenigma font package, and the Bitstream Vera Sans font for the little bits everywhere else. My greatest regret is not using a monospace font for the actual on-screen text, even though it looks really cool.

My source image was this Gameboy photo.

Posted Tue Sep 15 17:21:17 2009

I got my deck of 8-Bit Tarot in the mail today; this post is to chronicle my first impressions. If you aren't in the mood, perhaps you will like these 8bit music videos (my favorites, for varying reasons, are "Video Computer System", "Total Control", and "L-V---SC-LD-RTH-ND--TH" -- I typically love YMCK but these videos aren't their best).

The 8-Bit Tarot, designed by Indigo Kelleigh and published on, fuses the vibrant, digital, and modern-retro aesthetic of 8-bit with the mysticism and tradition of tarot. The deck itself is $30; for comparison, a standard Rider-Waite deck will cost you about $12 on Amazon, and a Robin Wood deck $13-14. The deck is packed in a box made of thin cardboard, suitable for display but not for travel (at least by my own, somewhat destructive standards). The cards themselves are small, especially by tarot standards -- 2.5" by 3.5", or a little bit smaller than a playing card. "Standard" tarot cards are more like 3" by 5". You can get a sense of the art from the deck's homepage at Lunar Bistro. The cardstock is glossy, a little thin, but turns out to be quite stiff; due to the cards' size and stiffness, they are very difficult to shuffle.

A bit of background about tarot decks: the most well-known deck, the Rider-Waite deck, was designed in the early 1900s according to "traditional" tarot mysticism. Since then, there have been many other decks created by other people, most in the vein of the Rider-Waite deck -- the Robin Wood deck, which I typically use, is one of these, although somewhat more accessible. There are other "traditions" of tarot deck. The 8-Bit Tarot deck is more-or-less a reskin of the standard Rider-Waite deck. More information about all of this can be found at the Tarotpedia.

Now, for the cards themselves -- they're freakin' gorgeous. I don't usually care for Rider-Waite, but it really pops off the cards here. The art is cartoony and fun, even on traditionally negative cards like the Devil or the Tower. One annoyance is that the artist's initials, "ik", are on every card in the lower-right. This isn't unheard of for Tarot decks -- most cards in the Robin Wood deck are signed or initialed, but in a pixelated medium the initials are large and distracting.

The deck hews very very closely to the Rider-Waite designs. As a fan of the Robin Wood deck, I found this somewhat disappointing. A lot of the cards resonated more deeply with me in the Robin Wood deck -- I keep looking at the Five of Pentacles and thinking, "No, they're supposed to be lepers".

But it's not just that I prefer the conventions of the Robin Wood deck -- the 8-Bit Tarot deck had a real opportunity to push the envelope a lot more, and make something really innovative and 8-bit at its heart, instead of a new coat of paint on something old. However, if Mr. Kelleigh had chosen to go that route, we might have ended up with a gimmick-y "Mario Tarot" or something equally tasteless. It seems to me that leaning on the Rider-Waite deck makes the 8-Bit Tarot deck solid and authentic, but some 8-bit hardliners out there may feel that the use of lo-fi aesthetics in this deck is in fact quite gimmicky, particularly the use of suit icons at the top of each card ("[cup] X 4" for the Four of Cups, for example).

My biggest complaint is the quality of the cardstock. Just shuffling the cards is a real challenge. I had to resort to the old three-piles trick. By and large I'd say this deck is a solid piece of art and a completely valid tarot deck.

Posted Sat May 30 12:12:23 2009

free software audio

Producing music with Free Software isn't all that hard, but as with other Free Software you need to know what you're doing. I don't. But I've learned a bit in wandering around and seeing what I can do. This article is meant to give an overview of what I think I know.

Samples or notes

Audio software can take one of two views of the audio it works with. One, the audio can be a very long and complex waveform, represented by a long list of samples. An audio CD is like this -- each second, 44100 samples are read off of the CD and played. Sample-based file formats are WAV, MP3, OGG. The variation in these samples creates pressure on air, which creates vibrations, which creates music. Two, the audio can be notes, which are "played" in some way by the software. If you've ever seen an antique player piano, it has a note-based view of music. Some device reads off notes and, depending on each note, takes some kind of action (presses a different key, or something). Note-based file formats are MIDI, IT, MOD.

Speeding up waveform-based audio will increase the frequency of the vibrations, and thus the pitch as well as the tempo. Imagine spinning a record around with your finger, faster and faster. Speeding up note-based audio only increases the tempo. I used to have a jack-in-the-box which played "Pop Goes the Weasel" when you cranked it -- cranking it faster made it play faster. If you ever had something like that, imagine that.

If you're working with waveform-based audio, your first choice is likely to be Audacity. If you're working with note-based audio, your first choice is likely to be Rosegarden.

Waveform-based audio

Waveform-based programs are good for recording from a mic, or doing postproduction on different tracks, but they're not optimal for creating sounds from scratch or composing music, so I'll skip over them.

Suggested tags: sound::waveform

Note-based audio

In note-based audio, notes are entered in some way, and then some software goes through the list of notes and plays them. Sometimes playing is done in the same program that edits the notes; sometimes it's done in a separate program.


  • sequencer: "a device or piece of software that allows the user to record, play back and edit musical patterns" (Wikipedia).

Suggested tags: sound::notes. There's already a works-with::music-notation, but to me this suggests a staff and clefs and things like that, which isn't true for programs like nyquist, which don't use any musical notation at all. Alternately, sound::sequenced or sound::MIDI might be more concise, but to me sound::notes encompasses all of the concepts that sound::sequenced and works-with::music-notation leave out.

Editing notes

Notes are arranged in patterns, which can be moved around, copied, or re-used, depending on the program. Editing a pattern is generally done in one of the following interfaces.


Music notation, with staffs, clefs, bars, and all the assorted musical notes. Classically-trained musicians might recognize and appreciate this kind of notation, but new musicians might not like it.


Suggested tags: sound::interface:musical-score or maybe sound::interface:notation?

Piano roll

Piano roll interfaces are like a big grid. On the left side of the grid, there are all the keys of a piano. Each key on the piano is aligned with a row of the grid. The grid extends to the right. Each note is shown as a rectangle on the grid, with its position showing both its pitch and its time, and its width showing its duration.


Suggested tags: sound::interface:pianoroll


Originally, the term "tracker" applied to a certain kind of electronic music program, meant for editing certain tracked music formats: MOD, XM, IT. These formats are meant for sample-based synthesis (see below) and have a certain distinctive row-based interface. The interface is general enough to apply to other programs -- shaketracker is a MIDI sequencer that uses this interface style.

A tracker interface looks like a screen full of numbers. Each row of numbers represents a note. Time goes down. The numbers in each column has a different meaning.


(A screenshot of Aldrin's tracker mode. Conceptually the unused columns are for envelopes, pitch bends, and so forth, but I couldn't make any of them work.)

Suggested tags: sound::interface:tracker

Tracker software

FastTracker 2 is a well-known piece of electronic music software from the early 90s. It's got a tracker interface and does its own sample-based synthesis. Soundtracker is a FastTracker clone.

ImpulseTracker is another DOS-era tracker. The .IT format comes from this tracker. SchismTracker and Cheesetracker are both clones of IT.

Suggested tags: sound::tracker-clone (if this is even necessary)


Once you have some patterns, you lay them out. Each program has different ways of doing this.


Once you have notes, you have to convert them to sounds. This is the domain of synthesizers. Some programs do both note-based editing and synthesis, like Aldrin, Beast, or LMMS, but there are also separate synthesizers like Fluidsynth, TiMidity, and ZynAddSubFX which can do synthesis for other programs.

Suggested tags: sound::synthesis


In sample-based synthesis, a sound file in WAV or other format is made for the instrument, and this "sample" is pitch-shifted to make different notes. Sample-based synthesizers often have options like using only part of the sample, looping it, etc.

Suggested tags: sound::synthesis:sample-based


There are other programs that do other forms of synthesis.

  • ZynAddSubFX: generic "software synthesizer", capable of producing many different kinds of sounds.
  • OM: "modular synth" in which components are connected to create a "synthesizer".
  • Aeolus/horgand: synthesizers that emulate the sound of a pipe organ.


Some protocols and acronyms:

  • MIDI channels. jackd handles routing MIDI messages. This is most common when talking to hardware synthesizers, or getting input from hardware MIDI devices. MIDI events include "the pitch and intensity of musical notes to play, control signals for parameters such as volumue, vibrato and panning, cues and clock signals to set the tempo." (Wikipedia.)
  • LADSPA. An API that is used by audio plugins; lots of programs support it (audacity does, and so does OM, in very different ways), and there are a ton of plugins (in blop, caps, cmt, fil-plugins, mcp-plugins, swh-plugins, and tap-plugins). LADSPA plugins generally transform audio in some way.
  • DSSI. "DSSI is an API for audio plugins.. It may be thought of as LADSPA-for-instruments, or something comparable to VSTi." (dssi homepage) Much like MIDI, information regarding what notes to play is sent to a DSSI plugin; however, unlike MIDI, the application sending the notes, gets the waveforms back, so that effects can be done on the resulting sounds. [This should be possible to do using OM, but I haven't actually successfully done it yet.] Note, however, that using jack, you can connect the output of one program to the input of another program, and do similar things like that. This may be why DSSI hasn't taken off quite so much.
  • ll-scope: a DSSI plugin which displays audio via an oscilloscope view
  • jack-rack: LADSPA effects plugins, hooks up with jack; "turns your computer into an effects box"

Tags: Are audio::ladspa and audio::dssi useful here? Not sure.

Realtime kernel support

Most audio programs communicate using jackd, which is a sound server meant for low-latency performance and arbitrary connections. Using some client (for example qjackctl), you can connect arbitrary ports on programs to other ports on other programs. In this way, you can route the output of your synthesizer through other effects plugins. jackd is cool but it's not used much outside of professional audio, so you likely haven't used it before. The number one stumbling block to getting jackd to work for you is setting up realtime support in your kernel.

Realtime support is actually not "hard" realtime support, as you might find in QNX. In this context, realtime just means "low latency". Low latency is important, because jackd wants to send samples around your computer at 44100 times per second (or whatever), and if it can't get a sample in on time, it'll sound bad.

Low latency support has two aspects: PREEMPT_RT, and realtime-lsm. PREEMPT_RT is a patch, maintained by Ingo Molnar and various others, which makes the kernel "fully preemptible", i.e. the kernel can be interrupted in many more circumstances than it used to. This supports lower latency by reducing the amount of time jackd will have to wait to interrupt the kernel. realtime-lsm is a module that grants extra permissions to processes, like jackd, that want priority on computing time. See LWN for more information on realtime support in Linux.

PREEMPT_RT is slowly being folded into the mainline kernel, and it may be that by the time you read this, kernels will by default support preemption. But if it doesn't, you'll have to:

  • grab the PREEMPT_RT patch from
  • apply it
  • make sure your config is correct
  • build kernels as you ordinarily do

realtime-lsm is available as a module, and on a Debian system you can apt-get install realtime-lsm, and use module-assistant to build it.

Posted Mon Sep 3 10:21:28 2007

"Reply-to munging considered harmful" considered infuriating

I'll admit right out that I'm "new" to email (meaning that I've only been using it for ten or twelve years, never implemented an MUA or an MTA, and don't administrate any mail servers), but there's this meme titled "Reply-To" Munging Considered Harmful (started by one Chip Rosenthal) that I would like to take issue with. The meme has managed to worm its way into mailing list administrators worldwide, and I even thought I would be one of them, waving the standard of "well-behaved Internet software" as I marched on. But it has not come to pass.

There are two mailing lists I'm on. One, the pygame-users mailing list, munges the Reply-To header. Another, the pyode-user mailing list, does not. On every mail I have replied to on the pyode mailing list, I have sent a follow-up to the list saying "oops, sorry Person-I-Replied-To for the duplicate mail, this was supposed to go to the list too". This hasn't happened once on the pygame mailing list. Mr. Rosenthal asserts that reply-to munging "adds nothing". This is not true. In my case, it adds relief from embarassing "shit I fucked up with the email thing" notes and copies sent to lists.

In most computer-mediated fora, if someone says something, and you'd like to respond, it is expected that the response will go to the same place that the original did (meaning: sent to the same people). Mr. Rosenthal asserts that any reasonable mailer has two functions, one for "reply" and one for "group reply", and that any idiot user can be taught to press "g" if they want to respond publically, and "r" if they want to respond privately. But a user expects that a "reply" will be sent to everyone relevant; it is the private responses that are special, not the public ones.

Back in the '90s, you frequently found that some AOL user had sent you and 400 other people a message, and one of those 400 other people had responded with "I don't want to read this crap", and another of those 400 other people had written back with "lol", etc., with each message growing longer and more meaningless until you gave up and got a different email account. This was entirely the result of making "reply-all" the default function. Idiot users who didn't know better would send replies to people who didn't need them. So for the most part I think making "reply privately" the default function for end-user software is a defensible choice; it prevents unwanted noise. But when you start a mailing list, you want to make the default reply function (on a per-message basis) "send to list"; it is for this purpose that reply-to munging is helpful.

This is the case for reply-to munging. Let's go through Mr. Rosenthal's arguments against.

  • It breaks the reply-privately function. It's not gone, only more difficult. This isn't always a make-or-break thing; if nobody ever replies privately, then it's a tradeoff that should be considered.

  • Administrators shouldn't dictate that all replies go to the list. This is a straw man. Administrators know best what the tone and flavor of their mailing list is. If it's primarily technical discussion, or a digital gathering space that doesn't usually wander off into private email discussions, then absolutely they should have the power to make replies go to the list by default.

  • Reply-to munging can make it difficult or impossible to find the sender of a message. As someone who is new to email, I'm not familiar with the uses Mr. Rosenthal offers: if someone is sending a message that they haven't authored, typically they indicate this fact in the body of the email and include the content of the message as a quote. If someone is sending email from one address, but would like to recieve email at a different address, they should change the From: header to match the Reply-To: header. (I know it's really more complicated than that, but I think this approach is viable.)

  • Munging reply-to headers penalizes those who use good software. Mr. Rosenthal writes, "If a few people need to type in a full reply address so that everybody else can use all the features of their mailer, I say, 'Fine!'" To this I reply: if a few people cannot use all the features of their mailer so that everybody's replies go sensible places by default, I say "Fine!"

  • Munging reply-to headers makes more work if you want to reply privately. On the pyode mailing list, hitting "Reply" will reply To: the person who sent the mail; hitting "Reply all" will go To: the sender and Cc: the list. Most users (including technically inclined ones) will assume this means that the sender will get a duplicate email from the list, and so will scratch out the To: line and change the Cc: to a To:. So not munging reply-to headers leads to users doing just as much work, assuming they even remember to reply publically. (Otherwise they have to go to their copy of the sent message and send a copy to the mailing list.)

    In fact, Mailman is intelligent enough to not send a duplicate email in this case, but frankly this is a huge surprise to me considering that Mailman sends passwords in plaintext emails. However, if B replies to A via the list L, the reply will be To: A and Cc: L. If C replies to this email, it will be To: B and Cc: A and Cc: L. This is just ridiculous.

  • A user expects a garden-variety reply to be private. This is just not true.

  • A user may expect a reply to be sent privately, only to discover it going out as public, which can lead to embarassment or worse. I have never seen this happen, and it has never happened to me.

  • If we munge reply-to headers, users will demand a third "reply" option which ignores reply-to headers but addresses an email to the sender of a message. This is complicated, but: if you already accept that there is a category of "private" replies and a category of "public" replies, why shouldn't the "private" reply in this case write to the sender? Anyhow, there's already a third type of "reply" called Reply-To-List, and it doesn't work, so I don't see what good Mr. Rosenthal's suggestion is.

In an ideal world, people would never make mistakes, and they would know that their mailer has reply-private and reply-public modes, and would invoke the correct ones all the time. That isn't the world we live in. In a slightly more plausible world, mailers would have a default "reply" mode which intelligently decides who gets a copy, based on who got the original message, whether it's part of a mailing list, and how dumb the user is, and explicit "private" and "public" modes in case the software's guess is wrong. We don't live in that world either. We also don't live in a world where users expect replies to be private by default. Munging reply-to can be a terrible thing, or it can be benign or beneficial. List administrators should not blindly follow Mr. Rosenthal's suggestion, but think about their users and what is likely to be best. For the lists I administrate, I'm enabling munging.

Lastly, email has grown to be a hugely complicated organism, made up of many users, many pieces of mail software, and many standards. We can't throw the whole thing out and start fresh; even if we did, how would replies work, given that many users are prone to noise generation? Reply-to munging is just a patch, but I think it's the best solution we have, and I'm going to use it when it's called for.

Posted Mon Aug 6 13:55:43 2007
combined log format CGI

Fist Farm needed a CGI that generated Combined Log Format for, which he was working on at the time. It's surprisingly hard to find one that does just that!

He wanted to call it from inside an SSI, but the ideas should be the same.

Here's what I came up with.

#! /usr/bin/perl

$logfile = "log.blah";

use POSIX;
my $time = POSIX::strftime("[%d/%b/%Y:%H:%M:%S +0000]", gmtime);
my $line = sprintf("%s %s %s %s \"%s\" %s %s \"%s\" \"%s\"\n",
                   $ENV {REMOTE_ADDR} || "-", # %h
                   $ENV {REMOTE_IDENT} || "-", # %l
                   $ENV {REMOTE_USER} || "-", # %u
                   $time,              # %t
                   "$ENV{REQUEST_METHOD} $ENV{REQUEST_URI}", # %r
                   "-", # %>s  (status code)
                   "-", # %b (size)
                   $ENV {HTTP_REFERER} || "-",
                   $ENV {HTTP_USER_AGENT}) || "-");
open LOG, ">>$logfile";
print LOG $line;
close LOG;

print "Content-type: text/html\n\n";
print "";

Note that it's impossible to guess the status code or the size of the request from a CGI; I just left them blank.

To call using an SSI, I did:

<!--#include virtual="blah.cgi" -->

But Fist Farm recommends:

<!--#exec cgi="blah.cgi"-->

Posted Mon Jun 11 15:27:22 2007
Spring Semester, 2118

Although the Halloween party started at 8, I was in the library working on a paper until 10. I didn't have a costume so I didn't really feel like going, and besides that I didn't know any of the people at the frat that was hosting it. I'd hoped to write another few hundred words that night while everyone else was out partying, but when I found myself alone my thoughts turned, as they usually did, towards me, and how it didn't matter how many words I wrote because I wasn't very smart to begin with. Eventually I decided no matter how long I tormented myself in a study cell, I wasn't going to put any more words down that night, so I went to the party anyhow. As expected, I didn't know anyone there, so I found myself sitting on an old plaid couch nursing a beer and watching people dancing, talking, pairing off and going elsewhere.

When she walked in, of course, she caught pretty much every eye in the room. You could hear the noises that people make when they stop making out because something extraordinary is going on. She was dressed in a full-body powder blue jumpsuit, and her face and hair had been colored to match. Only her lips were red -- the rest of her was that sharp powder blue, marred only by the telepath badge above her left breast. She must have felt all those heads turn towards her, and maybe she even picked up on the lewd sorts of things people think when they've been drinking, but at no point did a blush ever darken those faint blue cheeks.

She gathered up the grace that the pressure of stares can provide. She walked regally over to the fridge, got herself a drink, and sat down in the easy chair next to the couch. She was closer to me now and I could see her jumpsuit better. It had a zipper down the front and loose fabric at the wrists. Her hands were blue too; had she gotten a full-body cosmetic or was she just careful?

I continued to look her over as she glanced about the room. I had no idea whether she was reading minds or merely checking out the scene (this was before the different marks on telepath badges were introduced). She moved slowly, as though careful or thorough with her gaze, and never changed expression. Eventually her gaze came to rest on me.

"Hi," she said. She smiled. "My name is Kimberly, but everyone calls me Kim."

"Hello, Kim," I said. "My name is Nate."

"Hello, Nate." She continued to stare at me, though after a few seconds her smile faded. Eventually I realized that she must be reading my mind, though I didn't feel anything. I waited, silent, for her to finish.

"You're pretty messed up, aren't you, Nate?" she said.

"Yeah," I said, thinking about how glad I was to be at a party, not alone, distracted from my numerous failings.

"I like being in your mind," she purred. She moved to sit next to me on the couch.

"It's a good mind, I like it, it suits me well," I gibbered, dumbstruck by her attention.

A short while later, we paired off and went elsewhere ourselves. It turned out to be a whole-body cosmetic after all.


I didn't realize she was in my section of Social Dynamics until the next day. As I entered the discussion room, she smiled and waved at me. She still had blue hair, but she had gotten rid of the rest of the cosmetic. I'm pretty sure I blushed and waved back. I sat in the open seat behind her.

"Hi Nate!" she giggled, leaning her head back to look at me upside-down. "Great party last night, huh?"

"Hi Kim," I said. "I must admit I wasn't paying too much attention to the party." She giggled at that too. She was very pretty when she giggled. "The TA just sent out our grades on the last test -- how did you do?" When I asked that, her face lost all traces of mirth. I wasn't an esper myself, but the serious look didn't bode well for her grades. She righted herself and turned around in the seat.

"I'm kind of embarassed about it, actually," she said.

"Say no more," I assured her, ever the charmer. "You know, maybe we could have a study session some time."

"I don't know," she said. "I don't really like studying, you know?" She giggled again. I wondered if there was anything behind that pretty giggle. "But maybe we can go for coffee instead."

"Well, Kim, that sounds like fun. Maybe when finals come around I can talk you into that study session."

"I doubt it!" she said, and tossed her blue hair. "I'm very stubborn, you know."

Just then the TA came in, a smile on his face and a spring in his step. He casually went over some of the mistakes he'd found in the papers we'd turned in. He exuded a sense of superiority, but it had been worth it to see him one-foot-in-the-grave last week, when he was still grading. He said someone in the class had gotten an A-plus on the paper, but I found that hard to believe. As he rambled sonorously, I looked at the back of Kim's head. She was staring off to one side, presumably into the middle distance. If she hadn't been doing well in class, why wasn't she paying better attention? Maybe I had hooked up with a ditz. After all, she was certainly pretty enough. Not that you had to be pretty to be a ditz, I thought quickly. This was college, and we were all just trying to look sophisticated. I was urbane enough to allow for the possibility that a ditz could be ugly.

What have you gotten yourself into, Nate? I thought.


The next day I met for lunch with my friend Connie.

"You should break up with her," she said, as soon as I mentioned Kim.

"We just went for coffee," I protested. "What, are you afraid she's gonna take over my body or erase my memories or something like that?"

"Don't be ridiculous," Connie said. "But you should break up with her."

"I'm not even dating her yet!" I tried to describe what had happened at the party and after class as best as I could. "We just stayed and talked. Although, to be honest, I get the feeling she's not very bright. Whenever I mentioned classes, she clumsily changed the subject."

"You need to break up with her," Connie repeated. "She can read your mind! Doesn't that scare you? She knows all your dirty little secrets, and she's never even been to your apartment."

"I don't know, Connie. Maybe it means she'll be able to understand me, without all those stupid words that get in the way. Don't you want to be understood by whoever you're dating?"

"You know," Connie said, looking thoughtful, "I had a friend back home who was a lesbian. I thought, wow, that's gotta be so great. Your partner isn't some insensitive guy with a different sex drive than you. It's some woman who knows exactly what you're going through, biology and all. But I asked my friend, hey, do you ever fight? Or do you just understand each other all the time? And she gave me this look, right? and said, oh, we fight from time to time. And when we fight, it's no holds barred. We fight for blood. And we can hurt each other so much worse, because we understand each other." She gave me a withering look, the look her friend must have given her. "Listen, I think it's great you're dating, you know, a telepath. And I don't go around throwing rocks and calling anyone names or anything like that. But I just think a relationship has its secrets, and maybe there are some things you don't want her to know. I'm not just talking about your inability to dance or your filthy apartment, you know."

"Thanks, Connie," I said. "She's seen me without any clothes on, and then she still wanted to have coffee with me. I have to say I don't think I'm afraid of what else she might know about me." Of course, I really hoped that Kim did know all there was to know about me, and love me anyhow, or know how to fix me. But I wasn't going to admit that to Connie.

"Well, it's your funeral. Or insane asylum or whatever." Connie focused on her sandwich.


Kim and I got closer quite naturally, and we began sitting side by side in Social Dynamics. She did see my filthy apartment and we did go dancing, quite a few times. But then a paper or test would come up.

"Hey Nate, want to swing by my place for dinner? I'm cooking lasagna." She meant really cooking, cutting up ingredients and applying heat the old-fashioned way, not just entering selections in a menu. She was even dressed for the occasion. She was wearing an apron and was covered in flour. It was like a white version of the all-blue outfit I saw her in on the night we met.

"God, that sounds good, Kim, but there's this paper due Tuesday."

"Aww, damn," she said. "Well, more leftovers for me, I guess."

"Come over and let's work on the paper together."

"Oh, I can't do that, Nate, the tomatoes would go to waste. Do you know how hard it is to get real tomatoes? It ought to be a crime to waste them."

"They'll make more. Come on, let me see your paper, maybe I can help you with it."

"That's sweet, Nate, but I don't want to look at my paper! I want to eat lasagna. With you. Are you sure you can't come by?"

"Sorry, baby. I really need to get through another two thousand words tonight if I'm gonna have enough time to revise. Some of us aren't able to just wiggle our nose and read Hendrick's mind, you know?"

Her eyes went wide at the suggestion. "Me? I would never! That would be completely unethical. Besides, he's always focusing too hard on the brunette in the third row to think anything useful." She giggled. "You know the one? With the huge lips?"

I couldn't help but laugh. "All right, Kim. I'll see you in class, then." I disconnected, and hoped she really would pull it together before the semester ended.


"What's it like, Kim? Being a telepath?"

"It's lots of fun!" Giggle. "The first part, knowing how people feel, it's not as different as you might think. It's sort of like the women's intuition people always talk about, except a little better. Then there's when you do a deep scan. That's like going into someone's bedroom, seeing all the things around them that they take for granted. Thoughts half-finished because they've thought them a million times before. The things they value, scattered here and there because they didn't know you were going to be visiting."

"Did you ever try to lie, and hide your telepathy?"

"You remember those esper tests we took in first grade? The guy who did mine was so scary. I knew he was an esper, and I thought if I was an esper he was going to take me away with him and I'd never see my family again. So when he asked me, you know, How many fingers am I holding up, Can you tell me what I'm thinking, I told him I had no idea. They told me he himself was an esper but I didn't realize what that meant." We shared a laugh at her younger self's frightened misunderstanding. "But he didn't take me away, so I thought he believed me. And then the next week, I got a badge, and I was so proud! It was my very first piece of grown-up jewelry. I think I still have it at home somewhere."

"Have you looked at that analysis matrix for Dynamics yet?"

"Oh, Nate! You're the worst at pillow talk. Let's go back to talking about me, that was great!" Giggle.


When I got back my C-plus in Social Dynamics, I thought Hendricks was a jerk but concluded that it was probably a fair grade. It had been a hard class. The TA published some grade statistics -- I was barely over the median. Some overachiever had gotten an A and spoiled the curve for the rest of us. Damn, I needed a better grade than C-plus to pull up my GPA. Hopefully I'd do better in one of the other classes I was taking.

I went to Kim's to tell her the news. She wasn't there but her AI let me in. I ordered a beer from the automat and, once it synthesized, lay on her bed and sipped it. I wasn't thinking about anything in particular when I heard Kim come in the front door.

"Command: Check messages. Audio only." she said, walking over to the kitchen to get something for herself. The AI followed her with the sound, so I didn't hear anything. "Next," she said. "Blah, blah, blah.. next."

She walked into the bedroom where I was just as the next message started. She didn't notice me where I was, but I heard the message: "Congratulations, Kim, you made the Dean's List. Again. Be well. Bye."

"Dean's List?" I asked, maybe more forcefully than I should have. Kim spun around and grabbed at her chest. "Oh my God, Nate, don't do that! You scared the shit out of me!" She giggled again. "How nice of you to surprise me like this! Man, you would not believe --"

"Dean's List?" I asked again. "How did you make Dean's List? You're the worst student in the entire Dynamics class! I only got a C-plus! You couldn't have done better than me!"

The giggle was gone now. She was staring at me calmly, rationally. She ran her hand through her hair, sighed. "Actually, Nate, remember the student who got an A? That was me."

"Oh my God." I closed my eyes, leaned back against the pillows. "You cheated, didn't you? I can't believe it. How could you? Don't you understand what this means? If they find out --"

"I didn't cheat, Nate." She spoke quietly. "You can't just mind-read an entire paper. Do you think I want to spend any more time with that TA than I have to? He's full of himself. Hendricks is even more full of himself and he's a lecher too. I wrote those papers myself."

"You lied to me," I said, scrambling out of bed. "I can't believe this. I'm out of here." I left the beer and broke into a run. I got back to my apartment sweaty and out-of-breath. "Command: lock, all," I gasped. I collapsed into a chair at the kitchen table. It took me some time before my heart rate returned to normal, but then I put my head on the table, covered with my scalp with my hands and hoped for the world to end.


Kim contacted me that night through the comm system.

"Is it that frightening that someone could be smarter than you?" she asked.

"It isn't about that," I said, though secretly I was afraid it was. "You lied to me."

"Oh, this is about the fact that you thought I was dumb. And that's supposed to be my fault. The truth is that I really am embarassed about my grades. It's hard enough being a telepath, let alone being the top student at this school too. But that wasn't good enough -- not only am I supposed to tell you the truth, I have to make you think it too."

"Don't give me that," I snapped. "You acted a certain way, knowing what I'd think. You encouraged me to believe you were a -- a --"

"A bimbo?"

"Yeah. You can't claim any moral high ground just because you never said the words. You're a telepath, you knew what I'd think! And you said exactly what it took to make me think it!"

"What was I supposed to say, Nate? Hi, I'm Kim, and while you've been staring at my chest I've been thinking about the next four hundred words I have to write for my paper due Thursday? I'm not responsible for other people's misunderstandings. I don't go around correcting people all the time. If you really wanted to know something about me, you should have asked flat out!"

"Oh, so now I'm supposed to ask explicitly about everything? Is it true that you're in my Social Dynamics class, or did you sneak in? For that matter, do you even go to school here or do you just go to the parties?"

"Don't get snippy, Nate.."

"How should I get? Don't you think lying is different for telepaths? Don't you think that manipulating someone into a misunderstanding of the world is bad, no matter what words you use? How can you expect me to trust you when you've misled me all along?"

"What difference would it have made? Did you want me to double-check your work? Provide you with crib sheets? Drop my pencil if the answer to question one was true?"

"Is that what you think I'm like? That I'm really that shameless?"

"What I know," she said, "is that you care an awful lot about your grades. And that you like to feel more intelligent than other people. Regardless of what I did or did not do, we had a happy few months together. Isn't that enough?"

"It's over, Kim," I said, with a finality I didn't feel. I don't know if she could read my mind over the commnet, but her expression didn't change, and she sat quietly for a few moments.

"Nate," she began, "I've lived with the thoughts of strangers for 20 years. In all that time, I've found out that people lie. They lie all the time. They lie to themselves, loved ones, strangers. And I didn't even do that. I dodged one question about my grades and I giggled a lot. If that's too much dishonesty for you, heaven help you if you ever fall in love." She disconnected.

I don't know if she graduated, moved away, or simply avoided me. Maybe I guessed right and she never really went to school there after all. I didn't see her there ever again. Life goes on.

Kim was right. Any woman who loves me is going to know me well enough to lie to me. Isn't love about trusting the other person anyway?

Connie says I did the right thing, that it's impossible to know whether a telepath of that caliber is lying to you. Connie thinks a miserable truth is better than a happy fiction any day of the week, and that I managed to get the miserable truth. But sometimes, when the library is empty, I think back to Kim and wonder if maybe I'd have been better served by loving the blue-suited woman who knew me well enough to lie to me.

Posted Thu Jun 7 10:46:14 2007

Blue Sky design by Jonas John.