Month of November 2003

More cirCLe CD news, or lack thereof: spent most of the week getting to grips with the Debian kernel packaging stuff, and building initrds, and El Torito boot images. Then spent most of the weekend so far writing up what I'd done (look out for forthcoming Linux Journal article, with any luck).

The simplest way to do moderately sensible removable media support seems to be a combination of hotplug (to notice when new devices appear/disappear), supermount (to notice when/whether these devices actually contain disks) and xfe: a file manager which doesn't require a whole desktop infrastructure to be present before it will so much as run. I remember using supermount an awful long time ago (I guess it would have been about 1995, when I still used floppies for anything): I'm amazed it still exists and is available for current kernels. Kudos to Stephen Tweedie and all the hackers who've worked on it since then.

I don't often use a file manager: a shell with tab completion does most of what I want to do with files. After playing with xfe for a few minutes to see if it works, though, I notice my home directory is significantly less cluttered with random files I'll never use again. Perhaps I should use a file manager.

I have had enough, for the time being, of USB hotplug. I cobbled together a plausible though grotesquely kludgey way to do automounting even in the presence of partitioned devices: it's an autofs "program" map which when asked for the device "list" scans /proc/partitions, assuming that /dev/[sh]dan is a partition of [sh]da, and populates subdirectories of symlinks to the actual mountable devices. See, I told you I was ill.

No, the good news is that it's time to put another release of araneida together. The feature that people are actually wanting is that this one will compile cleanly on current SBCL (so you can stop sending me bug reports about sb-unix::sigpipe vs :sigpipe), but the feature I will be foisting on them additionally is that the rather strange export-server thing is gone, along with another few global variables. These days instead you can create an http-listener (which may be threaded or may use serve-event) which does the actual listening-on-a-port business and has a root-handler slot containing the handler to which it dispatches requests. We're also going to be handling reverse-proxied requests a little more correctly: it should now be possible for the front-end box to be on a different hostname as well as a different port number, which wasn't previously the case if my reading of the code is correct (i.e. if it did work it was a coincidence). How it will work is that you install your existing handlers as if you were using the external host, then add a `internal-redirect-handler' (which I haven't thought of a sensible name for yet, much less implemented) for requests to the internal URL: this creates a child request with a suitably rewritten url and dispatches that.

When will it be released? When I've done the redirect handler and ideally also decided where to hang the ssl key/certificate information that used previously to be on the server object.

I have spam filtering again, and this time it's filtered server-side. This makes me happy, or at least less readily depressed when I get up and see 200 new mails of which approximately 5 are at all likely to be interesting to me.

My mail is delivered to a shell host using SMTP. I pull it onto my laptop using fetchmail and "ssh shell /usr/sbin/imapd" when I want to read it. Then emacs Gnus splits it into folders for me. I like this setup: I have all my mail archives to hand locally when I'm away from the net, and if I have some form of shell access but no way to get my laptop on the net I can still read new mail using mutt on the server. So I added procmail on the server to pipe through spambayes then filter spam into its own folder, then added a line in fetchmailrc on the client to download that folder (not automatically, but just in case) then stuck two new keys in gnus to retrain spambayes when it misses something - by, uh, ssh into the server host and spitting the article contents into sb_filter.py. Yeah, I know, it's tacky.

(defun spambayes-train-spam (&optional arg)
  (interactive (gnus-interactive "P"))
  (let ((gnus-default-article-saver 
	 (lambda (x) 
	   (gnus-summary-save-in-pipe
	    "ssh eval.metacircles.com /home/dan/python/bin/sb_filter.py -s -f")))
	(gnus-save-all-headers t))
    (gnus-summary-save-article arg t)))

As is common with any kind of Unix mail setup stuff there are seven zillion possible different ways to do it and I tried several before settling on this one. Most of the wasted time was in trying to get Gnus to do disconnected IMAP support, which was basically pain. Some of it was installing Python from source three times (the debian stable packages are too old to run spambayes): the first two builds were missing dbm and zlib respectively owing to missing header files.

It seems to be working pretty well, so far, and should definitely makes mornings a bit less insta-bad-mood.

Neil Gaiman being asked about his Duran Duran biography in an interview:

Whenever I do things because I want to do it and because it seems fun or interesting and so on and so forth, it almost always works. And it almost always winds up more than paying for itself. Whenever I do things for the money, not only does it prove a headache and a pain in the neck and come with all sorts of awful things attached, but I normally don't wind up getting the money, either. So, after a while, you do sort of start to learn [to] just forget about the things where people come to you and dangle huge wads of cash in front of you. Go for the one that seems interesting because, even if it all falls apart, you've got something interesting out of it. Whereas, the other way, you normally wind up getting absolutely nothing out of it.

Of course, it works better if what you want to do is write books like American Gods than if your idea of "fun and interesting" is, say, staying in bed until mid-afternoon. Or possibly even hacking free Lisp software.

New Araneida finally out of the door.

Bcc: dan
To: lispweb@red-bean.com
Subject: ANNOUNCE: release Araneida 0.80
X-Draft-From: ("nnml+private:mail.lists.sbcl-devel" 3030)
From: Daniel Barlow 
Date: Mon, 10 Nov 2003 05:25:01 +0000
Message-ID: <874qxckd76.fsf@noetbook.telent.net>
-text follows this line-
<#secure method=pgpmime mode=sign>

Araneida 0.80 has been released to ftp.linux.org.uk and is now making
its way to whichever CCLAN mirrors are actually still maintained.
From the release notes:

 New HTTP-LISTENER abstraction drastically improves the threaded
  variant of Araneida, and also replaces a lot of older ad-hocky code
  and global variables.  The following are at least obsolete, and may
  not even exist any more

  - The 'server' stuff: export-server etc
  - root-handler
  - install-{thread,serve-event}-handlers, remove-handlers

 Raising HTTP errors from handlers is now accomplished in a far
  cleaner fashion, by signalling CL conditions.  (This also means that
  parent handlers may handle these conditions to provide e.g. custom
  404 pages).  A condition exists for each 4xx and 5xx HTTP response code

 Add REFRESH header to REQUEST-SEND-HEADERS; sends the not-quite-HTTP
  'Refresh: ' header (really, not part of HTTP/1.0 or 1.1: it appears to
  have been a Netscape extension)

 Fix packaging problem in 0.72

 ... and even compiles (in fact, only compiles) in current SBCL versions


This version of Araneida requires source code changes in your
application - although not terribly pervasive ones; unless there are
vastly more Araneida users than I'm aware of, I felt it would be
quicker for users to update their configuration than for me to provice
legacy compatibility routines.  There is a complete and working
example of Araneida configuration in doc/example.lisp: it should be
about three lines for most people.

Feedback appreciated.  


-dan

 

   http://web.metacircles.com/cirCLe_CD - Free Software Lisp/Linux distro

Just back from listening to Neil Gaiman read at Borders. The event listing on his web site didn't mention signing, so I foolishly assumed there would be none; when I left the event there was a queue of better-prepared people halfway across the floor.

Fun, though. He read "Wolves in the Walls", then answered questions. " Where do I get my ideas from? Confluence. You should be very careful asking that question, because 90% of authors will make up a joke answer". Favourite characters: Delirium (Sandman), Crowley - or possibly Death (Good Omens), Wednesday (American Gods). No, there probably won't be another Pratchett collaboration. His Babylon 5 episode was pretty much entirely his own work. And his sinister beard makes him look, well, sinister. But also kind of cool (as if he didn't already).

So I finally upgraded my desktop to 2.6 (2.6.0-test9), at least, after fiddling a bit so that it's still using the old OSS-style sound module instead of spiffy new Alsa. This is because spiffy new Alsa doesn't work, as far as I can tell: there seems to be no way to persuade my motherboard to put the video card (Voodoo3) and the onboard audio in different interrupts: ye olde technologie trident.o can cope with sharing, but ALSA snd-trident makes some really really evil noises every time I select a menu item in Mozilla. Anyway, that's an exploration for another day.

What this means immediately is that it's time to play with futexes.

Kernel interfaces vs Glibc/POSIX: discuss.

One thing that working on SBCL threads has really brought home to me is that there is sometimes much better taste in API design in the Linux kernel than there is in the POSIX standards that the GNU/Linux project as a whole seeks to follow.

clone() vs POSIX threads is one example: either I can have this nice clean interface where the receiver of a signal is the person who asked for it or otherwise induced it, or I can have some weird monstrosity where signals are sent randomly to any thread that hasn't blocked them. But I'm sure I've talked about that before, at the UKUUG conference if nowhere else.

I think I'm finding similar issues in futexes. The system call support for futexes is, at base, really really simple: you have

• one operation that tests whether some word in memory is a given value, and goes to sleep in a queue in the kernel if so.
• another operation that wakes some specified number of tasks which are waiting on that queue.

But then, you have the "approved" method of accessing it from userspace (futex(4)): basically a counting semaphore which uses these syscalls when the lock is contended. But we didn't want a counting semaphore. And even that you're not supposed to use directly, instead relying on your trusty NPTL implementation - which means FFI bindings to pthread_* functions in glibc, which means function call overhead - even in the uncontended case - if they really are functions, and screwing around with glue code if as I suspect half of them turn out to be inlined or macros.

So, considered in isolation, I think good taste requires that I call sys_futex() directly (modulo that I'm not sure it didn't get deprecated and replaced with a separate futex_* call for each different operation sometime in 2.5). Considered in the context of programs that do FFI to libraries that expect the full layered NPTL experience, though, will we interoperate?

At root the question is: are we building a Lisp that runs on a Linux system, or a Lisp that runs on a GNU/Linux system? It's ironic that I can use Richard Stallman's nomenclature to describe the difference: the Linux system that he's not involved with seems pretty language-agnostic, but the GNU system, which was originally created by this Lisp advocate and former Lisp Machine hacker, seems to only really like C programmers. Good luck finding the value of SIGRTMIN in any other language, anyway. (Yes, the Glibc behaviour described there is certaintly allowed by POSIX. It's not particularly helpful and I don't think it really holds up against the Principle of Least Surprise, but I can't argue that the SUS says "integral expression" and not something more helpful like "constant expression")

Last time I was seriously immersed in a looking-for-jobs frame of reference was a few years ago, when things were different. These days the job ads I see are mostly from agencies, and don't offer a lot of possibility for purposeful parody that's in any way distinguishable from actual real advertisements.

Looks like it's a day for linking to stuff I wrote already. So, here's my opinion on ESR's "Art of Unix Programming".

4

I should say a few words about futexes. Adding futex support to sbcl (using sys_futex, counting semaphores be damned) was mostly a straightforward job taking a couple of afternoons.

Unfortunately, it seems to have exposed a bug somewhere in GC handling or stopping-the-world issues around GC. GC itself is wrapped in a without-interrupts, which uses a mechanism very similar to pseudo-atomic to delay incoming signals: when the cleanup from without-interrupts detects that a signal was supposed to have happened, it calls int 3, and the SIGTRAP handler then runs the handler for the stored signal. At least, that's how it's supposed to work. Right now it looks like the kernel has decided that the SIGTRAP can't be delivered to the process in the normal fashion and instead it would be a good idea to SIGKILL all of our threads. This is probably some kind of stack corruption. Attaching a debugger to the right thread in the right place may be interesting.

I'm not sure exactly what to write here, but I feel an urge to whinge a bit about the IT job market in the UK right now. Translations:

If they say Unix, they mean enterprise or ISP sysadmin experience: Veritas, EMC, clustering, etc.

If they say C, they mean embedded or they mean C++.

If they say OOD, they also mean C++.

If they say threads, they mean pthreads.

If they say Java, they mean J2EE.

If they say Perl, they mean entry level.

I don't think anyone has said Lisp yet, but when they do, they mean AI.

Do I sound bitter? Very well, then, I sound bitter.

Quick note: the futex problem is not actually futex-related. Linux 2.6 behaves differently to 2.4 in what it does when int 3 happens while a SIGTRAP is masked.

Quick note 2: Araneida 0.81 (bug fixes only) to start happening tonight when I get back from dinner. I need a better test suite for that thing.

Quick followup re 2.4/2.6 SIGTRAP difference (another reason the kernel is better than glibc: when you ask questions you actually get replies from people working on it)

The cirCLe CD is on hold pending a rethink. Response from the survey showed lots (well, some few tens) of existing CL users who thought it would be neat, plus a few people who might want to try it for curiosity's sake (personal favourite: the person who wanted a Genera clone for $25) but neither of these is sufficient evidence of a sustainable market. Maybe I'm asking the wrong questions; almost certainly I'm asking them in the wrong place.

So, Marketing 101: products have features but customers actually want benefits. (Unless, for some reason, the feature in question is XML, but I digress). So, what are the benefits of Lisp? What's our USP?

I think the answer is interactive development: in particular for Web application development, where the usual state of the art seems to be either error reporting on the web page itself (which is a start, provided you can turn it off in production - do you want your customers reading your backtraces?) or dumping stuff in a log file. So I've been experimenting with SLIME and Araneida with the intention that (a) errors in Araneida handlers can be debugged using the Slime debugger, (b) Slime can connect across the network (SSH-tunnelled) to a long-lived server process. Some progress to report on (a): it works, but nothing is in CVS yet

Also today, set up a Paypal account: dan@metacircles.com. Not altogether sure what I'm hoping that people will send me money for: if you've solved a problem (or just enjoyed) using CLiki or SBCL (threads, sockets, CLX, Alpha, PPC) or Araneida or cclan or asdf or detachtty or ... and want to fund further development, now you have a mechanism. It might also be a good idea if you sent me email saying which of these you're using: not that your $40 is going to significantly influence my development whim, but in the aggregate your collective $4000 might.

Thanks, by the way, go to Erik Enge for renewing the cliki.net domain when i managed to miss the reminders for it.

Today, in Oxford

metacircles is now offering per-incident paid-for support for SBCL and related software. Presently this has less to do with actual customer demand and more to do with distracting the people who point to lack of commercial support as a reason not to use free CL predicted customer demand.

There is a fix in SBCL CVS sb-bsd-sockets for the writing-to-closed-files bug that SLIME users sometimes experience.

Last week, Frank Neubüser found the notes I made last year on Oracle Wallet Manager. By spooky co-incidence, I had occasion on Monday to revisit this experience and revise my notes in that light: you can now find the new version (and Frank's email) over at metacircles

OK, more content on web.metacircles.com: Scripting Emacs is an article I wrote for Linux User & Developer magazine last spring (published last summer, as is the way with magazine lead times). The "& Developer" part of their magazine doesn't get as far as their web site, so here's a small bit of it on mine.

I have written other stuff for them too, which I might post sooner or later. That said, the CLiki article is probably out of date, and the SBCL threads article doesn't tell you anything that reading the paper from my UKUUG presentation wouldn't. Of course it, too, is out of date. That's the price of progress. Or something.

Just spotted this on lisp-p.org: Thoughts about Game & Simulation Programming Architecture. haven't read it yet, it's in the queue

Preceding it in said queue:

• make QUIT work sensibly when background threads exist
• have threads removed from the interactive-threads list when they die
• investigate some problem with OpenGL bindings

I recently - after only five years ownership of this Voodoo3 - did the necessary fiddling to get hardware-accelerated 3d (turns out that I seem to be mostly immune to Tuxracer) so that OpenGL thing looks like it could be interesting.

So the latest released version of CLiki doesn't work in current SBCL, and the instructions are out of date w.r.t the current Araneida. Planning to release a new version which fixes both these problems imminently (as in, probably about Monday).

There's also a ton of other changes, which is why I'm preannouncing it. Anyone who wants to play with collaborative content editing this weekend is invited to check it out from CVS: instructions at http://cvs.telent.net/. There's a new example.lisp file which should get you up and running with minimum effort: I'm pretty sure CLiki gets easier to install with every new version.

Bug reports, suggestions, queries welcome. From the NEWS file -

New in CLiki 0.4.1

 Update README file to correspond to Araneida 0.81, and add
  example.lisp demonstrating a simple CLiki installation

 Change CLIKI-INSTANCE class graph.  Now we have CLIKI-VIEW as a parent
  of CLIKI-INSTANCE.  The approximate division is that CLIKI-VIEW deals
  with the visual aspects and CLIKI-INSTANCE deals with updating and
  saving data

 CLIKI-SKIN is a new CLIKI-VIEW subclass demonstrating a way to make
  a second look and feel for an existing cliki.

 New CLIKI-PAGE-SURROUND method replaces CLIKI-PAGE-HEADER and
  CLIKI-PAGE-FOOTER.  WITH-PAGE-SURROUND macro allows any response
  handler to reasonably conveniently use the standard surround for
  some cliki it's associated with

 New class AUTHED-CLIKI does cookie-based authentication for clikis that 
  you don't want just anyone to edit.  Badly, though.  Feel free to send
  patches

 At last, proper escaping of page pathnames.  Page titles may now contain
  interesting characters like #\#, #\., #\+ etc

  If you have any pages stored in files with even mildly interesting 
  names, they will change: what was "Foo Bar" is now "Foo=20Bar", for
  example.  The old names are still read, and updated to new names on
  SAVE-PAGE (i.e. after edits)

 #\Space and #\_ are treated as equivalent in page names, just as #\A
  and #\a are.

 SAVE-PAGE is now a GF specialkised on CLIKI-VIEW.  Might be usable
  for e.g. database-backed clikis.

 Bug fix: admin/recent-changes.dat is created if not found

 Some HTML cleanup

I also fixed the QUIT behaviour of SBCL to be ... well, different again. This probably broke sb-aclrepl and anyone else creating new sessions (ELI, I think), but the good news is that they can replace their old crufty code with new slightly less crufty code.

So, reviewing how far I got today, I realise that I didn't make it as far as "remove threads from the session on exit". We need some way of notifying the thread that created the exiting thread that its child exited, then it needs to clean up. Briefly, then

1. Remove CLONE_PARENT from clone() flags. We don't need it now we're not doing stop-for-GC with ptrace(). Add instead an RT signal called, say, SIG_THREAD_EXIT, so that our thread death signals queue up when lots of children exit at once.

2. Add a C handler for thread exit, which does normal pseudo-atomic stalling then calls a Lisp handler.

3. Write the Lisp handler to do appropriate surgery on SESSION and foreign-call destroy_thread.

Since we're here, we may as well make the new handlers call-behind. Since we're doing that, we may as well add a general Lisp interface for establishing call-behind handlers, and have a look at the existing handlers to see what could be converted to the new style

Many of the existing handlers just drop into the debugger with an appropriate message, but they expect the sigcontext and stuff to be on the stack so they can dig into it for PC and so on. This is not the case for a call-behind handler: Something Must Be Done. I'm quite tempted to make them all call some internal interface and repurpose the existing enable-interrupt to do call-behind. User code won't (shouldn't) notice the difference, after all.

Removing CLONE_PARENT also means that (a) the parent thread has getting on for nothing at all to do whatsoever, and it might even be possible to get rid of it completely, and (b) that the system might get a bit easier to port to BSD. But apparently FreeBSD doesn't have POSIX RT signals, so still not actually possible.

That's not how it ended up, though. Faced with the prospect of doing lots of Unixy waitpid() stuff without the benefit of SB-GROVEL, I punted slightly. We do all that in the C handler, then call into Lisp to run handle-thread-exit which frobs session, then destroy_thread() runs from C. Much simpler, and doesn't involve interesting design work to

• copy the siginfo somewhere safe that call-behind handlers can find it
• decide what to do about signal masking during the execution of the Lisp handlers

Now committed in 0.8.6.11 anyway. CLONE_PARENT is no more.

SLIME and Araneida: screenshot here and announcement here

TODO:

• Some way to automatically set the optimization qualities in SBCL that lasts for longer than the current file. When the current file is .sbclrc, that's not a very long time
• Write up what the qualities do anyway somewhere (the manual would be a good place) because I can never remember.
• That OpenGL thing from the other day
• Another bug (I think it's a PFD thing, but) that I can't remember
• SLIME-over-networks (preferably sshified)

Another one for the neglected dialog box collection

Found while looking for other stuff: a rather literal xref port for SBCL

OK, in principle we have working SLIME over SSH, using the client side of attachtty/detachtty. We tell slime to attachtty hostname:/path/to/socket instead of connecting over the network, then we hack up swank a little to accept connections on a local socket instead of a TCP socket.

Why a local socket? To cater for the situation that the remote host may have more than one user. Access to unix sockets can be controlled by filesystem permissions, which saves us from having to come up with some authentication protocol in slime.

Still a fair amount of cleanup to be done (slime-disconnect tends to kill the lisp, which could be considered a Bad Thing), but I'll see if I can get committage some time soon.

Not impressed by: emacs apparently wiring together stdout and stderr of subprocesses even when you've said you want a pipe not a tty. Does this look silly to anyone else?

  (setq slime-net-process
        (let ((process-connection-type nil)) ; pipe
          (start-process-shell-command "SLIME Lisp" nil "attachtty" path "2>/dev/null")))

I am wondering if there is anyone anywhere who's using the define-page macro in Araneida. Revisiting its implementation now, it occurs to me that (a) it still uses the old handler model, (b) I don't understand how it works.

I think it might be good to set up a mailing list for Araneida/CLiki users, just so I have some chance of reaching more than three of them at a time

I have some kind of phone interview thing tomorrow morning, so I really should be asleep right now. But

• Small SLIME hackage so that the SBCL backend also works with Helmut's new code for choosing from a list when multiple definitions of a function are found. Right now this works for generic functions: M-. one of them and you get a list of all the methods

• Better support for conditional GET in Araneida. Now when you call request-send-headers it returns the HTTP response code it sent, and when you call it with a sensible :last-modified argument and :conditional t, it may turn your 200 response into a 304. In which case, you get to go home early without having to do that tedious database query.

• Using which, I managed to straighten out the conditional GET in Stargreen a little. Stargreen is an Araneida application but a very old one in parts - possibly dating right back to the time before Araneida was so named - and it uses bits of the system that I've forgotten exist and I strongly suspect nobody else has ever dared try.

Interview went not so well: feedback was that I was "wary of perl", which is somewhat embarrassing given that it's one of my three favourite programming languages. Need to work on that a little, perhaps. Hmm.

Added another LU&D article pointer to the metacircles site: this one was about CLiki. It's giving me ideas for an article on setting up SBCL to do web development with Araneida: "0 to (port) 80 in 3000 words".

Though, conceded, probably port 8000 if you're sensible. I would tend to avoid running SBCL as root.

Hopefully, you can see from these lists that Perl provides a rich set of interfaces to low-level operating system details. Why is this ``what Perl got right''?

It means that while Perl provides a decent high-level language for text wrangling and object-oriented programming, we can still get ``down in the dirt'' to precisely control, create, modify, manage, and maintain our systems and data.

There's a lesson here for CL, though I'm not sure what it is. I certainly don't buy the idea that POSIX APIs are the best possible interface - not that I think this is where Randal is coming from anyway: he's clearly arguing from pragmatism. Filename manipulation, for example, is usually done in Perl by a mess of regular expressions that probably don't even always work (does . match newline?) which I don't really want to wish on anyone. On the other hand, the convenience is undeniable.

(Aside: POSIX APIs are not even necessarily, the things that Perl actually provides access to: how long did it take Perl to get reliable signal handlers?)

To some extent, SB-POSIX and similar things (will) solve this problem for SBCL. We still have a certain impedance mismatch imposed by the language itself, though: for example, NIL is the only false value in CL; 0 and 0e0 and "" are true. That said, Perl isn't entirely restricted to the C standard datatypes either: ioctls often require more fiddling with pack() and unpack() than the mere mortal should entertain.

This blog has no comment-publishing facility: it's all done by hand. Feedback is welcome, though: if you would like to get in touch, email me on comment2010 @ telent . net and I will publish your thoughts (unless you tell me you don't want me to) as an addendum to the article, provided only that in my view they are interesting, amusing, relevant, accurate, or (best) some combination of the four. Relevant links are welcome in support of your argument: spam and untargeted requests for help are not.