More cirCLe CD news, or lack thereof: spent most of the week getting#
Sun, 02 Nov 2003 15:00:57 +0000
More cirCLe CD news, or lack thereof: spent most of the week getting
to grips with the Debian kernel packaging stuff, and building initrds,
and El Torito boot images. Then spent most of the weekend so far
writing up what I'd done (look out for forthcoming Linux Journal
article, with any luck).
The simplest way to do moderately sensible removable media support
seems to be a combination of hotplug (to notice when new devices
appear/disappear), supermount (to notice when/whether these devices
actually contain disks) and xfe: a file manager which doesn't require
a whole desktop infrastructure to be present before it will so much as
run. I remember using supermount an awful long time ago (I guess it
would have been about 1995, when I still used floppies for anything): I'm
amazed it still exists and is available for current kernels. Kudos
to Stephen Tweedie and all the hackers who've worked on it since then.
I don't often use a file manager: a shell with tab completion does
most of what I want to do with files. After playing with xfe for a
few minutes to see if it works, though, I notice my home directory is
significantly less cluttered with random files I'll never use again.
Perhaps I should use a file manager.
I have had enough, for the time being, of USB hotplug#
Thu, 06 Nov 2003 04:51:48 +0000
I have had enough, for the time being, of USB hotplug. I cobbled
together a plausible though grotesquely kludgey way to do automounting
even in the presence of partitioned devices: it's an autofs "program"
map which when asked for the device "list" scans /proc/partitions,
assuming that /dev/[sh]dan is a partition of [sh]da, and populates
subdirectories of symlinks to the actual mountable devices. See, I
told you I was ill.
No, the good news is that it's time to put another release of
araneida together. The feature that people are actually wanting is
that this one will compile cleanly on current SBCL (so you can stop
sending me bug reports about sb-unix::sigpipe vs
:sigpipe), but the feature I will be foisting on them
additionally is that the rather strange export-server thing is
gone, along with another few global variables. These days instead you
can create an http-listener (which may be threaded or may use
serve-event) which does the actual listening-on-a-port business and
has a root-handler slot containing the handler to which it dispatches
requests. We're also going to be handling reverse-proxied requests a
little more correctly: it should now be possible for the front-end box
to be on a different hostname as well as a different port number,
which wasn't previously the case if my reading of the code is correct
(i.e. if it did work it was a coincidence). How it will work is that
you install your existing handlers as if you were using the external
host, then add a `internal-redirect-handler' (which I haven't thought
of a sensible name for yet, much less implemented) for requests to the
internal URL: this creates a child request with a suitably rewritten
url and dispatches that.
When will it be released? When I've done the redirect handler and
ideally also decided where to hang the ssl key/certificate information
that used previously to be on the server object.
I have spam filtering again, and this time it's filtered server-side#
Fri, 07 Nov 2003 14:09:40 +0000
I have spam filtering again, and this time it's filtered server-side.
This makes me happy, or at least less readily depressed when I get up
and see 200 new mails of which approximately 5 are at all likely to be
interesting to me.
My mail is delivered to a shell host using SMTP. I pull it onto my
laptop using fetchmail and "ssh shell /usr/sbin/imapd" when I want to
read it. Then emacs Gnus splits it into folders for me. I like this
setup: I have all my mail archives to hand locally when I'm away from
the net, and if I have some form of shell access but no way to get my
laptop on the net I can still read new mail using mutt on the server.
So I added procmail on the server to pipe through spambayes then
filter spam into its own folder, then added a line in fetchmailrc on
the client to download that folder (not automatically, but just in
case) then stuck two new keys in gnus to retrain spambayes when it
misses something - by, uh, ssh into the server host and spitting the
article contents into sb_filter.py. Yeah, I know, it's tacky.
(defun spambayes-train-spam (&optional arg)
(interactive (gnus-interactive "P"))
(let ((gnus-default-article-saver
(lambda (x)
(gnus-summary-save-in-pipe
"ssh eval.metacircles.com /home/dan/python/bin/sb_filter.py -s -f")))
(gnus-save-all-headers t))
(gnus-summary-save-article arg t)))
As is common with any kind of Unix mail setup stuff there are seven
zillion possible different ways to do it and I tried several before
settling on this one. Most of the wasted time was in trying to get
Gnus to do disconnected IMAP support, which was basically pain. Some
of it was installing Python from source three times (the debian stable
packages are too old to run spambayes): the first two builds were
missing dbm and zlib respectively owing to missing header files.
It seems to be working pretty well, so far, and should definitely
makes mornings a bit less insta-bad-mood.
Neil Gaiman being asked about his Duran Duran biography in #
Sat, 08 Nov 2003 05:00:13 +0000
Neil Gaiman being asked about his Duran Duran biography in
an interview:
Whenever I do things because I want to do it and because it seems fun
or interesting and so on and so forth, it almost always works. And it
almost always winds up more than paying for itself. Whenever I do
things for the money, not only does it prove a headache and a pain in
the neck and come with all sorts of awful things attached, but I
normally don't wind up getting the money, either. So, after a while,
you do sort of start to learn [to] just forget about the things where
people come to you and dangle huge wads of cash in front of you. Go
for the one that seems interesting because, even if it all falls
apart, you've got something interesting out of it. Whereas, the other
way, you normally wind up getting absolutely nothing out of it.
Of course, it works better if what you want to do is write books
like American Gods than if your idea of "fun and interesting" is, say,
staying in bed until mid-afternoon. Or possibly even hacking free
Lisp software.
New Araneida finally out of the door#
Mon, 10 Nov 2003 05:25:22 +0000
New Araneida finally out of the door.
Bcc: dan
To: lispwebred-bean.com
Subject: ANNOUNCE: release Araneida 0.80
X-Draft-From: ("nnml+private:mail.lists.sbcl-devel" 3030)
From: Daniel Barlow telent.net>
Date: Mon, 10 Nov 2003 05:25:01 +0000
Message-ID: <874qxckd76.fsf@noetbook.telent.net>
--text follows this line--
<#secure method=pgpmime mode=sign>
Araneida 0.80 has been released to ftp.linux.org.uk and is now making
its way to whichever CCLAN mirrors are actually still maintained.
From the release notes:
- New HTTP-LISTENER abstraction drastically improves the threaded
variant of Araneida, and also replaces a lot of older ad-hocky code
and global variables. The following are at least obsolete, and may
not even exist any more
- The 'server' stuff: export-server etc
- root-handler
- install-{thread,serve-event}-handlers, remove-handlers
- Raising HTTP errors from handlers is now accomplished in a far
cleaner fashion, by signalling CL conditions. (This also means that
parent handlers may handle these conditions to provide e.g. custom
404 pages). A condition exists for each 4xx and 5xx HTTP response code
- Add REFRESH header to REQUEST-SEND-HEADERS; sends the not-quite-HTTP
'Refresh: ' header (really, not part of HTTP/1.0 or 1.1: it appears to
have been a Netscape extension)
- Fix packaging problem in 0.72
- ... and even compiles (in fact, only compiles) in current SBCL versions
This version of Araneida requires source code changes in your
application - although not terribly pervasive ones; unless there are
vastly more Araneida users than I'm aware of, I felt it would be
quicker for users to update their configuration than for me to provice
legacy compatibility routines. There is a complete and working
example of Araneida configuration in doc/example.lisp: it should be
about three lines for most people.
Feedback appreciated.
-dan
--
http://web.metacircles.com/cirCLe_CD - Free Software Lisp/Linux distro
Yay. I can probably go to bed
now.
Just back from listening to Neil#
Mon, 10 Nov 2003 21:01:31 +0000
Just back from listening to Neil
Gaiman read at Borders. The event listing on his web site didn't
mention signing, so I foolishly assumed there would be none; when I left
the event there was a queue of better-prepared people halfway across
the floor.
Fun, though. He read "Wolves in the Walls", then answered
questions.
"
Where do I get my ideas from? Confluence. You should be
very careful asking that question, because 90% of authors will make up
a joke answer". Favourite characters: Delirium (Sandman), Crowley -
or possibly Death (Good Omens), Wednesday (American Gods). No, there
probably won't be another Pratchett collaboration. His Babylon 5
episode was pretty much entirely his own work. And his sinister beard makes him look, well, sinister. But also kind of
cool (as if he didn't already).
So I finally upgraded my desktop to 2.6 (2.6.0-test9), at least, after#
Thu, 13 Nov 2003 21:19:24 +0000
So I finally upgraded my desktop to 2.6 (2.6.0-test9), at least, after
fiddling a bit so that it's still using the old OSS-style sound module
instead of spiffy new Alsa. This is because spiffy new Alsa doesn't
work, as far as I can tell: there seems to be no way to persuade my
motherboard to put the video card (Voodoo3) and the onboard audio in
different interrupts: ye olde technologie trident.o can cope with
sharing, but ALSA snd-trident makes some really really evil noises
every time I select a menu item in Mozilla. Anyway, that's an
exploration for another day.
What this means immediately is that it's time to play with futexes.
Kernel interfaces vs Glibc/POSIX: discuss#
Fri, 14 Nov 2003 13:05:11 +0000
Kernel interfaces vs Glibc/POSIX: discuss.
One thing that working on SBCL threads has really brought home to
me is that there is sometimes much better taste in API design in the
Linux kernel than there is in the POSIX standards that the GNU/Linux
project as a whole seeks to follow.
clone() vs POSIX threads is one example: either I can have this
nice clean interface where the receiver of a signal is the person who
asked for it or otherwise induced it, or I can have some weird
monstrosity where signals are sent randomly to any thread that hasn't
blocked them. But I'm sure I've talked about that before, at the
UKUUG conference if nowhere else.
I think I'm finding similar issues in futexes. The system call
support for futexes is, at base, really really simple: you have
- one operation that tests whether some word in memory is a given
value, and goes to sleep in a queue in the kernel if so.
- another operation that wakes some specified number of tasks
which are waiting on that queue.
But then, you have the "approved" method of accessing it from
userspace (futex(4)): basically a counting semaphore which uses these
syscalls when the lock is contended. But we didn't want a
counting semaphore. And even that you're not supposed to use
directly, instead relying on your trusty NPTL implementation - which
means FFI bindings to pthread_* functions in glibc, which means
function call overhead - even in the uncontended case - if they really
are functions, and screwing around with glue code if as I suspect half
of them turn out to be inlined or macros.
So, considered in isolation, I think good taste requires that I
call sysfutex() directly (modulo that I'm not sure it didn't get
deprecated and replaced with a separate futex* call for each
different operation sometime in 2.5). Considered in the context of
programs that do FFI to libraries that expect the full layered NPTL
experience, though, will we interoperate?
At root the question is: are we building a Lisp that runs on a
Linux system, or a Lisp that runs on a GNU/Linux system? It's ironic
that I can use Richard Stallman's nomenclature to describe the
difference: the Linux system that he's not involved with seems pretty
language-agnostic, but the GNU system, which was originally
created by this Lisp advocate and former Lisp Machine hacker,
seems to only really like C programmers. Good luck finding the value
of SIGRTMIN in any other language, anyway. (Yes, the Glibc
behaviour described there is certaintly allowed by POSIX. It's not
particularly helpful and I don't think it really holds up against the
Principle of Least Surprise, but I can't argue that the SUS says "integral expression" and not something more helpful
like "constant expression")
Last time I was seriously immersed in a looking-for-jobs frame of#
Mon, 17 Nov 2003 22:46:23 +0000
Last time I was seriously immersed in a looking-for-jobs frame of
reference was a few years ago, when things were different. These days the job
ads I see are mostly from agencies, and don't offer a lot of
possibility for purposeful parody that's in any way distinguishable
from actual real advertisements.
Looks like it's a day for linking to stuff I wrote already#
Tue, 18 Nov 2003 15:23:44 +0000
Looks like it's a day for linking to stuff I wrote already. So,
here's my opinion on ESR's "Art of Unix Programming".
4
I should say a few words about futexes#
Wed, 19 Nov 2003 12:30:43 +0000
I should say a few words about futexes. Adding futex support to sbcl
(using sys_futex, counting semaphores be damned) was mostly a
straightforward job taking a couple of afternoons.
Unfortunately, it seems to have exposed a bug somewhere in GC
handling or stopping-the-world issues around GC. GC itself is wrapped
in a without-interrupts, which uses a mechanism very similar to
pseudo-atomic to delay incoming signals: when the cleanup from
without-interrupts detects that a signal was supposed to have
happened, it calls int 3, and the SIGTRAP handler then runs
the handler for the stored signal. At least, that's how it's supposed
to work. Right now it looks like the kernel has decided that the
SIGTRAP can't be delivered to the process in the normal fashion and
instead it would be a good idea to SIGKILL all of our threads. This
is probably some kind of stack corruption. Attaching a debugger to
the right thread in the right place may be interesting.
I'm not sure exactly what to write here, but I feel an urge to whinge#
Thu, 20 Nov 2003 18:33:01 +0000
I'm not sure exactly what to write here, but I feel an urge to whinge
a bit about the IT job market in the UK right now. Translations:
If they say Unix, they mean enterprise or ISP
sysadmin experience: Veritas, EMC, clustering, etc.
If they say C, they mean embedded or they mean C++.
If they say OOD, they also mean C++.
If they say threads, they mean pthreads.
If they say Java, they mean J2EE.
If they say Perl, they mean entry level.
I don't think anyone has said Lisp yet, but when they do,
they mean AI.
Do I sound bitter? Very well, then, I sound bitter.
Quick note: the futex problem is not actually futex-related#
Sat, 22 Nov 2003 18:54:37 +0000
Quick note: the futex problem is not actually futex-related. Linux
2.6 behaves differently to 2.4 in what it does when int 3
happens while a SIGTRAP is masked.
Quick note 2: Araneida 0.81 (bug fixes only) to start happening
tonight when I get back from dinner. I need a better test suite for
that thing.
Quick followup re 2.4/2.6#
Sun, 23 Nov 2003 01:49:43 +0000
Quick followup re 2.4/2.6
SIGTRAP difference (another reason the kernel is better than
glibc: when you ask questions you actually get replies from
people working on it)
The cirCLe CD is on hold pending a rethink#
Tue, 25 Nov 2003 15:37:44 +0000
The cirCLe CD is on hold pending a rethink. Response from the
survey showed lots (well, some few tens) of existing CL users who
thought it would be neat, plus a few people who might want to try it
for curiosity's sake (personal favourite: the person who wanted a
Genera clone for $25) but neither of these is sufficient evidence of a
sustainable market. Maybe I'm asking the wrong questions; almost
certainly I'm asking them in the wrong place.
So, Marketing 101: products have features but customers
actually want benefits. (Unless, for some reason, the feature
in question is XML, but I digress). So, what are the benefits
of Lisp? What's our USP?
I think the answer is interactive development: in particular
for Web application development, where the usual state of the art
seems to be either error reporting on the web page itself (which is a
start, provided you can turn it off in production - do you want your
customers reading your backtraces?) or dumping stuff in a log file.
So I've been experimenting with SLIME and Araneida with the intention
that (a) errors in Araneida handlers can be debugged using the Slime
debugger, (b) Slime can connect across the network (SSH-tunnelled) to
a long-lived server process. Some progress to report on (a): it
works, but nothing is in CVS yet
Also today, set up a Paypal account: dan@metacircles.com. Not
altogether sure what I'm hoping that people will send me money for: if
you've solved a problem (or just enjoyed) using CLiki or SBCL
(threads, sockets, CLX, Alpha, PPC) or Araneida or cclan or asdf or
detachtty or ... and want to fund further development, now you have a
mechanism. It might also be a good idea if you sent me email saying
which of these you're using: not that your $40 is going to
significantly influence my development whim, but in the aggregate your
collective $4000 might.
Thanks, by the way, go to Erik Enge for renewing the cliki.net
domain when i managed to miss the reminders for it.
Today, in Oxford#
Wed, 26 Nov 2003 22:10:18 +0000
Today, in Oxford
metacircles is now
offering per-incident
paid-for support for SBCL and related software. Presently this
has less to do with actual customer demand and more to do with
distracting the people who point to lack of commercial support as a
reason not to use free CL predicted customer demand.
There is a fix in SBCL CVS sb-bsd-sockets for the
writing-to-closed-files bug that SLIME users sometimes experience.
Last week, Frank Neubüser found the notes I made last year on
Oracle Wallet
Manager. By spooky co-incidence, I had occasion on Monday to
revisit this experience and revise my notes in that light: you can now
find the new version (and Frank's email) over at
metacircles
OK, more content on web.metacircles.com: Scripting Emacs#
Thu, 27 Nov 2003 04:34:58 +0000
OK, more content on web.metacircles.com: Scripting Emacs
is an article I wrote for Linux User & Developer magazine last
spring (published last summer, as is the way with magazine lead
times). The "& Developer" part of their magazine doesn't get
as far as their web site, so here's a small bit of it on mine.
I have written other stuff for them too, which I might post sooner
or later. That said, the CLiki article is probably out of date, and
the SBCL threads article doesn't tell you anything that reading the
paper from my UKUUG presentation wouldn't. Of course it, too, is out of date.
That's the price of progress. Or something.
Just spotted this on lisp-p.org: #
Thu, 27 Nov 2003 16:16:36 +0000
Just spotted this on lisp-p.org:
Thoughts about Game & Simulation Programming Architecture.
haven't read it yet, it's in the queue
Preceding it in said queue:
- make QUIT work sensibly when background threads exist
- have threads removed from the interactive-threads list when
they die
- investigate some problem with OpenGL bindings
I recently - after only five years ownership of this Voodoo3 - did
the necessary fiddling to get hardware-accelerated 3d (turns out that
I seem to be mostly immune to Tuxracer) so that OpenGL thing looks
like it could be interesting.
So the latest released version of CLiki doesn't work in current SBCL,#
Fri, 28 Nov 2003 04:37:17 +0000
So the latest released version of CLiki doesn't work in current SBCL,
and the instructions are out of date w.r.t the current Araneida.
Planning to release a new version which fixes both these problems
imminently (as in, probably about Monday).
There's also a ton of other changes, which is why I'm preannouncing
it. Anyone who wants to play with collaborative content editing this
weekend is invited to check it out from CVS: instructions at http://cvs.telent.net/. There's a
new example.lisp file which should get you up and running with minimum
effort: I'm pretty sure CLiki gets easier to install with every new
version.
Bug reports, suggestions, queries welcome. From the NEWS file -
New in CLiki 0.4.1
- Update README file to correspond to Araneida 0.81, and add
example.lisp demonstrating a simple CLiki installation
- Change CLIKI-INSTANCE class graph. Now we have CLIKI-VIEW as a parent
of CLIKI-INSTANCE. The approximate division is that CLIKI-VIEW deals
with the visual aspects and CLIKI-INSTANCE deals with updating and
saving data
- CLIKI-SKIN is a new CLIKI-VIEW subclass demonstrating a way to make
a second look and feel for an existing cliki.
- New CLIKI-PAGE-SURROUND method replaces CLIKI-PAGE-HEADER and
CLIKI-PAGE-FOOTER. WITH-PAGE-SURROUND macro allows any response
handler to reasonably conveniently use the standard surround for
some cliki it's associated with
- New class AUTHED-CLIKI does cookie-based authentication for clikis that
you don't want just anyone to edit. Badly, though. Feel free to send
patches
- At last, proper escaping of page pathnames. Page titles may now contain
interesting characters like #\#, #\., #\+ etc
If you have any pages stored in files with even mildly interesting
names, they will change: what was "Foo Bar" is now "Foo=20Bar", for
example. The old names are still read, and updated to new names on
SAVE-PAGE (i.e. after edits)
- #\Space and #\_ are treated as equivalent in page names, just as #\A
and #\a are.
- SAVE-PAGE is now a GF specialkised on CLIKI-VIEW. Might be usable
for e.g. database-backed clikis.
- Bug fix: admin/recent-changes.dat is created if not found
I also fixed the QUIT behaviour of SBCL to be ... well, different
again. This probably broke sb-aclrepl and anyone else creating new
sessions (ELI, I think), but the good news is that they can replace
their old crufty code with new slightly less crufty code.
So, reviewing how far I got today, I realise that I didn't make it as#
Fri, 28 Nov 2003 04:47:07 +0000
So, reviewing how far I got today, I realise that I didn't make it as
far as "remove threads from the session on exit". We need some way of
notifying the thread that created the exiting thread that its child
exited, then it needs to clean up. Briefly, then
- Remove CLONEPARENT from clone() flags. We don't need it now
we're not doing stop-for-GC with ptrace(). Add instead an RT signal
called, say, SIGTHREAD_EXIT, so that our thread death signals queue
up when lots of children exit at once.
- Add a C handler for thread exit, which does normal pseudo-atomic
stalling then calls a Lisp handler.
- Write the Lisp handler to do appropriate surgery on SESSION and
foreign-call destroy_thread.
Since we're here, we may as well make the new handlers call-behind.
Since we're doing that, we may as well add a general Lisp interface
for establishing call-behind handlers, and have a look at the existing
handlers to see what could be converted to the new style
Many of the existing handlers just drop into the debugger with an
appropriate message, but they expect the sigcontext and stuff to be on
the stack so they can dig into it for PC and so on. This is not the
case for a call-behind handler: Something Must Be Done. I'm quite
tempted to make them all call some internal interface and repurpose
the existing enable-interrupt to do call-behind. User code
won't (shouldn't) notice the difference, after all.
Removing CLONE_PARENT also means that (a) the parent thread has
getting on for nothing at all to do whatsoever, and it might even be
possible to get rid of it completely, and (b) that the system
might get a bit easier to port to BSD. But apparently FreeBSD doesn't
have POSIX RT signals, so still not actually possible.
That's not how it ended up, though#
Fri, 28 Nov 2003 16:01:39 +0000
That's not how it ended up, though. Faced with the prospect of doing
lots of Unixy waitpid() stuff without the benefit of SB-GROVEL, I
punted slightly. We do all that in the C handler, then call into Lisp
to run handle-thread-exit which frobs session, then
destroy_thread() runs from C. Much simpler, and doesn't involve
interesting design work to
- copy the siginfo somewhere safe that call-behind handlers can
find it
- decide what to do about signal masking during the execution of
the Lisp handlers
Now committed in 0.8.6.11 anyway. CLONE_PARENT is no more.
SLIME and Araneida: screenshot here and #
Sat, 29 Nov 2003 22:39:28 +0000
SLIME and Araneida: screenshot here and
announcement
here
TODO:#
Sun, 30 Nov 2003 00:59:23 +0000
TODO:
- Some way to automatically set the optimization qualities in SBCL
that lasts for longer than the current file. When the current file is
.sbclrc, that's not a very long time
- Write up what the qualities do anyway somewhere (the manual would
be a good place) because I can never remember.
- That OpenGL thing from the other day
- Another bug (I think it's a PFD thing, but) that I can't remember
- SLIME-over-networks
(preferably sshified)
Another one for the neglected dialog box#
Sun, 30 Nov 2003 01:48:46 +0000
Another one for the neglected dialog box
collection
Found while looking for other stuff: #
Sun, 30 Nov 2003 03:29:09 +0000
Found while looking for other stuff:
a rather literal xref port for SBCL
OK, in principle we have working SLIME#
Sun, 30 Nov 2003 16:36:46 +0000
OK, in principle we have working SLIME
over SSH, using the client side of attachtty/detachtty. We tell
slime to attachtty hostname:/path/to/socket instead of
connecting over the network, then we hack up swank a little to accept
connections on a local socket instead of a TCP socket.
Why a local socket? To cater for the situation that the remote
host may have more than one user. Access to unix sockets can be
controlled by filesystem permissions, which saves us from having to
come up with some authentication protocol in slime.
Still a fair amount of cleanup to be done (slime-disconnect tends
to kill the lisp, which could be considered a Bad Thing), but I'll see
if I can get committage some time soon.
Not impressed by: emacs apparently wiring together stdout and
stderr of subprocesses even when you've said you want a pipe not a tty.
Does this look silly to anyone else?
(setq slime-net-process
(let ((process-connection-type nil)) ; pipe
(start-process-shell-command "SLIME Lisp" nil "attachtty" path "2>/dev/null")))
I am wondering if there is anyone anywhere who's using the define-page#
Mon, 01 Dec 2003 17:29:47 +0000
I am wondering if there is anyone anywhere who's using the define-page
macro in Araneida. Revisiting its implementation now, it occurs to me
that (a) it still uses the old handler model, (b) I don't understand
how it works.
I think it might be good to set up a mailing list for
Araneida/CLiki users, just so I have some chance of reaching more than
three of them at a time