6043 calling sub-gc 6040 stopping for gc 6041 deferred stop for gc 6041 calling sub-gc 6042 stopping for gc There was a terrible ghastly silence
Deadlock. Boom. I'm going to go to bed now, but I think in the morning, or reasonable analogue thereto, we can have this GC problem fixed.
OK, it looks like that was half the problem. Or, at least, one of the two problems.
Allocation is done in pseduo-atomic sections. When alloc() decides that it's time to GC, it uses the same deferred handler mechanism as an interrupt received during pseudo-atomic to schedule a collection as soon as the allocation itself is done. Problem is that it doesn't (didn't, anyway) check whether there was already a deferred handler to run, so the message that said "stop for another thread to gc us" got whapped by a message saying "now do a gc". It wouldn't break us to call gc from two threads at once - appropriate locking mechanisms are in place - but it doe hurt to not stop when people are waiting.
So, one down. The other one is that sometimes threads don't seem to wake up after gc, so after a few minutes of running, all our threads quietly come to rest waiting for a signal.
Earlier we asked "What would block SIGTRAP and SIGDEQUEUE?". wait-on-queue blocks SIGDEQUEUE temporarily while it frobs the waitqueue data before it can go to sleep. rundeferredhandler is called from the sigtrap_handler, and although we unblock the usual culprits before calling into Lisp, SIGTRAP is (along with SIGSEGV) not in that set.
I've added the good parts of this experimentation (without, I hope, the debugging cruft) to CVS under the tag atropos-branch. If you can deal with the shear abhorrence of all these signals, you're welcome to take a look.
It is clear why men and women have sexually dimorphic reproductive organs. But why did they evolve a sexually dimorphic digit ratio? Manning notes that it has been suggested that the male digit ratio pattern may be functional -- a longer ring finger may help to stabilize the third digit (the middle finger) when throwing objects, thus increasing throwing accuracy. This implies that the throwing accuracy required for successful hunting and/or tribal warfare was of sufficient importance to drive the evolution of this sexually dimorphic trait. While gathering, ancestral women presumably did not need this extra stability for the third finger. Today, this sex difference may be seen in male superiority in throwing darts. And, it would be interesting to know if men with lower digit ratios were better dart throwers and men with higher digit ratios.Michael Mills' review of Digit Ratio: A Pointer to Fertility, Behavior and Health by John T. Manning
Personally, I prefer to take this as evidence that guitar players are evolutionally superior.
OK, don't say I never ask you anything. If you'd like to find out why I keep rabbitting on about Lisp for web development (or possibly, you already know, and you'd like to help support one small part of the Common Lisp community), take a look at the cirCLe CD proposed product, and fill out the market research survey linked from it.
Some interesting comments on the cirCle CD already. I'll send out a mail to the announcement list in a week or so when it's a bit clearer what shape things are taking, but for the moment let's answer a few of the early questions here:
The votes for a simple ILISP setup are still coming in. Surprise surprise, yes.
I've been looking at Knoppix, and starting to learn enough about it to beat it into a shape I like. Knoppix is an ISO9660 image, which contains another, compressed, ISO filesystem. The inner filesystem is the root of the running system and contains about 2Gb of software. The outer filesystem gets mounted on /cdrom during the boot process and holds a few other files such as X root window images which are presumably put there for ease of alteration.
So, to hack it around you basically unpack the ISO, unpack the inner ISO, chroot into it and fiddle with dpkg/apt-get/whatever else, then pack it all back up again. There appear to be a number of graphical tools for this, but I'm using a shell script originally found at http://www.informatik.uni-koeln.de/ls_juenger/people/lange/software/mkmyknoppix but which is slowly mutating as I find other things I want it to do/not do.
The repacking phase takes about 20 minutes, which makes things a little tedious to test. Fortunately VMWare Workstation will boot from ISO images without needing to burn them first, so at least I don't need to add the CD writing time and cost of blank to that for every iteration. Unless Bochs can do something similar (and emulate svga and some species of ethernet device) I can see having to pay for this sometime in the next 30 days. Why is it that all the software I buy these days is for emulation?
So, chief concern right now is licensing; I want all the source code to be available in a GPL-compliant way (enough of the Knoppix packages are GPLed that it makes no sense to try and separate out the source-required packages). So far I've managed to get the filesystem down to around 770MB uncompressed (no KDE, no applications) or 280Mb compressed. If dpkg is not telling fibs, sources for this will be about 600MB: gzipped tarballs, gzipped diffs and .dsc files, so I don't think that's going to compress significantly further. If I can find really not a lot more space it looks like we might actually be able to get the whole thing ona single disc, which will make life a lot simpler
What's slightly worrying is that I just noticed a /usr/lib/j2se directory that dpkg -S doesn't acknowledge the existence of. On the bright side, I can probably just delete it - nothing important will ever depend on java - but I'm concerned that there may be other stuff in there that the package system doesnt know about.
Sourceforge is just so awesome.
There's a project called SLIME, the Superior Lisp Interaction Mode (Enhanced). It's existed for about a month or so, and reportedly seeing a lot of hacking. Observe this mailing list archive, for example. Note several posts per day, and obvious evidence of ongoing development.
Now see that it all stops suddenly on 29th September. What possible reason might there be for this? Why, it was due to inactivity, of course. Sourceforge knows all.
On further investigation it turns out to be a little more complicated. Back in 2000 there was an unrelated project on sourceforge called Slime, which the author at the time decided to delete. Three years later, after the project name had been taken over by the current crew, Sourceforge presumably decide that this would be a good time to implement the deletion.
I'm semi-impressed that the same member of Sourceforge staff can forget what's going on in the week between Luke Gorrie's request to restore it and mine asking where it had gone. Although, I guess, they deal with too many projects to remember, and if the interface that SF staff use to search their support database is the same one as we the public have to put up with, I can quite understand their not wanting to touch it any more than they can get away with. I wouldn't.
Did I mention that people are asking for good Lisp/Emacs integration? This is why I'm looking at SLIME for the cirCLe CD; the options available seem to be ILISP, ELI or SLIME, and SLIME. None of them actually work properly with right now, but the last has (a) real momentum behind it, (b) intentions to run on SBCL at some future date - unlike ELI, which will always be a Franz thing at heart, and (c) a refreshingly low amount of cruft.
And that's "work properly" which it doesn't, not "work at all", which it does. After writing the following entry I found a CVS root of Slime as it was ten days ago on Luke Gorrie's web site, so I spent a few hours hacking at it. By means of some really rather ugly copy/paste programming, I have increased by about 30% the amount of duplicated code in SLIME (ugly but inescapable; the most likely outcome of any serious refactoring I could do right now would be unresolvable merge conflicts) and got something for which the basic functionality runs in SBCL.
;;; Cursory testing has found that the following appear to work ;;; ;;; * Symbol completion. ;;; * Evaluation of forms with C-M-x. ;;; * Compilation of defuns with C-c C-c. ;;; * File compilation with C-c C-k (but error parsing is broken) ;;; * Disassembling the symbol at point with C-c M-d. ;;; * Describing symbol at point with C-c C-d. ;;; * Macroexpanding with C-c RET. ;;; ;;; Things that aren't done yet: ;;; ;;; * Cross-referencing. ;;; * Apropos. ;;; * Test suite has not been attempted yet ;;; * Basic debugger functionality, jumping to frames is not implemented yet. ;;; * Compiler warnings aren't currently sent to the Emacs stream
Hey, more of this works than I was expecting.
SLIME rocks. Messrs Sourceforge, please hurry up and restore the project, I want to commit to its cvs tree.
;;; Cursory testing has found that the following appear to work ;;; ;;; * Symbol completion. ;;; * Evaluation of forms with C-M-x ;;; * Apropos. ;;; * Compilation of defuns with C-c C-c ;;; * File compilation with C-c C-k, apparently including error parsing ;;; * Disassembling the symbol at point with C-c M-d ;;; * Describing symbol at point with C-c C-d ;;; * Macroexpanding with C-c RET ;;; * find-definition works some of the time (need to use sb-introspect) ;;; ;;; Things that aren't done yet: ;;; ;;; * Cross-referencing. ;;; * I don't understand the test suite results, but some kind of problem ;;; is apparent ;;; * Basic debugger functionality, jumping to frames is not implemented yet.
So I'm reading through the file and digging in SBCL internals, with the intention of creating a supported interface to at least some of the functionality it needs.
SBCL is creating a "supported" interface to meet the needs of Lisp/editor integrations software (ELI, ILISP, SLIME etc). This will be an SBCL contrib package containing functions like VALID-FUNCTION-NAME-P, FUNCTION-ARGLIST, FIND-DEFINITION, etcThis list is for discussion and requests from the likely users of this interface, and also for Lisp implementation hackers who would like to implement the same or similar interface in their Lisp implementation.
Run, don't walk.
Another SLIME screenshot, with my trademark gimped arrows to add gratuitous colour. I don't think it's actually doing anything that it wasn't last time, but more of it's moved into sb-introspect now.
Ok, maybe magic spacebar isn't that exciting. In fact, I turned it off in ILISP because it's nigh-impossible to recover from if it goes wrong - which, when it has to access internal functionality, it will tend to. That's not a problem any more, right? The emacs-like apropos is cute though - and not my work, I must emphasize. All I'm doing is the SBCL glue.
Right, so let's see what's with the debugger, then. And add a slime command to load asdf systems.
Very quickly then, because I'd really rather be asleep: today's SLIME picture shows the expanding backtrace thing. No wavey arrows, because this laptop has a trackpoint thing instead of a mouse, making it far too painful to attempt drawing anything that's not a straight line.
I believe there's enough substance to the SBCL backend now that I can start using it for day-to-day work - or at least attempting to, to see where it falls over. I would also give you the latest news from Sourceforge ("Monday"), but their web site seems to be down right now.
But this week I've begun asking myself why I hope things might change. Because it doesn't really mean anything to me any more; I'm not the cutting age of some kind of technological revolution, I'm just more roadkill on the information superhighway. And I don't need to swallow shit from a twenty-something drone in a corporate office churning out propaganda for the profit factories of Jim Alchin, Michael Dell, Andy Grove, or Carly Fiorina. I don't need the money half as much as I need my self-respect.
Well, it's no longer Monday, and needless to say, there is still no Slime on Sourceforge.
>> as I sit here continually failing to checkout SBCL because >> sourceforge's anonymous CVS is so broken, I wonder if our destiny >> lies elsewhere? > > Well, I see on their Site Status page that they should be done with > the new CVS infrastructure this week - so maybe it's not such a big > deal.I have heard that before, mind you ...
The good news is, in the very short term, we will have a solution that will solve this issue. We are upgrading our hardware from one aging system to six new speedy boxes; this hardware is a portion of the new CVS infrastructure that we are implementing to improve CVS performance and reliability.
The solution will be in place by the time I write you next month.
- SOURCEFORGE.NET UPDATE - 2003-07-24 EDITION
First, in the last sitewide email I described our CVS performance issues and the solution we are introducing to fix them. The performance increase I spoke of (600%+ increase) is just days away from being deployed.
- SOURCEFORGE.NET UPDATE - 2003-08-29 EDITION
As of right now, we are working around the clock to get these new boxes online to give relief to the performance issues. Our primary CVS server is copying 15 million files to these six new machines. Since this box is also serving CVS checkins and checkouts to the community simultaneously, the data transfer has been slow going. During the last 4 days we have transfered 40% of the data. We expect the rest of the data to be transfered by Thursday of next week, and the new boxes online shortly there after.
- SOURCEFORGE.NET UPDATE - 2003-09-30 EDITION
:; cal October 2003 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 <- that Thursday. Yes, the one just gone. 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
"I love deadlines. I love the whooshing sound they make as they go by"
No slime today, because I haven't been working on it. No slime on sourceforge, and no explanation for its continued missingness. No slime on savannah: they rejected the application because the cvs tarball didn't have the right number of copyright messages. Not that we could have added them after the project had been created, or anything. No, they definitely need to be in place at project creation time.
For the record, I'm not at all sure that the requirement to include a copy of the GPL is appropriate in this case. The cited reason is that the user may not have one already; however, we are talking about Emacs lisp here - slime would be pretty useless without Emacs, which package tends as a rule to come with a GPL copy of its own.
The good news is that I think I've fixed the longstanding cliki bug with page names that have % or . or + or similar exciting characters in them. No, don't go and test it, I haven't released the new version or upgraded any of the public sites yet.
We have SLIME CVS again, now at common-lisp.net
Sourceforge also finally managed to recreate the project, two minutes after Luke sent them the support request saying "we don't need this any more, we've taken it elsewhere".
As you have requested the removal of your project, your project has now been set aside in a queue to be purged of data. This purge cycle typically occurs every two to three months. In the mean time, CVS data, project web content, mailing lists, file releases and cron jobs related to your project will remain intact
Of course, the project having been recreated empty, there isn't a lot of "CVS data, project web content, mailing lists, file releases and cron jobs" there in the first place.
cirCe CD News update. And it's not too late to fill out the survey (or just to mail me) if you're interested and haven't already.
Quick SLIME note: the SBCL port now uses SERVE-EVENT instead of threads, which means it'll work on platforms other than x86 Linux. Or should do, anyway.
Helmut has done some rather neat stuff for finding function callers/callees in the CMUCL port. Next job (next SLIME job, I mean) is to port that and put it in SB-INTROSPECT.
Picked up Neal Stephenson's Quicksilver in Borders this evening (link to Amazon.co.uk because the official site is some useless popup thing). I was moderately amused to find it in the Fiction section, not the SF section with all his previous works - yes, including the Steven Bury stuff. Immediate impression after 50 pages: his editor should have said a few words about the purple prose. After 100 pages either it's settled down a bit or I've got used to it, though.
This space reserved for a rant about Knoppix; specifically, how in the hell is anybody supposed to reliably find all the source code for the packages it's composed of - especially when it's based on Debian testing given that (as far as I know) there's no requirement on a Debian mirror to keep all previous versions of testing around indefinitely in case Knoppix users want the sources they're entitled to. Given how much of this stuff is GPLed, the commercial distributors (if not Knopper himself) are going to have a pretty unpleasant time of it when RMS finds out.
apt-get on the filesystem which will one day be the cirCLe CD (rather, disc 1 of the two disc circle cd set) has just decided it wants to upgrade over 100Mb of stuff. So, whether it will still actually work when it's burnt to CD, who knows?
With few exceptions, my language in this diary has tended to be more measured and less crude than, say, in real life. Or on irc (yes, yes, I know that's not the same thing). So, I'll just say I'm seriously investigating non-Knoppix options for a live Debian cd (currently looking at debootstrap, mkinitrd-cd, and discover) and leave you to fill in the blanks.
While I'm in this "if it moves, curse it" state of mind, I may as well give a shout out to vmware. I press ^L in less and it spends a minute repainting the screen: it's slower than windows telnet. I think the guest system may be running in some silly framebuffer mode, which is probably not helping.
More obviously silly is their decision to supply the vmware tools only as a fake cdrom, so that you can't install them in a guest that's running a live cd image, because it always installs the cd with the tools as the first cdrom device.
Knoppix anatomy in 60 seconds. You are not expected to remember this.
Recipe for a happy life
This is the rough order I did things in. The end result is Makefile-driven and completely repeatable (except for some questions from debconf that I haven't provided answers for yet, so are still being asked interactively)
Right now I am staring at a freshly booted (in vmware) cd image. It seems to have found my ethernet card, sound (not that Common Music is on the shortlist for cirCLe just yet, but eventually this'll be fun to play with), network, and usb card. If I run my cobbled together script for X configuration, I have a working X server. If I run my cobbled together script for ethernet configuration, it runs a dhcp client. I'm not overly impressed by etherconf, though. I think something ought to be able to dhcp first and set the default hostname/domain name based on what comes back. It's been years since I last ran a debian install from scratch, but I'm pretty sure it did this.
Next, write proper init.d scripts to do this detection, start X, run a session/window manager, and stuff. Next also, replace some more of the remaining bits of Knoppix: at the moment, we're copying /lib/modules straight out of the knoppix cloop fs, which is kind of cheesy - would much prefer to have (or at least know) the sources that correspond to our installed kernel.
Was this all a good idea? I think so, probably. We've lost some of the fabled Knoppix detect-all-hardware-on-the-planet juice (no support for isdn or pppoe any longer, nor for graphics cards that XFree86 4.2 doesn't support), but the whole thing is a lot more understandable, and (after we get our own kernel) runs from a Makefile with no need to start by untarring someone else's ISO image and changing it incrementally.
I'm testing bits of the cirCLe CD using Xnest and a chroot - it's a lot less painful than rebooting, and even somewhat less painful than compressing iso9660 images all the time to run vmware on. I was mildly amused to find just now that Xnest has a screen saver
cirCLe CD update, for everyone wondering. I'll probably make an announcement on the mailing list this week as well.
Having decided to ditch Knoppix I find that I can get a working Linux (running X, apache, sawfish, mozilla-firebird and emacs, which is more or less what's needed) into ~120Mb after compression. The sources for this lot weigh in at around 380Mb, so it's looking for the moment as if I can get the whole thing on a single disc. Some sacrifices have had to be made in the process: we autodetect your ethernet card, sound card (not that we use it for anything), mouse, video, etc, but if you have some more exotic hardware (usb cablemodem, isdn, pppoe thingy) we don't support that out of the box. I'd like to add support for wireless ethernet at least; we'll see how it goes.
Contrary to the initial announcement, it will use SLIME, not ILISP.
If you install it on a real system (as I hope that most serious users eventually will), the package names are chosen to avoid conflict with existing packages (unless you have existing packages whose names start circle-, in which case you still face problems), and it all goes into /opt/circle-2003-1, so in the worst case you can still dispose of it pretty easily with rm. There are actually only a couple of "vendor" packages in there: most of it is installed from source with asdf-install.
I've been playing with Linux USB storage support recently, and although it works most of the time I'm not altogether confident about its stability (yesterday's highlight: I managed to get a Zip disk and a USB floppy both pretending to be scsi1 at the same time). There will be some way from the live CD to mount existing filesystems and save work/preferences/settings, but depending on your hardware, the phase of the moon, etc, you may have to approach this with some degree of caution.
Current status: CD works and boots on my hardware. Packages created for sbcl, araneida, cl-ppcre, cliki, clx, detachtty, and split-sequence. None of them tested (some of them known to break, in fact). An optimistic timetable would be "end of November", a more realistic one would be "2004".
Oh, for the curious, I stopped using xnest after attempting to start it over a non-local connection. Now using VNC instead. This is basically for my own development purposes, but I'm very much tempted to leave it in on the finished CD (in a completely unsupported and undocumented fashion) just because I think it's kind of neat. All sotware has Easter eggs, right?
One of the things that makes programming easy is Hyperspec lookup from within emacs. One thing that makes it a bit nicer yet, on this underpowered-by-today's-standards laptop, is a lightweight web browser. Some people like one of those emacs browsers, but I've never been able to get used to them, so I use dillo. Here's a kludgey piece of elisp to make it use dillo for hyperspec lookup and stick with a "normal" browser (like Mozilla Firebird) for other links where javascript or SSL or what-have-you may be needed:
(defun browse-url-dillo (url &optional new-window) ;; new-window ignored "Ask Dillo to load URL. Default to the URL around or before point. Always starts a new instance of Dillo, ignoring new-windonw argument" (interactive (browse-url-interactive-arg "Dillo URL: ")) (start-process (concat "dillo:" url) nil (executable-find "dillo") url))(defadvice common-lisp-hyperspec (around wrap-dillo 0 (symbol) activate) (let ((browse-url-browser-function #'browse-url-dillo)) ad-do-it))
