I'm hacking threads for SBCL AMD64#

Sun Jan 30 00:19:50 2005

I'm hacking threads for SBCL AMD64. I've only just started on it, but there a couple of things I think I've decided on:

• The AMD64 sort of has segment registers insofar as %fs and %gs still exist. However, glibc is using %fs for its tls, and %gs is reportedly slower because the kernel uses it for something. So, as we have some GPRs these days, let's use one of them as base address for our thread data (specials &c)

• The original decision to use clone() instead of pthreads was largely made on not on any objection to the pthreads interfaces, but more on the basis that LinuxThreads (the then-current implementation of same) is/was pretty bloody awful. We no longer have to care about that, mercifully: NPTL is supposed to be actually quite good. So, let's see if we can use standard(ish) interfaces where appropriate.

Of course, there's still a big gap between "uses some pthreads interfaces" and "runs everywhere that pthreads does". Purists will point out that the pthread_t may be a non-integral type (e.g. a structure) and we shouldn't be comparing them with each other; I will respond by pointing out that we're still using POSIX RT signals, and (at least for the moment, until I can work out whether the pthreads locks are adequate) futexes. Which probably ties it down more effectively portability-wise than any weird architecture that uses a struct for pthread_t.

So far after an evening's hacking it compiles host-1 and target-1, then bails out very early in host-2 as I still haven't done any of the lisp stuff: need VOPs for set, bind, unbind, symbol-value etc.

Oh, before I forget: if you are running a highly loaded site on Araneida with a mod_proxy in front of it,

ProxyIOBufferSize 131072

(where 131072 comfortably exceeds the size of your largest likely response) seems to make a quite impressive difference to its speed/stability.

Hot property#

Wed Feb 2 10:45:33 2005

Hot property

From: Daniel Barlow <dan@telent.net>
Subject: [OxLUG] GPG key revocation

For anyone who'd signed my GPG key, please be aware that my laptop
was stolen at the weekend, so don't trust it any more.  The revocation
cert below has been sent to the keyservers.

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.2.5 (GNU/Linux)  
Comment: A revocation certificate should follow

iHAEIBECADAFAkIAkncpHQJsYXB0b3AgY29udGFpbmluZyBzZWNyZXQga2V5IHdh
cyBzdG9sZW4ACgkQHDK5ZnWQiRP18QCfZNMnZyOzdKon3mRKjt6EBn8WpwEAniE/
9qsy51ZzLlHbFyKellFpMas1
=U0GC
-----END PGP PUBLIC KEY BLOCK-----

For information, I claim that my new key is

:; gpg --fingerprint 1297D4B5    
gpg: NOTE: old default options file `/home/dan/.gnupg/options' ignored
pub  1024D/1297D4B5 2005-02-02 Daniel Barlow <dan@telent.net>
     Key fingerprint = 13D7 02AD 8665 27E0 8F71  5326 776F D7C0 1297 D4B5
sub  2048g/DE0A0FD9 2005-02-02 [expires: 2010-02-01]

but of course I don't expect you to believe that without the whole
face-to-face authentication hassle, so ...
[...]

If you see me, feel free to ask for a copy of my fingerprint

There has also been some AMD64/pthreads progress, btw#

Wed Feb 2 10:53:05 2005

There has also been some AMD64/pthreads progress, btw. It now gets as far as creating a core, and dies early in cold-init with some `4 is not a list' error that I haven't fully figured out yet.

MORE PTHREADS#

Mon Feb 7 11:50:14 2005

MORE PTHREADS. sbcl-devel subscribers will already have seen the current state - it appears to work superficially but there are still holes. Most obviously right now, there's some evil recursive bustup in internal-error support, so any attempt to run an undefined function will cause SBCL to come to a short and sticky end.

The previous `4 is not a list' was caused by a literal 4 that should have been an 8 - or probably even an n-word-bytes, and then we found that assembly trampolines must be 8 byte aligned so that their addresses look like fixnums to the GC.

NO MORE EVIL RECURSIVE BUSTUP#

Tue Feb 8 00:38:33 2005

NO MORE EVIL RECURSIVE BUSTUP. Problem was due to calling pthread_getspecific in call_into_lisp without first saving some registers it wanted to trash. Juho helpfully pointed out another crappy-editing-induced potential bug, too. Obviously it's bedtime.

I don't think I've mentioned skating yet all year#

Tue Feb 8 00:58:28 2005

I don't think I've mentioned skating yet all year. Well, I've been skating. Lots. Um. I joined the Londonskaters Speed Team, so some of that skating could even be dignified with the name "training".

Here, in case you cared, are two graphs derived (by gpsman: I haven't resuscitated gpskate yet as I want to get a working threaded sbcl on my dekstop for it first) from my gps data on evenings last week:

This is a so-called "Long Slow Skate" from last Wednesday: ten laps of a loop concentrating on different aspects of technique on each lap. You only see 9 and a half here as the gps is slow to get a fix while moving, and it was too cold to stop.

This is the log from marshalling the Friday Night Skate two days later.

You will observe

• that something is putting these in the wrong timezone, as both started at 20:00, not 21:00. Doesn't seem to be a problem with the data at least as far as the gpx file goes: maybe gpsman thinks it's in France.

• the effect of skating on roads with tall buildings both sides - the signal drops out in places, and there are some odd spikes in others. The big triangular section at 20:45 on Friday was probably the half time break.

• that most of marshalling a street skate is a matter of repeatedly overtaking the skate, blocking a side road or junction, waiting for the skate to go past, and repeating. Stop go stop go stop go.

I've just registered for the European Common Lisp Meeting in April: flights, hotel,#

Mon Feb 14 22:57:16 2005

I've just registered for the European Common Lisp Meeting in April: flights, hotel, conference, the works. I'll be arriving on Friday so I can do the Amsterdam Friday Night Skate first...

In other news#

Tue Feb 15 00:24:56 2005

In other news...

• I got an author contract from Linux Journal for an article I wrote for them about eighteen months ago. Frankly, I'd given up on ever hearing from them a while ago (they didn't reply to my "ping" email), and I'm not sure what to do now. The article was about how to make your own live CD out of bits of Debian unstable: I'm not at all sure that the procedure described in it would still work, and I'd really find it hard to believe that nothing that's come along in the meantime is an improvement. Like, say, gnoppix-based-on-ubuntu.

• Got new wheels; need more. My skates use a rather lame "hi-lo" setup where the front two axles are nearer the ground than the back two, meaning that the wheels in the front set should be 4mm smaller. So on Sunday I got new 76s to replace the rather triangular-looking ones on the front, then on fitting them found that they were actually larger than the supposed 80s in the back. Some wheel swapping ensued and I now have 76 on the back and previously-80-now-probably-72 on the front. Tested with a few minutes tottering around the serpentine road, then the sunday stroll down to Battersea Park and back in the face of a strong headwind.

• I came very close to writing a Perl script on Saturday, then just in time came to my senses and realised that the machine it'll be installed on doesn't even have the DBI/DBD modules installed yet and why sully it? So, I'm fiddling on and off with database APIs until I find something I like.

  (with-query (q (istr::*database-handle* "select * from country"))
     (loop
      (let ((row (get-row istr::*database-handle*  q)))
        (unless row (return))
        (format t "hello ~S~%" row))))
  

  As previously discussed, I don't like O-R mapping much. But nor can I remember the pg.lisp interfaces, and when I do go to look them up I find that the efficient one (pg-for-each, uses cursors) not only wants to execute a callback function for each row, but doesn't pass the attribute names to that callback.

• Pthreaded AMD64 SBCL mostly works but doesn't yet have interrupt-thread, and gives wrong error messages when you ask it for backtraces. I don't know how much time I have for hacking it in the next two weeks, though; it's all looking a bit busy around here.

I feel compelled to share this, just because it took so long#

Sat Feb 19 02:06:24 2005

I feel compelled to share this, just because it took so long. It may yet prove to be utterly useless.

I remarked in my previous entry that I was playing with database APIs in an attempt to avoid Perl: this is related. If you look at the Perl Cookbook, or even read a lot of Perl code written by people who have, you find that over half the recipes basically read incoming data (from a database query or a file or something), whack it into a hash or hash of hashes, then when they've collected everything they iterate over or otherwise traverse the hash contents to get the data out in a more useful format. The classic example is finding the n most common words in a file: for each word we read we look it up in the hash and either add it with value 1 if not found, or increment the value if it was. At the end of the file we have a hash of word frequencies we can easily iterate over, sort or whatever.

You could do that in CL with hash tables but, well, try it and you'll see it ends up being a lot more verbose - especially if you want to have "multidimensional" tables - hashes inside hashes.

(gethash "quux" (gethash "bar" (gethash "foo" table)))

and that's without checking that the intermediate values exist. And hashtables don't print in readable format either. And there's no relationship between the order you put elements in and the order thay come out in - not even a dependable happens-to-work-this-way-but-the-same-way-everywhere relationship as Perl gives you.

Plists are nice, though. They're readably printable. I happen to like that they have fewer parens than alists. They don't randomly shuffle themselves around when you're not looking. You can use destructuring-bind to pick them apart conveniently, too (perhaps this is a more compelling advantage than some of the previous). So, I wanted nestable plists. And as it's now 2:33AM I can't be bothered to relate the whole epic voyage of define-setf-expander discovery (some other time, perhaps) so I skip straight to the answers. First, an example

DBS> (let ((l (list :foo 1 :bar 2)))
  (setf (ref l :foo :ban :barry) 17)
  (setf (ref l :boz) 14)
  (format t "-= ~A ~%" (ref l :foo :ban))
  l)

-= (BARRY 17)
(:BOZ 14 :FOO (:BAN (:BARRY 17)) :BAR 2)

and here's the code. Some of the hard bits were taken from SBCL's define-setf-expander form for getf.

(defun ref (plist &rest keys)
  (reduce #'getf keys :initial-value plist))

(defun %put-ref (new-value plist key &rest more-keys)
  ;; not quite Perl-style autovivification, but we do create
  ;; appropriate list structure for intermediate keys that can't be found
  (unless (listp plist) (setf plist nil))
  (let* ((sub (getf plist key))
         (val (if more-keys
                  (apply #'%put-ref new-value sub more-keys)
                new-value)))
    (if sub
        (progn (setf (getf plist key) val) plist)
      (list* key val plist))))

(define-setf-expander ref (place &rest props
                                  &environment env)
  ;; %put-ref may cons new structure or mutate its argument.
  ;; all this magic is just so that we can
  ;; (let ((l nil)) (setf (ref l :foo :bar) t))
  (multiple-value-bind (temps values stores set get)
      (get-setf-expansion place env)
    (let ((newval (gensym))
          (ptemps (loop for i in props collect (gensym))))
      (values `(,@temps ,@ptemps )
              `(,@values ,@props )
              `(,newval)
              `(let ((,(car stores) (%put-ref ,newval ,get ,@ptemps)))
                 ,set
                 ,newval)
              `(ref ,get ,@ptemps)))))

I'd like to add a postscript to the preceding: this is not set in#

Sat Feb 19 02:44:32 2005

I'd like to add a postscript to the preceding: this is not set in stone. (1) I'll probably hack it for some looser equality predicate than EQ. (2) To cope with uses like (incf (ref thing :el)) it would be very nice to pass an optional default. Now is however not the time to decide which arg the default will be, but it's looking kind of non-obvious at present.

Do POSIX subcommittees ever look at each others' work#

Thu Feb 24 21:31:53 2005

Do POSIX subcommittees ever look at each others' work? The implementation of interrupt-thread is made more complicated by it not being possible to sigqueue a thread, and likewise not being possible to pthread_kill and pass any data to the thread being signalled. So, we need something held per-thread to indicate what code to interrupt it with, and then some stupid locking regime so that it can be interrupted from two other threads at once without something weird happening. OK, it's not rocket science, and scalability arguments probably don't matter too much as we don't really expect interrupt-thread to be used in expected-to-be-fast code anyway. It's still annoying, though.

I still haven't bought a new laptop; I'm still trying to figure out whether I actually need one or not: so far, the only requirement I've identified is to be able to IRC in front of the TV from time to time, and that's not pressing really. I did also wonder for a while whether I could use a PDA, but after thinking a while longer, maybe I can get away with a text file instead.