Month of July 2003

You're all wondering how it went, I know. I compromised. Current thinking: it's still sometimes convenient to pass rows around from one function to another, and this might just as well be in an object of some kind as a property list or similar ad hoc structure - because then we don't have to rewrite all those silly methods to format it into silly functions to format it. I am still convinced, however, that trying to do anything with the identity of these objects, or keeping them for any length of time, is Wrong. All trace of object caching is gone.

In the process I've spent a fair amount of time staring at bits of SQL and running queries through the analyser; changing types here, adding indices there, etc etc. (Why is my creditcard issue number a bignum? I find it hard to imagine that'd get often get much byond 9, never mind (ash 1 31)). So, the overall impact is good.

Still alive, not that you'd notice. A couple of URLs for your reading pleasure:

• IEEE's "Body of Knowledge" for Software Engineering

  The SWEBOK is written as the basis for licensing exams for professional software engineers. If your state requires you to get a license to practice software engineering (and more will, if they are convinced that they can create fair exams based on a consensus document from the profession), the SWEBOK is the document you will have to study.

  I am most familiar with SWEBOK\x{2019}s treatments of software testing, software quality and metrics. It endorses practices that I consider wastefully bureaucratic, document-intensive, tedious, and in commercial software development, not likely to succeed. These are the practices that some software process enthusiasts have tried and tried and tried and tried and tried to ram down the throats of the software development community, with less success than they would like.

• Tom Robinson on the music industry

  By far the most important factor in a successful artist's career has to do with identity and focus - in a word, PACKAGING - the one area musicians most often neglect. [...]

  For music fans, wearing an Oasis T-shirt or getting a Skunk Anansie tattoo is a public statement about who they are. Others impress their friends with Pavarotti or Sting CD's at a dinner party, while Billy Bragg has always carefully packaged himself for people who loathe packaging...

I recently got the Stargreen update installed and running (not that you can tell the difference; it's all backend stuff), so in theory I have some more free time to write here. As soon as I have everything else done that I've been putting off while I finished that (UKUUG paper, another article, a proposal) I'll have more time to write this anyway.

In the meantime, continuing the theme of objectful vs relational operations:

The reaction at the time was that [the relational model]'s going to be inefficient, and it's going to use more instructions and more disk I/Os than IBM's Information Management System (IMS), the first commercial hierarchical, structured database management system. In addition, the industry was moving from batch-transaction processing that dealt with sets of records to online transaction processing that dealt with individual records ("give me that bank account").

Ted's ideas were extremely controversial. The core issues were about how computers would be used in the future and what was important to optimize. This was not a PC world. This was a mainframe world of business data processing.

DP The controversy was whether anybody would pay for such inefficient use of the computer?

JG Right, and whether we were headed toward a world of online transaction processing where nobody actually wanted to go through all the records. Everything was going to be online, everything was going to be incremental, and there wouldn't be this need for batch reporting.

From A Conversation with Jim Gray

OK, just one more link, this time to a McCLIM listener screenshot

It's looking a little less frenetic around here now, then.

On Moday I spent some time playing with Sawfish. The task was to have it listen to my modem (which lives on my voice line and is only ever used for faxes or dire emergencies) and popup a window with caller ID details whenever the phone rings; unfortunately it turns out that my modem doesn't do caller id anyway, and even supposing it did it's uncertain that my phone company (NTL, Oxford franchise) can turn it on.

I've been using Sawfish as a window manager for three years or something and still haven't actually tried its Lisp dialect. Initial impressions: nice general design, but handicapped by being based on a lousy Lisp. Painful development process: dying on SIGSEGV is not convenient when the window manager was the last command run from .xinitrc

Notes on McCLIM : day 1

I'm using SBCL approximately-0.8.1, telent CLX (CVS version, but it's near enough the current release) and McCLIM anon-cvs of approximately whenever this diary entry is made. SBCL is compiled with the :sb-thread feature. The assumptions I've made so far may be horribly horribly wrong, but it seems to be working. I probably won't come back and revise this at a later date as I find out more, so do be sure to read later diary entries if you stumble across this significantly later than I'm writing it and are expecting to depend on anything you learn here.

To build McCLIM, (load "system") then (asdf:operate 'asdf:load-op :clim-clx). Don't bother installing mk-defsystem, someone's passed this way with asdf already. OK, that might have been me in an earlier life

It's visibly faster than it used to be.

The first two chapters of the manual (in McCLIM Doc/) have a worked example, which is worth looking at. Note that the output from the parity command is only displayed very very briefly. If you'd like to actually see it, add the option

          :display-time nil

Some layout monkeying was needed, in define-application-frame. Note that

• the modus operandi seems to be that (1) the interesting panes get named in the :panes clause, then (2) the :layouts clause describes one or more arrangements for how to fit them together. In this case we have a single layout called default. The layouts clause actually introduces more panes for layout control, but they don't get names. vertically and suchlike are actually macros that expand into calls to (make-pane 'some-pane-type ...): although it looks fairly declarative at the front, that's just a front.

• we've chosen a non-Courier font. This is Required :-)

• scrolling in the layout puts a scroller-pane around our journal - i.e we now have scroll bars. We disable the scrollbar for the inner journal (there was only a vertical scrollbar to start with) because it was always coming out with the thumb the same size as the pane, so no actual scrolling movement was available from it.

Our goal here is a new frontend for the metacircles bookkeeping system (actually, the real goal is just to print out a couple of invoices, this whole clim thing is so much sidetrack. But anyway) so we really want most of the application pane to be showing the same thing most of the time, instead of being dribbled into by various commands. So, remove :display-time again and replace with :display-function 'display-journal. Now we just have to write a display-journal method that does the business.

display-journal depends quite heavily on some kludgey macros I wrote a couple of weeks ago to sit atop Eric Marsden's pg.lisp. You don't see them here, and they aren't yet available anywhere else (though will be, someday, when they're cleaner). Regardless, it should be fairly obvious what they do.

Look at the presentation types! I don't know if this is actually a sensible use for them: I haven't worked out how to get these not to be higlighted when moused-over yet; I thought the :single-box nil ould do that, but it doesn't. In the meantime, notice that you need to define a presentation type, then a bunch of methods on it to tell the system how to present it and how to input it. We've only done the present method for these so far.

Fell into a what-strange-thing-is-clim-doing-now trap for too long, before realising that the problem was pg (or, just as likely, my glue for it) catching errors and throwing them away without telling me something had gone wrong. This made it difficult to work out why it had suddenly stopped wanting to print anything other than the column titles.

If you really care you can have a gander at the current source and/or the screenshot. Should note that it's not being entered for any artistic prizes, but at least it's not Courier.

There is a problem with the queuing/refresh. This manifests in two ways

• First, refresh doesn't always happen until the next time the application gets some events (mouse movement or keys or suchlike)

• Second, more rarely, something iffy in the bowels of SBCL or CLX makes it hang every so often (not usually all that often, but it varies). killall -CONT sbcl from the command line tends to unstick it in those cases.

Ok, that's enough for one evening.

CLiki visual makeover FAQ

1. Q: It's very nice. Did you do it?
   A: Yes, isn't it? No, we have Miles Egan to thank for all the pretty stuff. A certain amount of blame for butchering it to fit into the existing cliki framework(sic) may be attributed to me.

2. Q: The text is very small. Can you make it bigger?
   A: Yes. Any day now.

SBCL just changed the fasl file version, so I need to rebuild CLX and McCLIM. While it chugs away it might be time for some notes on McCLIM Day 2, except that there's practically no difference from day 1. I created a class for account, and a presentation type which presents as an account name. Then I decided to tackle some of the bugs

• The pg problem was that it was doing something akin to

  (handler-case 
      (foo) 
    (error (e) 
      (cleanup forms)
      (signal e)))
  

  Notice that the error is resignalled with signal instead of error. The difference between the two is that error will enter the debugger if the condition e is not handled, whereas signal will keep quiet and pretend nothing happened. I rewrote this to use an unwind-protect form instead.
• Spent some time looking at backtraces to decide where the periodic hangs were coming from. After deciding the bug was in SBCL itself, spent even more time staring hard at the queueing functions. Then rewrite them to have less obvious race conditions. I don't claim it's necessarily correct now, but I haven't managed to break it yet. It's probably slower. It might be detectably slower; it seemed to be quite slow when I was testing it, but I might have been imagining that. Or it migt have been swapping. This machine is usually swapping, after all. Anyway, changes committed, will be in 0.8.2 at the end of the month

I'm not going to cry too hard about the slowdown at the moment, because McCLIM has native-threads-unfriendly queuing, and the effect of that on top of the CLX native-threads-unfriendly thread synchronisation primitives means that there should be some fairly significant speedups to be made that don't involve staring at sbcl disassembly. In fact, as low-hanging fruit goes, I think that if we park the wheelbarrow underneath the tree and shake gently, this stuff wouldn't even fall far enough to bruise.

;;; from CLX
+sbcl
(defvar without-interrupts-sic-lock
  (sb-thread:make-mutex :name "lock simulating without-interrupts"))
+sbcl
(defmacro without-interrupts (&body body)
  `(sb-thread:with-recursive-lock (without-interrupts-sic-lock)
    ,@body))

;;; from McCLIM
(defmacro atomic-incf (place)
  `(sb-thread::with-spinlock (xlib::conditional-store-queue)
    (incf ,place)))

Words and symbols will pose no problems and a three-dimensional shape can be represented by a two-dimensional drawing. However, there are some interesting cases regarding trade marks for sounds and smells. At present it is clear that whatever the trade mark, its representation must fit into an 8 cm by 8 cm box on the application form.

The Stargreen web site, which I may have mentioned on these pages before - what? "only twice a week"? OK then - has a page which lists all the gigs in the system for a given month. Clicky arrows for next/previous months, etc.

Anyway, every so often, somebody (it might be a bot) decides either to click their way through every month until 2039, or to edit the url directly to go to 2039 or some other after-death-of-Dennis-Ritche date. Which causes an error, for which I get to see the backtrace in my dribble file. I wouldn't care too much - after all, it's fairly obvious there are not going to be any gigs booked that far in advance, these people are really just playing - but for completeness sake if nothing else it'd be nice to fix. After all, this is Lisp, one of whose boasts is that we don't have integer overflow problems when we exceed 32 bits.

 (decode-universal-time (+ (* 86400 365 50) (get-universal-time)))
debugger invoked on condition of type TYPE-ERROR:
  The value 2635786389 is not of type (SIGNED-BYTE 32).

The problem is with timezones. The only portable way to get timezone data is by calling the libc localtime() function with what the manual page describes as a calendar time - the integer number of seconds since 1970, stored as a time_t. Which is to say, probably 32 bits on most systems in use today. So, despite the fact that we can in principle handle time arbitrarily far in the future, we can't if we want to know if Summer Time is in effect.

So I was jyst about to write a long diary entry detailing what the various different things we could do at this point are, but instead you can read the mail I sent sbcl-devel. The reason for my sudden change of heart? I've just checked the tzfile manual page more closely and noticed that

       The above header is followed by tzh_timecnt four-byte  values  of  type
       long,  sorted  in ascending order.  These values are written in ``stan-
       dard'' byte order.  Each is used as a transition time (as  returned  by
       time(2)) at which the rules for computing local time change.

Which makes the compiled zoneinfo data useless beyond 2038 too. And practically nobody keeps the sources around. This is exactly the excuse I need to not implement option 4. I think 3.1 is most likely.