diary at Telent Netowrks

Puppets at work#

Fri, 01 Mar 2013 11:12:54 +0000

Another huge long gap between updates. I'd say that you might as well get used to it, but you probably are already. This is a short note to say that I have developed a Capistrano extension which runs Puppet as part of an app deploy, and you can read about it on the Simply Business tech blog

In other news, having confirmed that the X11 touchpad support is broken by design - the choice of whether to emulate mouse events (left click, middle click, drag etc) is set at a global level in the driver and cannot be overridden per client, so legacy mouse-only clients cannot meaningfully coexist with clients that usefully could handle touch events directly - I am working on a client implementation of the Wayland protocol in Clojure. I hope this will eventually will turn into a habitable text editor.

Still some way(land) to go#

Thu, 07 Mar 2013 22:01:54 +0000

Ignoring, because I can, this whole Ubuntu Mir thing completely, I have begun to learn stuff about the Wayland protocol (and about Clojure, with which I am still at the constantly-having-to-google-stuff stage). Some random notes on what I have learnt so far

First off: Java has no builtin support for talking to unix-domain (AF_FILE) sockets. And nobody seems to make a Maven-packaged library that does it either. This is a shame because Leiningen makes Maven manageable, but anything unmavened involved tedious mucking around. Eventually I did

:; apt-get install libunixsocket-java
:; cat project.clj
(defproject psadan "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :resource-paths ["/usr/share/java/unix-0.5.jar"]
  :url "http://example.com/FIXME"
  :jvm-opts ["-Djava.library.path=native/:/usr/lib/jni/"]
  :main psadan.core
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [org.clojure/data.zip "0.1.1"]
                 ])

which seems to be holding up. Then I opened a socket and tried reading from it in the hope of getting some lovely protocoly stuff 

(defn open-socket [name]
  (let [s (cx.ath.matthew.unix.UnixSocket. name)
        in (. s getInputStream)
        out (. s getOutputStream)
        ]
    {:socket s :input in :output out}))
(def socket (open-socket "/home/dan/private/wayland-0"))
(defn rd []
  (let [buf (byte-array 1024)]
    (. (:input socket) (read buf))
    buf))

and watched it hang. After some time looking at documentation and mucking about with strace to see if it was trying to read the full buffer instead of doing the short read I wanted it to, I eventually thought to use socat instead of clojure. It turns out that the client is expected to make the first request before the server sends anything, and with the use of strace weston-info I was able to find out what.

26335 sendmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"\1\0\0\0\1\0\f\0\2\0\0\0\1\0\0\0\0\0\f\0\3\0\0\0", 24}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 24

Time to start writing some code to parse wayland.xml so we can actually find out what this means. The usual way to do XML parsing in clojure seems to be using zippers and the easy examples seem to be somewhat lacking or slightly ungooglable. You need a project dependency on org.clojure/data.zip and a bunch of package requires, then you call clojure.zip/xml-zip on the return value of clojure.xml/parse and that gets you a zipper

(ns psadan.core
  (:require [clojure.xml]
            [clojure.data.zip :as dz]
            [clojure.data.zip.xml :as x]
            [clojure.walk]
            [clojure.zip :as z]))
;; [...]
(def the-protocol
  (->
   "/home/dan/wayland/source/wayland/protocol/wayland.xml"
   clojure.xml/parse
   z/xml-zip
   parse-protocol))

where I have conveniently left out the definition of parse-protocol and everything it calls because it's longwinded and tedious (but the code will be on github as soon as I'm not ashamed of it) but it might hypothetically do things like

(x/xml-> my-zipper :protocol :interface :request)

to descend the tree through <protocol> <interface> <request> and return all the elements. Use the similarly named x/xml1-> to get the first matching element only. The return values from these things are themselves zippers and you can call up, down etc - or xml-> again - to traverse the tree further, then eventually call node when you want to get the element itself. So e.g.

(defn parse-interface [i n]
  ;; n is a badly named zippered xml object thingy
  (let [el (z/node n)
        requests (map-indexed parse-message (x/xml-> n :request))
        events (map-indexed parse-message (x/xml-> n :event))
        enums (map parse-enum (x/xml-> n :enum))]
    {:index i
     :name (:name (:attrs el))
     :requests requests
     :events events
     :enums enums
     }))

So let's handwave over the details and take it on trust that I have parsed the whole file. There are two other things I discovered - mostly thanks to the #wayland IRC channel participants - about the wayland wire protocol that the docs don't mention:

Given which, we can attempt to parse that first client->compositor message

psadan.core> (parse-messages-from-buf (vec (.getBytes "\1\0\0\0\1\0\f\0\2\0\0\0\1\0\0\0\0\0\f\0\3\0\0\0")) :requests)
({:object-id 1, :bytes 12, :interface "wl_display", :message "get_registry", :opcode 1, :args (2)} {:object-id 1, :bytes 12, :interface "wl_display", :message "sync", :opcode 0, :args (3)})

Looks plausible so far ...

Next up, some code to create messages. And something, which may involve an atom, to map object ids to their corresponding interfaces as we learn them. After that, we find out what the Wayland FAQ really means by "shareable buffer"

Addendum, penned the following morning

Because this is supposed to be a log of what I am learning, and I had never previously learned about the -> operator (seen it, never used it), for completeness' sake:

(-> x fn1 fn2 fn3)
is the same thing as 
(fn3 (fn2 (fn1 x)))

Googling for clojure -> is not a terribly fulfilling endeavour, so it's helpful to know that it's also referred to as the 'arrow' operator, the 'threading' operator, and apparently also the 'thrush' operator. This blog article from Fogus explains further.

One more step along the Wayland#

Tue, 12 Mar 2013 13:57:03 +0000

Yesterday's lunchtime hacking was all about splitting the project into multiple files and getting it into git and onto Github - note that the mere fact of it being publically browsable does not imply that it will run, build, walk, make tea, perform any other useful function, or even forbear from exploding inside your computer and rendering the SSD to molten slag. Nor that I'm not still ashamed of it. It just keeps me slightly more honest.

Today I implemented enough of pack-message to be able to recreate the initial client->compositor message that we observed weston-info send last week. Still taking extraordinary liberties with signed vs unsigned longs, and plase note that all this code will work only on little-endian machines (there are any big-endian machines left?).

Lessons, puzzles

Leiningen does not need you to list the source files in your repository individually: it finds them magically. I believed otherwise for a while, but it turned out (slightly embarrassingly) that I had a parenthesis in the wrong place. My working hypothesis is that it assumes there is one namespace for each file, and that any reference to a namespace it doesn't know about it can be satisfied by loading a file with that name.

If I type (in-ns 'psadan.core) at the repl and that ns does not include a (:refer-clojure) form, I can't use the symbols in clojure.core at the repl. I have not observed a similar issue wrt uses of clojure.core/foo in core.clj itself, just at the repl.

atoms! An atom is dead simple, really - conceptually at least, if not also under the hood: it's a wrapper for an object that lets you look inside with deref and lets you change what's inside with swap!. For each connection we use an atom holding a mapping from object ids to the corresponding objects, which starts out holding the singleton object for wl_display and then needs to be updated each time we generate an object locally and each time we learn of a new object from the peer.

(defn open-connection [name]
  (let [s (cx.ath.matthew.unix.UnixSocket. name)
        in (. s getInputStream)
        out (. s getOutputStream)
        wl-display (global-object-factory)
        ]
    {:socket s
     :input in
     :output out
     :display wl-display
     :objects (atom (assoc {} 1 wl-display))
     }))
(defn remember-object [conn id object]
  ;; (swap r fn args...) gets the current value of the atom inside r,
  ;; which for the sake of argument we shall call oldval, then sets the atom
  ;; to the result of calling (fn oldval args...)
  (swap! (:objects conn) assoc id object)
  object)
(defn get-object [conn id]
  ;; foo is another way to write (deref foo)
  (let [o (get (:objects conn) id)]
    o))

I have probably not chosen the fastest possible way of building up the messages I plan to send, in terms of fiddling around sticking vectors of bytes together. Will worry about that later if it turns out to be a bottleneck (but suggestions are welcome).

There was not a lot of Wayland learning this time. In the next round we shall be sending it the messages we have so lovingly composed from whole cloth and see if it reacts the same way as it did when the same bytes were sent from weston-info

Finding a Way(land)#

Thu, 14 Mar 2013 13:32:01 +0000

In the next round we shall be sending it the messages we have so lovingly composed from whole cloth and see if it reacts the same way as it did when the same bytes were sent from weston-info

And the answer is ... yes, pretty much. We had to fix up our string parsing to make sense of the replies, but watch:

  1. 'psadan.core/connection psadan.core> (defn test-send-message [] (let [registry (conn/remember-object connection {:id 2 :interface (proto/find-interface-by-name :wlregistry)}) done-cb (conn/remember-object connection {:id 3 :interface (proto/find-interface-by-name :wlcallback)}) ] (conn/write-buffer connection (buf/pack-message connection (:display connection) :requests :get_registry [registry])) (conn/write-buffer connection (buf/pack-message connection (:display connection) :requests :sync [done-cb])) registry))

  1. 'psadan.core/test-send-message psadan.core> (test-send-message) {:id 2, :interface {:index 1, :name :wlregistry, :requests ({:index 0, :name :bind, :summary "bind an object to the display", :args ({:name "name", :type :uint, :interface nil} {:name "id", :type :newid, :interface nil})}), :events ({:index 0, :name :global, :summary "announce global object", :args ({:name "name", :type :uint, :interface nil} {:name "interface", :type :string, :interface nil} {:name "version", :type :uint, :interface nil})} {:index 1, :name :global_remove, :summary "announce removal of global object", :args ({:name "name", :type :uint, :interface nil})}), :enums ()}} psadan.core> (def received (read-buffer connection))
  2. 'psadan.core/received psadan.core> (pprint (map (fn [x] [(:object-id x) (:message x) (:args x)]) (buf/parse-messages received connection :events))) nil ([2 :global (1 "wldisplay" 1)] [2 :global (2 "wlcompositor" 2)] [2 :global (3 "screenshooter" 1)] [2 :global (4 "textcursorposition" 1)] [2 :global (5 "textmodelfactory" 1)] [2 :global (6 "wldatadevicemanager" 1)] [2 :global (7 "wlshm" 1)] [2 :global (8 "wlseat" 1)] [2 :global (9 "inputmethod" 1)] [2 :global (10 "wloutput" 1)] [2 :global (11 "wldrm" 1)] [2 :global (12 "wlshell" 1)] [2 :global (13 "desktopshell" 1)] [2 :global (14 "screensaver" 1)] [2 :global (15 "inputpanel" 1)] [2 :global (16 "workspacemanager" 1)] [3 :done (58)] [1 :delete_id (3)])

My interpretation of what's happening here is that we're sending to the server a 'tell object 2 about all your global objects' message, followed by a 'tell object 3 done when you're finished doing stuff' message, and as you can see from the output, the reply is a bunch of ids for global objects sent to object 2, a done event sent to object 3, and then a delete_id event for object 3 sent to object 1. I'm actually not sure why that last one triggers, as I don't think I asked it to. Perhaps it's just tidying up for me.

I'm also handwaving - if not actually handdrowning - a litle bit, because really ... are these :global messages sent to object 2 or from object 2? For the moment, I am using the two directions interchangeably, which is probably not a recipe for an easier future life, but in the meantime I can continue to tread water.

It's instructive, or at least reassuring, to compare this stuff with what weston-info says:

interface: 'wl_display', version: 1, name: 1
interface: 'wl_compositor', version: 2, name: 2
interface: 'screenshooter', version: 1, name: 3
interface: 'text_cursor_position', version: 1, name: 4
interface: 'text_model_factory', version: 1, name: 5
interface: 'wl_data_device_manager', version: 1, name: 6
interface: 'wl_shm', version: 1, name: 7
	formats: XRGB8888 ARGB8888
interface: 'wl_seat', version: 1, name: 8
	capabilities: pointer keyboard
interface: 'input_method', version: 1, name: 9
interface: 'wl_output', version: 1, name: 10
	x: 0, y: 0,
	physical_width: 1024 mm, physical_height: 640 mm,
	make: 'xwayland', model: 'none',
	subpixel_orientation: unknown, output_tranform: normal,
	mode:
		width: 1024 px, height: 640 px, refresh: 60 Hz,
		flags: current preferred
interface: 'wl_drm', version: 1, name: 11
interface: 'wl_shell', version: 1, name: 12
interface: 'desktop_shell', version: 1, name: 13
interface: 'screensaver', version: 1, name: 14
interface: 'input_panel', version: 1, name: 15
interface: 'workspace_manager', version: 1, name: 16

Top Wayland tip for today: it appears to be the case that you can make the C library clients log protocol exchanges to stderr by putting WAYLAND_DEBUG=client in the environment. When doing that it's clear to see that weston-info is making a couple of additional requests that we're not. We could add them, but I think the more pressing concern is to make it do something with the events we're getting already - if it's sending us details of global objects that we might need to know about, we should at a minimum be storing those details somewhere instead of throwing them away ...

The long way(land) round#

Mon, 18 Mar 2013 21:07:47 +0000

The latest round of psadan hacking was motivated by two goals

We went down a couple or three dead ends on our way to this goal, but eventually we settled on creating an agent responsible for listening to the connection (I called it channel, in the absence of any better ideas) and dispatching (using a handle-message multimethod) to some code appropriate for each kind of message we're receiving.

Learnings

Next up? At some point we need to decide whether sending messages should be done by the channel or whether it's OK to carry on doing that directly in whatever thread we happen to be in. But what would be much more fun is to see if we can actually render a window...

In mysterious way(land)s#

Thu, 21 Mar 2013 13:29:35 +0000

Why are our bind messages to wl_registry erroring out?

 
15:19 < daniels> yeah, wlregistrybind is a special case - i think it's the 
                 one non-autogenerated marshaller we have
15:32 < daniels> actually no, sorry, i'm lying through my teeth
15:32 < daniels> it's no longer autogenerated
15:32 < daniels> but the parser expands newid to interface + version + id
15:33 < daniels> it used to be hand-written, but is now autogenerated
15:33 < daniels> http://cgit.freedesktop.org/wayland/wayland/tree/src/scanner.c#n614
16:28 < jekstrand> danb: It is a somewhat-special case.  Basically, every time 
                   there's a newid that does not have any type information 
                   specified, two aditional pieces of information (interface 
                   name and version) get added.
16:29 < jekstrand> danb: That really should be documented, but it's not.  I 
                   had to grep through the scanner.c file to find it all.

Armed with this additional info the fix was fairly straightforward: I augmented the XML parsing/protocol generation to stick an additional pair of arguments into each message that contains a new_id and no interface.

Thanks to both daniels and jekstrand on irc for clearing it up.

⟪Feb 2013 Apr 2013⟫