diary @ telent

Respinning threads from Gotosocial#

Mon Aug 25 16:46:35 2025

Topics: fennel fediverse

I made a small Fennel script to extract the text from a GoToSocial thread and splat it into a text file. I have used it once and it seemed to work, so I'm going to explain it a bit.

It needs lua-http, rxi's json.lua, and net-url aka net.url aka neturl, all of which you can currently get from MS Github, if they haven't already replaced the whole site with an AI-mediated barrel of slop by the time you read this.

(local request (require :http.request))
(local url (require :net.url))
(local json (require :json))
(local {: view } (require :fennel))

lua-http might be a bit overkill here but I've used before and I went with what I know. I wrapped it here with a little send-request function so I didn't have underscores all over the place.

The last time I really dug into HTTP was about 2003, so this HTTP/2 concept that the request method is actually a header field with a colon in front of its name is ... unusual and new to me.

The url passed into this method comes from net-url and is actually a table with a metatable that defines __tostring. This is like a string when you print it, but not quite enough like a string when you pass it to lua-http, hence the cast.

(fn send-request [url method headers body]
  (let [r (request.new_from_uri (tostring url))
        h r.headers]
    (h:upsert ":method" method)
    (when body
      (r:set_body body))
    (each [k v (pairs headers)]
      (h:append k v))
    (let [(headers stream) (r:go)]
      (stream:get_body_as_string))))

(fn json-request [url method headers attrs]
  (let [body (json.encode attrs)]
    (-> (send-request url method headers body)
        json.decode)))

The next bit is pretty much a straight translation of the Gotosocial API login flow documentation from bash + curl into Fennel.

One surprise here was that the net-url API is very side-effecty. If you write

(let [u (url.parse "HTTPS://example.com")]
  (print (/ u "some" "path" "segments")))

It will print https://example.com/some/path/segments, as you might expect from skimming the docs, but it will also change the value of u, as you might not expect unless you'd actually read the docs (guess who didn't and only skimmed them). u:resolve is a more functional alternative.

(fn oauth-new-client [root-url client-name]
  (json-request
   (.. (root-url:resolve "/api/v1/apps"))
   :POST
   { :content-type "application/json" }
   {
    "client_name" "fetch-thread"
    "redirect_uris" "urn:ietf:wg:oauth:2.0:oob"
    "scopes" "read"
    }))

(fn oauth-access-token [root-url client_id client_secret code]
  (let [body {
	      : client_id
	      : client_secret
	      :redirect_uri "urn:ietf:wg:oauth:2.0:oob"
	      : code
	      :grant_type "authorization_code"
              }]
    (json-request
      (.. (root-url:resolve "/oauth/token"))
      :POST
      { :content-type "application/json" }
      body)))

(fn request-api-token [root-url]
  (let [{: client_id : client_secret } (oauth-new-client root-url "unroll.fnl")
        redirect-url (..
                      root-url
                      "/oauth/authorize?client_id=" client_id
                      "&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code&scope=read")]
    (io.stdout:write "login to the instance and then paste in the token here\n> ")

    ;; hardcoded firefox because `xdg-open` doesn't work for
    ;; me right now. That's something to fix another day.
    (os.execute (string.format "firefox %q" redirect-url))
    (let [code (io.stdin:read)
          {: access_token}
          (oauth-access-token root-url client_id client_secret code)]
      access_token)))

The net result of all that code will return an access token.

The access token can be used with the other API calls but not, as I originally thought it would, for fetching actual statuses like /users/dan/statuses/01K2Q3MHSGMHB2CFTRCE6FBP5T. That needs HTTP signatures (a.k.a "authorized fetch") which is something quite else and looks more involved than I wanted to get into at 12:30am.

However, we can fetch the status content in a different format using an API endpoint: /api/v1/statuses/01K2Q3MHSGMHB2CFTRCE6FBP5T gets the post itself and ..../context gets its ancestor and descendant statuses.

(fn fetch-post [root-url headers id]
  (let [url (root-url:resolve (.. "/api/v1/statuses/" id))]
    (json-request url :GET headers nil)))

(fn fetch-post-context [root-url headers id]
  (let [url (root-url:resolve (.. "/api/v1/statuses/" id "/context"))]
    (json-request url :GET headers nil)))

Our conversion to markdown is extremely rudimentary:

(fn format-post [post]
  (each [_ img (ipairs post.media_attachments)]
    (print (string.format "![%s](%s)" img.description img.url)))
  (print post.text))

and the rest is just calling things in order.

(fn extract-title [text]
  (->
   (or (string.match text "(.-)[%.%?%!\n]") text)
   (string.sub 1 60)))

(fn root-and-id [url-str]
  (let [u (url.parse url-str)]
    (values
     (u:resolve "/")
     (string.match u.path ".+/(.-)$")
     )))

(let [dotfile (string.format "%s/.rethread" (os.getenv "HOME"))
      (root-url post-id) (root-and-id (. arg 1))
      f (io.open dotfile :r)
      api-token
      (if f
          (f:read)
          (let [t (request-api-token root-url)]
            (with-open [out (io.open dotfile :w)]
              (out:write t)
              t)))
      headers { :authorization (.. "Bearer " api-token)}
      context  (fetch-post-context root-url headers post-id)
      post (fetch-post root-url headers post-id)
      first-post (or (. context.ancestors 1) post)]
  (print (.. "Title: " (extract-title first-post.text)))
  (print (.. "Slug: " post.id))
  (print (.. "Date: " (. first-post  :created_at)))
  (print "\n")
  (each [_ p (ipairs context.ancestors)]
    (format-post p))
  (format-post post)
  (each [_ p (ipairs context.descendants)]
    (format-post p)))

Aftercare

Some thoughts as they occur to me:

All of that is a very long-winded way of saying "expect to see more blog posts that are really just recycled fediverse threads", but I'm now on my fourth fediverse server and I won't put up again with saying goodbye to all my posts every time I change to a new instance. So, I am blogging for persistence (and some day, less shitty search).