Lambda Island

This one tool will transform how you program. It’s time to raise your standards!

Transcript

Programmers love to argue about which language or editor is better and cooler than anything that ever was or will be. Wars have been fought and empires have fallen in quarrels over whether Vim or Emacs is best. This is silly because while these are fine editors in their own right they are also toys that aren’t worth the electromagnetic charges they are stored as.

I will show you how to shed these training wheels and take your place among the rightfully smug, by raising your standards to the standard. The standard programming language being LISP, in its modern incarnation, Clojure, and the standard editor being ed(1), or as it is formally known: “The One True, The Right Honorable, The Standard Editor, Edward ‘ed(1)’ Editor”.

Because technologies sound more better when prefixed with “modern” I will teach you “modern ed(1)” which is really just regular ed but invoked with rlwrap and the -p option.

ed(1) is the only uncomplected editor. It singularly performs its job, not wasting your teletype’s ink on trivialities such as a prompt or error messages. It answers to your every whim, command after command, unless it is unable to do so, in which case it doesn’t waste words on your incompetence, but simply replies with “huh?”.

$ ed
help
?
quit
?
exit
?
^C
?
^C
^[[A^[[A
?
:wq
?
^Z
[1]  + 29559 suspended  ed
$ kill -9 29559
[1]  + 29559 killed     ed

While it may be that you have never heard of The One True, The Right Honorable, The Standard Editor, Edward ‘ed(1)’ Editor, it is a near certainty that ed is already installed on the computer you are using right now. You could verify this by typing ed in a terminal followed by RETURN, but since that may leave you attempting in vain to exit ed until the heat death of the universe you can verify its presence with which ed instead.

$ which ed
/bin/ed

Now I shall teach you ed(1).

Start modern ed(1) by invoking rlwrap ed -p'> '. Once you are inside ed you enter commands, one per line.

ed starts with an empty buffer. With a you enter insert mode, and any consecutive lines you enter get added to this buffer. Enter ‘.’ on an empty line to exit insert mode.

Use w to write to a file.

$ rlwrap ed -p'> '
> a
{:paths ["src" "test"]}
.
> w deps.edn
24

The p command will print the current line. 1p prints the first line. ,p prints all lines. The n command is like p, but also shows line numbers.

> p
{:paths ["src" "test"]}
> 1p
{:paths ["src" "test"]}
> ,p
{:paths ["src" "test"]}
> ,n
1    {:paths ["src" "test"]}

You can delete the current line with d or all lines with ,d. But by now your buffer is empty so ed will go “huh?”. Use h, short for “huh?” to see what’s up. With r you read a file, appending it to the buffer. u undoes the last edit.

> d
> ,d
?
> h
Invalid address
> r deps.edn
24
> r deps.edn
24
> ,p
{:paths ["src" "test"]}
{:paths ["src" "test"]}
> u
> ,p
{:paths ["src" "test"]}

s does a substitution on the current line. It’s followed by a /, a regular expression, another /, and the substitution. s only replaces the first match, add /g to replace all.

> s/}/
{:paths ["src" "test"]
> s/s/sh/g
> ,p
{:pathsh ["shrc" "tesht"]

Empty the buffer, add some lines of text, and admire your work.

A invocation like ,n consists of two parts, an address, and a command.

When omitted, the address defaults to ., which means the current line. When the command is omitted it defaults to the print command, p.

An address can be a line number, a range of lines, all lines, the first line to match a pattern, and a bunch of other things. You can move the “cursor” up or down with - or +, which can be repeated or combined with a number.

> ,d
> a
enter
some
lines
of text
.
> ,n
1       enter
2       some
3       lines
4       of text
> p
of text
> .
of text
> 2
some
> 2,3p
some
lines
> ,
enter
some
lines
of text
> ?lin?
lines
> -
some
> -
enter
> ++
lines
> -2
enter

f sets the name of the file you’re editing, so you can save it simply with write command w. e “edits” a file, reading it into the buffer, and setting it as the current file name, which you can check again with f.

> f words.txt
words.txt
> w
25
> e words.txt
25
> f
words.txt
> ,p
enter
some
lines
of text

A ! lets you run shell commands, % refers to the current file. With w ! you can pipe the current buffer to a command and. With r ! can append the output of a command to the buffer.

> ! echo hello
hello
!
> ! grep n %
 grep n words.txt
enter
lines
!
> w ! wc -l
4
25
> r ! date
29
> ,p
enter
some
lines
of text
Thu Mar 29 11:28:27 CST 2018

I guess that’s all you need to know. Oh yeah q or Control-D quits, unless your buffer has unsaved changes, then it will say “huh?”. Do q again and it will quit anyway.

> q
?
> q

I recently blogged about mayonnaise, since as a Belgian I am not only legally required to always carry chocolate, but also have a moral duty to educate the world about mayonnaise.

The thing to know about mayonnaise is that it is not sweet. There are mayonnaise-like sauces that are sweetened, maybe you even enjoy eating these at times, but don’t ever confuse them with The One True, The Right Honorable, The Standard Sauce, Mayonnaise.

Besides mayonnaise I also talked about a Clojure editor really needing two things, REPL integration, and Paredit. Of course ed(1) has both.

To start a Clojure REPL from ed(1), type !clj. Exit it again with Control-D.

> !clj
Clojure 1.9.0
user=> (+ 1 1)
2
user=>!
>

Let’s evaluate some code. To send the buffer to the REPL, use w !clj.

> a
(* 2 21)
.
> w !clj
Clojure 1.9.0
user=> 42
user=>
9
>

An essential operation is to evaluate an expression, and insert the result into the buffer. To do this first save your code, then pipe it to clj, reading back the result.

> f my_code.clj
> w
9
> r ! clj < %
36
> ,p
(* 2 21)
Clojure 1.9.0
user=> 42
user=>
>

Uhm yeah those prompts aren’t too elegant, let’s clean that up. Delete line number 2, and then on all lines remove that user=> prompt. That looks better.

> 2d
> ,s/user=> /

> ,p
(* 2 21)
42

So now I can write a function, there it is, and then send those lines to the REPL. So far so good, you can see Clojure defined the var.

So now I can switch to the REPL to try it out and… oh oh, this REPL isn’t keeping any state.

> a
(defn qsort [xs]
  (if (apply < xs) xs (recur (shuffle xs))))
.
> w ! clj
Clojure 1.9.0
user=> #'user/qsort
user=>
29
> !clj
Clojure 1.9.0
user=> (qsort [7 3 5])
CompilerException java.lang.RuntimeException: Unable to resolve symbol: qsort in this context, compiling:(NO_SOURCE_PATH:1:1)
user=>

That makes sense in a way, each call to clj starts a new process, so all state is lost. But this is easily fixable thanks to Clojure’s socket REPL. Let’s add a script to launch it.

Create “src”, and “bin” directories, one for Clojure code, and one for scripts. Save the code you have so far, and then empty ed’s buffer, add a shebang, and the clj invocation, pasing it the socket repl options. Now write the file and make it executable.

> ! mkdir src bin
> 4,5w src/quicksort.clj
> ,d
> f bin/socket_repl
> a
#!/bin/bash
clj -J-Dclojure.server.repl="{:port 5555 :accept clojure.core.server/repl}"
.
> w
88
> ! chmod +x %
!
>

The second script connects to it with netcat so you get an actual REPL. Save it to bin/repl, and flip on the executable bit.

> ,d
> a
#!/bin/bash
nc localhost 5555
.
> w bin/repl
30
> ! chmod +x bin/repl
!

Finally create a wrapper to invoke “modern ed”. rlwrap will provide read-line capabilities, allowing for line editing and command history. The -p'> ' option provides a prompt, your modem is probably fast enough to handle those extra bytes, and finally add bin to the $PATH so it’s easy to invoke helper functions like repl.

> ,d
> a
#!/bin/bash
PATH=`pwd`/bin:$PATH rlwrap -m ed -p'> ' $*
.
> w bin/edit
> ! chmod +x bin/edit

Alright, time to get cracking. In a separate terminal run bin/socket_repl, and run ed using bin/edit.

Now you can get a Clojure repl with ! repl. Eat your heart out CIDER.

$ bin/edit
> ! repl
user=> (+ 1 1)
2
user=> !
>

Let’s get back to our “quicksort”, edit the file and add a namespace declaration. Now let’s pipe this to the REPL, and then switch to the REPL and try it out. Nice!

Although, this REPL should be in the quicksort namespace now, but since we get a new socket REPL every time it always switches back to user. Let’s fix that.

> e src/quicksort.clj
> 0a
(ns quicksort)
.
> w ! repl
user=> 78
> ! repl
user=> (quicksort/qsort (shuffle (range 10)))
[0 1 2 3 4 5 6 7 8 9]

First change the socket repl to run on port 5554, and then create a new script called repl_glue. This one uses a glorious hack of my own devising which I call The Netcat FIFO Sandwich. The first netcat runs in a loop, accepting connections one after the other. The second netcat connects once to Clojure’s socket REPL. Whatever comes out of the first gets forwarded to the second, and through a named pipe the result gets piped back.

So it’s kind of like a demultiplexer, it takes multiple incoming connections, and pipes them all to a single outgoing connection.

Now restart the socket_repl script, cause it’s listening on another port now, and then open another terminal, and start bin/repl_glue.

> e bin/socket_repl
> s/5555/5554
> w
> ,d
> f bin/repl_glue
> a
#!/bin/bash
[[ -p .repl_fifo ]] || mkfifo .repl_fifo
nc -k -l localhost 5555 < .repl_fifo | nc localhost 5554 > .repl_fifo
.
> w
124
> ! chmod +x %

The repl script still works unmodified, but now each time you run it it reconnects to the same socket REPL, so state like the current namespace is preserved.

That’s already some solid tooling we have here, but shell scripts will only get you so far. To implement paredit stuff like “slurp” we’ll need a bit of help from Clojure itself.

> ,p
(ns quicksort)

(defn qsort [xs]
  (if (apply < xs) xs (recur (shuffle xs))))
> w !repl
user=> 78
> !repl
nil
quicksort=> quicksort=> #'quicksort/qsort
quicksort=> (qsort [7 5 3])
[3 5 7]
quicksort=> !

Create a user.clj file where you can put REPL helpers. Remember that earlier when we did an “eval-and-insert” we got those pesky REPL prompts. Turns out the regular REPL isn’t that great for tooling. Clojure’s socket REPL is pretty flexible though, so let’s add our own “exec” handler that simply reads a form, evaluates it, and exits.

Now you can add a “slurp” function that reads two forms, adds the second into the first, and pretty prints the result.

> ,d
> f src/user.clj
> a
(ns user
  (:require [clojure.pprint :as pprint]))

(defn exec []
  (eval (read)))

(defn slrp []
  (pprint/with-pprint-dispatch pprint/code-dispatch
    (pprint/pprint (concat (read) [(read)]))))
.
> w
197

Edit the socket_repl script again, adding a second socket REPL handler on port 6666 using user/exec as the :accept function.

The t command does a “transfer”, basically a copy-paste, then it’s just a matter of adding a trailing backslash, changing the port and function name, and removing the initial clj from the last line.

Save that, and restart the socket repl again.

> e bin/socket_repl
88
> ,n
1       #!/bin/bash
2       clj -J-Dclojure.server.repl="{:port 5554 :accept clojure.core.server/repl}"
> 2t2
> ,p
#!/bin/bash
clj -J-Dclojure.server.repl="{:port 5554 :accept clojure.core.server/repl}"
clj -J-Dclojure.server.repl="{:port 5554 :accept clojure.core.server/repl}"
> 2s/$/ \\
clj -J-Dclojure.server.repl="{:port 5554 :accept clojure.core.server/repl}" \
> 3s/clj/   /
> 3s/5554/6666
    -J-Dclojure.server.repl="{:port 6666 :accept clojure.core.server/repl}"
> 3s/clojure.core.*repl/user\/exec
    -J-Dclojure.server.repl="{:port 6666 :accept user/exec}"
> w

Finally you create the bin/slurp script that will act as an ed “macro”. It first echos (user/slrp), which will be evaluated by the exec function. We then pipe standard in into the REPL, so slrp can read the forms it will work on.

> ,d
> f bin/slurp
> a
#!/bin/bash
{ echo '(user/slrp)' ; cat - ; } | nc localhost 6666
.
> w

Let’s try that out! Open src/quicksort.clj again, and let’s add a :require stanza to the namespace declaration. Add it on line 1, so that it sits outside the ns form. Write the file, and delete those two lines from the buffer, but don’t save.

With sed (or “streaming ed”) you can print those two lines and pipe them to slurp, and finally “read” the result into the start of the buffer.

And there you have it, paredit for ed! Implementing the rest of paredit is left as an exercise to the viewer.

> e src/quicksort.clj
78
> 1,3p
(ns quicksort)

(defn qsort [xs]
> 1a
(:require [clojure.string :as str])
.
> w
114
> 1,2d
> 0r ! sed -n 1,2p % | slurp
 sed -n 1,2p src/quicksort.clj | slurp
53
> ,p
(ns quicksort
  (:require [clojure.string :as str]))

(defn qsort [xs]
  (if (apply < xs) xs (recur (shuffle xs))))
> w
116

That’s it, I hope you enjoyed watching this episode as much as I did making it. While this episode is very much intended as a joke, I think there are actually some valuable lessons to take home.

One is that ed is not dead! The ed1conf twitter account has over 800 followers, and there’s an extended version of ed that’s also a web browser, called edbrowse. It was originally developed for blind users, but its scriptability has meant it’s found uses far beyond its initial user base.

This episode forced me to finally really learn ed, but I was a little stunned when not longer after I caught myself actually using it. It’s become just another tool in my Unix toolbox.

And that brings me to the real takeaway for me, the power of small composable tools: ed itself, Clojure’s socket REPL, Netcat, rlwrap, sed, and LISP-y building blocks like read and eval. They all serve as a reminder of the Unix philosophy of doing one thing well, and playing well together, and while forcing everything into a straightjacket of files and character streams can be frustratingly limiting, having such a uniform protocol does allow for things to be recombined in innovative and surprising ways. All it takes is a well stocked toolbox and a creative mind, and if only a single one of the incantations I showed becomes part of your own lexicon I will be more than happy.

I also see this as a reminder not to obsess over tools. Much of Unix itself was written with nothing but ed, and while tools grant us great powers, we shouldn’t forget that they are just that, tools. They don’t define us or our skills, and so we shouldn’t fall into the trap of letting our tools define us, or letting them give us a sense of superiority. The same is true of the languages we program in, and the tribal sense of belonging they provide.

That’s all I have, thanks for watching, and happy April Fools day.

Lambda Island Episodes are clear and focused videos demonstrating practical uses of Clojure, a modern LISP for the JVM. They are densely packed with information, but easy to follow, and will not just make you a better Clojure programmer, but a better programmer all around.

Visit lambdaisland.com for full details, including information on individual pricing and attractive team packages.

The Ultimate Dev Setup