Miguel

On Distributed Stuff

2019-09-17T12:10:31+00:00

– Feel free to correct any inaccuracies –

sources

www.bailis.org/blog/linearizability-versus-serializability/
https://dddpaul.github.io/blog/2016/03/17/linearizability-and-serializability/
https://aphyr.com/posts/313-strong-consistency-models
http://gvsmirnov.ru/blog/tech/2014/02/10/jmm-under-the-hood.html
http://preshing.com/20120625/memory-ordering-at-compile-time/
https://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html
https://www.logicbig.com/tutorials/core-java-tutorial/java-multi-threading/happens-before.html
internet

intro

Today I want to share what I learned about distributed systems.

There are a couple of notions that are intrinsic to any kind of system where the parties have to agree (or not) on values, the so-called consistency models.

Why would I want to talk about this topic, if there are smarter people already explaining it? I feel there’s a huge gap between people that are starting to research these concepts and people that already employ them everyday. Terms like consistency model, compare-and-swap, wait-free, processes, total ordering, histories, etc can be confusing to someone who’s starting, so I’ll try and fill the gap.

Consistency Models

Consistency models define how memory operations apply within a distributed system. Right off the bat, let’s dissect a bit:

Distributed means there’s more than one process (can be a thread, or a different host)
On a distributed system, operations (read or write) may take some time (there’s latency involved), and within that time the system is processing other reads and writes
Consistency models can be enforced by your RDBMS (MySQL) but also by your CPU; it doesn’t necessary apply just to multiple hosts
If consistency models are enforced correctly (ie, if they do what they say the do), they help developers reason correctly about the program

Linearizability

What linearizable means is that when you tell your distributed system to write a value and it ACKs the write, all subsequent reads will get the written value. Basically it says that all reads and writes should appear atomic from the point of view of the processes that read/write the value.

What would NOT be a linearizable execution? Here’s a snippet courtesy of Stack Overflow:

public class Main extends Thread {

    boolean keepRunning = true; //should be volatile

    public void run() {
        while (keepRunning) {}
        System.out.println("Thread terminated.");
    }

    public static void main(String[] args) throws InterruptedException {
        Main t = new Main();
        t.start();
        Thread.sleep(1000);
        t.keepRunning = false;
        System.out.println("keepRunning set to false.");
    }
}

keepRunning is shared between two threads (thread t and main thread)
main writes to shared value keepRunning
because we are not enforcing any consistency model, keepRunning is always true from main’s point of view; so the program won’t stop.

A way to fix this is to make keepRunning volatile, thus enforcing linearizability. Linearizable means that when a write to keepRunning is finished (by executing keepRunning = false), eventually the other thread will see the new value.

The volatile keyword takes care of that because it makes sure reads and writes are atomic. So why wouldn’t setting a variable be atomic?

Short digression on CPU architectures

Modern systems can have multiple CPUs and multiple representations of the same value; typically a CPU will have registers, caches (L1, L2, L3), and main memory (the DIMMs on your motherboard). Registers are much faster to access than L1, L1 is faster than L2 and so on. Besides, a single assembly operation has some latency, so the CPU makes all sorts of tricks to make the code fast, including pipelining, reordering, speculative execution, etc.

Compilers can also reorder generated assembly as long as they deem it safe. So how to fix this conundrum? Well, it depends on the language and the runtime (and the architecture). In case of JVM-based languages, the JVM has your back. The Java Language Specification introduced semantics that define what to expect, the so-called within-thread as-if-serial semantics, meaning that for a single Thread, the program order will be what that Thread observes:

new Runnable(() -> {
  a = 1;
  b = 2;
  c = 0;
  // this thread will observe value in the program order
});

BUT, what happens between threads?

Happens-Before & synchronizes-with

Happens-before is a guarantee that the language & runtime gives about the behaviour of a program. It’s a guarantee about the relation between reads and writes. It basically says that if instruction A has a happens-before relation with B, the results of A will surely be observed in B. It’s not about actually happening before in wallclock time, it’s about what happens-before (A and B) will be observed as if it happened before - the actual assembly can happen in any order.

AFAIK, synchronizes-with is happens-before within Threads.

So, in a single-threaded program, there’s a happens-before guarantee of program order, meaning that each source code line has a happens-before relation with the next line. Happens-before is transitive too.

What about multi-thread programs? The Java spec defines some ways of establishing a happens-before relation:

Single thread and program order

Monitor locks and unlocks (synchronized keyword)

  //T1
  synchronized(lock) {...}//monitor unlock
	
  //T2
  //monitor lock
  synchronized(lock) {...}

Thread start and all of thread’s actions
```
  Thread.start(() -> {a=1; b=2;})
```

Thread join and all of the joining thread’s actions

  //T1
  t = Thread.start(() -> {a=1; b=2;})
	
  //T2
  t.join()

Volatile writes and reads

  //T1
  volatile int a = 0;
  ...
  a = 1
	
  //T2
  if (a == 1) ...

Coming back, so what exactly does the JVM do?

By calling javap on the classfile javap -h -p out.Main you will see this:

  volatile boolean keepRunning;
    descriptor: Z
    flags: ACC_VOLATILE

But can we see the actual assembly?

-XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -Xcomp -XX:CompileCommand=compileonly,*Main

I got sidestepped while trying to see the assembly, had to follow this guide to allow the JVM to print the assembly: https://github.com/liuzhengyang/hsdis

...
0x0000000109ca4697: lock addl $0x0,(%rsp)     ;*putstatic keepRunning
                                                ; - test.Main::main@21 (line 17)

Clojure, Emacs and Cider

2015-11-17T21:10:31+00:00

Or, what you need to know before diving into clojure’s REPL.

Intro

I’ve been coming to clojure lately, and I finally decided to give emacs a go. Unfortunately, setting up emacs for clojure is a PITA, for lots of reasons.

Let’s try to shed some light on that.

Terminology

Clojure: a programming language from the lisp family. Plenty about that on the internet
Leiningen: a clojure runner, not unlike npm or maven or whatever. Actually its different because the config file is written in clojure itself. It’s basically a command-line utility to interact with clojure programs, which include launching a network repl, or nREPL, for interacting with your clojure code
Emacs: an editor written in some kind of lisp. This is important, because you customize emacs by writing lisp (which is a different from clojure)
- major modes: determines the editing behaviour of the current buffer. Only 1 active at any time
- minor modes: alters behaviour in well-known way. Many active at same time
- init.el: where the customization of emacs starts. Should be in ~/.emacs.d/init.el
- Cider: a set of emacs modes for working with clojure Cider makes use of a lib called org.clojure/tools.nrepl

Setting up

There are already alot of tutorials on clojure and leiningen. We’re gonna dive into emacs and cider. For emacs, I followed this guide (actually the braveclojure site is excellent for clojure also), but you will need to tweak some things to make sure everything runs smoothly.

Note 1: the guide doesn’t have the link to the git repo that you should clone, right now its still online at https://github.com/flyingmachine/emacs-for-clojure so you should git clone https://github.com/flyingmachine/emacs-for-clojure ~/.emacs.d. I’m guessing the authors are prepping for the book release so they changed the text to reflect that.

Note 2: When you clone that repo, you are actually getting some packages besides the configuration. If you ls ~/.emacs.d/elpa you will see alot of folders.

Note 3: ~~Don’t forget to edit your ~/.lein/profiles.clj and include cider-nrepl:~~

{:user {:plugins [[cider/cider-nrepl "0.10.0-SNAPSHOT"]}} ;; ignore, see Note 4

Note 4: CIDER used to require you to keep its dependencies in your lein profiles. But CIDER and cider-nrepl have the same version numbers so its easy to keep them in sync. You are putting a hardcoded 0.10-snapshot but CIDER is 0.14.0 on melpa stable and 0.15 on melpa. But CIDER now injects its dependencies so stating them is no longer required, but only serves to explicitly mismatch the frontend (CIDER) from the backend (cider-nrepl)

What it is exactly that I’m doing here?

You’re:

using emacs as an editor
using leiningen with a plugin called cider-nrepl
using an emacs package called cider to send clojure code to the repl and get results, all from inside emacs.

I honestly don’t know exactly how it works because I’m a clojure noob. This is how I think it works.

when you launch a repl (lein repl for instance), and you have the cider-nrepl plugin the plugin hooks onto the repl
when you call cider-jack-in you are launching a repl behind the scenes
cider is “just” for editing files (in emacs lingo, buffers)
cider-nrepl takes care of communicating with the repl, ie, sending clojure code from emacs to the repl and getting results, exceptions, etc

Careful. Emacs uses cider, which is a major-mode. Cider communicates with the nrepl, so cider and nrepl must be version-compatible. You have to be careful with the cider and nrepl version you are using, otherwise you will have warnings and the repl may not work. This means you have to pick cider from the proper repo; if you clone the emacs settings mentioned above, you are already getting a specific cider version

Melpa, Elpa, Marmalade

You can grab packages (plugins) for emacs from different repos. There’s Melpa, Elpa and Marmalade. Here’s a link explaining the differences. Make sure you grab cider from the proper repo, and that the cider (actually cider-nrepl) version matches with the cider-nrepl you configure in your leiningen.

It took me a while to figure out that different repos have different versions (ex: stable vs snapshot). What I actually did was remove all package folders from ~/.emacs.d/elpa and add all repos in ~/.emacs.d/init.el. Emacs will install packages on the first run

;; Define package repositories
(require 'package)
(add-to-list 'package-archives
             '("elpa" . "https://elpa.gnu.org/"))
(add-to-list 'package-archives
             '("marmalade" . "http://marmalade-repo.org/packages/") t)
(add-to-list 'package-archives
             '("tromey" . "http://tromey.com/elpa/") t)
(add-to-list 'package-archives
             '("melpa" . "http://melpa.milkbox.net/packages/") t)
(add-to-list 'package-archives
             '("melpa-stable" . "http://stable.melpa.org/packages/") t)

And below:

;; The packages you want installed. You can also install these
;; manually with M-x package-install
;; Add in your own as you wish:
(defvar my-packages
 '(;; makes handling lisp expressions much, much easier
   ;; Cheatsheet: http://www.emacswiki.org/emacs/PareditCheatsheet
   paredit

   ;; key bindings and code colorization for Clojure
   ;; https://github.com/clojure-emacs/clojure-mode
   clojure-mode

   ;; extra syntax highlighting for clojure
   clojure-mode-extra-font-locking

   ;; integration with a Clojure REPL
   ;; https://github.com/clojure-emacs/cider
   cider

Emacs, Cider, Repl

REPL: When you are using cider, you are actually sending code to the REPL for evaluation, and getting results back. If you change a file, you have to either reload the classpath or send the code again

Let’s get this show on the road.

Start a new project:

$ lein new testdrive
$ ls testdrive

You should see a bunch of files, which include project.clj as well as src and test folders. There should be a core.clj under testdrive/core/ that looks like this:

(ns testdrive.core)
(defn foo
  "I don't do a whole lot."
  [x]
  (println x "Hello, World!"))

Lets go to emacs:

$ emacs .

Navigate to the core.clj file with arrows and enter key. Now we want to fire a repl through cider. Press Alt+X (in emacs lingo, its M-x), then write cider-jack-in and then ENTER.

The repl will start. Now you try to call the foo function from within the repl:

user> (in-ns 'testdrive.core)
#object[clojure.lang.Namespace 0x79a385c4 "testdrive.core"]
testdrive.core> (foo 1)
CompilerException java.lang.RuntimeException: Unable to resolve symbol: foo in this context, compiling:(/tmp/form-init5405367346234268941.clj:1:1) 

WHAAAAAT? I hate you emacs/clojure/repl.

Note: please see the shortcuts at the end on how to navigate emacs, split the frame, etc. Its much easier to split frame and switch between the source and the repl.

Not so fast. When you fire the repl, it just spins a clojure repl, that’s all. You haven’t loaded anything yet (I mean, eval’d). So now you want to load the core.clj onto the repl. There’s a bunch of cider shortcuts for this, I do Ctrl-c Ctrl-k (again, in emacs lingo its C-c C-k) (the Ctrl key is not released). The cider readme states that this means “load (eval) the current buffer”.

Now if you go to the repl again and try

testdrive.core> (foo 1)
1 Hello, World!
nil

YEEAAAH.

If you don’t want to go to the repl, you can write this in core.clj

(foo 1)

And press C-c C-e. You should see the result on the same line:

(foo 1) => nil

Now change the foo function to “return” something (as you know, in clojure the last result is the return value).

(defn foo
  "I don't do a whole lot."
  [x]
  (println x "Hello, World!")
  333)

Try again with C-c C-e:

(foo 1) => nil

WHAAAAAAT? FUUUUUUU!

Remember: the repl has the old function definition. If you change the function, you have to eval it again. Position the cursor at the end of the new definition for foo and eval it through C-c C-f (or any other way fwiw). Now try to call foo again through C-c C-e (so that it shows the result in our buffer):

(foo 1) => 333

Note: when using cider, the position of the cursor is important to determine what is going to be evaluated

Basically, what this allows you is to load and run all the code that you want without ever leaving the repl (emacs in this case). Cider has alot of goodies for interactive development:

C-c , for running tests for a given namespace (eg: if you’re in the testdrive.core buffer/namespace, it will run testdrive.core-test which correspondes to the file testdrive/core_test.clj)
We already seen C-c C-e for eval’ing and displaying the results (kinda like lighttable)
C-c C-x for reloading the classpath (eg: you changed some code, and want to run tests)
and others

Caveat: if you add a dependency on project.clj, you have to restart cider because the classpath is defined only once at startup (its a java thing).

Shortcuts

I recommend the following pages:

Please kindly correct me if you find any bugs. You can raise an issue: https://github.com/mping/mping.github.io/issues And finally, stay tuned. I’ll try and post some more clojure interactive development goodies.

Clojurescript and nodejs

2015-02-13T16:49:49+00:00

Here’s a little snippet for driving express from clojurescript (run in cljs repl):

(ns express_sample
  (:require [cljs.nodejs :as node]))

(def express (node/require "express"))
(def app (express))

(defn -main [& args]
  (doto app
    (.get "/" (fn [req res]
                (.send res "Hello World")))
    (.listen 3000)))

(-main)

Docker experiences

2015-01-26T12:10:31+00:00

Some thoughts on my experiences with docker.

Intro

Lately my friend convinced me to enter the Docker bandwagon, and for the last few days I’ve read about most of the tools on Docker out there. Turns out I should have started from the basics, although tutum.co seems very nice, once it is production-ready. fig and dokku also look cool.

But I started from scratch.

Docker

Docker terminology is a PITA. Here’s my mental guide

images are like vm images, which you customize mostly through a Dockerfile
- these are built in a tree-like, so docker reuses “checkpoints” for images that share something (eg: nginx and mongo based on ubuntu: ubuntu will be shared, so you just download ubuntu one time)
images are normally tagged for easy identification
- an example is the dockerfile/java image that has tags for (oracle open)jdk6/7/8
containers are processes that run images
you can run containers in daemon mode, but for getting started I found it’s easier to run and Ctrl+C to quit it, otherwise you’ll have a bunch of running containers without noticing
you should read the Docker docs carefully.

Why Docker

You’ll get a better answer elswhere, but for me it’s:

isolation (and security, and multiple versions etc etc)
easy replicability(is this an english word?) of images across machines
scalability is nice, but not for small apps like mine
- frameworks like deis, flynn.io, etc promess to easily scale a cluster up and down through the use of technologies such as docker

Docker concepts and tricks

####You should (generally speaking):

either build an image from a Dockerfile, or pull an image from docker index
use a container per service. Example:
- 1 container for mongo
- 1 for nginx
- 1 for java running tomcat or whatever
the rationale for this is, generally speaking, separation of concerns.
so, you should use a different image for each service; although I’m also running eg: wordpress from a single docker image (includes wordpress + mysql). I’m a practical guy.
share data with docker volumes: docker --v src_folder:target_folder_in_container
either link containers to expose eg: mongo to the main app container through docker run --name src and docker run --link src:src or use some kind of Docker ambassador pattern
- A stupid thing is that if you remove an image through its id, the name stays behind. So if you ran an image named weband stopped the container, you will have to do docker rm web before starting another container with the same name
docker run -d dameonizes the container, I only pass the -d flag when I’m sure that the command is correct
If you’re running docker through eg: git hooks, don’t use docker -t, otherwise the script won’t wait for the docker command to finish

####You can (you should know this):

enter a running container through docker exec -t -i CONTAINER_ID /bin/bash
enter a stopped container through docker run ... --entrypoint=/bin/bash ..., regardless of the containers CMD or ENTRYPOINT
parse docker inspectwith jq

Deploying an app

For this example I’ll be using CoreOS on Digital Ocean. Here’s my referrer if you want to give it a try: https://www.digitalocean.com/

My app consists of

a mongodb image
an nginx image
a clojure image, this is essentially an image to run a fatjar

Additionally, I needed

a socat image
a lein image
careful, see http://www.martinklepsch.org/posts/running-a-clojure-uberjar-inside-docker.html
the lein image would later become a leiningen/node/bower/maven image, because my app is clojure on the backend, angular on the frontend with a node/bower toolchain to build the js.

Let’s get these points one by one.

MongoDB

We want to run mongodb, but we want the data out of the container. For that, we’ll use docker volumes. We also want to expose the mongod service to other containers. For that, we’ll use container linking. Your preferred image should expose a volume, check the image’s Dockerfile:

VOLUME ["/data/db"]

By now you should either have pulled a mongo image or created your own Dockerfile and an image based on it.

docker run -P -v /home/mongo/data:/data/db --name mongo some/mongo

If your dockerfile has EXPOSE instructions, the -P will randomly expose the ports on the host container. Don’t worry you only need to know the port if you want to connect to it. If you’re running the latest docker, you can enter this container with docker exec and inspect the db. Otherwise, you can try some tutorials to set up nsenter or simply connect through the exposed ports:

3ce6046e2ee4        mping/mongo:latest        "mongod"               44 hours ago        Up 44 hours         0.0.0.0:49155->28017/tcp, 0.0.0.0:49156->27017/tcp   mongo

In this example you have local ports 49155, 49156 binded to the container’s 28017,27017ports which are mongo ports. I’m guessing the docker way would be to launch another mongo container and connect to the server, but I just use docker exec -it 3ce6046e2ee4 /bin/bash and voilá.

App

Assuming we have a docker image for our app (more on this later), we just need to run it and instruct docker to use the mongo service. Notice that the DATABASE_URL is set to an address called mongo. This will work because when you link docker containers, the /etc/hosts file has an entry that sets the proper ip for the name.

 docker run -p 3000:3000 \
	--name web \
	--link mongo:mongo \
	-h mysite.com \
	-e DATABASE_URL=mongodb://mongo:27017/myapp \
	-e ENV=prod \
	mydocker/image

nginx

Assuming you have a proper nginx image, you just need to run it and link it to the conf files. The interesting bit is setting an upstream to web in sites-available; remember that when you link containers, docker will update the container’s /etc/hosts with the ip for the linked container. So web will resolve to the running container named web.

upstream app_server {
		server web:3000;
}

Afterwards, just run it:

docker run -d -h mysite.com -p 8888:80 -p 4443:443 \
	--link web:web \
	-v `pwd`/nginx/sites-enabled:/etc/nginx/sites-enabled \
	-v `pwd`/nginx/certs:/etc/nginx/certs \
	-v `pwd`/nginx/logs:/var/log/nginx \
	local/nginx

What’s all this hassle for? Well, for the most of it, you get to have docker images that are ready to run and are not specifically configured to a particular configuration. You can spin another mongo or nginx and use it in different projects, as long as the volumes and links are correct.

CoreOS

CoreOS on DO has a fedora image that you can run, which has most of linux utilities (eg: nano). It’s available on /usr/bin/toolbox and it mounts the whole drive as a docker volume under /media/root.

Docs are here: https://coreos.com/docs/cluster-management/debugging/install-debugging-tools/

Clojure

I’m experimenting with lein uberimage, a plugin that generates a docker image for your uberjar. The main advantage of having an image, instead of building a jar and generically running it, is that you get your own app image that can be pushed to a repo, rolled back, etc.

Unfortunately, I hit alot of roadblocks:

The uberimage plugin expects a docker API on localhost:2375; on my dev machine (ubuntu), I had to update the docker daemon to run on 127.0.0.1 aside with the unix socket. Careful not to expose the docker api on all interfaces (don’t use 0.0.0.0)
```
 less /etc/init/docker
 ...
 DOCKER_OPTS="--host=tcp://localhost:2375 --host=unix:///var/run/docker.sock"
 ...
```
If you’re building an image from a container (CoreOS forces you to do everything from a container), you’ll need to communicate with the host docker api to let the leiningen plugin push the image. I had to install a socat image and expose it with a name. This is actually a good practice that allows you to expose the host docker api to a give container, either by exposing the socket through volumes or the http api through the named container docker-http
docker socat

run the image in the background

 docker run -d -p 2375:2375 --volume=/var/run/docker.sock:/var/run/docker.sock --name=docker-http someimageof/lein
 ...

when specifying the docker api, use the named docker: docker run … someimageof/lein uberimage -H http://docker-http:2375 -t mydocker/image

 docker run -i -t --link docker-http:docker-http -v /var/run/docker.sock:/tmp/docker.sock:rw -v /home/core/ezcode.deploy:/project someimageof/lein uberimage -H http://docker-http:2375 -b someimageof/jdk8 -t mydocker/image

But… But…

I lost the whole weekend trying to run a !"#&"@ jar. The fix that worked for me was this:

java -Xmx512m -noverify -jar -Djava.security.egd=file:/dev/urandom target/myjar-standalone.jar

So how did I got there? First I used jps to see where the launching was being stuck:

jps -l
15990 sun.tools.jps.Jps
15980 target/ezcode-0.1.0-standalone.jar

Then I used jstack to inspect the stack of the running java application:

jstack 15980
	...
	java.lang.Thread.State: RUNNABLE
		at java.io.FileInputStream.readBytes(Native Method)
		at java.io.FileInputStream.read(FileInputStream.java:220)
		at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:493)
		at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:117)
		at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
		at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
		- locked <0x00000007d8b0ee08> (a sun.security.provider.SecureRandom)
		at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
		- locked <0x00000007d8b0ee60> (a java.security.SecureRandom)
		at java.security.SecureRandom.next(SecureRandom.java:455)
		at java.util.Random.nextInt(Random.java:189)

Some thread on some secureRandom call or whatever caught my eye. Googling about it, I found this: http://docs.codehaus.org/display/JETTY/Connectors+slow+to+startup which had the workaround at the bottom:

Set the system property: -Djava.security.egd=file:/dev/urandom
Make sure that the java.security file contains this setting:
	securerandom.source=file:/dev/urandom

And this only worked on jdk8. I couldn’t even apply this in jdk7.