Inside Docker I
Overview
This is an atypical introduction to docker. It aims to learn more about the internals of lxc and aufs by experimenting and analyzing what docker is doing behind the scenes. The inspiration for this post comes from my favourite article from sysadvent - Down the ls Rabbit Hole. It reflects how I like to try to get understanding of various systems from a black box/trace approach before diving into the code.
Docker has had a lot of attention, and I wanted to get a deeper understanding of how it works. I’m starting with pretty minimal knowledge on how docker works, I’ve done the introduction tutorial and attended a Meetup where I saw a 101. I know it uses lxc and aufs under the hood but I’ve not done much with those at all. I wanted to see if we can understand what’s going on using standard Linux system tools.
Requirements
Pre-requisites - install virtualbox and vagrant as per the docker installation.
Create a clone of the main docker repo, we won’t use it right away but in a later post I intend to examine the source some more.
git clone https://github.com/dotcloud/docker.git
cd docker
vagrant up
If you’ve not used docker before take some time to run the hello world example. I’m currently assuming you’re running as root on the vagrant image (eg sudo -s).
docker run ubuntu /bin/echo hello world
hello world
Going deeper
Lets take a closer look at the Hello World example by using our old friend strace.
strace -e trace=file,network,write -s2048 -tt -fF -o /tmp/docker.out docker run ubuntu /bin/echo hello world
This is what I get:
1782 10:45:58.022405 execve("/usr/bin/docker", ["docker", "run", "ubuntu", "/bin/echo", "hello", "world"], [/* 20 vars */]) = 0
1782 10:45:58.135032 open("/proc/sys/net/core/somaxconn", O_RDONLY|O_CLOEXEC) = 3
1782 10:45:58.135549 read(3, "128\n", 4096) = 4
1782 10:45:58.136000 read(3, "", 4092) = 0
1782 10:45:58.136823 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
1782 10:45:58.137154 setsockopt(3, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
1782 10:45:58.137356 bind(3, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
1782 10:45:58.138761 socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
1782 10:45:58.139251 setsockopt(4, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
1782 10:45:58.139782 bind(4, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
1782 10:45:58.272733 stat("/usr/local/sbin/docker", 0xc2000d8000) = -1 ENOENT (No such file or directory)
1782 10:45:58.273302 stat("/usr/local/bin/docker", 0xc2000d8090) = -1 ENOENT (No such file or directory)
1782 10:45:58.274013 stat("/usr/sbin/docker", 0xc2000d8120) = -1 ENOENT (No such file or directory)
1782 10:45:58.274566 stat("/usr/bin/docker", {st_mode=S_IFREG|0755, st_size=5770760, ...}) = 0
1782 10:45:58.275128 stat("/usr/local/sbin/docker", 0xc2000d8240) = -1 ENOENT (No such file or directory)
1782 10:45:58.275643 stat("/usr/local/bin/docker", 0xc2000d82d0) = -1 ENOENT (No such file or directory)
1782 10:45:58.276159 stat("/usr/sbin/docker", 0xc2000d8360) = -1 ENOENT (No such file or directory)
1782 10:45:58.276577 stat("/usr/bin/docker", {st_mode=S_IFREG|0755, st_size=5770760, ...}) = 0
1782 10:45:58.278058 stat("/home/vagrant/.dockercfg", 0xc2000d8480) = -1 ENOENT (No such file or directory)
1782 10:45:58.279033 socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
1782 10:45:58.279570 setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
1782 10:45:58.281848 connect(3, {sa_family=AF_FILE, path="/var/run/docker.sock"}, 23) = 0
1782 10:45:58.282319 getsockname(3, {sa_family=AF_FILE, NULL}, [2]) = 0
1782 10:45:58.282688 getpeername(3, {sa_family=AF_FILE, path="/var/run/docker.sock"}, [23]) = 0
1782 10:45:58.283017 write(3, "POST /v1.4/containers/create HTTP/1.1\r\nHost: \r\nUser-Agent: Docker-Client/0.5.3\r\nContent-Length: 335\r\nContent-Type: application/json\r\n\r\n{\"Hostname\":\"\",\"User\":\"\",\"Memory\":0,\"MemorySwap\":0,\"CpuShares\":0,\"AttachStdin\":false,\"AttachStdout\":true,\"AttachStderr\":true,\"PortSpecs\":null,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":null,\"Cmd\":[\"/bin/echo\",\"hello\",\"world\"],\"Dns\":null,\"Image\":\"ubuntu\",\"Volumes\":{},\"VolumesFrom\":\"\",\"Entrypoint\":[],\"NetworkDisabled\":false}", 470) = 470
1782 10:45:58.284722 read(3, "HTTP/1.1 201 Created\r\nContent-Type: text/plain; charset=utf-8\r\nContent-Length: 21\r\nDate: Sun, 25 Aug 2013 10:45:58 GMT\r\n\r\n{\"Id\":\"6b4b8793687d\"}", 4096) = 143
1782 10:45:58.286191 socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
1782 10:45:58.286529 setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
1782 10:45:58.287292 connect(3, {sa_family=AF_FILE, path="/var/run/docker.sock"}, 23) = 0
1782 10:45:58.287629 getsockname(3, {sa_family=AF_FILE, NULL}, [2]) = 0
1782 10:45:58.287989 getpeername(3, {sa_family=AF_FILE, path="/var/run/docker.sock"}, [23]) = 0
1782 10:45:58.288605 write(3, "POST /v1.4/containers/6b4b8793687d/start HTTP/1.1\r\nHost: \r\nUser-Agent: Docker-Client/0.5.3\r\nContent-Length: 35\r\nContent-Type: application/json\r\n\r\n{\"Binds\":null,\"ContainerIDFile\":\"\"}", 181) = 181
1782 10:45:58.290946 read(3, 0xc200132000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
1782 10:45:58.293395 read(3, "HTTP/1.1 204 No Content\r\nContent-Type: text/plain; charset=utf-8\r\nContent-Length: 0\r\nDate: Sun, 25 Aug 2013 10:45:58 GMT\r\n\r\n", 4096) = 124
1782 10:45:58.294246 socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
1782 10:45:58.294400 setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
1782 10:45:58.295089 connect(3, {sa_family=AF_FILE, path="/var/run/docker.sock"}, 23) = 0
1782 10:45:58.295432 getsockname(3, {sa_family=AF_FILE, NULL}, [2]) = 0
1782 10:45:58.295774 getpeername(3, {sa_family=AF_FILE, path="/var/run/docker.sock"}, [23]) = 0
1782 10:45:58.296514 write(3, "POST /v1.4/containers/6b4b8793687d/attach?logs=1&stderr=1&stdout=1&stream=1 HTTP/1.1\r\nHost: \r\nUser-Agent: Docker-Client/0.5.3\r\nContent-Length: 0\r\nContent-Type: plain/text\r\n\r\n", 174) = 174
1782 10:45:58.297613 read(3, 0xc200135000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
1782 10:45:58.298491 read(3, "HTTP/1.1 200 OK\r\nContent-Type: application/vnd.docker.raw-stream\r\n\r\n", 4096) = 68
1782 10:45:58.298962 write(1, "", 0) = 0
1782 10:45:58.299223 read(3, 0xc200135000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
1782 10:45:58.301381 read(0, <unfinished ...>
1786 10:45:58.463570 read(3, "hello world\n", 4096) = 12
1786 10:45:58.464443 write(1, "hello world\n", 12) = 12
1786 10:45:58.465872 read(3, 0xc200135000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
1788 10:45:58.475465 read(3, "", 4096) = 0
As we can see we have a REST API over a unix socket. The REST API is lnked to from the docs home page.
The API is fully documented, from the strace output we can see that we’re using 1.4
Sadly this means we can’t use curl in the default configuration. You can enable an HTTP connection via a command line flag.
In our git repository lets see if we can figure out what is going on
git grep /usr/bin/docker
hack/release/make.sh: /usr/bin/docker -d
hack/release/make.sh: cp bundles/$VERSION/binary/docker-$VERSION $DIR/usr/bin/docker
packaging/ubuntu/docker.upstart: /usr/bin/docker -d
packaging/ubuntu/lxc-docker.prerm:if [ "`pgrep -f '/usr/bin/docker -d'`" != "" ]; then /sbin/stop docker; fi
Reading the sources we can run a port so lets edit /etc/init/docker.conf and change the script to
/usr/bin/docker -d -H 127.0.0.1:4243
Reload/restart docker via initctl and now we can use curl as we would normally
curl -s http://localhost:4243/v1.4/containers/json | python -m json.tool
[
{
"Command": "/usr/bin/python3 -m http.server",
"Created": 1377427941,
"Id": "a86b320f3c71a7473d06f2184a955d7f76ef3dfb113b0f22b4a8cd8a544f0793",
"Image": "ubuntu:12.10",
"Ports": "8000->8000",
"SizeRootFs": 0,
"SizeRw": 0,
"Status": "Ghost"
}
]