on debugging and gathering information about containers

After having set up the basic environment for kubernetes and docker, I will explore into options on getting some information (eg logfiles, configurations) of containers. The main focus will be debugging of container configuration and/or containerized applications, so I am not primarily interested in the application’s logfile. And I assume that /dev/log is bound to the host via docker -v /dev/log:/dev/log.

First of all, let me give a short overview of some of the options available for “logging with docker”. There has also been a long discussion on the docker-dev mailing list

For the purpose of debugging a container’s configuration or the abnormal termination of an application I will try to gather some information:

  • from outside the container, like network configuration, disks, cgroups, other processes, other logs
  • from inside the container, docker’s own logfiles of the container
  • and from inside the container, like network configuration, cgroups, …

Gathering different kinds of information for an operating system environment has been solved before, and I will choose sosreport as the base for my work. sosreport is included in every major Linux distribution, including Debian GNU/Linux, Fedora and Red Hat Enterprise Linux.

From outside a containers sosreport seems to do a pretty complete job, looking at the ps and netstat output we can reckon that docker daemon is running, ip_addr tells us that the docker0 interface has been set up correctly… so, we are set to go. sosreport also contains all the syslog (or systemd-journald nowadays) entries so we could inspect potential problems on dockerd startup.

Having a look at more docker specific information is harder. sos plugins for docker specific informations are targeted for milestone 3.2 or are available via github. However these plugins just gather some basic information. I have started opening requests for enhancements with the sos upstream project to get some more detailed docker information.

Taking a peek inside the container is not covered by sosreport by now. In the upcoming days I will explore some options using nsenter and execute sosreport within the container and will followup on this article too.

stay tuned…


Posted at 16° 35' on 2014-08-13 | Filed Under: docker, technology | read on

Setting up my Kubernetes Environment

To set up the Kubernetes environment I described earlier, I deployed a docker registry on raw-0, all data is contained within a volume container:

git clone
cd Fedora-Dockerfiles/registry
docker build --rm --tar="goern/registry" .
docker run --name=registry-data -v /var/lib/docker-registry fedora:20 true # to hold our data
docker run --name=registry -d -p 5000:5000 --volumes-from=registry-data goern/registry

See also at Scott’s howto. Testing if the registry and dns and docker works via

[root@raw-2 ~]# docker pull raw-0.localdomain:5000/goern/mongo Pulling repository raw-0.localdomain:5000/goern/mongo a79fce803e4c: Download complete 511136ea3c5a: Download complete c69cab00d6ef: Download complete 88b42ffd1f7c: Download complete 8d8d8f2d03f7: Download complete a3c6facc68a0: Download complete daed86b3b2da: Download complete 07bbb4fa70f5: Download complete e6c22cdf122e: Download complete [root@raw-2 ~]#

The next thing I deployed was a simple mongo container I build and pushed to my local registry using the same pattern as above.


[root@raw-0 ~]# cat mongo.json
 "id": "mongo",
 "desiredState": {
 "manifest": {
 "version": "v1beta1",
 "id": "mongo-1",
 "containers": [{
 "name": "master",
 "image": "raw-0.localdomain:5000/goern/mongo",
 "ports": [{
 "containerPort": 27017,
 "hostPort": 27017
 }, {
 "containerPort": 28017,
 "hostPort": 28017
 "labels": {
 "name": "apache"
[root@raw-0 ~]# kubernetes-kubecfg -c mongo.json create pods
I0809 12:03:27.817122 03020 request.go:220] Waiting for completion of /operations/2

Name Image(s) Host Labels
———- ———- ———- ———-
mongo raw-0.localdomain:5000/goern/mongo / name=apache

[root@raw-0 ~]#

Posted at 11° 53' on 2014-08-11 | Filed Under: docker, kubernetes, technology | read on