on debugging and gathering information about containers

After having set up the basic environment for kubernetes and docker, I will explore into options on getting some information (eg logfiles, configurations) of containers. The main focus will be debugging of container configuration and/or containerized applications, so I am not primarily interested in the application’s logfile. And I assume that /dev/log is bound to the host via docker -v /dev/log:/dev/log.

First of all, let me give a short overview of some of the options available for “logging with docker”. There has also been a long discussion on the docker-dev mailing list

For the purpose of debugging a container’s configuration or the abnormal termination of an application I will try to gather some information:

  • from outside the container, like network configuration, disks, cgroups, other processes, other logs
  • from inside the container, docker’s own logfiles of the container
  • and from inside the container, like network configuration, cgroups, …

Gathering different kinds of information for an operating system environment has been solved before, and I will choose sosreport as the base for my work. sosreport is included in every major Linux distribution, including Debian GNU/Linux, Fedora and Red Hat Enterprise Linux.

From outside a containers sosreport seems to do a pretty complete job, looking at the ps and netstat output we can reckon that docker daemon is running, ip_addr tells us that the docker0 interface has been set up correctly… so, we are set to go. sosreport also contains all the syslog (or systemd-journald nowadays) entries so we could inspect potential problems on dockerd startup.

Having a look at more docker specific information is harder. sos plugins for docker specific informations are targeted for milestone 3.2 or are available via github. However these plugins just gather some basic information. I have started opening requests for enhancements with the sos upstream project to get some more detailed docker information.

Taking a peek inside the container is not covered by sosreport by now. In the upcoming days I will explore some options using [nsenter][11] and execute sosreport within the container and will followup on this article too.

stay tuned…

 

[11]: http://man7.org/linux/man-pages/man1/nsenter.1.html