so… the past weeks I have been in Google land, specifically: working on Google Compute Engine (GCE). As part of the containerization of IT I started using an Atomic Host on GCE. This was basically and exercise to learn some of the GCE tools. But the important part: it gives a reusable Atomic Host with endless compute power. I put together a basic example of a web server running as a container, orchestrated by kubernetes…
In the upcoming weeks, I will use this type of set up to expolore
- multi instance networking with kubernetes and
- troubleshooting containers, aka. sosreport and docker
After having set up the basic environment for kubernetes and docker, I will explore into options on getting some information (eg logfiles, configurations) of containers. The main focus will be debugging of container configuration and/or containerized applications, so I am not primarily interested in the application’s logfile. And I assume that
/dev/log is bound to the host via
docker -v /dev/log:/dev/log.
First of all, let me give a short overview of some of the options available for “logging with docker”. There has also been a long discussion on the docker-dev mailing list…
- using logstash – not having syslog within the container and utilizing a forwarder
- or fluentd – to parse the docker logs on the host’s disk
- or shipping it off to logentries
- or have a volume to bound to store the logs
For the purpose of debugging a container’s configuration or the abnormal termination of an application I will try to gather some information:
- from outside the container, like network configuration, disks, cgroups, other processes, other logs
- from inside the container, docker’s own logfiles of the container
- and from inside the container, like network configuration, cgroups, …
Gathering different kinds of information for an operating system environment has been solved before, and I will choose sosreport as the base for my work. sosreport is included in every major Linux distribution, including Debian GNU/Linux, Fedora and Red Hat Enterprise Linux.
From outside a containers sosreport seems to do a pretty complete job, looking at the
netstat output we can reckon that docker daemon is running,
ip_addr tells us that the docker0 interface has been set up correctly… so, we are set to go. sosreport also contains all the syslog (or systemd-journald nowadays) entries so we could inspect potential problems on dockerd startup.
Having a look at more docker specific information is harder. sos plugins for docker specific informations are targeted for milestone 3.2 or are available via github. However these plugins just gather some basic information. I have started opening requests for enhancements with the sos upstream project to get some more detailed docker information.
Taking a peek inside the container is not covered by sosreport by now. In the upcoming days I will explore some options using nsenter and execute sosreport within the container and will followup on this article too.