Debugging most Linux programs usually involves checking log files, which can be a complex process. When running in a containerized environment under Docker though, you’ll need to use more specific tools to debug apps in production.
Where Are Logs Stored?
The simple answer is that Docker stores container logs in its main storage location, /var/lib/docker/. Each container has a log specific to their ID (the full ID, not the shortened one that’s usually displayed) and you can access it like so:
/var/lib/docker/containers/ID/ID-json.log
That’s where they’re stored, but since they’re in a JSON format, they’re not easily readable, and having to use the full container ID is annoying. Docker provides a built in command for viewing them:
docker logs -f e4bd48ef3103
Here, the -f flag will keep the prompt open and “follow” any new entries in the file. You can also --tail the file, or use --timestamps to display the log time, or use --until and --since to filter based on time.
If you’re using Docker Compose, you can use the log command from that to view all logs easily:
docker-compose logs
However, one thing you’ll notice is that this is STDOUT and STDERR, which is useful for lots of things, but only shows you the console output of the entrypoint specified by “CMD” in the Docker file. Many apps have their own dedicated logging systems, which often log to files like /var/log/nginx/access.log. Accessing logs like this is still possible from the host side through Docker.
Viewing Logs From Apps Inside Containers
Depending on the container, this might not be necessary. For example, the default NGINX container is set up to send its Docker logs to STDOUT to make log checking easier. It does this with a symlink from /dev/stdout to the log file, and you can set up something similar for your containers.
RUN ln -sf /dev/stdout /var/log/nginx/access.log && ln -sf /dev/stderr /var/log/nginx/error.log
But, if you want to check out specific files inside a container, you can do so. Docker provides the exec -it command to allow you to run any command inside any running Docker process. Using this, you can tail a log file inside a Docker container:
docker exec -it e4bd48ef3103 tail -f log.txt
Because this lets you run any command, you can use journalctl or any other debugging strategies you want, as long as you preface it with docker exec -it. You can even run /bin/bash if you want to hop in and poke around.
A more permanant solution that plays nicer with host services is to use a Docker volume mount. You can bind a directory like /var/log/nginx to a volume that’s visible from the host. First, create a new volume:
docker volume create nginx-logs
And run the container with the --mount:
docker run -d --name devtest --mount source=nginx-logs,target=/var/log/nginx nginx:latest
If you’re using Docker Compose, the process can be automated:
version: "3.0"
services:
web:
image: nginx:latest
ports:
- "80:80"
volumes:
- nginx-logs:/var/log/nginx/
volumes:
nginx-logs:
This way, log files will be directly ingestable by any log aggregation services on the host machine.
Viewing Docker Daemon Logs
If you instead want to view logs specific to the overall Docker service on your server, and not any specific containerized app, you’ll want to view the journalctl logs:
sudo journalctl -fu docker.service
This is where it’s stored on most systems, but it’s in a different spot on some:
- Amazon Linux:
/var/log/docker - CentOS/RHEL:
/var/log/messages | grep docker - macOS:
~/Library/Containers/com.docker.docker/Data/log/vm/dockerd.log - Windows:
AppDataRoamingDockerlogvmdockerd.log
