Debugging

Once you have your application running in Kubernetes, it's important to know how to also debug and inspect it further in Kubernetes.

In this section, we'll learn how to debug applications in Kubernetes using kubectl.

Debugging tools within kubectl

First, let's take a look at what kubectl provides us that we can use to debug our software.

Viewing resources

The primary way to view what resources you have available is to use get and describe commands.

# List all the running Pods
$ kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
demo-747c7c7f78-57tpk   1/1     Running   0          10m
demo-747c7c7f78-lpt7h   1/1     Running   0          10m
demo-747c7c7f78-qx55s   1/1     Running   0          10m

# List all the running Pods that have label app = demo-pod
$ kubectl get pod -l app=demo-pod
NAME                    READY   STATUS    RESTARTS   AGE
demo-747c7c7f78-57tpk   1/1     Running   0          10m
demo-747c7c7f78-lpt7h   1/1     Running   0          10m
demo-747c7c7f78-qx55s   1/1     Running   0          10m

# Watch live updates to the running Pods
$ kubectl get pod -w

# Show more Pod details (including event log)
$ kubectl describe pod demo-747c7c7f78-57tpk
Name:         demo-747c7c7f78-57tpk
Namespace:    tutorial
Priority:     0
...

We can also use this to view other resources too.

# List all deployments
$ kubectl get deployments
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
demo   3/3     3            3           12m

# List all services
kubectl get services
NAME   TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)   AGE
demo   ClusterIP   100.66.240.247    <none>        80/TCP    13m

# Show more details on demo deployment
$ kubectl describe deployment demo
Name:                   demo
Namespace:              tutorial
CreationTimestamp:      Wed, 25 Nov 2020 09:35:57 +0200
Labels:                 app=demo
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=demo-pod
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 1 max surge
Pod Template:
...

There are many more resource types you can view and many of them have short identifiers you can use to reduce typing. Some of these resources will be covered in future training sections. For example, you could list deployments with command kubectl get deployments or kubectl get deploy.

Type	Identifier	Short identifier
Pod	pods	po
Deployment	deployments	deploy
Service	services	svc
Secret	secrets	secret
ConfigMap	configmaps	cm
DaemonSet	daemonsets	ds
StatefulSet	statefulsets	sts
Role	roles	role
RoleBinding	rolebindings	rolebinding

We can also fetch the information in different formats using the -o parameter in the get command.

# Get the demo deployment in YAML format
$ kubectl get deploy demo -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
...

# Get all the Pod IP addresses for Pods with label app = demo-pod
$ kubectl get pod -l app=demo-pod -o 'jsonpath={ .items[*].status.podIP }'
100.101.38.203 100.103.228.203 100.108.251.12

Additionally, with Deployments, we can check if Kubernetes has successfully launched all the Pods in the Deployment.

$ kubectl rollout status deploy demo
deployment "demo" successfully rolled out

Logs

In Kubernetes, it's recommended to output all logs to the container application's STDERR or STDOUT streams. We can view these streams using the kubectl logs command.

# Show the logs for Pod named demo-747c7c7f78-57tpk
$ kubectl logs demo-747c7c7f78-57tpk
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
100.101.38.1 - - [25/Nov/2020:07:36:11 +0000] "GET / HTTP/1.1" 200 612 "-" "kube-probe/1.19" "-"
...

Occasionally, we want to view logs from multiple Pods such as all the Pods from a single Deployment. For this purpose, we can use the label filter -l.

# Show all the logs from Pods with label app = demo-pod
$ kubectl logs -l app=demo-pod

We can also limit how many log entries to print out.

# Show the last 3 log lines from each Pod with label app = demo-pod
$ kubectl logs --tail 3 -l app=demo-pod

If we want to continuously view new logs as they appear, we can use the -f flag (short for "follow"). It works similar to how the tail -f command works with files.

# Continuously show the new logs from all Pod with label app = demo-pod
$ kubectl logs -f -l app=demo-pod

In the future training sections, we'll learn how we can view these logs in a centralized logging platform.

Pod CPU and memory usage

We can view the CPU and memory usage of Pods using kubectl top command.

# CPU and memory usage for all Pods
$ kubectl top pod

# CPU and memory usage for Pod matching label app = demo-pod
$ kubectl top pod -l app=demo-pod

# CPU and memory usage for a single Pod
$ kubectl top pod demo-747c7c7f78-57tpk

In the future training sections, we'll learn how we can set limits for CPU and memory usage as well as view them in Grafana.

Running one-off containers

Occasionally, you may want to run one-off containers in the Kubernetes cluster to test out things in ad-hoc way.

For example, we could also check how our demo deployment responds to HTTP requests using a curl container.

$ kubectl run --rm -it --restart=Never --image=curlimages/curl my-curl -- http://demo/
<!DOCTYPE html>
<html>
...
pod "my-curl" deleted

Here's what the command parameters do:

--restart=Never: Once the containers in the Pod stop, the Pod is not restarted.
--rm: Once the Pod is stopped, it gets removed automatically.
-it: We enable STDIN and TTY for the Pod, so that we can receive the command output directly to our terminal window.
--image=: Select the Docker container image we want to run.
--: Separator between the Pod spec arguments and the container arguments

We could also check how domain names are resolved inside the cluster using netshoot.

$ kubectl run --rm -it --restart=Never --image=nicolaka/netshoot --command my-nslookup -- nslookup example.org
Server:     100.64.0.10
Address:    100.64.0.10#53

Non-authoritative answer:
Name:   example.org
Address: 93.184.216.34
Name:   example.org
Address: 2606:2800:220:1:248:1893:25c8:1946

pod "my-shell" deleted

Here the --command flag is used for overriding the container command (i.e Docker ENTRYPOINT) which is set to /bin/sh by default. Instead, the first parameter after -- is used as the container command.

Executing arbitrary commands in Pods

Sometimes, you may need to run some commands in Pods for diagnostic purposes. We can use kubectl exec to achieve this.

WARNING! Only use exec commands for diagnostic purposes! Never rely on them for production use.

For example, we could check out the files we serve from the demo application.

$ kubectl exec -it demo-747c7c7f78-lpt7h nginx -- ls /usr/share/nginx/html
50x.html  index.html

$ kubectl exec -it demo-747c7c7f78-lpt7h nginx -- cat /usr/share/nginx/html/index.html
<!DOCTYPE html>
<html>
...

Here's what the parameters do:

-it: We enable STDIN and TTY for the Pod, so that we can receive the command output directly to our terminal window.
demo-747c7c7f78-lpt7h: Name of the Pod to run the command in.
nginx: Name of the container in the Pod to run the command in. This refers to name field used in the containers section of the Pod specification. This is only needed in cases where there's multiple containers running in the Pod. When there's only container in the Pod, you can leave this out.

Another handy use for kubectl exec is to check what environment variables are set in the container using the env command.

$ kubectl exec -it demo-747c7c7f78-lpt7h nginx -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=demo-747c7c7f78-lpt7h
TERM=xterm
...

Warning

Shell access in Pods is generally considered a security risk and therefore it's recommended to be avoided.

Accessing a Pod from a local port (i.e. port forwarding)

Once in a while, you may want to open a direct access to a Pod from your computer. You can achive this using port forwarding. This can be useful when you want to interact with your service without exposing it using Ingress.

WARNING! Only use port forwarding commands for diagnostic purposes! Never rely on them for production use.

Let's set up a port forward from our local port 9999 to one of the demo app Pods port 8080 using the kubectl port-forward command. If port 9999 is not open in your computer, feel free to select any other open port.

$ kubectl port-forward pod/demo-747c7c7f78-lpt7h 9999:8080
Forwarding from 127.0.0.1:9999 -> 8080
Forwarding from [::1]:9999 -> 8080

Let's leave the session open, and open a new terminal window where we can test the connection with curl. Alternatively, you can just open the address http://localhost:9999/ in your web browser.

$ curl localhost:9999
<!DOCTYPE html>
<html>
...

You can stop the port forward session with key combination Ctrl+C (or Ctrl+Break in Windows).

We can also open port forward sessions on the service endpoints, too. Let's set up a port forward from our local port 9999 to the demo service port 80.

$ kubectl port-forward svc/demo 9999:80
Forwarding from 127.0.0.1:9999 -> 8080
Forwarding from [::1]:9999 -> 8080

We can again test this with the curl command or web browser as shown above. Once you're done, you can stop the port forward session with key combination Ctrl+C (or Ctrl+Break in Windows). Note that the port forward session might automatically disconnected after a while, if there's no data moving through it.

Finding documentation

Kubectl can also be used for browsing various pieces of documentation.

YAML manifests

If you need explanation for Kubernetes YAML manifest fields, you can use the kubectl explain command.

# Explain what fields a Pod manifest has
kubectl explain pod

# Explain what fields a Pod container has
kubectl explain pod.spec.containers

# Explain what fields a Deployment spec has
kubectl explain deployment.spec

Alternatively, you can find the same details from Kubernetes API reference page.

kubectl commands

Each command has extensive help documentation that you can print out by adding the -h flag to the command. If you need help for kubectl command, just add the flag to the end of the command. Here's few examples:

$ kubectl get -h
$ kubectl describe -h
$ kubectl create -h
$ kubectl create deployment -h

Things that are often set up incorrectly

When working with Kubernetes, these are the things that are often accidentally set up incorrectly. If your application never starts, keeps restarting, or you're not able to communicate with your app, it's most likely because of these.

Docker container / Pod

Container doesn't start listening to the ports you expected it to.
Container doesn't have the permission to run as non-root user.

Deployment

Pod template labels and selector match labels don't match together.
Container image name points to the incorrect or non-existing image
Liveness or readiness probes point to the wrong endpoint.
You have not given enough memory or CPU to the container.
The Secrets and ConfigMaps the Deployment tries to use don't exist or don't contain the expected key-value pairs.

Services

Selector doesn't match the desired Pods.
Target ports don't match the ports that the Pods use.

Ingress

Wrong Ingress class is used. This is often the case when there's multiple Ingress implementations in the cluster.
The backend service name and/or port don't match the desired Service name and/or port.

Checklist for troubleshooting

Now that we have a list of tools we can use to debug our applications, we just need to know when to use them. The more and more you use Kubernetes the better you know when to use which command. Before you get there, you can get started with this checklist.

Can I access my service successfully via Ingress?
- Open the browser or use curl to contact your app's hostname you set up in Ingress.
- If you get a 404, check the Ingress resource's ingressClassName and host.
- If you get a 503, check the Ingress resource's backend service name and port.
Can I access my Service?
- Run ephemeral container and try to curl your service IP
- Check that Service is mapped correctly to the deployment
Has my deployment rolled out correctly?
- Does kubectl rollout status deploy myapp return OK?
- Do you have Pods ready in kubectl get deploy myapp list?
- Use kubectl describe on the failing Pods to investigate what's wrong.
- Use kubectl logs on the failing Pods to investigate if there's any error logs printed when the Pods start.
- Use kubectl exec to view that environment variables and files are in the correct place.
Can I access my service successfully via port forward?
- Open a port forward to your app's service endpoint or Pod e.g. kubectl port-forward svc/myapp 9999 80
- Open the browser or use curl to contact the local port.
- Check that the ports and selector are configured correctly in the Service resource.
- Check that the app in the container starts listening the correct port.

A lot of times, application issues appear as errors in the Pods. When you view a Pod status using kubectl get or kubectl describe, you might see one of these statuses.

Status	What it means	What to do about it
ErrImagePull	Kubernetes failed to pull a Docker image for the Pod	Check that the Docker image name and tag is correct for your Deployment.
Failed	Your Pod failed to start or keep up	Check the logs for Pod to figure out why it fails: `kubectl logs <pod-name>`
CrashLoopBackoff	Your Pod has failed to start after multiple restarts	See above
Pending	Kubernetes failed to find a node that has enough resources to run your Pod	Either lower the resource requirements or create more Kubernetes worker nodes.
CreateContainerConfigError	Most commonly, you tried to reference a ConfigMap or a Secret that doesn't exist.	Check the ConfigMaps and Secrets referred in your Deployment, and make sure they exist in Kubernetes. Check for possible typos.
ContainerCreating	The Pod is still starting up. However, there's a few things that might prevent it from starting up properly. For example, if you try to mount volumes (ConfigMaps, Secrets, disks) that don't exist, your Pod will be stuck in this state.	Check that the volumes the Pod tries to mount exist.