Overview

• Studying for the Certified Kubernetes Administrator (CKA), one of the Kubernetes certification exams.
• Taking the "Certified Kubernetes Administrator (CKA) with Practice Tests" course on Udemy.
• Notes for future reference.

Course Structure

GitHub - kodekloudhub/certified-kubernetes-administrator-course

1. Introduction
2. Core Concepts
3. Scheduling
4. Logging & Monitoring
5. Application Lifecycle Management
6. Cluster Maintenance
7. Security
8. Storage
9. Networking
10. Design and Install Kubernetes
11. Install "Kubernetes the kubeadm way"
12. End to End Tests on Kubernetes Cluster
13. Troubleshooting
14. Other Topics
15. Lightning Labs
16. Mock Exams
17. Course Conclusion

Notes on important topics in course order.

1. Introduction

Skip.

2. Core Concepts

• Cluster Architecture
  • Master Node
  • ETCD
  • Kube-API Server
  • Kube Controller Manager
  • Kube Scheduler
  • ReplicaSet
  • Deployment
  • Namespace
  • Service
• Worker Node
  • Kubelet
  • Pod
  • Kube Proxy

Master Node

The node responsible for managing and interfacing with the Kubernetes cluster.

Worker Node

The node responsible for running application workloads in the Kubernetes cluster.

ETCD

ETCD itself is a distributed key-value store. In Kubernetes, information about Pods, Nodes, etc., is stored in ETCD. All information retrieved by kubectl get commands comes from the ETCD server.

Kube-API Server

A server that provides an API interface for communication between Kubernetes resources. Kube-API Server behavior during Pod creation:

1. Receives Pod creation API request. Authenticates the user.
2. Validates the API request.
3. Creates the Pod (Node not yet assigned at this point).
4. Notifies ETCD server of Pod creation; ETCD updates data; Kube-API Server notifies user.
5. Kube Scheduler monitors Kube-API Server periodically, detects an unassigned new Pod, decides which Node to place it on, and notifies Kube-API Server.
6. Kube-API Server notifies ETCD of the Node assignment.
7. Kube-API Server tells the Kubelet on the target Node to place the Pod.
8. Kubelet creates the Pod on the Node and instructs the container runtime (Docker) to deploy the application image.
9. Kubelet notifies Kube-API Server of the Pod placement; Kube-API Server notifies ETCD; ETCD updates data.

Kube Controller Manager

A server that manages multiple controllers such as container controllers and node controllers. Controllers continuously check status and take remediation actions.

Example: Node-Controller checks node status every 5 seconds. If a node's heartbeat stops for 40 seconds, it marks the node as Unreachable. If it stays Unreachable for 5 minutes, Pods on that node are deleted and rescheduled on healthy nodes.

Kube Scheduler

A server that decides which Node to place a Pod on. (Note: it is Kubelet's job to actually place the Pod on the Node.)

Kube-Proxy

In a Kubernetes cluster, all Pods can reach all other Pods via the Pod network — a virtual network spanning all nodes. Kube-proxy is a process running on each Node that creates forwarding rules (e.g., IP tables) whenever a new Service is created, enabling traffic to be forwarded from the Service IP to the Pod.

3. Scheduling

Kube Scheduler decides which Node to place a Pod on. Scheduling mechanisms:

• Taints & Tolerations
• Node Selectors
• Node Affinity
• Resource Requirements & Resource Limits
• Special cases:
  • DaemonSets
  • Static Pods
  • Multiple Schedulers

Taints & Tolerations

Taints (set on Nodes) act like insect repellent — Pods cannot be scheduled on Tainted nodes by default. Tolerations (set on Pods) act like protective gear, allowing a Pod to tolerate a Taint and be scheduled on that Node.

kubectl taint nodes <node-name> app=batch:NoSchedule  # Set Taint
kubectl taint nodes <node-name> app=batch:NoSchedule- # Remove Taint

spec:
  tolerations:
  - key: "app"
    operator: "Equal"
    value: "batch"
    effect: "NoSchedule"

Effects: NoSchedule, PreferNoSchedule, NoExecute.

Node Selectors

Schedules Pods to specific Nodes using Labels & Selectors.

kubectl label nodes <node-name> Size=Large

spec:
  nodeSelector:
    Size: Large

Node Affinity

Similar to Node Selectors but supports more complex rules (e.g., NOT, OR operators).

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: Size
            operator: In
            values:
            - Large
            - Small

Types: requiredDuringSchedulingIgnoredDuringExecution, preferredDuringSchedulingIgnoredDuringExecution, requiredDuringSchedulingRequiredDuringExecution.

Resource Requirements & Resource Limits

spec:
  containers:
  - name: app
    image: images.my-company.example/app:v4
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

CPU exceeding limits is throttled. Memory exceeding limits continuously results in Pod deletion.

DaemonSets

Automatically places one Pod on each Node whenever a new Node is added. Use cases: node monitoring, Kube Proxy deployment.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      containers:
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2

Static Pods

Pods that can run on a Worker Node even without a Master Node (no Kube API server). Kubelet periodically checks a specified directory and creates Pods based on YAML definition files found there.

Find the directory with: ps -ef | grep kubelet → look for --config → check staticPodPath.

Note: Static Pods can be viewed with kubectl get pods but cannot be deleted or edited with kubectl.

Multiple Schedulers

Custom schedulers can be created in Kubernetes and run alongside the default scheduler.

  - --leader-elect=true
  - --scheduler-name=my-custom-scheduler
  - --lock-object-name=my-custom-scheduler

4. Logging & Monitoring

Monitoring

As of 2018, there is no built-in monitoring tool in K8s. Open source solutions include Prometheus, Elastic Stack, DATADOG, dynatrace, and the Metrics Server.

The Metrics Server collects and aggregates metrics from nodes and pods. One Metrics Server per cluster.

git clone https://github.com/kodekloudhub/kubernetes-metrics-server.git
cd kubernetes-metrics-server/
kubectl create -f .
kubectl top node

Logging

kubectl logs -f <pod-name> <container-name> shows live stream logs.

5. Application Lifecycle Management

Rollout and Rollback

Rollout

Deployment strategies: Recreate (causes downtime), Rolling Update (no downtime, default).

kubectl apply -f <YAML-file>
kubectl set image deployment/<deployment-name> <container-name>=nginx:1.9.1
kubectl rollout status deployment/myapp-deployment
kubectl rollout history deployment/myapp-deployment

Rollback

kubectl rollout undo deployment/myapp-deployment

Configure Applications

Commands and Arguments

spec:
  containers:
    - name: ubuntu-sleeper
      image: ubuntu-sleeper
      command: ["sleep"]
      args: ["10"]

Environment Variables

Three methods: Plain Environment Variables, ConfigMaps, Secrets.

Plain Environment Variables

spec:
  containers:
  - name:
    image:
    env:
    - name:
      value:

ConfigMaps

Create a ConfigMap, then assign it to a Pod.

kubectl create configmap <configmap-name> --from-literal=<key>=<value>

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  APP_COLOR: blue
  APP_MODE: prod

spec:
  containers:
  - name:
     image:
     envFrom:
     - configMapRef:
          name:

Secrets

Stores environment variable values encoded in Base64.

kubectl create secret generic <secret-name> --from-literal=<key>=<value>

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
data:
  User: bxlZC=
  Password: cm9vDa=

Warning: Base64-encoded values are not secure on their own. Use encryption at rest.

Best practices:

• Not checking-in secret object definition files to source code repositories.
• Enabling Encryption at Rest for Secrets so they are stored encrypted in ETCD.

Multi Container Pods

spec:
  containers:
  - name:
    image:
  - name:
    image:

Init Containers

Run before the main container. If multiple initContainers are defined, they run in order. If any initContainer fails, the Pod restarts.

spec:
  containers:
  - name: myapp-container
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init-myservice
    image: busybox:1.28
    command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']

6. Cluster Maintenance

OS Upgrades

Drain Pods from a node before upgrading:

kubectl drain <node-name>     # Evict Pods and mark Unschedulable
kubectl cordon <node-name>    # Mark Unschedulable without evicting Pods
kubectl uncordon <node-name>  # Remove Unschedulable status

Kubernetes Software Versions

Format: v{Major}.{Minor}.{Patch}. Minor version adds features; Patch fixes bugs.

Cluster Upgrade Process

K8s version compatibility rules:

• Components must not be newer than the Kube API Server (except kubectl which can be one version newer)
• Kubelet and Kube Proxy can be up to two minor versions behind
• K8s supports only the three most recent minor versions
• Upgrade one minor version at a time

Kubeadm upgrade steps:

kubeadm upgrade plan
kubeadm upgrade apply v1.12.0
apt-get upgrade -y kubelet=1.12.0-00
systemctl restart kubelet
kubectl drain node01
apt-get upgrade -y kubeadm=1.12.0-00
apt-get upgrade -y kubelet=1.12.0-00
kubeadm upgrade node config --kubelet-version v1.12.0
systemctl restart kubelet
kubectl uncordon node01

Backup and Restore Methods

Three backup types: Resource configs, ETCD cluster, Persistent volumes.

K8s Resource Config Backup

kubectl get all --all-namespaces -o yaml > all-deploy-services.yaml

ETCD Cluster Backup

ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
--endpoints=https://[127.0.0.1]:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key

Restore:

service kube-apiserver stop
etcdctl snapshot restore snapshot.db --data-dir /var/lib/etcd-from-backup
systemctl daemon-reload
service etcd restart
service kube-apiserver start

7. Security

Authentication

Two account types: User (admin, developer) and Service Account (external applications).

K8s does not manage User accounts directly; it relies on external sources. Service accounts can be created with kubectl create serviceaccount sa1.

Authentication methods:

• Static Password File
• Static Token File
• Certificates
• External Identity Services (LDAP)

Static Password File

curl -v -k https://master-node-ip:6443/api/v1/pods -u "user1:password123"

Note: This method is not recommended. TLS Certificates are the preferred approach.

Static Token File

curl -v -k https://master-node-ip:6443/api/v1/pods --header "Authorization: Bearer <token>"

TLS Basics

PKI (Public Key Infrastructure) overview:

Goal: Enable encrypted communication between server and client.

Procedure:

1. Server sends its public key (as CSR) to CA.
2. CA signs the CSR with the CA private key and returns it to the server.
3. Server sends the signed certificate to the client.
4. Client verifies using the CA public key, extracts the server public key.
5. Client generates a symmetric key, encrypts it with the server public key, and sends it.
6. Server decrypts with its private key to get the symmetric key.
7. Subsequent communication is encrypted with the symmetric key.

TLS in Kubernetes

Three certificate types: server certificates, root certificates, client certificates.

Certificate files use .crt or .pem extensions; private key files use .key or -key.pem.

TLS in Kubernetes - Certificate Creation

# CA root certificate
openssl genrsa -out ca.key 2048
openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt

# Admin user client certificate
openssl genrsa -out admin.key 2048
openssl req -new -key admin.key -subj "/CN=kube-admin/O=system:masters" -out admin.csr
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt

Certificates API

To create a new user "jane":

openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr
kubectl get csr
kubectl certificate approve jane
kubectl get csr jane -o yaml

KubeConfig

$HOME/.kube/config stores cluster, context, and user definitions.

kubectl config view
kubectl config use-context dev-frontend

Authorization

Authorization methods:

• Node: allows Worker Nodes (Kubelet) to access Master Node
• ABAC: grants permissions per attribute (e.g., username)
• RBAC: grants permissions per role (developer, admin, etc.)
• Webhook: uses external tools like Open Policy Agent

RBAC Details

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
  resourceNames: ["blue", "orange"]

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
subjects:
- kind: User
  name: dev-user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Check access:

kubectl auth can-i create deployments --namespace prod --as dave

Cluster Roles and Cluster Role Bindings

For non-namespaced resources (Nodes, Namespaces), use ClusterRole and ClusterRoleBinding.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-administrator
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["create", "get", "list", "delete"]

Service Accounts

Used by external applications (Prometheus, Jenkins) that need cluster access.

kubectl create serviceaccount dashboard-sa
kubectl describe serviceaccount dashboard-sa
kubectl describe secret dashboard-sa

To assign a non-default Service Account to a Pod, add serviceAccountName to spec:. To prevent mounting the default Service Account, add autoMountServiceAccountToken: false.

Image Security

kubectl create secret docker-registry regcred \
    --docker-server=<your-registry-server> \
    --docker-username=<your-name> \
    --docker-password=<your-pword> \
    --docker-email=<your-email>

spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  imagePullSecrets:
  - name: regcred

Security Contexts

spec:
  containers:
  - name: sec-ctx-4
    image: gcr.io/google-samples/node-hello:1.0
    securityContext:
      capabilities:
        runAsUser: <User ID>
        add: ["NET_ADMIN", "SYS_TIME"]

Note: capabilities can only be set under spec: containers:, not spec:.

Network Policy

By default, all Pods in a Kubernetes cluster can communicate with each other.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: api-pod
    ports:
    - protocol: TCP
      port: 3306

NetworkPolicy supported: Kube-router, Calico, Romana, Weave-net. Not supported: Flannel.

8. Storage

Volume

When a container is deleted, its data disappears. Create a Volume and mount it to a host directory.

spec:
  volumes:
  - name: vol
    hostPath:
      path: /any/path/it/will/be/replaced
  containers:
  - name: pv-recycler
    image: "k8s.gcr.io/busybox"
    volumeMounts:
    - name: vol
      mountPath: /scrub

Note: hostPath is not recommended for security reasons. For multi-node setups, use NFS, GlusterFS, or similar.

Persistent Volumes

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0003
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 5Gi
  awsElasticBlockStore:
    volumeID: "<volume id>"
    fsType: ext4
  persistentVolumeReclaimPolicy: Retain

Persistent Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi

Reclaim policies: Retain (PV not deleted/reusable), Delete (PV deleted), Recycle (PV cleared for reuse).

Storage Class

Automatically provisions storage on clouds like AWS.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: aws-ebs-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1
  iopsPerGB: "10"
  fsType: ext4

9. Networking

DNS on Linux

/etc/hosts maps hostnames to IPs. /etc/resolv.conf sets DNS server IPs. /etc/nsswitch.conf determines resolution order (files → dns).

Note: nslookup only queries DNS; it does not check /etc/hosts. dig provides more detailed responses.

Core DNS

An open-source CNCF DNS server used in Kubernetes.

Network Namespace

ip netns add red
ip netns add blue
ip netns exec red ip link

Cluster Networking

Port numbers:

• Kube-API: 6443
• ETCD Server: 2379
• ETCD Client: 2380
• Kubelet: 10250
• Kube-scheduler: 10251
• Kube-controller-manager: 10252
• Services: 30000 - 32767

Service Networking

A Service is a virtual object. When a new Service is created, Kube-Proxy creates forwarding rules so traffic to the Service IP is forwarded to the Pod.

Proxy modes: userspace, iptables (default), ipvs.

Service types: ClusterIP, NodePort, LoadBalancer.

DNS in Kubernetes

DNS A record format:

• Service: <service-name>.<namespace>.svc.cluster.local
• Pod: <pod-ip-with-dashes>.<namespace>.pod.cluster.local

Ingress

A load balancer providing URL path-based routing with TLS termination support.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx-example
  rules:
  - http:
      paths:
      - path: /testpath
        pathType: Prefix
        backend:
          service:
            name: test
            port:
              number: 80

IP Address Range Verification

• Node: ip addr → check address range assigned to eth0
• Pod: ip addr → check address range assigned to CNI plugin interface, or check ipalloc-range: in logs
• Service: cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep service-cluster-ip-range

Design and Install a Kubernetes Cluster

Configure High Availability

Kube-API: Active-Active with load balancing. Controller Manager and Scheduler: Active-Standby.

ETCD topologies: Stacked (same node as other master components) or External (separate node).

ETCD in HA

Distributed database where all nodes hold identical data. Uses RAFT protocol for leader election. The number of nodes should be odd.

Install Kubernetes the kubeadm way

Steps:

1. Prepare Master and Worker Nodes
2. Install Docker container runtime
3. Install kubeadm
4. Initialize Master Node
5. Install network plugin
6. Join Worker Nodes

Troubleshooting

Application Failure

curl http://web-service-ip:node-port
kubectl describe service web-service   # Verify selector matches Pod labels
kubectl get pod
kubectl describe pod web
kubectl logs web
kubectl logs web --previous

Controlplane Failure

kubectl get nodes
kubectl get pods -n kube-system
kubectl logs kube-apiserver-master -n kube-system
service kube-apiserver status
sudo journalctl -u kube-apiserver

Worker Node Failure

kubectl get nodes
kubectl describe node <node-name>  # Check conditions, flags
top     # CPU usage
df -h   # Memory usage
service kubelet status
sudo journalctl -u kubelet
openssl x509 -in /var/lib/kubelet/worker-1.crt -text  # Check certificate expiry

Personal Notes

How to Find staticPodPath

• ps -aux | grep kubelet
• Find --config=/var/lib/kubelet/config.yaml
• cat /var/lib/kubelet/config.yaml
• staticPodPath: /etc/kubernetes/manifests

SCP Command

Transfer a file to a remote host:

scp test.txt node01:/etc/kubernetes