Running Azure IoT Edge on Kubernetes on real host

There is great documentation on running IoT Edge on Kubernetes, starting here: https://aka.ms/edgek8sdoc. I also wrote a quick blog about running Edge on Kubernetes on a single host here: Azure IoT Edge on Kubernetes with one command.

Today’s blog is about Edge on real (non development) Kubernetes clusters and the experience during an unplanned outage.

While this is not about teaching Kubernetes, it does walk you through the what, why and how to setup your own Cluster. At the end, I have some simple scripts that make this as simple as one command.

Note, for real HA, review: https://microsoft.github.io/iotedge-k8s-doc/examples/ha.html and the need for persistent volumes to hold certificates.

Requirements:

As discussed here: https://microsoft.github.io/iotedge-k8s-doc/examples/prereqs.html, here are the following requirements:

  • Kubernetes v1.12+
  • Helm 3
  • Docker or Moby
  • (required by Kubernetes Clusters) a Pod Network add-on

Setup Process:

The following steps were tested on an Ubuntu 18.04 on x64 hardware.

Master Node:

Install the Prerequisites and disable the swap file:

sudo swapoff -a
sudo sed -i -e '/swap.img/d' /etc/fstab
sudo apt update && sudo apt install -y unzip tree apt-transport-https jq curl wget

Install Docker or Moby:

sudo curl https://packages.microsoft.com/config/ubuntu/18.04/multiarch/prod.list > ./microsoft-prod.list
sudo cp ./microsoft-prod.list /etc/apt/sources.list.d/
sudo curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo cp ./microsoft.gpg /etc/apt/trusted.gpg.d/
sudo apt-get update
sudo apt-get install -y moby-engine moby-cli

Install Kubernetes and initialize the cluster:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
apt-add-repository "deb https://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update && sudo apt-get install kubeadm -y
sudo kubeadm init --pod-network-cidr=172.29.0.0/24 --ignore-preflight-errors=all

Install Calico Network Manager:

sudo kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml --kubeconfig=/etc/kubernetes/admin.conf

Install Helm 3:

wget https://get.helm.sh/helm-v3.1.0-linux-amd64.tar.gz
tar xvf ./helm-v3.1.0-linux-amd64.tar.gz
sudo mv ./linux-amd64/helm /usr/local/bin
rm -rf helm-v3.1.0-linux-amd64.tar.gz ./linux-amd64/

(optional) install K9 Visual:

wget https://github.com/derailed/k9s/releases/download/v0.15.2/k9s_Linux_x86_64.tar.gz
tar xvf ./k9s_Linux_x86_64.tar.gz
sudo mv ./k9s /usr/local/bin/
rm ./k9s_Linux_x86_64.tar.gz ./LICENSE ./README.md

Install IoT Edge with the Connection String

sudo kubectl create ns iotedge --kubeconfig=/etc/kubernetes/admin.conf
sudo helm install --repo https://edgek8s.blob.core.windows.net/staging edge-crd edge-kubernetes-crd --kubeconfig=/etc/kubernetes/admin.conf
sudo helm install --repo https://edgek8s.blob.core.windows.net/staging edge edge-kubernetes --namespace iotedge --kubeconfig=/etc/kubernetes/admin.conf --set provisioning.deviceConnectionString='YOURSTRINGHERE'

Note:

At the conclusion of creating the cluster, you will see a join command for joining worker nodes to the cluster. Copy this command

Worker Nodes:

Install the Prerequisites and disable the swap file:

sudo swapoff -a
sudo sed -i -e '/swap.img/d' /etc/fstab
sudo apt update && sudo apt install -y unzip tree apt-transport-https jq curl wget

Install Docker or Moby:

sudo curl https://packages.microsoft.com/config/ubuntu/18.04/multiarch/prod.list > ./microsoft-prod.list
sudo cp ./microsoft-prod.list /etc/apt/sources.list.d/
sudo curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo cp ./microsoft.gpg /etc/apt/trusted.gpg.d/
sudo apt-get update
sudo apt-get install -y moby-engine moby-cli

Install Kubernetes:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb https://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update && sudo apt-get install kubeadm -y

Join the Cluster (command given at the conclusion of creating the Cluster):

sudo kubeadm join 192.168.15.62:6443 --token yxrw*******7ytne --discovery-token-ca-cert-hash sha256:8df0*******8ae3e25c7 --ignore-preflight-errors=all

Automating it all with a single command:

On the Master, once you have a connection string, run:

wget -q -O - https://raw.githubusercontent.com/ksaye/AzureIoTEdgeOnKubernetes/master/clusterMaster.sh | sudo constr='YOURSTRINGHERE' bash

On a Worker Node, first run:

wget -q -O - https://raw.githubusercontent.com/ksaye/AzureIoTEdgeOnKubernetes/master/workerNode.sh | sudo bash

then Run (replacing the token, IP and has with the output from the Master:

sudo kubeadm join 192.168.15.62:6443 --token yxr*****tne --discovery-token-ca-cert-hash sha256:8df02******3e25c7 --ignore-preflight-errors=all

Seeing an unplanned failover in action:

Next we will see an unplanned failure in action.

To demonstrate this, I have deployed the “Temperature Sensor” module form the Azure IoT Edge Module Market Place. When deployed, Kubernetes placed the module on node2:

When viewed from Device Explorer, we see messages are coming in:

Next I simply unplugged node2, simulating an unplanned failure.

After 6 minutes, we can see that messages restart in Device Explorer, indicating that Kubernetes has re created the module on a different host:

Looking back at Kubernetes, we see that the “Temperature Sensor” and Edge Agent are flagged as terminated on node2 and have been started on node1.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s