参考官方文档：
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
在 debian12 上安装 kubenetes
在 debian12 上用 kubeadm 安装 kubenetes
- 1: 准备工作
- 2: 安装命令行
- 3: 初始化集群
- 4: 安装 dashboard
- 5: 安装 metrics server
- 6: 安装监控
1 - 准备工作
在 debian12 上安装 kubenetes 之前的准备工作
系统更新
确保更新debian系统到最新，移除不再需要的软件，清理无用的安装包：
sudo apt update && sudo apt full-upgrade -y
sudo apt autoremove
sudo apt autoclean
如果更新了内核，最好重启一下。
swap 分区
安装 Kubernetes 要求机器不能有 swap 分区。
参考：
开启模块
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# Apply sysctl params without reboot
sudo sysctl --system
container runtime
Kubernetes 支持多种 container runtime，这里暂时继续使用 docker engine + cri-dockerd。
参考：
https://kubernetes.io/docs/setup/production-environment/container-runtimes/
安装 docker + cri-dockerd
docker 的安装参考：
https://skyao.io/learning-docker/docs/installation/debian12/
cri-dockerd 的安装参考：
https://mirantis.github.io/cri-dockerd/usage/install/
从 release 页面下载：
https://github.com/Mirantis/cri-dockerd/releases
debian 12 选择下载文件
下载后安装：
sudo dpkg -i ./cri-dockerd_0.3.16.3-0.debian-bookworm_amd64.deb
安装后会提示：
Selecting previously unselected package cri-dockerd.
(Reading database ... 48498 files and directories currently installed.)
Preparing to unpack .../cri-dockerd_0.3.16.3-0.debian-bookworm_amd64.deb ...
Unpacking cri-dockerd (0.3.16~3-0~debian-bookworm) ...
Setting up cri-dockerd (0.3.16~3-0~debian-bookworm) ...
Created symlink /etc/systemd/system/multi-user.target.wants/cri-docker.service → /lib/systemd/system/cri-docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/cri-docker.socket → /lib/systemd/system/cri-docker.socket.
安装后查看状态：
sudo systemctl status cri-docker.service
如果成功则状态为：
● cri-docker.service - CRI Interface for Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/cri-docker.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-03-04 19:18:50 CST; 3min 25s ago
TriggeredBy: ● cri-docker.socket
Docs: https://docs.mirantis.com
Main PID: 2665 (cri-dockerd)
Tasks: 9
Memory: 15.0M
CPU: 21ms
CGroup: /system.slice/cri-docker.service
└─2665 /usr/bin/cri-dockerd --container-runtime-endpoint fd://
Mar 04 19:18:50 debian12 cri-dockerd[2665]: time="2025-03-04T19:18:50+08:00" level=info msg="Hairpin mode is set to none"
Mar 04 19:18:50 debian12 cri-dockerd[2665]: time="2025-03-04T19:18:50+08:00" level=info msg="The binary conntrack is not installed, this can cause failures in network conn>
Mar 04 19:18:50 debian12 cri-dockerd[2665]: time="2025-03-04T19:18:50+08:00" level=info msg="The binary conntrack is not installed, this can cause failures in network conn>
Mar 04 19:18:50 debian12 cri-dockerd[2665]: time="2025-03-04T19:18:50+08:00" level=info msg="Loaded network plugin cni"
Mar 04 19:18:50 debian12 cri-dockerd[2665]: time="2025-03-04T19:18:50+08:00" level=info msg="Docker cri networking managed by network plugin cni"
Mar 04 19:18:50 debian12 cri-dockerd[2665]: time="2025-03-04T19:18:50+08:00" level=info msg="Setting cgroupDriver systemd"
Mar 04 19:18:50 debian12 cri-dockerd[2665]: time="2025-03-04T19:18:50+08:00" level=info msg="Docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig>
Mar 04 19:18:50 debian12 cri-dockerd[2665]: time="2025-03-04T19:18:50+08:00" level=info msg="Starting the GRPC backend for the Docker CRI interface."
Mar 04 19:18:50 debian12 cri-dockerd[2665]: time="2025-03-04T19:18:50+08:00" level=info msg="Start cri-dockerd grpc backend"
Mar 04 19:18:50 debian12 systemd[1]: Started cri-docker.service - CRI Interface for Docker Application Container Engine.
安装 containerd
TODO：后面考虑换 containerd
安装 helm
参考：
https://helm.sh/docs/intro/install/#from-apt-debianubuntu
安装：
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
安装后取消 helm 的自动更新：
sudo vi /etc/apt/sources.list.d/helm-stable-debian.list
查看安装的版本：
$ helm version
version.BuildInfo{Version:"v3.17.1", GitCommit:"980d8ac1939e39138101364400756af2bdee1da5", GitTreeState:"clean", GoVersion:"go1.23.5"}
2 - 安装命令行
在 debian12 上安装 kubeadm / kubelet / kubectl
参考： https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
安装 kubeadm / kubelet / kubectl
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
假定要安装的 kubernetes 版本为 1.32:
export K8S_VERSION=1.32
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
开始安装 kubelet kubeadm kubectl：
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
禁止这三个程序的自动更新：
sudo apt-mark hold kubelet kubeadm kubectl
验证安装：
kubectl version --client && echo && kubeadm version
输出为：
Client Version: v1.32.2
Kustomize Version: v5.5.0
kubeadm version: &version.Info{Major:"1", Minor:"32", GitVersion:"v1.32.2", GitCommit:"67a30c0adcf52bd3f56ff0893ce19966be12991f", GitTreeState:"clean", BuildDate:"2025-02-12T21:24:52Z", GoVersion:"go1.23.6", Compiler:"gc", Platform:"linux/amd64"}
在运行 kubeadm 之前，先启动 kubelet 服务：
sudo systemctl enable --now kubelet
安装后配置
优化 zsh
vi ~/.zshrc
增加以下内容:
# k8s auto complete
alias k=kubectl
complete -F __start_kubectl k
执行：
source ~/.zshrc
之后即可使用，此时用 k 这个别名来执行 kubectl 命令时也可以实现自动完成，非常的方便。
取消更新
kubeadm / kubelet / kubectl 的版本没有必要升级到最新，因此可以取消他们的自动更新。
sudo vi /etc/apt/sources.list.d/kubernetes.list
注释掉里面的内容。
备注：前面执行 apt-mark hold 后已经不会再更新了，但依然会拖慢 apt update 的速度，因此还是需要手动注释。
常见问题
prod-cdn.packages.k8s.io 无法访问
偶然会遇到 prod-cdn.packages.k8s.io 无法访问的问题，此时的报错如下：
sudo apt-get update
Hit:1 http://mirrors.ustc.edu.cn/debian bookworm InRelease
Hit:2 http://mirrors.ustc.edu.cn/debian bookworm-updates InRelease
Hit:3 http://security.debian.org/debian-security bookworm-security InRelease
Ign:4 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.32/deb InRelease
Ign:4 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.32/deb InRelease
Ign:4 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.32/deb InRelease
Err:4 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.32/deb InRelease
Could not connect to prod-cdn.packages.k8s.io:443 (221.228.32.13), connection timed out
Reading package lists... Done
W: Failed to fetch https://pkgs.k8s.io/core:/stable:/v1.32/deb/InRelease Could not connect to prod-cdn.packages.k8s.io:443 (221.228.32.13), connection timed out
W: Some index files failed to download. They have been ignored, or old ones used instead.
首先排除是网络问题，因为实际配好网络代理，也依然无法访问。
后来发现，在不同地区的机器上 ping prod-cdn.packages.k8s.io 的 ip 地址是不一样的，
$ ping prod-cdn.packages.k8s.io
Pinging dkhzw6k7x6ord.cloudfront.net [108.139.10.84] with 32 bytes of data:
Reply from 108.139.10.84: bytes=32 time=164ms TTL=242
Reply from 108.139.10.84: bytes=32 time=166ms TTL=242
......
# 这个地址无法访问
$ ping prod-cdn.packages.k8s.io
PING dkhzw6k7x6ord.cloudfront.net (221.228.32.13) 56(84) bytes of data.
64 bytes from 221.228.32.13 (221.228.32.13): icmp_seq=1 ttl=57 time=9.90 ms
64 bytes from 221.228.32.13 (221.228.32.13): icmp_seq=2 ttl=57 time=11.4 ms
......
因此考虑通过修改 /etc/hosts 文件来避开 dns 解析的问题：
sudo vi /etc/hosts
添加如下内容：
108.139.10.84 prod-cdn.packages.k8s.io
这样在出现问题的这台机器上，强制将 prod-cdn.packages.k8s.io 解析到 108.139.10.84 这个 ip 地址，这样就可以访问了。
3 - 初始化集群
在 debian12 上初始化 kubernetes 集群
参考官方文档：
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
初始化集群
pod-network-cidr 尽量用 10.244.0.0/16 这个范围，不然有些网络插件会需要额外的配置。
cri-socket 的配置参考：
因为前面用的 Docker Engine 和 cri-dockerd ，因此这里的 cri-socket 需要指定为 “unix:///var/run/cri-dockerd.sock”。
apiserver-advertise-address 需要指定为当前节点的 IP 地址，因为当前节点是单节点，因此这里指定为 192.168.3.215。
sudo kubeadm init --pod-network-cidr 10.244.0.0/16 --cri-socket unix:///var/run/cri-dockerd.sock --apiserver-advertise-address=192.168.3.215
输出为：
[init] Using Kubernetes version: v1.32.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
W0304 20:23:50.183712 5058 checks.go:846] detected that the sandbox image "registry.k8s.io/pause:3.9" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [debian12 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.3.215]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [debian12 localhost] and IPs [192.168.3.215 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [debian12 localhost] and IPs [192.168.3.215 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 500.939992ms
[api-check] Waiting for a healthy API server. This can take up to 4m0s
[api-check] The API server is healthy after 3.00043501s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node debian12 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node debian12 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 8e5a3n.rqbqfbnvhf4uyjft
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.3.215:6443 --token 8e5a3n.rqbqfbnvhf4uyjft \
--discovery-token-ca-cert-hash sha256:183b3e9965d298e67689baddeff2ff88c32b3f18aa9dd9a15be1881d26025a22
根据提示操作：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
对于测试用的单节点，去除 master/control-plane 的污点：
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
执行：
kubectl get node
能看到此时节点的状态会是 NotReady：
NAME STATUS ROLES AGE VERSION
debian12 NotReady control-plane 3m49s v1.32.2
执行：
kubectl describe node debian12
能看到节点的错误信息：
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 04 Mar 2025 20:28:00 +0800 Tue, 04 Mar 2025 20:23:53 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 04 Mar 2025 20:28:00 +0800 Tue, 04 Mar 2025 20:23:53 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 04 Mar 2025 20:28:00 +0800 Tue, 04 Mar 2025 20:23:53 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Tue, 04 Mar 2025 20:28:00 +0800 Tue, 04 Mar 2025 20:23:53 +0800 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
需要继续安装网络插件。
安装网络插件
安装 flannel
参考官方文档： https://github.com/flannel-io/flannel#deploying-flannel-with-kubectl
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
如果一切正常，就能看到 k8s 集群内的 pod 都启动完成状态为 Running：
k get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-ts6n8 1/1 Running 7 (9m27s ago) 15m
kube-system coredns-668d6bf9bc-rbkzb 1/1 Running 0 3h55m
kube-system coredns-668d6bf9bc-vbltg 1/1 Running 0 3h55m
kube-system etcd-debian12 1/1 Running 0 3h55m
kube-system kube-apiserver-debian12 1/1 Running 1 (5h57m ago) 3h55m
kube-system kube-controller-manager-debian12 1/1 Running 0 3h55m
kube-system kube-proxy-95ccr 1/1 Running 0 3h55m
kube-system kube-scheduler-debian12 1/1 Running 1 (6h15m ago) 3h55m
如果发现 kube-flannel-ds pod 的状态总是 CrashLoopBackOff：
k get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-ts6n8 0/1 CrashLoopBackOff 2 (22s ago) 42s
继续查看 pod 的具体错误信息：
k describe pods -n kube-flannel kube-flannel-ds-ts6n8
发现报错 “Back-off restarting failed container kube-flannel in pod kube-flannel”：
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 117s default-scheduler Successfully assigned kube-flannel/kube-flannel-ds-ts6n8 to debian12
Normal Pulled 116s kubelet Container image "ghcr.io/flannel-io/flannel-cni-plugin:v1.6.2-flannel1" already present on machine
Normal Created 116s kubelet Created container: install-cni-plugin
Normal Started 116s kubelet Started container install-cni-plugin
Normal Pulled 115s kubelet Container image "ghcr.io/flannel-io/flannel:v0.26.4" already present on machine
Normal Created 115s kubelet Created container: install-cni
Normal Started 115s kubelet Started container install-cni
Normal Pulled 28s (x5 over 114s) kubelet Container image "ghcr.io/flannel-io/flannel:v0.26.4" already present on machine
Normal Created 28s (x5 over 114s) kubelet Created container: kube-flannel
Normal Started 28s (x5 over 114s) kubelet Started container kube-flannel
Warning BackOff 2s (x10 over 110s) kubelet Back-off restarting failed container kube-flannel in pod kube-flannel-ds-ts6n8_kube-flannel(1e03c200-2062-4838
此时应该去检查准备工作中 “开启模块” 一节的内容是不是有疏漏。
补救之后，就能看到 kube-flannel-ds 这个 pod 正常运行了：
k get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-ts6n8 1/1 Running 7 (9m27s ago) 15m
安装 Calico
查看最新版本，当前最新版本是 v3.29.2:
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/tigera-operator.yaml
TODO：用了 flannel， Calico 后面再验证。
4 - 安装 dashboard
安装 kubernetes 的 dashboard
安装 dashboard
参考：https://github.com/kubernetes/dashboard/#installation
在下面地址上查看当前 dashboard 的版本：
https://github.com/kubernetes/dashboard/releases
根据对 kubernetes 版本的兼容情况选择对应的 dashboard 的版本：
- kubernetes-dashboard-7.11.0 ，兼容 k8s 1.32
最新版本需要用 helm 进行安装：
helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard
输出为：
"kubernetes-dashboard" already exists with the same configuration, skipping
Release "kubernetes-dashboard" does not exist. Installing it now.
NAME: kubernetes-dashboard
LAST DEPLOYED: Wed Mar 5 00:53:17 2025
NAMESPACE: kubernetes-dashboard
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
*************************************************************************************************
*** PLEASE BE PATIENT: Kubernetes Dashboard may need a few minutes to get up and become ready ***
*************************************************************************************************
Congratulations! You have just installed Kubernetes Dashboard in your cluster.
To access Dashboard run:
kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443
NOTE: In case port-forward command does not work, make sure that kong service name is correct.
Check the services in Kubernetes Dashboard namespace using:
kubectl -n kubernetes-dashboard get svc
Dashboard will be available at:
https://localhost:8443
此时 dashboard 的 service 和 pod 情况：
kubectl -n kubernetes-dashboard get services
输出为：
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard-api ClusterIP 10.108.225.190 <none> 8000/TCP 2m5s
kubernetes-dashboard-auth ClusterIP 10.99.205.102 <none> 8000/TCP 2m5s
kubernetes-dashboard-kong-proxy ClusterIP 10.96.247.162 <none> 443/TCP 2m5s
kubernetes-dashboard-metrics-scraper ClusterIP 10.103.222.22 <none> 8000/TCP 2m5s
kubernetes-dashboard-web ClusterIP 10.108.219.9 <none> 8000/TCP 2m5s
查看 pod 的情况：
kubectl -n kubernetes-dashboard get pods
等待两三分钟之后，pod 启动完成，输出为：
NAME READY STATUS RESTARTS AGE
kubernetes-dashboard-api-7d8567b8f-9ksk2 1/1 Running 0 3m8s
kubernetes-dashboard-auth-6877bf44b9-9qfmg 1/1 Running 0 3m8s
kubernetes-dashboard-kong-79867c9c48-rzlhp 1/1 Running 0 3m8s
kubernetes-dashboard-metrics-scraper-794c587449-6phjv 1/1 Running 0 3m8s
kubernetes-dashboard-web-75576c76b-sm2wj 1/1 Running 0 3m8s
为了方便，使用 node port 来访问 dashboard，需要执行：
kubectl -n kubernetes-dashboard edit service kubernetes-dashboard-kong-proxy
然后修改 type: ClusterIP 为 type: NodePort。然后看一下具体分配的 node port 是哪个：
kubectl -n kubernetes-dashboard get service kubernetes-dashboard-kong-proxy
输出为：
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard-kong-proxy NodePort 10.96.247.162 <none> 443:32616/TCP 17m
现在可以用浏览器直接访问：
https://192.168.3.215:32616/
创建用户并登录 dashboard
创建 admin-user 用户：
mkdir -p ~/work/soft/k8s
cd ~/work/soft/k8s
vi dashboard-adminuser.yaml
内容为:
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
执行：
k create -f dashboard-adminuser.yaml
然后绑定角色：
vi dashboard-adminuser-binding.yaml
内容为:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
执行：
k create -f dashboard-adminuser-binding.yaml
然后创建 token ：
kubectl -n kubernetes-dashboard create token admin-user
输出为:
eyJhbGciOiJSUzI1NiIsImtpZCI6Ik9sWnJsTk5UNE9JVlVmRFMxMUpwNC1tUlVndTl5Zi1WQWtmMjIzd2hDNmcifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzQxMTEyNDg4LCJpYXQiOjE3NDExMDg4ODgsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiNDU5ZGQxNjctNWI5OS00MWIzLTgzZWEtNGIxMGY3MTc5ZjEyIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiZjMxN2VhZTItNTNiNi00MGZhLWI3MWYtMzZiNDI1YmY4YWQ0In19LCJuYmYiOjE3NDExMDg4ODgsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.TYzOdrMFXcSEeVMbc1ewIA13JVi4FUYoRN7rSH5OstbVfKIF48X_o1RWxOGM_AurhgLxuKZHzmns3K_pX_OR3u1URfK6-gGos4iAQY-H1yntfRmzzsip_FbZh95EYFGTN43gw21jTyfem3OKBXXLgzsnVT_29uMnJzSnCDnrAciVKMoCEUP6x2RSHQhp6PrxrIrx_NMB3vojEZYq3AysQoNqYYjRDd4MnDRClm03dNvW5lvKSgNCVmZFje_EEa2EhI2X6d3X8zx6tHwT5M4-T3hMmyIpzHUwf3ixeZR85rhorMbskNVvRpH6VLH6BXP31c3NMeSgYk3BG8d7UjCYxQ
这个 token 就可以用在 kubernetes-dashboard 的登录页面上了。
为了方便，将这个 token 存储在 Secret ：
vi dashboard-adminuser-secret.yaml
内容为:
apiVersion: v1
kind: Secret
metadata:
name: admin-user
namespace: kubernetes-dashboard
annotations:
kubernetes.io/service-account.name: "admin-user"
type: kubernetes.io/service-account-token
执行：
k create -f dashboard-adminuser-secret.yaml
之后就可以用命令随时获取这个 token 了：
kubectl get secret admin-user -n kubernetes-dashboard -o jsonpath="{.data.token}" | base64 -d
备注：复制 token 的时候，不要复制最后的那个 % 字符，否则会报错。
5 - 安装 metrics server
安装 kubernetes 的 metrics server
参考：https://github.com/kubernetes-sigs/metrics-server/#installation
安装 metrics server
下载：
mkdir -p ~/work/soft/k8s
cd ~/work/soft/k8s
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
修改下载下来的 components.yaml， 增加 --kubelet-insecure-tls 并修改 --kubelet-preferred-address-types：
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP # 修改这行，默认是InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls # 增加这行
然后安装：
k apply -f components.yaml
稍等片刻看是否启动:
kubectl get pod -n kube-system | grep metrics-server
验证一下，查看 service 信息
kubectl describe svc metrics-server -n kube-system
简单验证一下基本使用:
kubectl top nodes
kubectl top pods -n kube-system
参考资料
6 - 安装监控
安装 prometheus 和 grafana 以监控 kubernetes 集群
参考：https://github.com/prometheus-operator/prometheus-operator
https://computingforgeeks.com/setup-prometheus-and-grafana-on-kubernetes/