初探kubernetes集群

kubernetes ,

概览:

前期准备

节点

我准备部署4个节点:一主三从

操作系统: almalinux9 :almalinux

分配： master: 1, worker: 3

在管理节点hosts文件增加主机域名和ip的对应关系。

1
2
3
4
192.168.6.17 k8s-m1
192.168.6.21 k8s-w1
192.168.6.22 k8s-w2
192.168.6.23 k8s-w3

初始化

1
2
3
yum install epel-release -y
yum update -y
yum install nano vim tar socat net-tools ethtool wget mlocate bash-completion tree -y

运维工具安装

1
yum install ansible

编辑ansible配置文件,/etc/ansible/hosts,追加以下新内容

1
2
3
4
5
6
7
[master]
192.168.6.21

[worker]
192.168.6.31
192.168.6.32
192.168.6.33

ssh免密码

1
2
ssh-keygen
cat /etc/ansible/hosts | grep "192.168" | awk '{print $1}' | xargs -I {} ssh-copy-id {} # 就是自动化执行ssh-copy-id 192.168.6.[17,21-23]

检查一下

1
2
3
4
5
6
7
8
9
[root@k8s-m1 ~]# ansible all -m shell -a "uname -a"
192.168.6.23 | CHANGED | rc=0 >>
Linux k8s-w3 5.14.0-162.12.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jan 23 14:51:52 EST 2023 x86_64 x86_64 x86_64 GNU/Linux
192.168.6.17 | CHANGED | rc=0 >>
Linux k8s-m1 5.14.0-162.12.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jan 23 14:51:52 EST 2023 x86_64 x86_64 x86_64 GNU/Linux
192.168.6.22 | CHANGED | rc=0 >>
Linux k8s-w2 5.14.0-162.12.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jan 23 14:51:52 EST 2023 x86_64 x86_64 x86_64 GNU/Linux
192.168.6.21 | CHANGED | rc=0 >>
Linux k8s-w1 5.14.0-162.12.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jan 23 14:51:52 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

推送hosts

请检查出现类似本地回环的配置

1
ansible all -m copy -a "src=/etc/hosts dest=/etc/hosts"

sealos

我是用的是部署工具sealos来部署。所以先下载并安装，请安装官网最新版本。

1
2
wget https://github.com/labring/sealos/releases/download/v4.1.4/sealos_4.1.4_linux_amd64.tar.gz \
   && tar zxvf sealos_4.1.4_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin

防火墙配置

以下配置根据自己的需求来更改和参考

安装

1
ansible all -m shell -a "yum install iptables iptables-services -y"

编辑规则

1
2
3
4
5
6
7
8
9
[root@k8s-w1 ~]# cat /etc/sysconfig/iptables
# sample configuration for iptables service
# you can edit this manually or use system-config-firewall
# please do not ask us to add additional ports/services to this default configuration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

分发规则到剩余服务器

1
2
ansible all -m copy -a "src=/etc/sysconfig/iptables dest=/etc/sysconfig/iptables"
ansible all -m shell -a "systemctl enable iptables.service --now"

优化配置

1
2
ansible all -m shell -a "setenforce 0"
ansible all -m shell -a "ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime"

开始安装

1
sealos run labring/kubernetes:v1.25.7 labring/cilium:v1.13.0 --masters 192.168.6.17 --nodes 192.168.6.21,192.168.6.22,192.168.6.23 # 若没有配置ssh免密码 就追加 `-p <服务器集群密码>`

集群节点状态

1
2
3
4
5
6
[root@k8s-m1 ~]# kubectl get node -A
NAME     STATUS   ROLES           AGE   VERSION
k8s-m1   Ready    control-plane   86s   v1.25.7
k8s-w1   Ready    <none>          50s   v1.25.7
k8s-w2   Ready    <none>          50s   v1.25.7
k8s-w3   Ready    <none>          52s   v1.25.7

集群k8s状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@k8s-m1 ~]# kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   cilium-jntdj                       1/1     Running   0          64s
kube-system   cilium-kr586                       1/1     Running   0          64s
kube-system   cilium-lb7h5                       1/1     Running   0          64s
kube-system   cilium-operator-7d5855586f-j64pp   1/1     Running   0          64s
kube-system   cilium-vbn7t                       1/1     Running   0          64s
kube-system   coredns-565d847f94-jnnld           1/1     Running   0          85s
kube-system   coredns-565d847f94-l6swn           1/1     Running   0          85s
kube-system   etcd-k8s-m1                        1/1     Running   0          99s
kube-system   kube-apiserver-k8s-m1              1/1     Running   0          99s
kube-system   kube-controller-manager-k8s-m1     1/1     Running   0          98s
kube-system   kube-proxy-7k7l2                   1/1     Running   0          67s
kube-system   kube-proxy-8qsmz                   1/1     Running   0          67s
kube-system   kube-proxy-bt9r4                   1/1     Running   0          85s
kube-system   kube-proxy-s8dwf                   1/1     Running   0          69s
kube-system   kube-scheduler-k8s-m1              1/1     Running   0          99s
kube-system   kube-sealos-lvscare-k8s-w1         1/1     Running   0          59s
kube-system   kube-sealos-lvscare-k8s-w2         1/1     Running   0          60s
kube-system   kube-sealos-lvscare-k8s-w3         1/1     Running   0          61s

k8s cilium状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@k8s-m1 ~]# cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:          OK
 \__/¯¯\__/    Operator:        OK
 /¯¯\__/¯¯\    Hubble Relay:    disabled
 \__/¯¯\__/    ClusterMesh:     disabled
    \__/

Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 4, Ready: 4/4, Available: 4/4
Containers:       cilium             Running: 4
                  cilium-operator    Running: 1
Cluster Pods:     2/2 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.13.0: 4
                  cilium-operator    quay.io/cilium/operator:v1.13.0: 1

重命名节点role

1
2
3
kubectl label node k8s-w1 kubernetes.io/role=worker
kubectl label node k8s-w2 kubernetes.io/role=worker
kubectl label node k8s-w3 kubernetes.io/role=worker

修改成功

1
2
3
4
5
6
[root@k8s-m1 ~]# kubectl get node -A
NAME     STATUS   ROLES           AGE     VERSION
k8s-m1   Ready    control-plane   2m41s   v1.25.7
k8s-w1   Ready    worker          2m5s    v1.25.7
k8s-w2   Ready    worker          2m5s    v1.25.7
k8s-w3   Ready    worker          2m7s    v1.25.7

helm安装

1
sealos run labring/helm:v3.11.2

检查

1
2
[root@k8s-m1 ~]# helm version
version.BuildInfo{Version:"v3.11.2", GitCommit:"912ebc1cd10d38d340f048efaf0abda047c3468e", GitTreeState:"clean", GoVersion:"go1.18.10"}

kube-proxy平替(可选)

注意⚠️，请不要在没有充分调研的情况在生产环境实施该方案。

kube-proxy卸载

1
2
kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxy

iptables规则处理

1
ansible all -m shell -a "iptables-save | grep -v KUBE | iptables-restore"

clium卸载重装

⚠️ 请注意重新安装时，cilium版本要与原版本保持一致。

安装参数介绍Kubernetes Without kube-proxy

可以在这一步将Cilium Metrics同时开启，详情在另一篇文章。

若需要开启可观测监控，需要讲一下的配置添加cilium install的配置项中。

1
2
3
4
5
6
7
--helm-set prometheus.enabled=true  \
--helm-set operator.prometheus.enabled=true  \
--helm-set hubble.relay.enabled=true  \
--helm-set hubble.ui.enabled=true  \
--helm-set hubble.enabled=true  \
--helm-set hubble.metrics.enableOpenMetrics=true  \
--helm-set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_app\,destination_app\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}"

标准安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
cilium uninstall

cilium install --version 1.16.5  \
  --helm-set tunnel=disabled  \
  --helm-set bpf.masquerade=true  \
  --helm-set bpf.clockProbe=true  \
  --helm-set bpf.waitForMount=true  \
  --helm-set bpf.preallocateMaps=true  \
  --helm-set bpf.tproxy=true  \
  --helm-set bpf.hostRouting=false  \
  --helm-set autoDirectNodeRoutes=true  \
  --helm-set localRedirectPolicy=true  \
  --helm-set enableCiliumEndpointSlice=true  \
  --helm-set enableK8sEventHandover=true  \
  --helm-set loadBalancer.acceleration=native  \
  --helm-set enableK8sEndpointSlice=true  \
  --helm-set wellKnownIdentities.enabled=true  \
  --helm-set bandwidthManager.enable=true  \
  --helm-set hubble.enabled=false  \
  --helm-set installNoConntrackIptablesRules=true  \
  --helm-set egressGateway.enabled=true  \
  --helm-set endpointRoutes.enabled=false  \
  --helm-set kubeProxyReplacement=strict  \
  --helm-set loadBalancer.mode=dsr  \
  --helm-set nodePort.mode=dsr  \
  --helm-set k8sServiceHost=apiserver.cluster.local  \
  --helm-set k8sServicePort=6443  \
  --helm-set ipv4NativeRoutingCIDR=172.26.131.117/32  \
  --helm-set kubeProxyReplacementHealthzBindAddr='0.0.0.0:10256'  \
  --helm-set loadBalancer.serviceTopology=true

检查

Run kubectl -n kube-system exec ds/cilium -- cilium status --verbose

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init)
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.25 (v1.25.7) [linux/amd64]
Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEgressGatewayPolicy", "cilium/v2::CiliumLocalRedirectPolicy", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumEndpointSlice", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Strict   [eth0 192.168.6.17 (Direct Routing)]
Host firewall:          Disabled
CNI Chaining:           none
CNI Config file:        CNI configuration file management disabled
Cilium:                 Ok   1.13.0 (v1.13.0-c9723a8d)
NodeMonitor:            Listening for events on 8 CPUs with 64x4096 of shared memory
Cilium health daemon:   Ok   
IPAM:                   IPv4: 4/254 allocated from 10.0.1.0/24, 
Allocated addresses:
  10.0.1.104 (router)
  10.0.1.177 (kube-system/coredns-565d847f94-l6swn[restored])
  10.0.1.202 (kube-system/coredns-565d847f94-jnnld[restored])
  10.0.1.229 (health)
IPv6 BIG TCP:           Disabled
BandwidthManager:       Disabled
Host Routing:           BPF
Masquerading:           BPF   [eth0]   172.26.131.117/32 [IPv4: Enabled, IPv6: Disabled]
Clock Source for BPF:   ktime
Controller Status:      27/27 healthy
  Name                                  Last success   Last error   Count   Message
  bpf-map-sync-cilium_lxc               9s ago         never        0       no error   
  cilium-health-ep                      48s ago        never        0       no error   
  dns-garbage-collector-job             12s ago        never        0       no error   
  endpoint-1023-regeneration-recovery   never          never        0       no error   
  endpoint-2846-regeneration-recovery   never          never        0       no error   
  endpoint-930-regeneration-recovery    never          never        0       no error   
  endpoint-964-regeneration-recovery    never          never        0       no error   
  endpoint-gc                           3m12s ago      never        0       no error   
  ipcache-inject-labels                 2m58s ago      3m6s ago     0       no error   
  k8s-heartbeat                         12s ago        never        0       no error   
  link-cache                            19s ago        never        0       no error   
  metricsmap-bpf-prom-sync              7s ago         never        0       no error   
  resolve-identity-1023                 2m48s ago      never        0       no error   
  restoring-ep-identity (2846)          2m49s ago      never        0       no error   
  restoring-ep-identity (930)           2m49s ago      never        0       no error   
  restoring-ep-identity (964)           2m49s ago      never        0       no error   
  sync-endpoints-and-host-ips           49s ago        never        0       no error   
  sync-lb-maps-with-k8s-services        2m49s ago      never        0       no error   
  sync-policymap-1023                   38s ago        never        0       no error   
  sync-policymap-2846                   38s ago        never        0       no error   
  sync-policymap-930                    38s ago        never        0       no error   
  sync-policymap-964                    38s ago        never        0       no error   
  sync-to-k8s-ciliumendpoint (1023)     8s ago         never        0       no error   
  sync-to-k8s-ciliumendpoint (2846)     9s ago         never        0       no error   
  sync-to-k8s-ciliumendpoint (930)      4s ago         never        0       no error   
  sync-to-k8s-ciliumendpoint (964)      4s ago         never        0       no error   
  template-dir-watcher                  never          never        0       no error   
Proxy Status:            OK, ip 10.0.1.104, 0 redirects active on ports 10000-20000
Global Identity Range:   min 256, max 65535
Hubble:                  Ok   Current/Max Flows: 1517/4095 (37.05%), Flows/s: 9.12   Metrics: Ok
KubeProxyReplacement Details:
  Status:                 Strict
  Socket LB:              Enabled
  Socket LB Tracing:      Enabled
  Devices:                eth0 192.168.6.17 (Direct Routing)
  Mode:                   DSR
  Backend Selection:      Random
  Session Affinity:       Enabled
  Graceful Termination:   Enabled
  NAT46/64 Support:       Disabled
  XDP Acceleration:       Native
  Services:
  - ClusterIP:      Enabled
  - NodePort:       Enabled (Range: 30000-32767) 
  - LoadBalancer:   Enabled 
  - externalIPs:    Enabled 
  - HostPort:       Enabled
BPF Maps:   dynamic sizing: on (ratio: 0.002500)
  Name                          Size
  Non-TCP connection tracking   147590
  TCP connection tracking       295180
  Endpoint policy               65535
  Events                        8
  IP cache                      512000
  IP masquerading agent         16384
  IPv4 fragmentation            8192
  IPv4 service                  65536
  IPv6 service                  65536
  IPv4 service backend          65536
  IPv6 service backend          65536
  IPv4 service reverse NAT      65536
  IPv6 service reverse NAT      65536
  Metrics                       1024
  NAT                           295180
  Neighbor table                295180
  Global policy                 16384
  Per endpoint policy           65536
  Session affinity              65536
  Signal                        8
  Sockmap                       65535
  Sock reverse NAT              147590
  Tunnel                        65536
Encryption:                       Disabled
Cluster health:                   4/4 reachable   (2023-03-30T04:57:55Z)
  Name                            IP              Node        Endpoints
  kubernetes/k8s-m1 (localhost)   192.168.6.17    reachable   reachable
  kubernetes/k8s-w1               192.168.6.21    reachable   reachable
  kubernetes/k8s-w2               192.168.6.22    reachable   reachable
  kubernetes/k8s-w3               192.168.6.23    reachable   reachable

初始配置

默认存储

k8s存储支持多种模式：本地存储:hostPath/emptyDir，传递网络存储:iscsi/nfs，分布式网络存 储：glusterfs/rbd/cephfs，以及云存储等；
k8s默认容器如果重建，则容器中文件将丢失，为了解决这些问题，通常我们会将容器中需要持久化的文件存储到其他可持久化存储目录中。

安装主机依赖

我选择以nfs作为默认存储

1
sealos exec "yum install nfs-utils -y" # 集群节点都需要安装

配置共享(可选)

nfs服务主机 我主机有限：所以将nfs服务是由k8s-m1提供的，共享文件:在nfs服务主机文件/etc/exportes添加需要共享的目录以及允许的网段。

1
/data/share 192.168.6.0/24(insecure,rw,async,no_root_squash)

在服务器k8s-m1(跟随nfs磁盘)启动nfs服务

1
2
systemctl enable nfs-server
systemctl start nfs-server

添加默认存储

注：nfs镜像是拉取的上游(k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2)的镜像

1
2
3
4
5
6
7
8
9
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --set image.repository=ka1i137/sig-storage-nfs-subdir-external-provisioner \
  --set image.tag=v4.0.2 \
  --set storageClass.name=nfs-client \
  --set storageClass.defaultClass=true \
  --set nfs.server=192.168.6.11 \
  --set nfs.path=/mnt/data/k8s/NFS

验证

1
2
3
[root@k8s-m1 ~]# kubectl get sc -A
NAME                   PROVISIONER                                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   cluster.local/nfs-subdir-external-provisioner   Delete          Immediate           true                   20s

部署服务

创建命名空间

1
kubectl create namespace app

编写yaml

whoami.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  namespace: app
  labels:
    app: whoami

spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
          ports:
            - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: app
spec:
  type: NodePort
  ports:
    - name: http-80
      port: 80
      nodePort: 30888
      targetPort: 80
  selector:
    app: whoami

开始部署

1
2
3
4
5
6
7
8
9
[root@k8s-m1 app]# kubectl apply -f whoami.yaml 
deployment.apps/whoami created
service/whoami created
[root@k8s-m1 app]# kubectl get pod -A | grep "^app"
app  whoami-6bbfdbb69c-tss9n  0/1  ContainerCreating  0  22s
app  whoami-6bbfdbb69c-v26d4  0/1  ContainerCreating  0  22s
[root@k8s-m1 app]# kubectl get svc -n app
NAME     TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
whoami   NodePort   10.96.1.5    <none>        80:30888/TCP   36s

验证部署

1
2
3
4
5
6
7
8
9
10
11
[root@k8s-m1 app]# curl 192.168.6.17:30888 # ip为k8s集群ip其中一个都可以
Hostname: whoami-6bbfdbb69c-v26d4
IP: 127.0.0.1
IP: ::1
IP: 10.0.2.67
IP: fe80::5019:c7ff:fe03:42e9
RemoteAddr: 10.0.0.220:45186
GET / HTTP/1.1
Host: 192.168.137.11:30888
User-Agent: curl/7.76.1
Accept: */*

基本监控

metrics-server

yaml配置下载

1
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

若默认镜像无法下载,可以使用我转存的ka1i137/k8s-metrics-server:v0.6.2。

若显示tls错误，可以在启动参数添加- --kubelet-insecure-tls

1
2
3
4
5
6
7
8
9
10
spec:
  containers:
  - args:
    - --cert-dir=/tmp
    - --secure-port=4443
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --kubelet-use-node-status-port
    - --metric-resolution=15s
    - --kubelet-insecure-tls # 新增
    image: ka1i137/k8s-metrics-server:v0.6.2

dubernetes-dashboard

1
sealos run labring/kubernetes-dashboard:v2.7.0

编辑

开放访问

dashboard-adminuser.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

然后应用

1
kubectl apply -f dashboard-adminuser.yaml

最后获取登录token

1
kubectl -n kubernetes-dashboard create token admin-user

Tips 开放NodePort

1
kubectl edit svc -n kubernetes-dashboard   kubernetes-dashboard

可以看到svc的配置修改为如下。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-03-13T09:42:15Z"
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  resourceVersion: "20876"
  uid: f160d21f-2244-46d1-894c-d0cbd96383c2
spec:
  clusterIP: 10.96.3.50
  clusterIPs:
  - 10.96.3.50
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 30880 # 新增
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: NodePort  # 修改
status:
  loadBalancer: {}

k9s

官网：k9scli.io

Ingress

我使用的ingress是traefik，可以根据自己的具体需求选择其他的ingress。配置在另一篇文章。

Very Nice!

Mardan MDevOps Engineer

Hi there, I’m Mardan(ka1i).