k8s集群之kubernetes-dashboard和kube-dns组件部署安装

说明

最好先部署kube-dns，有些组合服务直接主机用hostname解析，例如redis主从，heapster监控组件influxdb、grafana之间等。

参考文档

https://greatbsky.github.io/KubernetesDoc/kubernetes1.5.2/cn.html

http://tonybai.com/2017/01/20/integrate-heapster-for-kubernetes-dashboard/

相关文档汇总如下

k8s集群安装部署

http://jerrymin.blog.51cto.com/3002256/1898243

k8s集群RC、SVC、POD部署

http://jerrymin.blog.51cto.com/3002256/1900260 

k8s集群组件kubernetes-dashboard和kube-dns部署

http://jerrymin.blog.51cto.com/3002256/1900508

k8s集群监控组件heapster部署

http://jerrymin.blog.51cto.com/3002256/1904460

k8s集群反向代理负载均衡组件部署

http://jerrymin.blog.51cto.com/3002256/1904463 

k8s集群挂载volume之nfs

http://jerrymin.blog.51cto.com/3002256/1906778

k8s集群挂载volume之glusterfs

http://jerrymin.blog.51cto.com/3002256/1907274

一，部署kube-dashboard插件

1，设置好谷歌IP否则dashboard镜像下载不了

[root@k8s-master ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.17.3.20  k8s-master  hub.yiche.io

172.17.3.7   k8s-node1

172.17.3.8   k8s-node2

10.1.8.2 redis-master

61.91.161.217 google.com

61.91.161.217 gcr.io      #在国内，你可能需要指定gcr.io的host，如不指定你需要手动下载google-containers. 如何下载其他文档说的方法挺多

61.91.161.217 www.gcr.io

61.91.161.217 console.cloud.google.com

61.91.161.217 storage.googleapis.com

2，创建pod

[root@k8s-master kubernetes]# wget https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml

[root@k8s-master kubernetes]# kubectl create -f kubernetes-dashboard.yaml 

deployment "kubernetes-dashboard" created

service "kubernetes-dashboard" created

[root@k8s-master kubernetes]# kubectl get service --namespace=kube-system

NAME                   CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE

kubernetes-dashboard   10.254.173.39   <nodes>       80:31595/TCP   2h

[root@k8s-master kubernetes]#  kubectl get pods --namespace=kube-system -o wide

NAME                                  READY     STATUS    RESTARTS   AGE       IP         NODE

kubernetes-dashboard-47555765-hnvp9   1/1       Running   0          2h        10.1.8.6   k8s-node2

3，验证

可以通过node节点IP加端口访问，这里的是http://172.17.3.7:31595或http://172.17.3.8:31595

4，安装headper插件方便监控流量图

https://github.com/kubernetes/heapster

发现要在k8s集群上跑heapster，需要安装InfluxDB与Google Cloud Monitoring and Google Cloud Logging

这是分开安装的资料，下面有个kube.sh脚本可以直接安装这两个相关组件

安装influxdb资料

https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb

安装google云监控和日志

https://github.com/kubernetes/heapster/blob/master/docs/google.md

git clone https://github.com/kubernetes/heapster

cd heapster/deploy/

注意修改一个配置，这样才能和dashboard连接起来

vim deploy/kube-config/influxdb/heapster-deployment.yaml 

        - /heapster

        - --source=kubernetes:http://172.17.3.20:8080?inClusterConfig=false

        - --sink=influxdb:http://monitoring-influxdb:8086

./kube.sh start

这里是自带监控脚本安装，安装完成后会生成

[root@k8s-master deploy]#  kubectl get pods -o wide  --namespace=kube-system

NAME                                   READY     STATUS    RESTARTS   AGE       IP          NODE

heapster-791010959-npzc9               1/1       Running   0          21s       10.1.8.7    k8s-node2

kube-dns-3019842428-tqjg8              3/3       Running   6          1d        10.1.8.10   k8s-node2

kube-dns-autoscaler-2715466192-gs937   1/1       Running   2          22h       10.1.8.5    k8s-node2

kubernetes-dashboard-47555765-szbw6    1/1       Running   1          3h        10.1.8.11   k8s-node2

monitoring-grafana-3730655072-s466b    1/1       Running   0          21s       10.1.89.2   k8s-node1

monitoring-influxdb-957705310-5rswg    1/1       Running   0          21s       10.1.89.6   k8s-node1

[root@k8s-master deploy]#  kubectl logs -f heapster-791010959-npzc9  -n kube-system

I0223 06:37:52.935620       1 heapster.go:71] /heapster --source=kubernetes:http://172.17.3.20:8080?inClusterConfig=false --sink=influxdb:http://monitoring-influxdb:8086

I0223 06:37:52.935815       1 heapster.go:72] Heapster version v1.3.0-beta.1

I0223 06:37:52.935957       1 configs.go:61] Using Kubernetes client with master "http://172.17.3.20:8080" and version v1

I0223 06:37:52.935976       1 configs.go:62] Using kubelet port 10255

二，部署kube-dns插件

1，下载配置文件

root@k8s-master ~]# mkdir -p /root/kubernetes

[root@k8s-master kubernetes]# ffor item in Makefile kubedns-controller.yaml.base kubedns-svc.yaml.base transforms2salt.sed transforms2sed.sed; do

    wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/$item

done

[root@k8s-master kubernetes]# ls

kubedns-controller.yaml.base  kubedns-svc.yaml.base  kubernetes-dashboard.yaml  Makefile  transforms2salt.sed  transforms2sed.sed

2，修改本地环境

[root@k8s-master kubernetes]# kubectl get svc |grep kubernetes  

kubernetes     10.254.0.1       <none>        443/TCP        6d

[root@k8s-master kubernetes]# DNS_SERVER_IP=10.254.0.10

[root@k8s-master kubernetes]# DNS_DOMAIN=cluster.local

修改transforms2sed.sed里面2个变量为实际网段及搜索域，修改前如下

[root@k8s-master kubernetes]# cat transforms2sed.sed 

s/__PILLAR__DNS__SERVER__/$DNS_SERVER_IP/g

s/__PILLAR__DNS__DOMAIN__/$DNS_DOMAIN/g

/__PILLAR__FEDERATIONS__DOMAIN__MAP__/d

s/__MACHINE_GENERATED_WARNING__/Warning: This is a file generated from the base underscore template file: __SOURCE_FILENAME__/g

修改后如下

[root@k8s-master kubernetes]# cat transforms2sed.sed 

s/__PILLAR__DNS__SERVER__/10.254.0.10/g

s/__PILLAR__DNS__DOMAIN__/cluster.local/g

/__PILLAR__FEDERATIONS__DOMAIN__MAP__/d

s/__MACHINE_GENERATED_WARNING__/Warning: This is a file generated from the base underscore template file: __SOURCE_FILENAME__/g

编译

[root@k8s-master kubernetes]# make

sed -f transforms2salt.sed kubedns-controller.yaml.base | sed s/__SOURCE_FILENAME__/kubedns-controller.yaml.base/g > kubedns-controller.yaml.in

sed -f transforms2salt.sed kubedns-svc.yaml.base | sed s/__SOURCE_FILENAME__/kubedns-svc.yaml.base/g > kubedns-svc.yaml.in

sed -f transforms2sed.sed kubedns-controller.yaml.base  | sed s/__SOURCE_FILENAME__/kubedns-controller.yaml.base/g > kubedns-controller.yaml.sed

sed -f transforms2sed.sed kubedns-svc.yaml.base  | sed s/__SOURCE_FILENAME__/kubedns-svc.yaml.base/g > kubedns-svc.yaml.sed

3，修改RC和SVC文件

#现在编辑文件kubedns-controller.yaml.sed删除（这是用#注释）volume的配置，以避免两个问题：

#1.错误: error validating "kubedns-controller.yaml.sed.bak": error validating data: found invalid field optional for v1.ConfigMapVolumeSource; if you choose to ignore these errors, turn validation off with --validate=false

#2.创建容器后, tail /var/log/messages, 显示错误: configmaps "kube-dns" not found 

[root@k8s-master kubernetes]# cat kubedns-controller.yaml.sed 

# Copyright 2016 The Kubernetes Authors.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Should keep target in cluster/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml

# in sync with this file.

# Warning: This is a file generated from the base underscore template file: kubedns-controller.yaml.base

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: kube-dns

  namespace: kube-system

  labels:

    k8s-app: kube-dns

    kubernetes.io/cluster-service: "true"

spec:

  # replicas: not specified here:

  # 1. In order to make Addon Manager do not reconcile this replicas parameter.

  # 2. Default is 1.

  # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.

  strategy:

    rollingUpdate:

      maxSurge: 10%

      maxUnavailable: 0

  selector:

    matchLabels:

      k8s-app: kube-dns

  template:

    metadata:

      labels:

        k8s-app: kube-dns

      annotations:

        scheduler.alpha.kubernetes.io/critical-pod: ''

        scheduler.alpha.kubernetes.io/tolerations: '[{"key":"CriticalAddonsOnly", "operator":"Exists"}]'

    spec:

#      volumes:

#      - name: kube-dns-config

#        configMap:

#          name: kube-dns

#          optional: true

      containers:

      - name: kubedns

        p_w_picpath: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.12.1

        resources:

          # TODO: Set memory limits when we've profiled the container for large

          # clusters, then set request = limit to keep this container in

          # guaranteed class. Currently, this container falls into the

          # "burstable" category so the kubelet doesn't backoff from restarting it.

          limits:

            memory: 170Mi

          requests:

            cpu: 100m

            memory: 70Mi

        livenessProbe:

          httpGet:

            path: /healthcheck/kubedns

            port: 10054

            scheme: HTTP

          initialDelaySeconds: 60

          timeoutSeconds: 5

          successThreshold: 1

          failureThreshold: 5

        readinessProbe:

          httpGet:

            path: /readiness

            port: 8081

            scheme: HTTP

          # we poll on pod startup for the Kubernetes master service and

          # only setup the /readiness HTTP server once that's available.

          initialDelaySeconds: 3

          timeoutSeconds: 5

        args:

        - --domain=cluster.local.

        - --dns-port=10053

        - --config-dir=/kube-dns-config

        - --v=2

        env:

        - name: PROMETHEUS_PORT

          value: "10055"

        ports:

        - containerPort: 10053

          name: dns-local

          protocol: UDP

        - containerPort: 10053

          name: dns-tcp-local

          protocol: TCP

        - containerPort: 10055

          name: metrics

          protocol: TCP

#        volumeMounts:

#        - name: kube-dns-config

#          mountPath: /kube-dns-config

      - name: dnsmasq

        p_w_picpath: gcr.io/google_containers/k8s-dns-dnsmasq-amd64:1.12.1

        livenessProbe:

          httpGet:

            path: /healthcheck/dnsmasq

            port: 10054

            scheme: HTTP

          initialDelaySeconds: 60

          timeoutSeconds: 5

          successThreshold: 1

          failureThreshold: 5

        args:

        - --cache-size=1000

        - --server=/cluster.local/127.0.0.1#10053

        - --server=/in-addr.arpa/127.0.0.1#10053

        - --server=/ip6.arpa/127.0.0.1#10053

        - --log-facility=-

        ports:

        - containerPort: 53

          name: dns

          protocol: UDP

        - containerPort: 53

          name: dns-tcp

          protocol: TCP

        # see: https://github.com/kubernetes/kubernetes/issues/29055 for details

        resources:

          requests:

            cpu: 150m

            memory: 10Mi

      - name: sidecar

        p_w_picpath: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.12.1

        livenessProbe:

          httpGet:

            path: /metrics

            port: 10054

            scheme: HTTP

          initialDelaySeconds: 60

          timeoutSeconds: 5

          successThreshold: 1

          failureThreshold: 5

        args:

        - --v=2

        - --logtostderr

        - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A

        - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A

        ports:

        - containerPort: 10054

          name: metrics

          protocol: TCP

        resources:

          requests:

            memory: 20Mi

            cpu: 10m

      dnsPolicy: Default  # Don't use cluster DNS.

 

 

[root@k8s-master kubernetes]# cat kubedns-svc.yaml.sed

# Copyright 2016 The Kubernetes Authors.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Warning: This is a file generated from the base underscore template file: kubedns-svc.yaml.base

apiVersion: v1

kind: Service

metadata:

  name: kube-dns

  namespace: kube-system

  labels:

    k8s-app: kube-dns

    kubernetes.io/cluster-service: "true"

    kubernetes.io/name: "KubeDNS"

spec:

  selector:

    k8s-app: kube-dns

  clusterIP: 10.254.0.10

  ports:

  - name: dns

    port: 53

    protocol: UDP

  - name: dns-tcp

    port: 53

    protocol: TCP  

[root@k8s-master kubernetes]# vim /etc/kubernetes/controller-manager  #若忘记修改可能会报错，默认参数为空：No API token found for service account "default"

KUBE_CONTROLLER_MANAGER_ARGS="--service-account-private-key-file=/var/run/kubernetes/apiserver.key --root-ca-file=/var/run/kubernetes/apiserver.crt"

4，创建RC和SVC

为保证创建能够成功，最好先在node节点下载如下镜像

docker pull gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.12.1

docker pull gcr.io/google_containers/k8s-dns-dnsmasq-amd64:1.12.1

docker pull gcr.io/google_containers/k8s-dns-sidecar-amd64:1.12.1

创建

[root@k8s-master kubernetes]# kubectl create -f kubedns-controller.yaml.sed

deployment "kube-dns" created

[root@k8s-master kubernetes]# kubectl create -f kubedns-svc.yaml.sed

service "kube-dns" created

验证

[root@k8s-master kubernetes]#  kubectl get svc --namespace=kube-system

NAME                   CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE

kube-dns               10.254.0.10     <none>        53/UDP,53/TCP   7m

kubernetes-dashboard   10.254.173.39   <nodes>       80:31595/TCP    5h

[root@k8s-master kubernetes]#  kubectl get pods --namespace=kube-system -o wide

NAME                                  READY     STATUS    RESTARTS   AGE       IP         NODE

kube-dns-3019842428-cds2l             3/3       Running   0          8m        10.1.8.4   k8s-node2

kubernetes-dashboard-47555765-hnvp9   1/1       Running   1          5h        10.1.8.6   k8s-node2

[root@k8s-master kubernetes]# kubectl cluster-info

Kubernetes master is running at http://localhost:8080

KubeDNS is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

5，node节点使用dns解析服务

kubelet里面kubelet_args原来是空，现在加入dns服务器配置，然后重启kubelet服务

[root@k8s-node1 ~]# cat /etc/kubernetes/kubelet|grep -Ev "^#|^$"

KUBELET_ADDRESS="--address=0.0.0.0"

KUBELET_PORT="--port=10250"

KUBELET_HOSTNAME="--hostname-override=k8s-node1"

KUBELET_API_SERVER="--api-servers=http://172.17.3.20:8080"

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-p_w_picpath=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

KUBELET_ARGS="--cluster_dns=10.254.0.10 --cluster_domain=cluster.local"

重启

[root@k8s-node1 ~]# systemctl restart kubelet && systemctl status kubelet

验证

6，可以进入容器验证

这个容器所在node1节点修改了DNS参数，所以里面容器resolv.conf文件修改了，nslookup也能找到redis-master了。

[root@k8s-master kubernetes]# kubectl get pods -o wide |grep redis

redis-master-qg457             1/1       Running   1          6d        10.1.8.2    k8s-node2

redis-slave-40tjp              1/1       Running   1          5d        10.1.8.3    k8s-node2

redis-slave-q5kj0              1/1       Running   0          20h       10.1.8.9    k8s-node2

redis-slave-r6mg2              1/1       Running   0          59m       10.1.89.4   k8s-node1

[root@k8s-master kubernetes]# kubectl exec -ti redis-slave-r6mg2 -- bash

root@redis-slave-r6mg2:/data# apt-get update

root@redis-slave-r6mg2:/data# apt-get install dnsutils

root@redis-slave-r6mg2:/data# cat /etc/resolv.conf 

search default.svc.cluster.local svc.cluster.local cluster.local

nameserver 10.254.0.10

options ndots:5

root@redis-slave-r6mg2:/data# nslookup redis-master

Server:10.254.0.10

Address:10.254.0.10#53

Non-authoritative answer:

Name:redis-master.default.svc.cluster.local

Address: 10.254.6.192

root@redis-slave-r6mg2:/data# ping gcr.io

PING gcr.io (64.233.187.82): 48 data bytes

对比另外一个容器所在node2节点没有修改DNS参数

[root@k8s-master kubernetes]# kubectl exec -ti redis-slave-r6mg2 -- bash

root@redis-slave-q5kj0:/data# apt-get update

root@redis-slave-q5kj0:/data# apt-get install dnsutils

root@redis-slave-q5kj0:/data# cat /etc/resolv.conf 

nameserver 203.196.0.6

nameserver 203.196.1.6

options ndots:5

root@redis-slave-q5kj0:/data# nslookup redis-master

Server:203.196.0.6

Address:203.196.0.6#53

** server can't find redis-master: NXDOMAIN

root@redis-slave-q5kj0:/data# ping gcr.io

PING gcr.io (64.233.187.82): 48 data bytes

可以看到redis-slave想要和redis-master通信需要有DNS支持，如果没有可以在/etc/hosts里面绑定redis-master的容器IP也行，但是2个容器外部解析和上网都没有问题

7，安装dns-horizontal-autoscaling插件，在集群里负责扩展DNS服务器

[root@k8s-master kubernetes]# wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml

[root@k8s-master kubernetes]# kubectl create -f dns-horizontal-autoscaler.yaml

deployment "kube-dns-autoscaler" created

[root@k8s-master kubernetes]# cat dns-horizontal-autoscaler.yaml |grep p_w_picpath

        p_w_picpath: gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.0.0

[root@k8s-master kubernetes]# kubectl get configmap --namespace=kube-system

NAME                  DATA      AGE

kube-dns-autoscaler   1         15m

[root@k8s-master kubernetes]# kubectl get deployment --namespace=kube-system

NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE

kube-dns               1         1         1            1           2h

kube-dns-autoscaler    1         1         1            1           16m

kubernetes-dashboard   1         1         1            1           1d