kubectl scale命令可以来实现 Pod 的扩缩容功能，但是这个毕竟是完全手动操作的，要应对线上的各种复杂情况，我们需要能够做到自动化去感知业务，来自动进行扩缩容。为此，Kubernetes 也为我们提供了这样的一个资源对象：Horizontal Pod Autoscaling（Pod 水平自动伸缩），简称HPA，HPA 通过监控分析一些控制器控制的所有 Pod 的负载变化情况来确定是否需要调整 Pod 的副本数量，这是 HPA 最基本的原理：
通过 kubectl autoscale 命令来创建一个 HPA 资源对象，HPA Controller默认30s轮询一次（可通过 kube-controller-manager 的--horizontal-pod-autoscaler-sync-period 参数进行设置），查询指定的资源中的 Pod 资源使用率，并且与创建时设定的值和指标做对比，从而实现自动伸缩的功能。
Metrics Server
在 HPA 的第一个版本中，我们需要 Heapster 提供 CPU 和内存指标，在 HPA v2 过后就需要安装 Metrcis Server 了，Metrics Server 可以通过标准的 Kubernetes API 把监控数据暴露出来，有了 Metrics Server 之后，我们就完全可以通过标准的 Kubernetes API 来访问我们想要获取的监控数据了：
1
|
https://10.96.0.1/apis/metrics.k8s.io/v1beta1/namespaces/<namespace-name>/pods/<pod-name>
|
比如当我们访问上面的 API 的时候，我们就可以获取到该 Pod 的资源数据，这些数据其实是来自于 kubelet 的 Summary API 采集而来的。不过需要说明的是我们这里可以通过标准的 API 来获取资源监控数据，并不是因为 Metrics Server 就是 APIServer 的一部分，而是通过 Kubernetes 提供的 Aggregator 汇聚插件来实现的，是独立于 APIServer 之外运行的。
聚合API
Aggregator 允许开发人员编写一个自己的服务，把这个服务注册到 Kubernetes 的 APIServer 里面去，这样我们就可以像原生的 APIServer 提供的 API 使用自己的 API 了，我们把自己的服务运行在 Kubernetes 集群里面，然后 Kubernetes 的 Aggregator 通过 Service 名称就可以转发到我们自己写的 Service 里面去了。这样这个聚合层就带来了很多好处：
- 增加了 API 的扩展性，开发人员可以编写自己的 API 服务来暴露他们想要的 API。
- 丰富了 API，核心 kubernetes 团队阻止了很多新的 API 提案，通过允许开发人员将他们的 API 作为单独的服务公开，这样就无须社区繁杂的审查了。
- 开发分阶段实验性 API，新的 API 可以在单独的聚合服务中开发，当它稳定之后，在合并会 APIServer 就很容易了。
- 确保新 API 遵循 Kubernetes 约定，如果没有这里提出的机制，社区成员可能会被迫推出自己的东西，这样很可能造成社区成员和社区约定不一致。
安装
所以现在要使用 HPA，就需要在集群中安装 Metrics Server 服务，要安装 Metrics Server 就需要开启 Aggregator，因为 Metrics Server 就是通过该代理进行扩展的，不过我们集群是通过 Kubeadm 搭建的，默认已经开启了，如果是二进制方式安装的集群，需要单独配置 kube-apsierver 添加如下所示的参数：
1
2
3
4
5
6
7
|
--requestheader-client-ca-file=<path to aggregator CA cert>
--requestheader-allowed-names=aggregator
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-User
--proxy-client-cert-file=<path to aggregator proxy cert>
--proxy-client-key-file=<path to aggregator proxy key>
|
如果 kube-proxy 没有和 APIServer 运行在同一台主机上，那么需要确保启用了如下 kube-apsierver 的参数：
1
|
--enable-aggregator-routing=true
|
对于这些证书的生成方式，我们可以查看官方文档：https://github.com/kubernetes-sigs/apiserver-builder-alpha/blob/master/docs/concepts/auth.md。
Aggregator 聚合层启动完成后，就可以来安装 Metrics Server 了，我们可以获取该仓库的安装资源清单：
1
2
3
4
|
helm repo search |grep metrics-server
aliyuncs/metrics-server 4.1.4 0.3.6 Metrics Server is a cluster-wide aggregator of ...
bitnami/metrics-server 5.10.12 0.5.2 Metrics Server is a cluster-wide aggregator of ...
stable/metrics-server 2.11.4 0.3.6 DEPRECATED - Metrics Server is a cluster-wide a...
|
使用helm方式选择bitnami仓库下的metrics-server直接部署到系统命名空间中
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
helm upgrade --install metrics-server bitnami/metrics-server/ --set apiService.create=true -n kube-system
Release "metrics-server" does not exist. Installing it now.
NAME: metrics-server
LAST DEPLOYED: Wed Feb 16 13:53:52 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: metrics-server
CHART VERSION: 5.10.12
APP VERSION: 0.5.2
** Please be patient while the chart is being deployed **
The metric server has been deployed.
|
等部署完成后，可以查看 Pod 日志是否正常：
1
2
3
4
5
6
|
kcp -n kube-system -l app.kubernetes.io/instance=metrics-server
NAME READY STATUS RESTARTS AGE
metrics-server-8658dd79dc-b8kxg 0/1 Running 0 4m36s
kcl metrics-server-8658dd79dc-b8kxg -n kube-system
......
E0216 05:53:55.727704 1 scraper.go:139] "Failed to scrape node" no such host"
|
我们可以发现 Pod 中出现了一些错误信息：xxx: no such host，我们看到这个错误信息一般就可以确定是 DNS 解析不了造成的，我们可以看到 Metrics Server 会通过 kubelet 的 10250 端口获取信息，使用的是 hostname，我们部署集群的时候在节点的 /etc/hosts 里面添加了节点的 hostname 和 ip 的映射，但是是我们的 Metrics Server 的 Pod 内部并没有这个 hosts 信息，当然也就不识别 hostname 了，要解决这个问题，有两种方法：
第一种方法就是在集群内部的 DNS 服务里面添加上 hostname 的解析，比如我们这里集群中使用的是 CoreDNS，我们就可以去修改下 CoreDNS 的 Configmap 信息，添加上 hosts 信息
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
ready
hosts {
10.101.16.232 node01
10.101.16.233 node02
}
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods verified
fallthrough in-addr.arpa ip6.arpa
}
autopath @kubernetes
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: "2020-08-07T08:05:11Z"
name: coredns
namespace: kube-system
|
这样当在集群内部访问集群的 hostname 的时候就可以解析到对应的 ip 了，另外一种方法就是在 metrics-server 的启动参数中修改 kubelet-preferred-address-types 参数，如下：
1
2
|
args:
- --kubelet-preferred-address-types=InternalIP
|
在helm默认配置文件values.yaml中修改args参数
1
2
3
4
5
|
## extraArgs:
## kubelet-insecure-tls: true
## kubelet-preferred-address-types: InternalIP
##
extraArgs: {}
|
修改为如下配置：
1
2
3
4
5
6
|
## extraArgs:
## kubelet-insecure-tls: true
## kubelet-preferred-address-types: InternalIP
##
extraArgs:
kubelet-preferred-address-types: InternalIP
|
这里使用第二种方式，然后重新安装
1
2
3
4
5
|
kcp -n kube-system -l app.kubernetes.io/instance=metrics-server
NAME READY STATUS RESTARTS AGE
metrics-server-76ff9c6b67-7hgdb 0/1 Running 0 54s
kcl metrics-server-76ff9c6b67-7hgdb -n kube-system
E0216 06:08:56.705571 1 scraper.go:139] "Failed to scrape node" err="Get: x509: certificate has expired or is not yet valid: current time 2022-02-16T06:08:56Z is after 2021-11-17T01:36:58Z"
|
因为部署集群的时候，CA 证书并没有把各个节点的 IP 签上去，所以这里 Metrics Server 通过 IP 去请求时，提示签的证书没有对应的 IP（错误：x509: cannot validate certificate for 10.101.17.1 because it doesn’t contain any IP SANs），我们可以添加一个–kubelet-insecure-tls参数跳过证书校验：
1
2
3
4
5
6
7
|
## extraArgs:
## kubelet-insecure-tls: true
## kubelet-preferred-address-types: InternalIP
##
extraArgs:
kubelet-preferred-address-types: InternalIP
kubelet-insecure-tls: true
|
然后重新安装即可成功！可通过如下命令来验证
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
|
kcp -n kube-system -l app.kubernetes.io/instance=metrics-server
NAME READY STATUS RESTARTS AGE
metrics-server-569df6976c-wlshv 1/1 Running 0 34s
I0216 06:14:12.663402 1 serving.go:341] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0216 06:14:13.528767 1 secure_serving.go:202] Serving securely on [::]:8443
kubectl get apiservice | grep metrics
v1beta1.metrics.k8s.io kube-system/metrics-server True 22m
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"|jq |head -n 9
{
"kind": "NodeMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"metadata": {
"name": "cn-beijing.10.101.16.232",
"creationTimestamp": "2022-02-16T06:17:53Z",
kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
cn-beijing.10.101.16.232 672m 16% 10224Mi 69%
cn-beijing.10.101.16.233 1260m 31% 9259Mi 63%
cn-beijing.10.101.16.235 459m 11% 8463Mi 57%
cn-beijing.10.101.16.236 1114m 27% 10666Mi 72%
cn-beijing.10.101.16.242 1179m 29% 8038Mi 54%
cn-beijing.10.101.16.243 637m 31% 5324Mi 78%
cn-beijing.10.101.16.244 500m 25% 5425Mi 79%
cn-beijing.10.101.16.245 610m 15% 10709Mi 73%
cn-beijing.10.101.16.253 246m 12% 4358Mi 63%
cn-beijing.10.101.17.0 1767m 44% 10135Mi 68%
cn-beijing.10.101.17.1 1506m 37% 9340Mi 62%
|
现在我们可以通过 kubectl top 命令来获取到资源数据了，证明 Metrics Server 已经安装成功了
基于CPU
现在我们用 Deployment 来创建一个 Nginx Pod，然后利用 HPA 来进行自动扩缩容。资源清单如下所示：（hpa-nginx.yaml）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
|
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx-test
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
labels:
app: nginx
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: NotIn
values:
- cn-beijing.10.101.16.235
- cn-beijing.10.101.16.236
containers:
- image: nginx:1.18.0-alpine
imagePullPolicy: IfNotPresent
name: nginx
livenessProbe:
httpGet:
path: /
port: web
initialDelaySeconds: 30
periodSeconds: 5
readinessProbe:
httpGet:
path: /
port: web
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 2
lifecycle:
preStop:
exec:
command: ["/bin/bash","-c","sleep 120"]
ports:
- containerPort: 80
name: web
protocol: TCP
|
然后直接创建Deployment
1
2
3
4
|
kubectl apply -f hpa-nginx.yaml
kcp -l app=nginx -n kube-system
NAME READY STATUS RESTARTS AGE
nginx-test-7cb4d568f5-brf4w 1/1 Running 0 2m7s
|
现在创建一个HPA资源对象，可以使用kubectl autoscale命令来创建
1
2
3
4
5
|
kubectl autoscale deployment nginx-test --cpu-percent=10 --min=1 --max=10 -n kube-system
horizontalpodautoscaler.autoscaling/nginx-test autoscaled
kubectl get hpa -n kube-system
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-test Deployment/nginx-test <unknown>/10% 1 10 1 34s
|
此命令创建了一个关联资源 nginx-test 的 HPA，最小的 Pod 副本数为1，最大为10。HPA 会根据设定的 cpu 使用率（10%）动态的增加或者减少 Pod 数量。
当然我们依然还是可以通过创建 YAML 文件的形式来创建 HPA 资源对象。如果我们不知道怎么编写的话，可以查看上面命令行创建的HPA的YAML文件：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
annotations:
autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2022-02-16T07:37:10Z","reason":"SucceededGetScale","message":"the
HPA controller was able to get the target''s current scale"},{"type":"ScalingActive","status":"False","lastTransitionTime":"2022-02-16T07:37:10Z","reason":"FailedGetResourceMetric","message":"the
HPA was unable to compute the replica count: missing request for cpu"}]'
creationTimestamp: "2022-02-16T07:36:54Z"
name: nginx-test
namespace: kube-system
resourceVersion: "869738033"
selfLink: /apis/autoscaling/v1/namespaces/kube-system/horizontalpodautoscalers/nginx-test
uid: 7c5cb797-8a7d-45ee-bede-cb8be7a59926
spec:
maxReplicas: 10
minReplicas: 1
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: nginx-test
targetCPUUtilizationPercentage: 10
status:
currentReplicas: 1
desiredReplicas: 0
|
然后我们可以根据上面的 YAML 文件就可以自己来创建一个基于 YAML 的 HPA 描述文件了。但是我们发现上面信息里面出现了一些 Fail 信息，我们来查看下这个 HPA 对象的信息：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
Name: nginx-test
Namespace: kube-system
Labels: <none>
Annotations: <none>
CreationTimestamp: Wed, 16 Feb 2022 15:36:54 +0800
Reference: Deployment/nginx-test
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 10%
Min replicas: 1
Max replicas: 10
Deployment pods: 1 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: missing request for cpu
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 12s (x12 over 3m) horizontal-pod-autoscaler missing request for cpu
Warning FailedComputeMetricsReplicas 12s (x12 over 3m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu
|
我们可以看到上面的事件信息里面出现了 failed to get cpu utilization: missing request for cpu 这样的错误信息。这是因为我们上面创建的 Pod 对象没有添加 request 资源声明，这样导致 HPA 读取不到 CPU 指标信息，所以如果要想让 HPA 生效，对应的 Pod 资源必须添加 requests 资源声明，更新我们的资源清单文件：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx-test
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
labels:
app: nginx
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: NotIn
values:
- cn-beijing.10.101.16.235
- cn-beijing.10.101.16.236
containers:
- image: nginx:1.18.0-alpine
imagePullPolicy: IfNotPresent
name: nginx
livenessProbe:
httpGet:
path: /
port: web
initialDelaySeconds: 30
periodSeconds: 5
readinessProbe:
httpGet:
path: /
port: web
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 2
lifecycle:
preStop:
exec:
command: ["/bin/bash","-c","sleep 120"]
ports:
- containerPort: 80
name: web
protocol: TCP
resources:
requests:
memory: 50Mi
cpu: 50m
|
然后重新更新 Deployment，重新创建 HPA 对象：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
kubectl apply -f hpa-nginx.yaml
deployment.apps/nginx-test configured
kubectl get pods -o wide -l app=nginx -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-test-8496b9bbf6-vqntz 1/1 Running 0 48s 10.101.5.27 cn-beijing.10.101.16.233 <none> <none>
kubectl delete hpa nginx-test -n kube-system
kubectl autoscale deployment nginx-test --cpu-percent=10 --min=1 --max=10 -n kube-system
horizontalpodautoscaler.autoscaling/nginx-test autoscaled
kubectl describe hpa nginx-test -n kube-system
Name: nginx-test
Namespace: kube-system
Labels: <none>
Annotations: <none>
CreationTimestamp: Wed, 16 Feb 2022 15:47:14 +0800
Reference: Deployment/nginx-test
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 2% (1m) / 10%
Min replicas: 1
Max replicas: 10
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events: <none>
|
现在可以看到 HPA 资源对象已经正常了，现在我们来增大负载进行测试，我们来创建一个 busybox 的 Pod，并且循环访问上面创建的 Pod：
1
|
kubectl run -it --image busybox test-hpa --restart=Never --rm /bin/sh
|
下图可以看出，HPA已经开始工作:
1
2
3
4
5
6
7
8
9
10
11
12
|
kubectl get pods -l app=nginx --watch -n kube-system
NAME READY STATUS RESTARTS AGE
nginx-test-8496b9bbf6-9w8jh 1/1 Running 0 46s
nginx-test-8496b9bbf6-gh644 1/1 Running 0 46s
nginx-test-8496b9bbf6-gq95r 1/1 Running 0 31s
nginx-test-8496b9bbf6-hn9qb 1/1 Running 0 31s
nginx-test-8496b9bbf6-jgzhx 1/1 Running 0 46s
nginx-test-8496b9bbf6-kvrkg 1/1 Running 0 31s
nginx-test-8496b9bbf6-rrbbm 1/1 Running 0 31s
nginx-test-8496b9bbf6-vphl7 0/1 Running 0 15s
nginx-test-8496b9bbf6-vqntz 1/1 Running 0 28m
nginx-test-8496b9bbf6-vphl7 1/1 Running 0 17s
|
我们可以看到已经自动拉起了很多新的 Pod，最后定格在了我们上面设置的 10 个 Pod，同时查看资源 hpa-demo 的副本数量，副本数量已经从原来的1变成了10个：
1
2
3
|
kubectl get deploy nginx-test -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-test 10/10 10 10 44m
|
查看 HPA 资源的对象了解工作过程:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
Name: nginx-test
Namespace: kube-system
Labels: <none>
Annotations: <none>
CreationTimestamp: Wed, 16 Feb 2022 15:47:14 +0800
Reference: Deployment/nginx-test
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 2% (1m) / 10%
Min replicas: 1
Max replicas: 10
Deployment pods: 10 current / 10 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooManyReplicas the desired replica count is more than the maximum replica count
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 28m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 22m horizontal-pod-autoscaler New size: 1; reason: All metrics below target
Normal SuccessfulRescale 6m21s horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 6m6s horizontal-pod-autoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 5m50s horizontal-pod-autoscaler New size: 9; reason:
Normal SuccessfulRescale 5m20s horizontal-pod-autoscaler New size: 10; reason: cpu resource utilization (percentage of request) above target
|
同样的这个时候我们来关掉 busybox 来减少负载，然后等待一段时间观察下 HPA 和 Deployment 对象：
1
2
3
4
5
6
|
kubectl get hpa -n kube-system
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-test Deployment/nginx-test 2%/10% 1 10 1 43m
kubectl get deploy nginx-test -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-test 1/1 1 1 57m
|
可以看到副本数量已经由 10 变为 1，当前我们只是演示了 CPU 使用率这一个指标，在后面的课程中我们还会学习到根据自定义的监控指标来自动对 Pod 进行扩缩容。
基于内存
HorizontalPodAutoscaler 是 Kubernetes autoscaling API 组的资源，在当前稳定版本 autoscaling/v1 中只支持基于 CPU 指标的缩放。在 Beta 版本 autoscaling/v2beta2，引入了基于内存和自定义指标的缩放。所以我们这里需要使用 Beta 版本的 API
现在我们用 Deployment 来创建一个 Nginx Pod，然后利用 HPA 来进行自动扩缩容。资源清单如下所示：（hpa-mem.yaml）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
|
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx-test
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
labels:
app: nginx
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: NotIn
values:
- cn-beijing.10.101.16.235
- cn-beijing.10.101.16.236
volumes:
- name: increase-mem-script
configMap:
name: increase-mem-config
containers:
- image: nginx:1.18.0-alpine
imagePullPolicy: IfNotPresent
name: nginx
livenessProbe:
httpGet:
path: /
port: web
initialDelaySeconds: 30
periodSeconds: 5
readinessProbe:
httpGet:
path: /
port: web
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 2
lifecycle:
preStop:
exec:
command: ["/bin/bash","-c","sleep 120"]
ports:
- containerPort: 80
name: web
protocol: TCP
volumeMounts:
- name: increase-mem-script
mountPath: /etc/script
resources:
requests:
memory: 50Mi
cpu: 50m
securityContext:
privileged: true
|
这里和前面普通的应用有一些区别，我们将一个名为 increase-mem-config 的 ConfigMap 资源对象挂载到了容器中，该配置文件是用于后面增加容器内存占用的脚本，配置文件如下所示：（increase-mem-cm.yaml）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
apiVersion: v1
kind: ConfigMap
metadata:
name: increase-mem-config
namespace: kube-system
data:
increase-mem.sh: |
#!/bin/bash
mkdir /tmp/memory
mount -t tmpfs -o size=40M tmpfs /tmp/memory
dd if=/dev/zero of=/tmp/memory/block
sleep 60
rm /tmp/memory/block
umount /tmp/memory
rmdir /tmp/memory
|
由于这里增加内存的脚本需要使用到 mount 命令，这需要声明为特权模式，所以我们添加了 securityContext.privileged=true 这个配置。现在我们直接创建上面的资源对象即可：
1
2
3
4
5
|
kubectl apply -f increase-mem-cm.yaml
kubectl apply -f hpa-mem.yaml
kubectl get pods -l app=nginx -n kube-system
NAME READY STATUS RESTARTS AGE
nginx-test-f45596f8b-pbx5c 1/1 Running 0 2m48s
|
然后需要创建一个基于内存的 HPA 资源对象：（hpa-mem.yaml）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: nginx-test
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: memory
targetAverageUtilization: 60
|
要注意这里使用的 apiVersion 是 autoscaling/v2beta1，然后 metrics 属性里面指定的是内存的配置，直接创建上面的资源对象即可
1
2
3
4
5
6
|
kubectl apply -f hpa-mem.yaml -n kube-system
horizontalpodautoscaler.autoscaling/nginx-hpa created
kubectl get hpa -n kube-system
kubectl get hpa -n kube-system
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx-test <unknown>/60% 1 5 0 15s
|
到这里证明 HPA 资源对象已经部署成功了，接下来我们对应用进行压测，将内存压上去，直接执行上面我们挂载到容器中的 increase-mem.sh 脚本即可：
1
2
3
4
5
|
kexec -it nginx-test-f45596f8b-pbx5c /bin/sh -n kube-system
/ # source /etc/script/increase-mem.sh
dd: error writing '/tmp/memory/block': No space left on device
81921+0 records in
81920+0 records out
|
然后打开另外一个终端观察 HPA 资源对象的变化情况：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
kubectl get hpa -n kube-system
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx-test 86%/60% 1 5 2 3m27s
kubectl describe hpa nginx-hpa -n kube-system
Name: nginx-hpa
Namespace: kube-system
Labels: <none>
Annotations: CreationTimestamp: Wed, 16 Feb 2022 17:00:34 +0800
Reference: Deployment/nginx-test
Metrics: ( current / target )
resource memory on pods (as a percentage of request): 6% (3153920) / 60%
Min replicas: 1
Max replicas: 5
Deployment pods: 2 current / 2 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 115s horizontal-pod-autoscaler New size: 2; reason: memory resource utilization (percentage of request) above target
kubectl top pod nginx-test-f45596f8b-pbx5c -n kube-system
NAME CPU(cores) MEMORY(bytes)
nginx-test-f45596f8b-pbx5c 1m 3Mi
|
可以看到内存使用已经超过了我们设定的 60% 这个阈值了，HPA 资源对象也已经触发了自动扩容，变成了两个副本了：
1
2
3
4
|
kubectl get pods -l app=nginx -n kube-system
NAME READY STATUS RESTARTS AGE
nginx-test-f45596f8b-pbx5c 1/1 Running 0 13m
nginx-test-f45596f8b-qdsxr 1/1 Running 0 4m42s
|
当内存释放掉后，controller-manager 默认5分钟过后会进行缩放，到这里就完成了基于内存的 HPA 操作。
