kubekey3.1.9离线安装k8sv1.33.0
目前kubekey最新版本为4.x，根据官网社区版步骤生成离线安装包，安装harbor时报错：
[harbor]: failed to push image: failed to push image “quay.io/tigera/operator:v1.34.5” to remote: POST “https://dockerhub.kubekey.local/v2/tigera/operator/blobs/uploads/”: response status code 401: unauthorized: unauthorized to access repository: tigera/operator, action: push: unauthorized to access repository: tigera/operator, action: push 即使手动安装harbor然后执行镜像推送步骤也报一样的错，感觉是kubekey的版本问题社区链接
暂时未解决，官方指导很多镜像都需要FQ访问，我用花了很长时间，做出来的离线包还有问题，所以不推荐用4.x版本
下面记录下离线安装经验（原文参考原文）：
环境配置就不赘述了，参照原文就行了（主机名那一步不用操作，修改config-sample.yaml时需要填节点name，kk会自动修改设备的hostname）
1、下载kubekey：https://github.com/kubesphere/kubekey/releases/download/v3.1.9/kubekey-v3.1.9-linux-amd64.tar.gz 解压就行了
2、创建manifest.yaml 创建离线安装包
./kk create manifest --with-kubernetes v1.33.0 --with-registry
cat manifest.yaml
apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
name: sample
spec:
arches:
- amd64
operatingSystems: []
kubernetesDistributions:
- type: kubernetes
version: v1.33.0
components:
helm:
version: v3.14.3
cni:
version: v1.2.0
etcd:
version: v3.5.13
containerRuntimes:
- type: docker
version: 24.0.9
- type: containerd
version: 1.7.13
calicoctl:
version: v3.27.4
crictl:
version: v1.29.0
docker-registry:
version: "2"
harbor:
version: v2.10.1
docker-compose:
version: v2.26.1
images:
- registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.9
- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.33.0
- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.33.0
- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.33.0
- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.33.0
- registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.9.3
- registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20
- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.4
- registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4
- registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4
- registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.27.4
- registry.cn-beijing.aliyuncs.com/kubesphereio/typha:v3.27.4
- registry.cn-beijing.aliyuncs.com/kubesphereio/flannel:v0.21.3
- registry.cn-beijing.aliyuncs.com/kubesphereio/flannel-cni-plugin:v1.1.2
- registry.cn-beijing.aliyuncs.com/kubesphereio/cilium:v1.15.3
- registry.cn-beijing.aliyuncs.com/kubesphereio/operator-generic:v1.15.3
- registry.cn-beijing.aliyuncs.com/kubesphereio/hybridnet:v0.8.6
- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-ovn:v1.10.10
- registry.cn-beijing.aliyuncs.com/kubesphereio/multus-cni:v3.8
- registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:3.3.0
- registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:3.3.0
- registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.9.6-alpine
- registry.cn-beijing.aliyuncs.com/kubesphereio/kube-vip:v0.7.2
- registry.cn-beijing.aliyuncs.com/kubesphereio/kata-deploy:stable
- registry.cn-beijing.aliyuncs.com/kubesphereio/node-feature-discovery:v0.10.0
## ks-core
- docker.m.ixdev.cn/kubesphere/ks-apiserver:v4.1.3
- swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubesphere/ks-console:v4.1.3
#- docker.m.ixdev.cn/kubesphere/ks-console:v4.1.3
- docker.m.ixdev.cn/kubesphere/ks-controller-manager:v4.1.3
- docker.m.ixdev.cn/kubesphere/kubectl:v1.27.16
- docker.m.ixdev.cn/kubesphere/redis:7.2.4-alpine
- docker.m.ixdev.cn/kubesphere/haproxy:2.9.6-alpine
- swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/openksc/ks-extensions-museum:v1.1.6
#- registry.cn-beijing.aliyuncs.com/kubesphere/ks-extensions-museum:v1.1.6
## metrics-server
- docker.m.ixdev.cn/kubesphere/metrics-server:v0.7.0
- docker.m.ixdev.cn/kubesphere/addon-resizer:1.8.20
registry:
auths: {}
我这里要安装kubesphere，不安装的话有的镜像不需要
然后执行以下命令创建离线安装包：
export KKZONE=cn
./kk artifact export -m manifest-sample.yaml -o kubesphere.tar.gz
#执行时可能会报错有的镜像需要登录才能下载，有的镜像找不到docker.m.ixdev.cn和swr.cn-north-4.myhuaweicloud.com这些域名的镜像都是我自己在网上找的
#如果有的镜像docker login后能手动pull，可以修改最后的auths，如
auths:
"docker.io":
username: ruizhouh #如果username为纯数字需要加上引号
passward: ********
#有的镜像如swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubesphere/ks-console:v4.1.3项目是ddn-k8s，后面push镜像的步骤会报错找不到项目，在创建harbor项目时将该项目添加进脚本即可，其他镜像同理
3、创建config-cample.yml 安装harbor
./kk create config --with-kubernetes v1.33.0 -f config-sample.yaml
cat config-sample.yaml
kind: Cluster
metadata:
name: sample
spec:
hosts:
- {name: node1, address: 192.168.10.11, internalAddress: 192.168.10.11, user: app, password: "*******"} #用户提前创建
- {name: node2, address: 192.168.10.12, internalAddress: 192.168.10.12, user: app, password: "*******"}
- {name: node3, address: 192.168.10.13, internalAddress: 192.168.10.13, user: app, password: "*******"}
roleGroups:
etcd:
- node1
control-plane:
- node1
worker:
- node1
- node2
- node3
registry:
- node3
controlPlaneEndpoint:
## Internal loadbalancer for apiservers
# internalLoadbalancer: haproxy
domain: lb.kubesphere.local
address: ""
port: 6443
kubernetes:
version: v1.33.0
clusterName: cluster.local
autoRenewCerts: true
containerManager: docker
#etcd:
# type: kubekey
network:
plugin: calico
kubePodsCIDR: 10.233.64.0/18
kubeServiceCIDR: 10.233.0.0/18
## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
multusCNI:
enabled: false
registry:
type: "harbor"
auths:
"my.harbor.com": #私有harbor域名，可在/etc/hosts中映射
username: admin
password: Harbor12345
privateRegistry: "my.harbor.com"
namespaceOverride: "kubesphereio"
skipTLSVerify: true
plainHTTP: true
registryMirrors: []
insecureRegistries: ["0.0.0.0/0","::/0"]
addons: []
./kk init registry -f config-sample.yaml -a kubesphere.tar.gz安装harbor
4、创建harbor项目
vim create_project_harbor.sh
#!/usr/bin/env bash
url="https://dockerhub.kubekey.local" # 修改为真实镜像仓库地址
user="admin"
passwd="******" ## 修改为真实的密码
harbor_projects=(
ks
kubesphere
kubesphereio
coredns
calico
flannel
cilium
hybridnetdev
kubeovn
openebs
library
plndr
jenkins
argoproj
dexidp
openpolicyagent
curlimages
grafana
kubeedge
nginxinc
prom
kiwigrid
minio
opensearchproject
istio
jaegertracing
timberio
prometheus-operator
jimmidyson
elastic
thanosio
brancz
prometheus
ddn-k8s #上面镜像列表中存在的项目在这里添加
)
for project in "${harbor_projects[@]}"; do
echo "creating $project"
curl -u "${user}:${passwd}" -X POST -H "Content-Type: application/json" "${url}/api/v2.0/projects" -d "{ \"project_name\": \"${project}\", \"public\": true}" -k # 注意在 curl 命令末尾加上 -k
done
chmod +x create_project_harbor.sh
./create_project_harbor.sh
./kk artifact image push -f config-sample.yaml -a kubesphere.tar.gz #推送镜像到harbor，可能报错找不到项目，在上面脚本中添加即可
5、安装k8s
./kk create cluster -f config-sample.yaml -a kubesphere.tar.gz --with-local-storage --skip-push-images
查看更多可用参数，可以执行以下命令：
./kk create cluster -f config-sample.yaml -a kubesphere.tar.gz --help
参数说明：
- --skip-push-images ：跳过预推送镜像
- --with-packages ：安装操作系统依赖（需要 ISO 文件,前面有安装socat这些就不用加这个参数，如果有OS离线补丁包可以挂载后自定义yum或apt本地源，然后加上这个参数自动安装依赖）
- --with-local-storage ：部署本地存储（Local PV Provisioner）
#安装报错Error: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 192.168.10.11:2379: connect: connection refused，
#执行journalctl -xeu etcd.service有报错node1 etcd[32536]: {"level":"fatal","ts":"2026-01-19T17:48:51.28967+0800","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:267"}
#需要修改/etc/etcd.env中的ETCD_INITIAL_CLUSTER_STATE=new 然后重新执行安装
到此k8s部署完成
浙公网安备 33010602011771号