kubernetes（十二） 部署 harbor 私有仓库

安装下载依赖包

  1. 安装docker-compose

    从 docker compose 发布页面下载最新的 docker-compose 二进制文件,本文以1.25.4为例

    cd /opt/k8s/work/harbor/

wget https://github.com/docker/compose/releases/download/1.25.4/docker-compose-Linux-x86_64

mv docker-compose-Linux-x86_64 /opt/k8s/bin/docker-compose
chmod a+x  /opt/k8s/bin/docker-compose
export PATH=/opt/k8s/bin:$PATH

  2. 下载harbor二进制文件

    从 harbor 发布页面下载最新的 harbor 离线安装包，本文以v1.9.4为例

    cd /opt/k8s/work/harbor/

wget https://github.com/goharbor/harbor/releases/download/v1.9.4/harbor-offline-installer-v1.9.4.tgz

tar -xzvf harbor-offline-installer-v1.5.1.tgz

  3. 导入 docker images
    导入离线安装包中 harbor 相关的 docker images

    cd /opt/k8s/work/harbor/harbor
docker load -i harbor.v1.9.4.tar.gz

以http形式启动

  1. 修改harbor.yml文件
    修改 hostname、data_volume属性值

    cd /opt/k8s/work/harbor/harbor

cp harbor.yml harbor.yml.bak

vim harbor.yml

    修改信息如下

    diff harbor.yml harbor.yml.bak
5c5
< hostname: 192.168.0.107
---
> hostname: reg.mydomain.com
40c40
< data_volume: /data/k8s/harbor/data
---
> data_volume: /data

  2. 加载和启动 harbor 镜像

    cd /opt/k8s/work/harbor/harbor
mkdir -p /data/k8s/harbor/data
chmod 777 /var/run/docker.sock /data/k8s/harbor/data
./install.sh

    启动日志

    
Note: docker version: 18.09.6

Note: docker-compose version: 1.25.4
# Configuration file of Harbor

[Step 1]: loading Harbor images ...
Loaded image: goharbor/harbor-core:v1.9.4
Loaded image: goharbor/clair-photon:v2.1.0-v1.9.4
Loaded image: goharbor/harbor-portal:v1.9.4
Loaded image: goharbor/nginx-photon:v1.9.4
Loaded image: goharbor/chartmuseum-photon:v0.9.0-v1.9.4
Loaded image: goharbor/prepare:v1.9.4
Loaded image: goharbor/redis-photon:v1.9.4
Loaded image: goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.4
Loaded image: goharbor/notary-server-photon:v0.6.1-v1.9.4
Loaded image: goharbor/harbor-log:v1.9.4
Loaded image: goharbor/harbor-db:v1.9.4
Loaded image: goharbor/harbor-jobservice:v1.9.4
Loaded image: goharbor/harbor-registryctl:v1.9.4
Loaded image: goharbor/notary-signer-photon:v0.6.1-v1.9.4
Loaded image: goharbor/harbor-migrator:v1.9.4


[Step 2]: preparing environment ...
prepare base dir is set to /opt/k8s/work/harbor/harbor
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /secret/keys/secretkey
Generated certificate, key file: /secret/core/private_key.pem, cert file: /secret/registry/root.crt
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir



[Step 3]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-db     ... done
Creating redis         ... done
Creating harbor-portal ... done
Creating registry      ... done
Creating registryctl   ... done
Creating harbor-core   ... done
Creating harbor-jobservice ... done
Creating nginx             ... done

✔ ----Harbor has been installed and started successfully.----

Now you should be able to visit the admin portal at http://192.168.0.107.
For more details, please visit https://github.com/goharbor/harbor .

    查看启动状态

    root@master:/opt/k8s/work/harbor/harbor# docker-compose ps
  	Name                     Command                  State                 Ports
---------------------------------------------------------------------------------------------
harbor-core         /harbor/harbor_core              Up (healthy)
harbor-db           /docker-entrypoint.sh            Up (healthy)   5432/tcp
harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp
redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp
registry            /entrypoint.sh /etc/regist ...   Up (healthy)   5000/tcp
registryctl         /harbor/start.sh                 Up (healthy)

    • 如果有不是healthy状态的对象，到/var/log/harbor/目录下查看对应对象的日志

    • harbor官方提供的启动文件中容器的日志类型是syslog，不支持用docker logs查看

      
ls /var/log/harbor/

core.log jobservice.log portal.log postgresql.log proxy.log redis.log registryctl.log registry.log

	```

浏览器访问 http://192.168.0.107,用账号 admin 和 harbor.yml 配置文件中的默认密码 Harbor12345 登陆系统。
![](https://img2018.cnblogs.com/blog/1918821/202002/1918821-20200214210112961-1493493575.png)

  1. 创建一个新的项目

  2. docker命令拉取和上传镜像

    1. 执行login

      root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://192.168.0.107/v2/: dial tcp 192.168.0.107:443: connect: connection refused

    2. 因为docker命令默认采用https和API交互，而我们的harbor是http的，所以不能执行，需要在/etc/docker/daemon.json中追加insecure-registries配置

      {
    "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"],
    "insecure-registries": ["192.168.0.107"],
    "max-concurrent-downloads": 20,
    "live-restore": true,
    "max-concurrent-uploads": 10,
    "data-root": "/data/k8s/docker/data",
    "log-opts": {
      "max-size": "100m",
      "max-file": "5"
    }
}

    3. 重新启动docker服务

      root@slave:~# systemctl restart docker

root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

    4. 向harbor中刚创建的私有项目里面上传镜像

      root@slave:~# docker tag k8s.gcr.io/pause:3.1 192.168.0.107/k8s/pause:3.1
root@slave:~# docker push 192.168.0.107/k8s/pause:3.1
The push refers to repository [192.168.0.107/k8s/pause]
e17133b79956: Pushed
3.1: digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d size: 527

    5. 浏览器查看

    6. 从harbor中下载镜像

      root@slave:~# docker rmi  192.168.0.107/k8s/pause:3.1
Untagged: 192.168.0.107/k8s/pause:3.1
Untagged: 192.168.0.107/k8s/pause@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
root@slave:~# docker pull  192.168.0.107/k8s/pause:3.1
3.1: Pulling from k8s/pause
Digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
Status: Downloaded newer image for 192.168.0.107/k8s/pause:3.1

以https形式启动

下列操作的工作目录/opt/k8s/work/harbor/harbor是解压harbor离线安装文件后 生成的 harbor 目录

  1. 停止harbor（如果没有启动过，则跳过该步骤）

    cd /opt/k8s/work/harbor/harbor

root@master:/opt/k8s/work/harbor/harbor# COMPOSE_HTTP_TIMEOUT=200 docker-compose down -v
Stopping nginx             ... done
Stopping harbor-jobservice ... done
Stopping harbor-core       ... done
Stopping registryctl       ... done
Stopping redis             ... done
Stopping harbor-portal     ... done
Stopping harbor-db         ... done
Stopping registry          ... done
Stopping harbor-log        ... done
Removing harbor-log        ... done
Removing network harbor_harbor

    • 执行docker-compose down -v后，有可能对应的容器还在，可以执行强制停掉容器

      docker ps | grep harbor | awk '{print $1}' | xargs -I {} docker rm -f {}

  2. 创建 harbor nginx 服务器使用的 x509 证书，本文档采用cfssl工具生成，也可以利用openssl生成，具体步骤参考harbor-openssl

    1. 创建harbor用的CA根证书

      1. 创建配置文件

        cd /opt/k8s/work/harbor/
cat > harbor-ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "harbor": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF

      2. 创建证书签名请求文件

        cd /opt/k8s/work/harbor/

cat > harbor-ca-csr.json <<EOF
{
  "CN": "harbor-ca",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "NanJing",
      "L": "NanJing",
      "O": "k8s",
      "OU": "system"
    }
  ],
  "ca": {
    "expiry": "87600h"
 }
}
EOF

      3. 生成自签名的根证书,分发证书

        cd /opt/k8s/work/harbor/

cfssl gencert -initca harbor-ca-csr.json | cfssljson -bare harbor-ca

ls harbor-ca*pem

mkdir -p /etc/harbor/cert

mv harbor-ca*pem harbor-ca-config.json  /etc/harbor/cert

    2. 创建harbor用的证书

      1. 生成证书请求文件

        cd /opt/k8s/work/harbor/

cat > harbor-server-csr.json <<EOF
{
  "CN": "harbor",
  "hosts": [
    "127.0.0.1",
    "192.168.0.107"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "NanJing",
      "L": "NanJing",
      "O": "k8s",
      "OU": "system"
    }
  ]
}
EOF
        • hosts 字段指定授权使用该证书的当前部署节点 IP

      2. 生成证书

        
cfssl gencert -ca=/etc/harbor/cert/harbor-ca.pem \

-ca-key=/etc/harbor/cert/harbor-ca-key.pem
-config=/etc/harbor/cert/harbor-ca-config.json
-profile=harbor harbor-server-csr.json | cfssljson -bare harbor-server

		ls harbor-server*pem
		
		mv harbor-server*pem /etc/harbor/cert
		
		```

  1. 编辑配置文件

    修改 hostname、data_volume属性值,去除默认的http方式，追加https的配置

    cd /opt/k8s/work/harbor/harbor

vim harbor.yml

    修改内容如下

    diff harbor.yml harbor.yml.bak
5c5
< hostname: 192.168.0.107
---
> hostname: reg.mydomain.com
8c8
< #http:
---
> http:
10c10
<   #  port: 80
---
>   port: 80
13c13
< https:
---
> # https:
15c15
<    port: 443
---
> #   port: 443
17,18c17,18
<    certificate: /etc/harbor/cert/harbor-server.pem
<    private_key: /etc/harbor/cert/harbor-server-key.pem
---
> #   certificate: /your/certificate/path
> #   private_key: /your/private/key/path
40c40
< data_volume: /data/k8s/harbor/data
---
> data_volume: /data

  2. 生成配置文件（如果没有启动过，则跳过该步骤）

    cd /opt/k8s/work/harbor/harbor

./prepare

    执行日志

    prepare base dir is set to /opt/k8s/work/harbor/harbor
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registry/root.crt
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/core/env
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

    修改生成文件的访问权限

    cd /opt/k8s/work/harbor/harbor
chmod -R 777 common

  3. 启动harbor

    如果是首次启动 执行

    cd /opt/k8s/work/harbor/harbor

mkdir -p /data/k8s/harbor/data
chmod 777 /var/run/docker.sock /data/k8s/harbor/data
./install.sh

    否则执行

    docker-compose up -d

  4. 查看启动状态

    root@master:/opt/k8s/work/harbor/harbor# docker-compose ps
  Name                     Command                  State                          Ports
---------------------------------------------------------------------------------------------------------------
harbor-core         /harbor/harbor_core              Up (healthy)
harbor-db           /docker-entrypoint.sh            Up (healthy)   5432/tcp
harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp
redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp
registry            /entrypoint.sh /etc/regist ...   Up (healthy)   5000/tcp
registryctl         /harbor/start.sh                 Up (healthy)
    • nginx 暴露了两个端口http的80和https的443，使用http访问80会自动重定向到https

  5. 浏览器访问 https://192.168.0.107,用账号 admin 和 harbor.yml 配置文件中的默认密码 Harbor12345 登陆系统。

    • 其中的k8s是在http模式下创建的项目，直接启动https的可参照上面创建

  6. docker命令拉取和上传镜像

    如果之前已经执行过docker login，配置修改后需要重新docker login，需要删除 /root/.docker/config.json文件

    如果在/etc/docker/daemon.json中配置过insecure-registries，需要去掉,并重新启动docker服务

    1. 执行login

      root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107

WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://192.168.0.107/v2/: x509: certificate signed by unknown authority

	```
	错误是说我们用的证书是自签名的证书，签发证书机构未经认证
	解决方法是将签署 harbor-server 证书的 CA 证书拷贝到 /etc/docker/certs.d/192.168.0.107 目录下（需要在所有要访问harbor的节点上都执行此操作）
	
	```
	root@slave:~# mkdir -p /etc/docker/certs.d/192.168.0.107
	root@slave:~# scp root@192.168.0.107:/etc/harbor/cert/harbor-ca.pem /etc/docker/certs.d/192.168.0.107/ca.crt
	root@192.168.0.107's password:
	harbor-ca.pem                        100% 1306   283.7KB/s   00:00
	root@slave:~# ls /etc/docker/certs.d/192.168.0.107/
	ca.crt
	
	```
	
	重新执行login
	
	```
	root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
	WARNING! Using --password via the CLI is insecure. Use --password-stdin.
	WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
	Configure a credential helper to remove this warning. See
	https://docs.docker.com/engine/reference/commandline/login/#credentials-store
	
	Login Succeeded
	
	```

  1. 上传镜像

    
root@slave:~# docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
kubernetesui/metrics-scraper   v1.0.3              3327f0dbcb4a        2 weeks ago         40.1MB
busybox                        latest              6d5fcfe5ff17        7 weeks ago         1.22MB
coredns/coredns                1.6.6               cc4d8e8c6169        2 months ago        40.8MB
k8s.gcr.io/pause               3.1                 da86e6ba6ca1        2 years ago         742kB
192.168.0.107/k8s/pause        3.1                 da86e6ba6ca1        2 years ago         742kB
kubeimage/pause                3.1                 da86e6ba6ca1        2 years ago         742kB
nginx                          1.9.1               94ec7e53edfc        4 years ago         133MB
root@slave:~# docker tag busybox:latest 192.168.0.107/k8s/busybox:latest
root@slave:~# docker push 192.168.0.107/k8s/busybox:latest
The push refers to repository [192.168.0.107/k8s/busybox]
195be5f8be1d: Pushed
latest: digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6 size: 527

  2. 通过浏览器查看刚上传的镜像

  3. 从harbor下载镜像

    root@slave:~# docker rmi 192.168.0.107/k8s/busybox:latest
Untagged: 192.168.0.107/k8s/busybox:latest
Untagged: 192.168.0.107/k8s/busybox@sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6
root@slave:~# docker pull 192.168.0.107/k8s/busybox:latest
latest: Pulling from k8s/busybox
Digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6
Status: Downloaded newer image for 192.168.0.107/k8s/busybox:latest

kubernetes集群中使用私有的harbor镜像仓库

  1. kubelet配置

    kubernetes以pod为管理单元，而不是docker容器，在创建pod时，会先启动一个名为k8s.gcr.io/pause:3.1的镜像。

    该镜像存在于google的镜像仓库中，国内不能直接访问，需要FQ下载，如果集群新加节点，或者节点上这个镜像被清理掉，将导致pod不能启动

    一般做法是搭建私有镜像仓库，把这个镜像push到我们的私有镜像仓库，再配置kubelet使用我们私有仓库中的pause镜像

    具体做法是修改kubelet服务的启动文件:/etc/systemd/system/kubelet.service,在其中追加--pod-infra-container-image属性，属性值指向自己私有仓库中的pause镜像

    例如

    --pod-infra-container-image=192.168.0.107/k8s/pause:3.1

    重启kubelet服务

    systemctl restart kubelet

  2. kubernetes和harbor进行认证

    kubernetes提供了各种方式来访问各种镜像仓库，具体参考Using a Private Registry,本文采用配置节点的方式来使用私有仓库

    1. 在集群的任意一个节点上登陆harbor

      root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

      成功后会生成认证文件/root/.docker/config.json

      
root@slave:~# ls /root/.docker/
config.json
root@slave:~# cat /root/.docker/config.json
{
        "auths": {
                "192.168.0.107": {
                        "auth": "YWRtaW46SGFyYm9yMTIzNDU="
                }
        },
        "HttpHeaders": {
                "User-Agent": "Docker-Client/18.09.6 (linux)"
        }
}

    2. 配置kubelet使用此认证文件

      默认情况下，kubelet会从如下路径中查找私有仓库的认证文件

      • {--root-dir:-/var/lib/kubelet}/config.json
      • {cwd of kubelet}/config.json
      • ${HOME}/.docker/config.json #须在环境变量文件中为 kubelet 显式设置 HOME=/root
      • /.docker/config.json

      此处我们将私有仓库的认证文件copy到kubelet的root目录下，此目录在kubelet.service中进行设置
      例如

      --root-dir=/data/k8s/k8s/kubelet

    3. copy私有仓库的认证文件到kubelet的root目录（所有运行kubelet，需要访问harbor的节点都需要），本环境中有两个节点

      cp /root/.docker/config.json /data/k8s/k8s/kubelet/config.json

scp /root/.docker/config.json root@192.168.0.114:/data/k8s/k8s/kubelet/config.json

  3. 验证

    创建一个pod,image 指定私有仓库里面的镜像

    cd /opt/k8s/yml

cat > private-busybox.yml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: private-busybox
spec:
  containers:
  - name: private-busybox
    image: 192.168.0.107/k8s/busybox:latest
    command:
      - sleep
      - "3600"
EOF

    创建pod

    root@master:/opt/k8s/yml# kubectl create -f private-busybox.yml

    查看运行状态

    root@master:/opt/k8s/yml# kubectl get pod | grep private-busybox
private-busybox                     1/1     Running   0          108s

    通过docker命令，查看启动pod对应的容器的镜像名称

    root@master:/opt/k8s/yml# docker ps | grep -e IMAGE -e private-busybox
CONTAINER ID        IMAGE                                                    COMMAND                  CREATED              STATUS                  PORTS                                                           NAMES
c2b43d9d9b3e        192.168.0.107/k8s/busybox                                "sleep 3600"             About a minute ago   Up About a minute                                                                       k8s_private-busybox_private-busybox_default_98a4d4bd-fd46-4d9d-8ef9-95845ce7e53d_0
a4a84461d5fe        192.168.0.107/k8s/pause:3.1                              "/pause"                 About a minute ago   Up About a minute                                                                       k8s_POD_private-busybox_default_98a4d4bd-fd46-4d9d-8ef9-95845ce7e53d_0                   1/1     Running   0          5s

    可以看到启动的容器对应的镜像都是我们自己的私有仓库中的镜像

posted @ 2020-02-14 21:20  高枫_henu  Views(510)  Comments(0)  Edit  收藏  举报