Harbor镜像仓库（含clair镜像扫描） - 完整部署记录

Harbor镜像仓库（含clair镜像扫描） - 完整部署记录

 

 

 Harbor环境部署的要求：系统版本在Centos7.5以上、内核版本在4.4X以上、ip_forward路由转发功能要打开。

一、环境准备

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@k8s-harbor01 ~]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
  
[root@k8s-harbor01 ~]# uname -r
4.4.232-1.el7.elrepo.x86_64
  
[root@k8s-harbor01 ~]# echo 1 > /proc/sys/net/ipv4/ip_forward
[root@k8s-harbor01 ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
[root@k8s-harbor01 ~]# sysctl -p
  
[root@k8s-harbor01 ~]# systemctl stop firewalld && systemctl disable firewalld && firewall-cmd --state
  
[root@k8s-harbor01 ~]# vim /etc/sysconfig/selinux
SELINUX=disabled
[root@k8s-harbor01 ~]# getenforce
Disabled
  
[root@k8s-harbor01 ~]# python --version
Python 2.7.5

　　

二、安装Docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
提前下载二进制安装包docker-18.09.6.tgz到/usr/local/src路径下，解压安装
[root@k8s-harbor01 ~]# cd /usr/local/src/
[root@k8s-harbor01 src]# ll docker-18.09.6.tgz
-rw-r--r-- 1 root root 48047231 Oct 19  2019 docker-18.09.6.tgz
[root@k8s-harbor01 src]# tar -zvxf docker-18.09.6.tgz
  
[root@k8s-harbor01 src]# cp docker/* /usr/local/bin/
[root@k8s-harbor01 src]# chmod 755 /usr/local/bin/*
  
/usr/local/bin默认已经加到系统环境变量中
[root@k8s-harbor01 src]# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
  
编辑docker启动文件
注意"WorkingDirectory"路径要和/etc/docker/daemon.json文件中的data-root、exec-root路径一致
[root@k8s-harbor01 src]# cat > /etc/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
  
[Service]
WorkingDirectory=/data/docker
Environment="PATH=/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin"
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/local/bin/dockerd
ExecReload=/bin/kill -s HUP
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process
  
[Install]
WantedBy=multi-user.target
  
EOF
  
授执行权限
[root@k8s-harbor01 src]# chmod 755 /etc/systemd/system/docker.service
  
编辑docker 配置文件
编辑docker 配置文件
[root@k8s-harbor01 src]# mkdir -p /etc/docker && mkdir -p /data/docker/data && mkdir -p /data/docker/exec
[root@k8s-harbor01 src]# cat > /etc/docker/daemon.json << EOF
{
    "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"],
    "insecure-registries": ["docker02:35000"],
    "max-concurrent-downloads": 20,
    "live-restore": true,
    "max-concurrent-uploads": 10,
    "debug": true,
    "data-root": "/data/docker/data",
    "exec-root": "/data/docker/exec",
    "log-opts": {
      "max-size": "100m",
      "max-file": "5"
    }
}
  
EOF
  
启动 docker 服务
[root@k8s-harbor01 src]# systemctl daemon-reload && systemctl enable docker && systemctl restart docker
[root@k8s-harbor01 src]# systemctl status docker|grep Active
   Active: active (running) since Wed 2020-08-12 13:41:07 CST; 28s ago
  
查看 Docker 版本号
[root@k8s-harbor01 src]# docker --version
Docker version 18.09.6, build 481bc77

　　

三、安装Docker-Compose

1
2
3
4
5
6
7
8
9
10
11
下载docker-compose二进制执行文件
百度网盘下载地址：https://pan.baidu.com/s/1er0rM0vxEubYOLHx7LI62A
提取密码：eer9
[root@k8s-harbor01 ~]# cd /usr/local/src/
[root@k8s-harbor01 src]# curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
[root@k8s-harbor01 src]# cp docker-compose /usr/local/bin/
[root@k8s-harbor01 src]# chmod 755 /usr/local/bin/*
  
查看 docker-compose 版本号
[root@k8s-harbor01 ~]# docker-compose --version
docker-compose version 1.26.0, build d4451659

　　

四、部署Harbor镜像仓库

1）HTTPS证书自签
如果线上环境有已购买好的HTTPS证书可以直接拿过来用，如果没有，就在Harbor本机进行HTTPS证书自签。这里Harbor本机ip地址是172.16.60.238

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
生成CA证书私钥
[root@k8s-harbor01 ~]# openssl genrsa -out ca.key 4096
  
生成CA证书
[root@k8s-harbor01 ~]# openssl req -x509 -new -nodes -sha512 -days 3650 \
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.60.238" \
 -key ca.key \
 -out ca.crt
  
生成服务器证书
1）生成私钥
[root@k8s-harbor01 ~]# openssl genrsa -out 172.16.60.238.key 4096
  
2）生成证书签名请求（CSR）
[root@k8s-harbor01 ~]# openssl req -sha512 -new \
    -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.60.238" \
    -key 172.16.60.238.key \
    -out 172.16.60.238.csr
  
3）生成一个x509 v3扩展文件（两种方式根据情况二选一）
####################################################################################
第一种方式：域名
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
  
[alt_names]
DNS.1=172.16.60.238
DNS.2=yourdomain
DNS.3=hostname
EOF
####################################################################################
第二种方式：IP
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = IP:172.16.60.238
EOF
####################################################################################
  
这里选择第二种的IP方式
[root@k8s-harbor01 ~]# cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = IP:172.16.60.238
EOF
  
4）使用该v3.ext文件为您的Harbor主机生成证书
[root@k8s-harbor01 ~]# openssl x509 -req -sha512 -days 3650 \
    -extfile v3.ext \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -in 172.16.60.238.csr \
    -out 172.16.60.238.crt

　　

2）提供证书给Harbor和Docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
1）将服务器证书和密钥复制到Harbor主机上的certficates文件夹中
根据自己实际环境需求创建Harbor的certficates文件夹
[root@k8s-harbor01 ~]# mkdir -p /data/cert/
[root@k8s-harbor01 ~]# cp 172.16.60.238.crt /data/cert/
[root@k8s-harbor01 ~]# cp 172.16.60.238.key /data/cert/
  
2）转换 172.16.60.238.crt 为172.16.60.238.cert，供Docker使用。
Docker守护程序将.crt文件解释为CA证书，并将.cert文件解释为客户端证书。
[root@k8s-harbor01 ~]# openssl x509 -inform PEM -in 172.16.60.238.crt -out 172.16.60.238.cert
  
3）将服务器证书，密钥和CA文件复制到Harbor主机上的Docker certificate文件夹中。
记住必须首先创建适当的文件夹
[root@k8s-harbor01 ~]# mkdir -p /etc/docker/certs.d/172.16.60.238/
[root@k8s-harbor01 ~]# cp 172.16.60.238.cert /etc/docker/certs.d/172.16.60.238/
[root@k8s-harbor01 ~]# cp 172.16.60.238.key /etc/docker/certs.d/172.16.60.238/
[root@k8s-harbor01 ~]# cp ca.crt /etc/docker/certs.d/172.16.60.238/
  
4）重新启动Docker
[root@k8s-harbor01 ~]# systemctl restart docker
[root@k8s-harbor01 ~]# systemctl status docker
  
5）将名为"ca.crt"的CA证书下载到本地电脑，然后安装证书。
这样就可以在本地电脑的浏览器里正常访问https地址的Harbor了（证书可被信任）

　　

3）安装Harbor
到 Harbor的GitHub仓库的Release页面 , 下载最新的在线安装包

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
这里下载Harbor V2.0.2版本的安装包
[root@k8s-harbor01 ~]# cd /usr/local/src/
[root@k8s-harbor01 src]# wget https://github.com/goharbor/harbor/releases/download/v2.0.2/harbor-online-installer-v2.0.2.tgz
[root@k8s-harbor01 src]# tar -zvxf harbor-online-installer-v2.0.2.tgz
[root@k8s-harbor01 src]# mv harbor /opt/
  
修改harbor配置信息
[root@k8s-harbor01 src]# cd /opt/harbor/
[root@k8s-harbor01 harbor]# cp harbor.yml.tmpl harbor.yml
.........
........
hostname: 172.16.60.238
  
# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80
  
# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/cert/172.16.60.238.crt
  private_key: /data/cert/172.16.60.238.key
........
........
harbor_admin_password: Harbor@123456
........
........
data_volume: /data
  
  
运行install.sh, 注意运行时加上--with-clair 选项，启动clair镜像扫描功能
[root@k8s-harbor01 harbor]# ./install.sh --with-clair
........
........
✔ ----Harbor has been installed and started successfully.----
  
出现上面的信息，说明Harbor已经安装完成了。
  
查看harbor启动情况。
docker-compose 命令必须要在harbor安装目录 (这里就是/opt/harbor)路径下才能执行。
[root@k8s-harbor01 harbor]# docker-compose ps
      Name                     Command                  State                          Ports
---------------------------------------------------------------------------------------------------------------
clair               ./docker-entrypoint.sh           Up (healthy)   6060/tcp, 6061/tcp
clair-adapter       /home/clair-adapter/entryp ...   Up (healthy)   8080/tcp
harbor-core         /harbor/entrypoint.sh            Up (healthy)
harbor-db           /docker-entrypoint.sh            Up (healthy)   5432/tcp
harbor-jobservice   /harbor/entrypoint.sh            Up (healthy)
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp
redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp
registry            /home/harbor/entrypoint.sh       Up (healthy)   5000/tcp
registryctl         /home/harbor/start.sh            Up (healthy)
  
查看harbor镜像
[root@k8s-harbor01 ~]# docker images
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
goharbor/redis-photon           v2.0.2              e547529bb6a1        3 weeks ago         72.3MB
goharbor/clair-adapter-photon   v2.0.2              9ec8853dc3cb        3 weeks ago         62MB
goharbor/clair-photon           v2.0.2              73885002dda7        3 weeks ago         171MB
goharbor/harbor-registryctl     v2.0.2              9f8b7bb0f1ff        3 weeks ago         101MB
goharbor/registry-photon        v2.0.2              eac8c5fc9ca8        3 weeks ago         83.6MB
goharbor/nginx-photon           v2.0.2              eee4771b916c        3 weeks ago         43.6MB
goharbor/harbor-log             v2.0.2              b2db762a6c3a        3 weeks ago         82.1MB
goharbor/harbor-jobservice      v2.0.2              3960e027ccb9        3 weeks ago         164MB
goharbor/harbor-core            v2.0.2              de2495b944cf        3 weeks ago         145MB
goharbor/harbor-portal          v2.0.2              90088a0e64a9        3 weeks ago         52.5MB
goharbor/harbor-db              v2.0.2              81e98a7af097        3 weeks ago         161MB
goharbor/prepare                v2.0.2              7e804db05454        3 weeks ago         160MB
  
确保harbpr启动后的80和443端口都起来了
[root@k8s-harbor01 harbor]# lsof -i:80
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 3095 root    4u  IPv6  26027      0t0  TCP *:http (LISTEN)
[root@k8s-harbor01 harbor]# lsof -i:443
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 3082 root    4u  IPv6  26015      0t0  TCP *:https (LISTEN)
  
到这里就可以访问harbor了,访问地址为：https://172.16.60.238
用户名为admin，密码为配置文件中定义的"Harbor@123456"

查看clair镜像扫描器

 

Habor 服务启停
注意：如果harbor.yml配置修改了，要先执行"./prepare"命令进行配置载入，然后再重启harbor服务。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
查看Habror
docker-compose ps
  
启动Harbor
docker-compose start
  
停止Harbor
docker-compose stop
  
重启Harbor
docker-compose restart
  
另外：
Harbor还可以通过down和up命令去停止和启动，
只不过这种方式是删除、创建的关停和启动。
docker-compose down -v
docker-compose up -d

　　

五、客户端登录Harbor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
在Habror客户端机器（如k8s的node节点、harbor节点）配置登录：
  
默认情况下，在客户端登录Habor是会报错的：
[root@k8s-node01 ~]# docker login 172.16.60.238
Authenticating with existing credentials...
Login did not succeed, error: Error response from daemon: Get https://172.16.60.238/v2/: x509: certificate signed by unknown authority
  
原因： 客户端登录Harbor，https证书不被信任。
  
解决办法：下面两种方法选其一
1）方法一
将Harbor服务器证书，密钥和CA文件复制到Harbor客户主机上的Docker certificate文件夹中
[root@k8s-node01 ~]# mkdir -p /etc/docker/certs.d/172.16.60.238/
[root@k8s-node01 ~]# cd /etc/docker/certs.d/172.16.60.238/
[root@k8s-node01 172.16.60.238]# rsync -e "ssh -p22" -avpgolr 172.16.60.238:/etc/docker/certs.d/172.16.60.238/* ./
[root@k8s-node01 172.16.60.238]# ll
total 12
-rw-r--r-- 1 root root 2053 Aug 19 14:34 172.16.60.238.cert
-rw-r--r-- 1 root root 3243 Aug 19 14:34 172.16.60.238.key
-rw-r--r-- 1 root root 2033 Aug 19 14:34 ca.crt
  
重启docker服务
[root@k8s-node01 172.16.60.238]# systemctl restart docker
[root@k8s-node01 172.16.60.238]# systemctl status docker
  
再次验证登录harbor
[root@k8s-node01 172.16.60.238]# docker login 172.16.60.238
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
  
Login Succeeded
  
2）方法二
配置docker服务的daemon.json文件，添加"insecure-registries"参数，表示忽略ssl证书认证。
[root@k8s-node01 ~]# vim /etc/docker/daemon.json
........
    "insecure-registries": ["https://172.16.60.238"],
  
重启docker服务
[root@k8s-node01 ~]# systemctl restart docker
[root@k8s-node01 ~]# systemctl status docker
  
再次验证登录harbor
[root@k8s-node01 ~]# docker login 172.16.60.238
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
  
Login Succeeded
  
========================================================================
另外，注意客户端机器登录Harbor时，只要首次登录需要输入用户名和密码。
登录成功后的信息默认保存到/root/.docker/config.json文件里。
下次登录时就不用再输入harbor用户名和密码了，直接读取config.json文件内容
[root@k8s-node01 ~]# cat /root/.docker/config.json
{
        "auths": {
                "172.16.60.238": {
                        "auth": "YWRtaW46SGFyYm9yQDEyMzQ1Ng=="
                }
        },
        "HttpHeaders": {
                "User-Agent": "Docker-Client/18.09.6 (linux)"
        }
  
  
[root@k8s-node01 ~]# docker login 172.16.60.238
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
  
Login Succeeded

　　

六、Harbor镜像扫描

选中镜像，进行漏洞扫描

如果扫描出漏洞，在漏洞报告了会告知漏洞当前版本和修复版本，按照修复版本修复即可。

 

修复方法：
可以依据当前基础镜像做Dockerfile，使用"yum update -y 漏洞所属软件名" 进行升级操作，然后再重新做一个基础镜像。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
1) 编译Dockerfile
升级原来centos7.7基础镜像里报出来漏洞的软件
[root@k8s-harbor01 ~]# cat Dockerfile
FROM 172.16.60.238/kevin/centos7.7:latest
RUN yum update -y sqlite \
&& yum update -y nss-util \
&& yum update -y nss-sysinit \
&& yum update -y dbus-libs \
&& yum update -y bind-license \
&& yum update -y nss \
&& yum update -y nss-softokn \
&& yum update -y dbus \
&& yum update -y nss-softokn-freebl \
&& yum update -y nss-tools \
&& yum update -y bash \
&& yum update -y python-libs \
&& yum update -y python \
&& yum update -y bind-license \
&& yum update -y expat \
&& yum update -y libxml2-python \
&& yum update -y libxml2 \
&& yum update -y shared-mime-info \
&& yum update -y libcurl \
&& yum update -y file-libs \
&& yum update -y curl
  
2）制作新的基础镜像
[root@k8s-harbor01 ~]# docker build -t 172.16.60.238/kevin/centos7.7:updatev1 .
  
3）上传到Harbor仓库
[root@k8s-harbor01 ~]# docker push 172.16.60.238/kevin/centos7.7:updatev1

将修复好漏洞的新基础镜像上传到Harbor仓库，再扫描新镜像，发现漏洞已修复。

*************** 当你发现自己的才华撑不起野心时，就请安静下来学习吧！***************
 
分类: Docker/K8S
posted @ 2021-11-29 14:38  技术颜良  阅读(462)  评论(0)    收藏  举报