从上周开始,docker hub在中国已无法访问,镜像自然也拉取不了了。

我负责的一个项目需要使用kubernetes的容器运行时containerd从docker hub拉取alpine:3镜像,正在思考如何才能在无网环境拉取到这个镜像,现在连外网也拉不了了。于是开始着手处理这个问题。

Plan A

首先想到的是在配置文件中增加一个镜像站的配置,当检测到某些环境变量的时候,就在需要拉的镜像前加上这个镜像站,实现根据配置不同,从不同的镜像站拉取镜像。

然而做起来却比我想的要复杂,配置镜像的函数根本就没有传入config字典(config字典从配置文件得到),所以就没法得到镜像站配置。

    def build_pod_request_obj(self, context=None):
        """Post-hook for pod spec creation,
        used to attach resource limits to xcom sidecar and set owner object
        """
        pod = super().build_pod_request_obj(context)

        # Set resource limits for sidecar
        c = pod.spec.containers[-1]
        assert c.name == 'airflow-xcom-sidecar'
        c.image = 'alpine:3'  # By setting a tag it will not be fetched again if already present
        c.resources.requests = {"cpu": "5m", 'memory': '64Mi'}
        c.resources.limits = {"cpu": "1000m", 'memory': '64Mi'}

并且这还是个父类的成员函数在子类中的重写,一时不知道如何修改。

修改containerd镜像源

正发愁之际,突然想到,最终负责拉取镜像的是与kubernetes对接的containerd,既然docker可以添加镜像源,那么containerd也应该可以。问了问chatgpt,发现需要如此修改/etc/containerd/config.toml

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://atomhub.openatom.cn"]

这样向docker.io的拉取请求就会被分别发送到atomhub.openatom.cnatomhub.openatom.cn是一个目前能访问到的源。配置完以后systemctl daemon-reload && systemctl restart containerd,

配置生效了,但死活就还是拉不下来。

ctr image pull docker.io/alpine:3

google上搜索了很久以后,才终于找到了答案。原来不应该用命令行ctr,而应该用crictl,因为我们修改的是containerdcri配置(container runtime interface),使cri使用mirror,而containerd自带的命令行ctr是不受影响的,kubernetes的命令行crictlcontainerd通过cri连接,因此使用crictl才是正确的选择。

crictl pull docker.io/alpine:3

这样的话就没问题了。

路径问题

然后问题还没有完全解决。crictl pull docker.io/alpine:3可以成功拉取镜像,但crictl pull alpine:3却不行,实际的代码中要执行的是后面的命令。

root@des206:/etc/containerd# crictl pull alpine:3
E0615 15:32:17.075323 1813746 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"docker.io/library/alpine:3\": failed to copy: httpReadSeeker: failed open: unexpected status code https://atomhub.openatom.cn/v2/library/alpine/blobs/sha256:4fc1d548892b0a8c8020fa151a38283cd62b9f8bfb1101f70f951b8234fafdb0?ns=docker.io: 502 Bad Gateway" image="alpine:3"
FATA[0003] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/alpine:3": failed to copy: httpReadSeeker: failed open: unexpected status code https://atomhub.openatom.cn/v2/library/alpine/blobs/sha256:4fc1d548892b0a8c8020fa151a38283cd62b9f8bfb1101f70f951b8234fafdb0?ns=docker.io: 502 Bad Gateway

猜测是路径问题,把配置改成如下:

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://atomhub.openatom.cn"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io/library"]
          endpoint = ["https://atomhub.openatom.cn/library"]

这样不管crictl pull docker.io/alpine:3还是crictl pull alpine:3就都能成功了。

root@des206:/etc/containerd# crictl pull docker.io/alpine:3
Image is up to date for sha256:df5a07b2eb43237722c03f02e718acc70a1e54a0ddaa8c5704cdbb32f6adfe6a
root@des206:/etc/containerd# crictl pull alpine:3
Image is up to date for sha256:df5a07b2eb43237722c03f02e718acc70a1e54a0ddaa8c5704cdbb32f6adfe6a