aws ecs 理解元数据和mock本地测试环境
资料
- Under the Hood: Task Networking for Amazon ECS
- amazon-ecs-local-container-endpoints
- A Guide to Locally Testing Containers with Amazon ECS Local Endpoints and Docker Compose
- Amazon ECS 容器代理自检
- Amazon ECS 任务元数据端点
本文讨论了ecs 任务获取元数据和凭证的逻辑和测试方法
任务元数据和凭证的端点是由ecs agent提供的。ecs agent使用host模式启动，能够阻止ecs agent启动的容器访问http://169.254.169.254获取实例元数据，并避免docker0上的链接和流量争用。
ecs agent提供了一个 API （自检），用于收集有关正在运行该代理的容器实例以及在该实例上正在运行的相关任务的详细信息
$ curl -s http://localhost:51678/v1/metadata | jq .
{
"Cluster": "xxxxxx",
"ContainerInstanceArn": "arn:aws-cn:ecs:cn-north-1:xxxxxx:container-instance/xxxxx/8293f9feb64d409fa069d329e63ba984",
"Version": "Amazon ECS Agent - v1.62.2 (*a1a5ecbc)"
}
ecs 元数据
注意：为任务指定 IAM 角色时，该任务的容器中的 Amazon CLI 或其他开发工具包只使用该任务角色提供的 Amazon 凭证，它们不再从 Amazon EC2 或它们运行所在的外部实例继承任何 IAM 权限
ecs元数据端点有多个版本，目前支持的有2，3和4
-
版本4（>1.39.0，> 1.54.0 for windows using awsvpc），容器元数据，docker统计信息
任务元数据和联网速率统计数据将发送到 cw container insight
-
版本3（>1.21.0，> 1.54.0 for windows using awsvpc），容器元数据，docker统计信息
从 代理版本 1.21.0 开始，代理将称为
ECS_CONTAINER_METADATA_URI的环境变量注入任务中的每个容器 -
版本2（fargate没有）（>1.17.0，> 1.54.0 for windows using awsvpc），容器元数据，docker统计信息
查看ecs 实例上容器注入的环境变量如下
"ECS_CONTAINER_METADATA_URI_V4=http://169.254.170.2/v4/ef186256-092c-49b5-ace1-7c426b4aaaf2",
"AWS_EXECUTION_ENV=AWS_ECS_EC2",
"AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/af07beab-d714-4b4a-aa27-75168bbe0c66",
"ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3/ef186256-092c-49b5-ace1-7c426b4aaaf2",
获取元数据路径
$ curl $ECS_CONTAINER_METADATA_URI_V4 #元数据
$ curl $ECS_CONTAINER_METADATA_URI_V4/task #任务元数据
$ curl $ECS_CONTAINER_METADATA_URI_V4/taskWithTags #带标签的任务元数据
$ curl $ECS_CONTAINER_METADATA_URI_V4/stats #docker统计信息，cpu，内存，网络数据收发
$ curl $ECS_CONTAINER_METADATA_URI_V4/task/stats #任务中容器的docker统计数据
获取凭证
# curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
{
"RoleArn":"arn:aws-cn:iam::xxxxxx:role/ecsTaskRole",
"AccessKeyId":"ASIAQRIBWRJKBUNH7I67",
"SecretAccessKey":"Gm0dkB3zpoumsj/IVVcEKXXg8M8mC6JmGQuScFD0",
"Token":"xxxxxxxxxxx",
"Expiration":"2022-11-21T17:48:24Z"
}
ecs agent在启动时为了使容器使用task role设置了iptable规则，因此访问127.0.0.1同样能获到元数据
$ sysctl -w net.ipv4.conf.all.route_localnet=1
$ iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679
$ iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679
访问169.254.170.2实际是访问agent获取信息
#ecs agent 监听51679端口
$ sudo netstat -ntulp | grep 51679
tcp 0 0 127.0.0.1:51679 0.0.0.0:* LISTEN 4162/agent
ecs容器代理通过任务元数据端点提供了检索任务元数据和docker统计指标的方法，可以在官方文档找到完整输出
bash-4.2# curl $ECS_CONTAINER_METADATA_URI_V4
{
"DockerId":"4bedfcb6e1df59992d7e579b11edfaed6b987c2ce67bf9c937d9b287a5f937b3",
"Name":"amazonlinux",
"DockerName":"ecs-amazonlinux-4-amazonlinux-fc9ecebdc0d7c9ac7300",
"Image":"amazonlinux:latest",
"ImageID":"sha256:6e809582795f51280dda491769531ca101af7ce73ff67ec039597b1f000fef8c",
"Labels":{
"com.amazonaws.ecs.cluster":"xxxxxx",
"com.amazonaws.ecs.container-name":"amazonlinux",
"com.amazonaws.ecs.task-arn":"arn:aws-cn:ecs:cn-north-1:xxxxxx:task/xxxxxx/10b941d27ed9472681f226ae63b34771",
"com.amazonaws.ecs.task-definition-family":"amazonlinux",
"com.amazonaws.ecs.task-definition-version":"4"
},
"DesiredStatus":"RUNNING",
"KnownStatus":"RUNNING",
"Limits":{
"CPU":2,
"Memory":0
},
"Type":"NORMAL",
"ContainerARN":"arn:aws-cn:ecs:cn-north-1:xxxxxx:container/xxxxxx/10b941d27ed9472681f226ae63b34771/e832efc3-3529-472d-b145-92a754f79ab3",
"Networks":[
{
"NetworkMode":"bridge",
"IPv4Addresses":[
"172.17.0.5"
]
}
]
}
获取的统计信息如下
{
"read": "2022-12-10T04:05:36.877932601Z",
"preread": "2022-12-10T04:05:35.857775774Z",
"pids_stats": {
"current": 2
},
"blkio_stats": {
"io_service_bytes_recursive": [],
"io_serviced_recursive": [],
"io_queue_recursive": [],
"io_service_time_recursive": [],
"io_wait_time_recursive": [],
"io_merged_recursive": [],
"io_time_recursive": [],
"sectors_recursive": []
},
"num_procs": 0,
"storage_stats": {},
"cpu_stats": {
"cpu_usage": {
"total_usage": 471454000,
"percpu_usage": [
236453619,
235000381
],
"usage_in_kernelmode": 50000000,
"usage_in_usermode": 370000000
},
"system_cpu_usage": 183090590000000,
"online_cpus": 2,
"throttling_data": {
"periods": 0,
"throttled_periods": 0,
"throttled_time": 0
}
},
"precpu_stats": {
"cpu_usage": {
"total_usage": 471325043,
"percpu_usage": [
236417821,
234907222
],
"usage_in_kernelmode": 50000000,
"usage_in_usermode": 370000000
},
"system_cpu_usage": 183088550000000,
"online_cpus": 2,
"throttling_data": {
"periods": 0,
"throttled_periods": 0,
"throttled_time": 0
}
},
"memory_stats": {
"usage": 1527808,
"max_usage": 5378048,
"stats": {
"active_anon": 458752,
"active_file": 0,
"cache": 0,
"dirty": 0,
"hierarchical_memory_limit": 268435456,
"hierarchical_memsw_limit": 9223372036854771712,
"inactive_anon": 0,
"inactive_file": 0,
"mapped_file": 0,
"pgfault": 8118,
"pgmajfault": 0,
"pgpgin": 4785,
"pgpgout": 4662,
"rss": 401408,
"rss_huge": 0,
"total_active_anon": 458752,
"total_active_file": 0,
"total_cache": 0,
"total_dirty": 0,
"total_inactive_anon": 0,
"total_inactive_file": 0,
"total_mapped_file": 0,
"total_pgfault": 8118,
"total_pgmajfault": 0,
"total_pgpgin": 4785,
"total_pgpgout": 4662,
"total_rss": 401408,
"total_rss_huge": 0,
"total_unevictable": 0,
"total_writeback": 0,
"unevictable": 0,
"writeback": 0
},
"limit": 4072448000
},
"name": "/ecs-amazonlinux-4-amazonlinux-86ced49c948ed8aed001",
"id": "e61b98b2996af65daa69f422256ce25e66d809e9de5bf553dc16760cb874032a",
"networks": {
"eth0": {
"rx_bytes": 10902,
"rx_packets": 55,
"rx_errors": 0,
"rx_dropped": 0,
"tx_bytes": 4854,
"tx_packets": 59,
"tx_errors": 0,
"tx_dropped": 0
}
}
}
在本地mock ecs容器环境
该工具的使用场景如下，当没有aws账号或者跨系统开发时需要依赖容器数据端点，可以使用该工具mock。但是这个工具貌似不支持v4版本的端点
A Guide to Locally Testing Containers with Amazon ECS Local Endpoints and Docker Compose
- Testing a container that needs credentials to interact with AWS Services
- Testing a container which uses Task Metadata
- Testing a multi-container app which uses the awsvpc or host network mode on Docker For Mac and Docker For Windows (in linux mode)
- Testing multiple containerized applications using local service discovery
在开始mock之前需要对网路进行配置，有以下两种方式
- 使用docker自定义网络（更推荐）
- 像ecs agent一样设置iptables规则将流量导入local container endpoint容器中
这里使用自定义网络配置，看一下docker-compose.yaml文件内容
version: "2"
networks:
#定义自定义网络，网段为169.254.170.0/24，桥接模式
credentials_network:
driver: bridge
ipam:
config:
- subnet: "169.254.170.0/24"
gateway: 169.254.170.1
services:
# 为容器提供凭证的服务
ecs-local-endpoints:
image: amazon/amazon-ecs-local-container-endpoints
volumes:
# 挂载docker.sock文件
- /var/run:/var/run
# 挂载凭证配置
- $HOME/.aws/:/home/.aws/
environment:
# 使用宿主机的默认凭证
AWS_PROFILE: "default"
networks:
credentials_network:
# 这个ip在aws cli和aws sdk中写死了
ipv4_address: "169.254.170.2"
# 我们的应用程序的配置
app:
image: amazonlinux:latest
command: sleep infinity
depends_on:
- ecs-local-endpoints
networks:
credentials_network:
ipv4_address: "169.254.170.3"
environment:
AWS_DEFAULT_REGION: "cn-north-1"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds"
ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"
在一台普通的ec2实例上，启动环境进入amazonlinux容器查看，环境变量已经注入
# env
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/creds
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
AWS_DEFAULT_REGION=cn-north-1
ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3
mock 凭证
在容器内安装aws cli工具查看当前的凭证，确实获取到了宿主机的凭证
# aws sts get-caller-identity
{
"UserId": "xxxxxxxxx:i-xxxxxxx",
"Account": "xxxxxx",
"Arn": "arn:aws-cn:sts::xxxxxx:assumed-role/MyEc2xxxxxAccess/i-xxxxxx"
}
mock 元数据
访问任务元数据端点，mock的数据可以拿到
$ curl $ECS_CONTAINER_METADATA_URI
{
"DockerId": "d8bc2359f93bf89a4a25560c76965d3ff91f4e82c1aeee8b0b8414f5dd28ae86",
"Name": "mock-app-1",
"DockerName": "mock-app-1",
"Image": "amazonlinux:latest",
"ImageID": "sha256:6e809582795f51280dda491769531ca101af7ce73ff67ec039597b1f000fef8c",
"Labels": {
"com.docker.compose.config-hash": "7f1cb58c41f4f568d23c307f248ec102fcfac153932a6a26c726013b79aace70",
"com.docker.compose.container-number": "1",
"com.docker.compose.depends_on": "ecs-local-endpoints:service_started",
"com.docker.compose.image": "sha256:6e809582795f51280dda491769531ca101af7ce73ff67ec039597b1f000fef8c",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "mock",
"com.docker.compose.project.config_files": "/home/ec2-user/amazon-ecs-local-container-endpoints/mock/docker-compose.yaml",
"com.docker.compose.project.working_dir": "/home/ec2-user/amazon-ecs-local-container-endpoints/mock",
"com.docker.compose.service": "app",
"com.docker.compose.version": "2.4.1"
},
"DesiredStatus": "RUNNING",
"KnownStatus": "RUNNING",
"Limits": {},
"Type": "NORMAL",
"Networks": [
{
"NetworkMode": "mock_credentials_network",
"IPv4Addresses": [
"169.254.170.3"
]
}
]
}
mock awsvpc网咯模式
使用awsvpc模式启动的任务独占一张网卡，因此相同任务中的容器可以通过lcoalhost进行通信。将容器启动在同一个网络空间下，需要pause容器进行连接（类似于kubernetes的pod）
构建pause容器
$ cat dockerfile
FROM amazonlinux:latest
RUN yum install -y iptables
CMD iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679 \
&& iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679 \
&& iptables-save \
&& /bin/bash -c 'while true; do sleep 30; done;'
$ docker build -t local-pause:latest .
之后我们创建的所有容器的网络空间都应该在pause容器中
version: "2"
services:
ecs-local-endpoints:
image: amazon/amazon-ecs-local-container-endpoints
volumes:
- /var/run:/var/run
- $HOME/.aws/:/home/.aws/
environment:
ECS_LOCAL_METADATA_PORT: "51679"
HOME: "/home"
# 修改网络模式
network_mode: container:local-pause
app:
image: amazonlinux:latest
command: sleep infinity
depends_on:
- ecs-local-endpoints
network_mode: container:local-pause
environment:
ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3/containers/app"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds"
在运行之前首先需要启动pause容器将基本的网络空间建立，使用-p设置在任务定义中容器需要expose的端口
$ docker run -d -p 8080:8080 --name local-pause --cap-add=NET_ADMIN local-pause
进入app查看网络信息，可以看到ecs-local-endpoints监听的端口，此时容器可以通过localhost相互通信
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
24: eth0@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::51679 :::* LISTEN -
浙公网安备 33010602011771号