AWS学习笔记（十三）：Developing on AWS, FSx, Storage Gateway

AWS EC2 Instance Metadata

• AWS EC2 Instance Metadata is powerful but one of the least known features to developers
• It allows AWS EC2 instances to "learn about themselves" without using an IAM Role for that purpose.
• The URL is http://169.254.169.254/latest/meta-data
• You can retrieve the IAM Role name from the metadata, but you CANNOT retrieve the IAM Policy

Amazon FSx

• Launch 3rd party high-performance file systems on AWS
• Fully managed service

Amazon FSx for Windows (File Server)

• EFS is a shared POSIX system for Linux systems.
• FSx for Windows is a fully managed Windows file system share drive
• Supports SMB protocol & Windows NTFS
• Microsoft Active Directory integration, ACLs, user quotas
• Built on SSD, scale up to 10s of GB/s, millions of IOPS, 100s PB of data
• Can be accessed from your on-premise infrastructure
• Can be configured to be Multi-AZ (high availability)
• Data is backed-up daily to S3

Amazon FSx for Lustre

• Lustre is a type of parallel distributed file system, for large-scale computing
• The name Lustre is derived from "Linux" and "cluster"
• Machine Learning, High Performance Computing (HPC)
• Video Processing, Financial Modeling, Electronic Design Automation
• Scales up to 100s GB/s, millions of IOPS, sub-ms latencies
• Seamless integration with S3
  • Can "read S3" as a file system (through FSx)
  • Can write the output of the computations back to S3 (through FSx)
• Can be used from on-premise servers

AWS Storage Gateway

• Bridge between on-premises data and cloud data in S3
• Use cases: disaster recovery, backup & restore, tiered storage

File Gateway

• Configured S3 buckets are accessible using the NFS and SMB protocol
• Supports S3 standard, S3 IA, S3 One Zone IA
• Bucket access using IAM roles for each File Gateway
• Most recently used data is cached in the file gateway
• Can be mounted on many servers
• Integrated with Active Directory (AD) for user authentication

Volume Gateway

• Block storage using iSCSI protocol backed by S3
• Backed by EBS snapshots which can help restore on-premises volumes!
• Cached volumes: low latency access to most recent data
• Stored volumes: entire dataset is on premise, scheduled backups to S3

Tape Gateway

• Some companies have backup processes using physical tapes (!)
• With Tape Gateway, companies use the same processes but, in the cloud
• Virtual Tape Library (VTL) backed by Amazon S3 and Glacier
• Back up data using existing tape-based processes (and iSCSI interface)
• Works with leading backup software vendors

Storage Comparison

• S3: Object Storage
• Glacier: Object Archival
• EFS: Network File System for Linux instances, POSIX filesystem
• FSx for Windows: Network File System for Windows servers
• FSx for Lustre: High Performance Computing Linux file system
• EBS volumes: Network storage for one EC2 instance at a time
• Instance Storage: Physical storage for your EC2 instance (high IOPS)
• Storage Gateway: File Gateway, Volume Gateway (cache & stored), Tape Gateway
• Snowball / Snowmobile: to move large amount of data to the cloud, physically
• Database: for specific workloads, usually with indexing and querying