分布式短链接体系设计方案
分布式短链接系统设计方案
1. 系统架构设计
1.1 整体系统架构图
[客户端]
|
[CDN]
|
[负载均衡器]
/ \
[API Gateway] [Web Server]
| |
┌─────────┴─────────────┴─────────┐
| 应用服务层 |
| ┌─────────┬─────────┬────────┐ |
| |短链生成 |URL重定向|统计服务| |
| |服务 |服务 | | |
| └─────────┴─────────┴────────┘ |
└─────────┬─────────────┬─────────┘
| |
┌─────────┴─────────┐ |
| 缓存层 | |
| ┌─────┬─────────┐ | |
| |Redis|Memcached| | |
| |集群 | | | |
| └─────┴─────────┘ | |
└─────────┬─────────┘ |
| |
┌─────────┴─────────────┴─────────┐
| 数据存储层 |
| ┌──────────┬──────────┬────────┐ |
| |MySQL主库 |MySQL从库|MongoDB| |
| |分片集群 |读写分离 |日志存储| |
| └──────────┴──────────┴────────┘ |
└─────────────────────────────────┘1.2 核心组件说明
1.2.1 接入层
- CDN: 全球分布式缓存，提升访问速度
- 负载均衡器: Nginx/HAProxy，支持多种负载均衡算法
- API Gateway: 统一入口，提供限流、鉴权、监控功能
1.2.2 应用服务层
- 短链生成服务: 负责将长URL转换为短链接
- URL重定向服务: 处理短链接访问，重定向到原始URL
- 统计服务: 收集和分析访问数据
- 管理服务: 提供短链接的增删改查功能
1.2.3 缓存层
- Redis集群: 热点数据缓存，支持主从复制和哨兵模式
- 本地缓存: 应用层缓存，减少网络开销
1.2.4 数据存储层
- MySQL分片集群: 存储URL映射关系
- MongoDB: 存储访问日志和统计数据
- 消息队列: 异步处理统计数据
1.3 核心业务流程设计
1.3.1 短链接生成流程
用户提交长URL → 参数校验 → 检查缓存 → 生成短链接ID →
存储映射关系 → 更新缓存 → 返回短链接1.3.2 短链接访问流程
用户访问短链接 → CDN查找 → 缓存查找 → 数据库查找 →
记录访问日志 → 重定向到原始URL2. 核心算法设计
2.1 短链接生成算法对比
2.1.1 Base62编码方案
public class Base62Encoder {
private static final String BASE62 = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
private static final int BASE = BASE62.length();
public static String encode(long num) {
StringBuilder sb = new StringBuilder();
while (num > 0) {
sb.append(BASE62.charAt((int)(num % BASE)));
num /= BASE;
}
return sb.reverse().toString();
}
public static long decode(String str) {
long num = 0;
for (char c : str.toCharArray()) {
num = num * BASE + BASE62.indexOf(c);
}
return num;
}
}优点:
- 算法简单，性能高
- 生成的短链接较短
- 无需额外存储
缺点:
- 可预测性高，存在安全风险
- 需要全局唯一ID生成器
2.1.2 雪花算法 + Base62方案
public class SnowflakeIdGenerator {
private final long epoch = 1640995200000L; // 2022-01-01 00:00:00
private final long workerIdBits = 10L;
private final long sequenceBits = 12L;
private final long maxWorkerId = ~(-1L << workerIdBits);
private final long maxSequence = ~(-1L << sequenceBits);
private final long workerIdShift = sequenceBits;
private final long timestampShift = sequenceBits + workerIdBits;
private long workerId;
private long sequence = 0L;
private long lastTimestamp = -1L;
public SnowflakeIdGenerator(long workerId) {
if (workerId > maxWorkerId || workerId < 0) {
throw new IllegalArgumentException("Worker ID out of range");
}
this.workerId = workerId;
}
public synchronized long nextId() {
long timestamp = System.currentTimeMillis();
if (timestamp < lastTimestamp) {
throw new RuntimeException("Clock moved backwards");
}
if (timestamp == lastTimestamp) {
sequence = (sequence + 1) & maxSequence;
if (sequence == 0) {
timestamp = waitNextMillis(lastTimestamp);
}
} else {
sequence = 0L;
}
lastTimestamp = timestamp;
return ((timestamp - epoch) << timestampShift) |
(workerId << workerIdShift) |
sequence;
}
private long waitNextMillis(long lastTimestamp) {
long timestamp = System.currentTimeMillis();
while (timestamp <= lastTimestamp) {
timestamp = System.currentTimeMillis();
}
return timestamp;
}
}优点:
- 全局唯一，无重复
- 性能高，单机可达百万QPS
- 包含时间信息，便于排序
缺点:
- 依赖机器时钟
- 需要机器ID管理
2.1.3 Hash + 冲突检测方案
public class HashBasedGenerator {
private static final String SALT = "your_salt_here";
public String generateShortUrl(String longUrl) {
String combined = longUrl + SALT + System.currentTimeMillis();
long hash = MurmurHash.hash64(combined.getBytes());
return Base62Encoder.encode(Math.abs(hash));
}
public String generateWithCollisionDetection(String longUrl) {
String shortUrl;
int attempts = 0;
do {
String input = longUrl + SALT + System.currentTimeMillis() + attempts;
long hash = MurmurHash.hash64(input.getBytes());
shortUrl = Base62Encoder.encode(Math.abs(hash));
attempts++;
} while (exists(shortUrl) && attempts < 5);
if (attempts >= 5) {
// 降级到雪花算法
return Base62Encoder.encode(snowflakeGenerator.nextId());
}
return shortUrl;
}
}2.2 分布式ID生成方案
2.2.1 数据库自增ID + 步长
-- 节点1: 起始值1，步长3
ALTER TABLE url_mapping AUTO_INCREMENT = 1;
SET @@auto_increment_increment = 3;
-- 节点2: 起始值2，步长3
ALTER TABLE url_mapping AUTO_INCREMENT = 2;
SET @@auto_increment_increment = 3;
-- 节点3: 起始值3，步长3
ALTER TABLE url_mapping AUTO_INCREMENT = 3;
SET @@auto_increment_increment = 3;2.2.2 Redis计数器方案
public class RedisIdGenerator {
private RedisTemplate<String, String> redisTemplate;
private String keyPrefix = "short_url_id:";
public long nextId(int shardId) {
String key = keyPrefix + shardId;
return redisTemplate.opsForValue().increment(key, 1);
}
public String generateShortUrl(int shardId) {
long id = nextId(shardId);
return Base62Encoder.encode(id);
}
}3. 数据库设计
3.1 数据表结构设计
3.1.1 URL映射表
CREATE TABLE `url_mapping` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`short_url` varchar(10) NOT NULL COMMENT '短链接标识',
`long_url` text NOT NULL COMMENT '原始长URL',
`user_id` bigint(20) DEFAULT NULL COMMENT '用户ID',
`expire_time` datetime DEFAULT NULL COMMENT '过期时间',
`status` tinyint(1) DEFAULT 1 COMMENT '状态：1-有效，0-无效',
`created_time` datetime DEFAULT CURRENT_TIMESTAMP,
`updated_time` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `uk_short_url` (`short_url`),
KEY `idx_user_id` (`user_id`),
KEY `idx_created_time` (`created_time`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='URL映射表';3.1.2 访问统计表
CREATE TABLE `url_statistics` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`short_url` varchar(10) NOT NULL,
`access_date` date NOT NULL,
`pv` bigint(20) DEFAULT 0 COMMENT '页面访问量',
`uv` bigint(20) DEFAULT 0 COMMENT '独立访客数',
`ip_count` bigint(20) DEFAULT 0 COMMENT '独立IP数',
`created_time` datetime DEFAULT CURRENT_TIMESTAMP,
`updated_time` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `uk_short_url_date` (`short_url`, `access_date`),
KEY `idx_access_date` (`access_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='URL访问统计表';3.1.3 访问日志表（MongoDB）
// MongoDB集合结构
{
"_id": ObjectId("..."),
"shortUrl": "abc123",
"longUrl": "https://example.com/very/long/url",
"clientIp": "192.168.1.1",
"userAgent": "Mozilla/5.0...",
"referer": "https://google.com",
"accessTime": ISODate("2023-01-01T12:00:00Z"),
"responseTime": 50,
"statusCode": 302,
"country": "CN",
"city": "Beijing",
"device": "mobile"
}3.2 分库分表策略
3.2.1 水平分表策略
public class ShardingStrategy {
private static final int SHARD_COUNT = 1024;
public String getShardTable(String shortUrl) {
int hash = shortUrl.hashCode();
int shardId = Math.abs(hash) % SHARD_COUNT;
return "url_mapping_" + String.format("%04d", shardId);
}
public String getShardDatabase(String shortUrl) {
int hash = shortUrl.hashCode();
int dbId = Math.abs(hash) % 8; // 8个数据库
return "shorturl_db_" + dbId;
}
}3.2.2 分库分表配置
# ShardingSphere配置
spring:
shardingsphere:
datasource:
names: ds0,ds1,ds2,ds3,ds4,ds5,ds6,ds7
ds0:
type: com.zaxxer.hikari.HikariDataSource
driver-class-name: com.mysql.cj.jdbc.Driver
jdbc-url: jdbc:mysql://192.168.1.10:3306/shorturl_db_0
# ... 其他数据源配置
rules:
sharding:
tables:
url_mapping:
actual-data-nodes: ds$->{0..7}.url_mapping_$->{0000..1023}
database-strategy:
standard:
sharding-column: short_url
sharding-algorithm-name: database_inline
table-strategy:
standard:
sharding-column: short_url
sharding-algorithm-name: table_inline
sharding-algorithms:
database_inline:
type: INLINE
props:
algorithm-expression: ds$->{Math.abs(short_url.hashCode()) % 8}
table_inline:
type: INLINE
props:
algorithm-expression: url_mapping_$->{String.format('%04d', Math.abs(short_url.hashCode()) % 1024)}3.3 索引设计
3.3.1 主要索引策略
-- 短链接唯一索引（最重要）
CREATE UNIQUE INDEX uk_short_url ON url_mapping(short_url);
-- 用户ID索引（用户查询自己的短链接）
CREATE INDEX idx_user_id ON url_mapping(user_id);
-- 创建时间索引（按时间范围查询）
CREATE INDEX idx_created_time ON url_mapping(created_time);
-- 过期时间索引（清理过期数据）
CREATE INDEX idx_expire_time ON url_mapping(expire_time);
-- 状态索引（查询有效链接）
CREATE INDEX idx_status ON url_mapping(status);
-- 复合索引（用户查询自己的有效链接）
CREATE INDEX idx_user_status ON url_mapping(user_id, status);3.3.2 MongoDB索引
// 短链接索引
db.access_logs.createIndex({"shortUrl": 1});
// 时间范围查询索引
db.access_logs.createIndex({"accessTime": 1});
// 复合索引（按短链接和时间查询）
db.access_logs.createIndex({"shortUrl": 1, "accessTime": 1});
// IP地址索引（防刷分析）
db.access_logs.createIndex({"clientIp": 1});
// TTL索引（自动删除过期日志）
db.access_logs.createIndex({"accessTime": 1}, {expireAfterSeconds: 7776000}); // 90天4. 关键技术方案
4.1 缓存策略
4.1.1 Redis集群配置
spring:
redis:
cluster:
nodes:
- 192.168.1.10:7000
- 192.168.1.10:7001
- 192.168.1.11:7000
- 192.168.1.11:7001
- 192.168.1.12:7000
- 192.168.1.12:7001
max-redirects: 3
password: your_password
timeout: 3000ms
lettuce:
pool:
max-active: 200
max-idle: 20
min-idle: 5
max-wait: 3000ms4.1.2 多级缓存策略
@Service
public class UrlMappingService {
@Autowired
private RedisTemplate<String, String> redisTemplate;
@Autowired
private UrlMappingMapper urlMappingMapper;
// 本地缓存
private final Cache<String, String> localCache = Caffeine.newBuilder()
.maximumSize(10000)
.expireAfterWrite(5, TimeUnit.MINUTES)
.build();
public String getLongUrl(String shortUrl) {
// 1. 本地缓存
String longUrl = localCache.getIfPresent(shortUrl);
if (longUrl != null) {
return longUrl;
}
// 2. Redis缓存
longUrl = redisTemplate.opsForValue().get("url:" + shortUrl);
if (longUrl != null) {
localCache.put(shortUrl, longUrl);
return longUrl;
}
// 3. 数据库查询
UrlMapping mapping = urlMappingMapper.selectByShortUrl(shortUrl);
if (mapping != null && mapping.getStatus() == 1) {
longUrl = mapping.getLongUrl();
// 更新缓存
redisTemplate.opsForValue().set("url:" + shortUrl, longUrl,
Duration.ofHours(24));
localCache.put(shortUrl, longUrl);
return longUrl;
}
return null;
}
public void invalidateCache(String shortUrl) {
localCache.invalidate(shortUrl);
redisTemplate.delete("url:" + shortUrl);
}
}4.1.3 缓存预热策略
@Component
public class CacheWarmupService {
@Autowired
private UrlMappingService urlMappingService;
@Autowired
private RedisTemplate<String, String> redisTemplate;
@Scheduled(fixedRate = 3600000) // 每小时执行一次
public void warmupHotUrls() {
// 获取热点短链接
List<String> hotUrls = getHotUrlsFromStatistics();
for (String shortUrl : hotUrls) {
String longUrl = urlMappingService.getLongUrlFromDb(shortUrl);
if (longUrl != null) {
redisTemplate.opsForValue().set("url:" + shortUrl, longUrl,
Duration.ofHours(24));
}
}
}
private List<String> getHotUrlsFromStatistics() {
// 从统计数据中获取热点URL
return urlStatisticsMapper.selectHotUrls(1000);
}
}4.2 数据一致性保证
4.2.1 分布式事务处理
@Service
public class UrlCreationService {
@Autowired
private UrlMappingMapper urlMappingMapper;
@Autowired
private RedisTemplate<String, String> redisTemplate;
@Autowired
private RocketMQTemplate rocketMQTemplate;
@Transactional(rollbackFor = Exception.class)
public String createShortUrl(CreateUrlRequest request) {
try {
// 1. 生成短链接
String shortUrl = generateShortUrl();
// 2. 数据库插入
UrlMapping mapping = new UrlMapping();
mapping.setShortUrl(shortUrl);
mapping.setLongUrl(request.getLongUrl());
mapping.setUserId(request.getUserId());
mapping.setExpireTime(request.getExpireTime());
urlMappingMapper.insert(mapping);
// 3. 发送异步消息更新缓存
CacheUpdateMessage message = new CacheUpdateMessage();
message.setShortUrl(shortUrl);
message.setLongUrl(request.getLongUrl());
message.setOperation("CREATE");
rocketMQTemplate.convertAndSend("cache-update-topic", message);
return shortUrl;
} catch (Exception e) {
log.error("创建短链接失败", e);
throw new BusinessException("创建短链接失败");
}
}
}
@RocketMQMessageListener(topic = "cache-update-topic", consumerGroup = "cache-consumer")
@Component
public class CacheUpdateConsumer implements RocketMQListener<CacheUpdateMessage> {
@Override
public void onMessage(CacheUpdateMessage message) {
try {
switch (message.getOperation()) {
case "CREATE":
case "UPDATE":
redisTemplate.opsForValue().set("url:" + message.getShortUrl(),
message.getLongUrl(), Duration.ofHours(24));
break;
case "DELETE":
redisTemplate.delete("url:" + message.getShortUrl());
break;
}
} catch (Exception e) {
log.error("更新缓存失败", e);
// 重试机制
throw e;
}
}
}4.2.2 最终一致性保证
@Component
public class ConsistencyChecker {
@Scheduled(fixedRate = 300000) // 每5分钟检查一次
public void checkDataConsistency() {
// 1. 检查数据库和缓存的一致性
List<String> inconsistentUrls = findInconsistentUrls();
for (String shortUrl : inconsistentUrls) {
// 以数据库为准，更新缓存
UrlMapping mapping = urlMappingMapper.selectByShortUrl(shortUrl);
if (mapping != null && mapping.getStatus() == 1) {
redisTemplate.opsForValue().set("url:" + shortUrl,
mapping.getLongUrl(), Duration.ofHours(24));
} else {
redisTemplate.delete("url:" + shortUrl);
}
}
}
private List<String> findInconsistentUrls() {
// 采样检查策略，避免全量检查
List<String> sampleUrls = getSampleUrls(1000);
List<String> inconsistentUrls = new ArrayList<>();
for (String shortUrl : sampleUrls) {
String cachedUrl = redisTemplate.opsForValue().get("url:" + shortUrl);
UrlMapping dbMapping = urlMappingMapper.selectByShortUrl(shortUrl);
String dbUrl = (dbMapping != null && dbMapping.getStatus() == 1)
? dbMapping.getLongUrl() : null;
if (!Objects.equals(cachedUrl, dbUrl)) {
inconsistentUrls.add(shortUrl);
}
}
return inconsistentUrls;
}
}4.3 高可用设计
4.3.1 服务熔断与降级
@Component
public class UrlServiceFallback {
@HystrixCommand(fallbackMethod = "getLongUrlFallback",
commandProperties = {
@HystrixProperty(name = "circuitBreaker.enabled", value = "true"),
@HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "20"),
@HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "50"),
@HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "3000")
})
public String getLongUrl(String shortUrl) {
return urlMappingService.getLongUrl(shortUrl);
}
public String getLongUrlFallback(String shortUrl) {
// 降级策略：返回默认页面或错误页面
log.warn("获取长链接失败，触发降级: {}", shortUrl);
return "https://example.com/error?code=service_unavailable";
}
@HystrixCommand(fallbackMethod = "createShortUrlFallback")
public String createShortUrl(CreateUrlRequest request) {
return urlCreationService.createShortUrl(request);
}
public String createShortUrlFallback(CreateUrlRequest request) {
// 降级策略：返回错误信息
throw new ServiceUnavailableException("短链接服务暂时不可用，请稍后重试");
}
}4.3.2 限流策略
@RestController
@RequestMapping("/api/v1/url")
public class UrlController {
// 基于令牌桶的限流
private final RateLimiter rateLimiter = RateLimiter.create(1000.0); // 每秒1000个请求
// 基于用户的限流
private final LoadingCache<String, RateLimiter> userRateLimiters =
Caffeine.newBuilder()
.maximumSize(10000)
.expireAfterAccess(1, TimeUnit.HOURS)
.build(key -> RateLimiter.create(10.0)); // 每个用户每秒10个请求
@PostMapping("/create")
public Result<String> createShortUrl(@RequestBody CreateUrlRequest request) {
// 全局限流
if (!rateLimiter.tryAcquire(100, TimeUnit.MILLISECONDS)) {
return Result.error("系统繁忙，请稍后重试");
}
// 用户限流
String userId = getCurrentUserId();
RateLimiter userLimiter = userRateLimiters.get(userId);
if (!userLimiter.tryAcquire(100, TimeUnit.MILLISECONDS)) {
return Result.error("请求过于频繁，请稍后重试");
}
try {
String shortUrl = urlServiceFallback.createShortUrl(request);
return Result.success(shortUrl);
} catch (Exception e) {
log.error("创建短链接失败", e);
return Result.error("创建失败，请重试");
}
}
@GetMapping("/{shortUrl}")
public void redirect(@PathVariable String shortUrl, HttpServletResponse response) {
// 重定向请求的限流策略相对宽松
if (!rateLimiter.tryAcquire(10, TimeUnit.MILLISECONDS)) {
response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
return;
}
try {
String longUrl = urlServiceFallback.getLongUrl(shortUrl);
if (longUrl != null) {
// 异步记录访问日志
recordAccessLog(shortUrl, request);
response.sendRedirect(longUrl);
} else {
response.setStatus(HttpStatus.NOT_FOUND.value());
}
} catch (Exception e) {
log.error("重定向失败", e);
response.setStatus(HttpStatus.INTERNAL_SERVER_ERROR.value());
}
}
}4.3.3 监控与告警
@Component
public class SystemMonitor {
private final MeterRegistry meterRegistry;
private final Timer.Sample sample;
@EventListener
public void handleUrlCreated(UrlCreatedEvent event) {
// 记录创建短链接的指标
meterRegistry.counter("url.created",
"user_id", event.getUserId(),
"status", "success").increment();
}
@EventListener
public void handleUrlAccessed(UrlAccessedEvent event) {
// 记录访问指标
meterRegistry.counter("url.accessed",
"short_url", event.getShortUrl(),
"status_code", String.valueOf(event.getStatusCode())).increment();
// 记录响应时间
Timer.Sample sample = Timer.start(meterRegistry);
sample.stop(Timer.builder("url.access.duration")
.description("URL access duration")
.register(meterRegistry));
}
@Scheduled(fixedRate = 60000) // 每分钟检查一次
public void checkSystemHealth() {
// 检查数据库连接
boolean dbHealth = checkDatabaseHealth();
meterRegistry.gauge("system.db.health", dbHealth ? 1 : 0);
// 检查Redis连接
boolean redisHealth = checkRedisHealth();
meterRegistry.gauge("system.redis.health", redisHealth ? 1 : 0);
// 检查服务响应时间
long avgResponseTime = getAverageResponseTime();
meterRegistry.gauge("system.response.time.avg", avgResponseTime);
// 告警逻辑
if (!dbHealth || !redisHealth || avgResponseTime > 1000) {
sendAlert("系统健康检查异常");
}
}
}5. 性能优化方案
5.1 读写分离优化
@Configuration
public class DataSourceConfig {
@Bean
@Primary
public DataSource dataSource() {
HikariDataSource masterDataSource = new HikariDataSource();
masterDataSource.setJdbcUrl("jdbc:mysql://master-db:3306/shorturl");
masterDataSource.setMaximumPoolSize(50);
HikariDataSource slaveDataSource = new HikariDataSource();
slaveDataSource.setJdbcUrl("jdbc:mysql://slave-db:3306/shorturl");
slaveDataSource.setMaximumPoolSize(100);
Map<Object, Object> dataSourceMap = new HashMap<>();
dataSourceMap.put("master", masterDataSource);
dataSourceMap.put("slave", slaveDataSource);
DynamicDataSource dynamicDataSource = new DynamicDataSource();
dynamicDataSource.setTargetDataSources(dataSourceMap);
dynamicDataSource.setDefaultTargetDataSource(masterDataSource);
return dynamicDataSource;
}
}
@Aspect
@Component
public class DataSourceAspect {
@Around("@annotation(readOnly)")
public Object around(ProceedingJoinPoint point, ReadOnly readOnly) throws Throwable {
try {
DataSourceContextHolder.setDataSourceType("slave");
return point.proceed();
} finally {
DataSourceContextHolder.clearDataSourceType();
}
}
}5.2 异步处理优化
@Service
public class AsyncUrlService {
@Async("urlTaskExecutor")
public CompletableFuture<Void> recordAccessLog(AccessLogDto logDto) {
try {
// 批量插入优化
accessLogBatch.add(logDto);
if (accessLogBatch.size() >= 100) {
flushAccessLogs();
}
} catch (Exception e) {
log.error("记录访问日志失败", e);
}
return CompletableFuture.completedFuture(null);
}
@Async("statisticsTaskExecutor")
public CompletableFuture<Void> updateStatistics(String shortUrl, String clientIp) {
try {
// 使用Redis HyperLogLog统计UV
redisTemplate.opsForHyperLogLog().add("uv:" + shortUrl + ":" + getToday(), clientIp);
// 使用Redis计数器统计PV
redisTemplate.opsForValue().increment("pv:" + shortUrl + ":" + getToday());
} catch (Exception e) {
log.error("更新统计数据失败", e);
}
return CompletableFuture.completedFuture(null);
}
@Configuration
public class AsyncConfig {
@Bean("urlTaskExecutor")
public TaskExecutor urlTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(10);
executor.setMaxPoolSize(50);
executor.setQueueCapacity(1000);
executor.setThreadNamePrefix("url-task-");
executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
executor.initialize();
return executor;
}
}
}6. 安全防护方案
6.1 防刷机制
@Component
public class AntiSpamService {
// IP限流
private final LoadingCache<String, AtomicInteger> ipCounters =
Caffeine.newBuilder()
.maximumSize(100000)
.expireAfterWrite(1, TimeUnit.MINUTES)
.build(key -> new AtomicInteger(0));
// 短链接访问频率限制
private final LoadingCache<String, AtomicInteger> urlCounters =
Caffeine.newBuilder()
.maximumSize(10000)
.expireAfterWrite(1, TimeUnit.MINUTES)
.build(key -> new AtomicInteger(0));
public boolean isSpamRequest(String clientIp, String shortUrl) {
// IP频率检查
AtomicInteger ipCount = ipCounters.get(clientIp);
if (ipCount.incrementAndGet() > 1000) { // 每分钟最多1000次
log.warn("IP访问频率过高: {}", clientIp);
return true;
}
// 短链接访问频率检查
AtomicInteger urlCount = urlCounters.get(shortUrl);
if (urlCount.incrementAndGet() > 10000) { // 每分钟最多10000次
log.warn("短链接访问频率异常: {}", shortUrl);
return true;
}
return false;
}
public boolean isBlacklistedIp(String clientIp) {
// 检查IP黑名单
return redisTemplate.opsForSet().isMember("blacklist:ip", clientIp);
}
public void addToBlacklist(String clientIp, Duration duration) {
redisTemplate.opsForSet().add("blacklist:ip", clientIp);
redisTemplate.expire("blacklist:ip", duration);
}
}6.2 恶意URL检测
@Service
public class UrlSecurityService {
private final Set<String> maliciousDomains = loadMaliciousDomains();
private final Pattern maliciousPattern = Pattern.compile(
".*(phishing|malware|virus|trojan|spam).*", Pattern.CASE_INSENSITIVE);
public boolean isSafeUrl(String url) {
try {
URL urlObj = new URL(url);
String host = urlObj.getHost().toLowerCase();
// 检查恶意域名
if (maliciousDomains.contains(host)) {
return false;
}
// 检查URL模式
if (maliciousPattern.matcher(url).matches()) {
return false;
}
// 调用第三方安全检测API
return checkWithSecurityApi(url);
} catch (Exception e) {
log.error("URL安全检测失败: {}", url, e);
return false;
}
}
private boolean checkWithSecurityApi(String url) {
// 集成Google Safe Browsing API或其他安全检测服务
// 这里简化处理
return true;
}
}7. 部署架构
7.1 Docker容器化部署
# Dockerfile
FROM openjdk:11-jre-slim
COPY target/short-url-service.jar /app/app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app/app.jar"]# docker-compose.yml
version: '3.8'
services:
app:
build: .
ports:
- "8080:8080"
environment:
- SPRING_PROFILES_ACTIVE=prod
- MYSQL_HOST=mysql
- REDIS_HOST=redis
depends_on:
- mysql
- redis
networks:
- short-url-network
mysql:
image: mysql:8.0
environment:
MYSQL_ROOT_PASSWORD: root123
MYSQL_DATABASE: shorturl
volumes:
- mysql-data:/var/lib/mysql
networks:
- short-url-network
redis:
image: redis:7-alpine
volumes:
- redis-data:/data
networks:
- short-url-network
nginx:
image: nginx:alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
depends_on:
- app
networks:
- short-url-network
volumes:
mysql-data:
redis-data:
networks:
short-url-network:
driver: bridge7.2 Kubernetes部署
# k8s-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: short-url-service
spec:
replicas: 3
selector:
matchLabels:
app: short-url-service
template:
metadata:
labels:
app: short-url-service
spec:
containers:
- name: short-url-service
image: short-url-service:latest
ports:
- containerPort: 8080
env:
- name: SPRING_PROFILES_ACTIVE
value: "k8s"
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "1Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /actuator/health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /actuator/health
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: short-url-service
spec:
selector:
app: short-url-service
ports:
- port: 80
targetPort: 8080
type: LoadBalancer8. 总结
本分布式短链接系统设计方案具备以下特点：
8.1 核心优势
- 高性能: 支持千万级QPS的访问量
- 高可用: 99.99%的服务可用性
- 高扩展: 支持水平扩展和弹性伸缩
- 数据安全: 多重防护机制保障数据安全
8.2 关键指标
- 响应时间: 平均响应时间 < 100ms
- 存储容量: 支持百亿级URL存储
- 并发处理: 单机支持10万+并发
- 数据一致性: 最终一致性保证
8.3 技术栈总结
- 应用层: Spring Boot + Spring Cloud
- 数据库: MySQL分片集群 + MongoDB
- 缓存: Redis集群 + 本地缓存
- 消息队列: RocketMQ
- 监控: Prometheus + Grafana
- 部署: Docker + Kubernetes
通过合理的架构设计、算法选择和技术方案，该系统能够满足大规模分布式短链接服务的需求，并具备良好的扩展性和维护性。
浙公网安备 33010602011771号