SERP API 错误处理 5 模式:retry / circuit breaker / fallback / 限流感知 / 降级

5 种错误模式

1. Retry with exponential backoff

import time
import random

def fetch_with_retry(url, headers, body, max_retries=3):
    for attempt in range(max_retries):
        try:
            r = requests.post(url, headers=headers, json=body, timeout=10)
            r.raise_for_status()
            return r.json()
        except requests.exceptions.Timeout:
            if attempt == max_retries - 1:
                raise
            wait = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait)
        except requests.exceptions.HTTPError as e:
            if 400 <= e.response.status_code < 500:
                raise  # 4xx 不重试
            if e.response.status_code == 429:
                retry_after = int(e.response.headers.get("Retry-After", 1))
                time.sleep(retry_after)
                continue
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)

2. Circuit breaker

class CircuitBreaker:
    def __init__(self, failure_threshold=10, cooldown=30):
        self.failure_count = 0
        self.last_failure_time = 0
        self.failure_threshold = failure_threshold
        self.cooldown = cooldown
        self.state = "CLOSED"  # CLOSED / OPEN / HALF_OPEN

    def call(self, func, *args, **kwargs):
        if self.state == "OPEN":
            if time.time() - self.last_failure_time > self.cooldown:
                self.state = "HALF_OPEN"
            else:
                raise Exception("Circuit open")

        try:
            result = func(*args, **kwargs)
            if self.state == "HALF_OPEN":
                self.state = "CLOSED"
                self.failure_count = 0
            return result
        except Exception:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.failure_threshold:
                self.state = "OPEN"
            raise

3. Fallback to alternative

def fetch_serp_with_fallback(query, params):
    try:
        return requests.post(SERPBASE_URL, json=params, headers=headers, timeout=10).json()
    except Exception as e:
        log_warning(f"SerpBase failed: {e}, fallback to cache")
        return get_cached_or_stale(query)  # 降级到缓存

4. Rate limit aware(看 429 头)

def fetch_serp_rate_aware(url, headers, body, max_retries=3):
    for attempt in range(max_retries):
        try:
            r = requests.post(url, headers=headers, json=body, timeout=10)
            r.raise_for_status()
            return r.json()
        except requests.exceptions.HTTPError as e:
            if e.response.status_code == 429:
                retry_after = int(e.response.headers.get("Retry-After", 1))
                # 利用 retry-after 时间,降级到缓存
                if retry_after > 60:
                    log_warning(f"Long rate limit: {retry_after}s, use cache")
                    return get_cached_or_stale(body.get("q"))
                time.sleep(retry_after)
                continue
            raise

5. Graceful degradation

def fetch_serp_or_fail(query, params, max_age=3600):
    try:
        return requests.post(SERPBASE_URL, json=params, headers=headers, timeout=10).json()
    except Exception:
        # 失败时降级到旧缓存
        cached = redis.get(f"serp:fallback:{hash(query)}")
        if cached:
            data = json.loads(cached)
            if (time.time() - data["ts"]) < max_age:
                return data["response"]
        # 缓存也过期,返 None 让上层用默认
        return None

5 模式组合使用

def robust_fetch(query, params):
    breaker = CircuitBreaker(failure_threshold=5, cooldown=30)

    for attempt in range(3):
        try:
            return breaker.call(
                fetch_serp_with_retry, 
                SERPBASE_URL, headers, params
            )
        except RateLimitError:
            time.sleep(60)
            continue
        except CircuitOpenError:
            # 熔断 → 降级
            return fetch_serp_fallback(query)
        except Exception:
            # 其他错 → 降级
            return fetch_serp_graceful_degradation(query, params)

90 天实测

错误类型	次数	retry 成功	circuit breaker 触发	fallback 生效
网络抖动	35	35(100%)	0	0
vendor 5xx	12	8(67%)	1 次	4 次
QPS 超限	8	6(75%)	0	2 次
4xx 客户端错	3	0(不重试)	0	0
总	58	49(85%)	1 次	6 次

85% 错误通过 retry 解决,15% 触发 fallback,实际真实失败 0。

4 个关键设计点

1. 4xx 不重试

# 错:任何错都重试
# 对:4xx(客户端错)不重试,5xx / 超时重试

2. retry 加 jitter

# 错:固定 1 秒退避
# 对:指数 + 随机 jitter,避免雪崩
wait = (2 ** attempt) + random.uniform(0, 1)

3. circuit breaker 不要太敏感

# 错:1 个失败就 open
# 对:5-10 个连续失败才 open

4. fallback 要 stale-while-revalidate

# 错:失败就返 None
# 对:失败时返旧数据 + 标记 stale,下次重新拉

与其他模式组合

• retry + circuit breaker:防止雪崩
• fallback + 降级:保证可用性
• rate limit aware:不浪费 retry 配额
• monitor + alert:长期观察模式

监控告警

指标	阈值	告警
失败率	> 1% / 5 分钟	Slack
circuit breaker open 次数	> 1 / 小时	Slack + oncall
fallback 触发率	> 5% / 5 分钟	Slack
auto-refund 率	> 5% / 1 小时	Slack(可能 vendor 故障)

100 次免费试用:serpbase.dev 注册,跑 1 周看自己的错误模式,再选 5 模式组合。