【爬虫逆向源码分析001】深入研究中国大学慕课的源代码，中国大学视频课件下载的逆向爬虫分析～
一、前言
中国大学MOOC（icourse163.org）是国内最大的慕课平台之一，汇聚了数百所高校的优质课程资源。然而，平台Web端仅支持PDF课件直接下载，视频内容不支持离线保存。对于网络条件不佳或需要反复观看的学习者来说，离线下载需求真实存在。
本文基于学无止下载器（xuewuzhi.cn）提供的源码分析框架，结合多个开源项目的逆向实践经验，系统梳理中国大学MOOC视频课件下载的技术原理，重点分析课程目录获取、视频签名生成、m3u8流下载等核心环节的逆向思路，供爬虫逆向技术爱好者学习交流。
二、整体下载框架设计
在深入具体接口之前，先理解一个完整的下载器需要解决哪些工程问题。学无止下载器源码（icourse163_pipeline.py）展示了一个规范的下载框架设计，虽然不是直接可用的平台接口实现，但其工程思想值得借鉴：
批量入口去重：将多个课程入口整理为批次，重复课程只进入一次，每个课程保留自己的章节层级和输出目录。
课程资源隔离：不同课程可能使用相同的课节ID，需用平台和课程范围共同标识资源，避免文件覆盖。
可恢复的文件写入：使用部分文件和修订记录恢复进度，检查文件长度与SHA-256哈希值，通过校验后原子替换正式文件。
任务依赖与有界并发：执行前检查重名路径、缺失依赖和循环依赖，只安排有限数量的任务并发执行。
进度记录、取消与重试：由调度器统一保存任务状态，暂时读取失败时退避重试，磁盘错误直接记录。
这套框架采用Task命名元组来抽象单个下载任务，包含key、source_id、path、size、digest、after等字段，通过ThreadPoolExecutor实现有界并发。核心思路是将"资源发现"与"文件传输"解耦——爬虫只负责获取资源的元数据（ID、签名、大小），真正的文件下载交给独立的下载引擎处理。
三、核心逆向环节分析
3.1 课程目录获取——DWR接口
中国大学MOOC的课程目录数据通过DWR（Direct Web Remoting）接口返回。DWR是一种Java服务端远程调用框架，请求格式和响应格式都有固定规范。
核心接口有两个：
获取期次课程目录：向CourseBean.getLastLearnedMocTermDto.dwr发送POST请求，传入termId参数，返回该期次下所有章节、课时的结构化数据，包括每节课的contentId和s.x.id（资源标识）。
获取课时详情：向CourseBean.getLessonUnitLearnVo.dwr发送POST请求，传入contentId和contentType，返回该课时的具体资源信息——对于视频返回videoId，对于PDF返回下载链接。
请求Payload中需要包含scriptSessionId、c0-scriptName、c0-methodName等DWR协议参数，这些参数在浏览器F12的Network面板中可以完整获取。
termId的提取相对简单，课程主页HTML中通常包含id : "数字"格式的数据，用正则表达式即可提取。
3.2 视频签名生成——getResourceToken
获取到videoId后，还不能直接拿到m3u8下载地址。中国大学MOOC的视频接口需要videoId + signature两个参数配合才能访问，且两者必须对应，否则无法访问。
signature的生成需要经过一个关键步骤——调用getResourceToken接口：
POST https://www.icourse163.org/web/j/resourceRpcBean.getResourceToken.rpc?csrfKey=xxx
Body: bizId={contentId}&bizType=1&contentType=1其中csrfKey来自登录后Cookie中的NTESSTUDYSI值，bizId对应课时的contentId，bizType=1和contentType=1为固定值。
这个接口返回的signature是一个经过加密的字符串，其生成逻辑在JavaScript中实现。实际的签名算法涉及时间戳、随机数、业务参数等元素的组合加密，不同时期平台可能调整算法，这也是下载器需要持续维护的根本原因。
3.3 视频流下载——m3u8与TS分片
拿到videoId和signature后，访问视频信息接口：
GET https://vod.study.163.com/eds/api/v1/vod/video?videoId={videoId}&signature={signature}&clientType=1该接口返回JSON格式的视频信息，其中包含多个清晰度级别的m3u8下载地址（标清、高清、超清等），每个地址对应一个m3u8索引文件。
m3u8文件本身是文本格式，内容为一系列TS分片的播放列表，每个TS分片时长约10秒。下载流程为：解析m3u8获取所有TS分片URL → 逐个下载TS分片 → 用FFmpeg合并为完整MP4文件。
需要注意的是，该接口要求使用HTTP/2协议，使用普通的requests库可能被拒绝，需要借助httpx等支持HTTP/2的客户端。
3.4 反爬与加密机制
中国大学MOOC在反爬层面做了多层防护：
调试器检测：打开开发者工具时会触发debugger死循环，需要停用断点才能正常调试。这是一种常见的JavaScript反调试手段。
模拟点击检测：平台会检测页面是否存在模拟点击类脚本的行为。
付费内容加密：付费课程的TS分片经过AES加密，m3u8文件中包含EXT-X-KEY标签指向密钥文件。即使下载了TS分片，没有密钥也无法播放。免费课程一般未加密，下载后可直接播放。
Cookie鉴权：所有接口请求都需要携带登录后的Cookie，包括NTESSTUDYSI、STUDY_SESS等关键字段，且平台会验证账号是否已参加该课程。
四、工程实现要点
一个稳定可用的下载器，除了接口逆向，还需要处理好以下工程问题：
断点续传：每个TS分片独立下载，记录已完成分片，中断后重新运行只下载未完成部分。学无止框架通过部分文件和SHA-256校验实现精确的进度恢复。
文件命名安全：课程标题和章节名称可能包含Windows不允许的字符（<>:"/\|?*），需要过滤和截断处理。学无止源码中的safe_name函数将非法字符替换为下划线，并检查保留文件名（CON、PRN、AUX等）。
多线程下载：TS分片数量可能上百个，串行下载效率低下。SigureMo的mooc-dl项目默认使用16个线程并发下载。但需注意控制并发数量，避免触发平台限流。
清晰的课程目录树：下载后的文件按"课程名/章/节/课时"的层级结构组织，方便离线观看时快速定位。mooc-dl项目支持通过file_path_template自定义路径模板。
# Author: Xuewuzhi
# Source: https://xuewuzhi.cn/icourse163_downloader#source-analysis
# from xuewuzhi.cn
# Python 3.5+; offline teaching example; no real media transport.
PLATFORM = {'client_classes': ['Icourse163_Batch'],
'client_files': ['Mooc/Courses/Course_Batch.py',
'Mooc/Courses/Mooc163/Icourse163/Icourse163_Base.py',
'Mooc/Courses/Mooc163/Icourse163/Icourse163_Batch.py',
'Mooc/Courses/Mooc163/Mooc163_Base.py'],
'family': 'batch',
'resource_lists': [],
'slug': 'icourse163'}
"""Offline task framework; adapters provide immutable, already-prepared bytes.
This module contains no network transport, platform authorization or decryption.
The data contract below is a teaching model, not a platform API response.
"""
from collections import Counter, deque, namedtuple
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
from hashlib import sha256
from pathlib import Path
from tempfile import TemporaryDirectory
import json
import os
import re
import threading
import time
Task = namedtuple('Task', 'key source_id path size digest after')
Task.__new__.__defaults__ = ((),)
Result = namedtuple('Result', 'key state detail')
SUCCESS = frozenset(('saved', 'skipped'))
class CatalogError(ValueError):
"""A catalog is incomplete, cyclic or internally inconsistent."""
class RetryableReadError(IOError):
"""A temporary interruption; the next attempt may resume saved bytes."""
class IntegrityError(ValueError):
"""The source does not match its declared immutable revision."""
class Cancelled(Exception):
"""Cooperative cancellation preserves unfinished work for a later run."""
def stable_key(*parts):
data = json.dumps(parts, ensure_ascii=True, separators=(',', ':'))
return sha256(data.encode('ascii')).hexdigest()
def safe_name(value):
name = re.sub(r'[<>:"/\\|?*\x00-\x1f]+', '_', str(value))
name = name.strip(' .')[:64].rstrip(' .')
if not name or name in ('.', '..'):
return 'untitled'
if name.split('.')[0].upper() in ('CON', 'PRN', 'AUX', 'NUL') or re.match(r'^(COM|LPT)[1-9](\.|$)', name, re.I):
name = '_' + name
return name
def atomic_json(path, data):
"""Publish state only after the replacement file has reached the disk."""
temporary = path.with_name(path.name + '.tmp')
with temporary.open('w', encoding='utf-8') as stream:
json.dump(data, stream, ensure_ascii=True, sort_keys=True, indent=2)
stream.flush()
os.fsync(stream.fileno())
os.replace(str(temporary), str(path))
class Journal:
"""One coordinator owns this journal; task workers never write it.
Completion records are observations, not proof that an output still exists.
Every run revalidates files, even when the journal says they were saved.
"""
def __init__(self, path):
self.path = Path(path)
self.path.parent.mkdir(parents=True, exist_ok=True)
self.data = {'schema': 1, 'tasks': {}}
if self.path.exists():
with self.path.open(encoding='utf-8') as stream:
data = json.load(stream)
if not isinstance(data, dict) or data.get('schema') != 1 or not isinstance(data.get('tasks'), dict):
raise ValueError('Invalid checkpoint journal')
self.data = data
def record(self, task, state, detail=''):
previous = self.data['tasks'].get(task.key, {})
self.data['tasks'][task.key] = {
'state': state, 'detail': detail, 'size': task.size,
'sha256': task.digest,
'runs': previous.get('runs', 0) + (1 if state == 'running' else 0),
}
atomic_json(self.path, self.data)
def build_plan(course, destination, selected_ids=None):
"""Assign paths before filtering; selection never changes lesson numbers."""
root = Path(os.path.realpath(str(destination)))
tasks, seen = [], {}
for chapter in course['chapters']:
for lesson in chapter['lessons']:
if not lesson['accessible']:
continue
for resource in lesson['resources']:
if selected_ids is not None and resource['selector_id'] not in selected_ids:
continue
identity = (course['app_id'], course['id'], resource['selector_id'], resource['revision'])
size, digest = resource['size'], resource['sha256']
if type(size) is not int or size < 0:
raise ValueError('Invalid resource size')
if not isinstance(digest, str) or not re.fullmatch(r'[0-9a-f]{64}', digest):
raise ValueError('Invalid resource checksum')
if identity in seen:
if seen[identity] != (size, digest):
raise ValueError('Conflicting resource metadata')
continue
seen[identity] = (size, digest)
folder = root / resource['output_root']
for name in chapter['folders']:
folder = folder / name
target = Path(os.path.realpath(str(folder / (resource['name'] + '.demo'))))
if root not in target.parents:
raise ValueError('Destination escapes the output directory')
tasks.append(Task(stable_key(*identity), (resource['app_id'], resource['id']), target, size, digest))
return tasks
def is_complete(path, task):
if not path.is_file() or path.stat().st_size != task.size:
return False
checksum = sha256()
with path.open('rb') as stream:
for chunk in iter(lambda: stream.read(64 * 1024), b''):
checksum.update(chunk)
return checksum.hexdigest() == task.digest
def remove_partial(path):
if path.exists():
path.unlink()
def check_cancel(stop):
if stop is not None and stop.is_set():
raise Cancelled('Run was cancelled')
def transfer(task, source, attempts=3, pause=time.sleep, stop=None):
"""Resume a revision, verify the whole file, then replace the destination.
Source.chunks(task, offset) must begin at exactly offset in the manifest's
immutable resource revision. A real transport would need to validate range
and revision responses. This example deliberately has no such transport.
"""
if attempts < 1:
raise ValueError('At least one attempt is required')
check_cancel(stop)
if is_complete(task.path, task):
return Result(task.key, 'skipped', str(task.path))
task.path.parent.mkdir(parents=True, exist_ok=True)
partial = task.path.with_name(task.path.name + '.part')
metadata = partial.with_name(partial.name + '.json')
identity = {'key': task.key, 'size': task.size, 'sha256': task.digest}
if metadata.exists():
try:
with metadata.open(encoding='utf-8') as stream:
previous = json.load(stream)
except ValueError:
previous = None
if previous != identity:
remove_partial(partial)
# A partial without metadata is still checked against the final digest.
atomic_json(metadata, identity)
for attempt in range(attempts):
try:
check_cancel(stop)
offset = partial.stat().st_size if partial.exists() else 0
if offset > task.size:
remove_partial(partial)
offset = 0
if offset < task.size:
with partial.open('ab') as output:
for chunk in source.chunks(task, offset):
check_cancel(stop)
if not chunk:
continue
if offset + len(chunk) > task.size:
raise IntegrityError('Source exceeded the expected size')
output.write(chunk)
offset += len(chunk)
output.flush()
os.fsync(output.fileno())
if offset != task.size:
raise RetryableReadError('Source ended before the expected size')
elif not partial.exists():
partial.touch()
check_cancel(stop)
if not is_complete(partial, task):
raise IntegrityError('Checksum mismatch; restart from byte zero')
partial.replace(task.path)
remove_partial(metadata)
return Result(task.key, 'saved', str(task.path))
except (RetryableReadError, IntegrityError) as error:
if isinstance(error, IntegrityError):
remove_partial(partial)
if attempt + 1 == attempts:
raise
delay = min(0.25 * (2 ** attempt), 2.0)
if stop is None:
pause(delay)
elif stop.wait(delay):
raise Cancelled('Cancelled during retry delay')
# Disk errors and programming errors are terminal, not read retries.
def validate_plan(tasks):
"""Reject collisions, missing prerequisites and cycles before writing files."""
by_key = {task.key: task for task in tasks}
paths = [os.path.realpath(str(task.path)).casefold() for task in tasks]
if len(by_key) != len(tasks) or len(set(paths)) != len(paths):
raise ValueError('Duplicate task identity or destination')
children = {key: [] for key in by_key}
degree = {}
for task in tasks:
if len(set(task.after)) != len(task.after):
raise ValueError('Duplicate prerequisite')
degree[task.key] = len(task.after)
for dependency in task.after:
if dependency not in by_key:
raise ValueError('Missing prerequisite')
children[dependency].append(task.key)
queue = deque(key for key in by_key if degree[key] == 0)
count = 0
while queue:
key = queue.popleft()
count += 1
for child in children[key]:
degree[child] -= 1
if degree[child] == 0:
queue.append(child)
if count != len(tasks):
raise ValueError('Cyclic task dependencies')
return by_key, children
def run_plan(tasks, source, workers=3, journal=None, stop=None):
"""Bound queued work, isolate failures and run dependents only after success."""
by_key, children = validate_plan(tasks)
degree = {task.key: len(task.after) for task in tasks}
ready = deque(task.key for task in tasks if not task.after)
results, pending = {}, {}
limit = max(1, min(workers, 4))
def finish(task, result):
results[task.key] = result
if journal is not None:
journal.record(task, result.state, result.detail)
for child in children[task.key]:
degree[child] -= 1
if degree[child] == 0:
ready.append(child)
with ThreadPoolExecutor(max_workers=limit) as pool:
while ready or pending:
while ready and len(pending) < limit:
task = by_key[ready.popleft()]
if stop is not None and stop.is_set():
finish(task, Result(task.key, 'cancelled', 'Run was cancelled'))
elif any(results[key].state not in SUCCESS for key in task.after):
finish(task, Result(task.key, 'blocked', 'A prerequisite did not finish'))
else:
if journal is not None:
journal.record(task, 'running')
future = pool.submit(transfer, task, source, stop=stop)
pending[future] = task
if pending:
done, _ = wait(pending, return_when=FIRST_COMPLETED)
for future in done:
task = pending.pop(future)
try:
result = future.result()
except Cancelled:
result = Result(task.key, 'cancelled', 'Run was cancelled')
except Exception as error:
result = Result(task.key, 'failed', type(error).__name__)
finish(task, result)
return [results[task.key] for task in tasks]
class MemorySource:
"""Only local fixture bytes; never resolves URLs or reads platform sessions."""
def __init__(self, blobs, chunk_size=8):
if chunk_size < 1:
raise ValueError('Chunk size must be positive')
self.blobs = blobs
self.chunk_size = chunk_size
def chunks(self, task, offset):
data = self.blobs[task.source_id]
for start in range(offset, len(data), self.chunk_size):
yield data[start:start + self.chunk_size]
def append_library_index(tasks, source, destination):
"""Publish a library index only after every selected resource has succeeded."""
root = Path(destination)
records = [{'path': task.path.relative_to(root).as_posix(), 'sha256': task.digest}
for task in tasks]
payload = json.dumps(records, sort_keys=True, indent=2).encode('ascii')
digest = sha256(payload).hexdigest()
source_id = ('demo-library', digest)
source.blobs[source_id] = payload
index = Task(stable_key('library-index', digest), source_id, root / 'library-index.demo',
len(payload), digest, tuple(task.key for task in tasks))
return tasks + [index]
def main():
course, source = demo_course()
with TemporaryDirectory(prefix='xuewuzhi-demo-') as destination:
tasks = build_plan(course, destination)
tasks = append_library_index(tasks, source, destination)
for task in tasks:
print(task.path.relative_to(destination).as_posix())
first = tasks[0]
first.path.parent.mkdir(parents=True, exist_ok=True)
first.path.with_name(first.path.name + '.part').write_bytes(source.blobs[first.source_id][:7])
for run_number in (1, 2):
# Reload state as a newly started process would; verify files again.
journal = Journal(Path(destination) / 'checkpoint.json')
report = run_plan(tasks, source, journal=journal)
counts = Counter(item.state for item in report)
print('Run {}: saved={}, skipped={}, failed={}, blocked={}, cancelled={}'.format(
run_number, counts['saved'], counts['skipped'], counts['failed'],
counts['blocked'], counts['cancelled']))
print('Checkpoint: {} tracked tasks'.format(len(journal.data['tasks'])))
# The temporary directory is removed after this offline demonstration.
# Normalize prepared catalog data; these are not platform endpoint schemas.
LIST_KINDS = {'video_list': 'video', 'audio_list': 'audio', 'pdf_list': 'document',
'ppt_list': 'document', 'doc_list': 'document', 'file_list': 'document',
'attach_list': 'document', 'html_list': 'article', 'text_list': 'article',
'sub_list': 'subtitle', 'practice_list': 'practice', 'clock_list': 'practice'}
def collect_pages(fetch, limit=100):
"""Finish explicit pagination; a short page may still have a successor."""
result, identities, signatures = [], {}, set()
for number in range(1, limit + 1):
page = fetch(number)
rows = page.get('items')
if not isinstance(rows, list) or type(page.get('has_more')) is not bool:
raise CatalogError('Page requires items and an explicit continuation flag')
signature = tuple((row['scope'], row['id']) for row in rows)
if not rows and page['has_more']:
raise CatalogError('Empty page declares more results')
if rows and signature in signatures:
raise CatalogError('Repeated page')
signatures.add(signature)
for row in rows:
identity = (row['scope'], row['id'])
if identity in identities:
if identities[identity] != row:
raise CatalogError('Conflicting metadata for one catalog identity')
else:
identities[identity] = row
result.append(dict(row))
if not page['has_more']:
return result
raise CatalogError('Pagination limit reached')
def fixture(blobs, identifier, title, kind, scope=None, payload=None):
scope = scope or PLATFORM['slug']
data = payload if payload is not None else ('Offline sample: ' + scope + '/' + identifier + '\n').encode('ascii') * 2
blobs[(scope, identifier)] = data
return {'id': identifier, 'scope': scope, 'title': title, 'kind': kind,
'revision': 'demo-v1', 'size': len(data), 'sha256': sha256(data).hexdigest()}
def normalize_tree(tree):
"""Support resource-list dictionaries and typed rows through one model.
Group position belongs to the complete catalog. Filtering inaccessible rows
or empty groups must not renumber the remaining chapters or lesson names.
"""
chapters = []
def walk(group, indexes=(), folders=(), trail=()):
identity = (group.get('scope', PLATFORM['slug']), group['id'])
if identity in trail or len(trail) >= 16:
raise CatalogError('Cyclic or excessively deep catalog')
resources = list(group.get('resources', []))
for key, kind in sorted(LIST_KINDS.items()):
resources.extend(dict(row, kind=kind, attachment=key in ('file_list', 'attach_list'))
for row in group.get(key, []))
lessons, seen = [], {}
counters = Counter()
for row in resources:
kind = row['kind']
if kind not in set(LIST_KINDS.values()) | {'live'}:
raise CatalogError('Unsupported normalized resource kind')
category = 'files' if row.get('attachment') else 'course'
counter = 'attachment' if category == 'files' else 'media' if kind in ('video', 'audio', 'live') else 'document'
resource_id = (row['scope'], kind, row['id'], category)
if resource_id in seen:
if seen[resource_id] != row:
raise CatalogError('Conflicting catalog occurrence')
continue
seen[resource_id] = row
counters[counter] += 1
if row.get('accessible') is False:
continue
sequence = '.'.join(map(str, indexes + (counters[counter],)))
brackets = '[]' if counter == 'media' else '##' if kind == 'article' else '()'
name = brackets[0] + sequence + brackets[1] + '--' + safe_name(row['title'])
selected = stable_key(tree['id'], identity, indexes, resource_id)
resource = dict(row, app_id=row['scope'], selector_id=selected,
output_root=category, name=name)
lessons.append({'title': row['title'], 'accessible': True, 'resources': [resource]})
if lessons:
chapters.append({'title': group['title'], 'folders': folders, 'lessons': lessons})
for position, child in enumerate(group.get('children', []), 1):
child_indexes = indexes + (position,)
prefix = '.'.join(map(str, child_indexes))
folder = '{' + prefix + '}--' + safe_name(child['title'])
walk(child, child_indexes, folders + (folder,), trail + (identity,))
walk(tree)
return {'id': tree['id'], 'app_id': PLATFORM['slug'], 'chapters': chapters}
def root_group(children):
return {'id': PLATFORM['slug'] + '-selection', 'title': 'Selected content', 'children': children}
def demo_course():
"""Different courses may reuse a lesson ID; scope every resource by course."""
blobs, groups, seen = {}, [], set()
entries = [('course-a', 'Course A'), ('course-b', 'Course B'), ('course-a', 'Course A')]
for course_id, title in entries:
identity = (PLATFORM['slug'], course_id)
if identity in seen:
continue
seen.add(identity)
scope = PLATFORM['slug'] + '/' + course_id
chapter = {'id': course_id + '-chapter', 'title': 'Chapter one', 'resources': [
fixture(blobs, 'lesson-01', 'Lecture', 'video', scope),
fixture(blobs, 'handout-01', 'Handout', 'document', scope),
]}
groups.append({'id': course_id, 'scope': scope, 'title': title, 'children': [chapter]})
return normalize_tree(root_group(groups)), MemorySource(blobs)
if __name__ == '__main__':
main()五、总结
中国大学MOOC的逆向下载涉及DWR接口调用、签名参数生成、m3u8流解析等多个技术环节。核心难点在于getResourceToken接口的签名算法逆向，以及HTTP/2协议适配。整体逆向思路可以概括为：
-
通过DWR接口获取课程目录和课时ID
-
利用csrfKey和contentId调用getResourceToken获取signature
-
用videoId+signature请求视频信息接口获取m3u8地址
-
解析m3u8并下载TS分片，FFmpeg合并输出
从工程角度看，一个成熟的下载器还需要在文件安全、断点续传、并发控制、目录组织等方面做好设计。学无止下载器提供的离线框架展示了良好的工程实践——将资源发现与文件传输解耦，通过哈希校验保证完整性，用有界并发控制资源消耗。
需要强调的是，本文的技术分析仅用于学习爬虫逆向与HTTP协议原理。下载的课程内容请仅用于个人学习，遵守平台的版权规定，不得用于商业用途或二次分发。
参考项目：
-
SigureMo/mooc-dl：https://github.com/SigureMo/mooc-dl
-
aialin/dl_icourse163：https://github.com/aialin/dl_icourse163
-
MoocDownloader（.NET版）：https://github.com/MoocDownloader/MoocDownloader
浙公网安备 33010602011771号