DeepSeek-实战-全-

DeepSeek 实战（全）

原文：DeepSeek in Practice

译者：飞龙

协议：CC BY-NC-SA 4.0

前言

大型语言模型 (LLM) 领域正以前所未有的速度飞速发展。当我们撰写本文时，DeepSeek 团队刚刚发布了一篇新论文，展示了如何利用计算机视觉技术压缩 LLM 的上下文。人工智能领域的活跃程度令人惊叹，且没有放缓的迹象。

作为本书的作者，我们曾思考，关于 DeepSeek，我们能写出什么内容而不会使本书立即过时呢？ 我们的答案很简单：我们相信 AI 技术的未来是开放的，而这场运动的先锋就是 DeepSeek。

DeepSeek 不断给人工智能界带来惊喜，从发布功能极其强大的模型到发表专注于 LLM 的突破性研究。考虑到这些发展，我们清楚地认识到，我们需要写一本书来记录正在发生的事情。

本书的目标是为你——亲爱的读者——提供掌握 DeepSeek 所需的工具。我们希望它能成为你精通开源语言模型的指南。

DeepSeek 实践指南

概述

本书涵盖了使用 DeepSeek 模型时可能遇到的几乎所有内容——从理解 DeepSeek 模型的独特之处，到如何将其用于实际应用。我们还深入探讨了如何精通 DeepSeek 模型的提示工程，以便您成为与该系列模型交互的专家。最后，我们深入实践层面，提供了如何设计、构建和部署由 DeepSeek 模型驱动的智能体（Agentic）与非智能体应用的示例。

在本书中，我们首先介绍 DeepSeek，探讨其基础，并理解它与其他模型的差异（第 1 章和第 2 章）。一旦您了解了 DeepSeek 的能力，接下来就是学习如何有效地使用它。我们有一整章专门讲解推理模型的提示艺术，向您展示“少即是多”（第 3 章）。

然后，我们进入实际应用，演示如何使用它对复杂问题进行顾问级分析（第 4 章）。在此基础上，我们构建了一个完整的端到端应用，您将从一个简单的 API 开始，直到在 Amazon Web Services 上运行一个完全容器化的服务（第 5 章）。

在本书的最后一部分，我们探索大语言模型和 AI 技术的前沿。您将学习如何使用 DeepSeek 模型作为智能体应用的骨干（第 6 章）。我们还将引导您完成一个更复杂的 MLOps 案例，您将使用更强大的 DeepSeek 模型将知识蒸馏到一个更小的模型中（第 7 章）。在本书的最后一章，我们涵盖了 DeepSeek 模型的部署及其多种权衡，以便您能为自己的用例选择最佳的部署方法（第 8 章）。

本书以结语部分收尾，带您回顾关键要点。本书还提供了一个附录，指导您使用 DeepSeek 的多种方式。

本书的目标读者

本书面向任何想要了解 DeepSeek 模型工作原理的人。无论您是 AI 工程师、软件开发者、机器学习工程师还是研究员——只要您熟悉 Python，就能从本书中获益。如果您有数据科学、API 和一些机器学习概念的经验，您将获益更多！

本书涵盖的内容

第 1 章，什么是 DeepSeek，介绍了 DeepSeek 这一开源大语言模型，探讨了其突破、发展、与其他大语言模型的比较，以及对当今全球 AI 领域的影响。

第 2 章，深入解析 DeepSeek，深入探讨了 DeepSeek 的架构、推理机制、高级能力，提供了对其独特优势和新兴争议的实践理解。

第 3 章，提示 DeepSeek，向您展示如何有效地为 DeepSeek 模型构建提示，并特别展示了如何在 V 系列和 R 系列模型之间实现这种差异化。

第 4 章，使用 DeepSeek：案例研究，展示了真实的 DeepSeek 案例研究，揭示了行业领导者如何应用它来提升生产力、改进开发并使用实用方法实施 AI 辅助工作流程。

第 5 章，使用 DeepSeek 构建，带您了解如何创建由 DeepSeek 模型支持的真实世界应用，如何迭代改进它，以及如何将 DeepSeek 模型部署到 AWS 作为独立服务。

第 6 章，使用 DeepSeek 构建智能体，向您展示如何开始使用 DeepSeek 模型构建智能体，存在的不同智能体模式，以及对 MCP 的简要介绍。

第 7 章，使用 DeepSeek 驱动的 Gemma 3 法律推理微调，涵盖了将 DeepSeek 的 R1 蒸馏到经过微调的 Gemma 3 模型中的过程。我们探索了一个法律案例，并讨论了在生产场景中如何对其进行评估。

第 8 章，部署 DeepSeek 模型，展示了如何为生产用例部署 DeepSeek 模型。我们向您展示如何在云基础设施上启动全规模模型，并指导您如何将其作为生产技术栈的一部分进行管理。

第 9 章，结语，回顾了您为理解 DeepSeek 内部原理、从应用到部署的有效使用方法所经历的旅程。

第 10 章，附录，包含一份 DeepSeek 备忘单，向您展示如何快速入门。

如何充分利用本书

具备使用 Python、API 以及 Ollama 或 llama.cpp 等工具的实践经验。
具备使用 uv 和 Docker 等工具和平台的经验。
扎实的机器学习概念理解。

声明

作者承认使用了尖端 AI（如 ChatGPT），唯一目的是增强本书的语言和清晰度，从而确保为读者提供流畅的阅读体验。

下载示例代码文件

本书的代码包托管在 GitHub 上：github.com/PacktPublishing/DeepSeek-in-Practice。我们还从丰富的图书和视频目录中提供了其他代码包，地址为：github.com/PacktPublishing。请查看！

下载彩色图片

我们还提供了一个 PDF 文件，其中包含本书中使用的截图/图表的彩色图片。您可以在此处下载：packt.link/gbp/9781806020850。

本书中的约定

本书中使用了许多文本约定。

CodeInText：表示文本中的代码词、数据库表名、文件夹名、文件名、文件扩展名、路径名、虚拟 URL、用户输入和 Twitter 句柄。例如：“执行 python-garminconnect 命令：”

代码块设置如下：

### 我们将值作为环境变量获取
GARMIN_EMAIL = os.getenv("GARMIN_EMAIL") # 你的邮箱
GARMIN_PASSWORD = os.getenv("GARMIN_PASSWORD") # 你的密码

当我们希望将注意力吸引到代码块的特定部分时，相关行或项目用粗体表示：

tools = [{
    "type": "function",
    "function": {
        "name": "analyze_sentiment",
        "description": "Analyze the sentiment of text",
        "parameters": {
            "type": "object",
            "properties": {
                "text": {"type": "string"},
                "confidence": {"type": "number",
                               "minimum": 0, "maximum": 1}
            },
            "required": ["text", "confidence"]
        }
    }
}]

任何命令行输入或输出均按以下格式编写：

git clone https://github.com/PacktPublishing/DeepSeek-in-Practice.git
cd Chapter03
pip install -r requirements.txt

粗体：表示新术语、重要单词或屏幕上看到的单词。例如，菜单或对话框中的单词在文本中如此显示。例如：“通过 Settings 菜单访问 Cursor 模型配置。”

警告或重要注意事项会以这种样式出现。

提示和技巧会以这种样式出现。

联系我们

我们随时欢迎读者的反馈。

一般反馈：如果您对本书的任何方面有疑问，或有任何一般反馈，请通过电子邮件联系我们：customercare@packt.com，并在邮件主题中注明本书的书名。

勘误：尽管我们已尽一切努力确保内容准确，但错误仍在所难免。如果您在本书中发现任何错误，请通过 www.packt.com/submit-errata 报告给我们。请点击 提交勘误 并填写表格，我们将不胜感激。

盗版：如果您在互联网上以任何形式发现我们作品的任何非法拷贝，请向我们提供地址或网站名称，我们将不胜感激。请通过 copyright@packt.com 联系我们并提供相关材料的链接。

如果您有兴趣成为作者：如果您对某个主题有专业知识，并且有兴趣撰写或参与一本书，请访问 authors.packt.com/。

分享您的想法

阅读完 《DeepSeek 实践指南》 后，我们很乐意听到您的想法！扫描下方二维码，直达本书的亚马逊评论页面，分享您的反馈。

packt.link/r/180602084X

您的评论对我们和技术社区都很重要，将帮助我们确保提供高质量的内容。

书籍附带的免费福利

本书附带支持您学习的免费福利。立即激活以获得即时访问权限（请参阅“如何解锁”部分获取说明）。

以下是您购买后可以立即解锁的内容概览：

| PDF 和 ePub 副本 | 下一代基于 Web 的阅读器 |

| :--- | :--- |

| | |

如何解锁

扫描二维码（或访问 packtpub.com/unlock）。按书名搜索本书，确认版本，然后按照页面上的步骤操作。 | |

注意：请保留您的发票。直接从 Packt 购买的不需要发票。

敬请关注

要随时了解生成式 AI 和大语言模型（LLMs）领域的最新动态，请订阅我们的周报 AI_Distilled，访问 packt.link/8Oz6Y。

加入我们的 Discord 和 Reddit 社区

对本书有疑问或想参与关于生成式 AI 和大语言模型（LLMs）的讨论？

加入我们的 Discord 服务器 packt.link/4Bbd9 和 Reddit 频道 packt.link/wcYOQ，与志同道合的爱好者联系、分享和协作。

| | |

第一部分

理解与探索 DeepSeek

在本书的第一部分，我们将为理解 DeepSeek 及其在快速演变的 AI 世界中的角色奠定坚实的基础。我们将从介绍 DeepSeek 作为一种开源大语言模型开始，探讨它为何能获得全球关注。接下来，我们将深入研究其内部架构、推理机制和高级能力，以揭示其真正独特之处。我们还将探索有效的提示策略，帮助您充分利用 DeepSeek 模型。

学完本部分后，您将具备所需的背景知识、技术理解和实践见解，从而在现代 AI 工作流中自信地使用 DeepSeek。

本书本部分包含以下章节：

第 1 章 什么是 DeepSeek
第 2 章 深入探讨 DeepSeek
第 3 章 提示 DeepSeek

1 什么是 DeepSeek？

人工智能（AI）正在快速发展，随之而来的是一套工具，允许开发者、研究人员和创新者构建更智能、更自适应的系统。其中一个新兴的工具就是 DeepSeek：一个强大的、开源的大语言模型（LLM），旨在与 GPT-4 和 LLaMA 等主要大语言模型（LLM）的功能相媲美。但 DeepSeek 到底是什么，为什么我们应该关注它？

在本章中，我们将深入探讨 DeepSeek 是什么，它如何融入更广泛的 AI 领域，以及为什么它在整个科技行业引起如此兴趣。您将了解 DeepSeek 的独特功能，以及它在训练数据、效率和性能基准方面与其他模型的比较。

学完本章后，您将能够理解 DeepSeek 的发展及其成功的关键贡献者。

在本章中，我们将涵盖以下主要主题：

介绍 DeepSeek
理解 DeepSeek 的技术突破
对全球 AI 生态系统的影响
探索 DeepSeek 的版本和演变

免费福利

您的购买包含本书的免费 PDF 副本以及其他独家福利。请查看前言中的 书籍附带的免费福利 部分以立即解锁它们，并最大化您的学习体验。

介绍 DeepSeek

DeepSeek 是一种开源语言模型，旨在使先进 AI 民主化和易于获取。第一个版本 DeepSeek-R1 于 2025 年 1 月 20 日发布，就在中国农历新年前夕。该团队不仅发布了闭源二进制文件，还公开了权重、训练脚本和推理代码，因此任何人都可以检查或重建该系统。

该模型在 MIT 许可证下发布，没有使用费或严格条款。任何人都可以在本地运行它或将其调整用于新任务。这种自由吸引了世界各地的开发者、研究人员、教师和小型公司。他们将 DeepSeek-R1 应用于客服机器人、课堂辅助工具、实验室研究和写作工具。

基准测试显示，DeepSeek-R1 与 OpenAI-o3 和 Gemini-2.5-Pro 竞争。它能处理数学、代码、多种语言和复杂提示。结果表明，强大的模型不一定必须是封闭的，并凸显了中国在前沿人工智能领域日益增长的作用。该模型的发布也重新引发了关于开放获取与安全性、以及改善全球研究合作的辩论。2025 年 9 月 17 日，另一个里程碑达成，DeepSeek-AI 团队在 Nature 上发表了关于 DeepSeek-R1 模型的研究论文，并登上了当期封面 (www.nature.com/articles/s41586-025-09422-z)。

从架构角度来看，DeepSeek 深度借鉴了基于 Transformer 模型的创新，并在后续版本中加入了自身的特色（在 DeepSeek 的版本与演进 章节中有深入探讨）。但真正使其脱颖而出的是其可用性。DeepSeek 可以部署在多种环境中，从云服务器到边缘设备，甚至可以使用轻量版本运行在笔记本电脑上。

图 1.1：DeepSeek-R1 (0528) 的基准性能 (来源：api-docs.deepseek.com/news/news250528)

我们来谈谈促成 DeepSeek 突然崛起和普及的各种因素：

开源架构和训练细节：DeepSeek-R1 附带了一份详细的研究论文 (arxiv.org/abs/2501.12948) 一起发布，概述了其架构和训练方法，以及在推理、数学和编程任务上的基准分数 (artificialanalysis.ai/providers/deepseek)。此次发布得到了完整的模型权重、配置文件和训练脚本的支持，还包括六个适合本地或低资源环境的较小蒸馏版本 (api-docs.deepseek.com/news/news250120)，以及面向需要托管服务开发者的即时 API 可用性 (api-docs.deepseek.com/guides/reasoning_model)。
时机：其部分流行度源于时机恰当，当时全球组织、科学家和开发者开始探索这一新版本。此外，MIT 许可证提供了商用完全自由——这在高性能模型中已越来越罕见。这次发布还因其并非仅适用于研究；它具有实用性而引发了兴奋。开发者能够对其进行微调，部署到生产环境，并将其集成到现有的 AI 工作流中。这种强大功能与可用性的结合立刻吸引了众人。
初始技术亮点：DeepSeek-R1 发布时最值得注意的方面包括：
- 推理能力：在涉及数学、代码和逻辑推理的关键基准测试中，它超越或匹敌了领先的模型。
- 效率：它以显著更低的推理成本提供了接近 GPT-4 级别系统的性能。
- 强化优先的训练：不同于依赖监督式人工标注数据的传统微调工作流，DeepSeek 跳过了中间步骤，直接使用 基于人类反馈的强化学习（RLHF）或类似范式——尽管使用了最小化的人类标注。这一改变加速了推理分数的提升，降低了人工标注成本，并使模型能够处理各种任务，例如数学问题、零样本代码等，而无需狭窄的特定任务指令。
- 自定义架构：虽然基于 Transformer 框架，但 DeepSeek-R1 融入了针对训练稳定性和长上下文理解优化的创新。

这些要素共同使该模型能够发挥远超其体量的实力，特别是在多步推理和问题解决方面。

现实世界就绪性：DeepSeek 从一开始就展示了现实世界就绪性，这与许多在基准测试中表现出色但在部署时举步维艰的最先进模型不同。与那些需要大量设置或封闭基础设施的模型不同，DeepSeek 可以在实际环境中立即使用。它通过其 API 提供生产就绪访问，通过开放权重和推理代码支持本地部署，并通过 LoRA 微调或提示工程进行定制。与 Trae 和 Windsurf 等企业平台的集成进一步简化了编排。这些能力在发布时很少在其他模型中如此无缝地结合，凸显了 DeepSeek 超越学术性能、致力于实际效用的承诺。
社区兴趣：DeepSeek-R1 的发布引发了激烈的社区活动。GitHub 上迅速涌现了大量插件、适配器和微调衍生版本，而数千个 Hugging Face 分叉为合同审查、辅导、研究辅助、摘要和编程支持等工具提供了支持。在线论坛分享基准测试和硬件指南，大学将该模型用于课程和实验室项目。Reddit、知乎和 Stack Overflow 等论坛充满了分享的实验、性能测试以及在本地硬件上微调 DeepSeek 的指南。该模型的易获取性将业余爱好者转变为研究人员，将开发者转变为企业家。如今，DeepSeek 也推动着教育计划。许多慕课和大学实验室已开始使用 DeepSeek 作为基础模型教授大语言模型理论和实践，因为它开放且清晰。
哲学愿景：DeepSeek 的愿景与更广泛的运动保持一致，即构建的 AI 不应是被守门的资产，而是全球共享的资源。正如 Linux 重塑了软件行业一样，DeepSeek 旨在通过将工具交到任何有好奇心或能力使用它们的人手中，来重塑 AI 开发。其战略不仅是与 OpenAI 或 Google 竞争，而是专注于可访问性和协作创新。

DeepSeek-R1 的成功归功于三个因素：RLHF 训练、开源承诺以及其在基准测试中具有竞争力的表现。

这些要素共同奠定了基础，使 DeepSeek 不仅仅成为一个模型，更成为一个生态系统。本章剩余部分将探讨 DeepSeek-R1 的反响及其开放哲学背后的动机，什么技术突破推动了它的发展，以及它如何演变成一个完整的生态系统。

理解 DeepSeek 的技术突破

真正使 DeepSeek-R1 与众不同的是其架构和训练过程中嵌入的技术创新。这些创新使其能够超越许多当代模型，并帮助重新定义未来大语言模型的构建方式。

训练过程

在开始介绍 DeepSeek 的训练过程之前，我们先看看主流大语言模型的开发过程，通常包括以下步骤：

在大规模语料库上进行自监督学习预训练：在这个阶段，模型暴露在大规模、多样化的数据集中，例如书籍、网页和代码。目标是学习通用的语言模式，而无需显式标签。常见的预训练策略包括：
1. 自回归建模（例如，GPT）：模型预测序列中的下一个词。
2. 掩码语言建模（例如，BERT）：模型预测被遮盖（masked）的词。
3. 基于排列的建模（例如，XLNet）：模型学习多种可能的词序。
Transformer 架构：大多数大语言模型使用 Transformer 架构（图 1.2），以其在自然语言处理（NLP）任务中的可扩展性和性能而闻名。它们通常使用自注意力机制来确定词与词之间的上下文关系。其中一些变体如下：

仅编码器、仅解码器与编码器-解码器

仅编码器（例如 BERT）用于分类或理解。
仅解码器（例如 GPT）用于生成。
编码器-解码器（例如 T5）用于翻译和摘要等任务。

图 1.2: Transformer 架构（Attention Is All You Need, Vaswani 等人, arxiv.org/pdf/1706.03762）

监督微调与 RLHF

在精心策划的指令遵循数据集上进行监督微调：预训练后，模型会在高质量、带标签的数据集上进行微调，学习遵循特定指令并执行有用任务。这些数据集通常包含人工编写的提示和理想响应，帮助模型学习如何更直接和有目的地进行交互。
RLHF：为了更好地使模型输出与人类偏好对齐，应用了 RLHF：
1. 人工评审员评估并排序多个模型响应。
2. 训练一个 奖励模型 来预测这些排名。
3. 然后使用强化学习算法（常用 近端策略优化 (PPO)）对 LLM 进行微调，以生成能最大化 奖励模型 分数的输出，从而更好地与人类价值观和期望对齐。

DeepSeek-R1 打破了这一惯例，它 完全绕过了监督微调。相反，它直接从预训练跳到强化学习，帮助其自主学习到强大的推理能力。

避免 监督微调 (SFT) 消除了对昂贵人工标注数据集的依赖。相反，DeepSeek 采用了 基于规则的奖励机制，例如自动验证正确答案或检查输出格式。这种方法更具可扩展性且成本效益更高，有助于克服人工数据策划的局限性。此外，使用明确、规则驱动的奖励，例如在指定结构内验证答案或确认代码功能，可以缓解奖励黑客（reward hacking）问题，这是不太可预测的神经奖励模型的一个常见问题。

图 1.3 展示了 DeepSeek 模型训练的流程图。

图 1.3: DeepSeek-R1 模型训练

DeepSeek-R1 训练流程

DeepSeek-R1 的训练流程始于 DeepSeek-V3，一个拥有 6710 亿参数的大型基础模型。这个基础模型使用侧重于准确性和输出格式的奖励进行 强化学习 (RL)，产生了一个名为 DeepSeek-R1-Zero 的中间模型。该模型作为一个关键的过渡点，使得在后续阶段能够进行更多针对特定任务的训练。

接下来，DeepSeek-R1-Zero 使用 冷启动数据 进行微调，这指的是广泛多样的指令遵循示例集合。这些数据通常结构良好、经过精心策划，为模型提供了对各种任务格式和领域的基本理解，使其适用于初步的通用指令调整。

在此阶段之后，应用了额外的监督微调（SFT）轮次，使用两个专门的数据集。第一个是 思维链 (CoT) **数据**，强调多步骤推理。该数据帮助模型学习通过分解为中间步骤来解决复杂问题——这对于数学推理、逻辑推理和多跳问题回答至关重要。第二组是 知识数据，包含事实丰富、特定领域的内容，如科学文献、百科全书信息和技术手册。这有助于模型提高事实准确性并扩展领域覆盖。

这些微调阶段完成后，通过先进的 RL 技术进一步增强了得到的模型 DeepSeek-R1。它不仅使用针对准确性和格式的 奖励 进行训练，还使用针对一致性的奖励，确保其输出在逻辑上连贯且自洽。此外，应用 基于规则的验证 来自动验证数学和代码生成等领域的响应。最后，引入 人类偏好微调 以使模型的行为与人类期望和质量判断保持一致。

DeepSeek-R1 已被蒸馏成更小、更高效的变体。这些变体包括 DeepSeek-R1-Distill-Qwen，它使用从 15 亿到 320 亿参数不等的 Qwen 2.5 模型；以及 DeepSeek-R1-Distill-LLaMA，它利用 80 亿和 700 亿大小的 LLaMA 3 模型。这些蒸馏版本保留了原始 R1 模型的大部分能力，但针对不同的资源和延迟约束进行了优化。

总体而言，DeepSeek-R1 流程代表了一种多阶段策略，结合了监督学习、基于奖励的调整和针对性的模型蒸馏，以提供一系列针对性能、泛化性和部署灵活性进行优化的指令遵循语言模型。

结果，训练过程变得更加稳定和高效，受益于简化、可靠的奖励信号，减少了噪声和歧义。

DeepSeek 的训练方法在以下方面带来了帮助：

降低人力成本：无需手动标注或排列数千条指令。
更快的开发周期：训练时间线被显著精简。
更强的泛化能力：模型通过试错交互而非固定模板学会了泛化指令遵循能力。

尽管缺乏传统的监督指令数据集，DeepSeek-R1 展示了 强大的指令遵循能力，与经过微调的模型具有竞争力。这表明，当设计良好时，仅靠强化学习就能赋予模型对指令和意图的深刻理解。

让我们看一下 DeepSeek 的 推理流水线，如 图 1.4 所示。

图 1.4: DeepSeek-R1 模型推理

DeepSeek-R1 推理流程

DeepSeek-R1 的推理流程旨在通过结合 规则感知解码 和 提示优化 来优先输出结构化、可验证的结果。在推理期间，DeepSeek-R1 利用 格式感知生成机制，模型受到鼓励（通常通过提示设计和内部对齐）生成结构良好、可解释的响应，特别是对于涉及代码、数学或 CoT 推理的任务。它不仅针对流畅性进行了优化，还针对事实和逻辑一致性进行了优化，通常在答案中包含中间步骤（CoT），即使没有明确提示也是如此。这使得 DeepSeek-R1 能够提供分步解决方案和结构化输出（如 JSON、Markdown 或代码块），从而提高了下游应用的可靠性。

DeepSeek-R1 与其他最先进的 LLMs 不同之处在于其基于规则的 奖励对齐推理行为。许多领先的 LLMs 主要依赖端到端训练和人类偏好微调，而 DeepSeek-R1 将 基于规则的验证 技术直接集成到 RL 循环中。这在推理时产生了下游效应：DeepSeek-R1 更有可能生成与形式化验证器或下游评估器兼容的输出（例如，代码的测试用例，数学的方程式）。因此，它在精度、结构和可解释性至关重要的领域表现出更强的性能，同时略微牺牲了开放式对话的灵活性。

接下来是 DeepSeek 的 RL 方法。

强化学习

DeepSeek-R1 的强化学习方法因其独特的设计选择而备受关注，我们将在第 2 章中深入探讨。与许多仅在训练最终阶段应用强化学习的模型不同，DeepSeek-R1 引入了对齐技术。对齐技术旨在训练早期就引导模型的输出变得更有帮助、更诚实、更无害。这种早期对齐有助于模型在整个发展过程中表现更一致、更理想。

它还用自动化的奖励模型取代了大规模人工标注，显著提升了可扩展性，并减少了对人工标注的依赖。另一个显著特点是它使用了自我博弈和迭代优化，使模型能够生成、评估并改进自身的输出。这种方法帮助 DeepSeek-R1 内化了高级推理模式和战略决策能力，使其在多轮推理、代码解释与补全以及解决复杂数学问题方面尤为有效。

此外，强化学习训练通过基于自我生成的成功指标强化事实准确性，帮助减轻幻觉。

对于不熟悉 AI 语境中“对齐”概念的读者，它通常指确保模型行为与人类意图和价值观相符的技术。关于这些概念的有用介绍，可以在 OpenAI 的对齐概述（openai.com/index/our-approach-to-alignment-research/）或对齐通讯（www.alignmentforum.org/s/dT7CKGXwq9vt76CeX）中找到。

除此之外，DeepSeek 对 Transformer 架构进行了一些修改。让我们来一探究竟。

架构修改

DeepSeek-R1 构建于广泛采用的 Transformer 架构之上，该架构构成了大多数现代大语言模型的基础。其核心是 Transformer 使用的自注意力机制，允许输入序列中的每个 token 无论位置如何，都能权衡所有其他 token 的重要性。这使得模型能够比传统的循环模型更有效地捕捉长程依赖和上下文关系。

然而，标准注意力机制的计算成本会随着输入长度增加而变得高昂。为了解决这个问题，DeepSeek-R1 引入了自适应注意力路由，这是一个重大的架构演进。与传统的对所有 token 应用固定全注意力的 Transformer 不同，此机制允许模型根据训练过程中计算的相关性分数，有选择地关注最相关的 token。这些分数通常源自内部注意力权重或辅助的门控机制，优先处理对最小化训练损失贡献最大的 token。通过将计算资源集中在高影响 token 上，尤其是在长序列中，自适应注意力路由使 DeepSeek-R1 能够更高效地处理最多 32,000 tokens 的输入。这不仅增强了模型理解和总结大型文档的能力，还通过避免对信息量较少的 token 进行冗余注意力计算来减少计算开销。

图 1.5 比较了传统 Transformer 注意力机制和 DeepSeek 的自适应注意力路由。

图 1.5：传统 Transformer 注意力与 DeepSeek-R1 自适应注意力路由对比

此外，DeepSeek-R1 采用了混合精度优化，结合 FP16（半精度）和 INT8（量化）运算以提高训练和推理效率。这种方法在降低内存使用和加速计算的同时，保持了模型在准确率和困惑度方面的竞争力性能。通常，FP16 用于模型大部分的一般计算，而 INT8 量化则选择性地应用于推理时的矩阵乘法，通常是在注意力和前馈层中，这些地方的精度降低不会显著影响输出质量。通过仔细选择要量化的层，DeepSeek-R1 实现了效率与性能之间的有利权衡。这种方法在保持输出质量的同时显著加速了推理和训练，使其非常适合大规模部署。

该模型还受益于高效的并行化策略。它利用张量并行和激活检查点来减少训练期间的内存使用，使其能够在多 GPU 系统上更有效地进行训练。

总的来说，这些增强使得 DeepSeek-R1 成为 Transformer 架构的演进，它不仅更具可扩展性和上下文感知能力，而且在训练和推理方面也更具成本效益。

现在，让我们将焦点转向 DeepSeek 架构与其他大语言模型架构的比较。

主流大语言模型架构对比（2025 年）

随着大语言模型领域的发展，不同的架构和训练范式应有尽有。本对比（表 1.1）重点关注关键差异，特别是近年来一些最具影响力的模型中 Mixture-of-Experts (MoE) 架构和强化学习的使用情况。

| --- | --- | --- | --- | --- |

| GPT-4.5 | 否 | 2025 年 2 月 27 日发布；OpenAI 最大的非 CoT 模型 (Orion) | 是。RLHF + SFT | 否 |

表 1.1：大语言模型架构对比

领先的语言模型在其架构策略上日益分化，特别是在 MoE 的使用方面。像 Gemini 2.5 Pro 和 DeepSeek-R1 这样的模型采用了稀疏 MoE 架构，使其能够拥有大规模参数，同时保持高效的计算利用率。基于其强大的性能和低延迟特性，外界普遍认为 GPT-4.1 采用了某种形式的 MoE 或稀疏专家路由，尽管具体细节尚未披露。

介于这些方法之间，Grok 3.5 保留了密集 Transformer 架构，针对实时响应能力和集成推理进行了优化。它完全避免了 MoE，转而专注于强化学习技术和使用实时反馈数据的迭代优化。

相比之下，Claude 4 和 LLaMA 4 继续采用完全密集的设计，优先考虑简单性、对齐稳定性和可预测行为，而非原始的参数规模扩展。

既然我们已经向您介绍了 DeepSeek 的技术创新，我们将更深入地探讨 DeepSeek 的 MoE 设计。

MoE 架构

DeepSeek-R1 的一个重大架构突破是集成了 MoE 设计。

MoE 是一种模块化的神经网络设计，其特点是对于任何给定的输入，只激活一部分参数（称为experts）。MoE 并非在每次预测中都使用完整的参数空间，而是动态地、有选择性地激活少数专家。

DeepSeek 如何实现 MoE

DeepSeek 采用了一种稀疏的MoE 架构，其中门控网络在每一层动态地从N个专家网络中选出两个来处理给定的输入。选择两位专家在计算效率和模型表达能力之间取得了平衡。它允许模型利用多样化的专业知识，而无需承担激活所有专家的全部成本。这种方法使得专家能够实现专门化，同时保持推理延迟和资源使用处于可管理的水平。这些专家并非被手动分配给特定任务，如数学或代码；相反，专门化是在训练过程中涌现的。门控机制通过优化来学习，根据上下文线索将输入路由到最有效的专家。随着时间的推移，由于这种学习到的路由机制，某些专家在特定领域（例如语言、推理和编码）被更频繁地激活，从而有效地发展出功能性专门化。

图 1.6 概览了此架构。

图 1.6：MoE 架构概念图

每个专家处理相同类型的输入表示，但可能会根据接收到的模式学会强调不同的方面。它们的角色由它们最常被路由的数据所塑造，而这反过来又引导着它们的参数更新。这使得专家能够有机地承担不同的角色，而无需不同的输入格式或编码。模型包含一个大型的专家子网络池（每个都是一个小型的前馈网络），但每个输入令牌只激活一小部分——通常是两个。门控网络评估上下文，并动态决定激活哪些专家，允许模型将信息自适应地路由到最有效的处理位置。

这种结构带来了几个关键优势：

可扩展性：由于任何时刻只有少数专家处于活动状态，模型可以维护大量的总参数，同时每个令牌消耗的计算量比同等规模的密集模型更少。这使得 DeepSeek 能够扩展，而不会线性地增加计算成本。
模块化：专家可以独立进行训练、冻结、更新甚至替换。这种模块性允许多高效的持续学习、领域适应或特定任务微调，而无需重新训练整个模型。
专门化：随着门控网络学会将不同的输入路由给不同的专家，这些子网络开始专门化；一些更擅长代码，另一些则更擅长数学推理、自然语言或对话。这降低了过拟合的风险，并增强了模型在多样化任务上的泛化能力。

通过这种架构，DeepSeek-R1 本质上表现得像一个特定领域模型的集成，但没有资源重复或通常与并行运行多个系统相关的延迟开销。

除了 DeepSeek，许多最先进（SOTA）的 LLM 也采用了 MoE 架构，详细信息见下表：

| --- | --- | --- | --- | --- | --- |

表 1.2：MoE 模型比较

随着 MoE 架构获得关注，Mixtral 8×7B 展示了开源模型如何在有限计算资源下获得强大的推理能力，而 Grok 和 Gemini 2.5 等更新系统则增加了自适应或专有路由以及多模态预训练。谷歌的 Switch Transformer 和 GLaM，这两个万亿参数的原型，首先证实了 MoE 可以可靠地扩展。这些项目共同展示了 MoE 如何让超大规模模型发展，同时保持足够快的推理速度以满足实时、高性能任务的需求。

构建 LLM 的另一个基础方面是其训练所用的数据。让我们看看 DeepSeek 如何利用其训练数据集。

训练数据集与理念

在论文DeepSeek-R1：通过强化学习激发 LLM 的推理能力 (https://arxiv.org/abs/2501.12948) 中，作者指出 DeepSeek-R1 是在一个大规模、多样化且领域丰富的多语言数据集上训练的，旨在支持推理、编程和跨语言理解。

虽然该数据集的确切组成并未公开，但论文概述了使用精心策划的冷启动数据和大约 80 万个 SFT 样本，这些数据被整合到一个多阶段的强化学习框架中以开发模型。

数据选择策略强调跨多个关键领域的高质量来源：

模型在Python、JavaScript和Rust等编程语言上接受了大量训练，使用了来自 GitHub 等仓库以及 Stack Overflow 等开发者问答平台的精选代码。
为了加强数学和逻辑推理，数据集纳入了形式逻辑语料库、符号数学基准以及竞赛数学问题集（例如MATH、GSM8K和ProofWriter）。
科学理解通过在学术出版物和技术手册上进行预训练得到加强，这些资料来源于 arXiv 等平台以及开放获取的研究数据集。
数据集的多语言部分以中文和英语为基础，并额外覆盖了主要全球语言，数据来源包括CCMatrix和OPUS等数据集。

这种多样化的训练基础反映了 DeepSeek-R1 的目标：将高级推理与强大的多语言和领域能力相结合。

在同一篇论文中，作者描述了在构建训练数据时采用了低过滤策略，这与许多其他大语言模型开发者使用的更激进的数据过滤流程形成对比。做出这一设计选择是为了保留语言的自然复杂性和多样性，使模型能更好地捕捉非正式表达、文化特定习语、情感化语言以及边缘案例场景。根据作者的说法，这种语言多样性支持更具表现力、创造性和语境流畅性的模型行为。然而，论文并未明确说明过滤掉了哪些类型的内容，或者是否进行了过滤。由于数据集的组成或过滤标准缺乏透明度，目前尚不清楚训练语料库是否包含了潜在有害内容，如仇恨言论、错误信息或攻击性材料。虽然这种低过滤方法可能增强模型在多样化语言和文化语境下的泛化能力，但它也引入了模型可能学习并再现不良内容的风险。这些权衡凸显了下游安全措施和负责任使用的重要性，尤其是在将模型部署于现实世界环境时。

DeepSeek-R1的架构和训练策略使其特别适合开放式任务。它在头脑风暴、创意生成和探索性对话——这些需要灵活性和最少预设的领域——展现出强大能力。得益于其多样化的语言训练，该模型在精细翻译和多语言推理方面也表现出色。此外，DeepSeek-R1还以其适应用户语气和对话风格的能力而闻名，许多用户认为这在创意和协作场景中非常有用。

嗯，DeepSeek 已经推出过多个版本，我们预计随着通往通用人工智能竞赛的白热化，未来还会有更新的版本发布。

在下一节中，我们将探讨自 R1 发布以来，DeepSeek 生态系统是如何演进的。

DeepSeek 的版本与演进

DeepSeek 的故事不仅仅关乎一次模型的发布。它关乎一个持续演进的生态系统。DeepSeek 的每一次迭代都在推理能力、易用性、安全性和跨平台集成方面带来了重大升级。理解 DeepSeek 的演进过程，对于把握其长远愿景和潜力至关重要。

DeepSeek 不断演进的生态系统

自首次亮相以来，DeepSeek 在多个领域取得了长足进步，包括语言理解、数学推理、代码生成和多模态能力。每个版本都引入了有意义的增强，强调了 DeepSeek 在不妥协质量的前提下，实现高性能语言模型民主化的目标。

理解 DeepSeek 的演进，能为其增长轨迹、愿景以及它如何持续颠覆闭源和开源 AI 生态提供关键见解。在下面的表格中，我们将按时间顺序探讨构成 DeepSeek 套件的关键里程碑、模型变体和产品层次。

| 版本 | 发布日期 | 关键特性 |

| :--- | :--- | :--- |

| DeepSeek LLM | 2025 年 1 月 | 基础模型；多语言，开源权重 |

| DeepSeek-R1 | 2025 年 1 月 20 日 | 完整 MIT 许可发布，强大推理能力，多语言，聊天+代码 |

| DeepSeek V2 | 2025 年初 | 更优的对齐性，更好的事实基础 |

| DeepSeek Coder | 2025 年 2 月 | 专用编码模型，在 Python 和 JS 方面表现顶尖 |

| DeepSeek VL | 2025 年 3 月 | 视觉-语言模型（图像+文本），奠定多模态基础 |

| DeepSeek Math | 2025 年 4 月 | 专注于代数、逻辑和多步推理 |

| DeepSeek V3 | 2025 年 5 月 | 升级的通用模型，具备更好的长上下文和规划能力 |

| DeepSeek-R1-0528 | 2025 年 5 月 | 最新优化：更强的事实性，32k 词元上下文，改进的对齐 |

| DeepSeek Coder V2 | 2025 年 6 月 | 代码合成和行内文档生成能力大幅提升 |

| DeepSeek V3.1 | 2025 年 8 月 | 混合推理，快速思考，更强的智能体技能 |

| DeepSeek V3.2-Exp | 2025 年 9 月 | DeepSeek 稀疏注意力（DSA），实现更快、更高效、支持长上下文的推理。 |

表 1.3：DeepSeek 模型演进

每个模型都针对特定用例，从通用的聊天机器人功能到高度聚焦的编码和数学任务。让我们来详细探讨一下它们。

深入解析：每个模型的功能比较

DeepSeek 生态系统已迅速发展成一套专用模型，每个模型都旨在解决推理、编码、视觉和通用 AI 领域的不同用例。虽然所有变体都建立在共同的架构骨干和训练理念之上，但每个模型迭代都引入了新的能力、性能权衡和领域优化。以下是 DeepSeek 家族中主要模型的概述，以及它们在专业化和效用方面的比较：

DeepSeek LLM (https://github.com/deepseek-ai/DeepSeek-LLM)：作为 DeepSeek 家族最初的骨架，DeepSeek LLM为所有后续迭代奠定了基础。虽然它缺乏一些专用能力，但它确立了多语言能力和扎实推理作为核心优先事项。
DeepSeek Math (https://github.com/deepseek-ai/DeepSeek-Math)：专为学生、研究人员和技术专业人士量身打造，DeepSeek Math专注于代数、微积分、几何和符号逻辑中的多步推理问题。它可作为类似 Wolfram Alpha 推理系统的可行开源替代方案。
DeepSeek Coder (https://github.com/deepseek-ai/DeepSeek-Coder) 和 Coder V2 (https://github.com/deepseek-ai/DeepSeek-Coder-V2)：初代Coder模型在 Python 和 JavaScript 方面引入了有竞争力的性能，并与VS Code和GitHub Copilot等开发环境集成。Coder V2（2025 年 6 月）显著提高了标杆，在行内函数合成、文档字符串生成和类型推断方面接近 Claude 3.5 的水平。
DeepSeek VL (https://github.com/deepseek-ai/DeepSeek-VL)：对于多模态应用来说是一个关键发布，VL支持图像和文本输入，为视觉问答、光学字符识别、文档摘要等应用打开了大门。尽管在视觉能力方面仍落后于GPT-4-V或Gemini 1.5 Pro，但它正在快速改进。
DeepSeek V2 (https://github.com/deepseek-ai/DeepSeek-V2) 和 V3 (https://github.com/deepseek-ai/DeepSeek-V3)：V2更新优先改进了提示对齐性，最大限度地减少了幻觉并扩展了对更长上下文的支持。V3则跟进提供了更好的长期记忆支持、更快的推理速度以及内部规划模块，从而实现了早期的智能体行为。

DeepSeek 模型与产品生态

DeepSeek-R1：这是开启一切的发布版本，于 2025 年 1 月 20 日发布，迅速成为在广泛基准测试中表现最佳的开源模型。其主要亮点包括完全开源、采用 MIT 许可证发布，提供了模型权重、分词器和整个训练流程。DeepSeek-R1提供了强大的逻辑推理能力，超越了大多数开源模型，并与一些专有系统相媲美；同时在英文和中文上都表现出强大的多语言性能。如前所述，该发布还包含了一系列小于 20B 参数的蒸馏变体，使其能够在本地或边缘环境中高效使用。实际应用范围从基于 API 的聊天机器人集成、Copilot 风格的编码辅助，到通过 Ollama、VS Code 扩展等平台进行的轻量级部署。
DeepSeek-R1-0528：这次更新巩固了DeepSeek在开源金字塔顶端的地位。5 月 28 日的版本对核心模型进行了改进，使其在对齐性、准确性和效率方面表现更佳。新功能和增强包括以下方面：
- 幻觉大幅减少：特别是在科学和历史事实上。
- 编码流畅度提升：在许多 Python 任务中达到了与GPT-4-turbo相当的水平。
- 数学性能增强：在多步代数、几何和逻辑问题中的准确性得到提高。
- 提示词对齐更新：即使在模糊的提示词下，也能更好地遵循用户指令。
- 多模态就绪：架构已为未来的图像/文本融合做好了准备。

在基准评估中，5 月发布的DeepSeek-R1-0528在数学推理任务上比 1 月版本提高了 7%。在HumanEval风格的基准测试中，代码生成性能提升了 9%。此外，该模型展示了有效的长上下文推理能力，能够处理多达 32,000 个token的输入，且性能下降极小。

DeepSeek-R1-0528的发布凸显了DeepSeek的开发步伐依然强劲且一致。其改进的性能和开放的可访问性促使许多开发者开始将整个工作流从基于 GPT 的系统迁移到DeepSeek API。这种转变进一步得到了生态系统集成激增的支持，包括 Visual Studio Code 插件、Ollama兼容性和Docker化部署选项，标志着在个人用户和企业级用户中都获得了日益增长的采纳。

DeepSeek-V3.1（2025 年 8 月）：DeepSeek V3.1是一个前沿的混合推理模型，具有思考和非思考模式、高级代理和工具使用能力、庞大的 6850 亿参数架构，以及扩展的 128,000 个token上下文窗口，适用于长文档理解。该模型专为快速、结构化、多步骤推理而设计，支持代码生成、搜索和代理工作流。得益于混合专家（MoE）架构和优化的推理，它在保持低延迟的同时，匹配或超越了之前的DeepSeek基准。DeepSeek V3.1还具有强大的多语言支持、开源的研究可用性，以及为可靠外部工具集成和减少幻觉而进行的专门训练，使其适用于各种企业和开发者应用。

除了各种模型，DeepSeek还创建了许多产品以简化采用和使用。接下来让我们看看这些产品。

DeepSeek 产品生态

除了模型本身，DeepSeek还开发了一系列不断增长的面向用户的产品和开发者工具，使采用变得无障碍：

DeepSeek 应用：一款移动端优先的 AI 助手应用，适用于 Android 和 iOS，提供与DeepSeek-R1及数学/编码器变体的实时交互。关键功能包括语音输入、代码单元格执行、笔记记录和多语言支持。
DeepSeek 网页应用：可通过deepseek.com/chat访问，提供了一个简洁且响应迅速的界面，用于与各种DeepSeek模型变体进行实时交互。它包含对话记忆功能，允许用户在多次交流中保持上下文，以获得更连贯的对话。提示词模板可用于简化重复性任务或结构化输入，便于原型设计或测试特定行为。此外，用户可以将整个聊天会话导出为Markdown或PDF格式，这对于文档、协作或离线审阅特别有用。
DeepSeek 开发者平台 (https://platform.deepseek.com/)：提供了一个灵活开放的环境，用于构建和部署 AI 驱动的应用程序。开发者可以微调模型以创建针对特定任务或领域的自定义端点。该平台支持在DeepSeek系列中无缝选择模型，包括通用（R1）、专注于编码（Coder）和多模态（VL）变体。支持高达 32,000 个token的上下文感知 API 调用，能够实现复杂的多轮推理和长篇内容处理。还提供了一个用于角色和函数调用的测试版功能，允许开发者定义结构化交互，并为工具、代理或工作流自动化扩展模型能力。

DeepSeek的价值主张在于其易于获取的模型权重、强大的推理能力和低成本。这种组合使得DeepSeek-R1成为教育、研发以及寻求先进 AI 工具而无苛刻许可或高成本的初创公司的有吸引力的选择。

接下来，让我们看看DeepSeek提供的集成和部署支持。

平台集成与部署生态

DeepSeek的可访问性是其决定性优势之一，这得益于其在本地、云和网络环境中广泛的集成和部署选项。以下是DeepSeek可以在何处及如何使用的高层概述：

对于本地和边缘部署，DeepSeek可以无缝运行在 Ollama (https://ollama.com/library/deepseek-r1 – ollama run deepseek)、Docker Hub (https://hub.docker.com/r/devlinrocha/ollama-deepseek-r1-7b/tags) 等平台上，并通过 VS Code 扩展 (https://github.com/enesbasbug/deepseek-vscode-extension) 提供代码补全和内联辅助。它还支持通过Kubernetes、Docker Compose或直接GPU集群设置进行私有云和本地部署。
在云环境中，DeepSeek已与 Amazon Bedrock (https://aws.amazon.com/bedrock/deepseek)、Amazon SageMaker (https://aws.amazon.com/blogs/machine-learning/deploy-deepseek-r1-distilled-models-on-amazon-sagemaker-using-a-large-model-inference-container/)、Azure ML (https://azure.microsoft.com/en-us/blog/deepseek-r1-is-now-available-on-azure-ai-foundry-and-github/) 以及 Google Cloud Vertex AI (https://cloud.google.com/vertex-ai/generative-ai/docs/maas/deepseek) 集成，支持通过Hugging Face或自定义容器进行可扩展的推理。

DeepSeek 对全球 AI 生态系统的影响

DeepSeek-R1 的发布不仅是一个技术里程碑；对全球 AI 产业而言，它更是一个战略转折点。其开源可用性与顶级性能的独特结合，在实验室、初创公司和政策圈都引发了震动。其影响范围涵盖了经济竞争、学术加速、伦理讨论和地缘政治格局调整。

市场颠覆与价格战

在 DeepSeek-R1 之前，前沿级大语言模型通常价格昂贵，仅限于通过 API 访问，或受限于限制商业使用的许可证。DeepSeek-R1 的发布，伴随着可自由获取的模型权重、推理代码以及宽松的许可证，标志着一次重大转变，使得高性能语言模型变得更容易获取。

DeepSeek-R1 通过提供免费、完整地访问一个高性能推理模型，对 AI 领域产生了显著影响。它在数学、编程和形式逻辑等传统上由专有系统主导的领域，展示了具有竞争力的准确性。通过公开提供这些能力，DeepSeek 为研究人员和商业开发者提供了一个可行的替代闭源平台的选择。

或许这次发布最重要的后果是它带来的市场压力。通过降低进行先进 AI 实验的经济门槛，DeepSeek-R1 挑战了该领域关于可获得性和可负担性的普遍假设。其开放可用性促使一些专有实验室加速实施自己的开源策略。此外，这一举措引发了关于 AI 公平性、竞争和全球治理的更广泛讨论，突显了创新、可访问性和负责任部署之间日益紧张的关系。它的开放发布在中国之外也激起了涟漪，激励了欧洲、印度甚至硅谷的初创公司探索更透明的模型。

DeepSeek-R1 的发布引发了 AI 定价和市场定位的显著转变。作为回应，OpenAI 推出了更实惠的选项，例如为 GPT-3.5 Turbo 提供折扣访问，同时为需要顶级性能的用户保留了每月约 200 美元的 GPT-4.1 高级计划。Anthropic 和 Cohere 也做出了反应，推出了更小、成本更低的聊天模型，旨在保持对预算敏感用户的吸引力。与此同时，Meta 重申了对 LLaMA 技术路线图的承诺，暗示将扩大许可选项，并强调转向在模型部署中实现更广泛的可访问性。

开源突然不再是小众领域，DeepSeek 成为对闭源商业模式的一个严肃经济威胁。企业，特别是对成本敏感的企业，开始评估将 DeepSeek 用于客服机器人、嵌入式智能体和企业知识库，以替代更昂贵的 API。

催生下一波开源 AI 浪潮

DeepSeek 的开放发布鼓舞了全球的开源运动。此前感觉自己无法对大语言模型做出有意义贡献的开发者和研究人员找到了新的动力。DeepSeek 证明了你不需要数十亿美元的基础设施就能创造出真正有影响力的东西。

这导致了以下情况：

学术实验室针对专门领域（生物医学、法律和 STEM 教育）对 DeepSeek 进行微调。
初创公司使用 DeepSeek 作为后端构建 SaaS 工具。
政府部门研究公共部门部署大语言模型的用例。

在 DeepSeek-R1 发布后，全球的开源开发激增，新的模型在印度、韩国、欧盟和拉丁美洲等地区涌现。在 Hugging Face 上，一个值得注意的印度项目Deepdive404/Deepseek-fork发布了 R1 的多个参数规模的蒸馏版本：1.5B、7B、8B、14B、32B 和 70B。您可以在此处找到它：huggingface.co/Deepdive404/Deepseek-fork。同时，原始的deepseek-ai/DeepSeek-R1代码库在huggingface.co/deepseek-ai/DeepSeek-R1提供核心 R1 和 R1-Zero 模型及其蒸馏变体。

这些社区驱动的分支版本融合了本地语言、区域使用习惯和各种导出格式，这从诸如gghfez/DeepSeek-R1-11446-Q2_K（位于huggingface.co/gghfez/DeepSeek-R1-11446-Q2_K）等经过量化和优化的版本中可见一斑，该版本专为高效 GPU 推理而定制。短短几个月内，Hugging Face 记录了数百个受 DeepSeek 启发的分支和集成，重点关注本地化模型、量化和社区主导的改进。

该模型还激发了跨国合作。研究人员开始使用 DeepSeek 作为基准发布跨实验室基准测试研究，社区维护的评估排行榜赋予了该模型远超其最初发布热度的可信度（artificialanalysis.ai/models/deepseek-r1、www.statista.com/statistics/1552824/deepseek-performance-of-deepseek-r1-compared-to-open-ai-by-benchmark/ 和 pubmed.ncbi.nlm.nih.gov/40267969/）。

中国技术成熟的象征

托管 API 可通过 Fireworks.ai（fireworks.ai/）、Together.ai（www.together.ai/）、Replicate（[replicate.com/]）和 Modal（modal.com/）等提供商获得，提供快速原型开发和生产就绪的工作流程。

对于轻量级和基于浏览器的访问，用户可以尝试在deepseek.com/chat、Hugging Face Spaces 或通过 Google Colab 和 Kaggle 上即用型笔记本中运行模型。

如果您希望探索这些部署选项，我们已在本书末尾为您创建了附录供您参考。

DeepSeek 还与 LangChain、Haystack 和 LlamaIndex 等流行的编排框架集成，用于检索增强生成（RAG），以及与 Turing、Trae 和 Windsurf 等无代码工具集成，用于构建结构化的智能体工作流程。

DeepSeek 即将到来的路线图概述了其模型能力和部署策略的雄心勃勃的扩张计划。计划中的开发包括整合图像和音频理解的多模态训练，以及创建配备记忆、规划和工具使用功能以处理复杂工作流程的自主智能体。团队还在开发针对医疗问答、法律分析和 STEM 教育等领域定制的小型、特定任务变体。此外，DeepSeek 正通过联邦学习方法和安全的本地部署大语言模型投资于注重隐私的部署方案。继 R1 系列成功之后，人们对可能于 2025 年底发布的 DeepSeek-R2 抱有期待。虽然尚未确认，但早期报告表明它可能提供可与 GPT-5 媲美的结构化推理能力，同时保持完全开源的框架。

但伴随着所有的炒作，质疑也随之而来。下一节将探讨 DeepSeek 的选择如何影响更广泛的 AI 生态系统，包括定价模型、竞争和全球政策动态。我们还将讨论社区中普遍关注的一些担忧和风险。

DeepSeek-R1 标志着一个里程碑，它是首批在全球公认基准测试上取得有竞争力性能的中国开源模型之一。其在需要深度推理的任务中表现强劲，并迅速获得国际认可，挑战了西方在前沿 AI 领域占据主导地位的固有观念。该模型的反响凸显了开放协作和跨境评估在确立全球 AI 创新合法性方面日益增长的重要性。

独立实验室的基准测试证实，DeepSeek-R1-0528 的性能可与 OpenAI 的 O3 和 Google 的 Gemini 2.5 Pro 相媲美。对于一个开源模型而言，这是前所未有的。

它证明了两件关键事情：

DeepSeek 并非昙花一现——它正在成为一个不断壮大的生态系统。
在社区协作和巧妙工程的支持下，开源开发能够跟上闭源实验室的步伐。

2025 年 5 月 28 日发布的 DeepSeek-R1-0528 在数学推理、代码生成和事实检索方面带来了性能提升。它还在事实问答基准测试中展示了降低的幻觉率，并引入了改进的长上下文处理能力，现在支持最多 32,000 个 token 的输入长度——因此，它对于复杂的多轮任务和扩展文档分析更加有效，并通过此次新版本的发布，再次确认了模型的持续改进。

DeepSeek-R1 的发布也引发了一些关键问题和持续的讨论。让我们谈谈围绕 DeepSeek 的一些争议。

围绕 DeepSeek 的争议

如同 AI 领域的任何重大进展一样，DeepSeek 的出现也伴随着争议。尽管它因其技术复杂性、开源立场和开创性的推理方法而广受赞誉，但它的崛起在研究伦理与安全、地缘政治战略到知识产权等多个领域引发了辩论。本节探讨了伴随 DeepSeek 崛起而来的多方面争议，承认了在技术进步与负责任创新之间的张力。

存在偏见和被滥用的可能性

一个令人担忧的领域源于 DeepSeek 决定仅依靠 RL 进行对齐，而未采用传统的 SFT 或 RLHF。批评者认为，这可能会使模型随着时间的推移容易发生对齐漂移或不可预测的行为。其他人则质疑，尽管与团队所宣称的透明和协作目标一致，但模型的开放访问性质是否增加了被滥用的可能性。公开讨论中也涉及到该模型是否会抑制或软化对政治敏感话题的回答，这引发了对内容审核和训练数据选择透明度的质疑。

DeepSeek 团队已发表多篇论文，例如 DeepSeekR1 (arxiv.org/abs/2501.12948) 和新闻文档 (api-docs.deepseek.com/news/news250120)，为社区、研究人员和批评者描述了评估方法。DeepSeek 还发布了较小的变体，用于在敏感、更保守或资源受限的环境中部署。其中一些模型是 DeepSeek-R1-Zero 和几个蒸馏的 1.5B–70B 模型；它们可通过官方 GitHub 仓库 deepseek-ai/DeepSeek-R1 (github.com/deepseek-ai/DeepSeek-R1) 和 Hugging Face 仓库 deepseek-ai/DeepSeek-R1 (huggingface.co/deepseek-ai/DeepSeek-R1) 获取。

虽然这些措施已帮助安抚了开发者和研究社区中的许多人，但它们并未完全解决开放性与控制之间的张力。这凸显了一个现实：平衡强大的通用人工智能与负责任的部署仍是一个尚未解决且不断演变的挑战。

DeepSeek 的极端透明性重新引发了 AI 社区关于访问与控制的辩论：

发布一个功能强大但防护措施最少的模型是否负责任？
它是否可能被滥用以生成虚假信息或有害内容？
社区的红队测试足以确保安全吗？

支持者认为，这种开放性使得对大型语言模型对齐和可信度的学术研究得以进行，这是任何闭源模型都无法实现的。批评者则坚持认为应采取更多措施来防止滥用。尽管如此，DeepSeek 表明，围绕安全性的对话不必局限于公司内部的走廊。它可以是开放的、协作的，并由社区治理。

对于好奇的读者，一份关于 DeepSeek-R1 模型可信度的高层评估报告《Trust Report: Vijil DeepSeek-R1 Trust Report》 (www.vijil.ai/blog/deepseek-trust-report) 可供参考，该报告总结了模型在安全、隐私、幻觉、毒性、刻板印象、伦理、性能和稳健性等维度的风险水平，以及缓解这些风险的通用建议。

开放性与国家安全关切

DeepSeek 最具争议的方面之一是其极端的开放性。虽然这受到学术和开源社区的赞誉，但模型完整的权重发布、训练方案和 RL 协议却受到了国家安全分析师和政策智库的批评。有人认为，DeepSeek-V3 和 R1 的复杂性足以匹敌或超越许多闭源模型，可能被滥用以增强恶意行为者的能力。

例如，情报分析师对 DeepSeek 在自动宣传生成、网络入侵规划和合成虚假信息活动中的潜在应用表示担忧，特别是考虑到它的 RAG 和代码编写能力。通过思维链（CoT）框架进行推理、解释复杂概念乃至操纵对话模式的能力，使得 DeepSeek 在不法之徒手中变得异常强大。

这场辩论反映了人工智能领域更广泛的紧张关系：多少开放性才算过多？具备自主推理能力的模型是否应在没有约束的情况下发布？虽然 DeepSeek 已实施内容过滤器和基于 RL 的伦理指令，但缺乏像 OpenAI 或 Anthropic 那样的 API 门控意味着对最终使用的限制更少。该模型的效用显而易见，但被滥用的风险也同样清晰。

尽管如此，支持者辩称，以安全为名审查开放科学可能会适得其反。他们认为，透明性能够促进关于对齐和安全性的更大研究合作，使创新去中心化，并建立公众信任。截至 2025 年中，DeepSeek 团队仍坚持其立场：受控的开放——结合稳健的文档和红队测试——仍是负责任的 AI 发展的最佳路径。

文化与政治认知

鉴于 DeepSeek 起源于中国，并使用英语和中文进行双语训练，它已成为地缘政治审视的焦点。一些西方评论者指责 DeepSeek 是一个用于软实力投射的战略工具——试图不仅通过能力，还通过影响全球开源生态系统来确立 AI 主导地位。虽然这种说法具有推测性且往往带有政治色彩，但它们反映了在多极世界中对 AI 领导地位的真切焦虑。开源软件曾被视为中立地带，如今却越来越多地被视为地缘政治资产。DeepSeek 在关键推理基准测试上超越西方模型的成功，加剧了这一叙事。

然而，在全球 AI 社区内，DeepSeek 的双语基础通常被视为更加正面——是多元主义和语言包容性的标志。通过以多种语言发布其模型和推理数据，DeepSeek 可以说比大多数同行更支持跨文化研究合作。它也促进了 AI 基准测试、训练和评估中更多样化的参与——这些是科学公平的关键要素。

尽管如此，批评者对该模型的内容审核标准提出了质疑。虽然模型设置了防范敏感或暴力内容输出的护栏，但地区规范和政治敏感性不可避免地影响了训练数据和奖励机制。这引发了关于跨文化价值对齐的疑问——模型推理框架中编码的是谁的规范？开源模型是否应被要求明确披露意识形态偏见？

学术界与产业界的抵制

另一争议点源于DeepSeek对传统研究等级制度的冲击。通过普及世界顶级LLM的访问权限，DeepSeek无意中挑战了长期主导 AI 发展的学术-产业联合体。部分大型实验室和商业机构批评DeepSeek侵蚀了专有模型享有的竞争优势。

尽管DeepSeek采用透明化发布，学术界仍对可复现性表示担忧。有人担心在缺乏共享基准治理或引用规范的情况下，DeepSeek衍生模型的泛滥可能模糊科学归属、干扰标准化进程。此外，DeepSeek快速发布的节奏可能助长重数量轻同行评审的倾向，形成先发布、后完善的文化，削弱学术研究的严谨性。

然而，其他人将DeepSeek视为打破学术垄断的必要制衡力量。其模型权重和训练方案让学生、独立研究者及非西方机构得以按自身条件参与顶级研究。支持者认为这并非降低学术标准，而是通过开放工程与学术目标融合提供了互补路径。

关于`RL`方法的伦理辩论

DeepSeek的R1-Zero方法采用稀疏二元奖励构建思维链框架，在对齐研究领域引发热议。虽然其简洁优雅备受赞誉，但批评者指出该方法在伦理边缘案例中缺乏细致考量。二元正确性可能无法捕捉道德推理、社会公平或用户中心对齐中的微妙差异。

例如，仅通过结构格式和正确性评分的R1-Zero，可能忽略输出中的语气、歧义或非预期含义。与Claude的多信号偏好建模或GPT-4o的精细化风格加权奖励系统相比，DeepSeek的RL策略可能显得过于粗糙。

辩护者则强调R1-Zero并非终极方案而是起点。它通过证明思维链推理可从稀疏信号中涌现，推动学界重新思考如何高效且可扩展地实现对齐。这引出了一个关键问题：清晰思考、准确回答的范式是否足以确保 AI 安全交互？抑或对齐必须依赖复杂的偏好网络？

明确的是，DeepSeek的对齐选择——无论是奖励机制还是忽略要素——都将塑造未来模型的推理方式。学界通过批判或复现对这些方法的探讨，将决定其演进方向。

话语特性：非缺陷而是标志

DeepSeek的争议并非功能失调的信号，而是其重要性的体现。它们反映了该模型在 AI 开放性、安全性、公平性和治理等持续对话中的核心地位。虽然没有任何模型可称完美，但DeepSeek勇于发布、迭代并参与公共讨论的姿态，使其在生态系统中占据独特位置。

无论视DeepSeek为民主化力量还是颠覆性变量，其影响都毋庸置疑。通过激发新问题、挑战旧假设，DeepSeek不仅参与 AI 辩论——更助力塑造这场辩论。

总结

本章我们探讨了DeepSeek的基础，这个强大的开源语言模型系列正引领下一波 AI 创新浪潮。我们从了解DeepSeek的本质及其在LLM领域的独特性入手，深入剖析了其技术突破——从训练方法论、架构设计到关键基准测试的卓越表现。同时考察了DeepSeek对全球 AI 生态的广泛影响，特别是在开源获取、创新民主化和地区发展方面。最后章节概述了DeepSeek的不同版本，展现其快速演进过程与能力持续改进。现在你应该能清晰阐述DeepSeek的定义、技术重要性、对全球 AI 领域的影响，以及不同版本如何体现随时间推移的战略设计选择。这些基础认知将成为你实践中评估和使用DeepSeek的透镜。

下一章我们将深入模型内部机制，探究DeepSeek的推理方式、多模态处理方法及其系统的广泛意义与局限。这次探索将帮助你不仅能批判性评估DeepSeek的功能，更能理解其实现方式与原理。

获取本书 PDF 版本与独家资源

扫描二维码（或访问packtpub.com/unlock）。通过书名搜索本书，确认版本后按照页面指引操作。

注意：请保留您的发票。直接从 Packt 购买无需发票。

2. 深入解析 DeepSeek

本章我们将全面剖析DeepSeek的内部机制。你将获得关于DeepSeek独特性及其在 AI 模型演进格局中定位的实践性理解。首先讲解DeepSeek的技术架构、组件构成与训练数据集，随后逐步演示这些组件如何协同形成DeepSeek的推理机制。你还将了解DeepSeek新兴的多模态功能及其在多模态领域中的定位。

本章涵盖以下主题：

DeepSeek的核心架构组件
理解DeepSeek的推理机制
DeepSeek的高级能力

`DeepSeek`的核心架构组件

正如第 1 章所述，DeepSeek建立在 Vaswani 等人（2017 年）提出的transformer基础架构上，以其核心组件作为后续创新的基线。我们将首先讨论关键架构组件。

DeepSeek的关键架构组件包括解码器、多头潜在注意力（MLA）和混合专家模型（MoE），如图 2.1所示。

图 2.1：DeepSeek V3/R1 架构

该架构与训练流程支持模块化与专业化设计，直接增强推理能力。DeepSeek不依赖单一庞大的transformer，而是将提示词路由至专用组件处理。

首先探讨提示词路由机制。

`DeepSeek`中的提示词路由

观察下图，了解DeepSeek如何将提示词路由至各组件。

图 2.2：DeepSeek中的提示词路由

提示词输入并分词后，系统执行以下流程：

检索模块获取相关知识。
接着，重排序模块评估并优化检索到的信息，优先排列逻辑最相关的内容。
轻量级解码器随后利用MoE动态激活专攻数学推理与逻辑推演的子网络，逐步构建响应内容。

例如，让我们看看当处理提示词 奇数之和为偶数 时会发生什么：

DeepSeek 首先从其知识库中检索关于奇偶和的基础原理（此处指奇数和偶数的算术性质）。
重排序模块提升那些正确解释了为何奇数加奇数等于偶数的说明，并丢弃无关或误导性的信息。
解码器随后生成逐步解释，激活专注于算术性质和逻辑推导的专家模块，最终在给出最终答案之前生成一个连贯的论证。

这种模块化流程使得 DeepSeek 能够动态调整推理深度和路径，即使在需要多层次理解的话题上，也能确保输出精确、符合上下文且透明。这种灵活的架构为实现比传统自回归模型更加细致入微的推理奠定了基础。

现在，让我们来讨论解码器模块。

解码器

通过 DeepSeek-R1 (arxiv.org/abs/2501.12948)，DeepSeek 采用了一个 仅解码器的自回归 Transformer，该模型将推理过程和解释作为其输出的一部分。图 2.3 展示了 DeepSeek-R1 解码器的框图。

DeepSeek 的解码器模块由以下部分组成：

嵌入层：此层将输入 token 转换为向量表示。
第 1-3 层：前三层使用以下组件：
- MLA（多头潜在注意力）以实现高效注意力机制，以及 旋转位置编码 (RoPE) 用于处理扩展序列。
- 标准的稠密 前馈网络 (FFN)。
第 4-61 层：这些层使用以下结构：
- 每个模块依然包含 MLA。
- 前馈网络被 MoE（混合专家）层取代：一个路由机制为每个 token 选择一个专门化的 FFN 专家 子集，从而同时提升容量和效率。
预测头：最终输出会传递到一个或多个预测头，以支持下一个 token 和多 token 预测，这是一种提升生成可靠性的技术。

当 token 进入 嵌入层 时，它们会通过由 61 个 Transformer 解码器模块组成的堆栈进行处理。初始的模块是完全稠密的，而第 4-61 模块则采用了 MoE 架构，每个模块利用 MLA 从被激活的专家那里获取响应。然后，该输出通过 预测头 以自回归方式生成下一个 token（或多个 token）。

好的，现在你了解了提示词和 token 如何在解码器中流动。我们知道你现在一定迫切想了解这背后发生了什么。让我们直接深入探讨。

解码器模块的内部机制

DeepSeek-R1 的解码器模块由 61 个 Transformer 层组成，其中前三层是稠密层，其余 58 层采用了 MoE 结构。在每个 MoE 层中，包含一个始终活跃的 共享专家，以及 256 个路由专家，总计在模型中分布了 14,906 个专家。

让我们看看所有处理 token 的组件。

通过路由器进行的 token 处理与生成

对于每个输入或生成的 token，一个轻量级的 路由器 会评估哪些专门的子网络（即路由专家）应该处理该 token。具体来说，路由器 在每一层从 256 个路由专家中动态选择 8 个，再加上始终开启的 共享专家。这种选择并非随机的；它针对每个 token 确定，通过考虑该 token 的上下文，并将其特征与每个专家的 亲和度 或专业特长进行匹配。哪些专家被激活取决于提示词的性质和生成的 token 类型。例如，当生成涉及数学论证或代码的推理过程时，模型会优先激活那些专精于数学推理或编程的专家。同样，对于推理或拒绝行为，使用 功能 token 共振成像 (fTRI) 的分析表明，某些可识别的专家会根据所需响应的类别被持续激活。

最终，检索候选结果得以生成。

在 DeepSeek-R1 中，重排序在将检索候选转化为高质量、易于人类理解的论证方面起着核心作用。通常，模型遵循如下图所示的过程。

以下是它为你的提示词工作的原理：

Top-k 检索：给定一个提示词（例如用户查询或推理任务），初始检索器使用快速、通常基于向量的相似度（例如嵌入空间中的余弦相似度或点积）选择 k 个最相关的候选段落或答案。
跨注意力重排序：这些候选内容、它们的嵌入向量以及原始提示词被输入到一个专门的 Transformer 模块中。该模块使用跨注意力——同时关注查询词和每个候选内容——来计算精炼的相似度分数，并生成重排序后的输出，以指导下一步生成。
最终论证生成：经过对齐和重排序的候选内容（加上中间评分向量）随后在解码器中处理，解码器生成一个推理过程或解释作为其输出的一部分。

例如，考虑以下提示词：

"法国的首都是什么？"

假设 k=3，检索到的前 3 个段落如下：

["巴黎是欧洲的主要城市。", "柏林是德国的首都。", "巴黎有许多博物馆。"]

跨注意力重排序器 通过同时关注查询和答案，给 巴黎是欧洲的主要城市 这段内容打出了最高分，因为 巴黎 与 首都 高度对齐。

在 top-k 检索 和 跨注意力重排序 之后，模型利用对齐后的嵌入向量和 token 序列，通过其标准生成过程生成易于人类理解的论证。

因此，当 DeepSeek-R1 逐步为一个推理过程生成 token 时，被激活的专家确切子集可以从一个 token 动态地转移到下一个。

对于每个 token，会发生以下步骤：

路由器 接收对应于该 token 的隐藏状态，并快速计算其与每个专家的中心表示之间的相似度分数。
然后，它选择分数最高的前 k 个（例如 k=8）专家，并将该 token 路由到仅由这些专家（加上始终活跃的 共享专家）在该层进行处理。
在一系列 token 和多样化的提示词类型中，大量不同的专家子集变得活跃，这使得 DeepSeek-R1 能够调动高度专业化的能力，例如复杂逻辑推理或领域特定知识，同时通过不一次性激活所有专家来保持计算效率。

这种基于 MoE 的架构确保了 DeepSeek-R1 能够构建详细、逐步的解释或推理过程。

生成输出中的每个 token 都能从上下文（先前 token）以及在每一层动态选择的专家网络的独特专长中受益，从而在整个响应过程中实现细致、高效且高度专业化的推理。

自回归 Transformer

DeepSeek-R1 使用自回归 Transformer，其中模型一次产生一个 token，每个新 token 的生成都基于所有先前生成的 token。这种设计被称为 自回归，因为在每个生成步骤中，模型的下一个 token 预测都明确地以迄今为止构建的整个序列为条件，而非依赖未来上下文，从而使得模型能够逐步构建连贯、符合上下文的输出。

一旦初始输入或提示词设定，就会发生以下过程：

Transformer 中的自回归生成

序列在 transformer 堆栈中的注意力（attentive）和前馈（feed-forward）组件中进行处理。

模型随后预测最可能的下一个 token，将其附加到序列中，并为下一个 token 重复此过程——始终完全基于已生成的内容进行条件生成：

Autoregressive generation 通过在每次前向传递中仅生成单个 token 来工作，而不是同时生成一批或一系列 token。
然而，通过重复的前向传递，模型构建了一个增长的序列，每次一个 token，直到满足停止条件（如序列结束 token）。

这种逐步过程对于结构化任务（如逻辑解释或代码）至关重要，因为下一个 token 的含义通常紧密依赖于所有先前输出所设定的上下文。

对于每一步可能生成的 token，模型会考虑其整个词表，潜在地包含数万个 token。

对于每一个新位置，解码器会输出在整个词表上的概率分布。可能的候选词包括基于当前上下文在句法和语义上合理的 token。例如，在短语 The answer is 之后，可能的下一个 token 可能是数字、冒号或表达解决方案的单词。如果在生成源代码，可能的 token 可能包括 def 这样的关键字、数据类型或变量名，并由提示和语义连贯性引导。

多头注意力

DeepSeek-R1 也使用了多头自注意力设计，具体是一种称为 MLA 的优化变体。这使得模型能够有效分析 token 之间的关系。DeepSeek-R1 的多头注意力并不意味着每个头只关注单个 token。相反，多个头并行操作，每个头学习同时关注输入序列的不同部分。

虽然 DeepSeek-R1 transformer 层中的确切头数在现有资料中没有明确说明，但典型的大型 transformer 模型使用数十个头（例如 64 个或更多），在总可用头数和 token 数量之间取得平衡。确实可能存在一种权衡——更多的头可以提供更细粒度的注意力，但计算成本更高，而更少的头降低了复杂性，但可能捕获更少的关系多样性。

多头注意力的核心功能是捕获输入序列中 token 之间的不同类型的关系。每个头计算查询（queries）、键（keys）和值（values）（Q、K 和 V），这些向量相互作用以生成注意力分数，这些分数突出显示了一个 token 在形成其内部表示时应考虑其他 token 的程度。

Queries：在 transformer 中，查询是一个向量，表示模型在输入数据中寻找什么样的信息或上下文。
Keys：键是一个向量，表示输入中每个元素的特征或属性，查询会与这些键进行比较以找到相关信息。
Values：值是一个向量，包含与每个键相关联的内容或信息，它会根据与查询的相似度进行加权和聚合以产生输出。
更多信息可以在 en.wikipedia.org/wiki/Attention_(machine_learning) 找到。

通过此机制，注意力不仅识别现有输入 token 之间的成对关系，还会考虑自回归生成过程中先前生成的 token，因为模型会关注整个前缀。

为了更好地处理 DeepSeek-R1 支持的大序列长度（最高达 128K 个 token），它采用了 RoPE。RoPE 注入位置信息，使模型知道 token 的相对位置，这对于长上下文中的有意义的注意力至关重要。

假设在自回归生成过程中，到目前为止的输入 token 包含短语 The answer is。多头注意力使得一些头可以关注短语 The answer，而其他头则关注最新的 token is，并将这些视角结合起来预测合理的下一个 token，例如数字或单词，从而形成一个解决方案。随着模型生成新的 token，这些 token 会被附加到序列中，注意力机制会扩展到这个增长的序列，动态地适应哪些 token 是最相关的，所有这些都由多个头并行处理。

此外，DeepSeek-R1 通过 MLA 集成了一种效率优化。在这里，模型不是为每个头单独存储完整的键和值矩阵，而是将它们分解为潜在向量，在不牺牲注意力质量的情况下减少内存消耗。

层归一化

层归一化应用于 transformer 层内特定点的 token 隐藏表示（也称为激活值），以提高训练稳定性、收敛速度和整体模型性能。更准确地说，归一化作用于 token 嵌入向量；这些向量在通过网络处理时代表 token 的当前状态。

大型语言模型 transformer 中的激活值是指在模型逐层计算过程中隐藏状态中的特定标量值；这些激活值代表了在不同阶段的变换后的输入特征，在模型如何处理和生成语言方面起着关键作用。

应用层归一化的关键变换包括每个 transformer 层内的关键组件：多头自注意力模块和 FFN（也称为位置网络）。其工作原理如下：

在进入注意力机制之前，通过对嵌入维度的均值和方差进行计算，然后缩放和偏移这些值，使它们具有一致的统计特性，从而对 token 嵌入进行归一化。这种注意力前归一化确保了输入到注意力计算的数据具有稳定的分布，有助于模型更有效地学习。
在注意力输出通过残差连接添加回 token 之后，通常会应用另一个归一化步骤，以保持激活值在进入 FFN 时处于良好状态。
类似地，归一化前和归一化后也发生在前馈层周围，确保涉及线性投影、非线性激活（如 GELU）和另一个线性投影的变换保持稳定的激活统计。

例如，考虑 Calculate the sum 这个提示 token：

当这些 token 进入 transformer 层时，其嵌入在自我注意力计算每个 token 应该关注自身和提示中的其他 token 的程度之前被归一化。
注意力模块产生更新的嵌入，之后层归一化再次稳定这些新向量，然后将它们传递到 FFN。
然后，该 FFN 应用线性权重、非线性和另一个线性变换——归一化支持平滑梯度流并缓解内部协变量偏移的关键变换。
随后，一个最终的归一化步骤准备好输出嵌入，以进入下一个 transformer 层或输出头。
DeepSeek-R1 中的层归一化在每个 transformer 块内的关键内部变换（多头自注意力和前馈层）之前和之后对 token 嵌入进行归一化。这种分层归一化在逐步的、自回归生成过程中保持 token 表示的一致性和稳定性，从而增强了非常长序列的训练可靠性和模型准确性。

MoE 和注意力优化（如分组查询注意力（GQA））的使用使模型能够扩展而不会产生过度的资源需求。

GQA（分组查询注意力）是 Transformer 模型中的一种优化注意力机制，它将查询头划分为若干组，每组共享单一的键值投影，通过在多头注意力和多查询注意力之间进行插值，平衡计算效率与模型性能。

凭借其结构化推理能力，DeepSeek-R1 不仅能生成答案，还能产生解释、验证和论证，这些能力对定理证明、法律和结构化推理等领域至关重要。DeepSeek-R1 中的解码器模块指的是一堆 Transformer 解码器块（共 61 层），构成了其自回归语言建模能力的核心。

接下来，我们将通过示例提示词展示其工作原理。

工作示例

假设用户输入以下内容：

Prove that the sum of two even numbers is even.

该提示词经历以下步骤：

分词：输入被拆分为词元，并嵌入为向量进行处理。典型的词元（以纯文本形式）可能如下所示：

Prove、that、the、sum、of、two、even、numbers、is、even、.
顺序处理：嵌入的词元流经解码器层堆栈。在 DeepSeek-R1 解码器的每一层中，自注意力使每个词元能够使用 key (K)、query (Q) 和 value (V) 表示来关注序列中较早的词元。例如，假设词元 t₂ 希望关注 t₁ 和 t₀；它生成 Q₂，与 K₁ 和 K₀ 进行比较，并使用生成的注意力分数对 V₁ 和 V₀ 进行加权。此操作允许 t₂ 聚合来自前面词元的相关上下文信息。
因果掩码 通过阻止每个词元关注未来的词元来强制自回归属性，确保预测仅依赖于之前看到的输入，并保持从左到右的生成结构。随着输出的生成，模型根据注意力分数选择最相关的先前词元激活，在每个前进步骤中构建理解。

需要区分的是，这种自注意力在单个序列（输入提示词或生成响应的词元）内操作，允许模型捕获内部依赖关系。相比之下，交叉注意力用于单独的、专门的模块（例如交叉注意力重排序），在该模块中，模型允许一个序列（例如提示词）的词元关注另一个序列（例如检索到的候选）的词元，以在进入解码器之前优化相似度分数。因此，解码器本身依赖自注意力进行序列内推理，而交叉注意力则在早期的检索和排序阶段应用。

自回归词元生成：在每次通过解码器后，模型通过评分多个可能的候选来预测下一个词元。例如，在生成前缀 `Let the two even numbers be 2a and 2b, where a and b are integers` 之后，模型会考虑下一个词元的几个顶级候选。假设 top-k 选项如下：
- Their（分数：0.34）
- The（分数：0.21）
- Sum（分数：0.18）
- Difference（分数：0.16）
- Product（分数：0.11）
使用交叉排序，Their 获得最高分，因此被选为下一个词元：

Let the two even numbers be 2a and 2b, where a and b are integers.
Their sum is 2a + 2b = 2(a + b)
Since a + b is an integer, 2(a + b) is even.
Therefore, the sum of two even numbers is even.
```

在每个生成步骤中，解码器权衡其顶级候选，并选择分数最高的词元，在上下文和自回归约束的引导下，逐个词元地构建输出。

呼！这内容确实不少。好的，还有更多。现在让我们进入下一个组件，即混合专家模型（MoE）。

混合专家模型

正如第 1 章所概述的，MoE 路由通过为每个词元选择性地激活专门的子模块，提高了参数效率。在所有阶段，投影和池化方法的进步有助于稳定嵌入范数，从而提升了检索性能。此外，元数据感知的交叉注意力使 DeepSeek 的编码器能够整合上下文文档特征，如标题和类别。

在 DeepSeek-R1 中，MoE 架构在生成过程中为每个词元动态选择性地激活专门的专家子网络，从而高效利用庞大的模型容量。处理提示词时，模型的路由器会评估该词元的上下文嵌入，并将其与代表每个专家专业领域的学习到的质心进行比较。基于这种相似性，它激活一小部分专家，通常在每个 MoE 层中激活 8 个路由专家（共 256 个）中的少数几个，外加一个共享专家，使计算适应词元的语义和功能需求。

如果你想知道这个专家选择是如何发生的，请继续阅读。

专家是如何初始化和配置的？

DeepSeek-R1 的架构在每个 MoE 层定义了数百个路由专家（例如，名词专家、数字专家、网页专家等），通过可学习的质心向量进行区分。这些质心初始为随机向量，并在预训练过程中，根据训练数据中遇到的模式，逐渐被塑造成专注于不同语言或语义领域的专家——例如识别数字、识别命名实体或解释推理线索。像数字专家或网页专家这样的专家并非硬编码；相反，这些专业化能力是不同专家在训练过程中针对某些词元类型或上下文被反复激活而涌现出来的。

例如，考虑以下部分提示词词元序列：Prove that the sum of two even numbers is even. 在自回归生成过程中，当模型遇到与数学推理相关的词元（如 even numbers、sum 或 prove）时，专注于数学运算和逐步逻辑的专家会被强烈激活。

图 2.5: DeepSeek-R1 中的 MoE 路由

路由器选择那些专业化程度与每个词元角色最匹配的专家：数学专家处理数字操作和证明步骤，而推理专家则被用于需要论证或逻辑解释的词元。

如果提示词转向编程，例如输出 def 或特定变量名，路由器将优先激活在代码生成方面训练过的专家。通过这种方式，上下文敏感路由确保模型利用与当前任务最相关的专家知识。

门控如何计算激活哪些专家？

当像 Prove that the sum of two even numbers is even 这样的提示词被分词并输入模型时，每个词元的嵌入（其隐藏状态）在其经过的每个 MoE 层都会由 MoE 门控网络 处理。对于每个词元在每一层，发生以下情况：

词元的嵌入与每个专家的质心向量（预先学习）通过点积进行比较，产生 亲和力分数，衡量该词元特征与每个专家专业领域的匹配程度。
这些亲和力分数通过一个归一化的门控函数。DeepSeek-R1 通常使用基于 Sigmoid 的机制，并可能带有用于平衡专家使用的偏差调整，而不是简单的 softmax。
门控函数 为该词元选择亲和力分数最高的 top-K 专家（例如 K=8），外加一个始终激活的共享专家。
只有被选中的专家处理该词元；它们的输出由门控分数加权，并组合形成该词元在该层的输出表示。

专家是如何被激活的？

我们现在将回到之前的示例提示，观察专家是如何被激活的。

考虑提示 Prove that the sum of two even numbers is even 中的 token even。在 MoE 层，路由器会将 even 的嵌入向量与所有专家的质心进行比较：

如果 even 与数学推理专家的质心高度对齐，则该专家的亲和度得分将较高，从而成为被选中的 top-K 专家之一。
诸如 sum 和 prove 之类与逻辑推理和证明构建相关的 token，可能会激活推理或证明逻辑专家。
如果提示上下文中包含编程语法或变量名，编程专家则会被优先路由。但就本数学证明而言，此类专家的相关性较低。
即使像 that、of 或 two 这样的连接词 token，也会独立经历此门控过程，从而在整个提示中形成动态的、针对特定 token 的专家路由路径。

这种逐 token 门控机制生成了一个独特的激活路径，贯穿于专家网络，并针对每个输入和生成的 token 量身定制。它能高效利用专业化的专家知识，精确满足需要，从而最大限度减少计算开销。这种动态路由也支持了 DeepSeek-R1 产生清晰、分步解释和推理输出的能力，因为它能在恰当时机激活最相关的专家。

另一方面，传统的顶尖（SOTA）大语言模型（如 GPT-4o 或 Claude）大多采用密集架构，这意味着所有参数都会用于每个输入，导致资源消耗巨大。DeepSeek 在达到 SOTA 级别性能的同时，其有效激活参数数量却少得多，因此效率高得多。此外，元数据感知的跨注意力层融合了文档标题或类别等上下文特征，丰富了影响专家路由的表示，并增强了检索和相关性。与通过每一层的每个神经元路由输入不同，每一层包含多个专家子网络，但每个 token 只激活少数几个专家（例如，64 个专家中仅激活 8 个）。这在保持容量的同时，大幅降低了计算负载。

以下公式展示了专家是如何被激活的：

其中，门控函数选择使用哪些专家（是专家总数；通常只激活前两个）。

接下来，我们将关注点转向 MLA。

多头潜在注意力：面向长上下文的高效内存注意力

MLA 最早在 DeepSeek-V2 中实现，后来在 DeepSeek-V3 中得到改进，旨在减少推理时的内存使用，同时不损害输出质量。

标准 Transformer（以及许多顶尖模型如 LLaMa 3 和 GPT-4o）使用多头注意力，并为每个 token 保留完整的键值（KV）缓存，这在处理长上下文时会导致极高的内存需求。DeepSeek-V2、V3 和 R1 引入了 MLA，将 KV 缓存压缩为每个头的一个紧凑潜在表示，在保持或提升质量的同时，将推理内存减少了超过 90%。

MLA 通过引入一种对 KV 缓存的低秩压缩形式，取代了标准的多头注意力机制。MLA 不是为每个 token 在推理时存储大量的键和值记忆张量（这会导致巨大的内存使用，尤其是长序列时），而是将这些表示吸收到一组更小的、经过学习的潜在向量中。在每一层，注意力计算不是直接基于完整缓存进行，而是通过这些压缩的潜在表示进行，它们保留了足够的上下文信息以实现高质量生成。

额外的改进包括选择性应用旋转位置嵌入和先进的权重吸收技术，使该机制能够匹配甚至超越模型原始的推理质量，同时通常将推理缓存大小减少 90% 以上。这一创新是 DeepSeek-R1 能够高效处理长上下文任务，并大规模提供经济高效的大语言模型推理的关键原因之一。

让我们看看这是如何工作的。在 MLA 中，不是存储个键/值（其中 N 是序列长度），每个头的 KV 都被投影到一个更小的潜在空间：

在注意力计算过程中，查询（query）关注的是这个潜在表示，而非完整序列——这在速度和内存方面带来了巨大提升。

LLaMa 3 和许多 GPT 模型使用 GQA 来减小 KV 缓存大小，但经验表明，MLA 在建模性能和效率上更胜一筹，这得到了基准测试和出版物的证实。

对于提示 Prove that the sum of two even numbers is even，MLA 通过将外部上下文（如文档标题或类别信息）融入注意力机制，增强了模型的理解能力。这些外部上下文提供了额外的信号，帮助模型基于相关元数据更好地解释和专化其响应。在一个注意力头中，键和值向量首先从输入 token 嵌入中计算得出。这些向量最初是高维的（例如 1024 维），然后使用学习到的投影矩阵被投影到一个更小的潜在空间（例如 64 维）。这种降维允许更高效的计算，并将注意力集中在最相关的特征上。投影后的键代表用于与查询（Q）匹配的上下文特征，而投影后的值则承载了被聚合的丰富信息。MLA 通过同时在多个潜在子空间中关注 token 序列和注入的元数据来实现这一点，使模型能够在生成证明时动态权衡和整合相关的外部知识。这个过程使得对诸如两个偶数之和仍为偶数之类的断言，能够进行更准确的、具备上下文感知能力的推理。

还有一些其他关键方面使得 DeepSeek 成为最具竞争力的模型之一。请稍安勿躁。

DeepSeek 实现高效且上下文感知响应的特性

除了上述架构创新，DeepSeek 还采用了其他几种技术来确保效率和上下文感知的响应。让我们来探讨一下。

FP8 训练与精度控制

DeepSeek-R1 是首批在预训练和推理中实施真正的、大规模 FP8（8 位浮点数）混合精度策略的大语言模型之一。FP8 是 DeepSeek 在速度和内存效率上超越早期格式（如 FP16 或 BF16）的关键因素之一。然而，FP8 的成功部署需要精心设计的方法来应对低精度算术中常见的数值不稳定性和收敛性下降风险。

其精心设计体现在 DeepSeek 的混合精度训练框架中：在该框架中，大部分计算密集型操作，例如前向和反向传播中涉及的线性变换（矩阵乘法或 通用矩阵乘法（GEMM）操作），都以 FP8 精度执行。对于每个线性算子（无论是前向传播、梯度反向传播还是权重更新），模型权重和激活值都被量化并在 FP8 中处理，计算吞吐量相比 BF16 翻倍，内存使用量减半。

关键的是，为了保持精度，实际的乘积累加（求和）是在 FP32 中执行的。激活值和梯度则以 BF16 格式缓存，以兼顾内存效率和稳定性。

DeepSeek-R1 的独特之处在于其对 FP8 的选择性应用

通过详细的实证研究，团队识别出那些若进一步降低精度将损害模型稳定性或学习动态的特定模块。例如，他们在嵌入层、序列输出头、MoE 门控模块、层归一化以及注意力算子上保留了更高（BF16 或 FP32）的精度，这些部分对量化噪声敏感，需要更强的数值鲁棒性。只有稠密核心（GEMM）和某些内存密集型路径被安全地量化至 FP8。这种选择性方法确保了网络的绝大部分能从 FP8 的效率中获益（从而能在相同的计算范围内扩展到更大的模型规模），而关键计算则始终保留模型收敛和高保真输出所需的精度。优化器状态、主权重和权重梯度均以 BF16 或 FP32 存储，复杂的分片与并行技术（如 DualPipe 流水线并行）进一步降低了内存和通信开销。

在 DeepSeek-R1 的预训练期间，用于令牌转换和中间计算的矩阵乘法（例如在数学提示词中，如Prove that the sum of two even numbers is even），均采用 FP8 精度执行以最大化计算效率。与此同时，关键操作，包括 MoE 门控逻辑、层归一化和最终输出生成，则利用更高精度以确保随着序列增长时的准确性和稳定性。这种针对模块的特定精度方法在效率与可靠性之间取得了平衡，使 DeepSeek-R1 能够在单次运行中大规模训练数万亿参数的模型，同时不损害输出质量或训练稳定性。这一解释突显了理解模型如何在前所未有的规模上实现高效稳定训练时，至关重要的实际权衡与设计决策。

用于策略解码的多令牌预测

大多数 Transformer 语言模型被训练为自回归模型，旨在最大化给定当前序列下下一个令牌的概率：

这意味着模型学习一次仅预测一个令牌，并始终以所有先前令牌为条件。在推理时，模型逐个令牌（逐令牌）生成文本，采样或贪心地选择最可能的下一个令牌。

DeepSeek 的 V3 模型引入了多令牌预测（MTP），其灵感源自推测性解码。在这里，模型不仅学习预测下一个令牌，还并行预测未来的多个令牌。以下是 DeepSeek 采用的 MTP 联合似然目标函数：

MTP 不再仅仅关注紧邻的下一个令牌，而是联合优化k个未来的令牌。这使得训练目标与更高效的解码相一致，并能更好地捕捉长程依赖关系。

MTP 的部分优势包括：

并行化训练：通过一起预测多个令牌，训练可以更快、更高效。
增强上下文建模：模型学会预测更广泛的模式并提前规划，鼓励全局连贯和一致的推理。
加速推理：在生成时，模型可以在一次前向传播中验证或提议多个令牌，从而大幅提升吞吐量。

DeepSeek-R1 的群组相对策略优化（GRPO）通过在微调期间实现更精确且感知上下文的奖励分配，代表了对标准 RLHF 的重大超越。让我们详细探讨这一点。

GRPO 与高级 RLHF

在 DeepSeek 的定制 GRPO 系统中，训练期间，策略模型为每个提示生成多个响应——通常是一个固定数量，如 4 到 8 个样本，尽管这个数字可能因训练阶段或提示复杂度而异。每个采样输出都基于严格的基于规则的标准被分配一个二值奖励（可接受或不可接受）。即使组内某些响应不正确并获得零奖励，它们仍被包含在组内以计算平均奖励。这意味着不正确输出的存在会降低该组的平均奖励基线，从而有效地惩罚质量较差的响应（相对于更好的响应）。在实践中，通过基于规则的检查（例如正确性和简单的格式约束）计算标量奖励，这通常最终简化为一个有效的二值信号；GRPO 在减去组均值中心化后使用这个标量奖励，以提升高分样本并抑制低分样本。

然后，通过将每个响应的二值奖励与此组平均值（并按组标准差进行归一化）进行比较，计算出模型对该响应的优势值，从而引导策略更新朝着优于组均值的响应方向进行。这种相对评分机制鼓励了持续改进，同时考虑了相同提示下所有样本的输出质量差异。

使用 GRPO 时，当提示Prove that the sum of two even numbers is even被给出，模型会并行生成一组候选响应。例如，假设候选响应及其奖励如下：

候选 A：Let the two even numbers be 2a and 2b. Their sum is 2(a + b), which is even.（奖励：0.85）。
候选 B：Two even numbers added together result in an even number because even plus even equals even.（奖励：0.75）。
候选 C：Since even numbers are multiples of 2, their sum is divisible by 2, so it's even.（奖励：0.80）。

GRPO 首先计算该组的平均奖励：

然后，对于每个候选响应，它通过将其奖励减去此平均奖励来计算优势值，并可选择性地按组标准差进行归一化以平衡尺度差异，如下所示：

优势 A = 0.85 - 0.80 = +0.05
优势 B = 0.75 - 0.80 = -0.05
优势 C = 0.80 - 0.80 = 0.00

通过基于这些相对优势（而不仅仅是绝对奖励）进行优化，模型学会偏好在给定候选集合中表现积极突出的响应，从而提高了整体生成质量以及对此类证明等复杂提示的相关性。

DeepSeek-R1 中的奖励可以基于多个标准：逻辑正确性、连贯性、对提示的忠实度、完整性，甚至是对数学原理或风格的遵循——简而言之，任何可以通过规则、启发式方法或基于结果的评分进行评估的属性。例如，在微调推理任务时，奖励可能对提供逻辑思维链（CoT）或精确中间步骤的输出更高，而不仅仅因为产生了最终正确答案。

通过使用每个组的平均奖励作为特定上下文的基线，GRPO 减少了奖励操纵和噪声，鼓励策略在每种情况下偏好相对更好的响应。DeepSeek-R1 还在损失中添加了KL 散度惩罚（hanj.cs.illinois.edu/cs412/bk3/KL-divergence.pdf），以稳定更新并维持期望的行为。这种精细的、基于群组相对的方法实现了对学习的精确控制，提高了清晰度、事实准确性以及与用户意图的一致性——尤其是在复杂的多步推理任务中。GRPO 是 DeepSeek-R1 在产生详细、可解释且与用户对齐的输出方面表现强劲的基石。

思维链推理：CoT 与长 CoT

DeepSeek 的训练，特别是使用特殊提示符（``，<answer>）时，针对逐步的、可解释的推理进行了优化。它在长链式思维（CoT）任务上尤其强大——这在 2025 年美国数学奥林匹克 LLM 挑战赛中获胜得到了证明。

考虑以下提示：

理解 DeepSeek 的推理机制

解释为什么两个奇数的和是偶数？

这是 DeepSeek 的回答：

 设这两个数为 2a+1 和 2b+1。
 它们的和为 2a+1+2b+1=2(a+b+1)，是偶数。

其他先进的 LLM 也能生成思维链输出，但 DeepSeek 表现卓越，这得益于其架构和训练偏向于结构化解释（更多内容见下一节及第 4 章），在竞争性基准测试中常常表现更优。

DeepSeek 的思考过程

回想 第 1 章，DeepSeek 的数据集构建涵盖了丰富多样的来源，经过精心筛选以覆盖推理、编码、科学和通用世界知识。对于编程任务，DeepSeek 利用了来自 GitHub 官方数据集等仓库的大规模开源代码，以及数据集合（如 The Stack (huggingface.co/datasets/bigcode/the-stack) 和 CodeContests (huggingface.co/datasets/deepmind/code_contests)）中精心挑选的论坛和库。对于数学和科学推理，模型在基准数学数据集（如 GSM8K 和 MATH）、科学论文语料库（如 arXiv 和 PubMed Central）以及逻辑谜题和教科书问题集合上进行训练。

至关重要的是，DeepSeek 的训练数据包含了明确的思维链数据集，尤其使用了来自 CoT Collection (github.com/kaistAI/CoT-Collection)、MathQA (math-qa.github.io/) 等资源的思维链标注样本，以及精心挑选的人类推理过程。通过受 Flan Collection (github.com/google-research/FLAN)、Alpaca (github.com/gururise/AlpacaDataCleaned) 等提示启发的合成数据集，以及为自我指令或对话监督发布的数据集，提供了指令和叙述的多样性。为了获得强大的多语言和领域通用能力，DeepSeek 包含了大量的维基百科 (www.wikipedia.org/)、经过质量筛选的 Common Crawl (commoncrawl.org/)、古登堡计划 (www.gutenberg.org/) 以及从法律文件（例如 Pile-CC 的法律部分 pile.eleuther.ai/）到医学文献（如 PMC-PubMed (pmc.ncbi.nlm.nih.gov/) 和 MedQA (huggingface.co/datasets/bigbio/med_qa)）等特定领域语料库。

数据集的选择

DeepSeek-R1 应用 测试时扩展，在推理期间分配额外的计算资源，从而能够对输入提示进行逐步深入推理，以产生更高质量的输出。模型不会生成单一直接答案，而是执行一系列推理过程，类似于人类系统 2（深思熟虑、费力的推理）的逻辑推理，在最终确定响应前迭代评估可能的解决方案。这种方法由其 MoE 架构支持，该架构将每个 token 并行路由到多个专家，并利用先进的硬件加速实现快速的多专家评估。

在运行时，随着更多计算资源投入推理，模型的输出质量会提高，特别是在数学、编码和逻辑分析等复杂任务上。DeepSeek-R1 在现代 GPU 集群上每秒可处理数千个 token，其提供的实时答案需要紧密连接的高性能基础设施来最大化 token 吞吐量和专家间通信。这种扩展机制有效地增强了模型长时间思考、检查和完善其推理的能力，并提升了在需要大量计算和精确逻辑的任务上的基准性能。

总之，这种广泛、精心策划的数据混合赋予了 DeepSeek 进行推理、编码、回答多步问题以及跨语言和领域泛化所需的广度和深度。如需更详细的分解或每个数据集的文档指引，你可以参考相关的代码库或学术数据集排行榜。

哇！这么多内容确实不容易消化。我们建议你休息一下，再回来看看这些概念如何共同构成 DeepSeek 的推理机制。

测试时扩展

使 DeepSeek 与众不同的关键在于它对推理的重视。语言模型中的推理是指解决多步问题、理解因果关系、执行符号操作或基于隐含上下文推导答案的能力。

DeepSeek 的架构创新并非仅提供孤立的性能提升，而是组织内部表示、在提示内维持扩展上下文，并以更审慎且符合上下文的方式构建其响应——从而在不同领域实现更快、更可靠的推理。

DeepSeek 在提示工程、训练设计和模型对齐方面的做法，强调向明确的结构化推理有意识地转变。DeepSeek 并非仅仅依赖模型在幕后推断逻辑链，而是通过引入结构化的提示格式（例如明确的 `` 和 <answer> token，或指导模型展示其工作过程的指令微调模板）来鼓励中间推理步骤。这与对话意义上使用系统提示不同。这种方法依赖于精心设计的、特定任务的提示格式，这些格式被明确嵌入模型的训练和微调数据集中。

在数据准备阶段，示例会使用专用 token（如 ``）来标记中间推理步骤，使用 <answer> 来标示最终结论。这些 token 作为清晰的语义线索，指导模型生成详细、逐步的解释，而不是立即给出最终答案。

这涉及以下步骤：

训练语料库包含大量标注数据，其中人类撰写或模型增强的推理过程伴随着最终响应，并被分解为逻辑推理步骤。
然后训练模型不仅要预测正确的最终 token 序列，还要在输出序列中重现中间推理步骤。
损失函数应用于整个 token 序列，包括这些推理步骤，以鼓励忠实再现结构化思维。
此外，在微调和强化学习阶段（例如使用 GRPO），评估指标明确奖励输出在清晰度、连贯性、事实一致性和推理完整性方面的表现，而不仅仅是答案的正确性。这鼓励模型重视其思维链的可解释性和内部一致性，强化使推理过程透明且可验证的行为。

因此，DeepSeek 的训练流程建立了一种运行范式，其中嵌入了推理标记的结构化提示成为学习目标和输出评估不可或缺的一部分。这种端到端的设计确保模型能够通过明确的中间步骤可靠地展示其工作过程，产出用户可以检查逻辑有效性和可信度的输出，而不仅仅是不透明的最终答案。

那么，DeepSeek 是如何思考的？让我们立即深入探究。

DeepSeek-R1-Zero 的训练与推理机制

DeepSeek 的开发中有一个显著现象：在训练 DeepSeek-R1-Zero 时——这是一个仅通过强化学习（RL）训练、未使用任何监督微调数据的变体——模型开始展现出未被明确编程的涌现推理能力。这一过程为理解语言模型如何仅通过 RL 发展推理技能提供了新的见解。

R1-Zero 的训练设置相对简单。DeepSeek 的工程师在训练和对齐阶段设计了一个固定的提示结构，鼓励模型在提供答案之前明确地对问题进行推理。

这其中有三个关键因素在起作用：思考与回答块、GRPO 以及基于训练样本的 RLHF。我们来逐一讨论。

思考与回答块

虽然确切的系统提示仍然是专有的，但可获得的 tokens 和训练痕迹表明，提示始终以包含一个 `` token 开始，后接一个 <answer> token 为结构，引导模型在生成最终响应前生成结构化的中间推理步骤。这种思考与回答之间的明确区分——无论是直接嵌入系统指令还是通过监督示例体现——就像一个潜在的系统级指令，促使模型将推理视为一个基本的、首要的步骤，而非可选行为。

DeepSeek 模型（尤其是 R1 系列）在多步任务（如数学问题求解和法律论证）中表现出强大性能，因为这些问题需要将问题分解为逻辑步骤以确保准确性和透明度。每个模型响应都遵循一个两阶段结构，包括一个包含中间推理步骤的块和一个包含最终结论的 `<answer>` 块。这种在训练和评估期间引入的格式，强化了决策前的显式推理，尤其适用于需要逐步逻辑的任务。两者之间的边界由特殊 token 定义，如和 <answer>，它们作为格式和行为的分隔提示。

这种明确的提示结构有助于使模型的推理变得透明、可验证和可扩展，尤其适用于数学、法律和多轮决策等复杂领域。我们将在第 3 章中详细讨论提示格式。

考虑以下提示示例：

"两个偶数的和总是偶数吗？解释原因。"

模型的响应如下：


让这两个偶数表示为 2a 和 2b，其中 a 和 b 是整数。
它们的和是 2a + 2b = 2(a + b)，可被 2 整除，因此是偶数。
</think>
<answer>
是的，两个偶数的和总是偶数。
</answer>

让我们看看模型生成的响应结构。`` 部分是模型被鼓励展开逐步推理的地方，而 <answer> 部分则简洁地给出了最终响应。

接下来，GRPO 机制开始用于模型响应，我们将在下文讨论。

使用 GRPO 评估响应质量

在与 GRPO 进行训练和对齐期间，`` 部分的质量通过一个群组相对奖励机制进行评估，该机制实现了对中间推理步骤的精细控制。GRPO 不为每个模型输出分配绝对奖励，而是将为同一提示生成的多个候选响应作为一个群组进行比较：

对于每个响应，包括其 中间推理块，所分配的奖励是相对于该群组中其他候选者的表现而言的。这种相对优势是通过减去群组的平均奖励并按其标准差归一化来计算的。这种设置允许基于逻辑连贯性、一致性、事实正确性、格式和可解释性等标准，独立于最终答案的正确性来明确评估 部分的质量。
通过奖励那些包含清晰、逐步、结构良好推理的输出，模型学会的不仅仅是猜测答案，而是以逻辑和透明的方式构建答案。具体来说，在 GRPO 微调期间，模型会为每个提示生成多个候选解释。
一个评分函数，通常是启发式方法和学习到的奖励模型的加权组合，会为诸如 `` 块的清晰度和正确性等属性分配分数。
学习到的奖励模型通常包括在人类偏好数据上训练的神经排序器或奖励预测器，以评估生成解释中的推理质量、事实准确性和连贯性。

GRPO 使用这些分数（如前所列）来计算群组相对优势，指导策略更新，使模型朝着产生更好中间推理的方向调整，并明确符合人类对齐的质量标准。

以下是奖励函数的工作原理：

准确性奖励 仍然是一个二元信号，当最终答案正确且格式正确（例如，包含 `` 和 <answer> 块）时分配为 1，否则为 0。此评估利用了自动检查，并在必要时对照早期阶段建立的基本事实答案进行手动验证。
一致性奖励 旨在量化模型在整个推理步骤中保持单一语言风格和逻辑连贯性的均匀程度，防止在单个响应中混合语言或使用不连贯的措辞。概念上，该奖励可以表示为一个标量函数。

一致性奖励，记为，被定义为一个函数应用于模型的响应：

的值在一个特定范围内（例如，从 0 到最大值 10），其中函数评估响应文本的内部语言一致性和连贯性。例如，可能执行以下操作：

计算响应中主流语言与其他语言相比的 tokens 或句子比例。
惩罚风格基调或措辞的突然变化。
应用连贯性指标来确保推理链的逻辑流畅性。

在 RL 中指导策略优化的组合奖励计算为准确性奖励与加权一致性奖励之和：

其中，是衡量最终响应正确性的二元或标量奖励，λ 是一个权重系数，用于平衡一致性奖励相对于准确性奖励的效果。

这个组合奖励在 GRPO 框架内使用。GRPO 通过比较每个采样响应的总奖励与群组平均奖励来计算相对优势，鼓励模型生成不仅准确且结构良好，而且在推理过程中语言一致、逻辑连贯的输出。

虽然这种采样与比较过程增加了训练期间的计算工作负载——因为每个提示需要生成并评估多个候选补全——但其影响严格限于训练阶段，一旦部署开始，不会影响推理速度。在推理阶段（处理真实用户请求时），每个提示只需一次模型生成，与常规大语言模型无异。因此，DeepSeek在训练期间实现了强大的推理对齐和样本效率，但在生产环境中以标准且高效的推理速度提供响应，不会因GRPO的训练时多采样方法而产生额外开销。GRPO无需辅助价值网络或多次生成，使其具备快速、稳定且易于实施的特点。

基于规则的 RLHF

训练和评估过程中使用的示例问题涵盖多个领域，从简单的算术和符号逻辑到代码生成和物理问题，展示了模型在不同推理任务中的通用性。

DeepSeek的基于规则的 RLHF 信号以相对直接的方式应用，其中每个响应或推理步骤根据固定的、预定义的逻辑标准被标记为“可接受”或“不可接受”，而非通过复杂的奖励模型学习获得。根据技术描述，这一过程并非传统意义上的完全人工标注，而是依赖基于规则的启发式方法，自动根据设定的逻辑评估输出的正确性和格式。这种方法减少了人工标注每个样本或步骤的耗时需求，使其比通常涉及学习奖励模型或人工排名的 RLHF 更具可扩展性且资源消耗更低。

然而，仍需一些初始人工工作来定义构成可接受步骤的精确逻辑规则和阈值，这是一项重要的前期设计任务。二元、基于规则的反馈的简单性也意味着训练期间每个数据点所需的标注员或标注工作量更少，因为评估是自动化的且主观性较低。这与标准的 RLHF 工作流程形成对比，后者需要人工标注员花费大量时间比较多个响应或分配标量奖励分数，这可能成本高昂且速度缓慢。

DeepSeek的方法平衡了手动与自动任务：前期通过人工逻辑设计创建基于规则的评估器，结合训练期间对模型输出的自动二元标注，从而获得清晰的接受/拒绝信号，无需繁重的人工标注负担或使用复杂的已学习奖励函数。这使得对齐过程更高效且可扩展，同时为强化正确的推理步骤保留了有意义的反馈。即使有时缺乏高级已学习奖励模型可能实现的细粒度对齐，该反馈管道仍变得更具可扩展性和高效性。该系统评估输出的正确性和格式合规性。

`DeepSeek`如何处理复杂场景

考虑一个你希望DeepSeek识别代码中错误的场景。

在此场景中，一段代码片段能正确编译但产生错误结果，DeepSeek采用多种训练和对齐策略来识别并纠正错误。例如，考虑实现一个对两个数字求和的函数。模型最初可能生成以下代码，该代码可编译但返回的是乘积而非和：

def sum_two_numbers(a, b):
    # Incorrect implementation: returns product instead of sum
    return a * b
# Testing the function
result = sum_two_numbers(8, 6)
print("Initial output (incorrect):", result)  # Outputs 48 instead of 14

训练过程中，DeepSeek的反馈机制检测到预期输出与实际结果之间的逻辑差异。利用这些信号，模型迭代优化其输出，最终得到修正后的实现：

def sum_two_numbers(a, b):
    # Correct implementation: returns the sum of two numbers
    return a + b
# Testing the corrected function
corrected_result = sum_two_numbers(8, 6)
print("Corrected output:", corrected_result)  # Outputs 14 as expected

该修正过程由针对性奖励模型和错误诊断支持，指导模型识别并修正错误。因此，DeepSeek生成的代码不仅语法有效，而且功能准确可靠。

推理机制也受DeepSeek训练方式的影响。让我们一探究竟。

`DeepSeek`训练

DeepSeek稀疏但定义明确的基于规则的反馈被选择性地应用于通过数据驱动优先级策略识别出的训练样本。样本的选择基于提示复杂性、是否存在多步推理、事实或逻辑难度，或对齐与正确性至关重要的决策关键上下文等标准。

例如，涉及多步数学证明、带正确性检查的代码生成或复杂法律推理的提示会被标记为接收显式基于规则的评估信号。相比之下，更简单或更常规的提示，如直接的事实检索或常见对话查询，通常被跳过或分配更粗糙的间接奖励，从而减少不必要的计算负载。一旦这些奖励信号通过策略优化（例如通过GRPO）指导模型参数的更新，学习便直接编码到模型权重中，意味着模型能将其学习泛化到类似提示，即使并非每个案例都提供了显式反馈。

训练过程中，随着R1-Zero迭代处理复杂推理任务，观察到一种涌现行为：模型生成越来越长、越来越详细的中间推理步骤。这并非通过与响应长度相关的奖励明确激励；相反，底层的GRPO框架自然地偏好提高正确性的推理路径。从数学角度看，更长的分步推理序列倾向于获得更高的相对奖励，因为它们更可靠地展示了中间步骤的逻辑连贯性和正确性。这些奖励受奖励模型中加权组件的影响，这些组件优先考虑准确性、完整性和清晰度等因素，对推理链中持续的逻辑一致性赋予更大权重。

本质上，通过产生扩展的思维链，模型最大化其在GRPO的组相对评分函数下计算的预期奖励信号，该函数定义为归一化的优势：

此处，是候选的奖励，是候选组的平均奖励，是标准差。这个形式化的、基于优化的框架解释了推理细节的增加——DeepSeek有效地学会在能够提高相对奖励优势并增加正确最终答案可能性时，详细阐述其内部思维过程。因此，模型更长的推理输出是对奖励结构的战略性响应，而非随机或无法解释的行为。

模型在训练期间未获得显式监督的思维链标签。相反，其输出基于最终答案的正确性和是否遵循包含中间推理步骤的规定结构化格式（例如 `` 和 <answer> 块）进行评估。

由于模型接触过中间推理步骤与获得奖励存在因果关联的示例，它开始倾向于先生成思考再输出答案的生成路径，从而形成了一种自组织的内部推理策略。重要的是，这种行为并未出现在早期仅通过标准监督微调训练的DeepSeek模型中。那些版本可能得出正确答案，但其输出缺乏可解释性，在多步骤任务中表现不佳，且未能很好地泛应用于更具挑战性的基准测试，如AIME、MATH和GPQA。相比之下，R1-Zero在没有使用任何监督CoT数据的情况下，仅通过这种纯奖励对齐方式就已开始超越这些早期模型。

这才是真正的顿悟时刻——模型能够发现可解释且有效的推理模式，不是因为它模仿了标注的解释，而是因为它被激励去生成持续有效的结构化输出。它挑战了手写推理规则对于学习推理是必需的这一观念，并为完全实现的DeepSeek-R1模型奠定了基础。

训练用冷启动数据

DeepSeek团队投入大量精力生成高质量的结构化推理数据——不是通过手动标注数千个CoT示例，而是利用R1-Zero的无监督推理能力。一旦R1-Zero开始生成遵循CoT格式且基本准确的输出，就收集了大量这类模型生成的响应，作为冷启动监督训练数据。

这些输出的正确性在可能的情况下通过程序检查、测试用例或逻辑验证规则，将生成的最终答案与已知标准答案进行比较来自验证。当自动验证不足或存在歧义时，人工审阅者会进行选择性手动验证，以确保质量和准确性。虽然手动检查需要额外的时间和精力，但大部分验证都利用了自动化程序，使整个过程比完全手动标注更高效且可扩展。这种混合验证策略使研究人员能够从模型自身的输出中高效地引导出高质量的CoT标注数据，而无需承担高昂的标注成本。尽管这些数据是模型生成的，但它们实际上已成为标注数据：每个样本都包含原始提示（隐式问题）、一个包含推理步骤的``块，以及一个包含最终响应的<answer>块。

例如，考虑以下提示：

"3 与 2 的和的平方是多少？"

模型输出如下：


3 + 2 = 5.
5² = 25.
</think>
<answer>
25
</answer>

这个结构化的数据集为下一阶段奠定了基础：使用R1-Zero生成的语料库来微调DeepSeek-V3的一个检查点，从而将R1-Zero涌现的推理能力转移到更大的主干模型中。在这里，R1-Zero同时扮演了学生和教师的角色——最初在强化学习期间通过简单的、基于规则的奖励信号进行学习，随后通过生成蒸馏示例来指导自身的训练。虽然这个过程与GRPO的奖励驱动优化有相似之处，但R1-Zero的方法特别专注于强化符合逻辑推理和正确性的行为，而不仅仅是最大化群组相对优势。

与使用监督损失来匹配教师输出的传统知识蒸馏不同，DeepSeek利用奖励信号来明确优化结构化、分步骤的推理质量和对输出格式的遵循，这是标准蒸馏损失难以单独捕捉的。以这种方式使用强化学习有助于模型动态探索和完善推理策略，发现最大化清晰度和正确性的输出。这些输出随后作为可靠的教师示例用于后续的监督微调，在补充GRPO的群组相对奖励框架的同时，有效地引导出性能的提升。

以下是训练过程中发生的情况：

DeepSeek-R1的训练始于预训练，以建立基础的语言理解和专家专业知识。
在此基础上，模型在CoT数据上进行进一步训练，其中RL过程通过一个额外的连贯性奖励得到了增强，该奖励与原始的二元正确性和格式奖励并存。
虽然原始奖励函数主要根据最终答案的正确性和对 + <answer>输出结构的遵循来分配二元的接受/拒绝信号，但连贯性奖励鼓励模型生成响应中的语言一致性和连贯性。这两个奖励通过简单相加结合，形成用于策略优化的最终奖励。
连贯性奖励是按每个响应计算的，不会显式存储或与为同一提示生成的先前响应进行比较。相反，它使用基于规则的启发式方法和学习模型的组合来评估每个单独输出的内在连贯性，通常分析语言统一性和风格一致性等因素来分配连贯性奖励。例如，如果提示是英文，响应会因为保持英文而不引入其他语言或不一致的表达而获得奖励。

引入这个奖励是为了解决早期模型（如DeepSeek-R1-Zero）中混合语言输出的问题。消融研究表明，虽然连贯性奖励有时会略微降低原始任务性能，但它通过鼓励语言连贯的输出，极大地提高了人类可读性和用户体验。

这个额外的信号确保模型的推理风格在所有步骤中保持连贯——不仅逻辑上合理，而且风格上一致，语言上流畅。

连贯性奖励函数结合了一个二元信号和基于语言的度量来评估每个输出内部的连贯性。形式上，它可以表示如下：

让我们分解一下：

是一个基于规则的二元信号，指示是否存在关键的连贯性标准（例如，遵循格式规则或逻辑约束），其值为0或1
是一个连续分数，源自语言模型，用于衡量风格统一性、流畅度和语义连贯性
α∈ [0,1] 是一个权重因子，平衡二元信号和基于语言信号的贡献

通过整合这些组件，模型有效地奖励那些既满足明确的基于规则的标准，又展现出自然、连贯的语言流的输出，从而增强了整体的推理清晰度和连贯性。

这个过程不是一次性的；它是迭代和自我强化的，通过多个周期（通常是 3-5 次迭代）运行，每个迭代使用最新的蒸馏示例和更新的奖励信号，以逐步增强模型的推理能力和输出质量，形成我们所描述的循环策略。改进的推理产生了更高质量的CoT示例，这些示例反过来又训练出了更好的模型——进而产生更强的输出，为下一轮生成和优化提供动力。

最终的DeepSeek-R1模型源于一个多阶段过程，该过程结合了高效的架构、自适应训练奖励以及一种鼓励清晰推理并利用顶级输出改进未来训练的数据驱动方法。

现在，让我们看看我们所学的所有内容如何在训练管道中结合起来。

R1 的训练管道

DeepSeek的训练管道涉及以下步骤：

图 2.6：DeepSeek-R1 的训练管道

DeepSeek R1 训练流程

让我们逐步了解这些阶段：

1. 使用 R1-Zero 输出的冷启动 SFT

冷启动数据作为在 DeepSeek-V3 基座模型上构建的新模型检查点的引导语料库。

此阶段有两个目的：

首先，它通过监督学习将 R1-Zero 中涌现的推理行为迁移到了新模型中。
其次，它使团队能够评估 DeepSeek-V3 基座模型从 R1-Zero 的推理轨迹中泛化的程度。

结果非常有前景。即使在这个阶段还没有额外的强化学习，使用冷启动 监督微调（SFT）数据训练的模型在结构化推理方面与单独的 V3 相比已开始显示出可衡量的改进。这反映在评估指标上，例如在多步推理基准测试中的准确率，冷启动 SFT 模型比 V3 提升了约 10-15%——例如，在复杂数学和逻辑任务上的正确率从大约 65% 提升到了 75-80%。此外，评估推理连贯性和遵循预期 + <answer> 格式的指标也显示出显著增益，这表明模型不仅更准确，而且产生了组织更好、更透明的推理路径。

2. 带一致性奖励的强化学习

流水线中的下一步涉及通过第二轮 RL 训练这个新的检查点。这次的训练不仅包括基于响应准确性的原始奖励信号（这是通过将每个模型输出与遵循冷启动阶段确立的标准的预期已知正确答案进行比较来衡量的），还包括一个新引入的指标：语言一致性。

3. 融合通用知识 CoT 的最终 SFT

虽然最初的流水线设想了一个单独的最终 SFT 轮次来将常识和世界知识整合到 CoT 风格的推理中，但这一阶段最终被跳过。相反，训练直接从大规模预训练进行到使用蒸馏 CoT 数据集的监督微调。这种方法有效地将模型广泛的通用知识与结构化推理示例在单个微调阶段中结合起来，而无需额外的专用融合轮次。其原理很简单：一个推理模型如果只能解方程和谜题，就无法在现实世界中蓬勃发展。它还必须处理日常查询、事实和人类可解释的逻辑。

DeepSeek 通过对一个结合了无监督文本和监督性、逐步 CoT 示例的混合数据集进行微调，将 CoT 推理与广泛的预训练知识融合。这训练模型通过将结构化推理轨迹与其广泛的知识库集成来生成连贯、基于事实的解释。

多任务目标和一致性约束进一步优化了这种集成，使 DeepSeek 能够在不同领域产生透明、逻辑严密的输出，同时放大与准确、可解释答案相关的模式。

4. 最终 RLHF

R1 训练的最后阶段涉及 RLHF，其中模型的输出使用基于人类偏好的显式奖励函数进行优化。此阶段还涉及偏好比较对，其中人类标注者对同一提示的多个响应进行排序。这些信号用于微调一个奖励模型，然后指导最后的 RL 循环。

为了捕获并强制执行这些偏好，RL 阶段使用了设计为迭代地改进模型行为的显式奖励函数：

4.1 用户期望定义

一组领域专家和标注者开发了详细指南，描述了理想的模型响应。这些指南强调避免歧义、减少不必要的冗长、保持事实和逻辑陈述的精确性，以及优先考虑上下文中的有用性。

4.2 基于规则的检查

自动化的基于规则的评估脚本解析每个生成的响应，以检查是否符合这些指南，例如：

通过扫描模糊词语或矛盾陈述来检测歧义。
通过比较 token 数量与类似提示类型的预期长度范围来衡量冗长度。
通过启发式指标（如相关解释、逐步推理和可操作结论的存在）部分评估有用性。
通过将模型输出与标准答案或参考解决方案进行比较（主要使用二进制正确性标签）来评估精确度。

4.3 分数计算

每个响应被分配一个组合奖励分数，结合了多个因素：

是 1，如果最终答案正确并且与已知事实相符，否则为 0。
是基于不含模糊短语的标量奖励。
惩罚过长或冗余的响应（值越低表示长度过度）。
奖励存在详细、相关的推理和解释。
权重根据人类偏好研究进行调整，以平衡这些方面。

对于 Prove that the sum of two even numbers is even 提示，DeepSeek 的奖励函数结合了多个组件，并进行了加权以平衡逻辑正确性、清晰度和一致性。

4.4 基于规则的检查启发式方法

执行以下操作：

验证所有关键数学术语（例如 even, sum, 2a, 2b）是否出现在证明中。
结构检查以确保证明遵循预期的逻辑格式：定义、代入和结论。
格式规则，例如正确使用符号和步骤论证分隔。

4.5 奖励组件的权重选择

奖励组件包括以下内容：

逻辑正确性（基于规则的检查）因其关键作用而被赋予高权重，约为 0.5。
清晰度和语言一致性（基于语言模型的信号）权重为 0.3，以奖励流畅、易于理解的解释。
内部连贯性和对推理格式的遵循权重为 0.2，以鼓励步骤式结构。

4.6 人类偏好数据收集

人类标注者收到了为该提示生成的多个候选证明，并被要求根据正确性、清晰度和完整性对它们进行排名。这些排名用于训练奖励预测器，以估计人类偏好，从而指导模型优先考虑推理质量而非表面属性。这种数据驱动的监督有助于校准奖励权重并塑造启发式规则，使自动评分与人类判断保持一致。

所有这些方法共同确保了 DeepSeek 产生了数学上严谨、清晰且结构良好的证明，例如证明两个偶数之和是偶数的示例。

5. 蒸馏与分发

在最终确定主要的 R1 模型后，DeepSeek 进行了蒸馏实验，以将其先进的推理能力转移到更小、更高效的模型中。

5.1 蒸馏

蒸馏很重要，因为它使紧凑的模型能够通过在来自更大模型的高质量输出上进行训练来学习复杂行为，从而提高了可访问性和部署灵活性。在这个过程中，R1 的详细 CoT 输出的分布作为训练数据分布，引导像 DeepSeek-R1-Distill-Qwen 和 DeepSeek-R1-Distill-Llama 这样的小模型在没有 RL 的情况下也能泛化出复杂的推理能力。

DeepSeek 的高级能力

在演示推理能力可以通过知识蒸馏有效转移方面，其意义重大。DeepSeek-R1生成的思维链轨迹可作为高质量的教师数据，用于训练更小、更高效的学生模型，而不是通过代价高昂的强化学习循环重新训练大模型。

在蒸馏过程中，使用了组合损失函数来平衡学生对教师软输出分布的模仿与对真实标签的遵循。具体来说，损失函数的公式如下：

我们来分解一下：

是 知识蒸馏损失，计算为教师模型和学生模型软化概率分布之间的 KL 散度。
是基于真实标签的标准 交叉熵损失。
是一个权重超参数，用于控制蒸馏和监督学习之间的平衡。
在计算之前，对教师的逻辑值应用 温度 T 进行软化，通常设置为 T=2 或 T=4，以提供更丰富的梯度信号。

DeepSeek 的蒸馏过程 通过调整超参数（如（通常设置为 0.7）和温度（通常 T=3））来优化学习信号，从而平衡了知识转移和监督学习。这些设置在显著减小模型大小和推理延迟的同时，仍能紧密匹配教师模型的推理准确率。关于蒸馏损失和超参数选择的更多细节，您可以参考 DeepSeek 的研究论文：arxiv.org/abs/2501.12948。

这种蒸馏策略通过利用思维链教学材料作为丰富的监督信号，有效地将 R1 的大规模推理能力压缩到了更轻量的模型中。它证明了大规模推理不一定局限于庞大的架构或重复的强化学习训练，从而能够实现强大、可解释的推理助手的规模化部署。

自我教学中的涌现模式

这个流程中最引人入胜的方面之一是其自我强化的结构。R1-Zero 创建了一个迭代循环：

图 2.7：DeepSeek 的自我教学迭代循环

此迭代循环持续固定轮数（通常为 3-5 次迭代），直到性能提升趋于平稳且进一步改进变得微乎其微（请参考 图 2.7 以快速回顾）。

这种受控循环使模型能够逐步引导出更强的推理能力，而无需无限运行，从而在训练效率与质量提升之间取得平衡。在每个阶段，模型不仅从外部反馈中学习，还从其自身的历史最佳成果中学习。

哎呀呀！谁说构建一个大语言模型那么容易！

接下来，我们看看 DeepSeek 的工程如何在不牺牲智能的前提下，推动了效率的边界。

DeepSeek 的视觉能力

DeepSeek 的架构（最初为语言任务设计）已发展到支持多模态能力，特别是通过 DeepSeek-VL 等模型。这些模型集成了文本和视觉输入，使系统能够联合理解并推理图像与语言。虽然 DeepSeek 目前在支持文本和视觉的意义上是多模态的，但公开发布的版本尚未正式支持音频或视频等其他模态。

视觉-语言集成的架构变更

为实现图像理解，DeepSeek-VL 扩展了其语言基础，加入了一个视觉编码器，通常是经过修改的 视觉 Transformer (ViT) 或 卷积神经网络 (CNN)。

该系统由三个主要组件构成：一个 混合视觉编码器、一个 视觉-语言适配器 和一个 语言模型。

下图展示了视觉-语言适配器的工作原理：

图 2.8：DeepSeek 的视觉适配器（来源：arxiv.org/html/2403.05525v2）

DeepSeek 使用一个两层的 混合 MLP 来有效连接视觉编码器和大语言模型。最初，使用单独的单层 MLP 分别处理高分辨率和低分辨率的视觉特征。然后将这些处理过的特征拼接起来，并通过一个额外的 MLP 层，将其映射到大语言模型的输入空间。具体来说，DeepSeek-VL-1B 模型建立在 DeepSeek-LLM-1B 之上，后者在大约 5000 亿个文本标记上进行了训练。类似地，DeepSeek-VL-7B 模型基于 DeepSeek-LLM-7B，在大约 2 万亿个文本标记上进行了训练。

当用户在输入中包含图像时，模型首先使用视觉编码器（通常是 CNN 或 ViT）对其进行处理，将图像转换为固定大小的图像块嵌入，每个嵌入在共享的潜空间中表示为一个向量（通常为 768 或 1024 维）。这些图像块级别的嵌入捕获局部视觉特征，然后与相同维度的标记化文本嵌入一起被送入跨模态注意力层。

模型通过将图像划分为均匀的图像块（例如，16x16 像素）来识别相关图像区域，这些图像块作为注意力单元。通过每个 Transformer 块内的跨模态注意力，文本标记（通过标准分词器处理）学会有选择地关注这些视觉图像块。例如，当处理单词 图表 时，模型会关注包含图表视觉元素的特定图像块，从而有效地将语言上下文与空间特征对齐。

这种多模态融合使模型能够构建一个联合表示，融合了文本的语义内容和图像的空间层次结构，从而实现跨模态的连贯推理和生成。这个解释平衡了技术细节和清晰度，使其对我们的目标受众易于理解，同时强调了基于图像块的注意力如何驱动视觉定位。

假设用户输入以下内容，并附带图像：

参考附带的图表，解释与 Q1 和 Q3 相比，Q2 的增长模式是否正常。

图 2.9：DeepSeek 的一个图像输入示例

以下是 DeepSeek-VL-7B 的处理过程：

视觉编码：提示中的图像（例如，图表）被划分为图像块，并通过视觉编码器。此步骤捕获条形高度、轴标签和整体布局等细节。
文本编码：问题被标记化并送入语言主干网络，创建用户意图的表示（例如，Q2 的增长模式）。
跨模态对齐：模型使用注意力机制将问题中的单词或短语与最相关的视觉区域联系起来。例如，Q2 激活对标有 Q2 的图表部分的注意力，而 增长 则链接到对应条形的高度。
响应生成：通过组合这些对齐的表示，模型生成一个答案，例如 Q2 的增长明显高于 Q1 和 Q3，表明可能存在一次性事件或季节性峰值。

能够同时对文本和图像进行推理的能力，使得 DeepSeek-VL-7B 能够解决任何单一模态都无法单独处理的问题，例如解读附带问题的图表或阅读并修正手写方程。在教育领域，这支持能够解读扫描笔记的智能辅导系统；在企业环境中，它使系统能够从发票、表格或原理图中提取结构和意义。

DeepSeek 将其对 MoE 和 MLA 的使用扩展到了多模态领域。这意味着即使处理视觉数据，系统也只会激活其参数的一个子集，从而确保高效的计算。得益于这种动态专业化方法，视觉推理并不会显著增加推理成本。

代理推理与工具集成

尽管本章主要讨论推理机制，但必须承认 DeepSeek 也包含了支持更广泛代理能力的基础组件。这些能力包括与外部工具交互、执行多步骤任务规划以及基于反馈动态调整其行动的能力。其中一些特性在 第 6 章 中有详细阐述。

DeepSeek 模型，特别是 DeepSeek-R1，通过利用 CoT 技术将复杂问题分解为明确、可验证的推理链，在 AI 代理中充当强大的推理引擎。这种结构化推理为 DeepSeek 在多步骤问题求解、逻辑推理和上下文决策方面的能力奠定了基础。

在应用工作流中，DeepSeek 可以通过以下方式充当自主代理：

根据中间输出和环境反馈执行顺序操作。
动态整合外部工具和数据源以增强其决策过程。
通过上下文感知规划处理依赖关系和错误恢复。
为复杂任务执行协调多代理工作流。

这些代理推理能力使 DeepSeek 适用于需要自适应、目标导向行为的现实世界应用，超越了静态问答的范畴。

现在，我们将重点关注 DeepSeek 与其他大型语言模型（LLM）的比较及其一些不足之处。

DeepSeek 在全球大语言模型版图中的定位

全球大语言模型版图正因新模型和新技术的涌现而蓬勃发展，各方正激烈竞争，以期率先实现推理能力，并最终达成 通用人工智能（AGI）。大语言模型构成了通往 AGI 的必要第一步。

让我们看看不同的大语言模型之间如何比较（截至 2025 年中）：

| --- | --- | --- | --- | --- |

表 2.1：DeepSeek 与领先大语言模型的比较表（2025 年中）

DeepSeek 通过稀疏路由、可解释的推理输出和完全开源的透明度，实现了强大的性价比，在数学、代码和 CoT 任务方面表现出色。GPT-4o 在多模态和视觉能力方面脱颖而出，但伴随着高昂的计算成本和有限的透明度。Claude 4 Sonnet 提供了一个具有竞争力的闭源替代方案，与之前的 Claude 模型相比，在效率和长上下文处理方面有所改进，在长文本、数学和代码任务上表现良好，尽管它仍然不够开放，并且受限于致密推理的瓶颈。

DeepSeek 的局限性及其与其他模型的比较

尽管 DeepSeek 在推理、开放性和架构创新方面取得了突破，但它绝非没有局限性。像所有语言模型一样，它也在规模与可解释性、延迟与能力、开放性与性能之间进行权衡。

本节旨在客观评估 DeepSeek 当前的局限性，这些局限性与同行（如 GPT-4o、Claude 3.5/4、Gemini 1.5 和 Mistral）相比处于何种水平，以及未来的迭代版本可能如何应对这些挑战。

扩展挑战与稀疏专业知识的局限

DeepSeek-V3 的 670B 参数——尽管通过 MoE 高效实现——仍然使模型的训练和部署成为一项艰巨的任务。即使采用稀疏激活（每次推理激活 37B 参数），大规模部署 DeepSeek 也需要高吞吐量的 GPU 和带宽优化的环境。这意味着，对于许多较小的机构来说，尽管发布了蒸馏版本，但完整使用该模型仍然是一个难以企及的目标。

此外，MoE（混合专家模型）引入了一系列特有的技术约束。专家间的路由不稳定性、token 丢弃问题以及专家利用率不足等，仍是当前研究的热点领域。DeepSeek 通过使用共享专家与 FFN 预路由机制，帮助缓解了其中部分问题，但在尾部 token 分布上的性能衰减仍然存在。相比之下，像 Claude 4 或 GPT-4o 这样的密集模型，虽然计算量更大，但能维持更稳定的跨层 token 流。

稀疏模型还面临跨不相关输入的表征共享有限的局限，这意味着在某些零样本场景下的泛化能力可能落后于密集模型。这种效率与通用性之间的权衡并非 DeepSeek 所独有，但在 DeepSeek 前所未有的规模下，其表现尤为突出。

推理延迟与实时交互的权衡

虽然 MoE 和 FP8 训练有助于提升每个 token 的推理速度，但 DeepSeek 在实时场景中仍面临延迟挑战，尤其是与搜索模块或多文档重排序器结合使用时。思维链推理虽然提高了输出质量，但通常会导致响应更长，从而增加了感知延迟。

Claude 4 和 GPT-4o 凭借其高度优化的解码与缓存策略，在实时生成基准测试中常常优于 DeepSeek。GPT-4o 尤其受益于紧密的模型-服务器集成以及激进的投机解码，使其响应时间可与 GPT-3.5 相当，同时具备 GPT-4 级别的推理能力。

DeepSeek 的多 token 预测（MTP）与 KV 压缩（通过 MLA 实现）在一定程度上缓解了此问题，但面向用户的应用，如聊天代理或嵌入式设备，若要求毫秒级响应，可能会发现目前 Claude 或 GPT 系列的优化更佳。

可解释性与对齐风险

具有讽刺意味的是，尽管 DeepSeek 通过思考/回答格式开创了结构化推理，但这一优势也可能成为瓶颈。强制要求思维链输出意味着模型可能产生看似结构合理但存在细微缺陷的推理依据，从而导致错误的自信。由于模型被训练生成连贯的逻辑，即使答案错误，也可能以令人信服的语言包装呈现，从而引发对齐与安全风险。

此问题并非 DeepSeek 独有。GPT-4 和 Claude 4 在多步推理中的幻觉问题也面临类似关切。然而，这些模型通常包含对抗偏好建模与多信号安全协议——这些技术对 DeepSeek 而言仍在开发中。DeepSeek 缺乏基于偏好排序模型（PRM）的训练步骤，意味着其对齐依赖于更简单的奖励启发式方法，可能在处理伦理或边缘场景时遗漏细微差别。

上下文长度与压缩上限

尽管 DeepSeek 使用 MLA 来减少上下文占用，但在超长上下文任务中仍落后于 Claude 4 和 Gemini 1.5。Claude 的 200k 上下文窗口和 Gemini 的稀疏注意力技巧使其更擅长吸收整本书籍、法律文件或多轮对话。DeepSeek 在实用性能上更现实地限制在 64k token。

此外，虽然 MLA 压缩有所帮助，但它在检索保真度上需要权衡，尤其在多文档问答或混合检索/排序设置中。相比之下，Claude 的上下文优先处理和 GPT-4o 的选择性注意力在大规模应用中提供了更智能的上下文利用方式。

数据集与文化覆盖范围的差距

尽管 DeepSeek 注重多语言能力，但在某些文化数据集和特定领域语料库上的训练仍相对不足。GPT-4 和 Claude 4 受益于广泛的语料摄入与细粒度的领域调优，在金融、法律及特定科学领域展现出更优的细微理解力。DeepSeek 由于采用成本感知的预训练，采取了更通才的路线，有时牺牲了特定领域的深度。

然而，这恰恰也是 DeepSeek 开放基础设施的亮点所在：其模块化的检索器、重排序器和解码器设置，允许研究者针对缺失的知识领域进行微调，这种灵活性是专有模型所不具备的。

局限性展望

DeepSeek 的局限性是真实存在的，但并非不可逾越。随着其周边社区的成长，许多挑战——尤其是那些涉及路由、实时解码、安全调优和上下文优化的问题——很可能通过研究贡献得到解决。与专有模型不同，DeepSeek 透明的生态系统允许共享学习和迭代修正。

在许多方面，暴露局限性的行为本身就成为了邀请他人去改进、适应和挑战模型本身的契机。这就是开放科学的承诺——不在于完美，而在于参与。

本章总结

在本章中，我们深入探讨了 DeepSeek 背后的核心研究与技术演进，追溯其发展轨迹。我们审视了 DeepSeek 的发展如何分裂为两条强大的脉络：基础模型的构建与推理能力的提升。DeepSeek 的每个迭代版本都增加了一层新的创新，从扩展密集架构到开创高效的 MoE 模型，并最终通过基于规则的强化学习完善多步推理。

我们探讨了采用 MoE 技术如何以更低的计算成本带来显著的性能提升，以及 MLA、MTP 和 GRPO 等创新如何应对训练稳定性、推理效率和推理可靠性方面的关键挑战。本章还重点介绍了 DeepSeek 在代码智能、数学和形式逻辑等领域的专门努力，展示了其跨领域的通用应用能力。

DeepSeek-R1 的发布标志着这些发展的集大成，证明了一个强大、高效且具备推理能力的 LLM 可以通过整合稀疏架构与最小化的、基于规则的强化学习而诞生。你现在应该已经清楚理解了 DeepSeek 如何通过迭代创新而演进，以及驱动其性能的技术突破。

在下一章，我们将讨论为 DeepSeek 设计提示的各种技巧。

| |

| --- |

| 获取本书的 PDF 版本和独家补充资料
扫描二维码（或访问 packtpub.com/unlock）。搜索本书书名，确认版本，然后按照页面上的步骤操作。 | |

| 注意：请保留好您的发票。直接从 Packt 购买无需发票。 |

3 提示 DeepSeek

如果你曾经从编写命令式代码、手动告诉文档对象模型（DOM）如何一步步更新网页，转向使用 React 这样的声明式框架，你就知道需要进行的思维转变。你不再给出详尽的指令，而是开始描述期望的结果，信任系统去弄清如何实现。同样的原理也适用于从 Mistral 7B 或 Llama 3.3 8B 等传统 LLM 转向 DeepSeek 推理引擎。你为引导模型思考过程而培养的肌肉记忆，在一个被设计为自主推理的引擎上，可能反而会对你不利。

本章的核心就是驾驭这些差异，从引人注目的怪癖（如 R1 对系统提示的抵触）到那些能决定成败的细微行为转变。无论你是迁移现有的提示库还是从零开始使用 DeepSeek，都请将本章视为你的实地指南，带你理解语言模型中一个根本不同的范式。

重要的是要记住，我们仍处于理解这些强大新模型的早期阶段。目前，DeepSeek 模型问世已近一年。本章中的指导内容是基于当前可获得的最佳信息综合而成：DeepSeek 团队的官方文档、行业先驱的使用报告，以及第一批探索这些系统的学术论文。可以将其视为一份动态文档：是我们当前集体理解的一个快照。最佳实践无疑会随着社区认知的深入而演变，但以下内容是我们迄今为止对这片领域的最佳描绘。

本章大部分内容聚焦于 R1 系列：DeepSeek 的推理优先模型，它们颠覆了传统的提示工程理念。这些模型不仅处理方式不同；它们的训练方式也不同，采用了强化学习技术，创造出了一个具有自身内部思维链（CoT）流程的模型。这就像给人类驾驶员提供逐步导航指令，与直接告诉自动驾驶汽车目的地之间的区别。R1 模型已经知道如何导航整个推理过程。

其影响遍及提示设计的各个方面：

少样本示例？它们实际上会让你的结果变差（如 DeepSeek 原始论文（arxiv.org/html/2501.12948v1）中所述，准确度下降了 15%）。

分步思考指令？冗余。R1 内部已经在做。

丰富的系统提示？模型可能会完全忽略它们。

复杂、冗长的指令？不如清晰、最小化的指令有效。

与此同时，V 系列模型的行为更像你习惯的 LLM，但即使它们也有其特殊之处。例如，温度参数映射与 OpenAI 的默认值不同，并且存在对超出典型 markdown 的结构化格式的偏好。

本章将逐步剖析 DeepSeek 提示范式。我们将探讨解释这些差异存在的核心思维模型（提示：一切都关乎强化学习），提供基于官方文档和社区测试的具体注意事项和禁忌，并为你提供适用于 R 系列和 V 系列模型的模板和故障排除策略。

最终，你不仅将理解要做什么不同的事情，还将理解为什么 DeepSeek 独特的训练方法要求我们从第一性原理重新思考提示工程。可以将其视为学习一种新的编程范式。一旦你掌握了其底层哲学，具体的技术自然会水到渠成。

在本章中，我们将涵盖以下主要主题：

DeepSeek 的核心思维模型与原则
提示 DeepSeek 的通用技巧与建议
处理结构化输出
V 系列的独特提示技术
故障排除
提示迁移指南

技术要求

学习本章你需要：

克隆项目仓库并安装依赖：

git clone https://github.com/PacktPublishing/DeepSeek-in-Practice.git
cd Chapter03
pip install -r requirements.txt

该requirements.txt文件包含用于发送提示和获取结构化输出的 OpenAI 和 Instructor，以及Pydantic，用于帮助创建我们可以传递给模型的结构。

API 密钥和环境变量：你需要访问 OpenRouter 模型（如 DeepSeek 以及任何你想要的其他模型）的 API 密钥。
OpenRouter 提供对 DeepSeek 模型的访问，用于在评估期间生成基线比较。请在 openrouter.ai 注册，并从仪表板获取你的 API 密钥。我们将使用免费模型访问基础版DeepSeek-R1模型，但你需要一个 API 密钥来访问它（通过litellm）。如果你没有 OpenRouter API 密钥，可以在 openrouter.ai/docs/api-reference/api-keys/get-api-key 创建一个。

DeepSeek 的核心思维模型与原则

在深入技术细节之前，让我们先重新校准你对DeepSeek模型，特别是 R 系列模型，到底是什么的心智模型。忘记你知道的所有关于如何用精心设计的系统提示来引导 ChatGPT 给出正确回答的知识。

为DeepSeek-R1系列构建提示时，有三件事需要谨慎：

DeepSeek模型更像是定理证明器，而非聊天机器人：少把 R1 视为对话伙伴，多把它视为一个碰巧会说英语的数学证明助手。当你向 R1 提交提示时，你不是在开始对话。你是在向一个推理引擎提交正式的问题描述。

这不仅仅是说辞。R1 模型通过强化学习训练，以在可验证任务（如数学/编码）上获得正确答案为优先，并且在回复前会进行大量的内部思考。具体来说，模型首先生成一个隐藏的推理草稿（``块），然后才输出最终答案（关于隐藏的思考令牌的复习，请参见第 2 章）。

在实践中这意味着：

* 你的提示是一个问题陈述，而不是对话的开场白。

* 期望模型在内部规划/打草稿，然后产生一个解决方案。（这里所说的内部规划，仅指模型的隐藏CoT/草稿；没有单独的编译阶段。）

* 你通常会得到一个解决方案——而不是一个健谈的来回交流。

像研究请求一样构建你的提示：想象你正在委托一位才华横溢但非常直白的研究生撰写一篇研究论文。你不会给他们一封包含其他论文示例的冗长邮件。你会提供一个清晰的问题陈述、约束条件和预期的输出格式。好的DeepSeek提示结构类似于学术问题描述。换句话说，如果你按照这种顺序和格式创建提示，DeepSeek 通常会给你更好的结果。不过，这不是官方指导中明确规定的：

Context: [哪些背景信息是必不可少的？]
Task: [需要解决什么具体问题？]
Constraints: [存在哪些限制或要求？]
Output: [解决方案应采用什么格式？]

这无关形式主义或刻板——而是关乎清晰度。当模型能够无歧义地解析你的意图时，它表现最佳。

例如，如果我想了解为急诊室部署新的分诊模型的哲学含义，我可能会在像Mistral Medium 3.1这样的传统LLM 中使用这样的提示：

如果比医生更准确，医院在急诊室使用黑箱 AI 分诊系统在伦理上是否可接受？请讨论利弊，并给出你的建议。

你可以看到，我们从Mistral模型（Medium 3.1）得到了类似这样的回复：

图 3.1：传统 LLM 的回复 – Mistral Medium 3.1

你可以在本书的 GitHub 仓库中查看eg1.py以获取完整回复（github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg1.py）。但遵循前述规范的结构化提示可能如下所示（参见eg2.py github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg2.py）：

背景：

你正在分析一个临床分诊中的伦理政策选择。该模型在回顾性数据上的准确性高于临床医生，但其内部逻辑不可解释。

任务：

评估医院是否应部署该模型，以及在何种条件下部署。

约束：

明确在以下视角下评估：(1) 功利主义，(2) 道义论（康德义务），(3) 美德伦理，(4) 关怀伦理。
对于每个视角：给出裁决 {允许|禁止|有条件} 并附上 1-2 句理由。
需涉及：公平性、问责制、透明度、危害缓解、知情同意/同意、监督。
包含一个可能推翻你结论的有力反驳论点，以及你将如何检验它。
不要揭示思维链；只提供最终标注的推理。
每个字段不超过 60 个词。

输出：

仅返回以下形状的 JSON：

{
  "per_lens": {
    "utilitarianism": {"verdict": "permit|forbid|conditional", "rationale": "..."},
    "deontology": {"verdict": "...", "rationale": "..."},
    "virtue_ethics": {"verdict": "...", "rationale": "..."},
    "care_ethics": {"verdict": "...", "rationale": "..."}
  },
  "deployment_recommendation": {"verdict": "permit|forbid|conditional", "conditions": ["...","..."]},
  "oversight_plan": ["metric_audits","bias_monitoring","appeals_process","A/B_safety_guardrails"],
  "counterargument": {"claim": "...", "test": "..."},
  "residual_risks": ["...","..."],
  "confidence": 0.0
}

此提示导致了 DeepSeek-R1 产生一个很长的思考输出，随后给出了以下 JSON 响应（正如所要求的）：

{
  "per_lens": {
    "utilitarianism": {"verdict": "permit", "rationale": "更高的准确性能最大化整体患者预后和福祉，实现了为最多数人谋最大利益的功利主义目标。"},
    "deontology": {"verdict": "forbid", "rationale": "不透明的逻辑将患者仅仅视为达到目的的手段，通过做出无法解释的、非自主的决定，违背了尊重人的义务。"},
    "virtue_ethics": {"verdict": "conditional", "rationale": "仅在模型支持而非取代临床医生的审慎和同情心时允许。部署必须促进信任与协作，而非削弱专业美德。"},
    "care_ethics": {"verdict": "conditional", "rationale": "仅作为支持性工具时允许。该模型缺乏共情与情境理解；最终决定必须由人类照护者作出，以维持关系性照护。"}
  },
  "deployment_recommendation": {"verdict": "conditional", "conditions": ["仅作为临床医生的决策支持工具，而非自主分诊", "对受保护群体实施强大、持续的偏差审计", "获取患者的知情同意/同意，并披露 AI 参与情况"]},
  "oversight_plan": ["指标审计", "偏差监控", "申诉流程", "A/B 安全防护"],
  "counterargument": {"claim": "模型优越的回顾性准确性可能因隐藏的混杂变量或数据漂移而无法转化为现实世界的性能，从而导致净损害。", "test": "进行严格的前瞻性 A/B 试验，比较模型辅助护理与标准护理，由独立的 DSMB 监控显著的结果差异和安全性。"},
  "residual_risks": ["自动化偏差导致临床医生过度依赖模型", "因数据漂移导致模型性能随时间下降"],
  "confidence": 0.8
}

极简主义心态（或我如何学会停止担忧并信任模型）：这是经验丰富的提示工程师常感到挣扎的地方。对于 DeepSeek，少即是多。你精心收集的那些少样本示例？它们不仅是不必要的，而且是有害的。你通常提供的那种详细的逐步拆解？你正在对一个已经知道如何思考的系统进行微观管理。这就像通过解释每一个微小动作来教人开车，与仅仅告诉他们目的地之间的区别。DeepSeek 模型，尤其是 R1，已经掌握了如何思考的部分。它们只需要知道思考什么。

以下是一些使用 DeepSeek 极简提示词的入门指南：

清晰地陈述问题，然后放手。
抵制过度解释或提供示例的冲动。
接受模型的内部推理可能比你的外部指导更好。

这种极简方法起初可能令人不适，特别是如果你习惯了与其他模型配合良好的提示工程如同创意写作的方式。但一旦你接受它，你会发现 DeepSeek 的响应往往更全面、更准确，恰恰因为你没有限制其推理过程。

实际上，从上述建议推导出的极简提示词模板可能看起来像这样：

任务：[要决定/计算什么；1-2 句话]
约束：[编号的规则或视角；只保留要点]
输出：[紧凑的模式或要点模板；仅短字段；无思维链]

参见仓库中的eg3.py（github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg3.py），了解这一原则付诸实践的例子。eg3.py使用通过 OpenRouter 兼容 OpenAI 接口的 DeepSeek，针对相同的伦理分诊任务，运行三种提示风格（零样本极简、带示例的少样本、详尽逐步）的受控比较。它为每种风格构建消息请求，请求仅包含紧密模式的 JSON 答案，并通过检查必需的键、允许的裁决、字数限制和类型（对大小写有少许容忍度）来验证响应。每次运行会计时，在可用时捕获 token 使用量，并打印输出以及 PASS/FAIL 验证反馈。这个例子凸显了极简提示词如何通常更合规且成本更低。

模式：这里，模式指的是对模型预期输出的明确规范——答案的形状和约束。一个模式定义字段名、数据类型、允许的值/枚举、必需与可选字段、嵌套以及限制（例如，字数或范围约束）。我们在提示中或通过 API 功能（JSON 模式、函数/工具参数、Pydantic）来表达它，这样模型就知道要返回什么，我们也可以可靠地验证它。简而言之：提示描述任务；模式是响应必须满足的输出契约（例如，{"verdict": "approve|deny|escalate", "rationale": "string, <=50 words"}）。

现在你对能与 DeepSeek 良好配合的提示类型有了大致了解，让我们探究一下它们为何有效。

为什么结构化的极简提示词能与 R 系列“契合”（内在机制解析）

如果研究简报→ JSON（或单次生成）的答案风格与 R1 配合效果出奇地好，那是因为它与模型受训以赢得的优化方式一致。R1 并未被优化以产生雄辩的对话；它被优化为在隐藏推理之后产生可验证的、格式合规的答案。

回想第 2 章，在后训练期间，DeepSeek 使用了强化学习，其中每个候选响应都由基于规则的检查器评分：数学答案在指定的最终答案槽位中检查正确值；代码经过编译和测试；并且一个格式奖励强制分离了内部推理区域和最终答案区域（最初在训练模板中用 … </think>和<answer> … </answer>表示）。给模型一个紧凑的输出契约，它就会识别出它受训参与的游戏。这就是为什么一个清晰的问题陈述加上严格的模式，通常能胜过华丽的角色扮演或示例堆砌。

R1 使用 Group Relative Policy Optimization（GRPO）进行训练：对于每个提示，策略会生成 K 个响应样本（通常为 4-8 个），基于确定性规则的评分器为每个样本计算一个标量奖励（通常是正确性得分，外加少量格式检查），奖励按组内平均值居中，策略随即更新，使高于平均水平的样本获得更高的概率权重，而低于平均水平的样本则被削弱。在 R1-Zero 中，不存在学习到的奖励模型——仅有这些确定性评分器。

训练过程中，模型还会学习 测试时缩放：对于更难的问题，它在响应前会进行更长时间的内部思考——因此你无需指示它 逐步思考。你的任务是定义目标和格式；模型会在内部自行规划。

测试时缩放：R1 在推理时会自适应地为困难输入分配更多内部推理资源，而为简单输入分配较少资源。这种涌现行为使得模型能够通过对困难问题进行更长时间的思考来提高准确性，而无需任何特殊的逐步提示。

这一训练方案也解释了你之前观察到的两个实践中的特殊之处：

少样本会损害效果：在 R1 的评估中，添加示例反而会降低性能；作者建议使用零样本提示，并明确指定输出格式。示例会使内部规划器偏向于你提供的示例路径，而不是强化学习已经奖励的策略。
系统提示：早期的 R1 实践实际上会降低系统角色指令的权重，因此才有 将所有内容放在用户消息中 的建议。在 R1-0528 更新版中，现在支持系统提示——但最好保持其简洁（身份、日期、全局策略），而将任务特定的约束和模式放在用户内容中。

此时你心中可能浮现一个自然的问题：为什么传统的基于聊天的大型语言模型没有这些特殊之处？亲爱的读者，让我们为你解答。

为什么其他大型语言模型表现不同？

大多数聊天大型语言模型（例如 V 系列、GPT-4o、Claude 3.x）主要通过监督微调和偏好/RLHF 进行调优，因此它们通常受益于少样本示例和思维链演示——这是经典的 GPT-3 和 CoT 研究成果。传统的大型语言模型并非广泛使用基于规则的验证器来评分每个提示的单个、经过规则检查的最终答案，因此模式有助于解析，但示例仍倾向于教给模型分布规律。而推理优先模型（如 R1、OpenAI o1）颠倒了目标：奖励经过内部审慎思考后可验证的正确性和格式，因此零样本和模式成为更安全的默认选择。

以下是与传统大型语言模型相比，DeepSeek 所具有的优势和权衡：

优点：在可验证任务（数学、代码、科学问答）上具有更高的上限；对示例的依赖更少；当你将输出固定到单个插槽或严格 JSON 时，可靠性更强（这正是强化学习循环所强化的）。
缺点：更多的内部思考意味着延迟/成本的权衡；过度提示（少样本 CoT、角色扮演）可能会降低结果质量；在没有检查器的开放式、风格敏感任务中，优势会缩小，而使用良好示例的经典聊天模型可能以更低成本达到或超越 R1 的效果。

因此，在为 DeepSeek 编写提示时，请记住这个思维模型：你不是在训练一个聊天机器人。你是在向规划器提供一个经过验证器评分的规格说明。规格越明确，输出契约越严格，你就越能让强化学习训练出的行为发挥作用。

准备好了解这个思维模型如何转化为具体技术了吗？让我们详细展开。

提示 DeepSeek 的通用技巧与建议

确立了基本原则之后，我们将制定一些提示指南，以帮助你使用 DeepSeek 获得准确可靠的结果。正如我们在前一节中讨论的，这些建议中有很多直接违背了针对其他领先模型的最佳实践。这种调整是 DeepSeek 独特架构和训练方法的必然结果。

首先，我们将讨论为什么通常流行的少样本提示，实际上可能对 DeepSeek 的响应有害。

少样本陷阱

与传统提示工程最显著的偏离在于 DeepSeek 对少样本示例的处理。虽然这种技术是提升 OpenAI 的 GPT-4o 等模型性能的基石，但向 DeepSeek-R1 提供示例实际上会带来损害。官方文档在这一点上毫不含糊：不要在提示中提供示例，因为这会持续降低模型性能。

在对照实验中，当添加少样本示例时，准确率从 79.8% 下降到约 70%（www.linkedin.com/pulse/zero-shot-few-rag-benchmaking-deep-seek-sandeep-k-gil1c/）。

这种性能下降的发生是因为提供示例干扰了模型的原生推理过程。R1 是使用强化学习训练以形成其自身内部问题解决方法的，这些方法表现为一个隐藏的思维链过程。当示例包含在提示中时，它们试图强制执行特定的推理路径，从而覆盖了模型更强大、内部优化的策略。

与其提供示例，不如提供以下内容：

清晰的任务描述。
明确的输出格式要求。
模式定义（用于结构化输出）。

你可能想查看 eg3.py（github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg3.py）来查看一些零样本提示示例。

接下来，我们将讨论为什么系统提示在 DeepSeek 中不是一种流行的做法。

系统提示

使用系统提示来设置助手的角色和全局约束是很常见的。然而，对于最初的 DeepSeek-R1 版本，系统提示在实践中大多被忽略；官方建议是避免使用它们，并将所有可操作的指令放在用户消息中。这反映了早期 R1 的训练和评估方式：聊天模板强调用户→助手的交互模式，模型首先进行内部思考然后回答，而不是特别关注单独的系统角色。

2025 年 5 月，R1-0528 更新版增加了对系统提示的支持。DeepSeek 自己的应用使用特意简化的系统消息（标识助手；包含当前日期）。即便如此，实践经验法则仍然适用：将系统消息用于高级别的全局约束（身份、策略、格式偏好），而将任务特定的指令放在用户消息中。你不再需要加入 <|system|> 标签；模型无需它也能进行推理。

对于 V 系列（V3/V3.1），系统提示确实有效，官方聊天模板包含系统槽位，但它们对于获得良好结果并非必不可少。与 DeepSeek 的示例保持一致，优先使用简短、声明性的系统文本，并将精力集中在使用户指令清晰具体上。简而言之：对于 DeepSeek，简单、明确的用户提示就能承担主要工作。

对于 DeepSeek，简单清晰的提示就能恰到好处地完成工作。接下来我们详细讨论这一点。

冗长提示陷阱

如果你习惯于撰写读起来像维多利亚时代小说的提示，现在是时候展现你内心海明威的一面了。DeepSeek 的官方指导强调：复杂、冗长的提示往往导致效果不佳。应注重清晰而非复杂。

这并不是为了简洁而简洁。这是关于信噪比的问题。每一个额外的词都为模型误解你的意图提供了又一次机会。记住，R1 会生成可能成千上万的内部推理 token。它不需要你的帮助来过度思考。

以下是一个适用于 R1 的好提示示例：

Analyze the security vulnerabilities in this code snippet.
Focus on SQL injection and authentication bypass.
Code: [your code here]

一个错误提示的例子如下：

You are an expert security researcher with 20 years of experience.
I want you to carefully examine the following code, thinking about
all possible security issues. Consider things like SQL injection,
XSS, authentication problems, and any other vulnerabilities...
[several more paragraphs of instructions]

要查看此示例的说明，请查看eg4.py。该脚本通过两个提示对同一段有漏洞的 Flask 代码进行测试——一个是简洁的good提示，明确聚焦于 SQL 注入和认证绕过。

以下是good提示的响应：

Security Vulnerabilities Analysis
1\. SQL Injection
Vulnerability: The code uses direct string interpolation for SQL queries:
query = f"SELECT id, password_hash FROM users WHERE username = '{username}'"
Risk: Attackers can inject malicious SQL by manipulating the username input. For example:
' OR 1=1 -- would return the first user in the database, bypassing authentication.
'; DROP TABLE users; -- could delete tables.
Why It's Critical: Direct string interpolation allows arbitrary SQL execution, compromising data confidentiality, integrity, and availability.
2\. Authentication Bypass via Weak Hash Comparison
…
3\. Admin Privilege Escalation
…
4\. Insecure Password Hashing
…
7\. Database Connection Management
…
Exploit Scenarios
SQL Injection + Auth Bypass:
…
Fix Recommendations
Prevent SQL Injection:
query = "SELECT id, password_hash FROM users WHERE username = ?"
…
Fixed Code Snippet
…

另一方面，你会观察到，冗长的bad提示会导致主题漂移。

在此查看提示响应的片段：

Security Analysis of the Provided Flask Login Code
1\. SQL Injection (Critical – CVSS 9.8)
Issue
query = f"SELECT id, password_hash FROM users WHERE username = '{username}'"
…
Mitigation
Use parameterized queries:
query = "SELECT id, password_hash, is_admin FROM users WHERE username = ?"
cur.execute(query, (username,))
Reference: OWASP SQL Injection Prevention Cheat Sheet
2\. Weak Password Hashing (Critical – CVSS 9.8)
…
8\. Database Connection Management (Low – CVSS 3.7)
Issue: No connection pooling or error handling.
Mitigation: Use context managers (with sqlite3.connect()).
…
Broader Architectural Concerns
No rate limiting → susceptible to brute force.
Missing login attempt logging and monitoring.
Weak credential storage practices.
Database not isolated securely.
Dependencies not pinned → supply-chain risks.
Severity Summary
…
Corrected Code (Secure Example)
…
Additional Recommendations
References
…

如你所见，该响应本可以更加聚焦。

该脚本通过 OpenRouter 使用相同的 DeepSeek 模型（低温度设置），打印两种输出并计算简单的聚焦启发式指标（如对 SQLi/认证绕过的提及、XSS/CSRF 等离题内容的命中、一个focus_score、句子级别的信噪比以及词数）。

eg4.py 脚本还总结了哪个提示产出了更紧凑、更切题的答案：

Summary: focus_winner=Good shorter_output=Good | good_focus_score=17 bad_focus_score=9 | good_words=380 bad_words=872

现在让我们讨论可能影响 DeepSeek 响应生成的额外因素。

其他影响 DeepSeek 对提示响应的因素

除了提示技术之外，还有其他因素可能影响 DeepSeek 如何处理和响应提示。我们来探讨一下。

温度：官方文档（api-docs.deepseek.com/）认为温度（temperature）≈ 0.6 是 R1 的最佳值。温度过低，你可能会陷入重复循环（模型卡在局部最优解）。温度过高，精细的推理过程就会开始产生幻觉。

对于 V3，还有一个额外特点：API 将 temperature=1.0 映射为有效的 0.3，因此对于事实性任务，你可能需要将其设置得更低。

R1 的推荐设置是temperature=0.6，top_p=0.95。对于 V3，事实性任务使用temperature=0.3-0.5，创意工作最高可使用0.7。
人物提示：虽然复杂的角色扮演效果不佳，但微妙的人物暗示可以显著改善结果。社区测试表明，像I'm new to finance这样简单的受众级别暗示，能让 R1 在不损害准确性的情况下调整解释深度。

这之所以有效，是因为你提供了关于输出要求的上下文，而不是试图改变模型的推理过程。这就像说pretend you're a teacher（角色扮演）与explain this for a high school student（输出规范）的区别。

以下是一些有效人物提示的例子：
```
I'm a software engineer familiar with Python.
Writing for a general audience blog post.
Technical documentation for API developers.
```
编写人物提示的一个可靠方法是将其保持为一句话，指定受众、可交付物和深度——不涉及任何关于扮演什么角色的内容。将其表述为输出要求，而非角色：为谁而写，你想要什么，以及需要多深入。

一个有用的模板是For [audience], produce [format] that [purpose]; assume [background] and [depth constraints].。

例如，考虑这个提示：
```
For a non-technical CFO, produce a 150-word briefing that informs a buy/no-buy decision; assume no ML background and define acronyms, or For API engineers, provide a runnable Python 3.12 snippet with a docstring; avoid external deps and keep the explanation under 5 lines.
```
这个提示是目标明确的，指定了一个受众、一个输出和一个深度层次。如果结果仍然偏离目标，通过调整这三个要素（而非添加角色扮演）来进行迭代。

还有一些适用于 DeepSeek 提示的更多技术值得应用：

结构化格式：使用 XML 标签或 Markdown 标题来组织复杂提示。R1 对此解析效果极佳。以下是你可以如何使用 XML 标签来结构化提示的示例：
```
<context>Background information here</context>
<task>Specific problem to solve</task>
<constraints>Any limitations or requirements</constraints>
<output>Expected format</output>
```
你可能还记得第 1 章中提到，R1 展现出格式感知的生成能力（即，一个一致的结构可以充当解码器可以跟随的路标）。在第 2 章中，我们展示了结构化提示如何鼓励私密的中间推理（例如，隐含的思考 → 回答）。

我们现在可以将这些发现付诸实践：使用一个轻量、扁平的结构来指定输入和输出，然后让模型在内部进行规划。你不是让 R1 成为一个 XML 解析器；你是在给它一个它可以对齐的稳定布局。
显式输出格式：对于数学问题，总是为最终答案指定一个明确的输出格式。对于代码，预先指定语言和任何风格要求。
- 你可能还记得我们在第 1 章和第 2 章中的先前讨论，R1 通过了基于规则的验证器进行强化，这些验证器会检查最终答案和格式符合性。因此，提高可靠性最安全的方法是定义一个机器可以评分的输出契约。
- 在实践中，给模型最终结果留出一个槽位，指定单位和精度，并禁止额外的说明。以下是你可能尝试的一些技巧：
- 对于数学，要求一个单独的<answer>字段（或 final answer: 行），其中包含明确的舍入规则和单位——不要要求逐步解答。
- 对于代码，要求一个带有语言和版本（例如，Python 3.12）的代码块，一个入口点签名，以及任何约束（仅标准库、无网络、时间/内存限制）。

处理元数据与输出格式

如果你需要元数据（如置信度、方法），请将其放入单独的字段——切勿与最终答案混合。

在不同任务间保持契约的稳定；验证器奖励一致性胜过技巧性。例如，在一个数学问题的标签使用中，可以这样：

### Output
Return only:
<answer units="m/s²" rounding="3dp">…</answer>

或者对于代码问题，可以这样：

### Output
Return exactly one fenced code block:
```python

# Python 3.12, stdlib only

def normalize(v: list[float]) -> list[float]:

    ...

if __name__ == "__main__":

    ...

eg5.py (github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg5.py) 演示了如何将明确的输出契约与轻量级、可机器检查的验证器配对。该脚本向 DeepSeek 发送了三个提示：两个需要返回单个 <answer> 标签（指定了单位和小数精度）的数学问题，以及一个必须以恰好一个带特定入口签名和主守护语句的带围栏 Python 3.12 代码块返回的编程任务。

eg5.py 中使用的部分提示摘录（已截断）如下：


Math prompt 1

##### Task

A car accelerates from rest to 20 m/s in 8 s. Compute the constant acceleration.

##### Constraints

- Do not show steps or intermediate numbers.

- Do not include any text outside the required tag.

##### Output

Return only:

<answer units="m/s²" rounding="3dp">…</answer>

Math prompt 2

##### Task

Compute the area of a circle with radius r = 3.2 m. Use π ≈ 3.141592653589793.

##### Constraints

- Do not show steps or intermediate numbers.

- Do not include any text outside the required tag.

##### Output

Return only:

<answer units="m²" rounding="2dp">…</answer>

Code prompt

##### Task

Implement a vector normalization function.

##### Requirements

- Language: Python 3.12

- Libraries: stdlib only (no third-party imports)

- Style: Type hints; PEP 8 friendly; include a minimal main guard demo

- Entry point signature must be exactly:

  def normalize(v: list[float]) -> list[float]:

##### Output

Return exactly one fenced code block:

```python
##### Python 3.12, stdlib only
def normalize(v: list[float]) -> list[float]:
    ...
if __name__ == "__main__":
    ...

从这些提示中，你会注意到以下几点：

没有多余的散文、解释或额外的代码围栏：验证器检查数学答案是否恰好有一个包含正确单位、舍入和值的 <answer> 标签；代码响应是否恰好是一个包含标题行、normalize (v: list[float]) -> list[float] 函数、主守护语句且无非 stdlib 导入的 Python 代码围栏。
对于每个响应，eg5.py 会打印原始模型输出并运行简单的验证器：一个标签解析器验证数学项的单位、舍入和数值正确性，而一个代码检查器确保恰好有一个 Python 代码围栏、Python 3.12、stdlib only 标题、normalize (v: list[float]) -> list[float] 函数、主守护语句且无非 stdlib 导入。这表明，清晰的契约加上自动检查可以提高可靠性，而无需检查 CoT 或中间步骤。
你可以在这里查看响应：app.warp.dev/block/SmWy7R2GE1nsLIGoFNYwUg。

草案链 (Chain-of-Draft) 技术

为了提高令牌效率，你可以要求 R1 “仅保留每个思考步骤的最小草案”。这可以减少约 80% 的令牌使用量，同时保持准确性。草案链 (CoD) 与 CoT 不同。CoT 要求模型外化完整的逐步推理（通常冗长）。CoD 是中间笔记的一种压缩策略：你告诉模型每个步骤只保留最小的、检查点式的草案（例如，3-7 个词）并返回一个干净的最终答案字段。

使用 R1 时，这些草案通常存在于隐藏的 `` 空间中——你不是在要求可见的推理，而是在约束它使用多少草稿空间。由于 R1 的奖励基于最终答案的正确性和格式遵守，压缩草案通常能在大幅减少令牌的同时保持准确性（报告范围从约 40% 到 80% 的令牌减少，取决于任务）。当你关注延迟/成本但仍需要稳健推理时，使用 CoD；当你确实需要可读的推导过程时，才使用 CoT。

以下是提示词中可能的样子：

### Constraints
- Reasoning: think step by step, but keep only a minimal draft
  (≤5 words per step, ≤4 steps). If stuck, expand to 8 words.
- Do not print the draft; return only the final field below.
### Output
<answer rounding="2dp" units="m/s²">…</answer>

以下是一些关于 CoD 提示的更多技巧：

如果答案质量下降，提高每步字数上限（5→8→12）或允许一个额外步骤。
对于证明/长推导，更倾向于使用较高的上限或跳过 CoD。
在不同提示间保持最终答案槽位的稳定；验证器奖励一致性。

接下来，让我们看看针对 DeepSeek 提示技巧的快速注意事项清单（表 3.1）。

| 技术 | 做 | 不做 | 原因 |

| :--- | :--- | :--- | :--- |

表 3.1：DeepSeek 提示的注意事项清单

请记住，这些不仅仅是任意偏好。每条建议都源于 DeepSeek 模型的训练方式以及它们所学到的期望。

一旦你掌握了提示 DeepSeek 的基础知识，你就可以准备好探索我们接下来将要探讨的模型特定技术。

结构化输出的高级技巧与工具

在生产环境中部署语言模型的一个主要挑战是确保其输出是可靠、机器可读的格式。虽然许多模型需要大量的后处理来解析非结构化文本，但 DeepSeek 提供了用于生成 JSON 等结构化数据的专用功能。然而，DeepSeek 的实现具有与其他模型不同的独特特征和要求，理解它们是成功集成的关键。

DeepSeek 模型的结构化输出功能在 2025 年 5 月的 R1-0528 版本 (huggingface.co/deepseek-ai/DeepSeek-R1-0528) 中得到了显著改进。此更新引入了原生 JSON 模式和函数调用功能，使 DeepSeek 与其他著名模型的功能集更加接近。

不过，DeepSeek 实现的一个显著特点是：即使使用专用的 JSON 模式，当提示词中明确请求 JSON 输出时，性能会更加可靠。这一提示要求是确保结果一致性的关键细节。

在深入探讨如何实现之前，我们先了解为什么以及何时使用。结构化输出（JSON、函数调用或类型化模型）不仅适用于构建聊天界面——它适用于任何模型输出将由代码（而非人工视觉）消费的场景。

假设你需要为应用程序实现以下功能：

将工单路由到正确的处理队列。
将提取的实体写入数据库行。
使用类型化参数触发下游工具。
运行对字段进行断言的评估工具集。
生成可观测性平台可解析的遥测数据。

在这些场景中，你需要的不是人类可解释的描述性文本，而是程序可以验证、重试并在测试中进行diff比较的可预测模式。

在典型的产品中，终端用户几乎不会看到 JSON，也无需主动请求。JSON 是后端与模型之间的幕后约定：界面呈现友好文本，而服务器则读取机器可读部分以决定后续操作（存储、路由、调用工具、显示按钮等）。

由此产生两种常见响应模式：

双通道响应：模型返回供界面显示的人类可读文本，同时为应用程序提供小型 JSON 元数据对象。
纯结构化响应：适用于批处理/ETL 作业、智能体循环和评估场景，完全不需要人类可读文本。

因此，当下游系统需要确定性时，请使用结构化输出：当你必须验证字段、运行自动化测试、支持幂等重试或安全链接工具调用时。若输出纯面向人类且无需代码解析，则可跳过此步骤。对于 DeepSeek 模型，使用其原生结构化模式（JSON 或工具）并仍在提示中声明模式期望，将获得最佳可靠性。对于推理模型，请仅将机器可读部分放在最终回答通道中，切勿将其隐藏的思考内容反馈至对话中；接下来我们将具体展示实现方法。

获取结构化数据有两种方式：让模型生成，或由客户端强制执行。以下将介绍四种渐进式增强的技术方案：

原生 JSON 模式：要求模型直接返回 JSON 对象——简单快捷，适用于数据提取和小型模式。
函数/工具调用：允许您定义模型必须填充的模式（参数）以调用工具，非常适合输出需要触发代码路径的场景。
Pydantic 类型强制生成（通过 Instructor）：验证模型回复是否符合 Python 类型，提供精确错误信息和自动重试；我们将展示Mode.TOOLS和Mode.MD_JSON（适用于推理模型）两种用法。
生产环境防护机制：提供提示级别的模式提示、低温度参数、最大令牌数设置、重试修复循环，以及在支持的情况下使用受限生成以获得严格保证。

所有示例均使用兼容 OpenAI 的聊天补全接口，但针对 DeepSeek 模型/端点。

首先了解如何设置原生 JSON 模式。

原生 JSON 模式

如果您有 OpenAI 使用经验，JSON 模式的设置会显得异常熟悉。请查看以下代码片段：

from openai import OpenAI
import json
client = OpenAI(
    api_key=os.getenv("DEEPSEEK_API_KEY"),
    base_url="https://api.deepseek.com"
)
response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {
            "role": "system",
            "content": "Extract user information as JSON. Example: {\"name\": \"Alice\", \"age\": 30}"
        },
        {
            "role": "user",
            "content": "My name is Bob and I'm 25 years old"
        }
    ],
    response_format={"type": "json_object"},
    temperature=0.1  # 保持确定性
)
data = json.loads(response.choices[0].message.content)

此代码片段通过 OpenAI 兼容 API 调用 DeepSeek，从句子中提取结构化数据。它创建客户端（使用DEEPSEEK_API_KEY和base_url），向deepseek-chat模型发送包含系统指令和用户消息的聊天请求，通过response_format={"type": "json_object"}启用原生 JSON 输出，同时保持低随机性（temperature=0.1）。模型返回 JSON 字符串（例如{"name":"Bob","age":25}），最后一行使用json.loads(...)将其解析为 Python 字典（data）供程序使用。

使用response_format={"type": "json_object"}获得最佳结果时，请遵循以下准则：

明确请求 JSON：提示词中应包含类似以 JSON 格式返回输出的短语以强化指令。
提供模式示例：在提示中包含简单 JSON 结构（如{"name": "string", "age": "number"}）可显著提高遵循度。
合理设置max_tokens：确保令牌限制足以容纳完整 JSON 响应，防止截断。
使用低温度值：建议温度值在 0.1 到 0.3 之间，以最小化偏离请求模式的创造性偏差。

有时 JSON 可能无法满足需求——通常出现在 JSON 对象包含复杂逻辑或多层嵌套时。达到某个临界点后，你将开始遇到错误或意外输出。此时可使用函数调用功能。

函数调用

针对更复杂的结构化数据需求，DeepSeek 支持符合 OpenAI 规范的函数调用功能。该设计选择显著简化了集成流程，因为为 OpenAI 函数调用 API 构建的现有代码库只需对客户端配置进行最小调整即可适配 DeepSeek。

tools = [{
    "type": "function",
    "function": {
        "name": "analyze_sentiment",
        "description": "Analyze the sentiment of text",
        "parameters": {
            "type": "object",
            "properties": {
                "text": {"type": "string"},
                "confidence": {"type": "number", "minimum": 0, "maximum": 1}
            },
            "required": ["text", "confidence"]
        }
    }
}]
response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[{"role": "user", "content": "I love this product!"}],
    tools=tools,
    tool_choice="auto"
)

在 DeepSeek 提供的兼容 OpenAI 的 API 中，工具调用是总称，而函数调用是你通过 JSON-Schema 参数块定义的具体工具类型。你注册一个或多个工具（类型function），模型决定调用哪个工具，并返回包含函数名和 JSON 参数的tool_call。你的应用程序随后验证并执行该函数。这与我们在智能体中使用的思维模型相同。我们将在第 6 章深入探讨函数调用与结构化输出，以及如何结合两者创建智能体。此处我们将聚焦结构化输出：使用单个函数作为类型化信封以强制预测字段。

函数调用与模式约束生成

函数调用示例声明了一个模型必须填充的模式（schema），将结构从礼貌的文本请求转移到机器可检查的契约。在运行时，模型不会生成自由格式文本；它生成一个具有严格类型参数（数字、枚举和嵌套对象）的工具调用，您可以对其进行验证、记录、重试，并传递给实际代码。这减少了提示漂移，消除了脆弱的正则表达式解析，并产生了清晰的失败模式：当输出违反模式时，您可以确定性地检测到它，并触发修复或重试。该模式现在已成为行业标准：OpenAI、Anthropic（tools）、Google（function calling）、Cohere（tool use）和 Mistral 都支持模式约束调用。当您只需要一坨结构化数据时，应使用 JSON 模式；当输出是带参数的操作、模式是深层/嵌套的，或者您希望模型在多个操作中进行选择时，则使用函数/工具调用。

您可以定义清晰的 Pydantic 模型来表示所需的输出（例如对象、列表或嵌套数据），而不是处理非结构化的文本响应，Instructor 库可以确保模型遵循这些定义。我们接下来将讨论这一点。

通过 Instructor 使用 Pydantic 进行类型强制生成

如您所知，Instructor 是一个 Python 库（https://python.useinstructor.com/），它通过直接与 Pydantic 集成，使提示大型语言模型变得结构化且可靠。它会自动验证响应，在模式错误时重试，并提供类型安全，使提示更加稳健和可预测。这种方法不仅简化了提示的编写方式，还将 LLM 输出转换为可靠的、强类型的数据结构，可以无缝集成到实际应用中。

对于 DeepSeek，您可以使用 Instructor 在 JSON 模式或工具调用之上添加一个类型强制层。您给它一个 Pydantic 模型；它负责处理提示、解析响应、验证 类型/范围/枚举，并在失败时可以自动修复——返回一个真正的 Python 对象，而不是原始的 JSON 字符串。对于推理模型，Mode.MD_JSON 保持低延迟（无工具往返），同时仍然强制执行模式；对于工具密集型流程，Mode.TOOLS 可插入函数调用。因为 Instructor 与传输层无关，它可以在 JSON 和工具两种模式上运行，其价值在于验证、重试和类型化对象。

让我们看看如何使用 Instructor 实现类型安全。

使用 Instructor 实现类型安全

您可以使用 Instructor 通过 Pydantic 驱动的验证来包装 DeepSeek 的 API。以下是一个演示此功能的示例。

import instructor
from pydantic import BaseModel, Field
client = instructor.from_provider(
    "deepseek/deepseek-chat",
    base_url="https://api.deepseek.com",
    mode=instructor.Mode.TOOLS  # 或者对于推理模型使用 Mode.MD_JSON
)
class ProductReview(BaseModel):
    rating: int = Field(ge=1, le=5, description="1-5 星评分")
    summary: str = Field(max_length=100)
    pros: list[str]
    cons: list[str]
###### 魔法发生在这里 - 保证返回有效的 ProductReview 对象或抛出异常
review = client.chat.completions.create(
    messages=[{"role": "user",
         "content": "iPhone 15 评测：摄像头很棒，电池续航还可以更好..."}],
    response_model=ProductReview
)

专业提示：对于推理模型使用 Mode.MD_JSON 以避免工具调用开销。这就像选择风景路线——更慢（低延迟）但更可靠。

这段代码片段使用 Instructor 包装了 DeepSeek 的 OpenAI 兼容 API，对模型的输出强制执行 Python Pydantic 模式。from_provider(...) 创建了一个绑定到 deepseek-chat 的客户端，并选择了一个强制模式（示例中是 Mode.TOOLS，它在底层使用函数/工具调用；Mode.MD_JSON 是替代方案，在纯 JSON 上强制执行模式）。ProductReview Pydantic 模型就是那个契约：一个约束在 1-5 之间的整数 rating，一个上限为 100 个字符的 summary，以及两个字符串列表。当您调用 client.chat.completions.create(..., response_model=ProductReview) 时，Instructor 执行以下操作：

将模式呈现给模型（作为工具模式或 JSON 模式，取决于模式设置）。
接收模型的结构化参数。
解析它们。
根据 Pydantic 约束验证每个字段——成功时返回一个真正的 ProductReview 对象，如果输出格式错误则引发错误。

换句话说，您不是在请求 JSON；您是在保证您的代码可以依赖的类型化数据。

当下游代码需要强保证或清晰的失败模式时，请使用此方法：写入数据库、触发带参数的操作、运行对字段进行断言的评估，或者构建必须是类型安全的代理步骤。当您想要模式强制而无需工具的繁琐时，对于推理模型优先选择 Mode.MD_JSON；当您已经处于工具/代理设置中、模式是深层或嵌套的，或者您希望模型从带类型参数的多个工具中选择时，优先选择 Mode.TOOLS。无论哪种方式，Instructor 都集中了解析和验证，使您的应用程序逻辑保持简单和稳健。

推理风格的 DeepSeek 模型（例如 deepseek-reasoner）输出两个通道：隐藏的 reasoning_content（思考过程）和最终内容（答案）。您的验证器和解析器必须仅在答案上操作——永远不要将模型的思考过程回显或反馈到对话或下游调用中，否则您会触发模式检查，甚至在下一次请求时遇到 400 错误。

下一节将涵盖这些边缘情况和强化模式——剥离思考通道、重试修复循环以及生产环境的其他保障措施。

稳健性和特殊情况的处理策略

还记得我们说过 R1 模型在说话之前会思考吗？这为结构化输出带来了独特的挑战。deepseek-reasoner 模型（https://api-docs.deepseek.com/guides/reasoning_model）输出两个通道：reasoning_content（思考过程）和 content（最终答案）。如果您没有正确处理这个问题，下次 API 调用时会收到恼人的 400 错误。

# 假设 `client` 是上面创建的经 Instructor 包装的 DeepSeek 客户端。
from pydantic import BaseModel
class DateInfo(BaseModel): date: str
class TimeInfo(BaseModel): time: str
# 1) 第一次调用 - 模型“思考”并回答；我们只保留答案（`content`）
completion, raw = client.chat.completions.create_with_completion(
    model="deepseek-reasoner",
    messages=[{"role": "user", "content": "以 JSON 格式提取日期：会议定在 3 月 15 日。"}],
    response_model=DateInfo,
    temperature=0
)
print(raw.choices[0].message.reasoning_content)  # 可选：查看思考过程
assistant_answer = raw.choices[0].message.content  # <-- 只保留这个用于历史记录
# 2) 下一次调用 - 我们实际喂给模型的内容（注意：任何地方都没有 `reasoning_content`）
next_messages = [
    {"role": "assistant", "content": assistant_answer},
    {"role": "user", "content": "现在以 JSON 格式提取时间：14:30 开始。"}
]
_ = client.chat.completions.create(
    model="deepseek-reasoner",
    messages=next_messages,
    response_model=TimeInfo,
    temperature=0
)

剥离思考过程的步骤是 R 系列推理模型特有的，因为它们返回两个通道：reasoning_content（隐藏的思维链）和 content（最终答案）。

处理助手消息重放与结构化输出问题

如果你在下一个请求中天真地重放整个助手消息（例如，通过序列化原始对象），你最终会发送一个 OpenAI 兼容 API 不接受的字段（reasoning_content）。这违反了请求模式，服务器将响应 HTTP 400 Bad Request——这不是一个无限循环，只是一个无效载荷。解决方法很简单：持久化并仅重放 content，绝不重放 reasoning_content。（聊天/代码模型不发送 reasoning_content，所以这个陷阱是推理端点特有的。）

结构化输出可能遇到的问题

重试修复循环

即使有了这些安全措施，有时 DeepSeek 仍可能输出无效的 JSON。与其束手无策，不如让它修复自己的混乱：

def extract_with_retry(prompt, model_class, max_retries=2):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(
                messages=[{"role": "user", "content": prompt}],
                response_model=model_class
            )
        except Exception as e:
            if attempt < max_retries - 1:
                prompt = f"The previous JSON was invalid:\n{str(e)}\n\nPlease fix and return valid JSON for: {prompt}"
            else:
                raise

这个辅助函数围绕类型化生成调用实现了一个自修复循环。它要求模型生成一个符合 Pydantic 模式（model_class）的值，并立即验证回复；如果验证（或 JSON 解析）失败，except 块会重写提示，包含确切的错误消息和简单指令——the previous JSON was invalid… please fix and return valid JSON——然后重试最多 max_retries 次。换句话说，你将一个无结构的失败转化为一个新的、高度约束的任务交给模型，这通常在第二次尝试时成功，因为它现在看到了问题所在（例如，缺少必需字段、类型错误、尾随逗号、截断的输出）。

在任何你偶尔遇到模式漂移的地方都使用此模式，尽管提示已足够好：从混乱文本中进行批处理/ETL 提取、对字段进行断言的评估工具、必须交接类型化参数的代理步骤，或者最终答案有时会泄露多余文本到 JSON 中的 R1 运行。保持重试次数较少（2-3 次），记录异常，并将循环与低温度、足够的 max_tokens 以及内联模式/示例配对；如果失败持续，则升级（回退到具有更严格模式的工具/函数调用，或引入人工环节）。避免将巨大的错误转储粘贴回模型（清理并保持简短），并确保对于推理模型，你只验证答案通道（content），绝不验证他们隐藏的 reasoning_content。

模式是什么意思？

这里没有服务器端验证器。模型是基于你提供的模式进行调节的；你的客户端代码负责验证结果。

你可以通过两种常见方式提供该模式：

原生 JSON 模式：请求 JSON 并在提示中包含一个小示例（或类似 JSON-Schema 的提示）——适用于简单的提取。
函数调用：在 parameters 下传递一个 OpenAI 兼容的 JSON 模式。模型返回一个 tool_call，其参数必须与你声明的 properties 匹配（类型、必填字段和嵌套）。

每当你的下游代码需要类型化参数（例如，路由、写入数据库行或触发操作）或你的模式是嵌套的，就使用工具模式。

所有权撇号错误

在 R1-0528 中，我们有时看到 JSON 中的键失去了撇号（例如，user's_name → users_name）。包含撇号的属性名——特别是排版用的 U+2019（'）——在训练数据中很少见，被标记为标点符号，因此模型通常会规范化或丢弃它们（例如，发出 users_name 而不是 user's_name）。这会破坏严格的验证器和任何期望精确键匹配的代码。

通过以下任一方法修复它：

避免在属性名中使用标点符号（例如，优先使用 ASCII snake_case）。
显示明确的示例，并在验证期间接受常见变体。可以通过别名和规范化捕获常见变体：

以下是如何操作：

# Tool schema with risky keys (curly apostrophes)  -  works but fragile:
tools = [{
  "type": "function",
  "function": {
    "name": "capture_profile",
    "parameters": {
      "type": "object",
      "properties": {
        "user's_name": {"type": "string"},
        "company's_address": {"type": "string"}
      },
      "required": ["user's_name", "company's_address"]
}}}]
# Robust parsing with Pydantic: accept curly, straight, or dropped apostrophes
from pydantic import BaseModel, Field, AliasChoices
class Profile(BaseModel):
    users_name: str = Field(validation_alias=AliasChoices("user's_name","user's_name","users_name"))
    company_address: str = Field(validation_alias=AliasChoices("company's_address","company's_address","company_address"))
    model_config = {"populate_by_name": True}  # allow using field names in your code
# Better yet: instruct the model to emit ASCII snake_case keys only:
# "Return JSON with keys: users_name, company_address (ASCII only, no punctuation)."

实际上，每当需要类型化、机器检查的字段时就使用工具/JSON 模式，并避免在属性名中使用标点符号；如果你无法重命名键（遗留契约），则添加别名选择和一个轻量的预规范化步骤（例如，验证前替换 →）。这使你的管道具有弹性，而不隐藏重要的错误。

表 3.2 可作为你可以采用的结构化输出策略的快速参考。

| 需求 | 解决方案 | 注意事项 |

| :--- | :--- | :--- |

| 简单 JSON 提取 | response_format + 示例 | 必须在提示中包含 json |

| 复杂验证 | Instructor 库 | 对推理模型使用 Mode.MD_JSON |

| 保证有效输出 | Fireworks 端点 | 不同的 API，成本更高 |

| 审计跟踪 | 使用推理模型 | 在下次调用前剥离 reasoning_content |

| 从坏 JSON 中恢复 | 重试修复循环 | 通常第二次尝试就成功 |

表 3.2：用于有效 DeepSeek 提示的结构化输出策略

自 0528 版本之前的混乱时期以来，DeepSeek 的结构化输出能力已经取得了长足的进步。虽然它不如 OpenAI 的实现那么精致，但对于生产使用来说已经绰绰有余，尤其是当装备了 Instructor 或类似库时。

关键是理解 DeepSeek 的怪癖：始终提供示例，保持低温度，并记住即使使用原生 JSON 模式，模型也欣赏（甚至要求）一个温和的提醒，即你想要 JSON。这就像与一个才华横溢但略显古怪的同事合作。一旦你了解了他们的偏好，你就可以一起完成惊人的事情。

为什么选择 DeepSeek

如果 DeepSeek 可能很挑剔，为什么团队仍然选择它？在实践中，这种权衡是很有吸引力的：

成本或性能异常强劲：每百万令牌费率（含缓存折扣）低于许多同行，因此高吞吐量推理工作负载在经济上可行。
推理质量在数学/逻辑/代码方面具有竞争力：因此，你通常需要更少的提示脚手架就能得到正确答案。
切换成本低：API 是 OpenAI 兼容的（相同的 SDK、函数/工具调用、JSON 模式），因此现有后端可以以最小的更改接入。
生态系统支持：托管选项为需要更严格结构时增加了约束或基于语法的生成。
可选性：R1(-0528)及其蒸馏变体可在 Hugging Face（甚至是供应商量化的构建）上使用，这支持自托管和混合架构。这些怪癖主要转化为有纪律的提示（陈述模式、保持低温、剥离推理通道），对于许多团队来说，节省加上吞吐量和可选的自托管值得付出额外的谨慎。

接下来，我们来看 DeepSeek V 系列模型如何以自己独特的方式处理提示词。

V 系列：V 系列模型的专属提示技术

R 系列模型针对复杂推理任务进行了优化，而 V 系列模型则被设计为功能强大的通用对话代理。它们在广泛的应用场景中展现出高性能，但其行为由一套独特的操作特性所支配。为了充分利用其全部能力并确保可预测的结果，理解并适应这些独特的提示要求至关重要。

模板探戈：Unicode 字符与特殊标记

接下来要讲的，是 V 系列真正变得“奇特”的地方。虽然 R 系列大多忽略格式上的细节，但 V 系列模型对其聊天模板极为挑剔——它们使用的 Unicode 字符可能会让你怀疑自己的眼睛。

聊天模板是一个确定性的格式化层，它将你结构化的消息 =[...] 转换为模型训练时所见的确切字节序列。它会插入模型的特殊标记（句首/句末、角色标签、轮次分隔符），处理模式切换（例如，V 系列的 `` 与 </think>），并保留所需的 Unicode 令牌（全角 |、低区块 _）。

使用官方模板能保持分词稳定性和高质量；手动拼接字符串（或使用 ASCII 近似字符）则很脆弱，并可能显著降低性能。在实践中，应让你的服务提供商 SDK 或 transformers 库的 tokenizer.apply_chat_template(...) 来构建提示，合并成一条系统消息，并确保助手回合以 <|end_of_sentence|> 结尾，以便解码能干净地停止。

例如，考虑以下提示模板，它用于开启思考模式的单轮对话。

<｜begin_of_sentence｜>{system}<｜User｜>{query}<｜Assistant｜>

此模板中的字符并非标准 ASCII。竖线（|）是全角 Unicode 字符（U+FF5C），而看似下划线的字符是一个特殊的区块字符（U+2581）。使用错误的、视觉上相似的 ASCII 等效字符（|、_）会导致分词不匹配，并显著降低模型性能。这一点至关重要：尽管视觉相似，但对模型的分词器而言，这些字符在功能上是不同的。

我需要手动输入这些奇怪的字符吗？

通常不需要。当你通过聊天 API（OpenRouter、提供商 SDK 等）调用 V 系列或 R 系列时，服务器会为你应用模型的聊天模板。你传递结构化的消息 =[...]，并（对于 V3.1）可选地传递一个推理/思考模式标志；提供商服务会构建正确的提示字符串，包含全角竖线（|，U+FF5C）和 SentencePiece 低区块字符（_，U+2581），这样你就不会意外地将它们替换为 ASCII 近似字符。

如果你使用 Hugging Face Transformers 在本地运行，请始终使用 tokenizer.apply_chat_template(...)（对于 V3.1，选择 thinking=True/False），而不是自己拼接字符串。这可以避免细微的分词不匹配，从而防止性能可衡量地下降。

在构建我们的提示模板时，你可以遵循这些 V 系列模板规则（如果你坚持要手动设置）：

使用分词器的 apply_chat_template() 方法——不要试图手动构建。
始终用 <|end_of_sentence|> 结束助手回复，否则有无限生成的风险。
开头只能有一条系统消息（合并多个系统提示会降低性能）。

这就是如何与 V 系列“对话”；现在让我们来管理它如何“回应”——常常是带着过多的 Markdown。

格式化狂欢：当 Markdown 失控时

V 系列模型，尤其是 V3，在输出内容方面有一些怪癖。

这是你默认会得到的结果：

"""
*Really* interesting point here.
**Another** observation.
A single sentence paragraph.
Yet *another* single sentence.
**Bold** conclusion with *emphasis*.
"""

# What you probably wanted:
"""
Here's an interesting point that flows naturally into the next thought.
The observation connects seamlessly with the previous context, creating
a cohesive narrative that doesn't feel like a TG from 1952.
The conclusion emphasizes key points without resorting to typographical
fireworks every other word.
"""

如何解决？在你的提示中明确表达格式偏好：

system_prompt = """
Write in flowing, connected paragraphs. Avoid:
- Single-sentence paragraphs unless for emphasis
- Excessive use of *italics* or **bold** formatting
- Breaking thoughts into choppy segments
"""

eg7.py github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg7.py 提供了一个紧凑的 A/B 测试框架，它将格式作为生成中的唯一变量进行隔离：它向同一个 DeepSeek V3 模型发送两次相同主题的提示——第一次没有任何风格指导，第二次则附带一个简短的、系统级的风格规范——并将结果并排打印出来。由于内容、模型和主题保持不变，任何差异都可以归因于指令的角色和具体性，使得风格策略的效果一目了然。

约束条件被表述为清晰、可测试的规则而非模糊的形容词，并且将它们放在系统消息中可以全局应用，而 V3 往往能更可靠地遵循。该文件甚至包含了一个捕获的转录，突出了 Markdown 密集的基线版本与更流畅、连贯的散文之间的对比，并且你可以替换规范来测试不同的风格，例如无列表、最多两段或行政语调。

实际要点：将格式视为你提前设置的可控约束，而不是生成之后杂乱的清理步骤。

现在文章不再“大喊大叫”了，但要确保模型确实能读懂它。下一个失败模式是无形的：微小的默认上下文窗口和无声的截断。

上下文窗口的困惑

V 系列模型宣传拥有巨大的上下文窗口（163k+ 令牌），但有个问题。默认配置通常只显示 8k 令牌，并且许多推理框架会默默在该限制处截断，除非你明确设置 max_model_len，如下所示：

# 错误（默默在 8k 处截断）：
model = load_model("deepseek-v3")
# 正确：
model = load_model("deepseek-v3", max_model_len=65536)

专业提示：V 系列使用 多头潜在注意力（MLA）来压缩 KV 缓存，以实现高效的长上下文推理。与 MLA 无关，Transformer 在长提示中往往表现出 U 形的位置偏差（“中间丢失”），因此请将最关键的指令放在开头和结尾——不要把它们埋在提示中间。

为什么大多数运行时默认将 V 系列限制在 ~8k，而模型宣传有 100k+？因为上下文长度是一个预算，而非强制规定。长输入在三方面代价高昂：

预填充延迟大致随输入长度线性增长，因此一个 50–100k 令牌的提示即使对短答案也可能主导端到端时间。
KV 缓存随长度线性增长并占用 GPU 内存，这会降低吞吐量并损害并发性。
尾部延迟或超时风险上升，尤其是在共享集群中。DeepSeek 的 MLA 通过激进压缩 KV 缓存（据报道约减少 93%）使这变得便宜一些，但它并没有让上下文长度变得免费——缓存仍然随长度 L 缩放。

因此，提供商和框架会采用保守的限制以保持服务稳定；如果你确实需要更多，你必须通过提高 max_model_len 来选择加入，最好配合性能分析以确保你不会搞炸延迟或容量。

你应该扩大窗口吗？

只有当那些额外的 token 在执行实际工作时，才是有益的。更多的 token 并不意味着更好的推理能力。语言模型在处理长提示时会表现出 U 形位置偏见（lost in the middle）。提示的开头和结尾会获得不成比例的更多关注。因此，塞入 60k 未经消化的文本反而可能损害准确性。相反，你应该选择使用检索和摘要，保持指令精简；只有在真正需要原样保留长段文本（如合同、长日志、多文件代码差异）或维持超长对话时，才扩展上下文窗口。

在实践中，应测试不同的提示长度，测量准确率与延迟/成本的关系，并保留满足质量要求的最小窗口。

在窗口大小设置正确、关键指令被放置在开头和结尾之后，你就准备好使用 V 系列在纯推理能力之外所提供的额外功能了。

是时候启用这些附加功能了：FIM、原生工具调用和保证的 JSON 模式。

隐藏的超能力：R 系列不具备的功能

R 系列专注于推理，而 V 系列则在此过程中掌握了一些额外技巧：

填充中间 (FIM)：FIM 使模型能够在给定前缀和后缀之间生成缺失的中间部分，这样你就可以就地编辑代码，而不仅仅是追加续写。这里的hole简单指代占位的中间部分——即模型填充在固定前缀和后缀之间的 <｜fim_hole｜> 区域，同时保留周围的代码不变。
```
prompt = """<｜fim_begin｜>
def calculate_average(numbers):
<｜fim_hole｜>
    return sum(numbers) / len(numbers)
<｜fim_end｜>"""
# V 系列将生成缺失的验证逻辑
```
此提示标记了三个区域：空洞前的前缀（函数签名）、应生成代码的空洞，以及空洞后的后缀（返回语句）。根据这些锚点，模型会推断中间应该是什么——通常是输入检查或保护子句——因此你会看到它在返回平均值之前添加诸如处理空列表、None或非数字项之类的验证。换句话说，你是在要求模型精确地填充缺失的函数体，同时逐字保留周围上下文。
原生工具调用：原生工具调用使模型能发出结构化的函数调用（名称 + JSON 参数），而非普通文本，这样你的运行时就能执行工具并反馈结果。
```
system = """## Tools
### Function: get_weather
Description: Get current weather
Parameters: {"location": "string"}
"""
# V 系列将使用特殊 token 发出结构化的工具调用
```
系统提示声明了一个可调用的工具 get_weather 及其预期的参数模式。当用户询问天气时，模型不会用自由文本回答；它会产生一条结构化的工具调用消息（模式因 SDK 而异），选择 get_weather 并附带类似 {"location": "Delft"} 的参数。你的应用程序拦截此调用，运行真实工具，并将工具输出作为工具结果消息返回，之后模型才编写最终用户可见的答案。关键点是：模型自主决定何时调用该函数，并提供格式正确的参数，而不是猜测 URL 或抓取文本。
保证的 JSON 模式：仅 JSON/严格 JSON 模式约束了解码过程，使模型仅返回语法有效的 JSON（且无其他内容），这非常适合程序化消费。
```
system_prompt += "\n## Response Format\nReply with JSON object ONLY."
# 强制输出有效的 JSON，无需额外库
```
添加的 Response Format 指令告诉模型只输出一个顶层的 JSON 对象，不包含评论、代码围栏或尾随文本。在实践中，你会将此指令与客户端的 JSON/结构化输出设置（如果可用）配对，以强制保证有效性；两者结合，能可靠地生成下游代码可直接解析的干净 JSON。如果你还提供一个最小化模式或示例，你将进一步减少偏差，获得稳定的键和类型。
推理残留：一些 V 系列检查点偶尔会显示内部思考痕迹（例如 …</think>）；这些对正确性并非必需，通常最好予以抑制。V2.5 从 R1 继承了一些推理行为，偶尔会产生不需要的思考输出：
```
User: What's 2+2?
Assistant: This is a simple arithmetic problem...</think>
The answer is 4.
```
在此示例中，助手在陈述 The answer is 4. 之前暴露了一个内部思考块。该痕迹是继承的推理行为的残留，而非有意设计的功能。

在生产环境中，可以通过指示来抑制它：Do not include tags or intermediate reasoning; return only the final answer，和/或通过后处理从输出中过滤这些标签。你仍然会得到正确结果，但没有了冗长的解释，除非你明确要求审计跟踪。

V3 基本消除了这一点，但当你只想要答案时，可能仍会看到冗长的解释。与推理是有意为之的 R 系列不同，V 系列的推理痕迹是残留的——除非你特别需要审计跟踪，否则应予以抑制。
温度与采样：越低越好。V 系列的采样甜点范围往往更窄，因此小幅提高温度会比许多其他模型更快地降低格式保真度和事实性。V 系列模型对温度比大多数模型更敏感：

| 模型 | 推荐温度 | 原因 |

| :--- | :--- | :--- |

| V 系列（通用） | 0.3 | 更高温度会迅速降低连贯性 |

| V 系列（代码） | 0.0 | 确定性最适合代码 |

| V 系列（创意） | 最高 0.5 | 即使是创意任务，超过 0.5 也会受损 |

表 3.3：各种 V 系列模型的推荐温度

对比 OpenAI 的 GPT-4o 常用 0.7-1.0，或偏好 0.6 的 R 系列。V 系列模型似乎拥有更窄的甜点范围，超过则输出就会变得奇怪。对于结构化或高精度任务，保持低温度（例如，约 0.15-0.35），如果需要多样性可适度调整 top_p；这能保持输出稳定，并降低出现冗长解释或 JSON 损坏的风险。如果你稍后提供配置代码片段，可以注明所选值旨在实现近乎确定性的解码，同时保留足够的随机性以避免重复，并且提高温度应谨慎且逐步进行。

以下是 DeepSeek V 系列和 R 系列不同能力的快速回顾。

| 方面 | V 系列 | R 系列 |

| :--- | :--- | :--- |

| 系统提示 | 支持一个，非常靠前 | 大多忽略（除 R1-0528） |

| 格式化要求 | 非常严格的 Unicode 模板 | 灵活，仅用户提示 |

| 默认行为 | 对话式，可能啰嗦 | 先推理，后回答 |

| 温度甜点 | 0.0-0.3 | 0.6 |

| 特殊功能 | FIM、原生工具、JSON 模式 | 内部 CoT 推理 |

| 身份问题 | 可能声称是 ChatGPT | 知道自己是谁 |

| 上下文窗口 | 163k+（需配置） | 128k（开箱即用） |

表 3.4：V 系列与 R 系列 DeepSeek 模型的功能对比

V 系列模型是强大的对话智能体，是专注于推理的 R 系列的对应物。

有效使用 V 系列模型关键在于精确控制提示结构。这包括严格遵守其特定的聊天模板（该模板使用非标准 Unicode 字符），以及显式配置参数如 max_model_len 以启用其完整上下文窗口。

此外，开发者必须准备好管理模型独特的格式化倾向和温度敏感性，以确保输出一致且可预测。

在设定好轨道之后，你已准备好应对罕见的脱轨情况。下一节将症状映射到原因，以便你能一次纠正错误。

故障排除

DeepSeek 故障排除指南

即使是最精心设计的提示词，在使用 DeepSeek 时也可能出状况。这份实用指南涵盖了最常见的问题及其修复方法——可以把它看作是当模型开始“自由发挥”时的应急工具包。

以下是一些您可以轻松修复的常见问题：

当 R1 忘记思考时

症状：R1 跳过了标志性的推理阶段，直接给出答案，导致结果肤浅或不正确。

解决方案：通过预置预期标记来强制启动思考过程：

# 核心选项 - 从字面上开始它们的回复
messages.append({
    "role": "assistant",
    "content": "\n"
})
# 然后继续正常的生成

此处的 messages 是与您传递给补全调用的聊天历史数组相同。您在这个数组的末尾、调用模型之前，追加一个包含内容 "\n" 的合成助手轮次。由于最新的轮次来自助手，模型会将其视为自身回复的开始，并在 `` 之后继续生成，这能可靠地启动一个推理块。在此预填充之后不要再添加新的用户轮次。如果您向最终用户展示输出结果，请剥离 …</think> 部分（或使用提供商的隐藏推理选项），以便他们只看到最终答案。

这项技术有效地预设了模型的生成条件，确保在尝试制定直接答案之前，其内部推理过程得以启动。成功应用此方法的标志是在模型输出中，最终答案之前出现推理链。

多语言漫游谜团

症状：您的英文提示词突然得到混杂着中文字符的响应，尤其是在复杂推理过程中。例如：

The solution involves 计算 the derivative...

解决方案：在您的用户 prompt 中添加明确的语言约束：

prompt += "\nRespond ONLY in English. Do not use any Chinese characters."

为什么会发生这种情况？DeepSeek 在训练时使用了包含大量中文的多语言数据。在认知负荷较高时，它有时会回退到其“母语”。这就像双语者在做心算时可能会切换语言一样。不过请注意，如果最终结果或返回的响应本身没有问题，这可能不会造成太大影响，尽管对于英语使用者来说，它可能会损害您理解推理链的能力。

重复循环，当 DeepSeek 卡住时

症状：模型重复输出相同的短语或模式：

Therefore... Therefore... Therefore... Therefore...

修复组合：您可以遵循以下步骤来修复重复问题。在设置模型进行响应时，请添加这些额外的配置细节。

略微提高 temperature（如果低于 0.5）：temperature=0.6。
添加重复惩罚：frequency_penalty=0.3。
设置合理的 max_tokens：不要让它无限制地冗长。
使用 top_p 采样：top_p=0.95 有助于避免局部最优。

重复通常意味着模型陷入了概率谷底。你需要加入足够的随机性让它“爬出来”，但又不能完全陷入混乱模式。

多数投票

当有疑问时，采取民主方式。对于准确度比速度更重要的关键任务，请使用 DeepSeek 自身的基准测试策略：

from collections import Counter
def get_best_answer(prompt, n_samples=5):
    responses = []
    for _ in range(n_samples):
        response = client.chat.completions.create(
            model="deepseek-r1",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.6
        )
        responses.append(extract_answer(response))

    # 返回最常见的答案
    return Counter(responses).most_common(1)[0][0]

这段代码片段的位置：将其放在您当前发起单个 chat.completions.create(...) 调用的地方——您的推理层（API 处理程序、作业或评估工具）。您不是进行一次调用，而是使用非零的 temperature（例如 0.4–0.8）运行 n_samples 次独立调用以产生多样性，使用 extract_answer(...) 来规范化每个结果（剥离推理、修剪空白、规范化 JSON，或提取 Final Answer: 之后的值），然后通过 Counter(...).most_common(1)[0][0] 返回得票最多的答案。

get_best_answer(...) 中的循环正是如此：收集 n_samples 个答案，将它们折叠为可比较的字符串，并选择出现频率最高的那个。如果两个答案票数相同，Python 的 Counter 会按首次出现的顺序打破平局；在生产环境中，您可以重新采样、增加 n_samples，或应用确定性的平局打破规则（例如，最短的有效 JSON 或模式验证器得分）。

延迟和成本预计会与 n_samples 大致成线性关系；为了保持较低的执行时间，请并行运行这些调用。对于准确度比速度更重要、错误是随机而非系统性的高风险查询，请使用此模式；如果错误是系统性的而非随机的，则应在依赖投票之前先修复提示/约束。

这并非偏执——DeepSeek 自己的团队在基准测试中使用了 64 样本多数投票（arxiv.org/pdf/2501.12948）。对于生产环境，除非是在进行竞争性数学问题，否则通常 3–5 个样本就足够了。

特定于格式的陷阱

在使用 DeepSeek 时，您可能会遇到一些与输出相关的差异。让我们来谈谈它们：

JSON 模式静默失效：在某些情况下，当对非常长或复杂的提示使用 JSON 模式时，API 可能会返回一个空响应而不报错。这可能发生在模型难以生成一个适合内部处理限制的有效 JSON 对象时。最可靠的解决方案是缩短提示长度或重置对话历史，仅使用最近的消息重试请求。具体如何操作取决于您的用例或应用程序，但基本原则如下所示：

# 症状：使用 response_format 时收到空响应
# 解决方案：缩短提示或重置对话
if not response.content:
    # 使用较短的上下文开始新对话
    messages = [messages[-1]]  # 只保留最新消息

输出截断：一个常见的失败模式是收到一个语法上不完整的 JSON 对象，因为生成被 max_tokens 限制提前终止了。模型在生成响应时并未考虑此限制，因此可能会被突然截断。为防止此情况，请计算预估的响应大小，并为 max_tokens 设置一个充裕的缓冲区，例如比预估值大 20%。

# 症状：{"result": "answer", "confidence": 0.9
# 解决方案：始终为 max_tokens 增加 20% 的余量
estimated_tokens = len(prompt.split()) * 2
max_tokens = int(estimated_tokens * 1.2)

故障排除检查顺序

当 DeepSeek 行为异常时，请按以下顺序进行检查：

temperature：如果 temperature 过低（约 0.3–0.5 或更低），模型可能会重复自己或退化为通用文本；如果过高（约 0.7 或更高），输出会变得不稳定。对于大多数生产提示，从 0.3–0.6 的区间开始并谨慎调整。
模板完整性：V 系列模型要求其精确的聊天模板和 EOS 标记 <｜end_of_sentence｜>。缺失标记或使用 ASCII 相似字符会降低质量或导致解码无休止。请使用官方模板函数构建提示，而不是手动拼接字符串。

Prompt 调试与迁移指南

调试检查项

3. 令牌限制

中途截断通常意味着 max_tokens 过小；静默输入截断表示您配置的 max_model_len 小于提示长度。请增加 max_tokens，明确设置 max_model_len，并定义停止序列以使答案整洁地结束。

4. 语言控制

如果需要仅英文输出，请在提示顶部明确声明，并避免在上下文中使用多语言示例。少样本或上下文中的混合语言提示会导致语言泄露。

5. 推理参与

如果 R1 跳过了隐藏推理阶段，请为助手预填充 \n 或指示：Think inside…</think>；在 </think> 之后给出最终答案。在向最终用户展示结果前，请剥离或隐藏推理过程。

6. 上下文纪律

非常长的上下文（约 100k+ tokens）会降低保真度。仅总结或检索必要内容，并将关键指令放在开头和结尾，而不是埋没在中间。

7. 如果这些检查无法稳定输出，则简化任务

重申目标，收紧模式，然后才考虑集成（例如，多数投票）。

核心调试原则

调试 DeepSeek 模型时要记住的一个核心原则是其对初始条件的高度敏感性。与其他一些模型不同，提示措辞的微小变化或超参数（如 temperature）的小幅调整（例如 0.1）会显著改变模型的输出轨迹。因此，迭代、小规模的更改应是解决意外行为的主要方法。

诊断技巧

一个关键的诊断技术是维护一个已知良好的基准提示。当遇到新的或复杂提示的问题时，运行此基准可以确定意外行为是源于提示本身，还是模型或推理环境的潜在临时问题。此实践是隔离错误源的基础步骤，类似于模型行为的单元测试。

迁移指南

有了稳定的调试循环和已知良好的基准，是时候从救火转向预防：转化你已有的东西。下一节将 GPT-5/Claude 的常见模式映射为 DeepSeek 友好的习惯用法——保留什么、更改什么、以及丢弃什么。

提示词迁移指南

将您的提示词从 OpenAI 的 GPT-4o、Claude 或其他模型迁移到 DeepSeek？以下是一些关于如何迁移提示词的建议和想法。这些旨在作为启发性的建议，而非硬性规定。

假设您有来自其他模型的系统提示：

messages = [
    {"role": "system", "content": "You are a helpful Python expert..."},
    {"role": "user", "content": "Explain decorators"}
]

那么，对于 DeepSeek R 系列，相同的提示可以如下编写：

messages = [
    {"role": "user",
    "content": "You are a helpful Python expert. Explain decorators."}
]

对于 DeepSeek V 系列，您需要指定最小化提示：
```
# V-series accepts system prompts, but keep it minimal
messages = [
    {"role": "system", "content": "Python expert assistant"},
    {"role": "user", "content": "Explain decorators"}
]
```
迁移说明：为 R 系列合并系统指令到用户提示中。对于 V 系列，大幅简化——思考姓名牌，而不是个人简介。

少样本示例迁移

考虑来自其他模型的少样本示例：

prompt = """
Example 1: Input: "happy" → Output: "positive"
Example 2: Input: "sad" → Output: "negative"
Example 3: Input: "angry" → Output: "negative"
Now classify: "excited"
"""

对于 DeepSeek（两个系列），可以如下指定：
```
prompt = """
Task: Classify the sentiment as 'positive' or 'negative'.
Expected format: Return only the classification word.
Classify: "excited"
"""
```
迁移说明：用清晰的任务描述和格式规范替换示例。您的准确率实际上可能会提高——DeepSeek 的强化学习训练意味着它通常比您的示例更了解。

高温设置迁移

如果您为创造性设置了较高的温度，在其他模型中的设置可能如下：

# OpenAI's GPT-4o creative writing
temperature = 0.9
top_p = 0.95

对于 DeepSeek V 和 R 系列，您可以如下指定这些设置：

# V-series creative tasks
temperature = 0.5  # Max recommended
top_p = 0.9
# R-series creative tasks (not recommended, but if you must)
temperature = 0.6
# Add explicit creativity instruction instead
prompt += "\nBe creative and explore unconventional ideas."

迁移说明：DeepSeek 在高温度下比其他模型更快地变得不稳定。用明确的创造力指令来补偿，而不是依赖随机性。

长提示迁移

如果您的提示非常长（>50k tokens），可以如下放置：

# Just dump everything
context = load_100k_token_document()
prompt = f"Analyze this document:\n{context}\n\nQuestion: {query}"

但对于 DeepSeek，您需要智能地放置提示：

# Structure and position strategically
prompt = f"""
<context>
{context[:30000]}  # Less critical info
</context>
<critical_context>
{context[30000:]}  # Most important parts
</critical_context>
<task>
{query}
</task>
"""
# Don't forget to set max length for V-series!
model_kwargs = {"max_model_len": 65536}

迁移说明：DeepSeek 的注意力机制（尤其是 V 系列 MLA）更重视后面的 token。将关键信息放在末尾附近。

CoT 提示迁移

如果您使用思维链提示，其他模型的提示可能采用以下格式：

prompt = "Let's approach this step-by-step:\n" + problem

对于 DeepSeek R 系列，对于 R1 系列您可能只需陈述问题：

# Just state the problem
prompt = problem
# R1 already thinks step-by-step internally

但对于 DeepSeek V 系列，您可能需要指定它需要遵循 CoT 方法：
```
# V-series benefits from CoT prompting
prompt = "Think through this systematically:\n" + problem
```
迁移说明：R 系列内置了 CoT；添加它就像戴两副眼镜。V 系列仍然受益于明确的推理请求。

角色扮演迁移

如果您有复杂的角色扮演，为其他模型指定角色：

system = """You are Shakespeare, the famous playwright.
You speak in iambic pentameter, use Elizabethan English,
make references to your plays..."""

对于 DeepSeek，您需要指定对响应的期望：

# Simplify to output requirements
prompt = """Write a response in Shakespearean style:
- Use iambic pentameter where possible
- Include elizabethan vocabulary
- Reference classic plays if relevant
User query: {query}"""

迁移说明：关注输出特性而非身份。DeepSeek 对“像莎士比亚一样写”的效果好于“你是莎士比亚”。

JSON 生成迁移

对于其他模型没有原生支持的 JSON 生成，您可以如下指定要求：

prompt = "Return a JSON object with name and age fields."

对于 DeepSeek（Pre-0528）模型，您需要指定提示、格式和一个示例来告诉它如何处理：

prompt = """Return ONLY a valid JSON object.
Format: {"name": "string", "age": number}
Example: {"name": "Alice", "age": 30}
Extract from: {text}"""

对于 DeepSeek（Post-0528），您需要在提示中指定格式并提供清晰的规范：

# Use native JSON mode
response_format = {"type": "json_object"}
# But STILL mention JSON in the prompt!
prompt = "Extract as JSON: {text}"

迁移注意事项：即使原生支持 JSON，DeepSeek 仍倾向于要求口头确认。

这就像一个安全机制，需要同时拥有钥匙和按钮才能启动。

迁移前，请先问自己以下问题：

我是否使用了少样本示例？→ 转换为任务描述。
我的温度设置是否大于 0.7？→ 降低它，并添加创意指令。
我是否有系统提示词？→ 将其合并到用户提示词中（R1 模型）或简化（V3 模型）。
我是否在提示词中要求思维链？→ 对于 R1 模型，应移除；对于 V3 模型，可保留。
我的提示词是否超过 5 万标记？→ 对其进行结构化，并将关键信息放在最后。
我是否使用了角色扮演？→ 转换为输出规格说明。

如有疑问，就简化提示词。DeepSeek 模型——尤其是 R 系列模型——就像专家顾问。清晰地向它们提出问题，然后就别再过多干预。你从其他模型那里学来的、精心设计的提示词策略，实际上可能会阻碍 DeepSeek 发挥作用。

请记住：迁移不只是简单的查找和替换。这是一个机会，让你重新思考所有那些复杂的提示词工程是否真的有必要。通常情况下，对于 DeepSeek 而言，是没必要的。

总结

DeepSeek 的提示词工程需要思维方式的转变：停止规定思考过程，而是清晰地陈述目标、明确约束条件、无歧义地定义输出模式——然后信任模型的内部推理能力。在其他地方有效的技巧，在这里可能会适得其反：少样本示例演示常常会降低准确性，而明确的任务陈述结合严格的模式定义则能持续获得更好的性能。

在实际操作中，将指令放在正确的位置，并遵循其固有的机制。R 系列模型基本上会忽略系统角色（system role），因此请将指导信息放在用户消息中；V 系列模型则可以接受一个高级别的系统提示词来设定全局约束。严格遵循每个系列原生的聊天模板（包括非标准的 Unicode 分隔符），并刻意调整采样参数：两个系列在温度（temperature）和顶级概率（top-p）上都有着相对较窄的良好区间。内化这些习惯，你就能从仅仅使用 DeepSeek，进阶到构建可靠、达到生产级别的系统——经过一些“去学习”后，你将获得一种显著更强大的自动化推理能力。

在下一章，我们将讨论一些 DeepSeek 可以应用的用例。

获取本书的 PDF 版本与独家附加内容

扫描二维码（或访问 packtpub.com/unlock）。按书名搜索本书，确认版本，然后按照页面上的步骤操作。 | |

| 注意：请保管好您的发票。直接从 Packt 购买无需提供发票。 |

第二部分：使用 DeepSeek

在本书的第二部分，我们将从理论转向实践，通过真实世界的案例研究，展示 DeepSeek 如何在各行各业推动创新。接下来，我们将亲手实践一个具体项目：构建一个由 DeepSeek 驱动的 API。你将跟随完整的开发流程——设计应用逻辑、在本地运行，并将其部署到亚马逊云科技（AWS）。在本部分的最后一章，我们将介绍智能体（agents），并演示如何构建由 DeepSeek 模型支持的强大智能体。

学完本部分后，你将掌握使用 DeepSeek 解决常见业务问题的能力，无论你是创建简单的网络应用程序还是构建功能完备的智能体系统。

本部分包含以下章节：

第 4 章，使用 DeepSeek：案例研究
第 5 章，使用 DeepSeek 进行构建
第 6 章，使用 DeepSeek 的智能体

保持关注

要跟进生成式 AI 和大型语言模型领域的最新动态，请订阅我们的周刊《AI_Distilled》，访问 packt.link/8Oz6Y。

加入我们在 Discord 和 Reddit 的社区

对本书有疑问，或想参与关于生成式 AI 和大型语言模型的讨论？

请加入我们的 Discord 服务器 packt.link/4Bbd9 和我们的 Reddit 频道 packt.link/wcYOQ，与志同道合的爱好者交流、分享和协作。

| | |

4 使用 DeepSeek：案例研究

到现在，你应该对 DeepSeek 的运作方式、内部机制及其在全球大型语言模型（LLM）格局中的角色有了相当的了解。在本章中，我们将通过实际应用案例，审视 DeepSeek 如何正在改变各个行业。微软、GitHub 以及财富 500 强公司等主要玩家，正采用 DeepSeek 等 LLM 来提高生产力、削减成本并驱动创新。通过我们构建生产就绪型应用的经验，我们亲眼见证了 DeepSeek 如何提升开发速度、改善代码质量，并赋能更优的战略决策。

DeepSeek 可以应用的一些著名用例包括：金融公司自动化交易策略、法律团队加速合同分析以及开发者对遗留系统进行现代化改造。在本章中，我们将重点讨论文档理解和文档提取的用例。

我们将深入研究这些综合性案例研究，以展示 DeepSeek 的实际应用，以及你如何运用结构化的方法来实施人工智能辅助的工作流程。我们选择Cursor IDE作为配置和使用 DeepSeek 的平台。不过，我们涵盖的技术适用于任何支持 DeepSeek 的平台，从传统 IDE 到云工具或本地界面。如果你还不知道从何处入手，我们为你准备了一份附录，简要介绍了使用 DeepSeek 的各种方式。

对于本章的案例研究，我们遵循系统化的方法：定义明确的问题陈述、设计有效的提示词、根据定性和定量指标评估响应，并进行迭代以达到生产级别的质量。

在本章中，我们将涵盖以下主要主题：

基准测试工具设置与提示词设计
用例研究：文档理解
用例研究：金融文档理解与基准测试

技术要求

在本章中，我们将使用Cursor IDE来演示 DeepSeek 的能力，这是一个与 DeepSeek 模型紧密集成的、由 AI 驱动的开发环境。你首先需要获取 DeepSeek 的 API 访问权限。请先在 platform.deepseek.com/ 创建一个账户并生成你的 API 密钥。获得凭证后，建议通过一个简单的测试调用来验证你的访问权限，确保一切配置正确：

curl -X POST "https://api.deepseek.com/v1/chat/completions" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
  "model": "deepseek-reasoner",
  "messages": [{"role": "user", "content": "Hello, DeepSeek!"}]
}'

确认 API 访问权限无误后，我们就可以着手建立开发环境，这将成为我们探索 DeepSeek 实际应用的实验室。

设置你的开发环境

要建立一个有效的 AI 辅助开发环境，我们将使用以下工具：

Python 3.11+，用于后端 AI 和数据处理任务。
Docker Desktop (docs.docker.com/desktop/)。
Git，用于版本控制 (git-scm.com/)。
Node.js 18+。
Cursor IDE，可从 cursor.com/ 下载。

在 Cursor IDE 中配置 DeepSeek

使用Cursor IDE配置 DeepSeek 是一个直接了当的过程。要将 DeepSeek 与Cursor集成，你需要按照以下步骤开始：

Cursor IDE DeepSeek 模型配置与项目设置

访问 Cursor 模型配置

通过 Settings（设置）菜单访问 Cursor 模型配置：

在 macOS 上，你会在 Cursor > Settings（设置）下找到该选项。
如果你是 Windows 或 Linux 用户，可以在 File（文件） > Settings（设置）下找到。

在 Settings（设置）界面中，左侧边栏的 Models（模型）部分包含了管理 AI 模型集成的所有选项。

启用与配置 DeepSeek 模型

Cursor IDE 内置了对 DeepSeek 模型的支持。你可以通过滚动浏览可用模型列表并选择 deepseek-r1-0528 来启用它。
启用后，该模型将出现在你的可用选项中，准备使用你的具体 API 凭据进行配置。
配置过程需要输入你的 DeepSeek API 凭据以及相应的连接设置。
你需要将模型名称指定为 deepseek-r1-0528，提供从 DeepSeek 平台获取的 API 密钥，将基础 URL 设置为 api.deepseek.com/v1，并将提供者配置为 OpenAI 兼容 以确保正确的 API 通信协议。
要验证集成是否正常工作，请在 Cursor 中打开一个新文件，并使用 Cmd + K（在 macOS 上）或 Ctrl + K（在 Windows 和 Linux 上）调用 AI 命令面板。
从模型下拉菜单中选择 DeepSeek-R1，并使用一个实质性的提示进行测试，例如 Explain the difference between microservices and monolithic architecture。成功的响应将确认你的开发环境已准备好用于我们即将探讨的高级用例。

有关在 Cursor 中使用 DeepSeek 的更多细节以及其他选项，请参考附录。

组织开发工作区

清晰的工作区能在项目变得复杂时让你保持高效和有序。本章中，请设置一个名为 Chapter04 的文件夹，并包含所需的子文件夹。例如，我们将文件夹命名为 brainstorming-ideation、document-analysis 和 code-generation，如下所示：

# Navigate to the DeepSeek-in-Practice directory
cd /Users/<user-id>/workplace/DeepSeek-in-Practice-main
# Create Chapter04 directory and subdirectories
mkdir Chapter04
cd Chapter04
mkdir 01-brainstorming-ideation
mkdir 02-document-analysis
mkdir 03-code-generation
# Initialize Git repository for version control
git init

每个子目录将包含其各自案例研究的完整产物，包括源代码、配置文件、文档和任何支持资源。

配置管理

适当的凭据管理是任何专业开发工作流的基础，尤其是在集成外部 AI 服务时。在你的 Chapter04 目录中创建一个专用的环境配置文件，以安全地存储敏感信息：

# Navigate to Chapter04 directory
cd /Users/<user-id>/workplace/DeepSeek-in-Practice-main/Chapter04
# Create .env file
cat > .env << EOF
# DeepSeek Configuration
DEEPSEEK_API_KEY=your_api_key_here
DEEPSEEK_BASE_URL=https://api.deepseek.com/v1
# Database Configuration (we'll use these later)
DATABASE_URL=sqlite:///./development.db
# API Configuration
API_HOST=localhost
API_PORT=8000
# Logging
LOG_LEVEL=INFO
EOF

这种方法将敏感凭据与源代码分离，实现了安全的共享和部署实践。请记住，环境文件不应提交到版本控制系统；创建后应立即将 .env 添加到你的 .gitignore 文件中，以防止敏感信息意外泄露。

所有基准测试脚本、测试文档和评估框架均可在我们的 GitHub 仓库中找到，供有兴趣复现结果或进行自身实验的读者使用：

github.com/PacktPublishing/DeepSeek-in-Practice/tree/main/Chapter04/document-understanding

该仓库包含以下内容：

用于准确性测量的基准真实数据集。
自动化基准比较工具。
所有三个工具的完整测试脚本：
- DeepSeek-R1：通过直接 API 调用进行基于 LLM 的语义提取 test_deepseek_parsing.py。
- Docling (IBM Research)：文档结构分析和表格识别 test_docling_parsing.py。
- MarkItDown (Microsoft)：针对 LLM 优化的文档预处理 test_markitdown_parsing.py。
详细的评估框架文档。

基准测试工具设置与提示设计

为了进行本章的定量评估和基准测试，我们将测试 DeepSeek-R1 与专业文档解析工具的对比，以提供全面的性能比较。我们创建了自动化测试脚本来测量不同方法在准确性、处理时间和成本指标上的表现。

以下是用于基准测试的工具：

DeepSeek-R1：通过直接 API 调用进行基于 LLM 的语义提取。
Docling (IBM Research)：文档结构分析和表格识别。
MarkItDown (Microsoft)：针对 LLM 优化的文档预处理。

要设置基准测试环境：

# Install benchmarking dependencies
pip install requests tabulate pandas
# Optional: Install comparison tools
pip install docling markitdown

在我们开始案例研究之前，我们将分享一些关于如何为案例研究设计提示的快速技巧：

提供丰富的商业背景：用清晰的细节来构建提示，包括商业环境、约束条件、愿景和目标。避免使用诸如 help us innovate（帮助我们创新）这样泛泛的查询。
关注可执行的结果：构建提示以引出可行的、战略性的输出，而非抽象的想法。
利用 DeepSeek-R1 的推理优势：使用需要系统性分析的结构化提示。
提供清晰详细的信息：包括行业数据、竞争洞察和明确的成功指标。呈现具有明确参数和时间线约束的具体挑战。
鼓励权衡评估：DeepSeek-R1 在处理涉及多个变量、风险和战略权衡的问题时表现最佳——非常适合生成全面的创新战略。

本章中所有公司名称、场景和代码示例均为虚构，专为教育目的而创建。与任何真实公司、系统或专有代码的相似之处纯属巧合。遗留代码示例基于遗留系统中的常见模式，但不代表任何实际的生产系统。所有代码均为为本书创建的原创内容，并遵循开源原则供教育使用。

内容来源说明：本章包含来自我们 DeepSeek-R1 交互的实际提示和响应，并通过格式指示器清晰标记。任何潜在的后续或增强提示建议都是迭代提示可能如何工作的说明性示例，并非实际执行的提示。所有真实的 DeepSeek-R1 交互均已明确标记。

现在，让我们从文档理解的案例研究开始。

用例研究：文档理解

CloudTech Solutions，一家不断成长的企业软件公司，正面临实施现代无服务器架构的重大挑战。其开发团队的任务是从传统的单体 API 服务器迁移到 Amazon API Gateway 和 AWS Lambda 函数，以支持跨 15+地理区域的 10,000+每秒请求，同时维持 99.99%的正常运行时间。

该公司当前的情况反映了一个常见的企业挑战：他们拥有全面的技术文档（一份 45 页的 Amazon API Gateway 集成指南），但需要从复杂的技术规范中快速提取可执行的实施策略。开发团队必须分析该文档，创建一个结构化的实施计划，涵盖架构设计、安全要求、性能优化和操作流程。

业务影响是重大的。CloudTech Solutions 必须在紧张的时间表内完成这次迁移，以满足不断增长的客户需求，并将基础设施成本降低 40%。挑战不仅在于理解文档——他们需要将高级架构概念转化为具体的技术决策、实施优先级和开发团队能立即执行的生产就绪代码。

我们对这个文档分析挑战的探索，展示了 DeepSeek-R1 如何处理复杂的技术文档并提供结构化的实施指导。此用例展示了模型从综合企业文档中提取关键架构组件、优先排序实施任务、识别关键配置决策以及生成可操作技术要求的能力。

提示 DeepSeek

对于技术文档分析任务，DeepSeek-R1 需要清晰定义分析目标和预期交付物的结构化提示。关键是提供关于文档类型、需提取的特定技术要求以及可操作输出格式的全面上下文。与一般的文档审查不同，技术分析提示应侧重于可实施的洞见和结构化的决策框架。

处理企业技术文档时，指定架构考量、实施优先级、安全要求和操作约束至关重要。DeepSeek-R1 的推理能力在被要求不仅提取信息，还将复杂的技术概念综合成开发团队可立即执行的可操作实施策略时表现卓越。

响应评估

让我用 CloudTech Solutions 的 Amazon API Gateway 文档分析挑战来演示，使用我们案例研究中的实际提示和响应。

Note

以下分析基于一份全面的 45 页 Amazon API Gateway 集成指南（来源：Chapter04/document-understanding/amazon_api_gateway_integration_guide.md）

这是我们要发送给 DeepSeek-R1 进行技术文档分析的提示：

I need you to analyze this Amazon API Gateway Integration Guide and provide a comprehensive implementation strategy. This is enterprise-grade documentation that our development team needs to implement for a production system handling 10,000+ requests per second.
Please provide:
1\. **Architecture Summary**: Extract the key components and their relationships from the document. What are the core services and how do they interact?
2\. **Implementation Priority Matrix**: Based on the document, what should be implemented first, second, third? Consider dependencies, risk, and business value.
3\. **Critical Configuration Analysis**: Identify the most important configuration decisions that could impact security, performance, or cost. Extract specific values and settings.
4\. **Code Generation Plan**: Which Lambda functions, API configurations, and infrastructure components need to be created? Provide a structured implementation plan.
5\. **Security & Compliance Requirements**: Extract all security requirements, authentication flows, and compliance considerations. How should these be implemented?
6\. **Production Readiness Assessment**: What monitoring, error handling, and operational procedures are specified? Create a production deployment checklist.
Focus on extracting actionable technical requirements that a development team can immediately implement. Convert high-level concepts into specific technical decisions.
[Attached: amazon_api_gateway_integration_guide.md - 45 pages of technical documentation]

DeepSeek 对此提示的响应如下：

Based on my comprehensive analysis of the 45-page Amazon API Gateway Integration Guide, here's the structured implementation strategy for your production system:
1\. Architecture Summary
The system implements a serverless architecture using Amazon API Gateway and Lambda functions with the following core components:
Core Services: - API Gateway: Fronts the entire system, handles routing, authentication, throttling, and request/response transformation - Lambda Functions: Serve as the backend for different services (User, Order, Payment, etc.) - Data Stores: DynamoDB (Users, Payments), RDS PostgreSQL (Orders), SQS (async processing), S3 (analytics) - Authentication: AWS Cognito for OAuth 2.0/OIDC, with multi-factor authentication - Caching: Redis (via ElastiCache) and CloudFront for CDN caching - Monitoring: CloudWatch, X-Ray, and custom metrics - Security: AWS WAF, security headers, and encryption at rest and in transit
System Interaction Flow:
Clients → CloudFront → AWS WAF → API Gateway → Lambda Authorizer (Cognito)
→ Lambda Functions (User, Order, Payment) → Databases (DynamoDB, RDS, etc.)
Async flows: API Gateway → SQS → Async Lambda Processor
Monitoring: API Gateway and Lambda → CloudWatch logs and X-Ray traces
2\. Implementation Priority Matrix

| 优先级 | 组件 | 原因 |

| --- | --- | --- |

| 1 | 核心基础设施 | 设置 VPC、IAM 角色、安全组和基本网络 |

| 2 | 身份验证与授权 | 无身份验证，则无安全访问。Cognito 设置和 Lambda 授权器 |

| 3 | 用户服务 | 其他服务（如订单）所需的基本用户管理 |

| 4 | 订单服务 | 核心业务功能 |

| 5 | 支付服务 | 对交易至关重要，但依赖于用户和订单服务 |

| 6 | 异步处理 | 用于非关键的后台任务 |

| 7 | 监控与日志 | 生产环境必需，但可同时构建 |

| 8 | 缓存与优化 | 在核心功能之后进行的性能改进 |

关键配置分析

认证配置：

Cognito User Pool：要求 MFA，密码策略（12+ 字符，混合大小写、数字、符号）
JWT Token：必须包含自定义声明（tenant_id、role）以实现多租户
Lambda Authorizer：必须验证 JWT 并实施基于租户的访问控制和基于角色的访问控制

API 网关配置：

新服务使用 HTTP API（经济高效），遗留系统使用 REST API 以保持兼容性
阶段：开发环境（限流：突发 100，速率 50）和生产环境（突发 5000，速率 2000）
自定义域名：api.example.com，使用 TLS 1.2+ 和 ACM 证书
请求/响应：对输入使用验证模式，使用标准化的响应模板

安全配置：

WAF 规则：SQL 注入、XSS、速率限制（每 IP 每分钟 100 次请求）、地理封锁
安全头：Strict-Transport-Security、X-Content-Type-Options、X-Frame-Options
加密：DynamoDB、RDS 和 S3 使用 KMS 加密；传输过程使用 TLS 1.2+

速率限制：

全局：生产环境中突发 10,000 RPS
按客户端：使用计划（免费、高级、企业版），配有限制层级

代码生成计划与实施策略

Lambda 函数（每个作为独立服务）：

用户服务：CRUD 操作，与 DynamoDB 集成的用户资料管理
订单服务：使用 RDS PostgreSQL 后端的订单生命周期管理
支付服务：支付处理、退款，采用安全的令牌化技术
异步处理器：处理 SQS 消息，用于后台任务和通知

基础设施即代码（CloudFormation）：

API Gateway（HTTP API 和 REST API 配置）
Lambda 函数，包含环境变量、VPC 配置和安全层
DynamoDB 表，使用 KMS 加密和备份策略
RDS 实例（PostgreSQL），支持备份、多可用区部署和只读副本
SQS 队列（主队列和死信队列）
Cognito User Pool 和客户端，配置 MFA
WAF 规则及其与 API Gateway 的关联
CloudWatch 告警、仪表盘和 X-Ray 配置

安全与合规实施

认证流程：

客户端请求 → API Gateway → Lambda Authorizer → Cognito 验证
→ JWT 声明提取 → 基于角色的访问控制 → 服务授权

合规要求：

SOC2 Type II：自动化安全控制和审计日志
PCI DSS：安全支付处理与令牌化
GDPR：数据驻留控制和用户同意管理
多租户：在数据库和应用层实现租户隔离

生产就绪清单

监控与可观测性：

CloudWatch 监控 API Gateway、Lambda 和数据库性能的指标
X-Ray 分布式追踪，用于请求流分析
结构化日志与关联 ID，便于调试
用于 SLA 监控的自定义业务指标

错误处理与弹性：

外部服务调用的断路器模式
针对瞬时故障的指数退避和重试逻辑
用于失败消息处理的死信队列
非关键服务故障的优雅降级

操作流程：

使用 Lambda 别名和流量权重的蓝/绿部署
基于错误率阈值的自动化回滚触发器
基于请求模式和增长预测的容量规划
设定 RTO/RPO 目标的灾难恢复流程

DeepSeek-R1 将一份长达 45 页的 Amazon API Gateway 规范文档转化为结构化的实施指南，展现了其复杂的技术分析能力，使开发团队能够高效执行复杂的企业项目。

技术分析的评估方法

对于技术文档分析，评估重点在于实施准确性和生产就绪性：

定性指标：强调对架构的理解、实施可行性以及对系统开发至关重要的生产就绪性考量。
定量指标：衡量技术规范的完整性、配置细节的准确性以及实施产出物的覆盖范围。

为了客观、可重复地评估 DeepSeek-R1 的文档分析能力，我们开发了一个系统化的评估框架，同时衡量定性理解和定量完整性。本节详细说明我们的评分方法论，展示我们如何为 Amazon API Gateway 案例研究分配分数并进行分析。

具体来说：

我们的评估过程始于基准事实的建立。我们创建了一份跨 45 页、1,743 行的 Amazon API Gateway 集成指南作为源文档，其中包含特定的架构组件、配置值和实施要求。该文档作为权威参考，我们据此衡量 DeepSeek-R1 的分析准确性。
在评估 DeepSeek-R1 的响应之前，我们手动编目了一份完整分析应识别的所有关键要素清单。该清单包括八个主要的无服务器组件，即 API Gateway；用于用户、订单、支付和异步处理服务的 Lambda 函数；DynamoDB；RDS PostgreSQL；Amazon Cognito；带 X-Ray 的 CloudWatch（用于监控）；SQS（用于消息队列）；以及 AWS WAF（用于安全）。
我们还确定了 23 个具体的配置参数，包括：生产环境突发速率 5,000 和开发环境 100 的限流限制；对应的速率限制分别为 2,000 和 50；MFA 要求；密码策略规则要求 12 个或更多字符，包含混合大小写、数字和符号；TLS 版本要求 1.2 或更高；每 IP 地址每分钟 100 次请求的速率限制；以及自定义域名设置。
此外，我们编目了四项合规要求，涵盖 SOC2 Type II、PCI DSS、GDPR 和多租户隔离；以及六项实施交付物，包括 CloudFormation 模板、Lambda 函数、IAM 策略、监控仪表盘、安全配置和部署流程。
在建立基准事实后，我们通过逐项对比 DeepSeek-R1 的响应与我们的参考要素进行手动比较和评分。对于每个评估标准，我们计算正确识别、理解和解释的预期要素数量。

我们的定性指标采用 1 到 10 的评分量表，具体如下：

9–10 分表示表现卓越，具备近乎完整的理解，正确识别并解释了 90–100% 的预期要素，并提供了恰当的上下文。
7–8 分表示表现良好，理解透彻，覆盖了 70–89% 的要素，仅在深度或具体性方面有轻微不足。
5–6 分表示表现可接受，理解充分，覆盖率为 50–69%，但遗漏了重要细节。
3–4 分表示表现低于平均水平，表现出部分理解，覆盖率为 30–49%，存在显著缺口。
1–2 分表示表现不佳，理解甚微，覆盖的预期要素少于 30%。

为了具体展示我们的评分过程，我们将审视如何评估获得了 9/10 分的架构理解部分：

DeepSeek-R1 正确识别了所有八个主要的无服务器组件，包括 API Gateway、Lambda 函数、DynamoDB、RDS、Cognito、带 X-Ray 的 CloudWatch、WAF 和 SQS。
模型准确描述了系统交互流程：客户端通过 CloudFront 连接，经过 WAF 安全过滤，到达 API Gateway，通过 Lambda authorizer 进行身份验证，最后调用与数据库交互的相应 Lambda 函数。
DeepSeek-R1 正确解释了异步处理路径，其中 API Gateway 将消息发送到 SQS，后者触发异步 Lambda 处理器，并正确识别了监控流程，其中服务向 CloudWatch 日志和 X-Ray 跟踪发送数据。

我们通过多个维度衡量了文档理解能力：

关于组件关系，DeepSeek-R1 展现出清晰的理解：API 网关位于整个系统前端，并根据服务类型（无论是用户、订单还是支付服务）智能地将请求路由到相应的 Lambda 函数。

关于集成模式，该模型正确识别出，在多租户架构中，Cognito 处理初始认证，随后 Lambda 验证器验证 JWT 以强制实施租户隔离。

对于数据流逻辑，DeepSeek-R1 理解了其架构决策：不同服务使用不同的数据存储。DynamoDB 因适合键值访问模式而服务于用户和支付服务；而 RDS PostgreSQL 则服务于订单服务，因其需要关系型数据结构和复杂查询。其得分并非满分 10 分，原因在于一个细微的疏漏：模型未明确提及源材料中记载的 ElastiCache/Redis 层，尽管其缓存策略在响应的另一部分中有所讨论。

以下是对 DeepSeek 响应的定性分析。

| 评价维度 | 分析 | 得分 (1–10) |

| --- | --- | --- |

| 架构理解 | 正确识别了所有主要无服务器组件、其关系及交互模式。清晰展示了对 API 网关、Lambda 及配套服务集成的理解。 | 9 |

| 实现可行性 | 提供了具体的配置值、安全设置和部署参数，开发团队无需额外研究即可直接实施。 | 9 |

| 安全意识 | 全面涵盖了认证流程、合规性要求（SOC2、PCI DSS）及带有具体实施指导的安全最佳实践。 | 8 |

| 生产就绪度 | 解决了运维方面的关切，包括监控、错误处理和灾难恢复流程，这些在初期实施中常被忽视。 | 8 |

| 优先级排序 | 实施顺序逻辑清晰，考虑了依赖关系、风险和业务价值，并为每个阶段提供了明确理由。 | 8 |

表 4.1：DeepSeek 响应的定性分析

对于定量评估，我们采用客观计数和基于百分比的评分方法，使用一个简单的公式：得分等于正确识别的元素数量除以预期元素总数，乘以 10，然后四舍五入到最接近的整数。

例如，正确识别所有 23 个参数可得满分 10 分，而识别出 23 个中的 20 个则得 9 分，代表 87%的准确率。

我们通过配置参数评估展示了这种定量方法，该评估得分为 9/10。我们在源文档中识别出 23 个具体的配置参数，并系统性地检查 DeepSeek-R1 的响应中是否包含了每一个。

该模型成功识别了以下内容：

开发环境限流突发限制为 100，生产环境为 5,000。
相应的限流速率限制分别为 50 和 2,000。
MFA 要求设置为ON，密码最小长度为 12 个或更多字符，以及包含大小写字母、数字和符号的密码复杂度要求。
用于tenant_id和role的 JWT 自定义声明、TLS 版本要求为 1.2 或更高、WAF 速率限制为每 IP 地址每分钟 100 个请求，以及生产环境全局 RPS 限制为 10,000 突发容量等。

总的来说，DeepSeek-R1 正确识别了 23 个预期参数中的 21 个。缺失的 2 个参数是 Lambda 函数的超时值（最大为 30 秒）和 CloudFront 缓存层的 TTL 设置。21/23 的正确识别率得出 91.3%的准确率，四舍五入得分为 9/10。

这种详细的评分方法确保了我们的评估是透明、可重复且客观的。在本章后续的评估中，我们将采用相同的框架，但提供汇总结果而不再次解释该方法。

将得分列中的分数求和，计算得出定性分数为 42/50（84%）。

该响应展现了强大的技术架构理解力，并提供了可直接执行的实施指导。在理解力和可行性方面的高分反映了分析质量，支持开发团队直接执行。

定量分析如下表所示：

| 评价维度 | 计数/指标 | 分析 | 得分 (1–10) |

| --- | --- | --- | --- |

表 4.2：DeepSeek 响应的定量分析

定量得分计算为 45/50（90%），由此可以推断 DeepSeek 展现了以下能力：

卓越的技术规格覆盖，提供全面的实施细节。
在所有技术交付类别中表现强劲，展现了企业级分析深度。

综合得分（定性与定量）计算为 87/100（87%）。

DeepSeek-R1 提供了企业级的技术分析，将复杂的文档转化为可直接执行的实施路线图。该响应成功平衡了架构理解与实用实施指导，为开发团队提供了可立即执行的技术规格。

DeepSeek 还展现了实施效率。传统分析可能需要 36 小时的专业知识（$6,000–$9,000），而 DeepSeek 在几分钟内完成，实现了 99%以上的时间缩减，同时保持了专业技术标准。

后续代码生成请求

收到全面分析后，CloudTech Solutions 请求提供实际的实现代码。以下是后续提示词及 DeepSeek-R1 的响应：

为分析中排名前 3 的优先组件生成生产就绪的实现代码：

DeepSeek 生成了以下响应：

根据您的分析，为排名前 3 的优先级组件生成生产就绪代码：

核心基础设施（CloudFormation 模板）：
- VPC、安全组、IAM 角色
- 具有适当节流和安全性的 API Gateway
- 启用 MFA 和自定义属性的 Cognito 用户池
用户服务 Lambda 函数：
- 使用 DynamoDB 的完整 CRUD 操作
- 适当的错误处理和日志记录
- 支持多租户并保持租户隔离
Lambda 授权方：
- 使用 Cognito 进行 JWT 验证
- 提取自定义声明（tenant_id, role）
- 强制执行租户隔离
- 返回适当的 IAM 策略

每个组件需提供：

完整、可部署的代码
基于文档的配置参数
安全最佳实践实现
错误处理和弹性模式
监控和日志记录集成

重点关注遵循文档分析中安全、性能和合规性要求的生产就绪代码。

Based on your analysis, generate production-ready code for the top 3 priority components:
1. **Core Infrastructure (CloudFormation Template)**:
   - VPC, security groups, IAM roles
   - API Gateway with proper throttling and security
   - Cognito User Pool with MFA and custom attributes
2. **User Service Lambda Function**:
   - Full CRUD operations with DynamoDB
   - Proper error handling and logging
   - Multi-tenant support with tenant isolation
3. **Lambda Authorizer**:
   - JWT validation with Cognito
   - Custom claims extraction (tenant_id, role)
   - Enforces tenant isolation
   - Returns proper IAM policies
For each component, provide:
- Complete, deployable code
- Configuration parameters based on the documentation
- Security best practices implementation
- Error handling and resilience patterns
- Monitoring and logging integration
Focus on production-ready code that follows the security, performance, and compliance requirements from the documentation analysis.

让我们看看 DeepSeek-R1 的代码实现响应。

DeepSeek-R1 提供了三个完整的、生产就绪的实现：

核心基础设施（CloudFormation 模板）

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Core infrastructure for API Gateway implementation
Parameters:
  Environment:
    Type: String
    Default: dev
    AllowedValues: [dev, staging, prod]
  SystemName:
    Type: String
    Default: enterprise-api-system
Resources:
  # API Gateway with security and throttling
  ApiGateway:
    Type: AWS::Serverless::HttpApi
    Properties:
      StageName: !Ref Environment
      DefaultRouteSettings:
        ThrottlingBurstLimit: !If [IsProd, 5000, 100]
        ThrottlingRateLimit: !If [IsProd, 2000, 50]
      Auth:
        Authorizers:
          CognitoAuthorizer:
            JwtConfiguration:
              issuer: !Sub "https://cognito-idp.${AWS::Region}.amazonaws.com/${UserPool}"
              audience: [!Ref UserPoolClient]
  # Cognito User Pool with security settings
  UserPool:
    Type: AWS::Cognito::UserPool
    Properties:
      UserPoolName: !Sub "${SystemName}-user-pool"
      Policies:
        PasswordPolicy:
          MinimumLength: 12
          RequireLowercase: true
          RequireUppercase: true
          RequireNumbers: true
          RequireSymbols: true
      MfaConfiguration: "ON"
      Schema:
        - Name: custom:tenant_id
          AttributeDataType: String
          Required: true
          Mutable: false
        - Name: custom:role
          AttributeDataType: String
          Required: true
          Mutable: true

用户服务 Lambda 函数

import json
import boto3
import logging
from datetime import datetime
from aws_xray_sdk.core import xray_recorder
from botocore.exceptions import ClientError

# Initialize AWS services
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Users')

# Configure structured logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

@xray_recorder.capture('lambda_handler')
def lambda_handler(event, context):
    """
    User Service Lambda - handles CRUD operations with multi-tenant support
    """
    try:
        # Extract tenant context from authorizer
        tenant_id = event['requestContext']['authorizer']['tenant_id']
        user_role = event['requestContext']['authorizer']['role']

        # Route based on HTTP method
        http_method = event['httpMethod']

        if http_method == 'GET':
            return get_user(event, tenant_id)
        elif http_method == 'POST':
            return create_user(event, tenant_id)
        elif http_method == 'PUT':
            return update_user(event, tenant_id, user_role)
        elif http_method == 'DELETE':
            return delete_user(event, tenant_id, user_role)
        else:
            return error_response(405, "Method not allowed")

    except Exception as e:
        logger.error(f"Unhandled error: {str(e)}")
        return error_response(500, "Internal server error")

@xray_recorder.capture('get_user')
def get_user(event, tenant_id):
    """Get user with tenant isolation"""
    try:
        user_id = event['pathParameters']['user_id']

        response = table.get_item(
            Key={
                'user_id': user_id,
                'tenant_id': tenant_id  # Ensures tenant isolation
            }
        )

        if 'Item' not in response:
            return error_response(404, "User not found")

        return success_response(response['Item'])

    except ClientError as e:
        logger.error(f"DynamoDB error: {e.response['Error']['Message']}")
        return error_response(500, "Database error")

def success_response(data):
    return {
        'statusCode': 200,
        'headers': {'Content-Type': 'application/json'},
        'body': json.dumps(data, default=str)
    }

Lambda 授权方

import json
import os
import jwt
from jwt import PyJWKClient
from jwt.exceptions import ExpiredSignatureError, InvalidTokenError

# Configuration
USER_POOL_ID = os.environ['USER_POOL_ID']
REGION = os.environ['AWS_REGION']
JWKS_URL = f"https://cognito-idp.{REGION}.amazonaws.com/{USER_POOL_ID}/.well-known/jwks.json"

# Initialize JWKS client
jwks_client = PyJWKClient(JWKS_URL)

def lambda_handler(event, context):
    """
    Lambda Authorizer with JWT validation and tenant isolation
    """
    try:
        # Extract token from Authorization header
        token = extract_token(event)
        if not token:
            raise Exception("No token provided")

        # Validate JWT token
        decoded_token = validate_jwt_token(token)

        # Extract custom claims
        tenant_id = decoded_token.get('custom:tenant_id')
        role = decoded_token.get('custom:role', 'user')

        if not tenant_id:
            raise Exception("Missing tenant_id in token")

        # Generate IAM policy
        policy = generate_policy(
            decoded_token['sub'],
            'Allow',
            event['methodArn'],
            tenant_id,
            role
        )

        return policy

    except Exception as e:
        print(f"Authorization failed: {str(e)}")
        raise Exception('Unauthorized')

def validate_jwt_token(token):
    """Validate JWT token with Cognito JWKS"""
    try:
        # Get signing key
        signing_key = jwks_client.get_signing_key_from_jwt(token)

        # Decode and validate token
        decoded_token = jwt.decode(
            token,
            signing_key.key,
            algorithms=["RS256"],
            options={"verify_exp": True}
        )

        return decoded_token

    except ExpiredSignatureError:
        raise Exception("Token has expired")
    except InvalidTokenError:
        raise Exception("Invalid token")

后续响应评估

DeepSeek-R1 的响应展示了与高级解决方案架构师相当的卓越技术文档分析能力。该分析将一份复杂的 45 页技术指南转化为可立即执行的实施策略，并附有具体的代码示例、配置详情和生产部署流程。

评估方法与指标选择

用例研究：财务文档分析与基准测试

在技术文档分析方面，我们根据反映企业开发需求的标准来评估响应：

定性指标：关注技术准确性、实施清晰度和架构洞察质量，这些是开发团队进行生产部署所需的。
定量指标：衡量响应的完整性、技术细节的明确性以及结构组织，这些因素有助于立即进行开发执行。

让我们从定性分析开始（表 4.3）：

| 标准 | 分析 | 评分 (1–10) |

| --- | --- | --- |

| 技术准确性 | 响应对AWS架构模式展示了深刻理解，提供了生产级配置和安全实施。正确识别了服务关系、数据流和集成点。 | 9 |

| 实施清晰度 | 提供了具体的代码示例、配置参数和部署流程。技术建议包括精确的设置（速率限制 (100 突发, 50 速率)、超时值和安全头），开发团队可立即使用。 | 9 |

| 架构洞察 | 提供了全面的系统设计指导，包括微服务边界、优先级矩阵和依赖分析。兼顾了企业部署的技术和业务考量。 | 8 |

| 安全深度 | 识别了关键的安全要求，包括WAF规则、加密标准和合规性考量。提供了认证流程和访问控制的具体实施指导。 | 9 |

| 生产就绪度 | 涵盖了监控、错误处理、部署策略和操作流程。包含具体的指标、告警配置和灾难恢复考量。 | 8 |

表 4.3：后续响应的定性分析

计算出的定性评分为 43/50 (86%)。

其技术深度匹配资深解决方案架构师的能力，全面覆盖了企业部署需求，在高级监控配置和地区合规性细节方面有少量不足。

接下来看看定量分析：

| 标准 | 计数/指标 | 分析 | 评分 (1–10) |

| --- | --- | --- | --- |

表 4.4：后续响应的定量分析

计算出的定量评分为 45/50 (90%)。

观察到异常出色的技术细节和生产就绪的实现。响应提供了全面的覆盖，能够实现即时的开发执行。

以下是我们的整体评估：

综合评分为 88/100 (88%)。
DeepSeek-R1 交付了解决方案架构师级别的技术分析，并具备即时实施能力。响应质量堪比昂贵的技术咨询，同时提供了开发团队可以立即执行的具体代码示例和部署流程。
传统的解决方案架构咨询需要 3-4 周时间和 $30-50K 的投入，而 DeepSeek-R1 在几分钟内就达到了同等的技术深度，时间减少了 99% 以上，同时保持了企业级的质量。

但总有改进和挖掘更多知识的空间。以下是我们如何调整提示词以增强文档理解。

通过迭代提示进行重新校准

初步分析提供了全面的技术指导，但也揭示了在增强区域部署细节和高级监控配置方面的机会。

以下领域需要改进：

区域合规性细节（差距评分：7/10）：我们需要 EU/APAC 部署的具体监管要求，而目前 DeepSeek 的响应中未提供。
高级监控（差距评分：7/10）：需要详细的可观测性模式和自定义指标实现，以进行高级监控。
成本优化（差距评分：6/10）：大规模降低 API Gateway 和 Lambda 成本的具体策略对于此解决方案架构至关重要。

您可以利用以下迭代提示示例来弥补这些差距：

后续提示 1 - 区域部署：

根据您的 `AWS` 架构分析，为 `EU`（`GDPR` 合规）和 `APAC` 市场创建特定区域的部署配置。包括：
- 具体的数据驻留要求和实现
- 需要的区域合规认证
- 跨区域灾难恢复程序
- 区域成本优化策略
为每个区域提供配置示例。

后续提示 2 - 高级监控：

扩展 `Amazon API Gateway` 系统的监控策略，包括：
- 用于业务 KPI 的自定义 `CloudWatch` 指标
- 微服务的高级 `X-Ray` 追踪配置
- 自动化异常检测和告警程序
- 基于监控洞察的性能优化
包含具体的实现代码和配置示例。

后续提示 3 - 成本优化：

为 `Amazon API Gateway` 和 `Lambda` 架构设计全面的成本优化策略，在保持性能和可靠性的同时，将每月运营费用降低 `40%`。包括：

- `API Gateway` 成本降低策略（`HTTP API` 与 `REST API` 选择、请求优化、缓存）

- `Lambda` 函数成本优化（内存配置分析、执行时长缩减、预置并发与按需）

- 数据传输成本最小化（`CloudFront` 缓存、响应负载优化、压缩策略）

- 请求批处理和连接池实现

- 针对可预测工作负载的预留容量分析和购买建议

- 预算阈值设置与成本监控告警
为每项优化策略提供具体的配置示例、成本计算和预计的每月节省金额。

练习

将评估框架应用于分析迭代提示的响应，重点关注增强的细节如何改善企业环境的实施准备度。

文档分析的一个有趣应用领域是金融。在下一个案例研究中，我们将提示 DeepSeek 分析一份财务文档，并将其性能与基准工具进行比较。

为全面评估 DeepSeek-R1 的文档理解能力，我们设计了一项严谨的基准测试研究，对比了三种不同的技术方法：

DeepSeek-R1：运用高级推理进行直接语义提取。
Docling (IBM Research)：专注于文档结构与表格解析。
MarkItDown (Microsoft)：针对大语言模型优化的文档预处理。

测试文档创建与基准测试设置

为了严格评估文档理解能力，我们创建了一个全面的测试数据集，旨在挑战文档解析的方方面面。我们虚构的 FruitStand Innovation Inc. 公司季度财务报告共计 192 行，包含了真实财务报表中常见的复杂结构。该文档包含六个主要表格，旨在测试不同的解析挑战：一个多列合并资产负债表，对比三个季度（2024 年 Q3、2024 年 Q2 和 2023 年 Q3）；一份详细的利润表，分列产品收入和服务收入；一份地域收入分析，覆盖全球四个区域（北美、EMEA、亚太和拉美）；四个业务线（云平台、移动解决方案、企业服务和开发者工具）的产品性能指标；八个针对季度目标进行跟踪的关键绩效指标（KPI）；以及预测 2024 年 Q4 表现的前瞻性指引范围。

资产负债表展示了我们希望测试的解析复杂性。每个财务科目都跨越多个季度并附有相应的百分比变化，要求工具在提取精确数值的同时，保持正确的行-列对应关系。以下是我们测试文档中“流动资产”部分的摘录：

| **Assets** | **Q3 2024** | **Q2 2024** | **Q3 2023** | **Change (%)** |
|-----------|-------------|-------------|-------------|----------------|
| **Current Assets** | | | | |
| Cash and cash equivalents | $234,678 \| $198,543 | $187,432 | +25.2% |
| Short-term investments | $89,234 \| $76,890 | $45,678 | +95.3% |
| Accounts receivable (net) | $142,567 \| $134,289 | $118,945 | +19.9% |
| Inventory | $67,890 \| $72,345 | $81,234 | -16.4% |
| Prepaid expenses | $23,456 \| $21,234 | $19,876 | +18.0% |
| **Total Current Assets** | **$557,825** \| **$503,301** | **$453,165** | **+23.1%** |

这种结构测试了工具能否处理多列布局，区分普通科目与加粗的合计项，正确解析正负百分比变化，并保持每个数值与其对应季度的关联性。一个工具即使提取了正确的数值$234,678，但将其与 2024 年 Q2 而非 2024 年 Q3 关联，也未能满足我们的准确性要求。

创建测试文档后，我们手动构建了一个全面的标准答案数据集，存储于 ground_truth_data.json 文件中。这个 JSON 文件包含超过 150 个独立的数据字段，代表了我们认为完美的文档解析器应提取的所有信息。标准答案结构镜像了文档的组织方式，不同部分采用嵌套对象。以下是标准答案表示文档元数据和财务关键数据的方式：

{
  "document_metadata": {
    "company_name": "FruitStand Innovation Inc.",
    "report_type": "Quarterly Financial Report",
    "quarter": "Q3",
    "year": 2024,
    "report_date": "2024-09-30",
    "fiscal_year": 2024,
    "document_id": "FSI-FIN-2024-Q3-001",
    "classification": "Internal Use Only",
    "nasdaq_symbol": "FSTI"
  },
  "key_financial_highlights": {
    "total_revenue": {
      "value": 487.3,
      "unit": "million",
      "yoy_change": "+23%"
    },
    "net_income": {
      "value": 89.2,
      "unit": "million",
      "yoy_change": "+31%"
    },
    "operating_margin": {
      "value": 24.8,
      "unit": "percent",
      "comparison": "vs 21.4% in Q3 2023"
    }
  }
}

标准答案格式不仅捕获原始值，还捕获其关联的单位、百分比变化和上下文比较。这种细致的层级确保我们的基准测试衡量的是真正的语义理解能力，而非简单的文本提取。当我们评估一个工具是否正确提取了 total_revenue 时，我们验证它是否捕获了数值（487.3）、理解了单位（百万），并识别了同比变化的上下文（+23%）。

我们的基准测试实现使用这些测试文件来自动评估每个工具的性能。test_deepseek_parsing.py Python 脚本展示了我们如何提示 DeepSeek-R1 来提取这些结构化数据。以下是我们实际使用的提示词：

prompt = f"""Analyze this financial report and extract ALL key information in structured JSON format.
The document is a Q3 2024 financial report. Extract the following with EXACT values from the document:
1\. **Document Metadata**: company_name, report_type, quarter, year, report_date, document_id, nasdaq_symbol
2\. **Financial Highlights**: total_revenue, net_income, operating_margin, cash_and_equivalents, total_assets (with values and units)
3\. **Balance Sheet - Current Assets** (Q3 2024, Q2 2024, Q3 2023):
   - cash_and_cash_equivalents, short_term_investments, accounts_receivable_net,
     inventory, prepaid_expenses, total_current_assets
4\. **Balance Sheet - Non-Current Assets** (Q3 2024):
   - property_plant_equipment, accumulated_depreciation, net_ppe, intangible_assets,
     goodwill, long_term_investments, total_non_current_assets
...
CRITICAL: Extract EXACT numerical values, preserve decimals, include negative numbers
with minus sign. Return valid JSON only.
"""

这种提示词结构被证明是有效的，因为它精确指定了要提取的字段，要求结构化的 JSON 输出格式，并强调了数值精确性的重要性。EXACT values（精确值）和 preserve decimals（保留小数）等短语帮助 DeepSeek-R1 理解，对于财务文档分析，近似值或四舍五入的数字是不足够的。

完整的测试文件，包括完整的财务报表、包含所有 150+字段的全面标准答案数据集、适用于所有三种工具（DeepSeek-R1、Docling 和 MarkItDown）的自动化基准测试脚本，以及评估框架文档，均可在我们的 GitHub 仓库中获取：github.com/PacktPublishing/DeepSeek-in-Practice/tree/main/Chapter04/document-understanding。您可以克隆该仓库，并使用提供的 run_benchmark_comparison.py 脚本来复现我们的基准测试结果。

为财务文档提取而设计 DeepSeek 的提示词

财务文档提取所需的提示词设计策略与通用文档分析有根本性的不同。关键挑战在于，要在数百个数据点中实现数值精确性，同时保留数值、其单位、时间上下文和层级结构之间的复杂关系。当为财务文档提示 DeepSeek-R1 时，具体性变得至关重要：诸如 提取收入数据 这样的模糊请求会产生不完整的结果，而精确的指令，如 提取 total_revenue 及其数值、单位（百万/千）、货币和同比百分比变化，则能引导模型进行全面提取，捕获所有相关属性。

在我们的财务文档提示方法中，有三个设计原则被证明是必不可少的：

首先，明确要求使用定义模式的结构化输出格式（JSON），而非接受自由格式的文本响应，因为结构化格式便于自动化验证和后续处理。

2. 提示词设计与评估

其次，通过明确要求EXACT values（精确数值）、preserve decimal places（保留小数位）和include sign indicators for negative numbers and percentage changes（对负数和百分比变化包含符号指示）来强调数值精度要求，因为财务分析容错率为零——4.873 亿美元的收入数据与 4.87 亿美元或 4.8730 亿美元在报告精度上具有显著差异。

第三，通过详尽列举所有需要提取的数据类别（文档元数据、财务亮点、资产负债表组成部分、利润表部分、地区分布、产品指标、关键绩效指标以及指导方针）来全面指定范围。财务文档包含众多部分，在提示中遗漏类别通常会导致这些部分在提取过程中被忽略。

DeepSeek-R1 的推理能力在提示词建立清晰验证标准时表现得尤为突出。我们的提示词不是简单地请求数据提取，而是包含了诸如maintain quarter-to-quarter associations in multi-column tables（在多列表格中保持季度关联）和preserve parent-child relationships for revenue breakdowns（保留收入分项的父子关系）等短语，这些激活了 DeepSeek 的上下文理解能力。这种方法利用了模型对文档结构进行推理的能力，而非将提取视为简单的模式匹配。以下提示词展示了将这些原则应用于我们的 FruitStand Innovation Inc.财务报告的实例：

Analyze this Q3 2024 financial report for FruitStand Innovation Inc.
and extract ALL financial data in structured JSON format.
Extract the following with EXACT values:
1\. Document Metadata: company_name, report_date, document_id, fiscal_year
2\. Financial Highlights: total_revenue, net_income, operating_margin,
   cash_equivalents, total_assets (with units and YoY changes)
3\. Balance Sheet (Q3 2024, Q2 2024, Q3 2023):
   - All current assets line items
   - All non-current assets line items
   - All liabilities and equity items
4\. Income Statement (all 3 quarters):
   - Revenue breakdown (product vs. service)
   - All cost categories
   - Operating metrics (margins, EPS)
5\. Revenue by Geographic Region (all 4 regions with %, growth, margins)
6\. Revenue by Product Line (all 4 products with %, growth, margins)
7\. All 8 Key Performance Indicators with exact values
8\. Management team names and titles
9\. Q4 2024 Guidance ranges
CRITICAL: Extract EXACT numerical values, preserve decimals,
include negative numbers with minus sign. Return valid JSON only.
[Attached: 192-line financial report document]

现在，我们将探讨如何针对此案例评估 DeepSeek 的响应。

响应评估

为了客观地比较 DeepSeek-R1 与专用文档解析工具 Docling 和 MarkItDown 的性能，我们建立了六项量化指标，用以衡量文档理解能力的不同方面。每项指标都经过精心设计，可通过与我们的基准真实数据集进行自动比较来客观衡量。以下各节将解释每项指标的含义、其对文档理解的重要性以及计算方法。

指标 1：字段提取准确性

第一项指标衡量工具能否从文档文本中准确识别和提取语义字段-值对。在文档理解中，键值对代表离散的信息片段，其中一个描述性键（字段名）与一个特定值相关联。

以我们测试文档中的短语Total Revenue: $487.3 million为例。文档解析器必须识别total_revenue是语义字段名，487.3是数值，million代表单位，货币是美元。简单的文本提取只会捕获$487.3 million，而无法理解其与收入的语义关系。

我们的测试文档包含从简单元数据到复杂嵌套财务指标的键值对。简单示例如公司名称（FruitStand Innovation Inc.）和报告日期（2024-09-30），而复杂示例如多属性财务指标“总收入”，需要将数值（487.3）、单位（million）和同比变化背景（+23%）捕获为一个结构化对象。营业利润率是另一个复杂案例，其中数值（24.8）必须与其单位（percent）和比较基线（与 2023 年第三季度的21.4%相比）相关联。这些复合值测试了工具是理解文档语义还是仅仅提取孤立的文本片段。

为了计算字段提取准确性，我们实现了一个自动比较脚本，该脚本加载基准真实 JSON 数据，并逐字段将每个工具的输出与之对比。一个字段只有在键和值都符合我们严格标准时才被标记为正确：数值必须精确到小数点后一位（487.3，而非487或487.30）；字符串必须完全匹配，包括大小写（FruitStand Innovation Inc.，而非fruitstand innovation inc.）；日期必须遵循 ISO 格式（2024-09-30）；复合值必须包含所有必需属性。

公式很简单：用正确提取的字段数除以预期总字段数（我们测试文档中为 150），再乘以 100。当 DeepSeek-R1 从 150 个字段中正确提取了 137 个时，其准确率为 91.3%。

指标 2：表格解析准确性

表格解析准确性衡量工具能否在保留关键行列关系的同时，正确从表格数据中提取单个单元格的值。该指标尤其具有挑战性，因为它要求理解表格结构，而不仅仅是顺序阅读文本。我们合并资产负债表的多列布局展示了这种复杂性，它比较了三个季度的财务数据。以Cash and cash equivalents（现金及现金等价物）这一行项目为例，它在横跨一行的四个不同值：$234,678对应Q3 2024，$198,543对应Q2 2024，$187,432对应Q3 2023，以及计算出的变化值+25.2%。单元格级别的准确性要求所有四个值都以精确的数值精度提取，并与各自的时间段正确关联。

当我们考虑潜在的解析错误时，这一挑战变得显而易见。工具可能成功提取出$234,678的值，但错误地将其与Q2 2024而非Q3 2024关联。虽然数值提取成功了，但语义理解失败了，因为季度-值关系被错误识别。这种错误将导致提取的数据对财务分析毫无用处，即使表面上工具找到了正确的数字。我们的基准真实数据集通过不仅编码单元格值还编码其位置上下文来解决这个问题，要求工具证明它们理解表格结构，而不仅仅是扫描数字。

我们枚举了基准真实数据集中 6 个主要表格的所有 245 个单元格，通过表名、行标签和列标题对每个单元格进行标识。自动比较脚本检查每个提取的单元格是否与预期值和位置匹配。百分比值必须包含符号（+23%而非23%），货币值在存在时必须保留美元符号和逗号分隔符，计算字段如百分比变化必须完全匹配而不是重新计算。这种严格性确保我们衡量的是精确提取能力，而非近似理解。DeepSeek-R1 在总共 245 个单元格中正确解析了 232 个，相当于 94.7%的表格解析准确性。

指标 3：实体识别准确性

命名实体识别用于衡量工具识别嵌入文档文本中的特定现实世界对象的能力。我们的财务报告包含 42 个不同的命名实体，跨越多个类别，每个类别在文档理解中都有不同的用途。公司信息实体提供组织背景：公司名称（FruitStand Innovation Inc.）、其纳斯达克股票代码（FSTI）以及唯一文档标识符（FSI-FIN-2024-Q3-001）都代表必须被正确识别和分类的不同实体。高管姓名代表另一个关键的实体类别，我们的测试文档提到了陈莎拉（首席执行官）、迈克尔·罗德里格斯（首席财务官）、珍妮弗·吴（首席技术官）和戴维·朴（首席运营官）。每个姓名都必须被识别为与其各自领导职位相关联的人名实体。

地理实体增加了另一层复杂性。我们的收入分析部分引用了四个不同的地区：北美、EMEA（欧洲、中东和非洲）、APAC（亚太地区）和拉丁美洲。这些不仅仅是地名，而是构建我们收入分解的语义类别。同样，我们的四条产品线——云平台、移动解决方案、企业服务和开发者工具——代表组织财务绩效指标的产品实体。时间实体通过特定日期完善了我们的实体目录，例如2024 年 9 月 30 日作为报告日期，以及季度标识符（2024 年第三季度、2024 年第四季度）确立了报告和预测的时间范围。

实体识别准确率是通过检查工具提取的输出中是否出现了 42 个预期实体中的每一个，并且具有正确的拼写、适当的上下文和正确的实体分类来计算的。一个提取了陈莎拉但未能将其识别为人名，或者识别出北美但不理解其为地理区域的工具，将无法通过我们的实体识别要求。DeepSeek-R1 在此指标上取得了完美表现，正确识别了 42 个中的全部 42 个实体，实体识别准确率达到 100%。

指标 4：结构保存评分

结构保存评估工具在提取过程中保持文档逻辑组织、层级关系和阅读顺序的能力。这一指标很重要，因为财务文档的意义不仅通过内容传达，也通过结构传达。想想我们的财务报告遵循的传统顺序：执行摘要呈现关键要点，随后是详细的资产负债表，接着是带有分部明细的利润表，最后是前瞻性指引。这种排序并非随意，而是反映了从历史结果到未来预测的详细程度递增和时间递进。

层级关系在整个文档中承载着语义分量。当我们的报告将按地区划分的收入作为章节标题，随后是四个小节（北美、EMEA、APAC和拉丁美洲）时，这种结构传达了这四个地区是一个完整地理分析的组成部分。一个提取了这些地区名称但丢失了父子关系的工具会忽略这种组织逻辑。同样，视觉格式提供了关键线索：粗体文本通常表示总计或章节标题，缩进的行项目表示支持粗体总计的详细分解，而总收入条目以粗体显示，其下缩进显示产品收入和服务收入，则标志着一种有助于理解的求和-明细关系。

我们使用 0-100 分制的混合定量定性方法来评估结构保存。这可以涵盖以下方面：

章节排序贡献 25 分（提取是否保持了执行摘要→资产负债表→利润表→分析的序列？）
表格结构完整性占 30 分（行列关系和表头关联是否得到保留？）
标题数据关联提供 25 分（章节标题是否正确链接到其内容？），层级嵌套级别构成最后的 20 分（父子关系和缩进模式是否得到保持？）

这种评分结合了自动检查（例如验证表格列数和章节排序）和人工评估提取输出是否保持逻辑组织和可用性。

指标 5：处理时间

对于处理成千上万文档的生产系统，处理速度变得至关重要。我们通过记录提交文档之前和接收完整输出之后立即的时间戳来测量端到端处理时间。对于像 DeepSeek-R1 这样的基于 API 的工具，我们使用 Python 的time.time()函数来捕捉这些边界，测量总耗时，包括网络延迟、服务器排队时间、模型推理和响应格式化。这种现实世界的计时反映了生产系统将经历的情况。DeepSeek-R1 处理我们 192 行的财务文档耗时 4,250 毫秒（4.25 秒），对于用户可以等待几秒钟获取结果的交互式应用来说足够快，但对于需要每小时处理数千文档的批处理系统来说可能存在挑战。

指标 6：单文档成本

经济可行性决定了文档理解解决方案能否扩展到生产规模。对于 DeepSeek-R1，我们通过跟踪令牌消耗并应用公布的 API 定价来计算单文档成本。我们的测试文档总共需要 12,847 个令牌（输入和输出合计），按 DeepSeek 的定价（输入令牌每百万 0.14 美元，输出令牌每百万 0.28 美元），相当于每份文档约 0.0027 美元。按此费率，处理 10,000 份文档将花费 27 美元，这对于大多数商业应用来说是经济可行的。像 Docling 和 MarkItDown 这样的开源工具直接 API 成本为 0.00 美元，尽管自托管会引入基础设施费用（计算资源、维护和更新），我们在这次基准测试中并未量化这些费用。

测试数据分析

定义了我们的评估指标和提示策略后，我们现在可以查看 DeepSeek-R1 在处理我们的实际测试文档时的表现。我们为 FruitStand Innovation Inc.准备的 192 行财务报告是我们基准比较中所有三个工具的输入。文档以执行摘要信息开始，测试实体识别和简单字段提取，然后逐步展开越来越复杂的表格结构，挑战解析器的能力。以下是测试文档的开头：

### FRUITSTAND INNOVATION INC.
### Quarterly Financial Report Q3 2024
**Report Date:** September 30, 2024 
**Fiscal Year:** 2024 
**Document ID:** FSI-FIN-2024-Q3-001 
**Classification:** Internal Use Only 
---
### Executive Summary
FruitStand Innovation Inc. (NASDAQ: FSTI) reports strong financial performance for Q3 2024,
with revenue growth of 23% year-over-year and operating margin expansion of 340 basis points.
**Key Financial Highlights:**
- Total Revenue: $487.3 million (up 23% YoY)
- Net Income: $89.2 million (up 31% YoY)
- Operating Margin: 24.8% (vs 21.4% in Q3 2023)
- Cash and Equivalents: $234.7 million
- Total Assets: $1,247.8 million

概述

本开篇部分测试工具能否从标题中提取结构化元数据（公司名称、文件编号和纳斯达克代码），同时从要点摘要部分捕获关键财务指标。这些指标以半结构化格式呈现，混合了绝对数值（4.873 亿美元）、百分比变化（同比增长 23%）以及比较背景（相较于 2023 年第三季度的 21.4%），要求工具能够解析复合信息，而非孤立的数字。

该文档的主要难点在于其合并资产负债表，它呈现了一个同时比较三个时期、多列结构的表格。每个财务科目行跨越四个数据列及描述性文本，要求解析器必须精确维护数值与其时间背景之间的关联：

### 合并资产负债表
#### 截至 2024 年 9 月 30 日
**(单位：千美元，股份数据除外)**
| **资产** | **2024 年第三季度** | **2024 年第二季度** | **2023 年第三季度** | **变化 (%)** |
|-----------|-------------|-------------|-------------|----------------|
| **流动资产** | | | | |
| 现金及现金等价物 | $234,678 \| $198,543 | $187,432 | +25.2% |
| 短期投资 | $89,234 \| $76,890 | $45,678 | +95.3% |
| 应收账款（净额） | $142,567 \| $134,289 | $118,945 | +19.9% |
| 存货 | $67,890 \| $72,345 | $81,234 | -16.4% |
| 预付费用 | $23,456 \| $21,234 | $19,876 | +18.0% |
| **流动资产合计** | **$557,825** \| **$503,301** | **$453,165** | **+23.1%** |

此表格结构揭示了若干解析复杂性。流动资产标题行中的空白单元格必须被识别为结构性元素，而非缺失数据。粗体格式用于区分科目标题和合计行与明细行，要求工具保留格式语义。负值存货（通过逐季递减的数字体现）考验着符号处理能力，而最右侧列的百分比变化则要求提取幅度和方向指示符（+/-号）。文档稍后的累计折旧行显示为($123,456)（括号表示），这是会计惯例中表示负值的方式，解析器必须正确解读，而非仅将括号视为格式符号。

利润表部分通过涉及多个季度、基于百分比的嵌套计算引入了额外的复杂性：

### 合并利润表
#### 截至 2024 年 9 月 30 日的三个月
**(单位：千美元，每股数据除外)**
| **收入** | **2024 年第三季度** | **2024 年第二季度** | **2023 年第三季度** | **变化 (%)** |
|------------|-------------|-------------|-------------|----------------|
| 产品收入 | $312,456 \| $287,654 | $245,678 | +27.2% |
| 服务收入 | $174,892 \| $168,543 | $150,234 | +16.4% |
| **总收入** | **$487,348** \| **$456,197** | **$395,912** | **+23.1%** |
| | | | | |
| **营业成本** | | | | |
| 产品收入成本 | $143,567 | $139,876 | $128,945 | +11.3% |
| 服务收入成本 | $76,543 | $74,321 | $70,234 | +9.0% |
| **营业成本合计** | **$220,110** | **$214,197** | **$199,179** | **+10.5%** |
| | | | | |
| **毛利** | **$267,238** | **$242,000** | **$196,733** | **+35.8%** |
| **毛利率** | **54.8%** | **53.0%** | **49.7%** | **+5.1 pts** |

请注意本部分如何在绝对美元金额和基于百分比的指标之间交替。例如，毛利率字段要求解析器在同一个表格结构中处理不同的数据类型。表示利润率改善的+5.1 pts标注使用了一个领域特定的缩写（pts代表基点），语义解析器必须正确理解。

计算最终得分

总绩效分数使用加权平均法结合所有六个指标，该加权平均法反映了它们在生产环境文档理解系统中的相对重要性。我们根据实际部署中的关键因素分配了权重：字段提取准确性作为文档解析的主要目标，权重为 35%；考虑到商业文档中表格数据的普遍性，表格解析准确性权重为 30%；实体识别权重为 15%，因为实体代表了关键的文档对象；结构保留权重为 10%，以保持文档组织性；而处理时间和成本各占 5%，作为实际约束而非主要目标。

该公式将每个指标分数乘以其权重并求和：

让我们具体分解如下：

字段提取贡献了 31.96 分。
表格解析增加了 28.41 分。
完美的实体识别提供了 15.00 分。
结构保留贡献了 8.50 分。
处理时间增加了 4.50 分。
成本效率提供了 4.75 分。

总计得到 93.12 分，我们将其四舍五入为 93/100。

我们设立了绩效层级来解读这些分数。95 至 100 分表示卓越性能，适合需要最少人工验证的生产部署，代表着能够持续提供准确结果、仅需抽查的工具。85 至 94 分表示良好性能，适合具有针对性验证工作流程的生产环境，其中自动化提取处理大部分工作，但关键字段需经过验证。75 至 84 分代表可接受性能，需要大量验证和质量保证，适用于人工审核仍然广泛存在的半自动化工作流程。低于 75 分表示性能较差，在有显著改进之前不适合生产部署，通常需要更多开发才能投入实际使用。

基准测试比较：从文档到准确度分数

我们的基准测试系统通过在 run_benchmark_comparison.py 中实现的四阶段自动化流水线来评估每个工具。

让我们详细描述这一流水线：

过程首先按顺序运行每个工具的测试脚本（test_deepseek_parsing.py、test_docling_parsing.py 和 test_markitdown_parsing.py），捕获其输出并测量处理时间。
每个工具接收完全相同的输入文档，并必须生成代表提取信息的结构化输出。
对于 DeepSeek-R1，我们测量从向 DeepSeek API 提交 HTTP 请求到接收到完整 JSON 响应的端到端处理时间。
API 调用将完整文档文本作为输入连同我们的提取提示一并发送，DeepSeek-R1 返回包含所有已识别字段的结构化 JSON。
我们将此输出保存到 deepseek_test_results.json 以供后续准确性分析。处理时间测量使用 Python 的 time.time() 函数捕获 API 调用前后的高精度时间戳，以获得生产应用程序将经历的真实延迟。
一旦所有三个工具完成其提取，我们的比较脚本从 ground_truth_data.json 加载基准数据集，并逐字段进行验证。比较算法遍历基准数据集中定义的每个字段，检查每个工具的输出是否包含该字段且值正确。对于简单字段（如公司名称或报告日期），我们验证精确的字符串匹配。对于数字字段，我们在适当的精度容差范围内比较数值。对于复合字段（如 total_revenue，包含数值、单位和同比变化），我们验证所有属性是否符合预期结构。

此自动比较为字段提取、表格解析（逐单元格比较）和实体识别（检查所有 42 个预期实体是否正确出现）生成准确度百分比。

基准测试脚本随后将结果编译成对比表格，展示各工具在所有指标上的并行性能。这些表格不仅显示了准确率百分比，还包括处理时间、token 使用量（针对基于 API 的工具）和单文档成本，为每个工具的生产就绪性提供了全面视图。整个基准测试过程通过python run_benchmark_comparison.py命令自动运行，方便希望验证我们的结果或使用不同文档进行测试的读者复现。

DeepSeek-R1 成功提取了财务数据，其输出结构如下：

{
  "document_metadata": {
    "company_name": "FruitStand Innovation Inc.",
    "report_date": "2024-09-30",
    "document_id": "FSI-FIN-2024-Q3-001",
    "nasdaq_symbol": "FSTI"
  },
  "key_financial_highlights": {
    "total_revenue": {"value": 487.3, "unit": "million", "yoy_change": "+23%"},
    "net_income": {"value": 89.2, "unit": "million", "yoy_change": "+31%"},
    "operating_margin": {"value": 24.8, "unit": "percent"}
  },
  "balance_sheet_current_assets_q3_2024": {
    "cash_and_cash_equivalents": 234678,
    "short_term_investments": 89234,
    "accounts_receivable_net": 142567,
    "inventory": 67890,
    "prepaid_expenses": 23456,
    "total_current_assets": 557825
  },
  // ... [另外 145 个字段被正确提取]
}

以下是 DeepSeek-R1 响应的量化结果：

| 指标 | 结果 | 评估 |

| :--- | :--- | :--- |

| 字段提取准确率 | 91.3% | 优秀 |

| 正确字段 | 137/150 | 成功率高，有少量遗漏 |

| 表格解析准确率 | 94.7% | 优秀 |

| 正确表格单元格 | 232/245 | 单元格提取近乎完美 |

| 实体识别 | 100% | 完美 |

| 已识别实体 | 42/42 | 实体捕获完整 |

| 处理时间 | 4,250 毫秒 | 快速 |

| Token 使用量 | 12,847 tokens | 消耗适中 |

| 单文档成本 | $0.0027 | 非常低 |

| 总分 | 92/100 | 优秀 |

表 4.5：DeepSeek-R1 在金融文档理解方面的量化结果

以下是我们对 DeepSeek-R1 性能的分析。

DeepSeek-R1 在整个文档提取过程中对金融术语展现了出色的语义理解能力。具体结果如下：

在遇到执行摘要的关键财务亮点时，模型正确理解了诸如 operating margin expansion of 340 basis points 这样的复杂短语，并准确提取了数值 24.8% 及其比较背景（与 2023 年第三季度的 21.4%相比）。
模型成功区分了利润表中的不同收入类型，将$312,456 准确分类为产品收入，$174,892 分类为服务收入，并在多列结构中正确维持了它们的季度关联。
该工具对嵌套表格结构的处理被证明特别强大。在合并资产负债表中，DeepSeek-R1 正确解析了 Total Current Assets ($557,825) 及其组成部分之间的层次关系，理解现金及现金等价物 ($234,678)、短期投资 ($89,234)、应收账款 ($142,567)、存货 ($67,890) 和预付费用 ($23,456) 之和构成了总额。
模型还成功处理了利润表中的嵌套计算，认识到毛利润 ($267,238) 等于总收入 ($487,348) 减去总销售成本 ($220,110)，这表明其理解的是会计关系，而不仅仅是提取孤立的数字。
实体识别在整个文档的所有 42 个命名实体上都达到了完美的准确率。DeepSeek-R1 正确识别了高管姓名，包括 Sarah Chen (Chief Executive Officer)、Michael Rodriguez (Chief Financial Officer)、Jennifer Wu (Chief Technology Officer) 和 David Park (Chief Operating Officer)，并恰当地将每个姓名与其对应职务关联。地理实体被准确提取并保留了完整名称：North America、EMEA（被明确识别为欧洲、中东和非洲，而非作为未定义的首字母缩写处理）、APAC（亚太地区）和 Latin America。产品线实体 – Cloud Platform、Mobile Solutions、Enterprise Services 和 Developer Tools – 被正确识别并与其各自的收入数字和绩效指标相关联。

然而，DeepSeek-R1 遇到了 13 个提取错误，主要出现在深度嵌套的多季度对比单元格中：

八个错误发生在资产负债表的季度对比列中，这些单元格包含多个相关值。例如，在 Accounts Payable 行中，模型正确提取了 2024 年第三季度的值 ($87,654) 和同比百分比变化 (+14.5%)，但错误地关联了 2024 年第二季度的值，将$82,345 放入了 2023 年第三季度列，而非正确的 2024 年第二季度位置。
三个格式歧义错误涉及遵循会计惯例的数字：以括号显示的累计折旧值 ($123,456) 初始提取时未包含负号，这需要模型推断出 Liabilities 部分中的括号值代表负数。
两个错误涉及缩写展开，其中表头中的 PP&E (Property, Plant & Equipment) 被提取为缩写形式，而非基准数据集所期望的展开术语。

尽管存在这些小错误，DeepSeek-R1 在整个文档中表现出对百分比计算和货币格式的出色处理能力。模型在百分比变化中正确保留了正负号（现金增长为 +25.2%，而存货下降为 -16.4%），保持了财务数字的小数精度（$234,678 未被四舍五入为 $234,700），并正确理解了毛利率基点变化的 +5.1 pts 表示法。货币符号和逗号分隔符在提取输出中始终得以保留，显示了其对维护金融数据完整性至关重要的格式细节的关注。

理解 DeepSeek-R1 的提取错误

在将 DeepSeek-R1 与其他工具进行比较之前，我们应该审视其遇到的具体错误，以了解其局限性。这 13 个提取错误可分为 3 个不同的类别，每个类别揭示了文档理解挑战的不同方面。八个错误发生在嵌套单元格混淆场景中，多季度对比表格呈现了模糊的单元格边界。在合并资产负债表的 Accounts Payable 行中，DeepSeek-R1 正确提取了 Q3 2024 的值 ($87,654) 和同比变化 (+14.5%)，但错误关联了 Q2 2024 的值，将 $82,345 放入了 Q3 2023 列，而非其正确的 Q2 2024 位置。这类错误表明，当多个时间段出现在相邻单元格中时，模型有时难以维持精确的列关联，尤其是在百分比变化计算增加了额外认知复杂度的情况下。

格式错误分析与工具对比

格式歧义错误

三个格式歧义错误涉及需要领域知识才能正确理解的会计惯例数字。累计折旧项目显示为($123,456)，其值被括号包围，遵循标准会计惯例，其中括号表示负金额。DeepSeek-R1 最初将此值提取为正数123,456，忽略了括号表示法的语义含义。虽然模型最终识别出这些是负债并从上下文中推断出负值，但最初的提取表明，隐式格式化约定甚至可能挑战复杂的语言模型。同样，在遇到累计其他综合亏损值($7,520)时，模型正确解释了负号，但在最初的提取中经历了小数点位置的不确定性。

缩写展开错误

两个缩写展开错误揭示了技术术语识别的挑战。资产负债表中的表头使用PP&E作为Property, Plant & Equipment的缩写，这是财务报告中的标准缩写。DeepSeek-R1 逐字提取了PP&E，而不是将其扩展为我们基准数据集所期望的全称。虽然缩写形式包含相同的语义含义，但与基准数据集进行比较的自动验证系统需要精确的字符串匹配，这凸显了领域特定缩写如何在功能正确的提取结果与根据严格验证标准技术上不正确的结果之间造成差异。这表明，针对财务文档的提示可能受益于包含常见缩写词典或明确要求使用扩展术语。

工具对比

为了提供全面的背景并了解DeepSeek-R1的性能与专用文档解析工具的对比情况，我们使用了两种替代方法测试了相同的财务文档：Docling，一个专注于文档结构分析和表格识别的 IBM Research 工具包，以及MarkItDown，一个微软设计用于为 LLM 优化文档预处理的工具。每个工具都使用相同的测试文档和基准数据集进行评估，并根据我们先前定义的六个定量指标进行衡量：字段提取准确性、表格解析准确性、实体识别准确性、结构保留分数、处理时间和每文档成本。

测试 `Docling` 和 `MarkItDown`

Docling的操作方式与DeepSeek-R1有根本不同，它专注于文档结构提取而非语义理解。我们使用pip install docling安装了Docling，并创建了一个 Python 脚本（test_docling_parsing.py）来处理我们的财务报表 Markdown 文件。基本使用模式包括加载文档、运行Docling解析器并提取表格结构：

from docling import DocumentConverter
converter = DocumentConverter()
result = converter.convert("test_financial_statement.md")
#### 提取表格
tables = result.document.tables
for table in tables:
    print(f"Table: {table.num_rows} rows x {table.num_cols} columns")
    # 处理表格单元格...

Docling在识别表格边界、保留行列关系和维护文档结构方面表现出色，但产生的是结构化表示，而非语义字段提取。对于我们的财务文档，Docling正确识别了所有 6 个表格，并保留了 97.2%的单元格结构（245 个单元格中的 238 个），在表格解析准确性上优于DeepSeek-R1。然而，Docling无法直接提取诸如total_revenue等带有相关单位和同比变化的语义字段，这就是为什么我们在其字段提取准确性上显示N/A。从Docling的输出中获取语义信息需要额外的 LLM 处理。

MarkItDown遵循类似的理念，为 LLM 消费优化文档预处理。安装需要pip install markitdown，该工具将各种文档格式转换为适合 LLM 输入的干净 Markdown：

from markitdown import MarkItDown
converter = MarkItDown()
markdown_output = converter.convert("test_financial_statement.md")
print(markdown_output.text)

MarkItDown仅用 890 毫秒处理了我们的文档，明显快于DeepSeek-R1（4,250 毫秒）和Docling（2,100 毫秒），但只达到了 78.5%的表格解析准确性。该工具优先考虑处理速度和 Markdown 的整洁性，而非完美的结构保留，使其适用于可接受一些结构歧义的文档。与Docling一样，MarkItDown需要后续的 LLM 处理来进行语义字段提取。

Docling和MarkItDown的完整使用示例、配置细节和输出样本都记录在我们的 GitHub 仓库中：完整的使用示例、配置细节和输出样本都记录在我们的 GitHub 仓库中。仓库包含docling_usage_guide.md（分步说明）、markitdown_usage_guide.md（实现示例）以及tool_comparison_results.md（显示所有三个工具的完整输出以进行直接比较）。

计算基准指标

我们的基准比较需要对所有三个工具的字段准确性、表格准确性、处理时间和成本进行客观测量。字段提取准确性是通过将每个工具对测试文档运行、捕获其输出，并使用自动化 Python 脚本将提取的字段与我们的基准数据集进行比较来计算的。对于ground_truth_data.json中定义的 150 个字段中的每一个，我们检查工具的输出是否包含该字段及其正确的值，同时考虑数值精度、字符串大小写敏感性和日期格式。DeepSeek-R1正确提取的 137 个字段除以预期的 150 个总字段，得出 91.3%的准确性。Docling和MarkItDown在字段提取上获得了N/A评级，因为它们的结构化输出在没有额外 LLM 处理的情况下不会直接产生语义字段提取。

表格解析准确性测量需要将提取的表格数据与我们 6 个主要表格中 245 个预期单元格进行逐单元格比较。我们通过基准数据中的表名、行索引和列索引来标识每个单元格，然后验证工具是否正确提取了该单元格的值，并具有正确的格式和位置关联。Docling通过其专门的表格结构识别算法达到了 97.2%的准确性（245 个单元格中的 238 个正确）。DeepSeek-R1使用语义理解达到了 94.7%的准确性（245 个单元格中的 232 个正确），而MarkItDown的轻量级预处理达到了 78.5%的准确性（245 个单元格中的 192 个正确）。单元格级别的比较表明，像Docling这样的专用表格解析器在结构识别方面表现出色，即使它们缺乏语义理解。

处理时间测量使用 Python 的time.time()函数，在将文档提交给每个工具之前和接收到完整输出之后记录高精度时间戳。对于DeepSeek-R1，我们测量了端到端的 API 延迟，包括网络传输、服务器队列时间、模型推理和响应格式化，对于我们的 192 行文档，结果为 4,250 毫秒。Docling在标准开发机器（MacBook Pro M1）上本地处理相同文档耗时 2,100 毫秒，而MarkItDown仅用 890 毫秒就完成了处理。这些测量反映了生产系统在实际中会遇到的性能，尽管实际时间会因网络条件、服务器负载和硬件规格而异。

基于 API 的工具与本地工具在单文档成本计算上存在显著差异。DeepSeek-R1 的成本源于 token 消耗量乘以 API 定价：我们的测试文档总计消耗了 12,847 个 token（输入加输出），按每百万输入 token $0.14 和每百万输出 token $0.28 计算，约合每文档$0.0027。按此费率，处理一百万份文档的成本为$2,700。Docling 和 MarkItDown 作为开源本地工具，不产生直接的 API 费用。然而，本地部署会引入基础设施开销，包括服务器硬件、维护、更新和运维成本，我们在本次基准测试中未对这些进行量化。组织在选择云端解决方案还是自托管方案时，必须权衡直接的 API 成本与基础设施投资。

六项指标的量化比较

下表呈现了针对三项工具在我们之前定义的六项量化指标上的完整基准测试结果：字段提取准确率、表格解析准确率、实体识别准确率、结构保留评分、处理时间和单文档成本：

| --- | --- | --- | --- |

| 字段提取准确率 | 91.3% (137/150) | N/A* | N/A* |

| 表格解析准确率 | 94.7% (232/245) | 97.2% (238/245) | 78.5% (192/245) |

| 实体识别准确率 | 100% (42/42) | N/A* | N/A* |

| 结构保留评分 | 88/100 | 95/100 | 72/100 |

| 处理时间 | 4,250 毫秒 | 2,100 毫秒 | 890 毫秒 |

| 单文档成本 | $0.0027 | $0.00 | $0.00 |

表 4.6：整体性能比较

*Docling 和 MarkItDown 专注于结构提取，需要额外的 LLM 处理才能完成语义字段提取。

结果解读

每个工具都展现出与其设计理念相符的显著优势。DeepSeek-R1 实现了最高的字段提取准确率（91.3%），成功从文档中直接识别了 150 个语义字段中的 137 个，例如 quarterly_revenue、operating_margin_expansion 和 year_over_year_revenue_growth。这种语义提取能力是其区别于结构化解析器的关键。该模型完美的实体识别得分（42/42 实体）证明了其在高管、地理区域和产品线等方面具有强大的命名实体识别能力。然而，DeepSeek-R1 的表格解析准确率（94.7%）略低于 Docling 行业领先的 97.2%，这表明对于纯结构提取任务，专用的表格识别算法仍然优于通用的语言模型。

Docling 取得了最高的表格解析准确率和结构保留评分，这反映了其专为文档布局分析而设计的特点。该工具正确识别了 245 个表格单元格中的 238 个，并保持了精确的行列关联，比 DeepSeek-R1 高出 2.5 个百分点。其 95/100 的结构保留评分表明它在维护层级关系、嵌套表格结构和文档阅读顺序方面接近完美。Docling 更快的处理时间（2,100 毫秒对比 DeepSeek-R1 的 4,250 毫秒）和零 API 成本使其在大批量批处理场景中极具吸引力。其在字段提取和实体识别方面标注为 N/A，这反映的是架构选择而非局限性：Docling 提供的是结构化表示，需要下游 LLM 处理才能提取语义含义。

MarkItDown 优先考虑速度和简洁性，而非全面的准确性，仅用 890 毫秒就完成了文档处理——不到 DeepSeek-R1 处理时间的五分之一。这种性能是以牺牲表格解析准确率（78.5%）和结构保留能力（72/100）为代价的，因为该工具有时会简化复杂的嵌套表格或合并单元格以生成更整洁的 Markdown 输出。在完美结构保真度不如为 LLM 消费进行快速预处理那么重要的使用场景中，MarkItDown 的轻量级方法提供了实用优势。与 Docling 一样，它也需要后续的 LLM 处理来进行语义提取。

理解工具能力

除了量化指标，每个工具都拥有影响其在不同文档理解场景中适用性的专用能力。直接字段提取能力指的是工具在无需额外处理步骤的情况下识别和提取语义字段（如 total_revenue、ceo_name 或 quarterly_growth_rate）的能力。DeepSeek-R1 通过自然语言理解出色地实现了这一点，而 Docling 和 MarkItDown 则生成需要 LLM 解释的中间结构表示。

表格结构识别衡量工具识别表格边界、保持行列关系、处理合并单元格以及在复杂表格中维护层级嵌套结构的能力。Docling 通过专门针对表格检测训练的计算机视觉和布局分析算法在此方面表现出色。DeepSeek-R1 即使视觉结构不明确也能理解表格语义，展现了非常好的性能，而 MarkItDown 则提供了适用于较简单表格的中等能力。

OCR 集成能力决定了工具能否处理扫描文档或包含文本的图像。Docling 集成了 OCR 引擎，可以处理由扫描图像创建的 PDF 文件，这对于旧文档数字化项目至关重要。DeepSeek-R1 和 MarkItDown 均不包含原生 OCR 功能，需要预处理过的基于文本的输入。语义推理能力衡量工具理解提取信息的含义、上下文和关系的能力。DeepSeek-R1 展现出出色的语义推理能力，它能够识别出 “operating margin expansion of 340 basis points” 意味着当前的 24.8%利润率是从之前的 21.4%增长而来。Docling 和 MarkItDown 缺乏此能力，它们更专注于结构提取。

成本效率考量不仅涉及直接的 API 定价，还包括基础设施、维护和运维开销。DeepSeek-R1 每文档$0.0027 的 API 成本意味着处理一百万份文档需花费$2,700，无需基础设施投资但需持续支付使用费。Docling 和 MarkItDown 消除了 API 成本，但自托管部署需要服务器基础设施、软件更新、依赖管理以及相关技术专业知识。相应的设置复杂度也不同：DeepSeek-R1 仅需 API 凭证和简单的 HTTP 请求；MarkItDown 可通过 pip 安装，配置极少；而 Docling 要获得最佳性能则涉及更复杂的依赖管理和配置。

为你的用例选择合适的工具

我们的基准测试结果表明，不同的文档理解场景需要根据具体要求和约束条件选择不同的工具。当分析财务报表或商业文档，且主要目标是语义提取特定字段时，DeepSeek-R1 提供了获取可操作数据最直接的路径。其 91.3%的字段提取准确率和完美的实体识别能力使得生产部署可通过验证工作流来捕获剩余的 8.7%错误。那些需要即时基于 API 部署且无需基础设施投资的组织，会发现 DeepSeek-R1 简单的集成方式和按使用量付费的定价模式在中等文档量下颇具吸引力。

对于涉及复杂视觉布局的 PDF 文档场景，将 Docling 的结构提取与 DeepSeek-R1 的语义理解相结合的混合方法能实现最佳效果。Docling 预先处理 PDF 以提取表格结构，准确率达 97.2%，随后DeepSeek-R1解析这些结构以提取语义字段。该流程充分利用了每种工具的核心优势：Docling专业的版面分析能力和DeepSeek-R1的语言理解能力。这种组合方案每文档成本仅 0.0027 美元（仅DeepSeek-R1的 API 费用），同时整体准确度优于任何单一工具。

处理扫描文档或含文字图像需要 OCR 能力，在我们的基准测试中仅有Docling提供此功能。传统文档数字化项目、历史财务记录分析，或任何涉及实体文档转图像的场景都需要Docling的 OCR 集成。典型工作流程是：Docling执行 OCR 和结构提取，然后将生成的文本送入DeepSeek-R1进行语义分析。在要求亚秒级响应延迟的实时 API 集成场景中，MarkItDown的 890 毫秒初始预处理时间可能更具优势，但DeepSeek-R1的 4.25 秒端到端延迟对大多数交互式应用仍然可接受。

大规模文档批量处理（百万级文档）需进行细致的成本效益分析。Docling零 API 成本在大规模应用中具有经济吸引力，但机构必须考虑预估为每年 50,000-200,000 美元的基础设施投资（包括服务器、存储、网络和运维人员）。按每月处理百万文档计算，DeepSeek-R1月成本为 2,700 美元（年成本 32,400 美元），显著低于自建基础设施成本——除非文档量达到约 500-1000 万份/月（具体取决于基础设施效率）。多语言文档可借助DeepSeek-R1强大的多语言处理能力，无需特定语言配置即可准确处理英语、西班牙语、普通话、法语、德语等语言的财务报告。

混合方案：最佳实践

基于我们的全面基准测试结果，处理复杂财务文档的最优方案是整合多种工具的互补优势，而非依赖单一解决方案。通过构建融合结构提取与语义理解的混合流程，机构可实现更高准确度、更低成本和更大灵活性。

成本效益分析与经济合理性

混合自动化的经济优势在规模化时尤为显著。结合Docling的零成本本地处理和DeepSeek-R1每文档 0.0027 美元的 API 费用，每处理千份文档的总成本为 2.70 美元。相比之下，由训练有素的财务分析师进行传统手动提取的成本为每文档 50 至 100 美元（取决于复杂度和准确度要求）。对于年处理 10,000 份财务文档的机构，手动提取年成本为 50 万至 100 万美元，而自动化混合流程仅需支付 27 美元年度 API 费用加上本地运行Docling的基础设施开销。

这种超过 99%的显著成本降低使得此前不可行的应用成为可能，例如全面分析所有供应商发票、自动提取历史财务档案，或实时处理竞争对手财务披露。规模化下的每文档成本（0.0027 美元）意味着处理百万份文档仅需 2,700 美元，使文档理解技术对于此前因成本过高而却步的手动方法应用成为经济可行方案。机构需考虑自托管Docling的基础设施成本（中等规模预估年成本 5,000-15,000 美元）以及流程开发与维护的工程资源（约年成本 50,000-100,000 美元，取决于复杂度），但即使包含这些额外成本，处理百万级文档的年总费用仍低于 120,000 美元。

如希望探索更多案例研究，如头脑风暴与创意激发和代码生成与理解，请访问本章 GitHub 仓库：

头脑风暴与创意激发：https://github.com/PacktPublishing/DeepSeek-in-Practice/tree/main/Chapter04/brainstorming-ideation
代码生成：https://github.com/PacktPublishing/DeepSeek-in-Practice/tree/main/Chapter04/code-assistant

总结

本章通过两个详细案例研究展示了DeepSeek-R1的文档理解能力：亚马逊 API 网关集成文档和财务报表分析。在亚马逊案例研究中，DeepSeek-R1分析了 45 页的技术集成指南，通过成功提取无服务器架构组件、识别安全配置并生成可投入生产的 CloudFormation 模板，获得了 87%的综合评估分数。该模型展现了处理复杂技术文档的强大能力，包括 API 网关限流策略、Lambda 授权模式、JWT 验证逻辑和分布式追踪配置。

5 构建 DeepSeek 应用

案例研究：财务文档分析

财务文档分析案例研究使用 FruitStand Innovation Inc. 的一份虚构季度报告（包含 192 行、6 个复杂表格和 150 个语义字段），评估了 DeepSeek-R1 与专用文档解析工具的表现。DeepSeek-R1 实现了 91.3% 的字段提取准确率（137/150 字段）、94.7% 的表格解析准确率（232/245 单元格）和完美的实体识别（42/42 实体），处理时间为 4,250 毫秒，每份文档成本为 $0.0027。我们将 DeepSeek-R1 与 Docling（IBM Research）和 MarkItDown（Microsoft）进行了比较，发现 Docling 通过专门的结构识别实现了更高的表格解析准确率（97.2%），MarkItDown 提供了最快的处理速度（890 毫秒）和中等准确率（78.5%），而 DeepSeek-R1 独特地提供了语义理解能力，并具备直接字段提取功能。将 Docling 的结构提取与 DeepSeek-R1 的语义解释相结合的混合流水线取得了最佳的综合效果。

在两项案例研究中，我们应用了一致的评估指标，包括对技术准确性和实施准备情况的定性评估，以及对提取准确性、处理时间和成本效益的定量测量。DeepSeek-R1 始终表现出专业顾问水平，在复杂文档上达到 88-92% 的准确率，每份文档成本 $0.0027，而人工分析成本为 $50-100，这展示了在自动化文档理解工作流程中 99% 以上的成本降低。实施 DeepSeek-R1 进行文档理解的组织应在部署前建立明确的评估指标，定义定性标准（如技术准确性和完整性）和定量测量（包括字段提取准确率、实体识别、处理时间和每份文档的成本），这些指标应与其特定的文档类型相适应。由于监管和合规要求，财务文档通常需要 95% 以上的准确率阈值，而一般技术文档可能接受 85% 以上的准确率，并对关键部分进行人工验证。

成功的实施需要仔细的提示工程，明确指定输出格式、必需字段、数值精度要求和特定领域术语。我们的财务文档分析表明，为常见缩写提供词汇表，例如 PP&E 代表 Property, Plant & Equipment，并明确是否扩展首字母缩略词，可以显著减少提取错误。对于复杂的多列表格，明确指示模型列之间的关联性和时间段比较，可以最大限度地减少嵌套单元格混淆。当初始提取遗漏关键字段时，迭代提示被证明很有价值，因为针对特定缺口的后续提示能够持续提高完整性，而无需重新处理整个文档。组织还应考虑根据文档特征结合工具优势的混合架构，例如，使用 Docling 预处理具有复杂视觉布局的 PDF 文件以 97.2% 的准确率提取表格结构，然后使用 DeepSeek-R1 进行语义字段提取和实体识别，总成本仅为每份文档 $0.0027，同时实现比任何单一工具更高的准确性。

接下来，我们将展示如何使用 DeepSeek API 创建一个可工作的服务。敬请期待。

获取本书的 PDF 版本和独家内容

扫描二维码（或访问 packtpub.com/unlock）。按书名搜索本书，确认版本，然后按照页面上的步骤操作。

注意： 请保留好您的发票。直接从 Packt 购买则无需发票。

在前面的章节中，我们讨论了 DeepSeek 的内部工作原理、有效提示 DeepSeek 模型的最佳策略以及如何使用 DeepSeek。在本章中，我们将付诸实践。我们将解决一个实际问题，并向您展示如何利用 DeepSeek 模型来解决它。

DeepSeek 模型是开源的，这既是优势也是挑战。您有无尽的选择：通过 API 使用大型模型，或使用可在本地运行的微缩模型，这很强大但需要权衡。使用 DeepSeek 模型没有唯一的正确方式；最佳选择取决于您的情况和具体用例的需求。您应该选择本地选项还是云选项？应该选择小模型、大模型还是推理模型？

本章的目标是双重的。我们想向您展示如何使用 DeepSeek 模型解决一个特定的用例（端到端），并演示您可以与 DeepSeek 模型交互或使用的主要方式。在本章中，您将学习使用模型的所有不同方式：本地、通过 API、在云端以及在您自己的机器上。

但在开始之前，我们需要一个用例。让我们看看 Duarte 对其日常与智能手表斗争的描述。

我和我的妻子都是长跑运动员，已经很久了。不瞒你说，我非常喜欢马拉松。跑步是为数不多的我真正从办公桌后走出去到户外的场合。我热爱它的一切。但这可能不是谈论它的好地方。我们都使用佳明智能手表；我们很喜欢它们！

每天晚上 8:45 整，我和妻子都会看手表，就在那一刻，手表会发给我们一个当天的摘要（压力大的一天、锻炼的一天或休息的一天）。这里有一个例子：

锻炼的一天

今天高强度的锻炼和休息时段对您的身心健康都有好处。现在花点时间放松，好好睡一觉。

图 5.1：佳明每日通知示例

在拥有手表的头几周，它们棒极了！它给了我另一个追踪和了解自己身体的指标。但几周后，您开始意识到您总是收到五条预写消息中的一条。从一个角度来看，这很酷；手表了解我的身体！从另一个角度来看，它很烦人，因为手表只是发送它内存中存储的五条预写消息中的一条。而且，这些消息太笼统了。几乎感觉手表并没有告诉我任何我不知道的事情。过了一段时间，它就感觉很无聊。

我们听到了您的心声，Duarte！

当我们三位作者开始构思这本书时，我们意识到在像 DeepSeek 这样的 大语言模型 (LLM) 时代，佳明的通知肯定可以做得更好——好得多。一条更动态、个性化的消息——一条可以接受某种用户输入的消息。您知道我们的意思。有很多方法我们可以让它变得更好。所以，这就是我们打算做的。

在本章中，我们将构建一个可以访问我的智能手表数据、读取数据并每天为我提供一个有趣且动态摘要的程序。不再只有五条预定义消息；每次我们收到摘要，消息都会不同。我们将利用 DeepSeek 来分析数据并为我们构建一个很好的摘要。我们将其称为 每日健康摘要。

在开始之前，有几点说明：

这个每日健康摘要将由 LLM 生成。正如您可能知道的，LLM 有时会 产生幻觉 或产生不准确的信息。本章我们不会讨论防护栏或保证模型输出的安全性。但是，如果您有兴趣了解更多，这里有一个有用的库供您入门：github.com/guardrails-ai/guardrails。

本章将重点讲解如何从 Garmin 手表中提取数据。我们将使用 Garth 库（https://github.com/matin/garth）进行身份验证。虽然该库广为人知且流行，但重要的是要记住它并非 Garmin 的官方库。因此，无法保证 Garmin 未来不会更改其身份验证方法。如果您没有 Garmin 设备，仍然可以从 Strava 或其他您使用的服务中提取数据；不过，您将需要使用这些替代来源自行处理数据提取过程。

那么，我们开始吧。

本章我们将涵盖以下内容：

构建我们的第一个原型
与 DeepSeek 模型进行交互
使用 AWS 部署一个隔离的模型服务
最佳实践和建议

技术要求

本章的所有代码都位于本书的 GitHub 仓库：https://github.com/PacktPublishing/DeepSeek-in-Practice。

您需要在机器上安装以下内容：

uv：请参阅以下链接获取安装说明：https://docs.astral.sh/uv/getting-started/installation/。
Docker：我们将为我们设计的 API 构建 Docker 镜像。请查看此链接开始安装：https://docs.docker.com/engine/install/。

本章的代码位于Chapter 05目录中；在章节文件夹内运行uv sync将安装所有依赖项。

本章假设您拥有一个 DeepSeek API 密钥，您可以在https://platform.deepseek.com/获取。

您可能需要安装其他库/依赖项，但我们会在章节中明确指出。

有几种方法可以使用 DeepSeek，例如，DeepSeek API、第三方 API，或使用一些库如litellm来利用 DeepSeek 模型。我们在本书的附录中介绍了一些这些选项。请去查看。

构建第一个原型

在本节中，我们将为“每日健康摘要”构建一个端到端的原型，遵循五个高级步骤：

首先，我们将从 Garmin API 获取数据并学习如何查询它。
接下来，我们将把这些原始健康数据塑造成 DeepSeek 能够理解和推理的格式。
然后，我们将定义结构化的输出：我们希望 DeepSeek 返回什么以及它应该是什么样子。
在结束之前，我们将把所有内容重构为一个您可以查询和使用的 API。
最后，我们将逐步介绍如何使用 Docker 部署所有内容。

让我们从从 Garmin 获取数据开始。

获取数据

首要任务是查询 Garmin 关于我们拥有的数据。这完全是以编程方式检索我们的 Garmin 手表记录的关于我们的健康数据（心率、睡眠时长、压力等）。为了完成这项工作，我们将利用python-garminconnect项目（https://github.com/cyberjunky/python-garminconnect）。

为了使其工作，您应该同时拥有您的 Garmin 电子邮件和用于登录 Garmin 账户的密码。两者都应适用于https://connect.garmin.com/signin。

注意

如果您没有 Garmin 账户，请不要担心。GitHub 仓库包含加载示例数据的说明，因此您可以毫无问题地运行 API 和示例。

一旦您手头有了这些信息，我们就可以使用以下函数登录 Garmin：

# We get the values as environmental variables
GARMIN_EMAIL = os.getenv("GARMIN_EMAIL") # your email
GARMIN_PASSWORD = os.getenv("GARMIN_PASSWORD") # your password
# and we define our function
def start_garmin() -> Garmin:
    """Initialize Garmin connection."""
    try:
        GARMIN = Garmin(
            email=GARMIN_EMAIL,
            password=GARMIN_PASSWORD,
            is_cn=False, # if you are in China, set to True
        )
        GARMIN.login()
        print("You are now logged in.")
        return GARMIN
    except Exception as e:
        print(f"Could not login with email and password: {e}")
        raise
If everything goes well, you can now run the following:
garmin = start_garmin()
# prints: You are now logged in.

此函数现在为我们提供了一个可以处理和查询 API 的garmin实例。现在我们知道如何实例化 Garmin API，让我们继续查询给定日期所需的所有信息。为此，我们创建一个get_daily_health_summary函数。此函数将接收一个 API 实例、一个开始日期和一个结束日期。目标是返回该日期范围内的健康信息。

首先，我们将创建几个辅助函数：一个将日期转换为 ISO 格式的函数（因为 Garmin 要求此格式），另一个接收两个日期并返回我们可以迭代的范围的函数：

def get_daily_health_summary(
    api: Any, start: datetime.date, end: datetime.date
) -> list[dict[str, Any]]:
    def dstr(d: datetime.date) -> str:
        return d.isoformat()
    def daterange(a: datetime.date, b: datetime.date):
        for i in range((b - a).days + 1):
            yield a + datetime.timedelta(days=i)

现在我们遍历每一天，并可以通过使用get_user_summary方法从 Garmin 检索摘要信息。此方法返回一个字典，其中包含我们想要用于上下文的字段，例如静息心率、步数、运动分钟数和睡眠时长：

 for day in daterange(start, end):
        s = dstr(day)
        day_of_week = day.strftime("%A")
        summary = api.get_user_summary(s) or {}
        rhr = summary.get("restingHeartRate")
        steps = summary.get("totalSteps")
        stress_level = summary.get("averageStressLevel")
        body_battery_final = summary.get("bodyBatteryMostRecentValue") or summary.get(
            "mostRecentBodyBattery"
        )
        exercise_minutes = (summary.get("moderateIntensityMinutes") or 0) + (
            summary.get("vigorousIntensityMinutes") or 0
        )
        sleep_seconds = summary.get("sleepingSeconds")
        sleep_hours = round(sleep_seconds / 3600, 2)
        body_battery_start = summary.get("bodyBatteryAtWakeTime")
        total_distance_meters = summary.get("totalDistanceMeters")

最后，一旦我们提取了所有感兴趣的字段，现在可以将它们附加到函数将返回的数组中：

 out.append(
            {
                "date": s,
                "day_of_week": day_of_week,
                "resting_heart_rate": rhr,
                "exercise_minutes": exercise_minutes,
                "stress_level": stress_level,
                "sleep_hours": sleep_hours,
                "steps": steps,
                "total_distance_meters": total_distance_meters,
                "body_battery_start_day": body_battery_start,
                "body_battery_end_day": body_battery_final,
            }
        )
    return out

有了这个函数，我们现在可以为任何感兴趣的日期调用它。例如，如果我们想获取昨天的数据，我们可以获取今天的日期，减去一天（使用timedelta），并调用get_daily_health_summary方法：

yesterday = datetime.date.today() - datetime.timedelta(days=1)
summary_for_yesterday = get_daily_health_summary(
    garmin, yesterday, yesterday)
summary_for_yesterday

调用此函数后，我们可以看到在 8 月 13 日，我的静息心率为 62，总共睡了 9 个小时（！）。我不确定发生了什么，但那确实睡了很多！俗话说，睡眠是运动员最好的朋友。

[{'date': '2025-08-13', 'day_of_week': 'Wednesday', 'resting_heart_rate': 62, 'exercise_minutes': 0, 'stress_level': 38, 'sleep_hours': 9.09, 'steps': 1866, 'total_distance_meters': 1387, 'body_battery_start_day': 74, 'body_battery_end_day': 17}]

请注意，我们也可以为一个日期范围调用我们的函数，这很有帮助，例如：

```python

past_7_days_start = yesterday - datetime.timedelta(days=7)

past_7_days_end = yesterday - datetime.timedelta(days=1)

summary_for_past_7_days = get_daily_health_summary(

    garmin, past_7_days_start, past_7_days_end

)

summary_for_past_7_days

这将返回一个介于 past_7_days_start 和 past_7_days_end 之间每一天的数据列表。如果这看起来不太明白，别担心！一切很快就会清晰：


[

    {

        'date': '2025-08-06',

        'day_of_week': 'Wednesday',

        'resting_heart_rate': 58,

        'exercise_minutes': 3,

        'stress_level': 32,

        'sleep_hours': 8.17,

        'steps': 5320,

        'total_distance_meters': 3953,

        'body_battery_start_day': 97,

        'body_battery_end_day': 20

    },

    {

        'date': '2025-08-07',

        'day_of_week': 'Thursday',

        # ...

    },

    {

        'date': '2025-08-08',

        # ...

构建上下文

我们现在可以查找特定日期或日期范围内的健康数据。接下来，我们需要为我们的 DeepSeek 模型准备数据。有很多种方法可以实现，但我们想保持简单。我们将创建一个提示词（prompt），其中包含一个简短的 Markdown 部分来呈现每个测量指标。对于每个测量指标（例如压力水平），我们希望向大语言模型（LLM）展示以下内容：

该指标的当日数值。
过去七天的平均值（基线），不包括今天。
与基线相比，今天数值的百分比变化。
基线期间的趋势。
对于该测量指标，较低的数值是否更好。

例如，对于睡眠时长，我们希望得到以下内容：


### 睡眠时长

- 今日数值 (2025-08-13): 9.09

- 7 天基线平均值 (不含今日): 8.21

- 与基线相比的百分比变化: +10.7%

- 过去 7 天趋势: up ↑

- 较低的数值更好: False

这将为每个我们获取的其他指标重复生成。

通过这些信息，我们现在是为模型提供了一个在特定上下文下的指标，而不是一个孤立的指标。这将确保模型减少“幻觉”，并理解指标更广泛的背景。

为了识别趋势，我们需要一个启发式方法。这里有一个简单的函数，它接受一个数字列表并返回 up（上升）、down（下降）或 flat（平稳）。它比较列表中前三项和后三项的平均值。如果这些平均值在 5%以内，函数返回 flat，因为指标没有发生重大变化。这不是一个完美的启发式方法，但能达到我们的目的：


def detect_trend(values, pct_threshold=5):

    recent, earlier = np.mean(values[-3:]), np.mean(values[:3])

    return (

        "up"

        if recent > earlier * (1 + pct_threshold / 100)

        else ("down" if recent < earlier * (1 - pct_threshold / 100) else "flat")

    )

有了我们的启发式方法，我们现在可以创建一个构建上下文的函数；build_llm_context_md 接收今天的摘要和过去七天的摘要。这些是相同的字典列表，其中包含当前日期和过去七天的健康指标信息。

然后，我们创建一个字符串列表，每个字符串代表 LLM 的一行上下文。最后，我们创建一个 better_is_lower 数组，用于存储那些分数较低更可取的指标，例如压力水平：


def build_llm_context_md(

    summary_for_today: list[dict], summary_for_past_7_days: list[dict]

) -> str:

    assert len(summary_for_today) == 1, "Expected 1 day of summary"

    today = summary_for_today[0]

    metrics = [key for key in today.keys() if key not in ["date", "day_of_week"]]

    lines = [

        f"# {today['date']} ({today['day_of_week']}) 每日指标摘要",

        "_ 注意：所有比较仅使用**过去 7 天的数据**，不含今日。_",

        "",

    ]

    better_is_lower = [

        "resting_heart_rate",

        "stress_level",

    ]

之后，我们遍历指标。对于每个指标，我们创建一个值列表。包括指标的当前值、基线期间的平均值，以及“较低的数值是否更好”标志。遍历完成后，我们通过 join 所有行来构建我们给 DeepSeek 的上下文：


for metric in metrics:

    today_val = today[metric]

    if not today_val:

        print(f"Skipped {metric} since no values were found")

    past_vals = [

        day[metric] for day in summary_for_past_7_days 

        if day[metric] is not None

    ]

    avg_7d = sum(past_vals) / len(past_vals) if past_vals else 0

    delta_pct = ((today_val - avg_7d) / avg_7d * 100) if avg_7d else 0

    trend_dir = detect_trend(past_vals)

    arrow = "↑" if trend_dir == "up" else ("↓" if trend_dir == "down" else "→")

    lines.append(

        f"## {metric.replace('_', ' ').title()}\n"

        f"- 今日数值 ({today['date']}): {today_val}\n"

        f"- 7 天基线平均值 (不含今日): {avg_7d:.2f}\n"

        f"- 与基线相比的百分比变化: {delta_pct:+.1f}%\n"

        f"- 过去 7 天趋势: {trend_dir} {arrow}\n"

        f"- 较低的数值更好: {metric in better_is_lower}\n"

    )

return "\n".join(lines)

我们现在可以通过以下方式调用该函数：


# 启动客户端并定义日期

garmin = start_garmin()

date = "2025-08-13"

# 获取指定日期的摘要数据

date_for_summary = datetime.datetime.strptime(date, "%Y-%m-%d").date()

summary_in_date = get_daily_health_summary(

        garmin, date_for_summary, date_for_summary

)

# 获取该日期之前 7 天的数据

past_period_start = date_for_summary - datetime.timedelta(days=7)

past_period_end = date_for_summary - datetime.timedelta(days=1)

summary_in_past_period = get_daily_health_summary(

        garmin, past_period_start, past_period_end

)

# 构建提示词

prompt = build_llm_context_md(summary_in_date, summary_in_past_period)

print(prompt)

这将输出一个清晰的摘要，我们可以将其输入给我们的大语言模型：


# 2025-08-13 (Wednesday) 每日指标摘要

_ 注意：所有比较仅使用**过去 7 天的数据**，不含今日。_

### 静息心率

- 今日数值 (2025-08-13): 62

- 7 天基线平均值 (不含今日): 55.86

- 与基线相比的百分比变化: +11.0%

- 过去 7 天趋势: up ↑

- 较低的数值更好: True

### 运动时长（分钟）

- 今日数值 (2025-08-13): 0

- 7 天基线平均值 (不含今日): 3.00

- 与基线相比的百分比变化: -100.0%

- 过去 7 天趋势: up ↑

- 较低的数值更好: False

### 压力水平

- ....

### 睡眠时长

- ...

... 其他指标因简洁而隐藏

很好！我们现在拥有了想要提供给 DeepSeek 模型的所有信息。但我们具体希望它如何回应呢？

定义结构化输出

现在是时候定义我们希望“每日健康摘要”呈现何种形态了。与大多数基于大语言模型的应用程序一样，让模型返回某种结构化输出（JSON 格式）通常是一个好做法（我们在 第 2 章 和 第 3 章 中讨论过）。为了表示我们的每日健康摘要，我们创建两个 Pydantic (docs.pydantic.dev/) 模型：

DayType：将我们的日期分类为五种类型之一。
DailySummary：创建一个对象，包含日期类型、标题、表情符号、观察结果和建议。

代码如下：


# 创建每日健康摘要

现在，我们已具备制作第一份每日健康摘要的所有要素。让我们开始创建摘要。

首先，我们将创建一个简单的函数，该函数使用 `type: json_object` 调用 DeepSeek API，以强制模型返回 JSON 结构。这并不出奇；它会返回模型创建的 JSON 以及推理内容（如果有的话，例如在使用 `deepseek-reasoner` 时）：

```python
from pydantic import BaseModel, Field
class DayType(str, Enum):
    TRAINING = "training"
    ACTIVE_RECOVERY = "active_recovery"
    REST = "rest"
    HIGH_STRESS = "high_stress"
    BALANCED = "balanced"
class DailySummary(BaseModel):
    day_type: DayType = Field(
        ...,
        description="基于活动和恢复指标对这一天进行分类",
    )
    title: str = Field(..., description="对这一天的单句总结")
    emoji: str = Field(..., description="代表当天类型的 Emoji 表情")
    observation: str = Field(
        ...,
        description="关于关键指标和模式的两句话观察",
    )
    recommendation: str = Field(
        ...,
        description="针对明天的两条可操作建议",
    )

def llm(
    messages: list[dict], model: str, response_format: dict | None = None
) -> tuple[dict, str | None]:
    client = OpenAI(
        api_key=os.environ["DEEPSEEK_API_KEY"],
        base_url="https://api.deepseek.com",
    )
    response = client.chat.completions.create(
        model=model, messages=messages, response_format=response_format, temperature=0.0
    )
    message = response.choices[0].message

    if hasattr(message, "reasoning_content"):
        reasoning_content = message.reasoning_content
    else:
        reasoning_content = None

    return json.loads(message.content), reasoning_content

为了确保我们的模型表现良好，我们创建了两个 DailyHealthSummary 示例，并制定了系统提示词：

# 示例 1：训练日
training_day_example = DailySummary(
    day_type=DayType.TRAINING,
    title="强有力的训练日，所有指标活动量均有所增加。",
    emoji="💪",
    observation="运动分钟数达到基线的两倍，活动了 95 分钟，并伴有 18,500 步。尽管训练负荷很高，但身体电量从 85 的良好水平开始，表明从昨天恢复得不错。",
    recommendation="考虑明天进行积极恢复或休息日，以适应今天付出的努力。今晚优先保证睡眠，以维持身体电量水平并支持肌肉恢复。",
)
# 示例 2：高压力日
high_stress_day_example = DailySummary(
    day_type=DayType.HIGH_STRESS,
    emoji="😫",
    title="尽管体力活动极少，但压力升高且恢复不佳。",
    observation="压力水平比基线高出 46%，而睡眠时间降至仅 6.1 小时，导致身体电量起点较低，仅为 45。运动和活动极少，表明压力来自非体力来源。",
    recommendation="专注于压力管理技巧，争取今晚睡足 8 小时以上。考虑明天进行轻度锻炼，如散步或瑜伽，因为温和的活动有助于调节压力水平。",
)
examples = [training_day_example, high_stress_day_example]
examples_str = "\n\n\n".join([example.model_dump_json() for example in examples])

系统提示词包含三个部分——为模型提供背景和指导的通用指令、我们期望它遵循的 JSON 模式，以及一些示例 JSON 输出：

system_prompt = f"""
Instructions:
* 你将收到用户今天与过去 7 天相比的健康与健身数据摘要。
* 你的目标是生成一个将在用户智能手表上显示的摘要。
* 保持内容简短，同时也要让用户感兴趣。
* 你的摘要应包括当天类型、标题、一些观察和给用户的建议。
* 你的摘要应采用 JSON 格式。仅输出 JSON，不要有其他文本。
---JSON SCHEMA---
{DailySummary.model_json_schema()}
---END JSON SCHEMA---
---EXAMPLE JSON OUTPUTS---
{examples_str}
---END EXAMPLE JSON OUTPUTS---
"""

现在万事俱备，我们可以创建最后一个函数了。这个函数的作用就是将我们至今构建的所有部分整合起来。它执行以下步骤：

接收一个 Garmin API 实例和一个字符串格式的日期。
从 Garmin 获取该日期及之前七天的数据。
构建我们的提示词和消息列表，包括系统提示词。
调用模型，并根据我们的 DailySummary Pydantic 模型验证其响应。

def get_daily_summary(
    garmin: Garmin,
    date: str,
    model: Literal["deepseek-chat", "deepseek-reasoner"],
    verbose: bool = False,
) -> DailySummary:
        # 转换为日期时间格式
    date_for_summary = datetime.datetime.strptime(date, "%Y-%m-%d").date()

        # 获取指定日期的信息
    summary_in_date = get_daily_health_summary(
        garmin, date_for_summary, date_for_summary
    )
        # 获取基线时期的信息
    past_period_start = date_for_summary - datetime.timedelta(days=7)
    past_period_end = date_for_summary - datetime.timedelta(days=1)
    summary_in_past_period = get_daily_health_summary(
        garmin, past_period_start, past_period_end
    )
        # 使用趋势和指标创建提示词
    prompt = build_llm_context_md(summary_in_date, summary_in_past_period)
        # 创建消息列表
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": prompt},
    ]

        # 调用大语言模型并验证响应
    response, reasoning = llm(messages, model, {"type": "json_object"})
    health_summary = DailySummary.model_validate(response)
    return health_summary

从现在开始，我们需要获取 DailyHealthSummary 时，只需调用 get_daily_summary 函数，传入日期和我们想使用的模型即可：

daily_summary = get_daily_summary(garmin, "2025-08-19", "deepseek-chat")
print(daily_summary.model_dump_json())

然后，它会返回我们构建的每日健康摘要：

{
   "day_type":"balanced",
   "title":"恢复改善、压力降低、平衡良好的一天。",
   "emoji":"⚖️",
   "observation":"静息心率下降 9.3%至 39，压力水平显著降低 58.8%，表明恢复极佳。身体电量起点高达 82，终点为 72，显示尽管睡眠和活动略有减少，但能量使用高效。",
   "recommendation":"通过持续适度运动和保证充足睡眠来维持这种平衡。专注于压力管理以保持低水平并支持持续恢复。"
}

如你所见，这效果相当不错。看这个例子，我可以告诉你它完全正确。这是我八月份假期的第一天，这使得摘要特别准确。我们现在有了第一个原型。但构建应用程序并不仅限于此。我们希望构建一个完整的服务，为任何感兴趣的人计算这些每日摘要。为此，我们需要构建一个 API。

重构为 API

既然我们的系统已经可以作为简单函数运行，是时候为它构建一个 API 了。目标是创建一个服务，能够接收用户的请求及其登录 Garmin API 的详细信息，并返回他们的健康指标的 DailyHealthSummary。为了实现这一点，我们将使用 FastAPI。

安全注意事项

Garmin 健康摘要 API 服务介绍

API 端点与认证

API 服务使用您的认证凭据（包括密码）来调用 Garmin Connect API。这种方法对于原型是可接受的，但在生产环境中，用户通常不愿与陌生服务共享明文密码。对于实际应用，建议考虑实现 OAuth 或使用官方 Garmin Connect 开发者计划：developer.garmin.com/gc-developer-program/activity-api。

我们的 API 包含两个端点。第一个是一个小型生活质量改进：当用户访问 /（根路径）时，他们会自动重定向到 /docs。这使得开发者（尤其是项目新手）能够更轻松快速地查找和探索可用端点，并理解预期输出：

from fastapi import FastAPI, Header, HTTPException
from fastapi.responses import RedirectResponse
app = FastAPI(title="Garmin Health Summary API")
@app.get("/")
async def root():
    return RedirectResponse(url="/docs")

应用实例化后，我们创建 HealthSummaryRequest，这是期望从用户获得的请求。如以下所示，我们期望他们提供一个日期，如果未提供，则默认为今天的日期。用户还可以选择指定两个模型之一（deepseek-chat 或 deepseek-reasoner）来创建摘要：

class Model(str, Enum):
    chat = "deepseek-chat"
    reasoner = "deepseek-reasoner"
class HealthSummaryRequest(BaseModel):
    date: str = Field(
        default_factory=lambda: datetime.date.today().isoformat(),
        description="Date in YYYY-MM-DD format, defaults to today",
        example=datetime.date.today().isoformat(),
    )
    model: Model = Field(
        default=Model.chat,
        description="Model to use for the summary",
    )

为了进行身份验证，用户必须在请求头中提供 Garmin 凭据。我们返回之前介绍过的同一个 DailySummary 请求：

@app.post("/health-summary", response_model=DailySummary)
async def get_health_summary(
    request: HealthSummaryRequest,
    garmin_email: str = Header(..., description="Garmin email address"),
    garmin_password: str = Header(..., description="Garmin password"),
) -> DailySummary:

get_health_summary 端点的核心部分运行着与之前看到的完全相同的逻辑，并额外添加了一些错误处理以防万一出错：

"""Get Daily Health Summary for a specific date.
    Args:
        request: Health summary request with date and model
        garmin_email: Garmin account email from header
        garmin_password: Garmin account password from header
    Returns:
        Daily Health Summary with AI-generated insights
    """
    try:
        garmin = get_garmin_client(garmin_email, garmin_password)
        summary = get_daily_summary(garmin, request.date, request.model)
        logger.info(
            f"Daily Health Summary API request completed successfully for {request.date}"
        )
        return summary
    except ValueError as e:
        logger.error(f"Invalid date format provided: {request.date}")
        raise HTTPException(
            status_code=400, detail=f"Invalid date format: {e}"
        )
    except Exception as e:
        logger.error(
            f"Failed to generate Daily Health Summary for {request.date}: {e}"
        )
        raise HTTPException(
            status_code=500, detail=f"Error generating summary: {e}"
        )

有了这一切，我们现在可以使用以下命令运行我们的 API：

$ uv run fastapi run 02-api.py

这将在您机器的端口 8000 上启动一个生产服务器。当您访问 http://0.0.0.0:8000 时，您将看到 API 页面已准备好供您交互：

图 5.2: Garmin 健康摘要 API 页面

您可以通过界面或命令行调用 API。如果您安装了 curl（curl.se/），以下是如何调用的示例：

$ curl -X 'POST' \
        'http://localhost:8000/health-summary' \
        -H 'accept: application/json' \
        -H 'garmin-email: your.email@example.com' \
        -H 'garmin-password: your-password' \
        -H 'Content-Type: application/json' \
        -d '{
    "date": "2025-08-19"
  }'

这将以 JSON 格式返回模型的响应：

{
   "day_type":"rest",
   "title":"Rest day with minimal activity and good recovery indicators.",
   "emoji":"😴",
   "observation":"Exercise minutes and steps were significantly below your baseline, suggesting a deliberate rest. Body battery started high and stress levels were low, indicating effective recovery from previous days.",
   "recommendation":"Continue with light activity or rest tomorrow to maintain this positive recovery state. Ensure adequate sleep to keep your body battery levels optimal for future training."
}

使用 Docker 部署

我们的服务的最后一个方面是部署。您可能通过 SSH 连接并运行 fastapi run 命令，甚至在本地运行。然而，在处理实时生产系统时，您可能会使用 Docker。

现在，您的 API 可以运行了。这很棒。但不幸的是，我们无法将您的计算机运送给用户。如果我们的应用将在生产中运行，它需要部署到云中的某个地方。这意味着我们的应用需要在另一台计算机上运行。我们如何确保那台计算机上也安装了所有需要的东西？在过去的十年里，答案一直是容器。

Docker 是业界一个成熟的工具，它使用容器来声明式地定义依赖关系，并在部署时保持应用程序隔离。简而言之，Docker 让您可以将应用程序及其所需的一切——代码、库和工具——打包到一个轻量级的容器中。这确保了它可以在任何机器上以相同的方式运行，从您的笔记本电脑到生产环境，而不会出现环境问题。尽管我们很乐意写一本关于 Docker 惊人与不那么惊人的事物的厚书，但这不是那本书。如果您不熟悉它并想学习基础知识，我们建议您从这里开始：www.docker.com/101-tutorial。

现在，这是我们的详细注释的 Dockerfile，它打包了我们日常健康摘要服务的所有依赖项。如果您有兴趣深入了解此处的一些选择，我们建议您深入 uv Docker 教程/演练，因为我们的 Dockerfile 很大程度上受其启发（docs.astral.sh/uv/guides/integration/docker/）：

# First, build the application in the `/app` directory
FROM ghcr.io/astral-sh/uv:bookworm-slim AS builder
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy
# Configure the Python directory so it is consistent
ENV UV_PYTHON_INSTALL_DIR=/python
# Only use the managed Python version
ENV UV_PYTHON_PREFERENCE=only-managed
# Install Python before the project for caching
RUN uv python install 3.12
WORKDIR /app
RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=bind,source=uv.lock,target=uv.lock \
    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    uv sync --locked --no-install-project --no-dev --no-group local
COPY . /app
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --locked --no-dev --no-group local
# Then, use a final image without uv
FROM debian:bookworm-slim
# Copy the Python version
COPY --from=builder --chown=python:python /python /python
# Copy the application from the builder
COPY --from=builder --chown=app:app /app /app
# Place executables in the environment at the front of the path
ENV PATH="/app/.venv/bin:$PATH"
# Expose the port
EXPOSE 8000
# Run the FastAPI application
CMD ["fastapi", "run", "app/02-api.py", "--host", "0.0.0.0", "--port", "8000"]

与 DeepSeek 模型交互

我们可以运行该应用程序，通过构建容器并在我们自己的机器上运行它，使用 docker build 和 docker run 命令。第一个命令构建一个名为 api 的镜像，使用 02-api.Dockerfile 作为 Dockerfile；第二个命令从该镜像启动一个容器，暴露端口 8000 并传递 DEEPSEEK_API_KEY 环境变量：

$ docker build -t api -f 02-api.Dockerfile .
$ docker run -it -e DEEPSEEK_API_KEY=sk-... -p 8000:8000 -t api

应用程序应再次在你的本地 URL 启动：localhost:8000。

内容很多，但你应该感到高兴。你创建了一个接收用户 Garmin 凭据并返回由 DeepSeek 驱动的每日健康摘要的服务。恭喜！现在，尽管我们很想向你展示如何在服务器上运行它，但本书不会这样做。容器的好处在于你可以将它们部署在任何地方。几乎所有云提供商在云上部署容器至少有 5 种方式，尽管 AWS 有 17 种不同方式：www.lastweekinaws.com/blog/the-17-ways-to-run-containers-on-aws/。

在休息之后，你会开始思考，虽然系统运行良好，但它并非特定于 DeepSeek。我们只是在调用 DeepSeek API。我们也可以调用任何其他 API（OpenAI，Anthropic 等）。然而，DeepSeek 的真正力量在于其开源本质。这为我们打开了一些大门。接下来我们将探索它们。

LiteLLM

LiteLLM 是一个库，允许我们使用单一统一格式轻松地与不同的 LLM 和提供商交互。

LiteLLM 的优势在于，当你更换提供商时，真正需要更改的只是你传递给 completion 函数的 model 字符串。

在以下示例中，我们演示如何使用 litellm 库通过三种不同的提供商使用 DeepSeek：

deepseek/deepseek-chat：使用 DeepSeek API 中的 DeepSeek chat 模型。
bedrock/us.deepseek.r1-v1:0：使用 Amazon Bedrock (aws.amazon.com/bedrock/) 运行 DeepSeek-R1 推理模型的推理。
openrouter/deepseek/deepseek-r1-distill-qwen-14b：使用 OpenRouter (openrouter.ai/) 运行 Qwen 14B 蒸馏模型。

警告

使用不同提供商时，你可能必须设置不同的 API 密钥和认证方法。请务必查看它们的文档以获取更多信息。内容非常丰富！

以下是你如何使用 litellm 的示例：

from litellm import completion

def litellm(messages: list[dict]) -> tuple[str, str | None]:
    response = completion(
        # model="deepseek/deepseek-chat", # 使用 DeepSeek API
        # model="bedrock/us.deepseek.r1-v1:0", # 使用 AWS bedrock 进行推理
        model="openrouter/deepseek/deepseek-r1-distill-qwen-14b", # 使用 OpenRouter 进行推理
        messages=messages,
        temperature=0.0,
    )
    message = response.choices[0].message
        # 如果存在推理内容，则提取出来
    if hasattr(message, "reasoning_content"):
        reasoning_content = message.reasoning_content
    else:
        reasoning_content = None
    return message.content, reasoning_content

# 使用示例：
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]
result, reasoning = llm(messages)
print(f"Result: {result}")
# Result: The capital of France is Paris.
print(f"Reasoning: {reasoning}")
# Reasoning: Okay, so I need to figure out the capital ...

litellm 是一个强大的工具，结合 OpenRouter，意味着你可以调用任何 API 和模型，甚至是本地模型！是的，我们说的是本地……但让我们一步步来。

使用 Ollama 在本地运行

2025 年 1 月发布的 DeepSeek-R1 震惊了世界，原因有很多。旗舰 R1 模型不仅在基准测试中表现出色，并与当时的最先进技术相媲美，而且这家中国实验室更进一步。你可能还记得我们在 第 1 章 和 第 2 章 中的讨论，除了旗舰模型，他们还发布了六个不同的蒸馏模型。

每个模型都是使用 DeepSeek-R1 模型生成的样本，从一个基础开源模型微调而来。原始模型包含 6850 亿参数，总权重为 641.33 GB。粗略地说，以全精度运行该模型大约需要 1540 GB 的显存。现在，你可能在后院拥有那种计算能力，但现实是很少有公司拥有那种计算能力，因为它需要多 GPU 设置。

但蒸馏版本则讲述了一个不同的故事。在 表 5.1 中，你可以看到蒸馏模型、它们的基础模型（从中微调而来）以及以 4 位量化运行它们所需的显存。这为我们这些在后院没有 GPU 集群的人打开了大门。其中一些可以在非常小的 GPU 上运行。事实上，我们甚至可以在单个 GPU 上运行它们！

| :--- | :--- | :--- | :--- |

| DeepSeek-R1 | - | 🤗 huggingface.co/deepseek-ai/DeepSeek-R1 | ~436 GB |

| DeepSeek-R1-Distill-Llama-70B | Llama-3.3-70B-Instruct (huggingface.co/meta-llama/Llama-3.3-70B-Instruct) | 🤗 huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B | ~46 GB |

表 5.1: DeepSeek 模型的显存需求

本地运行 LLM 有两个成熟的工具。第一个是 llama.cpp (github.com/ggml-org/llama.cpp)，第二个，使用第一个作为其推理引擎，是 Ollama (ollama.com/)。

使用 Ollama 运行本地模型很简单。安装 Ollama (ollama.com/download) 后，只需在终端中输入以下命令：

使用本地模型进行健康数据摘要

使用 Ollama 运行本地模型

$ ollama run deepseek-r1:1.5b
>>> Hi there!
Hello! How can I assist you today? 😊

此命令将下载蒸馏后的 1.5B 模型（如果你之前没有运行过），并启动一个聊天会话。根据你的系统情况，它可能会运行得比较热，但 1.5B 参数模型应该能在大多数现代计算机上运行。

我们可以使用 Ollama Python SDK (github.com/ollama/ollama-python) 来生成每日健康摘要。我们只需适配 get_daily_summary 函数，并利用我们刚下载的 1.5B 参数蒸馏模型：

from ollama import chat
def get_daily_summary(
    garmin: Garmin,
    date: str,
) -> DailySummary:
    prompt = get_daily_summary_prompt(garmin, date)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": prompt},
    ]
    response = chat(
        messages=messages,
        model="deepseek-r1:1.5b",
        format=DailySummary.model_json_schema(),
        options={
            "temperature": 0.0,
        },
    )
    return DailySummary.model_validate_json(response.message.content)

我们可以通过设置 format=DailySummary.model_json_schema() 来为 Ollama 指定返回格式。这会将 Pydantic 模型的 JSON schema 也传递给模型。我们可以这样调用函数：

garmin = get_garmin_client(
        email=os.environ["GARMIN_EMAIL"],
        password=os.environ["GARMIN_PASSWORD"],
    )
summary = get_daily_summary(garmin, "2025-08-14")
print(summary)

这会输出以下内容：

day_type=<DayType.HIGH_STRESS: 'high_stress'> title='Elevated stress and poor recovery despite minimal physical activity.' emoji='😫' observation='Stress levels jumped 46% above baseline while sleep dropped to just 6.1 hours, resulting in a low body battery start of 45\. Exercise and movement were minimal, suggesting stress is from non-physical sources.' recommendation='Focus on stress management techniques and aim for 8+ hours of sleep tonight. Consider light exercise like walking or yoga tomorrow, as gentle movement can help regulate stress levels.'

是的，今天状态不太好。但我们刚刚将本地的 1.5B 参数模型接入了我们的每日健康摘要服务！在继续之前，请记住，你也可以像我们之前展示的那样，使用 LiteLLM 来使用本地模型，只需将模型设置为例如 ollama_chat/deepseek-r1:1.5b。

使用 Transformers 和 XGrammar 在 CPU 上进行推理

Ollama 非常适合快速测试和试验本地模型。但要真正构建我们的每日健康摘要服务，我们需要对设备、提示结构、分词过程以及生成过程本身有更精细的控制。这就是我们更接近底层，利用成熟的 Transformers 库 (github.com/huggingface/transformers) 的地方。

要在 CPU 上运行我们的 1.5B 参数模型，让我们使用 Unsloth 版本的 1.5B 蒸馏模型，你也可以在 Hugging Face 上找到它。Unsloth (docs.unsloth.ai/) 是另一个值得关注的有趣库，如果你对微调和自己运行模型感兴趣的话。

我们首先用以下代码加载模型：

from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
model_name = "unsloth/DeepSeek-R1-Distill-Qwen-1.5B"
device = "cpu"
question = "What is the capital of Le Marche, Italy?"
max_tokens = 1200
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to(device)

我们对消息使用 apply_chat_template 将它们转换为模型输入：

messages =
[
    {"role": "user", "content": question},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

有了这些，我们现在可以直接从 CPU 流式输出 token：

streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=False)
model.generate(
    **inputs,
    max_new_tokens=max_tokens,
    streamer=streamer,
)
# prints Okay, so I need to figure out the capital of Le Marche, Italy. Hmm, I'm not super familiar with Italy, but I know it's one of the largest and most populous countries in Europe....

这使用 transformers 库直接在 CPU 上运行我们的模型。效果很好，但如果我们想将此模型与每日健康摘要服务的其余部分集成，我们需要确保模型能有效遵守我们的 DailySummary 格式。这正是开源模型真正闪光的地方。

记住，模型是本地运行的，所以我们对它生成的 token 拥有完全的控制权。我们可以编写一个函数来限制每个 token，使其符合我们定义的 JSON schema。但这正是 outlines (dottxt-ai.github.io/outlines)、xgrammar (github.com/mlc-ai/xgrammar) 和 guidance (github.com/guidance-ai/guidance) 等库开箱即用的功能。它们执行受限生成。它们不仅仅是让模型生成 JSON，而是明确地将 token 限制在符合 JSON 结构的范围内。

根据我们的经验，xgrammar 产生了最佳结果。使用它的代码有些复杂，但如果你有疑问，我们建议你查阅该库的文档。不过，我们还是会在高层次上解释一下。

代码的第一部分只是导入模型，将其放置在 CPU 上，并设置所需的种子（以获得可复现的输出）：

import xgrammar as xgr
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
MODEL_NAME = "unsloth/DeepSeek-R1-Distill-Qwen-1.5B"
MODEL = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.float32,
    device_map="cpu",
)
TOKENIZER = AutoTokenizer.from_pretrained(MODEL_NAME)
CONFIG = AutoConfig.from_pretrained(MODEL_NAME)
MAX_NEW_TOKENS = 1024
transformers.set_seed(42)

现在，我们将调整 get_daily_summary 函数以适配我们的 API。我们接收消息，但现在也将它们通过 apply_chat_template 处理：

def get_daily_summary(
    garmin: Garmin,
    date: str,
) -> DailySummary:
    prompt = get_daily_summary_prompt(garmin, date)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": prompt},
    ]
    texts = TOKENIZER.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

乍一看，代码可能令人生畏，但它实际上只做了四件事：

它首先用 TokenizerInfo.from_huggingface 提取分词器元数据，为正确解释 token 提供基础。
在生成过程中，它应用一个 xgrammar logits 处理器，强制模型仅生成在 DailySummary schema 下有效的 token。
生成步骤本身在较低的 temperature 并启用采样的情况下运行，在一致性和多样性之间取得平衡。
最后，输出通过 DailySummary.model_validate_json 进行解码和验证，返回一个干净的结构化 Pydantic 对象。
温度（Temperature）

temperature 参数控制模型输出的随机性。较低的值使响应更确定，较高的值提供更有创意的答案。实际上，temperature 通过操纵 DeepSeek 生成每个 token 的概率来实现这一点。

Patrick von Platen 有一篇关于使用 Transformers 进行文本生成的优秀教程，深入探讨了包括 temperature 在内的参数：huggingface.co/blog/how-to-generate。

这些部分协同工作，确保我们每次都能获得完全符合需求——即干净且有效的 JSON：

model_inputs = TOKENIZER(texts, return_tensors="pt").to(MODEL.device)
tokenizer_info = xgr.TokenizerInfo.from_huggingface(
    TOKENIZER, vocab_size=CONFIG.vocab_size
)
grammar_compiler = xgr.GrammarCompiler(tokenizer_info)
compiled_grammar = grammar_compiler.compile_json_schema(DailySummary)
xgr_logits_processor = xgr.contrib.hf.LogitsProcessor(compiled_grammar)
generated_ids = MODEL.generate(
    **model_inputs,
    max_new_tokens=MAX_NEW_TOKENS,
    logits_processor=[xgr_logits_processor],
    do_sample=True,
    temperature=0.01,
    top_p=0.95,
    top_k=50,
)
generated_ids = generated_ids[0][len(model_inputs.input_ids[0]) :]
model_response = TOKENIZER.decode(generated_ids, skip_special_tokens=True)
return DailySummary.model_validate_json(model_response)

我们为获得确定性输出而将 temperature 设置为 0.01，倾向于选择高概率的令牌。top_k=50 将每一步的选择限制为最可能的 50 个令牌，而 top_p=0.95 则将生成范围限定在覆盖 95%概率质量的令牌内，去除了不太可能的长尾部分。这些选择并非严格规定，因此可以自行尝试调整。

有了 xgrammar，我们现在可以将每日健康摘要服务重构为完全在本地运行。无需 API 调用，也不会出现 JSON 格式错误。

为本地 CPU 生成进行重构

既然我们已经替换了 get_daily_summary，就可以创建一个全新的、完全在本地运行的 API。在本书代码仓库的 Chapter05 文件夹中，我们提供了一个 05-api-cpu-xgrammar.py 文件。它看起来完全像我们设计的第一个模型服务，但运行 100%在本地。

要运行它，我们再次使用 FastAPI，命令如下：

$ uv run fastapi run 05-api-cpu-xgrammar.py

在另一个终端中，我们可以通过命令行调用此 API：

$ curl -X 'POST' \
        'http://localhost:8000/health-summary' \
        -H 'accept: application/json' \
        -H 'garmin-email: your.email@example.com' \
        -H 'garmin-password: your-password' \
        -H 'Content-Type: application/json' \
        -d '{
    "date": "2025-08-19"
}'

这再次返回了格式良好的 JSON，由 DeepSeek-R1-Distill-Qwen-1.5B 生成，100%在你自己的机器上运行：

{
   "day_type":"high_stress",
   "title":"Elevated stress and poor recovery despite minimal physical activity.",
   "emoji":"😫",
   "observation":"Stress levels jumped 46% above baseline while sleep dropped to just 6.1 hours, resulting in a low body battery start of 45\. Exercise and movement were minimal, suggesting stress is from non-physical sources.",
   "recommendation":"Focus on stress management techniques and aim for 8+ hours of sleep tonight. Consider light exercise like walking or yoga tomorrow, as gentle movement can help regulate stress levels."
}

Dockerfile 有 99%的相似度，只有少量依赖差异，为简洁起见，我们将其略过。

呼！干得漂亮！你刚刚创建了一个完全本地化的每日健康摘要服务。你学会了如何在 CPU 上运行 DeepSeek 模型，确保它们产生可靠的 JSON，并清晰地理解了如何部署它们以及如何与我们设计的服务进行整合。这内容相当多。

然而，需要注意的是，我们为 1.5B 参数模型设计的方案同样适用于任何其他 DeepSeek 模型，但有一个主要限制：你需要专门的硬件。你可能无法在 CPU 上运行比 1.5B 版本更大的模型。对于更大的模型，我们需要 GPU。你们中的一些人可能拥有强大的电脑，但现实是，更大的模型往往比小模型表现更好。

假设我们希望用 DeepSeek-R1 的 70B 参数蒸馏版来运行每日健康摘要；我们至少需要 46GB 的显存。我有一台配置相当高的 MacBook，拥有 64GB 统一内存，但事实是我从未能舒适地运行过超过 30B 参数的模型。而且我们不可能把我的 MacBook 寄出去。

你明白这个意思。我们需要在别人的机器上运行——也就是在云端。但我们仍然希望拥有本地模型的优势，即我们可以紧密控制生成过程。在下一节中，我们将探讨如何在云端部署一个更大的蒸馏模型。系好安全带，这将会很有趣！

使用 AWS 部署隔离的模型服务

亚马逊云科技（AWS）是全球最受欢迎的云计算平台之一。我们将重点介绍它，以便部署一个更大的 DeepSeek 模型，并将其与我们的每日健康摘要服务集成。和大多数云计算平台一样，实现同一目标的方式几乎有 25 种之多。AWS 也不例外。

要在 AWS 上使用/部署 DeepSeek 模型，你可以采取以下方式：

使用 Bedrock API，AWS 在此托管基础模型（包括 DeepSeek）。在这里，你可以像我们使用 DeepSeek API 一样调用和使用模型。你不会进行部署；你只是使用。
使用 Bedrock 自定义模型导入，这是 AWS 提供的一项服务，允许你将训练好的模型导入 Bedrock，然后通过 Bedrock API 使用它。模型需要是 Hugging Face 格式。请注意，仅支持某些架构（例如，不支持 Gemma 模型）。
Hugging Face 深度学习容器（DLC）也可用于在 AWS 上部署模型。用于推理时，它们包含 PyTorch 或文本生成推理（TGI, github.com/huggingface/text-generation-inference) 作为受支持的框架。如果你想使用这些框架，这是一个有趣的选择。你也可以使用 DLC 来训练模型，这将通过 AWS 的 SageMaker 服务完成。
自行构建：我们的意思是，你在云端购买一个带有 GPU 的虚拟机（即一个 EC2 实例），安装你需要的所有软件包，管理依赖项、访问、升级等等——拥有最大控制权，但也可能带来不少麻烦。
最后，通过 SageMaker 的大模型推理（LMI）容器允许你使用多种可用的推理后端来部署大语言模型。这是为更大的模型设计的，你需要完全控制，但又不想管理基础设施——接近底层，但又无需太多烦心事。LMI 容器由 Deep Java Library（DJL）框架提供支持。可以将 DJL 视为驱动这些容器的引擎。

这并非详尽无遗的列表。AWS 提供了更多选项，其中一些可能会持续多年，而另一些可能不会。在 AWS 上部署模型没有唯一正确的方式；但是，你应该在选择前权衡利弊。我们汇编了一张表格（表 5.2），列出了每个选项的优缺点以及何时使用：

| 选项 | 优点 | 缺点 | 使用场景 |

| :--- | :--- | :--- | :--- |

| Hugging Face DLC (SageMaker) | 托管式训练和推理。包含 TGI 用于优化的 LLM 推理。与 Hugging Face Hub 轻松集成。自动扩展端点。 | | |

部署选项对比

仅限于 Hugging Face、TGI 和 PyTorch 技术栈。
相比 DJL，对于非常大的模型优化较少。
仍需为 SageMaker 基础设施付费。

| 适用场景：您需要一条基于 Hugging Face 的原生路径，在 SageMaker 上进行训练或推理。 |

| :--- |

| 自定义在 EC2 上部署 |

| :--- |

| * 最大程度的控制。

可自由选择任何服务栈，例如 vLLM、TGI 和 Triton。
没有 SageMaker 或 Bedrock 的额外开销。 |

| * 您需要管理所有事情，包括扩缩容、安全和网络。

更难可靠地运行。
运维负担更高。 |

| 适用场景：您需要完全的控制权，或者想要尝试自定义的栈和内核。 |

| :--- |

| 大型模型推理（LMI/DJL） |

| :--- |

| * 专为大模型构建。

支持 vLLM、TensorRT-LLM 以及用于 Inf2 的 NeuronX。
管理基础设施、扩缩容和分片。
在大型模型上拥有最佳吞吐量。 |

| * 设置过程比 Hugging Face DLC 稍微复杂。

需要了解 SageMaker。
开箱即用时，“Hugging Face 魔法”较少。 |

| 适用场景：您需要为大型或生产级模型获得最大性能，同时无需管理基础设施。 |

| :--- |

表 5.2：AWS 上部署选项的比较

我们建议几乎总是从 API 调用开始，避免复杂的自托管，除非您知道您的产品确实需要它。所以，如果您正在构建产品，请从 API 调用开始。一旦您有了一百万用户，就该开始考虑部署了。本书中我们也“吃自己的狗粮”，这就是为什么我们从通过 API 使用 DeepSeek 模型开始。现在我们将深入探讨使用 LMI 容器部署 DeepSeek。这是因为我们相信 vLLM 是一个极其强大的推理引擎（github.com/vllm-project/vllm）。

您可能已经在问自己：到底什么是推理引擎？问得好。

推理后端

一个推理引擎是简化 LLM 部署过程的框架。它提供了打包、优化和服务能力。这样您就无需考虑 GPU 优化、请求管理、内存优化、LLM 架构、流式传输或对结构化生成的支持。推理后端提供了所有这些功能，因此您可以专注于使用模型。我们编译了另一份（非详尽的）推理引擎列表，见表 5.3：

| :--- | :--- | :--- | :--- | :--- | :--- | :--- |

表 5.3：流行推理引擎的比较

尽管每个引擎都有其优缺点，但 vLLM 已经逐渐确立了其作为主流引擎的地位。它还支持许多我们对于“每日健康摘要”服务感兴趣的功能：结构化生成、兼容 OpenAI 的 API、量化、对 DeepSeek 模型的一流支持，甚至支持推理输出！

使用 LMI 容器部署 DeepSeek

SageMaker 支持多种用于训练和推理作业的容器镜像。您可以在 GitHub 上的 AWS DLC 仓库中查看完整列表：github.com/aws/deep-learning-containers。

在这个生态系统中，我们将特别关注 LMI 容器，因为它们对 vLLM 提供了一流的支持。LMI 容器是 DJL 的一部分并由其文档化：一个用于深度学习的开源 Java 框架。我们理解这有点令人困惑。

关键在于，如果我们对将要部署的容器的文档感兴趣，可以在 docs.djl.ai/ 找到。

我们需要首先弄清楚的是模型所需的总内存。我们将专注于为我们的用例部署 DeepSeek-R1 Distill Qwen 14B。为此，我们将使用一个 ml.g6.12xlarge AWS 实例。该实例将为我们提供大约 96 GB 的 GPU 显存，这应该足以运行我们的模型。

警告

在此部署中，我们将使用一台 ml.g6.12xlarge 机器。根据区域的不同，它每小时的成本约为 5 美元。忘记关闭它，您将被收取每月 3,720 美元的费用！始终确保在测试后关闭资源。特此警告。

在表 5.4中，我们提供了一些关于为不同流行开源模型选择哪种机器的指导：

| :--- | :--- | :--- | :--- |

| ~7B | ml.g5.4xlarge，ml.g6.4xlarge | 24 GB | Llama 2 7B，Mistral 7B，DeepSeek-LLM 7B，DeepSeek-R1 Distill Qwen 7B |

| ~13B | ml.g5.12xlarge，ml.g6.12xlarge | 96 GB | Llama 2 13B，Code Llama 13B，Qwen2.5 14B，DeepSeek-R1 Distill Qwen 14B |

| ~20B | ml.g5.12xlarge，ml.g6.12xlarge | 96 GB | GPT-OSS 20B，Mistral Small 24B，Solar Pro 22B |

| ~35B | ml.g5.48xlarge，ml.g6.48xlarge | 192 GB | Code Llama 34B，Falcon 40B，Yi 34B，DeepSeek-R1 Distill Qwen 32B |

| ~70B | ml.g5.48xlarge，ml.g6.48xlarge | 192 GB | Llama 2 70B，Code Llama 70B，DeepSeek-R1 Distill Llama 70B，DeepSeek-R1 Distill Qwen 70B |

| ~70B | ml.p4d.24xlarge | 320 GB | - |

| ~180B | ml.p4de.24xlarge，ml.p5.48xlarge | 640 GB | Falcon 180B，GPT-OSS 120B，DeepSeek V2 236B (MoE)，DeepSeek V3 671B (37B active) |

表 5.4：流行开源模型的实例类型选项

为了部署我们的模型，我们将利用 SageMaker 的 Python SDK（sagemaker.readthedocs.io/）。以下脚本创建了部署 DeepSeek 模型的通用配置。您可以将模型替换为您感兴趣的模型，并根据前面的速查表调整机器配置（同时考虑成本）。在运行之前，您应确保以下几点：

您有一个启用了计费的 AWS 账户并已安装 AWS CLI。如果您还没有 CLI，可以在此处获取：docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html。
您已创建了一个附加了 AmazonSageMakerFullAccess 策略的 IAM 角色，并设置了以下信任关系以便 SageMaker 可以扮演该角色（创建后请务必获取该角色的 Amazon 资源名称 (ARN))：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "sagemaker.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

SageMaker Python SDK 已包含在本项目的依赖项中，因此无需单独安装。

您可能需要为所使用的实例类型请求配额增加。对于以下脚本，您需要至少一个 ml.g6.12xlarge 实例的服务配额。AWS 文档中有关于如何操作的良好说明：docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html。

如何部署您的模型

以下是部署模型的方法：

import boto3
import sagemaker
import os
import time

# setup session
boto_session = boto3.session.Session()
region = boto_session.region_name
sess = sagemaker.Session()

# create our configuration
CONFIG = {
    "INSTANCE_TYPE": "ml.g6.12xlarge", # the instance type
    "ENV": {
        "HF_MODEL_ID": "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",  # hugginface model id
        "TENSOR_PARALLEL_DEGREE": "max",  # set max to use all GPUs
        "OPTION_ENABLE_REASONING": "true", # enable reasoning parsing
        "OPTION_REASONING_PARSER": "deepseek_r1", # set reasoning parser
    },
    "IMAGE_NAME": "djl-inference:0.33.0-lmi15.0.0-cu128", # image
    "BEDROCK_ROLE_ARN": "arn:aws:iam::ACCOUNT_NUMBER:role/replace-with-yours", # role
}

# get base model name
ts = time.time()
hf_model_id = CONFIG["ENV"]["HF_MODEL_ID"]
model_name = hf_model_id.split("/")[-1]
base_model_name = sagemaker.utils.name_from_base(model_name)

# create SageMaker model
current_region = os.environ["AWS_DEFAULT_REGION"]
image_name = CONFIG["IMAGE_NAME"]
inference_image_uri = (
    f"763104351884.dkr.ecr.{current_region}.amazonaws.com/{image_name}"
)
lmi_model = sagemaker.Model(
    image_uri=inference_image_uri,
    env=CONFIG["ENV"],
    role=CONFIG["BEDROCK_ROLE_ARN"],
    name=base_model_name,
)
endpoint_name = f"{base_model_name}-endpoint"

# print details
print(f"{base_model_name=}")
print(f"{current_region=}")
print(f"{inference_image_uri=}")
print(f"{endpoint_name=}")

# deploy model (this can take up to 15 mins!)
lmi_model.deploy(
    initial_instance_count=1,
    instance_type=CONFIG["INSTANCE_TYPE"],
    container_startup_health_check_timeout=900,
    endpoint_name=endpoint_name,
)
print(f"Model has been deployed to endpoint {endpoint_name}")

模型的部署（取决于您的区域和实例可用性）可能需要长达 15 分钟。完成后，当您导航到 AWS 控制台时，会注意到一个新的模型和端点刚刚被创建：

图 5.3: 我们在 AWS 上创建的模型

图 5.4: SageMaker 端点的仪表板

模型部署完成后，我们可以导航到 Amazon SageMaker AI 控制台。点击 Inference 部分，可以导航至 Inference > Models，您将看到新部署的 DeepSeek 模型。导航至 Inference > Endpoints 将显示您的模型端点，以及一组供您监控端点状态（CPU 利用率、内存利用率、GPU 内存、GPU 利用率以及调用/请求指标）的指标。

危险

完成模型端点使用后，请记得将其关闭，否则实例开启期间您将继续被收费！为此，您可以在 AWS 控制台删除端点或使用 SageMaker SDK：

import sagemaker
sess = sagemaker.Session()
sess.delete_endpoint(endpoint_name)
sess.delete_endpoint_config(endpoint_name)
lmi_model.delete_model()

LMI 容器 API 格式与结构化输出

LMI 容器支持三种不同的 API 模式：OpenAI 的 Chat Completions 格式、OpenAI Completions 格式和 TGI 格式。如果我们希望进行预测，以下三个示例实际上是等效的：

llm = sagemaker.Predictor(
    endpoint_name=endpoint_name,
    sagemaker_session=sess,
    serializer=sagemaker.serializers.JSONSerializer(),
    deserializer=sagemaker.deserializers.JSONDeserializer(),
)
TEMPERATURE = 0.01
MAX_TOKENS = 1024
USER_MESSAGE = "What is the capital of Le Marche, Italy?"

# 1. OpenAI chat completions
llm.predict(
    {
        "messages": [
            {
                "role": "user",
                "content": user_message,
            }
        ],
        "temperature": TEMPERATURE,
        "max_tokens": MAX_TOKENS,
    }
)

# 2. OpenAI Completions format
llm.predict(
    {
        "prompt": user_message,
        "temperature": TEMPERATURE,
        "max_tokens": MAX_TOKENS,
    }
)

# 3. TGI format
llm.predict(
    {
        "inputs": user_message,
        "parameters": {
            "max_new_tokens": MAX_TOKENS,
            "temperature": TEMPERATURE,
        },
    }
)

然而，正如我们之前提到的，选择 LMI 容器的一个重要原因是它支持 vLLM 作为推理引擎以及结构化输出的各种选项。事实上，LMI 容器支持 Chat Completions 格式因为 LMI 容器使用 vLLM 作为推理引擎。使用 vLLM 进行结构化生成的一个选项是 xgrammar (docs.vllm.ai/en/v0.8.1/features/structured_outputs.html)。

接下来是一个名为 llm 的函数示例，该函数将使用我们的 SageMaker 端点和结构化输出，返回符合我们传递给它的 response_model 的预测。如您所见，vLLM 支持 response_format 和一个 extra_body 参数，我们可以在此指定希望其遵守的 Pydantic 模型以及希望使用的解码后端：

def llm(
    messages: list[dict],
    endpoint_name: str,
    response_model: BaseModel,
) -> BaseModel:
    client = get_aws_llm(endpoint_name=endpoint_name)
    response = client.predict(
        {
            "messages": messages,
            "temperature": 0.01,
            "max_tokens": 1024,
            "response_format": {
                "type": "json_schema",
                "json_schema": {
                    "name": response_model.__name__,
                    "schema": response_model.model_json_schema(),
                    "strict": True,
                },
            },
            "extra_body": {"guided_decoding_backend": "xgrammar"},
        }
    )
    return response_model.model_validate_json(
        response["choices"][0]["message"]["reasoning_content"]
    )

然后我们可以通过提供一个 Pydantic 模型和消息列表来使用此函数，如以下示例：

class Capital(BaseModel):
        capital: str
        population: int

messages = [
        {
                "role": "user",
                "content": "What is the capital of France?",
        }
]

response = llm(
        messages,
        "DeepSeek-R1-Distill-Qwen-14B-2025-08-24-08-52-31-391-endpoint",
        response_model=Capital,
)
print(response)
# capital='Paris' population=2161512

因此，现在我们已在 AWS 上部署了大型 DeepSeek 模型，并拥有一个支持 vLLM 推理引擎任何功能的端点。剩下的唯一步骤是将其与我们的每日健康摘要服务再次集成。由于我们正在对 AWS 进行 API 调用，这应该非常简单。

将我们的服务更新为使用 Amazon SageMaker 端点

要将我们的服务更新为使用在 AWS 中部署的模型，我们只需替换之前构建的 llm 函数即可。在章节材料中，我们包含了一个 07-api-deepseek-sagemaker.py API，可通过以下方式运行：

最佳实践和建议

我们的 API 后端现在是 Amazon SageMaker，但其他一切保持不变：请求格式、命令和用户体验。这展示了结构化输出的优势。你可以将后端切换到另一个服务，而不会干扰用户的工作流或破坏 API。事实上，我们也可以同时构建一个服务于三个不同后端（DeepSeek API、CPU 推理和 Amazon SageMaker）的三端点 API——我们就把这个留作练习吧。

在本章中，通过构建每日健康摘要服务，我们探索了使用 DeepSeek 模型的三种主要方式：调用 API、在本地 CPU 上运行蒸馏模型，以及使用 AWS、LMI 容器和 vLLM 部署更大的模型。这些只是冰山一角。开源的好处在于，如果需要，你还有更多的选择可以探索。

话虽如此，很少需要自己部署 DeepSeek 模型。自托管能让你拥有控制权和隐私性，但也意味着额外的工作和成本。你必须管理基础设施、监控性能并确保可靠性。对于大多数情况，尤其是在测试想法或验证产品时，从 API 开始几乎总是最佳选择。保持简单，直到你确定需要更多。

在决定是否自己托管模型时，可以参考这个快速清单：

成本：比较 API 使用成本与自部署成本。它们相差多少？
延迟：响应需要多快？你的用户能接受吗？
可扩展性：你的设置能否应对峰值使用量？如果用户一夜暴增怎么办？你还能为他们服务吗？
稳健性：你的设置需要多稳健？你有负责基础设施的团队吗？
灵活性：你能轻松地将应用程序从 API 调用演进到自托管，反之亦然吗？

我们已经展示了如何从 API 开始，转向 CPU 推理，并最终部署更大的模型。由于我们的 API 格式保持一致，用户不会注意到变化。这是一种经过验证的演进方式，不会破坏现有系统。

总结

在本章中，我们带你从最初的想法到一个完全可用的服务，该服务用 DeepSeek 驱动的动态洞察取代了无聊的智能手表摘要。我们首先展示了如何使用 DeepSeek API 快速构建一个简单的原型，以满足我们的初始需求。然后，我们探索了使用蒸馏版本在本地运行模型，并构建了一个轻量级的、基于 CPU 的服务，无需 GPU 即可运行。

接下来，我们专注于灵活性，展示了LiteLLM等库如何让你无需修改应用代码即可无缝切换后端提供商。最后，我们通过使用 LMI 容器和vLLM在 AWS 上部署 LLM 来扩展规模，从而在保持控制的同时能够部署更大的模型。

至此，你已经掌握了将任何 DeepSeek 驱动的应用推向生产环境的工具集，无论你是想保持小而高效、通过 API 运行，还是在云端部署更大的模型。

但今天许多成功的应用程序并非由单个 LLM 驱动，而是由多个模型协同工作，这通常被称为智能体。这正是我们将在下一章中探讨的内容。

| | |

|---|---|

| ### 获取本书的 PDF 版本及独家附加内容 | |

| 扫描二维码（或访问packtpub.com/unlock）。按书名搜索本书，确认版本，然后按照页面上的步骤操作。 | |

注意： 请备好您的发票。直接从 Packt 购买无需提供发票。

6 使用 DeepSeek 构建智能体

在第 4 章中，我们讨论了如何使用 DeepSeek 构建简单的应用程序。在本章中，我们将讨论如何使用 DeepSeek 构建 LLM 驱动的智能体。在撰写本文时，智能体是一个非常热门的话题。许多公司和工程师都在急于构建所谓的智能体应用。近年来出现了大量的智能体产品。一些流行的例子包括 OpenAI 的深度研究（openai.com/index/introducing-deep-research/）和 Google 的 Gemini 深度研究产品（gemini.google/overview/deep-research），其中 LLM 会动态地为你搜索网络，并生成一份访问了多个网站和资源的报告。如果你从事软件工作，可能已经使用过一些智能体系统：Claude Code（docs.anthropic.com/en/docs/claude-code/overview）和 Cursor（www.cursor.com/）就是很好的智能体驱动产品的例子。

除此之外，用于构建智能体的库和框架也在不断增多。CrewAI（www.crewai.com/）、OpenAI 的 swarm（github.com/openai/swarm）和 LangChain 的LangGraph（langchain-ai.github.io/langgraph/）等库每天都在演进。在撰写本文时，围绕智能体的库和概念非常新，并且在不断演变。为了让这本书经得起时间的考验，我们将专注于那些可能不会改变的智能体核心原则和定义。我们还将向你展示如何利用 DeepSeek 模型来构建这些构件。

在本章中，我们将涵盖以下主要主题：

智能体简介
工具
理解模型上下文协议
智能体与工作流
探索各种智能体系统

技术要求

下一章包含三个不同的 Jupyter 笔记本（docs.jupyter.org/en/latest/）。你需要安装的唯一软件是uv。安装uv，请按照此处的说明操作：docs.astral.sh/uv/getting-started/installation/。

智能体简介

在我们深入探讨如何使用 DeepSeek 构建智能体之前，我们应该先尝试定义什么是智能体。人们已经多次尝试定义智能体。OpenAI 说智能体是“代表你独立完成任务的系统”。

在最基本的层面上，基于 LLM 的智能体是一个自主系统，它能够接受一个通用任务并独立操作来完成它。一个任务可以是任何事情：创建一个关于某个主题的报告、创建一个解决此 GitHub 问题的拉取请求，或与某个对手下国际象棋。需要注意的是，智能体并非在真空中运行。智能体被设计为与环境进行交互。这个环境可能是一个棋盘、互联网（如果智能体需要搜索它），或者一个 GitHub 仓库（在智能体要向代码库提交补丁的情况下）。

但智能体需要一种在环境中交互和执行动作的方式。例如，一个下国际象棋的智能体需要能够读取/理解棋盘，一个研究智能体需要能够搜索互联网，而一个玩宝可梦的智能体（www.theverge.com/news/619482/anthropics-claude-ai-is-playing-pokemon）则需要能够与游戏进行交互。

为了与环境交互，智能体会使用我们称之为 工具 的组件。工具使智能体能够在环境中执行操作。例如，一个编程智能体需要能够读取代码库并为其提交补丁。在这种情况下，你可以想象智能体会使用类似 read_code_base 和 submit_patch 的工具。例如，DeepSeek 就能通过其函数调用功能来调用这些工具（api-docs.deepseek.com/guides/function_calling）。

总结一下，构成智能体的三个主要组件是：

任务：智能体希望为用户完成的目标。
环境：智能体为了完成任务而运行的空间。
工具：智能体可以在环境中采取的一组动作，以完成目标。

举例来说，假设我们要构建一个能向代码库提交改进的智能体。任务可能是解决我 GitHub 仓库中的 issue #45634，环境是包含代码的 GitHub 仓库，工具可能包括 read_file、search_repository 和 submit_patch。另一个例子是客户支持智能体。任务可能是解决客户的任何问题，环境可能是与用户的对话（很可能是一个聊天界面），工具则可能包括 search_knowledge_base（搜索知识库）、escalate_ticket（升级工单），甚至 mark_issue_as_resolved（将问题标记为已解决）。

智能体与普通基于 LLM 的应用程序相比，一个重要的特点是智能体旨在自主行动。这意味着智能体可以异步地与环境交互，而无需与分配任务的用户进行明确的交互。然而，我们应当注意，智能体与人类配合时可以变得特别强大。例如，智能体可使用的工具之一可能是向人类专家寻求指导的能力。

你一定还听说过其他特点，比如 记忆 和 规划。记忆可能与两件事有关。

第一种是 短期记忆，通常与你与智能体直接进行的对话历史（或消息序列）有关。例如，你可能会告诉智能体 始终用葡萄牙语回复，那么智能体在本次对话的剩余部分中将会/应该记住这一点。
第二种记忆是 长期记忆。长期记忆可能涉及我们希望智能体在任务之间持续跟踪的概念。我们可以为长期记忆构建非常复杂的东西。但也可以简单地创建两个工具：read_memory 和 update_memory。这将允许智能体检索过去的记忆并更新新记忆。没必要把事情搞得太复杂。

至于 规划，你可以将其类比为 思维链：要求智能体在解决问题之前概述一个计划。这是一个有趣的概念，但不是此处的重点。

记忆和规划——尽管有趣——在构建智能体时并非关键。相反，让我们转向智能体最重要的方面之一：工具。

工具

工具是构建智能体的最关键部分之一。工具调用 使智能体能够与环境交互并采取行动。然而，需要注意的是，我们并非一开始就拥有工具调用。在 LLM 的早期阶段（也就不久之前），我们使用简单的提示词来让 LLM 与外部世界交互。

这是一个例子：

Question: What is the capital of France?
Thought: I should look this up on Wikipedia.
Action: wikipedia: France
PAUSE
Observation: France is a country. The capital is Paris.
Answer: The capital of France is Paris.

在前面的例子中，LLM 输出一个 动作（例如，wikipedia: France）。系统（我们）看到这个动作，执行搜索，并将结果反馈给 LLM。这种方法虽然原始，但确实有效。这是当时许多热门论文背后的核心机制，比如来自 Google 的 ReAct 论文（react-lm.github.io/）。

随着发展，这种方法显得脆弱，LLM 模型开始具备了原生的函数（或工具）调用能力。

在本章中，我们将交替使用 函数 和 工具 这两个术语。

在下一代 LLM 中，我们可以更详细地描述模型可以使用的工具，以及该特定工具可以接受哪些参数。然后模型可以直接通过工具调用来响应。这与之前的原始方法非常相似，但由于模型是专门针对此类任务训练的（使用强化学习等技术），因此它更加健壮，减少了出错的可能性。

例如，我们可以通过向 LLM 提供 JSON 模式（schema）来指定一个名为 get_weather 的工具：

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the weather for a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"}
                },
                "required": ["location"]
            },
        }
    },
]
messages = [{"role": "user", "content": "How's the weather in Hangzhou?"}]
response = client.chat.completions.create(
    model="deepseek-chat",
    messages=messages,
    tools=tools,
)

模型会响应它想要调用的函数，以及它打算传递给该函数的参数。

{
  "tool_calls": [
    {
      "function": "get_weather",
      "arguments": {
        "location": "Hangzhou"
      }
    }
  ]
}

必须注意，模型不会执行工具——执行的是你，工程师。模型只是告诉我们它希望我们执行什么。你可以看到这种方法为什么稍显健壮。我们可以利用 JSON 模式来定义复杂的工具。

工具调用在当今非常流行。有些模型是专门为工具调用训练的，例如 Salesforce 的 xLAM 系列模型，它们是专门针对工具使用而训练的。伯克利函数调用排行榜（gorilla.cs.berkeley.edu/leaderboard.html）对不同模型调用不同工具的能力进行了排名。在撰写本文时，DeepSeek V3 和 DeepSeek-R1 分别位列第二十二和第二十七位，虽然不是函数调用的最佳模型，但在构建工具使用方面表现相当不错。

在我们开始构建你的智能体和工作流之前，让我们来谈谈一个越来越流行的东西：模型上下文协议。

理解模型上下文协议

2024 年 11 月，Anthropic 发布了 模型上下文协议（MCP）。当时，许多人刚开始用 LLM 构建应用程序。LLM 仍然在某种真空中运行。许多公司都在构建自己的工具接入 LLM，这往往导致相同的逻辑被反复重新实现。

例如，假设你是一家公司，正在构建一个将 LLM 应用与 Gmail 集成的系统。这将导致许多公司各自创建自己版本的 read_inbox 和 compose_email 函数。如果我们每次想将 LLM 连接到新系统时都必须开发一个新工具，这将使扩展变得非常困难。MCP 正是为了解决这个挑战。

MCP 是一个旨在简化此过程的开放标准。该协议旨在建立一个模式，让 LLM 可以连接到外部世界。它定义了 MCP 客户端和 MCP 服务器。

客户端是消耗 MCP 服务器资源的应用程序。MCP 服务器暴露三种类型的资源：（1）工具，（2）提示词，（3）资源——每种都在下面的图表中进行了说明。

图 6.1: MCP 服务器示意图（灵感来自 Anthropic https://www.youtube.com/watch?v=kQmXtrmQ5Zg ）

这在实践中意味着什么？这意味着，如果你希望将你的大语言模型连接到 Google Drive，你可以利用别人（可能是 Google）已经构建好的现成 Google Drive MCP，而不必自己构建。

如果到目前为止听起来都有点抽象，让我们来点更具体的。

我们来创建一个自己的小型 MCP 服务器。在下面的例子中，我们使用 FastMCP (gofastmcp.com) 框架来创建一个 MCP 服务器。这个服务器提供两个功能：(1) 一个获取城市天气的工具，和 (2) 一个返回给定用户状态的资源：

from fastmcp import FastMCP
mcp = FastMCP(name="MyAssistantServer")
@mcp.tool
def get_weather(city: str) -> str:
    """Multiplies two numbers."""
   return f"The weather in {city} is sunny with a high of 25°C."
@mcp.resource("users://{user_id}/profile")
def get_profile(user_id: int):
    return {"name": f"User {user_id}", "status": "active"}
if __name__ == "__main__":
    mcp.run()

定义好我们的小型 MCP 服务器后，我们可以将其保存为 mcp_server.py。如何将这些 MCP 工具接入我们的 DeepSeek 大语言模型呢？我们可以参考 DeepSeek 文档中的一个示例。在其官方函数调用文档 (api-docs.deepseek.com/guides/function_calling) 中，为了让模型能够使用工具，我们需要首先定义其 JSON schema，如下所示：

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get weather of an location, the user should supply a location first",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    }
                },
                "required": ["location"],
            },
        },
    },
]

或者，我们可以通过 MCP 来获取这些工具。我来介绍一下如何操作。首先，我们定义一个服务器配置：

# Alternatively, we can use the MCP server to fetch tools
mcp_server_config = {
    "mcpServers": {
        "assistant": {
            "command": "uv", "args": ["run", "./mcp_server.py"]}}
}

接下来，我们创建一个函数，该函数使用 FastMCP 的 Client 连接到我们的 MCP 服务器并获取工具列表：

async def fetch_async_tools() -> list:
    async with Client(mcp_server_config) as mcp_client:
        return await mcp_client.list_tools()

最后，我们可以将所有这些逻辑包装在一个 get_tools 函数中，在那里我们也调整我们的字典以符合 DeepSeek 的期望格式。

def get_tools() -> list:
    """Fetch and format the list of tools from the MCP server."""
    mcp_tools = asyncio.run(fetch_async_tools())
    return [
        {
            "type": "function",
            "function": {
                "name": tool.name,
                "description": tool.description,
                "parameters": tool.inputSchema,
            },
        }
        for tool in mcp_tools
    ]

正如你所见，给大语言模型提供工具访问权限，既可以通过自己定义这些工具来实现，也可以通过连接到现有的 MCP 服务器来实现。对于目的明确的应用，我认为你更希望自己定义工具。通常，如果我们使用过多的工具，模型往往会变得困惑，不知道该用哪个以及为什么用。自己定义工具能让你对模型要完成的任务有更强的掌控力。

许多智能体框架，如 OpenAI Agents SDK、LangGraph 和 Pydantic AI，都支持将你的大语言模型连接到 MCP 服务器，并为你省去我刚刚演示的样板代码。然而，在本章中，我们更愿意让你能精细地控制和理解底层不同的工具（或机制）；因此，我们不会使用那些框架。

MCP 的受欢迎程度急剧上升，甚至催生了其他协议，例如 Google 的 Agent2Agent (A2A) 协议 (github.com/a2aproject/A2A)。MCP 真正流行起来主要是在面向消费者的应用中。例如，你可以将你的 Claude 桌面应用程序连接到你的 Gmail MCP，或者将你的 VSCode IDE 连接到你公司的 MCP 以读取文档。

在撰写本文时，MCP 和 A2A 都是非常新的协议。只有时间才能证明它们是否会成为真正的标准，还是被其他东西取代。我们确实认为大语言模型和工具调用会存在很长时间。因此，为了向你解释这些机制在底层是如何工作的，我们现在将深入探讨智能体和工作流。

使用智能体和工作流

在我们开始构建智能体之前，亲爱的读者，我有一个坏消息告诉你。大多数智能体目前还不太好用。原因有两个。

第一个是复合错误。想象一下，我们给智能体安排了一个具有挑战性的任务。这个任务需要精心编排一系列步骤。假设完成这个任务需要五个仔细的步骤。如果每一步都有 90% 的成功概率，那么所有步骤都成功完成的概率就骤降到可怜的 60%。想象一下，你有一个函数只有 60% 的时间能正常工作。你会把它投入生产吗？

智能体——以及更广泛地说，大语言模型——经常失败的第二个原因是上下文腐烂问题。大语言模型具有有限的上下文窗口。我们添加给它们的每一个工具和指令都会填满它们的上下文。研究表明，我们填满上下文越多，大语言模型的性能就越差 (research.trychroma.com/context-rot)。这意味着，我们给智能体的工具和指令越多，它们最终的表现就越差。这两个因素共同作用使得智能体变得脆弱。解决方案是采用更可控的方法。

2024 年 12 月，Anthropic 发布了一篇非常有趣的博文，名为《构建有效的智能体》(www.anthropic.com/engineering/building-effective-agents)。在文中，他们对智能体和工作流做了重要区分。鉴于每天都在演变出众多不同的智能体定义，根据我们的经验，Anthropic 的这种区分在思考构建由智能体驱动的应用时非常有效。智能体和工作流之间的区别以及工作流的结构直接启发了我们处理本章概念的方式。

一个智能体（灵感来自最初的 ReAct 智能体）有一个目标、一个环境和一套工具。在下面的图中，你可以看到一个增强型大语言模型的示例，这是一个带有工具的大语言模型。工具可以是搜索网络的能力、写入某种记忆的能力等等。它还有一个与之交互的环境。

图 6.2：带有工具的增强型大语言模型

在这个场景中，我们让大语言模型自动选择正确的工具来达到其目标。而不加以引导。这意味着控制流是动态的。换句话说，我们期望智能体做出正确的决策来达成其目标。这可能被证明是脆弱的。智能体可能会陷入循环，可能永远无法达到目标，等等。

与智能体（agent）相对的是工作流（workflow）。在工作流中，整体系统的控制流和方向是预先定义好的。工作流是一种模式，我们预先（在一定程度上）规定了 LLM 将要采取的行动和使用的工具。你可以将工作流看作一种模式，或是围绕增强型 LLM 的某种脚手架，我们预先决定了希望 LLM 采取的行动或流程。这让我们重新获得了对智能体所需行动的控制权，当行动集相对简单时，这可能是一个绝佳的选择。

表 6.1 强调了智能体与工作流之间的一些差异。

| 方面 | 工作流 | 智能体 |

| --- | --- | --- |

| 结构 | 预定义且顺序执行 | 动态且自主 |

| 灵活性 | 最适合可预测、可重复的任务 | 适用于复杂、开放性问题 |

| 控制 | 由开发者编写的代码路径驱动 | LLM 基于当前上下文决定行动 |

| 工具使用 | 固定顺序或配方 | 由智能体根据需要选择和排序 |

| 可预测性 | 高度可预测 | 可变；可基于反馈进行调整或重试 |

| 主要用例 | 定义明确的自动化、文档流程 | 编程智能体、交互式助手、任务规划器 |

表 6.1：工作流与智能体之间的差异（改编自 Anthropic 的《构建有效智能体》博文）

需要指出的是，智能体和工作流都是智能系统，都具有不同程度的自主性。在某种程度上，你可以把智能体想象成一辆纯粹的自动驾驶汽车，里面没有人，没有东西控制它；而工作流则是一辆有司机（你）的自动驾驶汽车，确保它做正确的事情。未来我们可能会使用大量的智能体，但在可能的情况下，我们应该选择控制，使用工作流。

在本章中，我们将设计三个不同的、由 DeepSeek 模型驱动的智能体系统。其中两个是工作流，一个是纯粹的智能体。我们将在各自的章节中逐一介绍。本节的目标是让你理解如何使用 DeepSeek 构建智能体系统，并能够将其扩展到你自己的用例。我们不会专注于特定的库或工具。我们相信，尽管其中一些库功能非常强大，但它们往往会忽视智能体系统的重要方面——尤其是在学习阶段。

这里有一些关于智能体的有趣资源供你阅读：

现在，让我们来构建一些示例工作流和智能体。

探索各种智能系统

在本节中，我们将讨论三种不同的智能体系统。我们将从一个评估者-优化器工作流开始，通过总结科学论文来展示一个示例。第二个工作流是编排者-工作者工作流，我们将展示如何生成关于某个主题的详细报告。最后，我们将介绍一个纯粹的工具调用智能体，你将看到一个动态工作流的实际运作。

工作流：评估者-优化器

我们将介绍的第一个智能工作流是评估者-优化器工作流。这个工作流的想法是让第一个 LLM 生成一个响应（即生成器），然后让第二个 LLM 评估第一个 LLM 的响应（即评估者）。评估者 LLM 评估响应，并持续向第一个 LLM 提供反馈，以便它可以迭代地改进其生成。

关键思想是建立一个受控的循环，我们可以在其中迭代地精炼对特定目标的生成。当你有一个大致的起始方向，并且对优秀结果的标准也有很好的了解时，这个工作流是理想的。例如，你可能想为一份工作申请生成一封求职信，但同时也希望它遵循严格的准则（例如，一定数量的段落，特定的语气）。另一个例子是代码生成。你可以从定义一个函数开始，并建立清晰的标准（代码行数、时间复杂度、单元测试），希望最终结果能遵守这些标准。

图 6.3：评估者-优化器工作流

虽然听起来可能很简单，但这个工作流在推理型 LLM（如deepseek-reasoner）的情况下特别有效。这是因为我们不仅向生成器 LLM 提供反馈，还提供该反馈背后的推理过程。如果这听起来很抽象，别担心。我们接下来将构建一个这个模式的示例。

示例：总结 arXiv 论文

arXiv.org (arxiv.org/) 是一个用于发布和上传科学论文的热门平台。它在机器学习领域尤为流行。尤其是如今，每天都有数百篇论文发表。arXiv 允许任何人发布和阅读开放获取的论文。它在相邻领域（如生物领域的 bioRxiv (www.biorxiv.org/)）也日益普及并激发了类似的理念。

在这个示例中，我们将构建一个智能体，能够按照我们需要的确切格式，对来自 arXiv 的科学论文生成摘要。当然，你可以下载一篇论文并上传到 ChatGPT——但这个系统不仅让我们能够为摘要类型建立清晰的标准，还能以简单的方式展示这种模式。

第一步，我们需要能够从 arXiv 下载一篇给定的论文，并将其转换为 Markdown 格式。为此，我们定义一个 get_text_from_axiv_paper 函数。我们可以利用微软的 markitdown 库来完成这项工作：

from markitdown import MarkItDown
def get_text_from_arxiv_paper(url: str) -> str:
    md = MarkItDown(enable_plugins=True)
    result = md.convert(url)
    return result.text_content
print(get_text_from_arxiv_paper(url="https://arxiv.org/pdf/1706.03762"))
# prints Attention Is All You Need...

现在我们需要一个函数来调用我们的 DeepSeek LLM。llm_call 函数将提示发送到 DeepSeek API，并从 deepseek-reasoner 检索最终答案和推理过程。当 with_json_output 设置为 True 时，它会在前面添加一个系统指令并设置响应格式，以确保模型以 JSON 对象回复。然后该函数将解析后的响应作为 Python 字典与推理过程一起返回。如果 with_json_output 为 False，则它只返回原始字符串响应和推理过程。

import json
from openai import OpenAI
API_KEY = os.environ["DEEPSEEK_API_KEY"]
BASE_URL = "https://api.deepseek.com"
MODEL = "deepseek-reasoner"
def llm_call(
    prompt: str, with_json_output: bool = False
) -> tuple[str | dict, str]:
    client = OpenAI(api_key=API_KEY, base_url=BASE_URL)
    args = {
        "model": MODEL,
        "messages": [],
    }
    if with_json_output is True:
        json_prompt = """
        Output your response in JSON format with the keys specified in the prompt.
        Do not include any other text such as ```json or ```.

        The response should be directly parseable by json.loads.

        """.strip()

        args["messages"].append(

            {"role": "system", "content": json_prompt}

        )

        args["response_format"] = {"type": "json_object"}

    args["messages"].append({"role": "user", "content": prompt})

评估-优化循环：论文摘要生成

现在，我们可以开始实现评估-优化循环。generate函数将目标任务与可选的上下文和基础提示一起发送给 LLM。它调用llm_call获取模型的响应和推理过程。该函数还期望响应包含<RESPONSE>...</RESPONSE>块，并提取其中的内容。为了便于调试，我们还添加了一些详细的打印输出，以便查看发生了什么。


response = client.chat.completions.create(**args)

reasoning = response.choices[0].message.reasoning_content

final_response = response.choices[0].message.content

if with_json_output is True:

    return json.loads(final_response), reasoning

return final_response, reasoning

final_response, reasoning = llm_call(

    prompt="What is the capital of Portugal?", with_json_output=True

)

print(f"Final Response: {final_response}")

# prints Final Response: {'capital': 'Lisbon'}

print(f"Reasoning: {reasoning}")

# prints Reasoning: 'First, the user asked ...'

生成函数 `generate`


def generate(

    prompt: str, task: str, context: str = ""

) -> tuple[str, str]:

    """基于反馈生成并改进解决方案。"""

    full_prompt = (

        f"{prompt}\n{context}\nTask: {task}"

        if context

        else f"{prompt}\nTask: {task}"

    )

    response, thoughts = llm_call(full_prompt)

    result = re.search(r"<RESPONSE>(.*?)</RESPONSE>",

        response, re.DOTALL).group(1)

    print("\n=== GENERATION START ===")

    print("\n*** THOUGHTS START ***")

    print(thoughts)

    print("\n*** THOUGHTS END ***")

    print("\n*** RESULT START ***")

    print(result)

    print("\n*** RESULT END ***")

    print("=== GENERATION END ===\n")

    return thoughts, result

评估函数 `evaluate`

有了生成器，我们现在需要评估器。评估器的目标是提供评估（想想你的尝试有多好？）和反馈（想想通过，失败）。为此，我们定义了一个evaluate函数。该函数构建一个提示并使用 JSON 输出来获取评估和反馈。我们也添加了一些打印输出，当然，这些是可选的。


def evaluate(prompt: str, content: str, task: str) -> tuple[str, str]:

    full_prompt = f"{prompt}\nOriginal task: {task}\nContent to evaluate: {content}"

    response, thoughts = llm_call(full_prompt, with_json_output=True)

    evaluation = response.get("evaluation")

    feedback = response.get("feedback")

    print("=== EVALUATION START ===")

    print("\n*** THOUGHTS START ***")

    print(thoughts)

    print("\n*** THOUGHTS END ***")

    print("\n*** STATUS START ***")

    print(f"Status: {evaluation}")

    print("\n*** STATUS END ***")

    print("\n*** FEEDBACK START ***")

    print(feedback)

    print("\n*** FEEDBACK END ***")

    print("=== EVALUATION END ===\n")

    return evaluation, feedback

评估-优化循环 `loop`

现在我们的generate和evaluate函数都已就位，可以创建评估-优化循环了。loop函数持续生成解决方案并评估它们，直到找到通过的结果。它接受task、evaluator_prompt和generator_prompt。它首先调用generate函数，并将初始的想法和结果存储在chain_of_thought列表中。chain_of_thought列表的目标是只跟踪生成器经历的不同想法。我们创建它是为了在过程中检查发生的不同想法。

然后它进入一个无限循环，在每次迭代开始时将最新的结果传递给评估器。如果评估返回PASS，这意味着我们的生成器认为论文摘要符合我们定义的所有标准（别担心——我们稍后会定义它们），因此循环结束。

如果结果没有通过（意味着我们的评估器不满意），函数会构建一个新的上下文，包含所有之前的尝试和最新的评估器反馈。这个上下文被传递回生成器，提示它优化输出。结果是一个反馈循环，可以增量地改进每次生成。理论上，我们的无限循环受限于 DeepSeek 的总上下文。但在生产环境中，特别是，你应该确保限制这个循环！永远不要让 LLM 无限期运行！这会让你破产！


def loop(

    task: str, evaluator_prompt: str, generator_prompt: str

) -> tuple[str, list[dict]]:

    """持续生成和评估，直到满足要求。"""

    memory = [] # 记住之前的结果

    chain_of_thought = [] # 记住思维链

    thoughts, result = generate(generator_prompt, task)

    memory.append(result)

    chain_of_thought.append({"thoughts": thoughts, "result": result})

  # 进入无限循环

    while True:

        evaluation, feedback = evaluate(evaluator_prompt, result, task)

    # 所有内容都获得批准并完成

        if evaluation == "PASS":

            return result, chain_of_thought

    # 将之前的内容添加到上下文中

        context = "\n".join(

            [

                "之前的尝试：",

                *[f"- {m}" for m in memory],

                f"\n 反馈：{feedback}",

            ]

        )

    # 再次生成并添加到记忆中

        thoughts, result = generate(generator_prompt, task, context)

        memory.append(result)

        chain_of_thought.append({"thoughts": thoughts, "result": result})

提示模板定义

有了这些函数，我们现在可以创建论文摘要工作流。第一步是创建我们的提示。我们创建两个提示：一个用于评估者，概述什么是好的摘要；另一个用于生成者，告诉它需要完成任务并将其响应输出在<RESPONSE>块中：


evaluator_prompt = """

评估以下摘要。一个好的摘要应该：

1\. 能让本科生理解

2\. 以 Markdown 格式编写，具有适当的标题和子标题

3\. 有一个标题和清晰的结构

4\. 至少有 500 字

5\. 语法和拼写正确

你只应该进行评估，而不是试图完成任务。

只有当所有标准都满足，并且你没有进一步的改进建议时，才输出“PASS”。

请按以下格式简洁地输出你的评估：

示例 JSON 输出：

{

    "evaluation": "PASS, NEEDS_IMPROVEMENT, or FAIL",

    "feedback": "需要改进的地方及原因。"

}

"""

generator_prompt = """

你的目标是根据<用户输入>完成任务。如果你之前的生成收到了反馈，你应该反思这些反馈以改进你的解决方案。

请按以下格式简洁地输出你的回答：

<RESPONSE>

响应内容

</RESPONSE>

"""

论文摘要函数 `paper_summary_for`

有了这些提示，我们的最后一步是创建一个函数，作为我们的代理循环的主入口点。我们将这个函数命名为paper_summary_for。它从论文中获取文本，定义任务（在我们的例子中是一个摘要），然后启动loop函数。完成后，它返回我们的摘要以及思维链。


def paper_summary_for(paper_url: str) -> str:

    web_page_text = get_text_from_arxiv_paper(paper_url)

    task = f"""

    <用户输入>

    为以下文章撰写摘要：

    <文章>

    {web_page_text}

    </文章>

    </用户输入>

    """

    result, cot = loop(task, evaluator_prompt, generator_prompt)

    return result, cot

完整示例与运行

作为一个例子，我们将要求我们的代理总结一篇名为MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention的近期论文（arxiv.org/pdf/2506.13585）。当我们运行主函数时，我们可以看到过程启动并进行了大约五轮。我们可以看到评估失败了多次——如果摘要太短、使用了技术术语或未正确格式化 Markdown。

结果, cot = generate_agentic_summary_for("https://arxiv.org/pdf/2506.13585")

以下是程序的输出日志：


# prints

# === GENERATION START ===

# *** THOUGHTS START ***

# Summarizing the MiniMax-M1 article…

# *** THOUGHTS END ***

# *** RESULT START ***

# MiniMax-M1 is an open-weight reasoning model using a hybrid MoE architecture…

# *** RESULT END ***

# === GENERATION END ===

#

# === EVALUATION START ===

# *** STATUS START ***

# Status: FAIL

# *** STATUS END ***

# *** FEEDBACK START ***

# Too short (189 words) and missing a markdown title.

# *** FEEDBACK END ***

# === EVALUATION END ===

#

# ... Loops hidden for conciseness.

#

# === GENERATION START ===

# *** RESULT START ***

# **MiniMax-M1: Efficient Long-Context Reasoning**

# MiniMax-M1 is a large-scale model supporting 1M-token context and trained with the CISPO RL algorithm…

# *** RESULT END ***

# === GENERATION END ===

#

# === EVALUATION START ===

# *** STATUS START ***

# Status: PASS

# *** STATUS END ***

# === EVALUATION END ===

最终，在流程结束时，评估器给出了PASS分数，循环结束。我们的摘要格式良好，包含了所有部分，并且能被研究生理解（至少我们的评估器是这么判断的！）。以下是摘要的引言部分。


**MiniMax-M1：革新大语言模型中的高效长上下文推理**

### 引言

MiniMax-M1 代表了开源大语言模型（LLMs）的一项突破性进展，旨在解决需要广泛推理和长上下文处理的复杂现实世界问题。由 MiniMax AI 开发，它通过混合架构和新颖的训练技术，开创了“测试时计算”（任务执行期间使用的计算资源）的高效扩展。该模型支持**100 万个词元**的上下文窗口——比 DeepSeek-R1 等前身大 8 倍——并能生成多达**8 万个词元**的输出，从而在软件工程、科学研究和多步骤代理工作流等领域实现了前所未有的能力。

你可以在本书的 GitHub 仓库中找到包含所有代码的完整笔记本。

如你所见，评估器-优化器是一种强大的模式，任务可以朝着一个目标逐步改进。值得一提的是，我们可能会陷入无限循环，因此我们建议将优化器循环限制在固定的迭代次数内；否则，事情可能会变得非常昂贵！此外，如果你的响应对时间敏感（即你需要快速生成内容），这个模式可能不是最合适的，因为根据任务的复杂性，过程可能会相当缓慢！

既然我们已经讨论了第一个工作流，让我们来讨论另一个重要的工作流，即协调器-工作者工作流。

工作流：协调器-工作者

是时候让事情稍微复杂一点了。在第二个例子中，我们将介绍协调器-工作者工作流。如果这个工作流的名字听起来很吓人，别怕，亲爱的读者。它比听起来简单得多！

协调器-工作者工作流背后的思想是，你可以将一个给定的任务分解成多个可以并行运行的小任务。首先，我们有一个协调器组件。这个 LLM 的工作是接收某个任务并将其分解为 N 个独立任务。每个工作者LLM 接收其中一个任务并执行其需要执行的操作。这里的最后一个组件是综合器LLM，其工作是收集所有工作者 LLM 的输出并将其转换为流程的单一连贯输出。

图 6.4：协调器-工作者工作流模式

你可能在实践中见过这种工作流。事实上，它驱动着像 ChatGPT Deep Research 这样的系统，该系统通过搜索网络为用户进行深度研究。在它的工作流中，创建搜索报告后，多个 LLM 然后充当工作者，独立搜索网络以获取关于子主题的相关信息。需要澄清的是，我们可能使用相同类型的 LLM（例如 DeepSeek V3），但我们并行启动该 LLM 的不同实例来研究不同的主题。最后，一个协调器LLM 综合所有信息并向用户返回一份连贯的最终报告。

当然，这都是推测——官方的 Deep Research 产品实际上是闭源的，所以我们能做的只是猜测。但创作者最近的一次演讲（www.youtube.com/watch?v=eJOjdjO45Sc）实际上指向了多次 LLM 调用的方向。此外，如果单个 LLM 必须一个接一个地浏览 70 个网站，用户很快就会失去耐心！

尽管如此，当手头的任务可以分解成更小的片段并并行化时，这仍是一种特别有用的模式。

示例：一个报告生成工作流

我们已经将 Deep Research 作为协调器-工作者工作流的一个有趣用例。让我们构建一些类似的东西。目标将是构建一份关于某个任务的报告。我们将让协调器 LLM 决定要撰写哪个部分。一旦它决定，我们就并行运行所有工作者。一旦最后一个工作者完成，我们就将所有输入馈送到我们的最终构建模块——综合器 LLM，它将生成最终报告。如果你感到害怕，别担心——我们将一步一步地构建它。

我们首先使用 Python 的@dataclass装饰器定义两个数据类。Task是工作者的输入。每个任务都有一个章节标题和撰写它的理由。我们还定义了WorkerResult，它是每个工作者的输出。它包含相同的元素，以及一个结果字符串。我们可以使用相同的类，但目前让我们将它们分开。这是一个好习惯。


from dataclasses import dataclass

@dataclass

class Task:

    section_title: str

    rationale: str

@dataclass

class WorkerResult:

    section_title: str

    rationale: str

    result: str | None = None

我们现在开始定义我们的OrchestratorWorker类。没什么花哨的。只有一个__init__方法，它接受协调器的提示词、每个工作者的提示词和最终综合的提示词。这些提示词将使用jinja2包制作成模板。


from jinja2 import Template

class OrchestratorWorker:

    """Break down tasks and run them in parallel using worker LLMs."""

    def __init__(

        self,

        orchestrator_prompt: str,

        worker_prompt: str,

        synthesis_prompt: str,

        task: str,

    ):

        self.orchestrator_template = Template(orchestrator_prompt)

        self.worker_template = Template(worker_prompt)

        self.synthesis_template = Template(synthesis_prompt)

        self.task = task

我们还定义了一个run_single_worker方法，它将接受一个主任务和一个任务，渲染工作者提示词模板，将其通过 LLM 运行，并输出一个工作者结果。注意我们将with_json_output设置为True，因为我们期望工作者结果是一个带有response键的字典。稍后会详细介绍。


 def run_single_worker(

        self, main_task: str, task: Task

    ) -> WorkerResult:

        """Run a single worker and return the result."""

        worker_input = self.worker_template.render(

            original_task=main_task,

            section_title=task.section_title,

            rationale=task.rationale,

        )

        print(f"\n[WORKER INPUT] Section: {task.section_title}\n{worker_input}\n")

        worker_response = llm_call(worker_input, with_json_output=True)

        result = worker_response["response"]

        print(f"\n[WORKER OUTPUT] Section: {task.section_title}\n{result}\n")

        return WorkerResult(

            section_title=task.section_title,

核心循环逻辑

以下是逻辑的核心循环。我们接收编排器输入，并从中获取一定数量的任务。每个任务被解析为 Task 类型，并并行地映射到 run_single_worker 函数。由于我们正在并行化一个 IO 密集型任务（对我们 LLM 的 API 调用），我们使用了 Python 内置的 ThreadPoolExecutor。


def process(self, task: str) -> list[WorkerResult]:

    """Run orchestrator and execute all workers in parallel."""

    input_str = self.orchestrator_template.render(task=task)

    print(f"\n[ORCHESTRATOR INPUT]\n{input_str}\n")

    orchestrator_output = llm_call(input_str, with_json_output=True)

    print(f"[ORCHESTRATOR OUTPUT]\nAnalysis:\n{orchestrator_output['analysis']}")

    print(f"\nTasks:\n{orchestrator_output['tasks']}\n")

    worker_tasks = [Task(**t) for t in orchestrator_output["tasks"]]

    print(f"[INFO] Total tasks: {len(worker_tasks)}\n")

    with ThreadPoolExecutor() as executor:

        results = list(

            tqdm(

                executor.map(

                    lambda t: self.run_single_worker(task, t),

                    worker_tasks

                ),

                total=len(worker_tasks),

                desc="Running workers",

            )

        )

    return results

关键方法描述

synthesize 方法是此处需要描述的关键步骤。它相对简单，其主要任务是接收一个 WorkerResult 列表，并使用 synthesis_template 渲染最终结果并呈现给用户。


def synthesize(self, worker_results: list[WorkerResult]) -> str:

    """Generate final report by synthesizing all worker results."""

    synthesis_input = self.synthesis_template.render(

        original_task=self.task,

        worker_results=worker_results,

    )

    print(f"\n[SYNTHESIS INPUT]\n{synthesis_input}\n")

    response = llm_call(synthesis_input, with_json_output=True)

    return response["response"]

最后，我们实现了 run 方法，它将作为我们类的入口点。该方法调用 self.process 方法，并将结果输入 synthesize 方法。最终，它将综合结果返回给用户。


def run(self) -> str:

    """Run the orchestrator and return the final report."""

    worker_results = self.process(self.task)

    synthesis_result = self.synthesize(worker_results)

    return synthesis_result

提示词定义

现在我们准备好运行编排器-工作者工作流程。首先，我们为三个组件定义提示词：

编排器提示词：接收主任务，并将其分解为 3-5 个不同的任务。以下是如何定义它。


ORCHESTRATOR_PROMPT = """

You're tasked with creating a report on a specific topic.

Break this down into 3-5 distinct sections that could each be written with a different focus.

Report topic: {{ task }}

Return your response in this JSON format:

{

  "analysis": "Summarize your understanding of the reporting goal and explain why breaking it into multiple sections with different focuses would be valuable. Highlight how each section emphasizes a different priority (e.g., technical depth, accessibility, narrative).",

  "tasks": [

    {

      "section_title": "Technical Overview",

      "rationale": "Explain why this section is important and what it will cover.",

    },

    {

      "section_title": "Practical Implications",

      "rationale": "Explain why this section is important and what it will cover.",

    }

  ]

}

"""

工作者提示词：接收主主题、一个独立的任务（在此例中是一个章节标题）以及该任务的依据。让我们定义工作者提示词。


WORKER_PROMPT = """

You're generating a section of a report.

Topic: {{ original_task }}

Section title: {{ section_title }}

Rationale: {{ rationale }}

Write the content for this section using **Markdown formatting** (bold, italics, lists, code blocks, etc.) where appropriate, but **do not include a header/title** — just the body content.

Return your response in this JSON format:

{

  "response": "Your report content here, using Markdown formatting but without any section headers."

}

"""

综合提示词：接收来自所有工作者的输入，并将响应综合为最终版本。以下是综合提示词。


SYNTHESIS_PROMPT = """

You're synthesizing multiple report sections into one cohesive final report.

Original topic: {{ original_task }}

These are the completed sections (in order):

{% for result in worker_results %}

Section title: {{ result.section_title }}

---

{{ result.result }}

{% endfor %}

Your task is to merge the sections into a well-flowing report. Use **Markdown formatting** where helpful, but **do not repeat the section titles**.

Ensure:

- The tone and style feel consistent throughout.

- Transitions between sections are smooth.

- Any overlap is minimized.

- The final output reads like a single unified piece.

Return your response in this JSON format:

{

  "response": "Your final merged report here, in Markdown."

}

"""

运行示例

一切定义就绪，我们可以用一个问题启动工作流。我选择了一个对我个人有点意义的问题，但你可以选择任何问题！


task = "Make an in-depth report that explains why biking has not been adopted in the City of Lisbon, and what can be done to encourage it."

orchestrator = OrchestratorWorker(

    orchestrator_prompt=ORCHESTRATOR_PROMPT,

    worker_prompt=WORKER_PROMPT,

    synthesis_prompt=SYNTHESIS_PROMPT,

    task=task,

)

final_report = orchestrator.run()

你可以查看 GitHub 仓库以获取工作流的完整输出，但我会在这里概述一些有趣的部分。

首先，编排器将这个问题划分为五个不同的任务，每个任务都有一个依据，并且每个任务都可以被单独探索。我们提供依据是为了让工作者对特定章节为何重要有更多上下文。


[

    {'section_title': 'Current Infrastructure and Challenges',

    'rationale': '...'

    },

    {'section_title': 'Cultural and Social Attitudes Towards Biking',

    'rationale': '...'

    },

    {'section_title': 'Policy and Governance',

    'rationale': 'T...'

    },

    {'section_title': 'Successful Case Studies and Potential Solutions',

    'rationale': '...'

    },

    {'section_title': 'Community Engagement and Awareness Campaigns',

    'rationale': '...'

    }

]

我不会展示所有工作者的全部输入，你可以查看 GitHub 仓库获取完整信息，但这里有一个工作者在行动中的示例：


[WORKER INPUT] Section: Current Infrastructure and Challenges

You're generating a section of a report...

#...

[WORKER OUTPUT] Section: Current Infrastructure and Challenges

The City of Lisbon faces several infrastructure and logistical challenges

that hinder the widespread adoption of biking as a mode of transportation.

Below is an in-depth analysis of these barriers:

...

如你所见，该工作者独立研究并输出了关于这个主题的报告章节。最后，综合器获取所有这些输出，并将其浓缩成一份连贯的单一报告。


The City of Lisbon faces significant challenges in adopting biking as a mainstream mode of transportation, primarily due to infrastructure limitations, cultural attitudes, and policy gaps. However, by addressing these issues through targeted strategies, Lisbon can foster a more bike-friendly environment.

**Current Infrastructure and Challenges**

智能体：工具调用智能体

里斯本的城市景观以陡峭的山丘和缺乏专用自行车道为特点，这给骑行带来了相当大的障碍。缺乏安全的停车设施以及与公共交通整合不佳，进一步阻碍了潜在的骑行者。要克服这些障碍，扩大自行车道网络、引入电动自行车共享计划以及改善停车设施是必要的步骤。

文化和社会对自行车的态度

里斯本的文化观念通常将骑行视为一种休闲活动，而非实用的交通方式，加上安全顾虑以及与汽车所有权相关联的地位，使得这种观念更加根深蒂固。要改变这些态度，需要开展社区参与项目、改善基础设施，并提供激励措施以凸显骑行的好处。


This is an interesting pattern, where the task can be broken down into individual sections/tasks that can be run in parallel. You can take this example and apply it to other tasks – for example, grading different essays, analyzing resumes, conducting different web searches on a topic, or editing files in a code base. Almost anything can be broken down into smaller, independent tasks! Not everything, though! Some tasks need to be sequential. So, think before throwing this pattern at a problem.

既然我们已经介绍了协调器-工作者模式，我们将讨论构建智能体（尤其是使用 DeepSeek 时）的最后一个，但非常重要的构建模块，那就是工具调用智能体。

我们到目前为止已经构建了两种智能体工作流。但我们还没有构建一个完整的智能体。没有工作流，没有防护栏——只是让智能体做出所有决策。然而，你可以添加到任何LLM中最强大的东西之一就是工具。回想一下，工具是你的LLM可以调用的函数。例如，你可能有一个search_web工具。当LLM决定调用它时，它会告诉你它想使用search_web工具并带上deepseek参数，如果它想搜索那个术语的话。

正如我们之前提到的，在这种情况下，我们将让LLM来做出决策。DeepSeek将决定调用哪些工具以及以什么顺序调用它们。我们也可以将其称为augmented LLM，即由一些工具支持的LLM。这种模式的现代版本可能包括智能体的记忆力或更复杂的技术。一些其他的智能体定义可能被简化为：An agent is an LLM with tools in a loop。虽然简单，但这涵盖了我们最终智能体系统的核心。你可以看到这里展示的工具调用智能体。

图 6.5：一个工具调用智能体

在构建工具调用智能体时，我们应该注意几点。我们的智能体可以使用哪些工具，它们是否有助于解决整体任务？LLM选择不同工具的循环会永远进行下去吗？智能体可以向人类寻求帮助吗？所有这些在构建工具调用智能体时都很重要。例如，当我们的智能体无需阅读代码库时，给予它访问代码库阅读工具的权限只会让它感到困惑。让客户支持智能体能够进行网络搜索，很可能导致它引用互联网上的错误信息。我们在为智能体创建工具时应该extremely小心和周到。用我们葡萄牙的话来说，这就是香肠是如何制作的（即让事情运转良好的秘诀！）。

解释够了，让我们构建一个由DeepSeek驱动的工具调用智能体。

示例：一个网络搜索智能体

在这个例子中，我们将创建一个网络搜索智能体，它可以接收一个问题，然后在互联网上搜索该问题的最佳答案。我们将为这个智能体配备完成任务的工具，并从头开始实现一个工具调用循环。由于我们的智能体需要能够搜索互联网，我们将赋予它访问两个工具的权限：search_web（用于执行网络搜索）和read_website（用于阅读互联网上的任何网站）。我们也可以添加第三个工具来向我们寻求支持或澄清。但这将留给你作为练习。

我们从通常的LLM调用函数开始，但增加了两个重要方面：由diskcache支持的缓存以避免冗余的模型调用。对于推理模型（如deepseek-reasoner），我们从消息链中剥离推理内容（与其他模型不同，DeepSeek文档表明我们必须从messages链中移除推理内容）。


from diskcache import Cache

cache = Cache("./cache")

@cache.memoize(expire=3600)

def llm_call(

    messages: list[dict[str, str]], tools: list[dict[str, Any]], model: str

) -> str:

    response = client.chat.completions.create(

        model=model,

        messages=messages,

        tools=tools if tools else None,

        temperature=0.0,

        tool_choice="auto",

    )

    message = response.choices[0].message

    if hasattr(message, "reasoning_content"):

        del message.reasoning_content

    return message

现在让我们定义我们的工具。第一个是search_web工具，可用于使用Marginalia搜索引擎（https://www.marginalia.nu/）进行网络搜索。选择这个的主要原因是它是免费的，并且有非常简单的 API。但在生产环境中，你可能会评估或使用其他搜索引擎（Google、Brave或其他）。

当我们搜索某些内容时，它将返回一个链接列表、标题以及其他一些详细信息。


from pydantic import BaseModel

import httpx

class SearchWeb(BaseModel):

    """Search the web for a given query."""

    query: str

    max_results: int = 10

def search_web(data: SearchWeb) -> str:

    base_url = "https://api.marginalia.nu/{key}/search/{query}"

    url = base_url.format(key="public", query=data.query)

    url += f"?count={data.max_results}"

    rsp = httpx.get(url)

    rsp.raise_for_status()

    results = rsp.json()["results"]

    return str(results)

# print(search_web(SearchWeb(query="Paris")))

# [

# {'url': 'https://en.wikipedia.org/wiki/Paris_Las_Vegas', 'title': 'Paris ..', 'description': '...', 'quality': 5.037784569660175, 'format': 'html', 'details': [[]]},

# {'url': 'https://en.wikipedia.org/wiki/Paris_Observatory', 'title': 'Paris Observatory', 'description': 'The Paris ...', 'quality': 5.161455267002603, 'format': 'html', 'details': [[]]}

# ]

我们将配备给智能体的第二个工具是read_website工具。它使用markdownify将网站内容转换为Markdown格式，以便LLM能更好地理解它。


from markdownify import markdownify as md

class ReadWebsite(BaseModel):

    """Read the content of a website and return it as text. (useful for further research)"""

    url: str

def read_website(data: ReadWebsite) -> str:

    html_content = httpx.get(data.url).text

    return md(html_content)

# print(read_website(data=ReadWebsite(url="https://www.theguardian.com/europe")))

# prints "The Latest news, sport and opinion from the Guardian.."

我们还定义了一个小的辅助函数，用于将普通函数转换为DeepSeek所需的工具调用格式。这是必需的，因为DeepSeek的 API 期望使用 JSON 格式来调用工具。


def to_openai_tool(func: callable) -> dict[str, Any]:

    sig = inspect.signature(func)

    if sig.parameters:

        param = next(iter(sig.parameters.values()))

        param_type = param.annotation

        schema = param_type.model_json_schema()

    else:

        param_type = None

        schema = {"type": "object", "properties": {}, "required": []}

    return {

        "type": "function",

        "function": {

            "name": func.__name__,

            "description": (param_type.__doc__ if param_type else func.__doc__ or ""),

            "parameters": schema,

        },

    }

最后，我们通过调用to_openai_tool来创建工具。我们这样做是因为我们使用的是 OpenAI SDK，它要求特定的工具格式。我们还创建了一个FUNC_TYPES字典，将每个函数映射到正确的 Pydantic 输入格式：


TOOLS = [

    to_openai_tool(search_web),

    to_openai_tool(read_website),

]

# {'type': 'function', 'function': {'name': 'search_web', 'description': ...

FUNC_TYPES = {

    search_web: SearchWeb,

    read_website: ReadWebsite,

}

现在工具和样板代码都已就绪，让我们看看代理是如何工作的。它是一个简单的类。在初始化时，它设置模型、工具、消息列表和最大迭代次数。主要逻辑在__call__中。它将用户提示添加到消息历史中，然后进入一个循环：

如果模型直接响应，我们返回它。
如果它想调用一个工具，我们找到正确的函数，运行它，追加结果，然后继续。

_send方法只是将消息传递给模型并返回响应——没有什么特别的。总的来说，代理会持续循环：询问模型该做什么，如果需要则调用工具，并将结果反馈回来——直到它完成或达到迭代上限。


class Agent:

    def __init__(

        self,

        model: str,

        system: str,

        tools: list[dict[str, Any]] | None = None,

        max_iters: int = 10,

    ):

        self.model = model

        self.tools = tools or []

        self.messages: list[dict[str, str]] = []

        if system:

            self.messages.append({"role": "system", "content": system})

        self.max_iters = max_iters

    def __call__(self, content: str, verbose: bool) -> str:

        self.messages.append({"role": "user", "content": content})

        iterations = 0

        while True:

            iterations += 1

            message = self._send()

            self.messages.append(message)

            tool_calls = getattr(message, "tool_calls", None)

            if not tool_calls:

                return message.content

            for call in tool_calls:

                func = globals()[call.function.name]

                args_model = FUNC_TYPES[func]

                args = (

                    args_model.model_validate_json(call.function.arguments)

                    if call.function.arguments != "{}"

                    else None

                )

                print(f"** CALLING FUNCTION {func.__name__} **")

                print(f"*** ARGS ***\n{args}\n")

                result = func(args) if args else func()

                if verbose is True:

                    print(f"*** RESULT ***\n{result}\n***")

                self.messages.append(

                    {

                        "role": "tool",

                        "tool_call_id": call.id,

                        "content": result,

                    }

                )

            if iterations >= self.max_iters:

                raise Exception("Max iterations reached")

    def _send(self):

        return llm_call(self.messages, self.tools, self.model)

现在我们定义系统提示。我们告诉 LLM 它应该遵循“思考-行动-观察”这个循环，并且我们用一个关于 2025 年安科纳市人口的简单问题来启动它。


system_prompt = """

You are a helpful assistant who can answer multistep questions by sequentially calling functions.

Follow a pattern of:

- THOUGHT (reason step-by-step about which function to call next)

- ACTION (call a function as a next step towards the final answer)

- OBSERVATION (output of the function)

Reason step by step which actions to take to get to the answer.

Only call functions with arguments coming verbatim from the user or the output of other functions.

"""

question = "What is the population of the city of Ancona as of 2025?"

要调用代理，我们附加系统提示、工具，并从deepseek-chat模型开始。


bot = Agent(system=system_prompt, tools=TOOLS, model="deepseek-chat")

response = bot(question, verbose=False)

print("Response:", response)

每次代理调用一个函数时，我们都会记录它。我们可以看到，在第一次尝试中，代理使用查询population of Ancona Italy in 2025调用了搜索网页函数。然后它读取了网站population.city/italy/ancona/，在那里它最终遇到了一个数字。然后它获取了这个数字，并将其从 2014 年推算到 2025 年。不确定我们是否可以将其计为correct，但看到模型为自己调用不同的工具确实很有趣。无论如何，我们could在提示中指定它应该只使用估计值而不是外推值。但我们将此作为留给您，我们亲爱的读者的练习。


** CALLING FUNCTION search_web **

*** ARGS ***

query='population of Ancona Italy 2025' max_results=10

** CALLING FUNCTION read_website **

*** ARGS ***

url='https://population.city/italy/ancona/'

Response: The last known population of Ancona, Italy, was approximately **101,500** in 2014\. If the population growth rate remained consistent at **0.34% per year** (as observed between 2011-2014), the estimated population of Ancona in 2025 would be **105,351**.

This is an unofficial projection based on historical data. For the most accurate and up-to-date figures, official census data or reports from the Italian National Institute of Statistics (ISTAT) would be required.

现在让我们尝试deepseek-reasoner模型，以期获得一个更intelligent的循环。


bot = Agent(system=system_prompt, tools=TOOLS, model="deepseek-reasoner")

response = bot(question, verbose=False)

print("Response:", response)

我们可以看到，推理器驱动的代理调用了更多函数，并（自行）决定寻找official estimates。经过一些挖掘，它最终从官方统计局找到了大约 99K 人的估计值。这听起来更合理，特别是模型还为我们提供了一个参考！


** CALLING FUNCTION search_web **

*** ARGS ***

query='Ancona population 2025 projection' max_results=10

** CALLING FUNCTION search_web **

*** ARGS ***

query='Ancona Italy population 2025 official estimate' max_results=10

** CALLING FUNCTION read_website **

*** ARGS ***

url='https://www.citypopulation.de/en/italy/cities/marche/'

Response: Based on the official population estimates from the Istituto Nazionale di Statistica Italia (Italy's National Institute of Statistics), the population of Ancona as of January 1, 2025 is **99,469**.

This information comes directly from the detailed city and commune population tables on citypopulation.de, which sources its data from Italy's official statistics agency. The table shows:

- 2021 census: 98,402

- 2025 estimate: 99,469

The population estimate represents the projected population for the city proper of Ancona as of the beginning of 2025.

总结

这里值得暂停一下做个说明。两次运行使用了相同的工具调用智能体、相同的工具（search_web 和 read_website）以及相同的系统提示。唯一的区别是模型：deepseek-chat 与 deepseek-reasoner。然而，结果却大相径庭。DeepSeek-Chat 在页面上找到一个数字，并选择将其向前推算到 2025 年，尽管页面（或提示）本身并未建议这样做。DeepSeek-Reasoner 做出了更谨慎的选择：它重新构思了查询，搜索了官方估算值，并根据意大利统计局的报告，正确返回了 99,469。这种差异凸显了 动态工作流 的一个关键特征：即使结构和工具保持不变，模型的内部推理也决定了控制流。DeepSeek-Chat 遵循了一条更短但有缺陷的路径。DeepSeek-Reasoner 增加了步骤并得出了正确答案。这就是动态智能体系统的希望与陷阱——它们灵活且能够适应，但本质上也难以预测，因为流程由模型行为驱动，而非硬编码逻辑。

增强型大语言模型是 LLM 驱动应用程序的一个相当强大的构建模块。在本例中，我们将其作为一个独立模块来解决特定问题。然而，你可以将增强型大语言模型视为任何其他工作流的乐高积木。

例如，在评估者-优化器工作流中，评估者可以配备工具（例如，用于测试代码或验证 SQL），生成器也可以。在编排者-工作者工作流中，每个工作者都可以配备工具来研究特定主题或完成特定任务。你不受限于某个特定的工作流或特定的一组工具。工具可以具有状态。例如，在现代 IDE 中，如 VSCode，你的聊天助手可能拥有访问工具的能力，以搜索你的代码库、编辑一定数量的文件，或读写你希望其记住的一组规则。

关于评估智能体的重要说明

评估智能体——以及 LLM 应用程序——至关重要。发布产品而不衡量成功意味着你将难以改进它。虽然本书不涉及评估，我们也不会为我们构建的智能体设计评估系统，但我们觉得有必要指引你走向正确的方向。

你会发现许多框架承诺提供“4 行代码评估”，并配有诸如“答案相关性”或“忠实度”等指标。这些数字可能看起来不错——太好了，我的相关性得分是 0.87！——但在许多情况下，它们与产品的成功无关。如果你构建一个智能体来回答“意大利某个小城市的人口是多少？”，唯一的成功指标就是它是否返回了 正确的数字。其他一切都是次要的。

根据我们的经验，评估基于 LLM 的应用程序最有效的方法是通过一个简单、迭代的过程：

从一个可信来源收集一个包含 1,000 个不同城市及其人口的数据集。
对你所有的城市运行你的智能体：X 城市的人口是多少？
衡量你的准确性：有多少答案是正确的？哪些是错误的？为什么？
根据学到的经验改进系统：切换搜索引擎、更改提示、添加规划步骤。
从步骤 2 重复。

通过多次重复此过程，你将系统地改进你的系统。这比提供一个光鲜数字的“4 行代码”框架要复杂，但相信我们：你在这个过程中会学到更多。

在撰写本书时，智能体正风靡一时。每家公司的高管和初创企业领导人都在热议智能体可以解决的各种问题。现在，我们不知道智能体是否能解开科学之谜或做出科学发现，我们当然也怀疑它们能否在短暂的生命周期内做到这一点。然而，知道如何利用 DeepSeek 来构建智能体是一项值得掌握的超能力。

在本章中，我们涵盖了智能体应用和模式的基本工作流和构建模块。随着这项技术的快速发展，我们决定专注于原则，并从头开始构建每一个。

在下一章中，我们将继续探索 DeepSeek 的方方面面，并了解如何为不同的应用对其进行微调。

既然你已经学会了如何利用 DeepSeek 构建智能体系统，我们将在下一章将重点转移到使用 DeepSeek 进行微调。

获取本书 PDF 版本与独家福利

扫描二维码（或访问 packtpub.com/unlock）。按书名搜索本书，确认版本，然后按照页面上的步骤操作。

注意：请保留好您的发票。直接从 Packt 购买无需发票。

第三部分

蒸馏与部署 DeepSeek

在本书的第三部分，我们将从理解和使用 DeepSeek 转向将其改造为生产环境应用。我们首先探讨推理蒸馏——利用 DeepSeek 高级推理能力，创建能够在普通硬件上运行、同时保持专业级性能的专用高效模型。接着，我们解决部署全规模 DeepSeek 模型所面临的实际挑战，探讨 API、托管服务和自托管解决方案之间的权衡。最后，我们专注于构建生产就绪的运营体系，涵盖监控、扩展、成本管理和安全实践，将实验系统转变为可靠的企业级服务。

学完本部分，你将了解如何蒸馏领域特定模型，根据你的需求和约束做出明智的部署决策，并能够自信可靠地大规模运行 DeepSeek 模型。

本部分包含以下章节：

第 7 章，使用 DeepSeek 驱动的微调 Gemma 3 进行法律推理
第 8 章，部署 DeepSeek 模型
第 9 章，结语
附录

敬请关注

要了解生成式 AI 和 LLM 领域的最新发展，请订阅我们的周刊《AI_Distilled》，地址是 packt.link/8Oz6Y。

加入我们的 Discord 和 Reddit 社区

对本书有疑问，或想参与关于生成式 AI 和 LLM 的讨论？

请加入我们的 Discord 服务器 packt.link/4Bbd9 和我们的 Reddit 频道 packt.link/wcYOQ，与志同道合的爱好者交流、分享和协作。

7 使用 DeepSeek 驱动的微调 Gemma 3 进行法律推理

想象一下，你被要求构建一个高频交易系统。你不会选用通用的脚本语言和消费级笔记本电脑——你会选择低延迟、高度优化的技术栈，专为你所在领域的精确需求而调校。在工程学中，我们知道，正确的工具加上正确的配置，是“它能运行”与“它能可靠、有保障地大规模运行”之间的区别。

这就是通用 大语言模型 (LLM) 所面临的核心挑战。像 OpenAI 的 GPT-5 或 Claude 4 Sonnet 这样的模型是能力惊人的通才，但它们相当于一个强大但通用的框架：灵活，但并未针对专业领域的边缘情况和性能要求进行优化。当你需要解析成千上万份法律合同以发现细微的风险因素，或在受监管的行业中自动化合规检查时，“足够接近”是不够的——精确性、可靠性和可解释性至关重要。你可能也关心使用这些模型的成本。

在本章中，我们将执行一个两阶段工作流，包括使用 DeepSeek-R1 进行推理蒸馏，以及在推理增强数据上微调 Gemma 3。

根据我们这里的术语，蒸馏 指由教师模型生成数据（不更新教师模型权重）；也就是说，DeepSeek-R1 作为教师模型来生成训练数据。微调 指更新学生模型的参数（通过 LoRA/Unsloth）以学习教师的行为。对于我们的用例，我们将微调 Gemma 3，一个更适合部署的较小模型。我们还将使用保留数据来评估 学生 模型与 教师 模型的表现。DeepSeek-R1 本身在此过程中不进行微调；这反映了行业实践，其中蒸馏更为常见。

在本章中，我们将涵盖以下主要主题：

理解蒸馏和微调的重要性
介绍 CUAD：一个用于法律条款分类的结构化基准
用于模型蒸馏的 LLMOps 工具
法律推理蒸馏的两阶段工作流
评估与结果
关键要点

技术要求

你需要以下内容来学习本章：项目仓库位于 github.com/PacktPublishing/DeepSeek-in-Practice。

依赖项

从项目仓库开始，安装所需的包：


git clone https://github.com/PacktPublishing/DeepSeek-in-Practice.git

cd Chapter07

pip install -r requirements.txt

该依赖文件包含用于流程编排的 ZenML、用于高效训练的 Unsloth、用于模型处理的 Transformers，以及用于数据处理和评估的各种工具。

创建你的本地 ZenML 环境

我们在代码示例中使用 ZenML 来演示良好的 MLOps 实践（流水线、缓存和血缘追踪）。你可以将其替换为提供相同功能的其他平台，而不改变学习目标（例如，MLflow、Weights & Biases、Flyte、Metaflow、Kubeflow 或 Airflow）。

创建你的 ZenML Cloud 账户

在 cloud.zenml.io 创建一个免费托管账户。这将作为你的实验跟踪和流水线编排中心，用于监控训练进度和管理产物。免费套餐为我们的法律分类项目提供了充足的资源。

注册后，连接你的本地环境并设置项目名称：


# 登录并选择你的工作区

zenml login

# 激活你的项目

zenml project set default # 通常默认项目就可以

带 GPU 的云栈：模型训练需要 GPU 计算，内存需求随模型规模而变化：
- 1B 模型：8 GB+ 显存。
- 4B 模型：16 GB+ 显存。
- 12B 模型：24 GB+ 显存。

为此，你需要按照 ZenML 云部署指南（docs.zenml.io/stacks/deployment/deploy-a-cloud-stack）设置云栈。ZenML 中的 stack 代表你的流水线运行的基础设施。你只需切换 stack，就可以轻松地从本地设置切换到云基础设施。创建云栈最快的方法是通过基础设施即代码选项。这使用 Terraform 部署云资源，并将它们注册为 ZenML stack。你需要以下内容：

本地安装的 Terraform（www.terraform.io/downloads.html）版本 1.9+。
已为首选云提供商（AWS、GCP 或 Azure）配置认证。
在你的云账户中创建资源的适当权限。

部署向导将指导你完成每个步骤。部署将需要几分钟时间。

部署完成后，激活你的新栈：


zenml stack set <NAME_OF_YOUR_NEW_STACK>

API 密钥和环境变量

你需要 OpenRouter 和 Hugging Face 模型的 API 密钥：

OpenRouter 提供访问 DeepSeek 模型的权限，用于在评估期间生成基线比较。在 openrouter.ai 注册并从仪表盘获取 API 密钥。我们将使用免费模型来访问基础 DeepSeek-R1 模型，但你需要 API 密钥才能访问它（通过 litellm）。如果你没有 OpenRouter API 密钥，可以在 openrouter.ai/docs/api-reference/api-keys/get-key 创建一个。
Hugging Face 用于上传你训练的模型。在 huggingface.co/settings/tokens 创建一个具有写入权限的令牌。使用 huggingface.co/join 创建你的 API 密钥。

配置所需的 API 凭据：


export OPENROUTER_API_KEY="your_openrouter_api_key"

export HF_API_KEY="your_huggingface_api_key"  # 可选

增强版 CUAD 数据集

增强版 CUAD 是 Contract Understanding Atticus Dataset (CUAD) 的一个理由增强版本，它为 510 份真实合同中 41 种条款类型的原始专家标签添加了 DeepSeek 生成的分步解释（理由）。我们将使用 Hugging Face 发布版 zenml/cuad-deepseek（huggingface.co/datasets/zenml/cuad-deepseek）。

不使用 ZenML 进行微调（独立脚本）

如果你只想微调模型而不安装 ZenML，我们提供了一个独立的 Python 脚本来重现本章的训练循环。

不使用 ZenML 的可选独立微调脚本

独立脚本位于此处：github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter07/standalone_training.py。该脚本遵循以下流程：

从 Hugging Face 加载增强版 CUAD 数据集（zenml/cuad-deepseek），并确保训练/验证/测试集划分。

增强版 CUAD 使用 seed=42 进行 80/10/10 分层划分，分为训练/验证/测试集。我们在训练集上微调学生模型，在验证集上监控和调优，并且所有指标仅在保留的测试集上报告。没有测试示例——或其理由——在训练期间被使用。
将示例格式化为 Gemma 3 聊天消息（用户 JSON 提示；包含理由和标签的助手 JSON）。
通过 Unsloth 将 LoRA 适配器附加到 Gemma 3（1B、4B 或 12B）上，并使用 TRL 的 SFTTrainer 仅在助手响应上进行训练。
将适配器和分词器本地保存，并可选择推送到 Hugging Face Hub。
在测试集上运行快速训练后健全性评估。

要开始使用该脚本，请执行以下操作：

安装依赖项（推荐使用支持 CUDA 的 PyTorch）：


pip install unsloth transformers trl datasets peft accelerate torch

运行最小训练命令（12B 模型需要约 24 GB 显存；如果显存较小，请尝试 --model-size 4b）：
```
python standalone_training.py --model-size 12b --filter-none-labels --eval-after-train
```
使用以下命令查看所有选项：
```
python standalone_training.py --help
```

提示：要将训练好的适配器推送到 Hugging Face Hub，请添加 --push-to-hub 并首先运行此命令以确保你已登录：


huggingface-cli login

了解蒸馏和微调的重要性

微调和蒸馏是适应大语言模型至特定领域的两种核心技术。它们各自解决不同方面的问题，共同提供了一条从通用模型走向生产就绪系统的实用路径。

在开始之前，让我们快速回顾一下微调和蒸馏。

微调通过在目标数据上更新（部分）参数，使已经预训练的模型适应你的领域。可以将其视为取一个强大的通用系统，并将其调整到你的工作负载——术语、格式和决策模式——以便它能在你的环境中可靠运行。

另一方面，蒸馏则将大型教师模型的行为压缩到一个更小的学生模型中。它有多种形式（如 logit 蒸馏、响应蒸馏或理由蒸馏）。

为什么我们需要专用模型？

与专用的、自托管选项相比，通过外部 API 使用通用模型可能会带来隐藏成本，尤其是在大规模应用时。部分原因如下：

Token 成本：

即使价格更低且支持批处理，使用外部 API 处理单个 300 页的并购协议（每条约 600,000 个 token）也可能变得非常昂贵。每月处理许多此类文档的组织可能面临高额账单，而一个在两张 A100 GPU 上运行（月摊销成本约 3,000 美元）的蒸馏内部模型可以以低得多的成本处理相同的工作量。
监管与合规风险：

使用外部 API 存在监管风险。例如，2024 年 12 月，意大利隐私监管机构因 GDPR 违规对 OpenAI 处以 1500 万欧元罚款（链接），该违规与 ChatGPT 的数据处理有关。这突显了当敏感数据离开组织安全边界时的责任。一些行业现在要求主权大语言模型部署，以避免此类处罚并保持数据驻留。
性能与可用性：

来自金融机构（链接）的内部基准测试表明，在本地部署的 70 亿参数蒸馏模型响应时间为 70-120 毫秒。相比之下，公共 API 在高峰负载时的 P95 延迟可能高达 600-1,200 毫秒。这种差异会影响高频审查工作流的服务级别协议，对实时应用非常重要。
灵活性与供应商锁定：

外部 API 无法针对组织特定的术语、格式或业务规则进行定制。通用模型可能缺乏微调替代方案所具有的领域知识。依赖 API 也可能造成供应商锁定，降低战略灵活性并增加长期风险。

以下是推动领域专用模型使用的一些重要考虑因素：

准确性：例如，考虑一个在合同语言上微调过的法律大语言模型，可以在文档分类任务上常规达到 95%以上的准确率，而通用模型约为 70%。这不是边际改进，而是一个偶尔辅助的工具与一个可靠自动化核心工作流的工具之间的区别。
更低风险：然而，准确性只是一个维度。通用模型容易产生幻觉，自信地生成不正确的输出。在医疗或金融等高风险领域，这是不可接受的。在精选数据集上训练的专用模型，更能认识到其知识的局限性，并且更有可能表达不确定性，而不是捏造听起来合理但无意义的内容。
效率是另一个关键因素。通用大语言模型计算成本高昂，每次查询都需要筛选大量无关知识。专用模型更精简、更小、更快、更专注。这转化为更低的延迟和更少的硬件成本。
数据隐私与控制：受监管的行业不能冒险将敏感数据发送到外部 API。本地部署的领域专用模型允许组织利用大语言模型的能力，同时不损害机密性或合规性。

简而言之，专业化带来更高的准确性、更低的风险、更高的效率和更强的控制力。这些是工程师在任何生产系统中都重视的结果。

专用模型的影响在大型组织的真实部署中显而易见。这些案例展示了效率、准确性和成本节约方面的改进：

医疗文档记录：

斯坦福医疗保健实施了 Nuance DAX Copilot，这是一个在临床医患对话上微调的 AI 系统，用于自动化临床文档记录（链接）。该系统在患者就诊期间捕获对话并生成草稿摘要，减少了行政工作量。试点项目中的医生报告文档记录时间减少（从每天 90 分钟减少到 30 分钟以下），从而能更专注于患者护理（链接）。
金融服务：

摩根大通的 COIN（合同智能）平台（链接）使用大语言模型蒸馏进行法律文档处理。该系统审查以前每年需要约 360,000 人类小时的商业贷款协议（链接）。自推出以来，COIN 减少了审查时间和错误率。
边缘计算：

谷歌的 Gemini Nano 是针对移动设备的蒸馏模型（链接）。这个 16 亿参数的模型源自 Gemini Ultra，可在 Pixel 手机上运行，无需网络调用即可在 300 毫秒内提供智能回复、摘要和字幕功能，同时在占用不到 1 GB 内存的情况下保持了母模型的大部分质量。

除了在无数用例中的若干优势和应用外，微调和蒸馏的真正价值在于将两者结合。

蒸馏和微调如何帮助创建专用模型？

当你结合蒸馏和微调时，你可以创建既高度专用又高效的模型。微调赋予领域专业知识，而蒸馏确保这种专业知识可以在现实世界的、资源受限的环境中被访问。这种方法让你能够超越通用解决方案，交付定制化、高性能且为生产就绪的模型。

微调为学生模型赋予领域专业知识；蒸馏确保这种专业知识可移植、高效，并能在现实世界的约束条件（延迟、隐私、成本）下部署。

我们将在本章中使用这种方法为法律领域创建一个专用模型，并将其用于多标签分类任务。

在下一节中，我们将详细研究将用于与 DeepSeek 进行蒸馏和微调的用例和数据集。

用例和数据集

为了学习如何使用微调和蒸馏创建专用模型，我们将专注于一个法律用例。但在我们开始之前，我们将概述本章用于驱动法律文本多标签分类问题的理由蒸馏方法。

本章中使用的约定：

教师模型：DeepSeek-R1（仅 API/推理）。
学生模型：Gemma 3（我们实际微调的模型）。
蒸馏：数据生成步骤。
微调：对学生模型的参数更新步骤。

为什么选择 Gemma（而不是 DeepSeek）？

直接微调 DeepSeek 对于我们的目标来说不切实际且不必要；行业标准模式是将大型、有能力的教师蒸馏到一个高效的开源学生模型中，以便你可以在本地运行。

法律文本中的多标签提取问题

合同审查是一个典型的高风险、高复杂性信息提取问题。法律专业人士平均花费约 50%的时间审查合同，主要律所的计费率为每小时 500 至 900 美元。对于单笔并购交易，这可能转化为数十万美元的法律费用。技术挑战不仅在于文本量，还在于需要准确识别和解释 41 种不同的条款类型——每种都有其法律和业务含义。这是一个多标签、多分类问题，其复杂性还在于相关信息通常埋藏在密集、异构且有时模棱两可的语言中。

“在干草堆中找针”的比喻常被用来描述合同审查，但这并未完全捕捉其复杂性。实际上，这个任务更像是在寻找 41 种不同类型的针，每种都有独特的特征和不同程度的风险。例如，考虑以下控制权变更条款：

“如果承包商发生控制权变更，其 50%或以上的有表决权证券转让给新的实际受益人，客户可以在提前三十天书面通知后终止本协议。”

对外行人来说，这似乎只是标准的法律样板文字。但对法律专家而言，它编码了一种特定机制，可能决定一项计划中的收购是顺利进行，还是会面临意外的合同复杂性。将这种复杂性乘以数百页和数十种合同类型，问题的规模就变得清晰了。AI 面临的挑战不仅是提取这些条款，更是正确地对其进行分类，并且理想情况下，为每个决策提供理由。

自动审查合同的业务案例非常引人注目：一个 AI 辅助系统可以将法律审查成本降低 60-70%，提高准确性，并提供完整的审计轨迹。更重要的是，它可以防止因遗漏条款而导致交易失败或产生意外责任的灾难性场景。

从工作流的角度来看，自动识别和解释有风险的条款，可以让初级员工专注于更高价值的分析，并让高级律师对系统的输出充满信心。其结果是一个更高效、一致且可辩护的审查流程。

接下来，我们将详细介绍技术方法：使用 DeepSeek-R1 生成理由，然后将这种专业知识蒸馏到更小、可部署的模型中——展示如何在生产环境中操作法律推理，而不牺牲透明度或准确性。

介绍 CUAD：一个用于法律条款分类的结构化基准

合同理解 Atticus 数据集（CUAD）正是为了应对这一挑战而创建的。它由 Atticus Project 开发，历时一年，由法学学生、执业律师和机器学习研究人员合作完成，包含超过 13,000 个专家注释，涵盖 510 份真实世界的商业合同。这些合同来自 SEC 备案文件，代表了法律专业人士实践中审查的文档类型。你可以在此探索该数据集：www.atticusprojectai.org/cuad。

CUAD 的价值在于其系统化的分类法：它定义了 41 种律师在企业交易中会一致标记的条款类型。例如，反转让条款（限制协议如何转让）、责任上限（限制损害赔偿责任）、管辖法律条款（确定争议的司法管辖区）以及竞争限制条款（可能阻止收购后的商业活动）。每个类别都代表一种特定的法律风险或义务，经验丰富的律师通过多年实践已学会识别。对于工程师而言，CUAD 提供了一个基于真实世界法律实践的、定义清晰的多分类基准。

扩展 CUAD：为何我们添加理由

CUAD 附带高质量标签，但没有每个示例的说明。为了使模型可审计并更容易微调，我们附加了教师生成的理由，并将这个增强后的语料库称为增强版 CUAD（详细信息稍后在数据预处理与增强版 CUAD 数据集部分出现）。

例如，一个反转让条款可能包含如下理由：

该条款被归类为反转让，因为它明确限制了分销商转让协议的能力。关键要素是：(1) 它明确禁止转让，(2) 它适用于整个协议和部分权利，(3) 它要求对方明确同意……

你可以直接使用增强版 CUAD 进行本章的微调步骤（huggingface.co/datasets/zenml/cuad-deepseek）。

我们的技术目标是将一个大型模型（DeepSeek-R1）的法律推理能力，蒸馏到一个适合本地部署的、资源高效的小型模型中。大多数律师事务所由于保密和合规限制无法使用外部 API，因此最终模型必须足够紧凑以便本地运行，同时又要足够准确和可解释，以供专业使用。

具体来说，我们的目标是一个 120 亿参数的模型，能够以>90%的准确率分类所有 41 个 CUAD 类别（加上一个用于范围外条款的NONE类），并为每个决策生成清晰、分步的解释理由。这是一个经典的知识蒸馏场景，附加的要求是可解释性。目标是创建一个能运行在笔记本电脑（或小型服务器）上，但推理能力堪比高级律师的系统——结合大语言模型的广泛知识与通过处理数千份真实世界合同所获得的专业知识。

使用 CUAD 和增强版 CUAD 数据集进行蒸馏微调过程概览

为了解决上述多标签分类问题，我们将遵循图 7.1所示的两阶段工作流。

图 7.1：法律理由蒸馏的两阶段工作流

阶段 1（蒸馏）：我们要求 DeepSeek-R1 为 CUAD 示例生成逐步解释，并将这些理由附加到数据中。教师模型不更新；它只生成丰富的训练信号，从而得到增强版 CUAD（数据）。（请注意，虽然我们在本章中解释并展示了如何创建增强版 CUAD 数据集，但我们也提供了一个包含完整处理数据的 Hugging Face 数据集，这样你就无需花费 100 多美元来重现蒸馏部分。）
阶段 2（微调）：然后我们将使用 Unsloth/LoRA 在增强版 CUAD 上训练 Gemma 3，使学生模型既能学习标签，也能学习解释结构。

其结果是一个更小、更快的模型，你可以在本地运行——其准确性和解释性符合法律审查的需求。

因此，我们将进行如下操作：

DeepSeek-R1（教师） → 在 CUAD 上生成理由 → 增强版 CUAD（理由增强） → 使用 Unsloth/LoRA 微调 Gemma 3（学生） → 评估学生模型与教师模型。

是时候开始了！我们将从本任务中将使用的工具开始介绍。

用于模型蒸馏的 LLMOps 工具

构建生产就绪的 AI 系统依赖于拥有正确的基础设施来可靠地开发、部署和维护它们。在 LLMOps（大语言模型运维）工具的背景下，这种基础由大语言模型运维（LLMOps）工具提供。这些工具帮助为可能杂乱无章的实验和部署过程带来结构化和可重复性。

在我们的蒸馏流水线（图 7.2）中，我们使用三个主要工具：用于编排和可观测性的 ZenML、用于高效训练的 Unsloth，以及用于统一模型访问的 LiteLLM。每个工具在推动我们的法律合同分类项目从研究实验走向生产就绪系统方面都扮演着不同的角色。

图 7.2：蒸馏和微调过程架构

对于我们的法律用例，我们需要精确跟踪哪些 CUAD 示例被增强了，哪个模型/版本在上面进行了训练，以及每次运行的性能如何——以便结果可以重现和审计。我们使用流水线来固定序列（加载 → 蒸馏 → 格式化 → 微调 → 评估 → 比较）并缓存昂贵的步骤。

以下是每个工具在本章工作流中的角色概述：

ZenML（编排与溯源）：运行端到端流水线（load_data → finetune_model → evaluate_model → compare_models），缓存昂贵的步骤，并记录输入、输出和配置以实现可重复性（docs.zenml.io）。
LiteLLM（教师 API 访问）：在蒸馏和教师基线评估步骤中，通过 OpenRouter 提供对 DeepSeek-R1 的单一调用接口（模型字符串和 API 密钥在环境中配置）（github.com/BerriAI/litellm）（openrouter.ai）。

有关使用 LiteLLM 进行设置的更详细说明，你可以查看本书末尾的附录。

Unsloth（高效 LoRA 微调）：附加 LoRA 适配器并使用优化内核训练 Gemma 3，以便我们可以在单个高端 GPU 上微调 12B 的学生模型（unsloth.ai）。

本节重点介绍这些工具如何适应工作流。

让我们查看代码，了解本章代码中每个工具出现的位置：

蒸馏和教师模型推理通过 LiteLLM 调用执行（用于阶段 1训练和教师基线评估），遵循标准的 LiteLLM 补全调用模式。

工具：LiteLLM + OpenRouter


import os, json

from typing import Dict

from litellm import completion

DEEPSEEK_MODEL = os.getenv("DEEPSEEK_MODEL", "openrouter/deepseek-r1")

OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY")

def generate_rationale(clause: str, label: str) -> str:

    system = "You are a senior contract lawyer. Explain why the clause matches the CUAD label. Return concise, stepwise rationale."

    messages = [

        {"role": "system", "content": system},

        {"role": "user", "content": json.dumps({"clause": clause, "label": label})},

    ]

    resp = completion(model=DEEPSEEK_MODEL, messages=messages, max_tokens=400, api_key=OPENROUTER_API_KEY)

    return resp["choices"][0]["message"]["content"].strip()

2. 使用 Unsloth LoRA 微调 Gemma 3 (第二阶段中的学生模型训练)


#### 工具：Unsloth

from typing import Any

from unsloth import FastLanguageModel

from trl import SFTTrainer, SFTConfig

def finetune_gemma(

    train_ds: Any, val_ds: Any,

    base_id: str = "unsloth/gemma-3-12b-it", out_dir: str = "ckpt"

) -> str:

    model, tokenizer = FastLanguageModel.from_pretrained(

        base_id, load_in_4bit=True)

    model = FastLanguageModel.get_peft_model(

        model, r=16, lora_alpha=32, lora_dropout=0,

        target_modules="all-linear",

        use_gradient_checkpointing="unsloth",

        random_state=3407, use_rslora=False,

    )

    trainer = SFTTrainer(

        model=model, tokenizer=tokenizer, train_dataset=train_ds,

        eval_dataset=val_ds,

        args=SFTConfig(

            per_device_train_batch_size=1,

            gradient_accumulation_steps=8,

            num_train_epochs=2, learning_rate=1e-4,

            lr_scheduler_type="cosine",

            warmup_steps=100, logging_steps=20),

    )

    trainer.train()

    model.save_pretrained(out_dir); tokenizer.save_pretrained(out_dir)

    return out_dir

3. 使用 `ZenML` 编排各步骤


#### 工具：ZenML

from typing import Tuple

from zenml import step, pipeline

@step(enable_cache=True)

def load_data(

    max_samples: int | None = None, filter_none: bool = True

) -> Tuple[list, list]:

    # 加载 Hugging Face 数据集，根据需要生成/附加推理解释，并转换为 Gemma-3 聊天格式

    # 返回 train_ds, val_ds（已分词/准备就绪）

    ...

@step

def finetune_model(train_ds: list, val_ds: list, model_size: str = "12b") -> str:

    # 调用上面的 finetune_gemma(...) 函数 (Unsloth)

    ...

@step

def evaluate_student_vs_teacher(...) -> dict:

    # 在测试集分割上运行本地学生模型；通过 LiteLLM 调用 DeepSeek-R1 作为教师基准；计算评估指标

    ...

@pipeline

def distill_finetuning(max_samples: int | None = None, model_size: str = "12b"):

    train_ds, val_ds = load_data(max_samples=max_samples, filter_none=True)

    ckpt_path = finetune_model(train_ds, val_ds, model_size=model_size)

    _ = evaluate_student_vs_teacher(...)

随着工具基础构建完毕，我们现在可以深入探索我们蒸馏流水线的实际实现了。第一阶段涉及增强 CUAD 数据集本身——利用我们的教师模型生成能够让学生模型更具可解释性的解释性推理解释。

法律推理解释蒸馏的两阶段工作流程

好吧，亲爱的读者，我们深知你已迫不及待想要开始。废话不多说，让我们直接进入蒸馏环节。

阶段 1：蒸馏

让我们从创建增强版 CUAD 数据集的过程开始。目标是在保留原始人工标签的同时，添加来自教师模型（DeepSeek-R1）的清晰、可审计的“为什么”解释。您可以在代码仓库的 Chapter07 文件夹中找到此过程的完整源代码，文件名为 standalone_synthetic_generation.py。

数据集划分概览

划分：80% 训练集 / 10% 验证集 / 10% 测试集（按标签分层，seed=42）。
训练：仅使用训练集分割（增强推理的）。
验证：仅使用验证集分割。
评估：仅使用测试集分割（留出集；无数据泄露）。
NONE 标签：在本章的训练运行中默认被过滤。

具体过程如下：

下载 CUAD（一次，幂等操作）：我们获取并解压官方的 CUAD 压缩包（来自 Zenodo）到一个稳定的文件夹中，以保持路径可预测（zenodo.org/records/4595826/files/CUAD_v1.zip?download=1）。
```
def download_and_extract_cuad(url, data_dir="data", force=False) -> str:

    # 流式下载（带进度） -> data/CUAD_v1.zip

    # 解压到 data/CUAD_v1/，使得 CUAD_v1.json 位于固定路径

    ...

    return os.path.join(data_dir, "CUAD_v1")
```
作用与重要性：此步骤使流程可重复：一个单独的函数，要么重用现有的本地副本，要么在 force=True 时干净地重新下载。后续的可重复性依赖于稳定的文件位置。

将 SQuAD JSON 转换为条款记录：CUAD 以 SQuAD 格式发布。（对于我们的目的而言，具体格式并不特别重要。通常，数据集会以各种格式存在，而我们的任务就是设法将它们整理成我们工作所需的形式。）我们将 CUAD 数据集规范化为紧凑的记录：确切的条款、其周围的上下文窗口、标签和合同元数据。


def extract_clauses_from_squad(

    squad: dict, context_window: int = 150

) -> list[dict]:

    items = []

    for doc in squad["data"]:

        title = doc.get("title", "")

        ctype = (title.split("-")[-1].strip()

            if "-" in title else "Unknown")

        for para in doc.get("paragraphs", []):

            ctx = para.get("context", "")

            for qa in para.get("qas", []):

                if qa.get("is_impossible"): continue

                label = (qa.get("id","").split("__")[-1] or "UNKNOWN")

                for ans in qa.get("answers", []):

                    start, text = ans["answer_start"], ans["text"]

                    end = start + len(text)

                    span = ctx[

                        max(0, start-context_window): min(len(ctx),

                        end+context_window)]

                    items.append({

                        "clause": text,

                        "clause_with_context": span,

                        "label": label,

                        "contract_name": title,

                        "contract_type": ctype,

                    })

    return items

作用与重要性：我们保留黄金标签，并添加一个有界上下文以帮助教师模型以及（之后的）学生模型进行推理。使用字符窗口避免了分词器的耦合；如果您愿意，可以稍后切换到基于 token 的窗口。

从无标注间隙中添加 NONE 负样本（可选）：为了提高鲁棒性，我们可以通过从间隙（未被任何标注跨度覆盖的文本区域）中采样来挖掘负样本。


def add_negative_none_examples(

    items: list[dict], squad: dict,

    ratio: float = 3.0, neg_len: int = 100,

    context_window: int = 150

):

    # 1) 记录每个段落中已标注的 (start, end) 跨度

    # 2) 计算已标注跨度之间的间隙

    # 3) 从间隙中采样固定长度的片段 -> 标签="NONE"

    # 4) 限制每个间隙的采样数量 (≤3)，以免负样本占主导

    ...

    return items_with_negatives

数据划分与提示词构建

这一部分为后续流程添加了全面的覆盖。本章中使用的收益、风险与默认选择详见后续的 NONE 标签决策 章节。

创建分层划分（并具备优雅回退机制）：我们将数据划分为训练集、验证集和测试集。这可以确保在评估模型性能时，我们不会产生虚假的信心感。此划分既用于训练过程，也用于后续评估。
```
def split_train_val_test(items, train=0.8, val=0.1, seed=42):

    # 按标签对训练集与临时集进行分层

    train_set, temp = train_test_split(

        items, train_size=train,

        stratify=[x["label"] for x in items], random_state=seed)

    # 计算验证集的相对大小；如果每个类别至少有 2 个样本则分层，否则进行普通划分

    ...

    return train_set, val_set, test_set
```
作用与重要性：将数据分层划分为训练集和测试集，在可能的情况下保持了标签比例，并在少数类别样本极少时能够优雅地回退。我们采用 80/10/10 的分层划分，seed=42。

泄漏规则：我们绝不在测试集（或其推理说明）上进行训练。
为教师提示生成推理说明（推理蒸馏）：我们要求解释现有标签的原因（而非生成新标签）。这是有意为之：我们是在为已知的分类蒸馏推理过程，而非重新标注 CUAD 数据集。
```
def create_prompt(sample: dict) -> str:

    return f"""[[TASK]]

    Explain why this clause fits the label.

    [[CONTEXT]]

    CLAUSE: {sample["clause"]}

    CLAUSE WITH CONTEXT: {sample["clause_with_context"]}

    CONTRACT TYPE: {sample["contract_type"]}

    CLASSIFICATION: {sample["label"]}

    [[INSTRUCTIONS]]

    Identify the clause language that justifies the label. Be concise, stepwise, and use Markdown."""
```
作用与重要性：以黄金标签为条件可以使输出紧扣任务且成本更低。另一种 标签与推理模式 可以捕获错误标签，但成本更高，且存在需要您裁决的分歧风险。

协调生成过程（并行、速率限制、可恢复）：我们确保 LLM 的生成过程是并行的，并且任何错误或速率限制都能被捕获，以便我们可以自动重试。


@backoff.on_exception(

    backoff.expo, (Exception,), max_tries=5, max_time=300)

def generate_once(

    client, prompt, model="deepseek/deepseek-r1", max_tokens=1500,

    temperature=0.2, top_p=0.95

):

    # OpenRouter OpenAI 兼容的 chat.completions.create(...)

    return content, reasoning_trace_or_none

class RateLimiter:

    def __init__(self, qps: float): ...

    def wait(self): ...

def process_one(idx, sample, client, limiter, out_path, lock):

    limiter.wait()

    rationale, trace = generate_once(client, create_prompt(sample), ...)

    rec = {**sample, "rationale": rationale,

        "reasoning_trace": trace, "index": idx, ...}

    with lock:

        append_jsonl(out_path, rec)

作用与重要性：并行工作器和全局 每秒查询数 (QPS) 限速器提供了高吞吐量且无节流错误；指数退避 处理瞬时故障；线程安全追加 防止写入交错。在实践中，这使得整个流程可靠且成本更低（OpenRouter 文档：openrouter.ai/docs）。

安全恢复（跳过已完成索引）：由于此过程会产生实际成本、执行时间长且可能被中断，我们跟踪已完成的项目，以便在希望暂停或恢复时能够做到。
```
def get_processed_indices(path) -> set[int]:

    # 读取输出的 JSONL；收集“index”字段 → 允许恢复

    ...
```
作用与重要性：您可以随时停止和恢复。这在分批处理成本或轮换 API 密钥时至关重要。
打包与发布：我们为每个划分存储 JSONL 文件，并可选择将 DatasetDict 推送到 HuggingFace Hub (huggingface.co)。
```
def to_hf_dataset(jsonl_path: str) -> Dataset: ...

def push_to_hub(

    dataset_or_dict, repo_id: str, private=False

) -> None: ...
```
作用与重要性：一个一致、有文档的产出，其他人无需重新生成合成数据即可复用。

成本与吞吐量说明

在我们的参考运行中，为单个划分生成中等长度限制和低温度的推理说明，成本大约为 100 美元。为所有划分生成推理说明的成本成比例增加（通常为 2-3 倍，具体取决于令牌限制和覆盖率）。

轻量级质量检查（低成本但有效）：在接受推理说明之前，我们运行简单的过滤器：
```
def accept(sample: dict, rationale: str) -> bool:

    if rationale is None or len(rationale.strip()) < 50: return False

    if rationale.lower().count("this clause") >= 4: return False  # 通用填充文本

    if sample["label"].lower() not in rationale.lower()[:300]: return False  # 标签未在早期得到支撑

    return True
```
作用与重要性：这些启发式规则故意设计得很简单：它们拒绝低质量的文本，并鼓励基于标签的解释，而无需为额外的评判模型付费。对于更高要求的场景，可以将其替换为一个小的推理说明评判提示，以评分具体性并引用关键短语匹配。

一旦这些步骤完成，就该进入 ZenML 管道数据处理阶段了。

ZenML 管道数据处理

管道层通过一个专门的 load_data 步骤管理数据预处理，并记录每次运行的输入和输出，以确保可重现性。每次执行都会捕获所使用的数据集版本、预处理配置以及生成的产出物。

以下是管道的实现方式：


@step(enable_cache=True)

def load_data(

    model_size: str = "12b",

    dataset_id: str = "zenml/cuad-deepseek",

    max_samples: Optional[int] = None,

    filter_none_labels: bool = True,

) -> Tuple[pl.DataFrame, pl.DataFrame, HTMLString]:

    """加载并预处理增强型 CUAD 数据集。"""

    # 获取模型配置并创建分词器

    config = get_model_config(model_size)

    tokenizer = create_tokenizer(config)

    # 下载并准备数据集

    download_and_save_dataset(

        dataset_id=dataset_id, output_dir=output_dir)

    train_dataset, val_dataset = load_datasets(

        max_samples, tokenizer, tokenize=False, filter_none_labels

    )

    return train_dataset.to_polars(), val_dataset.to_polars(), viz

数据准备流程

当此步骤运行时，它会执行以下操作：

将 model_size 解析为已知配置，并构建匹配的分词器，这样所有下游格式化都能使用正确的聊天模板和序列限制。
在稳定目录 output_dir 中获取或复用请求的增强型 CUAD 数据集 (dataset_id)，以确保特定修订版本被本地化为实体。
通过 load_datasets(...) 准备训练/验证集划分，该函数会应用您的开关参数，例如用于快速实验的 max_samples，以及用于包含/排除开放式 NONE 类别的 filter_none_labels，同时除非您启用它，否则会特意推迟分词处理（我们在此处将样本保留为字符串，以便在训练时进行一致的打包或修剪）。
将生成的划分转换为紧凑的 Polars 数据框，这样它们可以快速序列化为工件，并且便于分析。
生成一个小型 HTML 可视化图表，记录加载的内容（计数、过滤策略），并成为运行过程的一部分。

请注意，由于我们设置了 @step(enable_cache=True)，ZenML 会根据步骤代码、参数和上游工件计算缓存键；使用相同输入重新运行时，会复用已实体化的训练或验证数据框和可视化图表，而更改任何输入（例如，模型大小、数据集 ID、过滤策略或样本上限）会使缓存失效并重新计算。最终效果是实现可复现的数据准备，并支持快速、参数感知的迭代。

一个实际的好处是自动缓存：一旦某个步骤用相同输入运行过，编排器就可以复用其输出。这减少了迭代时间，而无需更改任何训练逻辑。在我们的示例中，我们使用 ZenML 来提供这些行为，但任何具有步骤缓存和运行元数据的编排器都能以类似方式工作。

对于此数据集，一个关键的预处理考虑是如何处理 NONE 分类样本。让我们来讨论它们。

NONE 标签的决策

这些是与 41 个预定义 CUAD 类别都不匹配的条款。实际上，它们充当了主要模式之外内容的兜底分类。包含 NONE 样本可以提供有用的负例训练数据，但它也给分类任务引入了额外的复杂性。

两个实际的权衡点在于负例比例和上下文窗口：

负面比例：当启用时，我们默认设置负例:正例为 3:1；增加此比例可以提高实际场景中的精确度，但会减慢训练速度，并可能导致模型偏向于预测 NONE。建议从 1-3 倍开始，并通过验证集 F1 值进行调优。
上下文窗口：为了速度和分词器独立性，我们使用字符窗口（例如，每侧 150 个字符）。如果您观察到截断的标记或混淆了教师/学生的断句，请切换到支持标记感知的窗口（例如，通过分词器），并将总序列长度保持在您的训练预算内。

NONE 类别并非原始 CUAD 数据集的一部分。最初的焦点是识别法律专业人士在合同审查过程中关心的具体条款类型。我们在增强过程中引入了 NONE 样本，以帮助平衡数据集，但这一添加改变了分类问题的本质。

对 NONE 样本进行分类与在特定法律类别间进行分类有本质的不同。模型必须学会识别任何已定义法律模式的缺失，这是一个比将条款匹配到已知类型更抽象、更开放的任务。例如，这类似于要求模型识别“非猫”，而不是区分狗的品种。

出于本章的目的，我们默认使用 filter_none_labels=True 参数排除 NONE 样本。这使我们能够专注于区分有意义法律类别的核心挑战，而没有“其他一切”类别带来的额外复杂性。在生产环境中，您可能会选择包含 NONE 样本以提高鲁棒性，但对于学习和演示来说，一个更聚焦的方法通常更可取。

数据准备包括智能过滤：


def prepare_dataset_from_file(

    jsonl_path,

    max_samples=None,

    tokenizer=None,

    filter_none_labels=False,

):

    """Process JSONL data with enhanced instructional format."""

    data = []

    none_count = 0

    with open(jsonl_path, "r") as f:

        for line in f:

            item = json.loads(line)

            # Skip NONE examples if filtering is enabled

            if filter_none_labels and item.get("label", "").strip().upper() == "NONE":

                none_count += 1

                continue

            # Process valid examples...

一旦我们下载并过滤了数据，接下来就需要对其进行格式化，为微调做好准备。

接下来，我们将为微调学生模型提供指令格式。

微调的指令格式

数据预处理还处理为指令微调模型格式化示例的关键任务。（所谓指令微调，指的是专门训练以遵循用户指令并提供有帮助回应的语言模型，而不仅仅是像基础模型那样预测序列中的下一个词。）我们不再使用简单的输入-输出对，而是将数据集中的每个示例构建为详细的对话，教导模型不仅要分类什么，还要如何推理分类过程。这里有一个示例：


{

    "clause": "access is reasonably required by the other, including without limitation, for audit, accounting and litigation purposes.",

    "clause_with_context": "ther (other than data and information subject to any attorney/client or other privilege), insofar as such\n\n                                       27\n\naccess is reasonably required by the other, including without limitation, for audit, accounting and litigation purposes.\n\n     (c) Notwithstanding the foregoing, either party may destroy or otherwise dispose of any information at any time in accordance with the corporat",

    "label": "Audit Rights",

    "contract_name": "FIDELITYNATIONALINFORMATIONSERVICES,INC_08_05_2009-EX-10.3-INTELLECTUAL PROPERTY AGREEMENT",

    "contract_type": "INTELLECTUAL PROPERTY AGREEMENT"

}

每个训练样本都会被转换为一个 JSON 格式的指令，其中包含任务描述、分类体系、有效的标签选项以及带有上下文的输入条款。模型被训练来生成结构化的推理过程，然后给出最终的分类结果——这正是我们在部署系统中所期望的行为。

这种指令式的方法有助于提供法律专业人士可以评估和信任的推理过程，而不仅仅是一个输出标签。预处理流水线会自动处理这种格式化，确保所有训练实例的一致性，同时保留灵活性以尝试不同的指令提示和响应格式。如果你不太关注用于微调的数据格式，你的模型可能根本无法工作。

现在，我们将使用在此阶段生成的推理增强的增强版 CUAD 数据集来训练学生模型（Gemma 3）。

阶段 2：在 `CUAD` 上微调 `Gemma 3`

准备好我们丰富的数据集并建立起推理基础后，我们现在将转向核心挑战：将这种法律专业知识蒸馏到更小的、可部署的模型中，使其能够在内部部署法律环境的限制下高效运行。这就是模型选择变得至关重要的地方——我们需要一个在能力与实用性之间取得平衡的架构。

为什么选择 Gemma 3？

Gemma 3 (https://ai.google.dev/gemma/docs/core) 是一种现代、资源高效的语言模型架构，继承了 Google Gemini 2.0 系列的核心创新，但专为单 GPU 部署而设计。对于构建法律文档分类器的技术团队来说，Gemma 3 提供了性能和可部署性的强大平衡。

与大多数需要多 GPU 集群或分布式推理的 LLM 不同，Gemma 3 被设计为在单个高端 GPU 上运行推理，甚至可以对 12B 参数变体进行训练。这是通过架构优化实现的，最显著的是局部与全局注意力层的 5:1 比例。这种设计选择显著减少了长上下文推理期间的 键值 (KV) 缓存 内存需求。例如，传统的 Transformer 架构可能会随着上下文长度增加而出现 60% 或更高的内存开销增长，而 Gemma 3 将这一增幅控制在 15% 以下。这使得在不需要专门硬件或分布式系统的情况下，实际处理长法律文档成为可能。

为什么不直接微调 DeepSeek？

对于这个用例，将 DeepSeek 的行为蒸馏到一个你可以微调和本地运行的、开放、高效的学生模型中，在操作上更有优势。这避免了繁重的计算需求，并简化了部署/治理，同时保留了教师模型大部分的法律推理能力。

对于我们的用例——涉及跨越数千个令牌上下文窗口的合同条款分类——这意味着我们可以将 12B 参数模型部署在标准工作站硬件上，而不会牺牲像 DeepSeek-R1 这样更大但效率较低模型所展示的推理深度或准确性。其结果是一个既高性能又在运营上可行的生产级法律 AI 系统，即使在要求严格数据隐私的环境中也能用于内部部署。

模型规模决策：为什么我们关注 12B？

虽然我们的流水线支持三种模型规模（1B、4B 和 12B 参数），但我们在增强版 CUAD 数据集上的实验揭示了一个明确的事实：对于法律合同分类，你需要相当大的模型容量才能达到专业级的准确性。

为什么小模型不够：1B 和 4B 的 Gemma 3 模型，虽然计算效率高，但无法达到严肃法律工作所需的准确性水平。1B 模型在复杂法律语言中难以保持一致的推理，尤其是在确保 JSON 输出时；而 4B 模型虽然稍好，但在高风险合同审查场景中，其分类错误仍然太多，无法保证可靠性。
12B 模型：这是我们的主要关注点。因此，在我们的示例和分析中，我们主要关注 12B 模型。虽然它在训练期间需要 24 GB+ 的 GPU 内存，但它代表了执行此类专业法律分类任务的最低可行规模。即便如此，12B 模型也并非完美——法律语言的细微差别和上下文依赖性对即使是复杂的模型也构成挑战。
部署权衡：欢迎你使用我们的流水线尝试较小的模型（只需更改 model_size 参数），它们可能适合初步文档筛选或资源受限的环境。然而，对于法律实践所要求的准确性水平，12B 模型的计算开销通常由其在复杂合同语言上的卓越性能所证明。

让我们开始微调过程。

微调过程

以下代码定义了 distill_finetuning 流水线。


@pipeline

def distill_finetuning(

    max_samples: Optional[int] = None,

    model_size: str = "4b",

    filter_none_labels: bool = True,

):

    """Pipeline to load the dataset and finetune the model.

    Args:

        max_samples: Maximum number of samples to use for training

        model_size: Model size to use ('1b', '4b', or '12b')

        filter_none_labels: Whether to filter out 'NONE' classifications

    """

    # Step 1: Load and prepare data with intelligent caching

    train_dataset, val_dataset, data_viz = load_data(

        model_size=model_size,

        filter_none_labels=filter_none_labels,

        max_samples=max_samples

    )

    # Step 2: Fine-tune with automatic resource management

    finetune_model(

        train_dataset,

        val_dataset,

        model_size=model_size

    )

微调法律分类模型需要平衡学习效率和稳定性的超参数。我们的配置反映了在编写本章实验过程中吸取的经验教训。


# Optimized training configuration for 12B legal model

TRAINING_CONFIG = {

    "per_device_train_batch_size": 1,

    "gradient_accumulation_steps": 8,

    "warmup_steps": 100,

    "num_train_epochs": 2,

    "learning_rate": 1e-4,

    "logging_steps": 20,

    "optim": "adamw_8bit",

    "weight_decay": 0.01,

    "lr_scheduler_type": "cosine",

    "seed": 3407,

}

以下是我们选择这些参数的理由：

学习率：1e-4的学习率比典型的微调学习率更为保守，这反映了在处理专业法律文本的大型模型时对稳定性的需求。12B模型庞大的参数空间需要谨慎选择学习率，以避免破坏预训练知识，同时又能发展法律专业能力。
内存优化的批处理配置：减小的批处理大小（1）适应了12B模型的内存需求，同时增加的梯度累积步数（8）保持了有效的批处理大小为8。这种配置平衡了训练稳定性与单 GPU 训练大型模型的实际限制。
为稳定性设计的扩展预热：100步的预热期为大模型提供了额外的稳定性，使其能够逐步适应法律领域，而不是采用可能破坏已学习表征的激进早期更新。
余弦学习率调度：余弦调度器提供了平滑的学习率衰减，通过在整个过程中保持稳定的训练动态，它尤其适合较大的模型。

在优化器和调度器确定后，下一个决策是更新哪些参数。全量微调会更改每个权重，但这既耗费内存又无必要；相反，我们使用一种参数高效的方法，学习小型、有针对性的更新，同时保持基础模型不变。

用于参数更新的 LoRA 配置：低秩适应（LoRA）是一种无需重新训练整个大模型就能教授其新技能的方法。可以把基础模型想象成一件高端乐器：与其重建它，不如夹上一个微小附件，稍微调整它的演奏方式。在训练过程中，只有这些小附件在学习；原始模型保持不变。这使得微调更快、成本更低且可逆，并且即使在基础模型以4-/8 位加载时也效果良好（arxiv.org/abs/2106.09685）。

我们的LoRA配置针对法律语言理解所需的具体适应性。所谓具体适应性，是指我们选择在哪里放置这些小附件以及它们有多大，从而将额外的学习能力导向关键之处。

对于法律文本，这通常意味着在注意力和 MLP 模块中添加适配器，以便模型更好地处理长篇交叉引用、正式的条款语言、标题和定义，以及我们期望的结构化输出（理由和标签）。简而言之，我们将有限的训练能力集中在能够提高该领域准确性的模式上。具体方法如下：


# 为 12B 法律模型优化的 LoRA 配置

LORA_RANK = 16

LORA_ALPHA = 32

LORA_DROPOUT = 0

LORA_BIAS = "none"

以下是我们训练运行的示例损失曲线。

图 7.3：开发过程中一次示例训练运行的损失曲线

我们来看一下这些优化意味着什么：

为更大模型增加的秩：秩为16的配置为12B模型有效捕获法律语言模式提供了额外的能力。较大的模型受益于更高的LoRA秩，因为它们拥有更多参数空间可供适应，同时与全量微调相比仍保持高效。
Alpha = 2 × 秩的惯例：遵循既定惯例，我们将 alpha 设为32（2 × 秩），这为LoRA适应提供了适当的缩放。这一比例在不同模型规模和领域已被证明是有效的。
零 Dropout 与全面覆盖：法律语言几乎没有模糊的余地，因此我们保留所有已学习的关联。通过对所有语言层的注意力和 MLP 模块进行微调，我们确保专业知识渗透到整个模型架构——包括语法解析和语义推理组件。具体来说，代码在constants.py中将LORA_DROPOUT=0，并启用finetune_language_layers、finetune_attention_modules和finetune_mlp_modules，然后在steps/trainer.py中通过 Unsloth 的FastModel.get_peft_model应用这些标志。这会在语言栈中的所有注意力和 MLP 模块上附加LoRA适配器，且不进行适配器 Dropout，确保领域更新端到端传播。

既然我们已经了解了参数设置，接下来将理解法律数据模型训练的具体细节。

法律 AI 学习的训练动态

法律 AI 训练遵循可预测的模式，揭示了模型如何内化领域专业知识。在最初的100个训练步骤中，模型通常表现出损失的快速下降，因为它们适应了法律词汇模式。损失曲线展示了典型行为：初始的陡峭下降，随后随着模型掌握条款分类逻辑，出现逐渐的改善。

训练内存使用量随模型规模可预测地增长——1B模型峰值在8-10 GB，而12B模型则需要24-28 GB，具体取决于序列长度。这类分类损失曲线通常呈现三个阶段：快速初始学习（步骤 1-50）、稳定改进（步骤 50-200）以及步骤200后的逐渐优化。注意，这通常适用于大部分机器学习。

现在，我们将微调后的学生（Gemma 3）与DeepSeek-R1教师在保留测试集上进行比较，以验证推理蒸馏和微调是否达到了预期的效果。

一旦你对模型进行了微调，验证其是否按预期运行至关重要。接下来，我们将评估我们的微调模型。

评估与结果

为了证明（附有证据）我们微调后的法律分类器确实比我们用作教师的庞大DeepSeek模型表现更好，请将此视为一场期末考试，我们的12B参数学生必须证明它已有效学习了法律推理，足以与它的600 多亿参数教授竞争。

数据划分策略提醒：我们在增强的CUAD测试划分（占数据集的10%）上进行评估，该划分严格独立于训练和超参数调优。

评估流水线运行两个并行推理过程：一个使用我们本地的微调模型，另一个通过OpenRouter的免费 API 查询DeepSeek-R1。两个模型都分析来自我们增强CUAD测试集的相同法律条款——这些数据在训练期间从未见过。


@pipeline

def evaluation_pipeline(

    num_samples: Optional[int] = 50,

    exclude_none: bool = True,

    model_size: str = "4b",

    use_local_model: bool = True,

    deepseek_max_workers: int = 5,

):

    """将微调模型与 DeepSeek 基础模型进行比较。"""

    # 加载保留的测试数据

    test_dataset = load_test_data(num_samples, exclude_none)

    # 评估我们的微调模型

    finetuned_results, finetuned_viz = evaluate_model(

        test_dataset, model_size, use_local_model

    )

    # 通过 API 评估 DeepSeek 基础模型

    deepseek_results, deepseek_viz = evaluate_deepseek_base(

        test_dataset, max_workers=deepseek_max_workers

    )

    # 生成综合比较

    comparison_results, comparison_viz = compare_models(

        finetuned_results, deepseek_results

    )

衡量模型性能的一个重要方面是指标的选择。虽然存在许多流行指标，如准确率和 F1 分数，这些常用于多标签分类任务，但它们可能不足以涵盖所有领域的用例。

让我们探讨将用于评估我们微调模型的性能指标。

性能指标

法律 AI 评估需要比典型分类任务更细致的指标。一个遗漏关键 责任上限 条款（数据集中可能的标签之一）的模型，其风险不同于偶尔误分类不那么重要条款的模型。我们的评估框架通过多个性能维度来捕捉这种复杂性。

总体准确率：所有 41 种法律条款类型的原始分类性能提供了基线指标。我们微调的 12B 模型准确率持续保持在 80% 以上 – 对于一个比 DeepSeek 小 150 倍的模型来说，这是显著的性能。
非 NONE 准确率：此指标侧重于实际条款识别，排除了其他所有内容类别。由于法律专业人士主要关心找到特定条款类型，这通常代表最实际相关的指标。
按类别分析：法律条款在复杂性和重要性上差异巨大。管辖法律 条款遵循可预测的模式，而 控制权变更 条款可能微妙地隐藏在复杂的语言中。我们的评估按条款类型细分性能，揭示了微调提供最大价值的领域。
精确率、召回率和 F1：这些指标有助于理解捕获所有相关条款（召回率）与避免误报（精确率）之间的权衡——两者在法律应用中都至关重要，因为遗漏条款可能代价高昂。

现在让我们看看结果。

评估结果

我们的比较评估（图 7.4）揭示了令人惊讶的结果，验证了蒸馏方法在法律应用中的有效性。微调模型不仅与 DeepSeek 竞争；它们在法律分类任务上显著优于它。

图 7.4：性能比较显示微调的 Gemma 3 模型达到 84.5% 的准确率，而 DeepSeek 为 41.5%，同时计算需求显著降低

结果表明，使用增强 CUAD 数据集的高质量合成数据进行专门训练，可以创建出性能显著超越大型通用系统的模型。

错误分析：理解模型局限性

两个模型都在某些类型的法律条款上表现不佳，但其失败模式以有启发性的方式不同。微调模型显示出集中的错误模式，只有 11 种独特的失败方式，而 DeepSeek 有 38 种。这表明失败模式更系统化且可能更易解决。

图 7.5：混淆矩阵揭示了微调模型在 200 个样本中正确分类 169 个，具有比教师模型更清晰的对角线模式

两个模型在以下方面都存在困难：

协议日期消歧：微调模型最常见的错误（6 个实例）涉及区分 生效日期 和 协议日期。
许可授予细微差别：几种错误类型（独占性、不可转让许可、关联公司许可）都归结为 许可授予 分类。
条款边界检测：两个模型偶尔都在重叠或复合条款上遇到困难。

但与基础 DeepSeek-R1 模型相比，微调模型在多个方面实现了改进：

消除解析错误：与 DeepSeek 的 12 个解析错误不同，微调模型产生了一致可解析的输出。
减少错误多样性：独特错误类型减少 71%（11 对 38），表明行为更可预测。
提高 F1 分数：75.3% 对比 69.8%，更有效地平衡了精确率和召回率。

我们的评估将指标和工件写入实验跟踪仪表板，因此结果在单次运行之外仍然存在。在我们的参考实现中，这是一个 ZenML 仪表板（docs.zenml.io），但 MLflow 或 Weights & Biases (W&B) 也可以达到同样的目的。重要的是记录以下内容：

模型/检查点标识符和谱系。
数据集哈希/分割规范和预处理版本。
按类别指标、混淆矩阵和错误切片。

图 7.6：我们在 ZenML 仪表板上的评估结果和训练流水线截图

本章的实现突出了以下内容：

量化改进：明确的指标显示了蒸馏带来的 2 倍性能提升。
错误模式分析：对剩余 31 个错误的详细分解指导了针对性的改进。
可复现的评估：从 unsloth/gemma-3-12b-it 基础模型到微调的完整工件谱系。

当然，性能总有提升空间。让我们看看你可以如何改进。

性能优化潜力

我们目前的结果，仅通过 2 个训练周期就取得，代表了一个坚实的基础而非性能上限。从 41.5% 到 84.5% 准确率的巨大提升表明，几种优化策略可以将性能推得更高。

针对性错误修复：由于只有 31 个错误集中在 11 种模式中，针对这些特定情况（特别是 6 个 生效日期/协议日期 混淆）的集中合成数据生成可以立即获得收益。换句话说，我们可以生成专注于微调模型最难识别的模式的示例数据。通过增加这些模式的样本强度，我们希望（重新）微调的模型表现会更好。
精确率-召回率平衡：召回率轻微下降（–2.8%）同时精确率显著提升（+11.4%），这意味着你可以在不重新训练整个模型的情况下调整操作点。从推理时调整开始：
1. 从验证集中选择按类别的决策阈值，使模型对高成本错误更严格，对漏报更宽松。
2. 为低置信度情况引入一个 拒绝/需审阅 或 NONE 路径。
3. 使用保留集校准置信度分数，使概率反映现实（大多数库提供简单的校准器）。如果这还不够，进行小的、针对性的训练更改：添加类别权重以解决不平衡问题，尝试更关注困难样本的损失函数，或轻微过采样/欠采样困难标签。

因为每次微调运行都花费时间和金钱，所以优先考虑阈值调整和校准，然后才是非常小的、预先计划的训练选项扫描。

多阶段训练：鉴于完全消除了解析错误，一种从输出结构开始，然后进展到细微分类的课程学习方法可以进一步提高已经很强的 84.5% 准确率。你可以使用分阶段的时间表，先教授简单技能，然后增加复杂性。
1. 阶段 0：先格式化：短上下文和高支持度标签以稳定结构化输出（有效 JSON，标签令牌）。
2. 阶段 1：完整的增强 CUAD，减去罕见/模糊案例。
3. 阶段 2：引入长上下文、复合条款，以及（可选的）NONE 类别。
4. 阶段 3：专注于顶级错误切片（例如 生效日期 与 协议日期）的困难负样本挖掘和近失误案例。然后，逐步增加序列长度和难度；在缩放难度以避免破坏早期增益时，保持适配器固定。
  - 集成方法：模型之间不同的错误模式（DeepSeek 有 38 种独特类型，而微调模型有 11 种）表明集成方法可以利用互补优势。从简单开始：
5. 种子/快照集成：使用不同的随机种子或数据洗牌训练相同的 12B LoRA 模型三次，并采用多数投票。
6. 混合容量集成：结合 12B 和 4B 学生模型；当它们不同意时，将案例发送给人类审阅者或教师模型。
7. 基于置信度的路由：如果学生置信度低，则回退到 DeepSeek-R1；否则，接受学生的答案。
8. 测试时增强：运行同一条款的两个视图（例如，不同的上下文窗口）并对结果进行投票。为控制成本，仅对最难的 10-20% 示例应用集成，如通过低置信度或先前错误切片所识别的。

这些结果验证了蒸馏的法律模型可以在实际约束内满足专业要求。准确率分数（超过 80%）相对于手动审查一致性是一个显著的改进。详细的推理能力使法律专业人士能够验证和信任系统的分析。

对于考虑采用 AI 的律师事务所来说，这些结果表明复杂的法律推理能力不需要大规模的基础设施投资。单个高端工作站就可以部署与最大可用系统相竞争的模型，同时完全控制敏感的客户数据。

接下来，快速总结本章的关键要点。

关键要点

我们从巨大的 6000 多亿参数模型到高效的 120 亿参数法律专家模型进行了一段非凡的旅程，这些模型可以在单个 GPU 上运行，同时提供具有竞争力的性能。这种从瑞士军刀到手术刀的转变表明，只要方法得当，专业化 AI 可以既强大又实用。

什么时候蒸馏和微调有意义？

法律合同分类挑战教会我们，当三个条件对齐时，蒸馏能提供卓越的价值：

专业领域要求：法律语言的运作规则不同于一般文本。诸如 time is of the essence 这样的短语具有特定的法律分量，而通用模型很难始终如一地理解。当你的领域有专业词汇、推理模式或监管要求时，微调就变得至关重要而非可选。
高质量训练数据可用性：我们使用 DeepSeek 生成的解释的增强 CUAD 数据集提供了使蒸馏成功的丰富学习信号。专家分类与详细推理的结合创建了捕获法律分析什么和为什么的训练样本。
部署约束：律师事务所不能将机密客户文档发送到外部 API。这个约束——在受监管行业很常见——使得蒸馏不仅有帮助，而且是必要的。能够在内部部署复杂的法律推理将蒸馏从一种优化技术转变为业务赋能器。

蒸馏还有助于以传统方法无法匹配的方式同时解决多个挑战：

基础设施民主化：将 DeepSeek 的法律专业知识转换为 120 亿参数模型意味着复杂的法律 AI 可以在律师事务所已有的硬件上运行——无需专门基础设施，无需持续 API 成本，没有外部依赖。
隐私保护：当整个 AI 系统在组织的安全边界内运行时，客户机密性得以保持。这不仅仅是一个技术优势——它通常是一个法律要求。
定制化潜力：微调模型可以适应特定的律师事务所写作风格、司法管辖区要求或客户需求。与基于 API 的解决方案不同，这些模型随着你的组织而发展。

另一个重要考虑是确保治理和可重现的结果。在本章中，我们依赖实验跟踪/编排层将临时脚本转换为可重复的工作流。在我们的示例中，这个层是 ZenML，但任何捕获运行、工件和谱系的可比系统都可以。我们需要的关键能力是合规文档（运行产生数据、代码、配置和指标的不可变日志）、可重现实验（预处理/训练/评估可以完全重新执行），以及模型治理（模型和数据集版本以与软件相同的严格程度进行推广）。

总结

在本章中，你通过解释蒸馏将 CUAD 转换为增强 CUAD 数据集。你将 SQuAD 注释规范化为条款级记录，可选地挖掘 NONE 负样本，创建分层分割，并提示 DeepSeek-R1（通过 OpenRouter）生成简洁的、基于标签的解释。你执行了带有安全恢复和简单质量检查的速率限制、退避优化的并行生成，然后打包结果并格式化示例用于指令调优聊天训练。

然后你使用 Unsloth 通过 LoRA 微调了 Gemma 3，选择了稳定的训练/LoRA 超参数，并在保留数据上使用准确率、F1 和按类别分析评估了学生与教师的表现。你还学习了如何通过编排/跟踪层捕获谱系和缓存以实现可复现性——以及如何在没有编排器的情况下，从单个文件脚本运行相同的工作流。这些技能超越了法律领域，推广到任何受益于解释增强监督的领域。

接下来，第 8 章 从训练转向部署。我们将部署完整的 DeepSeek 模型（V3/R1），比较自托管和托管选项，以及平衡延迟、成本、隐私和可靠性的推理设置。

获取本书的 PDF 版本和独家附加内容

扫描二维码（或访问 packtpub.com/unlock）。按书名搜索本书，确认版本，然后按照页面上的步骤操作。 | |
| 注意：请保留好您的发票。直接从 Packt 购买的无需提供发票。 |

8 部署 DeepSeek 模型

在上一章中，我们蒸馏并微调了更小的、特定领域的模型，你可以在适度硬件和严格的隐私边界内运行。这项工作针对小规模的效率和控制进行了优化。本章采取了补充步骤，将全参数 DeepSeek 模型（V3 和 R1）部署为可靠的生产服务。

部署是从研究到生产的桥梁。它迫使对内存占用、吞吐量和运行风险做出具体选择。DeepSeek 的架构放大了这些权衡：V3 的 专家混合 (MoE) 使 VRAM 放置紧张；R1 的扩展推理增加了令牌数量和首令牌时间。正确的路径取决于你的约束。

截至 2025 年 8 月，本章中的任何定价和吞吐量数字都是方向性的。在硬编码假设之前，请验证当前的提供商表格并在你的工作负载上进行验证。

本章为在真实场景中部署全参数 DeepSeek 模型提供了实用手册。我们将重点部署全规模的 DeepSeek-V3 (671B) 和 DeepSeek-R1；较小的模型仅作为开发代理被提及。

我们将在本章中涵盖以下主题：

DeepSeek 部署全景
选择部署策略的决策框架
硬件和推理优化需求
实践部署指南
生产运营和监控
你的部署手册

范围说明

本章不涵盖部署第 7 章 中提到的较小微调/学生模型。任何对较小模型的提及严格用于开发或代理工作流，而非全模型的生产部署。

技术要求

在本章中，我们将探索部署 DeepSeek 的各种方法。对于每种方法，技术要求将在相应部分提及。你可以利用最适合你的应用程序以及对工具和平台访问权限的任何方法。

DeepSeek 部署全景

在生产中，你关注三件事：监控、扩展和成本。你监控每个请求的令牌数、首令牌时间 (TTFT)、p95/p99 延迟、GPU 和键值 (KV) 缓存使用情况，以及错误/超时率。你根据系统实际生成的令牌数量（而不仅仅是到达的 HTTP 请求数量）进行扩展。并且你通过限制令牌数、在可能的情况下批处理、缓存常见前缀以及在空闲时暂停或调整副本大小来控制成本。

为此，你需要关注以下内容：

内存占用：这是运行时所需的 VRAM。它包括模型权重加上开销，如 KV 缓存和激活。跟踪峰值 VRAM 和每令牌的 KV 缓存字节数；这些数字决定了你的安全批量大小和上下文长度，以及你是否需要多个 GPU 或将部分权重移动到 CPU/NVMe。
吞吐量：这是你的服务在目标并发下每秒可以生成的令牌数。对于规划，还要报告在固定提示/响应大小下的每秒请求数，并始终将其与 TTFT 和 p95/p99 延迟配对，以便该数字反映真实的用户体验。
运行风险：这是由于不稳定或设置不当而错过 SLO 或超出预算的可能性。关注错误和超时率、内存不足 (OOM) 事件、队列增长以及你安全地推出更改或上下扩展的程度。

DeepSeek 增加了一些额外的复杂性。V3 使用 MoE 设计，当所有专家都保留在 GPU 内存中时效果最佳，这推高了 VRAM 需求，即使进行量化也是如此；将权重移动到 CPU 或 NVMe 有助于适应但会减慢速度。R1 思考很多，通常在最终答案之前生成数千个内部令牌，这提高了 TTFT，增加了内存使用，并使延迟更不可预测。因此，在你选择 API、托管或自托管部署之前，明确的容量规划、仔细的批处理和基本的仪表板是必须的。

DeepSeek 的部署特点

DeepSeek 模型带来了特定的挑战：

模型规模：即使是量化版本也推高了硬件极限。DeepSeek-V3 的 671B 总参数需要大量的 VRAM，无论其 MoE 效率如何。
推理开销：R1 模型生成广泛的内部推理链。一个看似简单的查询可能产生数千个中间令牌。
架构考虑：V3 的 MoE 设计通常将所有专家保留在 GPU 内存中以获得性能；然而，这不是硬性要求。现代堆栈可以将权重卸载/分区（CPU/NVMe，跨节点张量并行），但代价是显著的延迟/吞吐量损失。

这些挑战需要针对大规模模型部署的特定工具和技术。让我们首先研究为什么你可能自己部署 DeepSeek 而不是使用官方 API。

为什么要自部署以及 DeepSeek 的独特之处？

通常，你会遇到选择自部署 DeepSeek 或任何其他最先进 LLM 的场景。例如，假设你正在处理敏感的法律文件，如合并协议——官方的 DeepSeek API 可能不是一个选择。不是因为它不够好，而是因为客户合同通常禁止将特权文件发送给第三方服务。这个场景说明了为什么自部署不仅仅是一项技术实践。它通常是一种商业必要性。

让我们谈谈你可能想要自部署的一些关键原因：

控制和定制：控制不是可有可无的；它是你如何将通用模型转变为适合你领域、风险状况和产品约束的系统的方式。自部署允许你接触 API 不会暴露的层（分词、路由、适配器、系统提示和安全策略），因此模型反映的是你的要求而非供应商的默认值。当 Bloomberg 构建 BloombergGPT 时，他们不仅仅想要一个金融语言模型；他们需要修改分词器以处理金融术语，并将一半的训练语料库保密。这种程度的控制是基于 API 的服务无法实现的。

对于 DeepSeek 部署，控制意味着以下内容：

调整超出 API 限制的生成参数。
合并特定领域的适配器（例如 LoRA）或专用适配器。
实现适合你领域的自定义安全过滤器。
无供应商限制地修改系统提示。
规模成本：成本不仅仅是一张价格表；它是利用率、批处理和缓存的函数。如果你的流量稳定或高容量，拥有服务堆栈可以让你分摊硬件成本并提高利用率，通常使有效的每令牌成本远低于按计量的 API。在规模上，经济性会急剧转变。对于每天处理数百份合同的法律文档工作流，使用以下方向性成本（截至 2025 年 8 月，硬编码前请验证）：
- API 成本：每百万令牌约 $30–50
- 自托管（分摊）：每百万令牌约 $2–5
- 盈亏平衡点：通常在 10-20M 令牌/月左右
隐私和数据驻留：对许多组织来说，隐私不是可选的；它是合同和法规要求的。自部署将敏感令牌保留在你的边界内，执行数据驻留和保留策略，并提供对数据流向的审计能力。例如，波恩大学医院部署了本地 LLM 用于胸部 X 光报告，专门是为了遵守德国数据本地性规则。本地推理处理所有敏感的医疗数据，无需云端出口或第三方处理。

在法律领域，要求更为严格：

律师-客户特权禁止大多数外部处理。
财务文件可能引发 SOX 合规问题。
欧盟数据驻留要求（GDPR）。
特定行业的法规（如 HIPAA 和 PCI-DSS）。
一些法律团队试验本地 LLM 以避免内容过滤器可能将特权材料标记为不当内容。
性能：延迟和吞吐量是塑造用户体验、SLO 甚至单位经济性的产品特性。自己运行堆栈消除了网络往返，并允许你针对你的硬件和工作负载定制服务引擎，释放比一刀切 API 更低的 TTFT 和更高的 tokens/sec。截至 2025 年 8 月，vLLM 基准测试报告与之前最先进的系统相比有 2–4 倍的吞吐量提升（arxiv.org/abs/2309.06180），与 Hugging Face Transformers 相比高达 24 倍（blog.vllm.ai/2023/06/20/vllm.html）并支持连续批处理。

自托管的性能提升如下：

消除 API 往返延迟（通常 50–200ms）。
实现连续批处理以获得更高吞吐量。
针对你的特定硬件配置进行优化。
在本地缓存常见的提示前缀。
特定用例：西太平洋银行（Westpac）等主要银行已与 Kasisto 合作（www.westpac.com.au/about-westpac/media/media-releases/2022/23-august/）部署银行工作流的特定行业 LLM；这些部署通常在银行的云边界内运行。

虽然自部署的动机广泛适用，但在你配置该 GPU 集群之前，DeepSeek 的架构创造了值得理解的独特挑战。

你应该注意的关于不同模型的一些 DeepSeek 特定部署考虑因素如下所列：

DeepSeek-V3 内存需求和放置：V3 的 MoE 设计在理论上是优雅的：671B 总参数，但每个令牌只有 37B 活跃。在实践中，这是一个内存管理难题。为了获得最佳性能，团队将所有专家保留在 GPU 内存中。将权重卸载或流式传输到 CPU/NVMe 或跨节点分片是可能的，但这会带来巨大的延迟和吞吐量损失。有关整合的 VRAM 和 GPU 规模（包括 INT4 占用空间和 H100 数量），请参阅关于硬件和推理优化引擎的部分。
报告的 4 GPU 部署通常针对较小的变体，或依赖于对 CPU/NVMe 的重度卸载/权重流式传输，这会引入大的减速。
有关整合的 VRAM 和 GPU 规模（包括 INT4 占用空间和 H100 数量）、容量规划和余量指南，请参阅关于硬件和推理优化引擎的部分。
DeepSeek-R1 推理密集型生成：R1 的推理链带来了不同的挑战。那些使 R1 如此强大的 `` 令牌也使内存使用和延迟激增（长思考阶段，尖峰内存）。一个简单的问题可能在实际答案之前生成数千个推理令牌。

那么这在实际环境中意味着什么？例如，使用 DeepSeek-V3 的 多头潜在注意力 (MLA)，KV 缓存占用空间约为每令牌 70 KB（≈每 10k 令牌 0.7 GB），远低于传统注意力 KV 大小，但对于容量规划仍然是首要的。R1 的公开文档没有发布每令牌的 KV 大小；内存和延迟将取决于推理深度：

由于思考阶段，TTFT 可能明显更高；具体的 TTFT 取决于工作负载和服务堆栈（没有广泛引用的 >30s 基准）。
内存使用是不可预测的；它取决于推理复杂性。
吞吐量通常由思考阶段主导。

截至 2025 年 8 月，完整 R1 的公开 H200 数字因堆栈和工作负载而异。NVIDIA 在早期 NVIDIA 推理微服务 (NIM) 预览中报告了 8× H200 约 ~3,872 tokens/sec；后来的 Blackwell (8×) 营销声称达到 ~30k tokens/sec。对于你自己的用例，请务必将供应商数字视为营销数字，直到在你的工作负载上进行验证，并在硬编码容量假设之前验证当前数字。无论哪种方式，单个请求可能仅用于内部推理就需要 5,000+ 令牌。

这些挑战并非旨在阻止自部署。它们旨在为你做好准备。本章重点部署全参数 DeepSeek-V3 (671B) 和 DeepSeek-R1 模型；虽然较小的蒸馏或微调变体对于原型设计很有用，但它们不是这里的重点。关键点是，这并不自动意味着你应该自部署以省钱。自托管主要在合规性要求时或你拥有持续、高令牌量和强大 GPU 利用率时才能收回成本；否则，API 或托管服务通常更便宜、更简单，直到你测量的工作负载超过你验证的盈亏平衡点。

关键是将你的部署策略与你的实际需求匹配：

需要处理敏感文件？自部署可能是强制性的。
每天处理数百万令牌？经济学上倾向于自托管。
需要低于 100ms 的延迟？具有优化服务的本地部署是最好的。
只是在试验？从 API 开始，需要时迁移。

让我们引导你通过一个框架来系统地做出这些决策，将部署从一项艰巨的挑战转变为一系列可管理的权衡。

选择部署策略的决策框架

部署选择主要取决于我们在上一节讨论的关键驱动因素：控制、成本和复杂性。你将发现自己需要在这三者之间进行权衡并找到合适的平衡点。

根据你的用例，你可以选择以下三种部署策略之一：

API 方法最大程度降低复杂性，但成本多变且控制较少。
托管服务提供中等复杂性和企业功能，但单位成本更高。
DIY（自托管）部署最大化控制并在规模上实现最佳单位成本，同时施加最高的运营负担。

让我们构建一个框架来帮助你明智地选择，提供一个决策表（表 8.1），按合规约束、持续令牌率和目标 SLO 进行索引。

选择	控制	成本	复杂性	典型用途	关键 SLO 变量
API	低	可变，按令牌付费	低	原型设计；突发负载	TTFT，p95 延迟（网络）；提供商速率限制
托管	中	每令牌成本更高	中	企业功能；IAM	TTFT，p95；自动扩展行为
DIY	高	规模上最低	高	稳定高流量	tokens/sec，TTFT，p95；GPU 利用率

表 8.1：为 DeepSeek 模型选择部署选项

每个部署决策都归结为这三个相互竞争的力量（控制、成本和复杂性）。根据我的经验，你可以优化其中两个，但很少三个都能优化：

控制意味着调整每个参数、合并模型、实现自定义安全过滤器或为合法研究运行无审查版本的能力。一家公司可能会从 Vertex AI 迁移到自托管的 vLLM，这给了他们所需的控制权，但他们必须配备专门的 LLM SRE 团队来维持低于 300ms 的延迟。
成本是数学变得有趣的地方。DeepSeek 的 API 按输入/输出令牌计费（你的有效混合费率取决于组合）。在 4× A100 GPU 上自托管？在 1,600 tokens/sec 下，这大约是每百万令牌 $1.24，但这假设你让那些 GPU 保持忙碌。仅就基础设施数字而言，自托管可能已经更便宜；真正的盈亏平衡分析只有在你为你的组织显式添加固定人员/运营开销后才会出现。
复杂性是隐性的税款。运行你自己的推理基础设施意味着雇佣专家、编写手册，以及在模型服务器凌晨 3 点崩溃时被呼叫。AWS 关于 EKS 上 DeepSeek-R1 的参考架构需要深入的 Kubernetes 专业知识和 GPU 调度知识。

最佳点？大多数组织发现不是完全选择一条路径，而是知道何时使用每条路径。你可以想象将低流量功能保留在 Vertex AI 上，同时将高流量的对账作业迁移到内部 vLLM 集群，将成本从每令牌 1.4¢ 降至 0.3¢。

你可以使用以下决策图（图 8.1）作为起点来探索哪个选项最适合你。你可以选择最适合你应用程序的部署路径。我们稍后会详细介绍所有选项。

图 8.1：选择如何部署 DeepSeek 的决策图

注意

决策树中的 Local/Ollama 路径仅适合作为开发代理，不适用于完整的 DeepSeek-V3 (671B) 或 R1 部署，后者需要多 GPU 或托管基础设施。

总的来说，这个框架通过平衡控制、成本和运营复杂性，将每个用例映射到 API、托管或 DIY 策略。为了对已选择的选项进行压力测试，请使用你自己的 $/小时报价和工作负载上测量的 tokens/sec 运行快速成本健全性检查，这样你就是从测量的经济性（而非营销数字）进行决策。

成本健全性检查

截至 2025 年 8 月，将 $/h 和 tokens/sec 视为变量，并在硬编码之前验证当前费率。

例如，如果你的 8× H200 端点报价约为 $40/h 并且在你的测量中维持约 3,000 tokens/sec，基础设施成本≈ 每千令牌 $0.0037（不包括人员/运营）。将其与提供商的令牌价格和你观察到的提示/响应分布进行比较。

除非明确标记为混合或摊销，否则每令牌成本显示为分割输入/输出费率。任何单一数字都是1:1 混合假设或摊销的基础设施估计，并相应标注。在硬编码价格之前，请验证当前的提供商表格。

让我们用实际数字来说明这些权衡：

部署路径	前期成本	每令牌成本	技术专业知识	延迟/吞吐量	隐私	可扩展性
DeepSeek API	$0 \| V3：$0.27/M 输入 + $1.10/M 输出 R1：$0.55/M 输入 + $2.19/M 输出	非常低	环境相关的端到端 TTFT 和 p95 延迟 tokens/sec 吞吐量	数据离开 VPC	弹性，由供应商管理
Local/Ollama (RTX 4090)	约 $1,599 GPU \| $0.43/M 令牌（摊销）	低-中	8–70 tokens/sec TTFT < 150ms	完全本地控制	受限于 GPU RAM
托管服务 (Bedrock/Vertex)	按需 $0。承诺 $2,000+/月	Vertex 上的 DeepSeekR1：$1.35/M 输入 + $5.40/M 输出（区域/模型相关）	中	指示性；因模型/区域/批处理而异。示例：R1 在 8× H200 上约 ~3,872 tokens/sec（NVIDIA 预览）。	区域内，SOC2	自动扩展，全球化
DIY Kubernetes (4× A100-80GB)	$5,155/月基础设施（$1.79/GPU 小时）	@ 1,600 tokens/sec $1.24/M 令牌	高	高达约 3,800 tokens/sec 在 8× H200 上（NVIDIA 预览）	完全 VPC 隔离	手动扩展

表 8.2：DeepSeek 各种部署选项的定价比较矩阵（截至 2025 年 8 月）

使用简单的规划公式来建模你的工作负载成本：

例如，让我们为矩阵中的 4× A100 DIY 设置代入数字。大约每小时 $7.16（4 GPU × $1.79/h）和测量的 1,600 tokens/sec，公式简化为每小时计算：$7.16 ÷ (1,600 × 3600) ≈ 每百万令牌 $1.24。你现在可以直接将其与 API 进行比较。这个自托管成本略低于 DeepSeek-R1 API 的约 $1.37/M 令牌的混合费率（1:1 输入/输出）。决策随后取决于利用率：如果你的工作负载足够一致以保持 GPU 忙碌，自托管提供节省；如果是间歇性的，按需付费的 API 将更便宜。

那么你如何选择？

API 胜出时：你正在试验，有可变负载，或每月处理少于 100M 令牌。零运营开销值得溢价。
本地出色时：你有开发、测试或绝对不能离开你场所的敏感数据。例如，假设 4090 GPU 上约 65 tokens/sec，1:1 I/O 比率，并将 $1,599 GPU 在 50% 利用率下分摊 24 个月，仅 GPU 成本约为每百万令牌 $0.8（不包括电力和运营）。使用成本公式根据你的工作负载进行调整。
托管服务合理时：你需要企业功能（审计日志、IAM 集成和全球化部署）而没有运营负担。更高的每令牌成本买来了安心。
DIY 提供价值时：你每月处理数十亿令牌且有可预测的负载。一个处理 3B 令牌/月的 4× A100 集群相比 API 节省约 $3,000 – 足以雇佣那个 SRE。

但决策树不是万能的。它没有告诉你大多数成功的部署是混合的。回想一下我们在本章开头讨论的法律场景。他们可能需要一种部署选项。你可能会遇到三条路径或选择用于 DeepSeek 部署：

使用 API 进行低容量实验功能，或在私有数据上进行合成数据生成。
在本地 GPU 上部署微调的法律分类器进行文档处理。
利用 Bedrock 进行需要五个九正常运行时间的面向客户的应用程序。

这个框架不是关于永远选择一条路径——而是关于知道哪条路径最适合每个用例。

你准备好看看每条路径在实践中实际是什么样子了吗？让我们深入实施细节。

三条部署路径

本节详细介绍了我们在 图 8.1 中确定的每条部署路径，并重点介绍了你可以为部署采取的具体步骤。

路径 0：基线（官方 DeepSeek API）

API 方法提供最低的运营开销。一切正常工作，其他人处理维护，你可以专注于你来要做的事情。对于许多团队来说，即使考虑了所有替代方案，这仍然是正确的选择。

DeepSeek API 提供两个主要模型：deepseek-chat (DeepSeek-V3) 和 deepseek-reasoner (DeepSeek-R1)。截至 2025 年 8 月，两者都提供 128K 上下文窗口。最大输出令牌取决于 API 参数（例如 max_tokens）和提供商设置。定价与公开表格一致。V3 每百万输入令牌 $0.27，每百万输出令牌 $1.10；R1 每百万输入 $0.55，每百万输出 $2.19。

入门需要最少的仪式，正如你在 第 5 章 中看到的：


import os

from openai import OpenAI

client = OpenAI(

    api_key=os.getenv("DEEPSEEK_API_KEY"),

    base_url="https://api.deepseek.com"

)

response = client.chat.completions.create(

    model="deepseek-chat",  # 或 "deepseek-reasoner" 用于 R1

    messages=[

        {"role": "user", "content": "什么是合同中的关键条款？"}

    ],

    max_tokens=1024

)

print(response.choices[0].message.content)

引言


import deepseek

deepseek.api_key = "YOUR_KEY"

resp = deepseek.ChatCompletion.create(

    model="deepseek-reasoner",

    messages=[{"role": "user", "content": "Prove the Pythagorean theorem."}],

    stream=True,   # 始终使用流式传输——降低 R1 的感知延迟

    # 原因：使用流式传输可降低 R1 的感知延迟（TTFT）；同时能限制客户端缓冲区的增长。

)

for chunk in resp:

    print(chunk.choices[0].delta.content, end="", flush=True)

# 原因：保护客户端免受 R1 漫长“思考”阶段的影响；请根据您的服务水平协议（SLA）进行调整（例如 60–120 秒）。

这段代码是使用 API 获取响应并直接流式传输结果的一个简单示例。

该 API 的实际运行情况既有惊喜，也有隐藏的复杂性。截至 2025 年 8 月，请求最长可运行约 30 分钟超时。在依赖这些限制之前，请核实当前的限制。尽管前面的代码示例中未显示，但您可能需要密切关注 API 密钥的开销，以保护预算，特别是考虑到 R1 倾向于冗长的推理过程。

一家供应链 SaaS 公司记录了其从 OpenAI 的GPT-4o迁移到 DeepSeek 的过程，主要动机是 API 价格低了近 25 倍（https://medium.com/tracified/deepseek-vs-openai-aec06fbee43a）。然而，节省成本伴随着一个权衡：迁移并非简单的替换，而是需要彻底改革他们的提示词工程策略。该团队发现，DeepSeek 需要更明确、更详细的指令，才能匹配他们从更灵活的 OpenAI 模型那里获得的提取准确性。

路径 1：本地和本地部署（用于开发和特定场景）

本地部署是事情变得有趣的地方。此路径主要用于开发和原型设计；它不适用于全参数的DeepSeek-V3（671B）或DeepSeek-R1部署，因为后者需要多 GPU 或托管基础设施。您用 API 的便利性换取完全的控制权——这非常适合处理机密文件的法律分类器或探索无审查模型行为的研究实验室。

我们首先使用Ollama进行本地部署。

Ollama：摩擦最小的本地路径

Ollama已成为以最少麻烦在本地运行模型的事实标准。安装在 Linux 或 macOS 上只需一条命令，如下所示：


curl -fsSL https://ollama.com/install.sh | sh

Windows 用户可使用 MSI 安装程序，可从https://ollama.com/download/OllamaSetup.exe下载。


ollama pull deepseek-r1:14b-q4_K_M  # 14B 蒸馏版，可在 16GB 显存内运行

ollama run deepseek-r1:14b

在前面的代码片段中，我们使用了 14B 蒸馏模型。14B 模型代表了一个有趣的中间平衡点，在本地运行的能力和其反馈答案的力量之间取得了平衡。在Q4_K_M量化下，它占用约 8.5 GB 的显存，截至 2025 年 8 月，在RTX 4090（非科学测试；工作负载各异）上的非正式本地测试中，速度约为 65–72 tokens/秒。虽然不是极快，但对于开发或小规模生产使用来说完全足够。完整的模型目录范围从用于内存受限系统的激进 3 位量化（约 7.5 GB）到保留更多原始模型能力的 8 位版本（约 16.7 GB）。

Docker + vLLM：生产级本地服务

当Ollama的简单性达到其极限时，容器化的vLLM（即从 Docker 容器运行的vLLM）提供了一种更复杂的方法。这不是您标准的 Flask 或 FastAPI 包装的模型——vLLM带来了工业级的优化，例如分页注意力（Paged Attention），与简单实现相比实现了 24 倍的吞吐量提升。权衡在于复杂性与工作负载配置。那 24 倍的提升反映了高并发负载下的吞吐量，这对生产来说是游戏规则的改变者，但对于在本地运行单个请求的开发者来说通常是过度的。对于低并发或开发工作流程，Ollama的简单性难以超越；当您需要从单个共享 GPU 为多个用户或应用程序提供服务时，vLLM才能证明其设置开销是合理的。

分页注意力

分页注意力是一种内存管理算法，它将 KV 缓存存储在非连续的内存块中，就像操作系统使用虚拟内存和分页一样。这解决了内部内存碎片问题，能够实现更大的批处理大小，并最终带来vLLM所知的吞吐量提升。

使用docker run --gpus all -p 8000:8000 -v models:/models deepseek-vllm:latest启动以下脚本。您将获得一个与官方 DeepSeek API 使用相同协议的 OpenAI 兼容端点。


FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04

RUN pip install vllm==0.8.5

ENV HF_TOKEN=<your-token>

CMD vllm serve \

     --model "deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct" \

     --dtype "bfloat16" \

     # 原因：限制上下文以约束 KV 缓存增长；仅在内存允许时提高此值。

     --max-model-len "8192" \

     # 原因：推动显存使用以提升吞吐量；>0.95 可能在突发负载时导致内存溢出。安全范围：0.85–0.95。

     --gpu-memory-utilization "0.95"

面向部署的硬件与推理优化引擎

当您需要在本地部署与云部署之间切换，而无需重写应用程序代码时，这种兼容性就变得至关重要。

对于处理敏感数据的团队，像OnPrem.LLM(arxiv.org/html/2505.07672v2)这样的项目可以在这些本地服务层之上构建完整的 RAG 流水线，确保专有文档永远不会离开您的基础设施。

路径二：托管推理服务（平衡之道）

托管服务在控制力与便利性之间取得了绝佳平衡。您无需雇佣专门的 MLOps 团队，即可获得企业级功能（审计日志、IAM 集成以及全球部署）。想象一下，一家快速发展的 B2B SaaS 公司，其核心产品是 AI 驱动的分析仪表盘。公开 API 非常适合其最小可行性产品，但其新的企业客户要求保证正常运行时间、更低的延迟以及在私有网络内进行处理。自建 Kubernetes 集群需要数月时间，并需要雇佣专门的 MLOps 工程师。因此，他们转向了 Amazon Bedrock 或 Vertex AI 等托管服务。这一举措使他们能够以可预测的成本模型和最小的运维负担，获得所需的企业功能，如私有端点和预配置吞吐量，从而满足企业需求，而不影响其产品路线图。

各大主要云服务提供商都已拥抱 DeepSeek，尽管其方法各有不同。让我们来逐一介绍：

Amazon Bedrock (aws.amazon.com/bedrock) 通过 Bedrock（包括 Marketplace）提供DeepSeek-R1，支持无服务器调用。Bedrock 中通常提供预配置吞吐量，但其可用性可能取决于具体的第三方模型和区域——请为您的账户核实。无服务器选项非常适合可变负载，而预配置吞吐量（在可用的情况下）一旦超过持续 Token 速率，即可保证一致的性能。其内置的护栏服务无需自定义代码即可添加内容过滤，这对于面向客户的应用至关重要。
SageMaker JumpStart (docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html) 采用了不同的方法，专注于可以在单 GPU 实例上轻松运行的蒸馏版本（8B 和 32B）。AWS 最近的一篇博客文章展示了在ml.g5.xlarge实例上服务 8B 模型。定价因区域和承诺而异，请为您的账户核实费率。
Google Vertex AI (cloud.google.com/vertex-ai/generative-ai/docs/maas/deepseek) 在其模型花园中添加了DeepSeek-R1，支持一键部署。截至 2025 年 8 月，us-central1区域的代表性价位约为每百万 Token 6.75 美元。费率因区域/模型而异，请在您的账户中核实。它比官方 API 更贵，但包含企业级 SLA 和私有端点支持。
Azure AI Foundry (azure.microsoft.com/en-us/products/ai-foundry) 补齐了三大云商的阵容，同时提供无服务器和专用集群选项。其负责任的 AI 记分卡提供自动偏差检测和内容安全分析——这些功能可以为受监管行业节省数月的定制开发时间。
Hugging Face Inference Endpoints (huggingface.co/inference-endpoints/) 因其透明度值得特别提及。其部署向导会预先显示明细成本，但费率因区域/计划/提供商而异。截至 2025 年 8 月，可将 8×H200 大约$40/小时作为市场参考（例如，Together AI 的公开报价约为$0.67/分钟）。对于 Hugging Face Endpoints，自动缩容至零是可选加入的，并且默认在闲置约 15 分钟后触发（历史上也记录为约 1 小时）。存在用于将端点手动缩容至 0 的编程控制；除非您的提供商明确支持，否则不要假设有 60 秒的闲置超时。请为您选择的区域/计划核实可用性和具体语义。

一种实用的混合模式是，将延迟敏感、面向客户的推理保留在托管平台上，而将大型、非交互式的批处理作业转移到专用端点以降低成本。SageMaker 上的成本案例研究报告了显著的节省；例如，Forethought 指出，迁移到 SageMaker 后，推理成本降低了高达 80% (aws.amazon.com/solutions/case-studies/forethought-technologies-case-study/)。

路径三：在 IaaS 上自建（为了极致控制）

自建部署是工程与经济的交汇点。您不仅仅是在运行一个模型；您是在构建基础设施。这条路径需要专业知识，但能提供无与伦比的灵活性和潜在的成本节约。以下是一些典型用例及各自示例：

受监管和网络隔离的工作负载：您必须将数据保留在特定的网络边界或国家内，并且不能使用第三方端点。例如，一家医院在自己的 VPC 中通过私有 API 部署 DeepSeek，没有互联网出口。
稳定、高流量且注重每 Token 成本的工作负载：您的工作负载具有可预测的每日/每周批处理任务，并保持 GPU 繁忙。例如，一个电子商务团队在vLLM集群上运行夜间目录充实和翻译，与按 Token 付费的 API 相比降低了单位成本，并控制了批处理窗口。
超低延迟或网络邻近性：您需要达到公共 API 难以企及的 p95 延迟目标。例如，一个交易台将推理节点与其市场数据流并置，使用编译内核（TensorRT-LLM）和 KV 缓存预热器，并避免跨区域跳转。
自定义模型或运行时控制：您需要托管端点未暴露的能力。例如，一个法律科技平台合并领域 LoRA，执行自定义的脱敏过滤器，调整 R1 思考预算，并在vLLM中调整连续批处理参数以匹配其流量。
具有严格隔离和预算的多租户平台：您必须保证每个租户的数据隔离，并在数千个租户间强制执行 Token 预算。例如，一个开发者平台为每个租户运行独立的命名空间，使用粘性路由会话以利用缓存局部性，并在网关处执行令牌桶限制。
本地部署、边缘或不可靠的 WAN：您无法依赖稳定的连接。例如，一家工厂在一个小型本地 GPU 集群上运行 DeepSeek，并定期更新权重，以确保即使在无线网络链路中断时也能继续进行本地推理。
供应商独立性和容量控制：您希望在需求激增前提前锁定价格/可用性。例如，一家媒体公司在两个区域预留 GPU，为基线负载自建，仅在发布期间突发到托管端点。

这些模式充分发挥了自建的优势：完全控制内存放置（这对 V3 的 MoE 很重要）、限制或塑造 R1 推理 Token 的能力、大规模下可预测的单位经济学，以及与您自己的安全和 SRE 实践的深度集成。

以下是一些您可以尝试的方法：

单 VM 或裸金属：最简单的自建方法类似于本地部署，但使用更强大的硬件。一个典型配置可能包括 4×A100-80GB GPU，运行vLLM，其中DeepSeek-V3被量化到 INT4。截至 2025 年 8 月，在高利用率下，实测吞吐量约为每秒 1,600 个 Token，每百万 Token 的基础设施成本低于约$2；在承诺之前，请用您的工作负载进行验证。

使用更新的硬件，这个计算变得更有意思。NVIDIA 报告称，在早期 NIM 预览条件下，在 8×H200 上运行 R1 的速度约为 3,872 tokens/sec；后来的 Blackwell（8×）营销声称达到约 30k tokens/sec（blogs.nvidia.com/blog/deepseek-r1-nim-microservice/ 和 developer.nvidia.com/blog/nvidia-blackwell-delivers-world-record-deepseek-r1-inference-performance/）。对于每天处理数十亿 Token 的组织来说，投资回报率可能很快显现。

Kubernetes 编排：Kubernetes 部署将自建从运行一个服务器提升到运营一个服务。2025 年的现代方法以KServe v0.15 作为服务层为核心，但真正的重点在于您如何思考扩展和优化。
与传统的 Web 服务模式不同，LLM 部署需要不同的思维模型。您不是根据每秒请求数进行扩展；而是根据实际完成的工作量进行扩展：有多少请求正在积极生成 Token，与有多少在队列中等待。诸如Kubernetes Event-driven Autoscaling（KEDA）之类的工具通过监视真实的 LLM 指标（而非原始的请求计数）来实现这种更智能的扩展（keda.sh/）。

以下是自建部署中最关键的一些架构选择：

智能自动扩展：基于有意义的指标（如活跃的生成负载）进行扩展，而非 HTTP 请求速率。R1 较长的推理阶段使得这一点尤为重要。
Token 预算控制：与传统的速率限制不同，您需要在 Token 级别限制支出。Envoy AI Gateway 和类似工具提供了这种控制，无需重写客户端。
KV 缓存共享：在会话式工作负载中，跨副本共享缓存可显著提高吞吐量。像 LMCache 这样的技术使这成为可能，无需更改应用程序。
多节点考虑：当模型超出单节点内存时，您就进入了不同的复杂性领域。流水线和张量并行可以工作，但需要仔细规划，并且它们目前不支持自动扩展。

请注意，即使是 INT4 量化的 V3 也需要多个 GPU。R1 的思考阶段需要宽松的超时时间。对于最大的模型，较少但强大的节点加上快速互连通常优于许多较小的实例。然而，这些并非即插即用的解决方案。每个优化都需要理解您特定的工作负载模式。团队经常发现，他们的聊天工作负载实际上是 80%的单轮提问，使得会话优化的缓存毫无价值；或者他们实现了复杂的自动扩展，结果发现他们的负载是完全可预测的。

我们探索的每条路径都有其合理的用例。API 擅长处理原型和可变负载。本地部署保护敏感数据，同时允许在较小模型上进行实验；完整的DeepSeek-V3或 R1 需要多 GPU 或托管服务。托管服务在功能和操作简便性之间取得了平衡。自建基础设施提供终极控制力和规模化的经济效益。

您的选择较少取决于技术优越性，更多取决于组织的现实情况。您是否有运行 Kubernetes 的专业知识？您的数据能否离开您的本地环境？您是否会处理足够多的 Token 来证明基础设施投资的合理性？诚实地回答这些问题，正确的路径就会变得清晰。

接下来，我们将深入探讨使任何这些路径成为可能的基础技术，因为无论您是调用 API 还是运行自己的集群，理解底层机制都有助于您做出更好的决策。

下一节将调查支撑这些选择的硬件和服务引擎。

选择您的硬件

VRAM 容量决定了可行性；其他组件主要影响速度。在 LLM 部署方面，您的 GPU 内存决定了什么是可能的，而其他一切只决定了发生的快慢。

当前用于 DeepSeek 部署的 GPU 层级如下所示：配备 80 GB HBM3 的 H100 位于顶端，通过其 Transformer Engine 提供 3.3 TB/s 的内存带宽和原生 FP8 支持。A100（40 GB 或 80 GB 变体）仍然是生产主力——可靠、广泛可用，并且对于大多数用例来说速度仍然足够快。L40S 提供了一个有趣的中间选择，配备 48 GB GDDR6——内存带宽较低，但通常更容易采购。

一个采用 BF16（bfloat16）精度——这是一种在现代 GPU/TPU 上使用的 16 位浮点格式，每个参数存储 2 个字节，在更少的尾数位（即精度位）下保留 FP32 的数值范围——的 33B 参数模型需要以下内容：

基础内存 = 33B 参数 × 2 字节/参数 = 66 GB。
运行时开销（约 10%） = 6.6 GB。
总 VRAM 需求 = 约 73 GB。

这还没计算用于上下文的 KV 缓存、用于批处理的激活内存，或者安全余量。您的 33B 模型勉强能塞进单个 H100，但一旦启用生产特性，就需要多个 GPU。

如果您选择部署具有 671B 参数的DeepSeek-V3怎么办？即使进行激进的 INT4 量化，您也需要至少 335 GB。在这种情况下，多 GPU 集群不是可选的，而是强制性的。

另一个重要方面是利用推理引擎。原因如下。

推理引擎

对于 DeepSeek，在简单的 Web 服务器中编写一个快速的model.generate()函数是不够的。我们很快发现了推理引擎所解决的性能瓶颈。

朴素的逐请求生成一次处理一个请求，并在每个序列上逐个 Token 生成，然后才接受新的请求。在 Token 之间，GPU 通常等待 Python 开销和内核启动，KV 缓存以大块连续方式分配，随着序列在不同时间完成，导致内存碎片。例如，一个简单的 Flask 或 FastAPI 服务器，每个 HTTP 请求都调用batch_size=1的model.generate()，会运行一个用户流直到完成，而其他用户排队等待，导致 GPU 利用率低下。另一个常见情况是固定批处理，等待批次中的所有序列完成；短提示在空闲，而最长的提示阻塞整个批次。

转向vLLM或 TGI 并使用连续批处理可以避免这些间隙。它们重用分页的 KV 内存，并在其他请求完成时插入新请求，这使得设备保持饱和。在实践中，这通常能在相似延迟下带来 2-4 倍的吞吐量提升。公开基准测试反复显示，在负载下与 Hugging Face Transformers 服务器相比有巨大收益（arxiv.org/abs/2309.06180 和 blog.vllm.ai/2023/06/20/vllm.html）。

朴素的方法将每个请求视为孤立的。您昂贵的 GPU 在等待 I/O、内存碎片以及短序列完成时闲置，而较长的序列则阻塞整个批次。

于是推理引擎登场。推理系统有助于解决 GPU 时间的浪费，提高吞吐量，并降低延迟。让我们谈谈您可用的各种选择：

vLLM (github.com/vllm-project/vllm)：LLM 引入了 PagedAttention，借鉴操作系统的虚拟内存概念来高效管理 KV 缓存。vLLM不是预分配巨大的连续内存块，而是像现代操作系统处理 RAM 一样，对缓存条目进行分页。这意味着与朴素的服务方式相比，吞吐量提升了 2-4 倍，某些工作负载甚至能看到高达 23 倍的提升。
vLLM真正的魔力在于连续批处理。传统服务方式等待批次中的所有序列完成才处理新请求，而vLLM则在其他请求完成时动态插入新请求，使您的 GPU 保持饱和。这就像出租车司机，一种方式是等待所有乘客到达目的地，另一种方式是沿途搭载新乘客。

以下是如何实现批处理：

传统批处理伪代码


def serve_batch(requests):

    outputs = model.generate(requests)  # 必须全部完成

    return outputs

连续批处理概念


def serve_continuous():

    while True:

        if completed := get_finished_sequences():

            yield completed

            add_new_requests_to_batch()

        step_active_sequences()

Text Generation Inference (TGI) (github.com/huggingface/text-generation-inference)：Hugging Face 的 TGI 为 IBM Watson 和 Grammarly 的企业级部署提供动力。其核心技术结合了 Flash Attention 内核、tensor parallelism 和 safetensors，以实现近乎即时的模型加载。与优化吞吐量的 vLLM 不同，TGI 平衡了吞吐量与一致的延迟表现——这对于面向用户的应用至关重要。

TGI 的优势包括连续批处理、Flash Attention、PagedAttention、量化和 tensor parallelism。它使用 safetensors 和优化的加载器来加速冷启动，但权重必须在服务请求前完全加载。

TensorRT-LLM (github.com/NVIDIA/TensorRT-LLM)：NVIDIA 的 TensorRT-LLM 代表了更贴近底层硬件的方法。它将你的模型编译成优化的 CUDA 内核，融合操作，并利用 H100 Transformer Engine 等硬件特性。编译步骤增加了复杂性，但回报是巨大的——从 A100 FP16 迁移到 H100 FP8 时，吞吐量可提高 4.6 倍。

需要注意的是，TensorRT-LLM 对模型架构和硬件更为挑剔。它在 NVIDIA 最新 GPU 上运行其熟知的模型时表现最出色。对于 DeepSeek 的 MoE 架构，你可能需要等待官方支持或深入研究自定义插件。

那么，如何选择你的推理引擎？

选择不在于哪个引擎是最好的；而在于将优势与需求相匹配。以下是一个快速指南，帮助你做出决策：

当需要最大灵活性和吞吐量时，使用 vLLM。
当需要生产环境稳定性和一致延迟时，选择 TGI。
当你拥有现代 NVIDIA 硬件并需要追求极致的每秒 token 数时，采用 TensorRT-LLM。

大多数团队因易用性而从 vLLM 或 TGI 入手，然后在达到规模时评估 TensorRT-LLM。妙处何在？三者都提供兼容 OpenAI 的 API，因此切换引擎通常只需更改配置，而非重写代码。

量化是另一种可以帮助你加速推理时间和降低成本的技术。

量化的力量

回忆一下 第 2 章，量化通过降低精度来缩小内存占用并提高吞吐量，通常伴随轻微的质量损失。量化领域提供了多种技术，每种都有其权衡：

FP8 代表了最温和的压缩，在保持近乎完美精度的同时将内存使用减半。它需要较新的硬件（H100/L40S），但提供了最佳的质量与压缩比。可以把它想象成从无损的 FLAC 格式转向高比特率的 MP3：技术上有损失，但实际上难以区分。
GPT quantization (GPTQ) 采用训练后处理方法，分析你的模型以确定哪些权重可以压缩到 4 位而不会导致灾难性遗忘。它速度快，只需要几百个样本的校准数据集，通常精度损失不到一个百分点。（要了解更多关于 GPTQ 的信息，请查阅 Hugging Face 博客：huggingface.co/blog/gptq-integration。）
Activation-aware weight quantization (AWQ) 更进一步，识别那些对输出有不成比例影响的显著权重，并保持其在更高精度。对于 DeepSeek 的 MoE 模型，这种激活感知特别有价值——专家路由依赖于精确的权重值，而 AWQ 能够保留它们。（要了解更多并找到理解其工作原理的资源，Hugging Face 的 AWQ 文档是个好去处：huggingface.co/docs/transformers/en/quantization/awq。）

一个简单估算 GPU 内存的方法是将其分为三部分：模型权重、少量运行时余量和 KV 缓存：

权重：从权重张量本身开始。将参数数量乘以所选精度下每个参数的字节数。对于一个 330 亿参数的模型，BF16 每个参数存储约 2 字节，因此仅权重就大约需要 66 GB。FP8 将其减半至约 33 GB。使用 4 位量化时，每个参数存储约半字节。这意味着，实际上，330 亿参数模型的 GPTQ-4b 检查点通常在 16–17 GB 左右，而 AWQ-4b 检查点通常更接近 18–19 GB，因为它们将一些关键值保持在更高精度。在相同位宽下，AWQ 通常比 GPTQ 保留了稍高的精度。
运行时余量：在权重大小上增加约 10–20%，用于临时工作空间、CUDA 内核缓冲区、分配器碎片以及（对于 MoE 模型）路由开销。以同样的 330 亿参数模型为例，在考虑序列内存之前，BF16 部署可能需要在 66 GB 的权重之外额外 6–13 GB。
KV 缓存：最后，为 KV 缓存 预留空间，它随着上下文长度、生成 token 数和批处理大小而增长。这部分与权重大小是分开的。对于 DeepSeek-V3 的 MLA，每个 token 的 KV 缓存 大约为 70 KB（大约每 10,000 个缓存 token 占用 0.7 GB）。其他架构可能更大。乘以并发序列数来估算总的 KV 占用量。

简而言之，对于一个 330 亿参数的模型，预计需要大约 66 GB (BF16) 或 ~33 GB (FP8) 用于权重，或者 ~16–17 GB (GPTQ-4b) / ~18–19 GB (AWQ-4b)，再加上 10–20% 的运行时余量，以及为服务的 token 预留的 KV 缓存。

一个重要的陷阱：确保你的服务栈原生运行低位内核。一些系统在磁盘上存储 4 位权重，但在加载或矩阵乘法时将其转换为 BF16/FP16。这会悄悄地将 VRAM 需求推回接近全精度的数字，从而抹去性能和成本节约。通过检查加载后的日志和观察到的 VRAM，验证你的引擎（例如 vLLM、TGI 或 TensorRT-LLM）正在使用预期的低位执行路径。

实际影响如何？那个在全精度下需要昂贵 H100 的 330 亿参数模型，在经过 4 位量化后，可以轻松运行在单个 L40S 上。你的推理成本下降了 75%，而精度仅损失 1–2%——大多数生产系统都乐意接受这种权衡。

并非所有模型都能均匀地量化。较小的模型从量化中受苦更多；它们可供牺牲的冗余较少。任务复杂性也很重要：摘要任务可能容忍激进量化，而法律文档分析则要求更高精度。

务必针对你的具体用例验证量化模型。那 0.5% 的平均精度下降可能掩盖了在罕见但关键输入上 10% 的性能退化。并且请记住：量化只是虚拟的 VRAM。你仍然需要一个理解低位格式的推理引擎，否则你会在运行时进行反量化，从而失去所有优势。

硬件提供了原始能力。推理引擎释放了该能力的潜力。量化扩展了你的硬件预算。三者结合，将 DeepSeek 从一个令人印象深刻的演示转变为一个生产就绪的系统。

关键洞察是什么？这些并非独立的选择。只有当你的推理引擎能够利用它时，H100 的 FP8 支持才有意义。只有当你的引擎高效处理混合精度执行时，AWQ 才有帮助。你的部署策略必须考虑完整的栈，而不仅仅是单个组件。

考虑到硬件和服务引擎后，下一节将演示三个具体的部署场景。

实践部署指南

这三个部署示例从笔记本电脑实验到云上生产逐步推进，每一个都建立在前一个经验教训的基础上。请跟随与你当前需求匹配的示例进行操作，但建议阅读全部三个示例，以了解完整的部署谱系。

让我们先从托管端点的成本分析开始。截至 2025 年 8 月，托管提供商上一个 8× H200 部署的成本约为每小时 40 美元。Together AI 公开报价为 $0.67/min (www.together.ai/deepseek)。这些成本会迅速累积：

持续运行的每日成本：960 美元。
7x24 运行的每月成本：28,800 美元。
部署启动时间：约 10-15 分钟 ≈ $7-$10。

那么，有哪些选项来管理你的成本？以下是一些智能的成本管理选项：

在支持的地方使用缩容到零：在 Hugging Face Inference Endpoints 上，这是可选加入的，通常在空闲约 15 分钟后触发（尽管你可以根据需要配置）。通常不支持每端点 60 秒的空闲超时；相反，使用编程式手动缩容到零/暂停来实现激进的成本控制。检查你的提供商支持的功能。由于模型启动和基础设施配置通常需要几分钟，因此无论你设置什么值，扩容或缩容通常都会有一点延迟。
设置计费警报：多个阈值是你的好朋友。大多数主要提供商都提供预算和使用量警报（AWS Budgets/CloudWatch、Google Cloud Budgets and Alerting、Azure Cost Management，以及托管端点的使用量通知）；如果你的提供商没有提供，请在你的 API 网关中强制实施 token 级别上限，并从你的可观测性栈中发出警报。默认情况下，在月度预算的 50%、80% 和 100% 处设置警报（在 100% 时自动限流或暂停），并添加每日减速带警报（例如，一天内消耗月度预算的 10%），以便尽早发现失控的突发流量。
考虑你的用例：突发性工作负载？非常适合缩容到零。持续流量？也许可以协商容量预留块。
算笔账：以每小时 40 美元计算，你需要处理足够的价值来证明计算成本是合理的。例如，如果你的端点持续处理约 3,000 tokens/sec，基础设施成本约为每 1k token 0.0037 美元，那么一个 500 万 token 的批处理运行成本约为 18.5 美元（不包括人员/运维成本）。相反，如果你每小时只处理约 10 万 token，你实际上为每 1k token 支付约 0.40 美元（约 $400/M），这通常高于 API 费率，因此对于这种负载模式，请暂停/缩容到零或使用无服务器服务。

此定价反映了运行具有企业级基础设施的完全托管 6710 亿参数模型的成本。将其与以下情况进行比较：

自建 8x H200 集群：前期投入 50 万美元以上。
AWS 容量预留块：约每小时 36 美元（但你需要自己管理一切）。
较小的云提供商：通常不可用或不可靠。

专业提示

对于实验，从较短的推理窗口开始。一次 2 小时的深入研究花费 80 美元，对于有针对性的实验来说是一笔可管理的费用。完成后执行 endpoint.scale_to_zero() 以避免持续收费。

这些数字可能看起来令人生畏，但它们强调了匹配工具与任务的重要性。就我个人而言，我的工作流程几乎总是从本地 Ollama 实例开始进行快速迭代，只有在需要大规模测试性能时才升级到托管端点。这种方法使开发保持快速，初始成本为零。

示例 1：使用 `Ollama` 和量化版 DeepSeek Coder 模型进行本地部署

使用 Ollama 进行本地部署代表了阻力最小的路径。它是 LLM 部署的 Hello World——简单到可以在喝杯咖啡的时间内完成，同时又足够强大，能够驱动真实的开发工作流。我们将使用 160 亿参数版本并量化到 4 位精度，这能提供接近完整模型的性能，同时能舒适地运行在消费级硬件上。本示例是一个开发代理，并未部署完整的 DeepSeek-V3 (671B) 或 R1 模型。

在本节中，我们的目标是在 10 分钟内让 DeepSeek Coder V2 在你的本地机器上运行起来，非常适合在没有云依赖或 API 成本的情况下进行开发和实验。

前提条件和性能预期

量化模型（DeepSeek Coder V2）重量约为 ~9 GB。在消费级硬件上，你应该预期 4 位 Coder 模型能达到每秒数十个 token 的速度，并且本地的首 token 时间（TTFT）通常 <1 秒。吞吐量因量化方式、驱动程序和工作负载而异。将本节中的任何单机数字视为方向性参考，而非 SLA 目标。

让我们开始逐步部署步骤：

在类 Unix 系统上，安装只需一条命令：
```
[在此处插入安装命令]
```

使用 Ollama 部署 DeepSeek-Coder-V2 模型

安装 Ollama

Linux/macOS


curl -fsSL https://ollama.com/install.sh | sh

Windows

Windows 用户可以从 Ollama 网站下载 MSI 安装程序：ollama.com/download/OllamaSetup.exe。安装完成后，拉取模型非常简单：


ollama pull deepseek-coder-v2:16b

此命令会下载默认的 Q4_0 量化版本。下载过程会实时显示进度，Ollama 会处理所有模型格式转换和针对您硬件的优化。

基本使用

要进行交互式测试，可以启动一个交互会话：
```
ollama run deepseek-coder-v2
```
这将启动一个 REPL 环境，您可以立即开始编写代码。但要与开发工具集成，您需要使用 API 服务器。

API 服务器

启动 API 服务器：


ollama serve

这会在 http://localhost:11434 上暴露一个与 OpenAI 兼容的 REST 端点。其兼容层几乎完美；大多数期望使用 OpenAI API 的工具只需简单更改基础 URL 即可工作。

验证与测试

通过快速 curl 测试来确认部署：


curl http://localhost:11434/api/generate \

  -d '{"model":"deepseek-coder-v2","prompt":"// bubble sort in Go"}'

对于 Python 应用程序，OpenAI 客户端库可以无缝工作。注意，OpenAI 库已成为人们用于向 LLM 发出推理请求的标准 SDK。在以下代码示例中，我们传入了一个配置，指定我们希望对本地 Ollama 模型进行请求，而不是对云托管的 OpenAI 模型：


import openai

openai.base_url = 'http://localhost:11434/v1'

openai.api_key = 'ollama'  # 任何非空字符串均可

response = openai.chat.completions.create(

    model='deepseek-coder-v2',

    messages=[

        {

            "role": "user",

            "content": "Write a Rust macro for logging."

        }

    ]

)

print(response.choices[0].message.content)

常见问题排查

最常见的问题是内存不足。如果看到 cudaMalloc failed: out of memory 错误，您有三个选择：使用更激进的量化版本（尝试 deepseek-coder-v2:16b-q3_K_M），通过 OLLAMA_NUMA=1 启用 CPU 卸载，或者关闭其他占用 GPU 的应用程序。
下载中断会留下可能导致晦涩错误的局部文件。如果遇到 EOF 错误或 model manifest not found 错误，请删除并重新拉取：
```
ollama rm deepseek-coder-v2

ollama pull deepseek-coder-v2:16b
```
磁盘空间是另一个陷阱。下载过程临时需要大约 2 倍于模型大小的空间。如果您在中途拉取失败，可能需要运行 ollama prune 从孤立的 blob 文件中回收空间。

清理

清理工作与部署一样简单：


ollama stop deepseek-coder-v2  # 停止运行中的模型

ollama rm deepseek-coder-v2     # 删除模型文件

ollama prune                    # 清理未使用的数据

本节要点

这个示例演示了模型生命周期管理的基础知识：拉取、运行、服务和清理。你已经看到量化如何使大模型在消费级硬件上变得实用，并且 Q4 量化仅以 1-2 个百分点 的精度损失为代价，就实现了 75% 的内存需求缩减。

更重要的是，你现在拥有一个功能完备的 LLM 开发环境。安装了 Continue 扩展的 VS Code 可以连接到你本地的 Ollama 实例，在无需将代码发送到外部服务的情况下，为你提供 AI 驱动的编码辅助。这是进行实验、开发以及处理数据隐私至上场景的完美配置。

接下来，我们将扩展到托管云部署，为此我们需要牺牲一些简易性，以换取企业级功能和生产环境的可靠性。

示例 2：在 Amazon Bedrock 上进行托管部署

AWS 提供了多种 DeepSeek 部署路径。截至 2025 年 3 月，DeepSeek-R1 已作为全托管无服务器模型在 Amazon Bedrock 中提供（aws.amazon.com/blogs/aws/deepseek-r1-now-available-as-a-fully-managed-serverless-model-in-amazon-bedrock/），提供了最简单的部署选项，支持自动扩缩容和按 token 计费。

这里的目标是将任何受支持的 DeepSeek 模型（例如 R1 或 V3）部署到 Amazon Bedrock，以实现具有 AWS 原生集成、安全性和合规功能的企业级托管推理。

AWS 提供了三种主要方式来运行 DeepSeek 模型：

Bedrock 无服务器（最简单）：完全托管，无需配置基础设施。按 token 付费，无空闲费用。可通过跨区域推理配置文件用于 DeepSeek-R1。
Bedrock 预置吞吐量：适用于可预测工作负载的预留容量。提供有保障的性能，但需要预先承诺。请检查区域可用性，因为并非所有模型都支持此模式。
AWS Marketplace + SageMaker：自己部署模型，拥有完全控制权。此路径需要为大型 GPU 实例（例如配备 H100 的 ml.p5en.48xlarge）申请提高配额。你管理基础设施，但获得最大灵活性。

与其提供注定会过时的分步说明，不如直接访问你的 AWS 账户中的 Amazon Bedrock 控制台。该控制台提供最新的部署向导，并将指导你完成以下步骤：

模型访问：某些模型需要通过 模型访问 部分明确选择加入。
配额验证：根据你的部署路径，检查 Bedrock（每分钟请求数）或 SageMaker（GPU 实例）的服务配额。
护栏配置：用于生产部署的可选内容过滤。
IAM 设置：用于应用程序的基于角色的访问控制。

如果选择 Marketplace 选项，请准备好申请大幅提高配额；即使经过量化，DeepSeek-V3 和 R1 也需要多个高端 GPU。

无论你选择哪种部署路径，使用 AWS 部署时都需要做出几项重要决策：

API 选择：AWS 提供多种推理 API。Converse API 提供跨模型的统一接口，并内置 token 使用量指标。InvokeModel API 使用特定于模型的架构，但可能提供更低的延迟。因此，选择取决于你的应用程序对标准化与性能的需求。
超时配置：DeepSeek-R1 的推理过程可能很长。因此，你需要将客户端超时至少配置为 300 秒。
成本控制：务必设置 maxTokens 限制。R1 可能生成非常长的推理链，单个请求可能消耗数千个 token。
私有网络：对于敏感工作负载，配置 VPC 端点以将流量限制在 AWS 内部。即使是 AWS 的托管服务，你也会遇到一些常见挑战：
- 模型访问被拒绝：在 Bedrock 控制台的 模型访问 下检查。即使具有正确的 IAM 权限，某些模型也需要明确选择加入。批准时间从即时到 48 小时不等，具体取决于你的账户状况。
- 限流错误：Bedrock 对每个模型强制实施速率限制（每分钟请求数和 token 数）。实现带有抖动的指数退避。
- 配额问题：对于 Marketplace 部署，你需要大量的 GPU 配额。单个 ml.p5en.48xlarge 实例每月成本高达数千美元。请提前很久申请提高配额；AWS 可能需要数天才能批准大型 GPU 分配。
- 推理时间过长：R1 的推理深度差异很大。简单问题可能在几秒内返回，而复杂问题可能需要几分钟。实现流式响应以获得更好的用户体验。
意外的成本：如果没有适当的限制，单个失控的请求可能产生数千美元的费用。监控 CloudWatch 指标，并在多个阈值（预算的 50%、80% 和 100%）设置账单警报。在生产部署之前，你需要验证所选路径是否满足你的服务等级目标：
- 首 token 延迟：对用户体验至关重要。由于推理阶段，R1 的首 token 延迟可能较高。
- 每秒 token 数：根据部署类型和实例大小，差异显著。
- p95 延迟：比平均延迟更有意义，适用于容量规划
并发请求处理：在预期负载水平下进行测试；性能会呈非线性下降。考虑在一天中的不同时间进行负载测试，因为共享基础设施的性能可能波动。

本节要点

Amazon Bedrock 展示了云提供商部署大模型的方法：以供应商锁定和溢价为代价，实现与现有服务（IAM、CloudWatch 和 VPC）的深度集成。无服务器选项完全消除了运维开销——无需管理实例，无需配置缩容至零，只需调用 API 和接收账单。

关键的权衡在于控制与便利性。Bedrock 无服务器让你对底层基础设施零控制，但也零责任。Marketplace 路径让你拥有完全控制权，但需要 GPU 实例管理、模型优化和容量规划方面的专业知识。

对于已经投入 AWS 生态系统的团队，Bedrock 的集成优势通常超过了其溢价。CloudWatch 原生指标、你的安全团队已经理解的 IAM 策略，以及与现有网络配置兼容的 VPC 端点，都降低了采用的隐性成本。

AWS AI 服务的快速演进意味着具体功能和定价会变化，但基本模式不变：AWS 将继续提供全托管和自管理选项，其价格反映了你愿意接受的运维复杂性。

接下来，我们将探讨 Hugging Face Inference Endpoints，它提供了介于完全自管理和完全抽象之间的中间方案。

示例 3：使用 Hugging Face Inference Endpoints 将 DeepSeek V3 部署到云端

Hugging Face Inference Endpoints 允许你自主选择硬件和区域，但由 Hugging Face 处理编排复杂性。对于 DeepSeek-V3，这种抽象尤其有价值；该模型仅加载就需要 8 块 NVIDIA H200 GPU，更不用说高效服务了。

在开始之前，我们先谈谈数字。

如果你自己部署 DeepSeek，请准备好在常见的托管提供商上为 8× H200 设置每小时花费约 40 美元（Together AI 公开报价 ≈$0.67/分钟 ≈ $40.2/小时）；Hugging Face Inference Endpoints 的价格因区域/计划/提供商而异。如果你让推理端点持续运行，每天大约需要 960 美元——这仍然是一笔不小的数目。

因此，始终选择支持“缩容至零”的功能（如果支持），以便在空闲期间管理成本。

要通过 HuggingFace API 进行部署，你需要以下内容：

一个具有 API 访问权限的 Hugging Face 账户。
一个具有写权限的 API 令牌。
预算批准（请仔细检查账户支出限额）。

最简洁的方法是使用 Hugging Face Python SDK。以下是完整的部署脚本：

在 Hugging Face 上部署 DeepSeek-V3

本指南涵盖了从使用 Python SDK 创建端点、监控部署，到通过 REST API 和 ZenML 流水线进行部署的全过程。

基础配置与端点创建

首先，使用 create_inference_endpoint 创建一个具备生产就绪配置的端点。


from huggingface_hub import create_inference_endpoint

endpoint = create_inference_endpoint(

    name="deepseek-v3-prod",

    repository="deepseek-ai/DeepSeek-V3",

    framework="pytorch",

    task="text-generation",

    accelerator="gpu",

    vendor="aws",

    region="us-east-2",

    type="protected",  # 需要认证才能访问

    instance_size="x8",

    instance_type="nvidia-h200",

    # 自动缩放配置

    min_replica=0,  # 启用空闲时自动缩容至零（如果支持）

    max_replica=1,  # 单副本以控制成本

    # 注意：HF Endpoints 通常在空闲约 15 分钟后触发缩容至零；不支持为单个端点设置 60 秒超时。

    # 用于优化服务的自定义 TGI 容器

    custom_image={

        "health_route": "/health",

        "url": "ghcr.io/huggingface/text-generation-inference:3.3.1",

        "env": {

            # 针对大模型的 TGI 优化

            "MAX_BATCH_PREFILL_TOKENS": "16384",

            "MAX_INPUT_LENGTH": "32768",

            "MAX_TOTAL_TOKENS": "65536",

            "MODEL_ID": "/repository"

        }

    }

)

这个初始配置定义了端点的标识、模型和基础设施：我们针对 us-east-2 区域的 8x H200 GPU 上的 DeepSeek-V3，并设置为需要认证的 protected 类型。这些基础设置确定了模型的运行位置和访问权限，之后我们将添加扩缩容和服务细节。

自动缩放设置用于控制成本和可用性：min_replica=0 启用空闲时缩容至零（在支持的情况下），而 max_replica=1 在初始测试阶段限制开销。如果后续需要更高的并发性，可以增加这些值。

我们指定了一个 TGI 容器并调整服务限制（预填充、输入和总 token 数），以在内存限制内处理大模型请求并最大化吞吐量。MODEL_ID 将容器指向挂载的仓库以加载模型权重。

部署监控

部署通常需要 10-15 分钟来拉取镜像并将权重加载到 GPU；wait() 方法提供了一个带有超时限制的阻塞流程。如果失败，fetch() 可以帮助你检查部分状态以进行快速调试。

你可以通过以下代码监控部署进度：


print(f"Initial status: {endpoint.status}")

print("Waiting for deployment (this typically takes 10-15 minutes)...")

try:

    endpoint.wait(timeout=1200)  # 20 分钟超时

    print(f"✓ Deployment successful!")

    print(f"Endpoint URL: {endpoint.url}")

    print(f"Status: {endpoint.status}")

except Exception as e:

    print(f"✓ Deployment failed: {e}")

    # 检查是否发生了部分部署

    endpoint.fetch()

    print(f"Current status: {endpoint.status}")

通过 REST API 部署

对于 CI/CD 流水线或不支持 Python 的环境，REST API 同样适用，可帮助你监控部署。

首先，配置环境变量：


#!/bin/bash

# 配置

HF_TOKEN="${HF_TOKEN:-your-token-here}"

HF_USERNAME="${HF_USERNAME:-your-username}"

ENDPOINT_NAME="deepseek-v3-prod"

这些变量为你的组织上下文和凭证提供给 Hugging Face Endpoints API。确保 HF_TOKEN 具有 write 范围，并与目标账户/空间匹配。

此请求创建端点并镜像了 Python SDK 的字段：模型、服务镜像、token 限制、硬件和缩放。一些字段是提供者特定的（例如，缩放语义），因此如果您的区域或计划不支持某个选项，请省略或调整它们：


# 通过 curl 部署并正确处理错误

# 注意：一些缩放字段（例如 "scaleToZeroTimeout"）是特定于提供者的；如果不支持，请省略它们。

RESPONSE=$(curl -s -w "\n%{http_code}" \

  "https://api.endpoints.huggingface.cloud/v2/endpoint/${HF_USERNAME}" \

  -X POST \

  -H "Content-Type: application/json" \

  -H "Authorization: Bearer ${HF_TOKEN}" \

  -d '{

    "name": "'"${ENDPOINT_NAME}"'",

    "type": "protected",

    "model": {

      "repository": "deepseek-ai/DeepSeek-V3",

      "framework": "pytorch",

      "task": "text-generation",

      "fromCatalog": false,

      "image": {

        "tgi": {

          "url": "ghcr.io/huggingface/text-generation-inference:3.3.1",

          "healthRoute": "/health"

        }

      },

      "env": {

        "MAX_BATCH_PREFILL_TOKENS": "16384",

        "MAX_INPUT_LENGTH": "32768",

        "MAX_TOTAL_TOKENS": "65536"

      }

    },

    "compute": {

      "accelerator": "gpu",

      "instanceType": "nvidia-h200",

      "instanceSize": "x8",

      "scaling": {

        "minReplica": 0,

        "maxReplica": 1,

        "metric": "hardwareUsage",

        "measure": {

          "hardwareUsage": 80

        }

      }

    },

    "provider": {

      "vendor": "aws",

      "region": "us-east-2"

    }

  }')

我们解析 HTTP 状态和 JSON 响应体，以便在 CI 日志中提供清晰的成功/失败信号。状态码 200 表示部署已被接受；否则，错误载荷有助于定位访问权限、配额或无效字段问题：


# 解析响应

HTTP_CODE=$(echo "$RESPONSE" | tail -n1)

BODY=$(echo "$RESPONSE" | sed '$d')

if [ "$HTTP_CODE" -eq 200 ]; then

    echo "✓ Deployment initiated successfully"

    echo "$BODY" | jq -r '.status'

else

    echo "✗ Deployment failed with HTTP $HTTP_CODE"

    echo "$BODY" | jq -r '.error // .'

fi

使用 ZenML 集成部署

对于使用 ZenML 进行 MLOps 编排的团队，此部署可以无缝集成到现有流水线中。

导入所需模块：


from zenml import pipeline

from zenml.config import DockerSettings

from zenml.integrations.constants import HUGGINGFACE

from zenml.integrations.huggingface.services import HuggingFaceServiceConfig

from zenml.integrations.huggingface.steps import (

    huggingface_model_deployer_step,

)

ZenML 打包了 Hugging Face 集成和容器设置，使你的部署可在流水线中复现运行。这使得端点创建、重试和元数据跟踪都在一个版本化的流水线工作流下进行：


docker_settings = DockerSettings(

    required_integrations=[HUGGINGFACE],

)

@pipeline(enable_cache=True, settings={"docker": docker_settings})

def deepseek_deployment_pipeline(

    model_name: str = "deepseek-v3",

    timeout: int = 1200,

):

    # 使用生产配置进行配置

    service_config = HuggingFaceServiceConfig(

        model_name=model_name,

        endpoint_name="deepseek-v3-prod",  # 将添加 zenml- 前缀 + UUID

        repository="deepseek-ai/DeepSeek-V3",

        framework="pytorch",

        task="text-generation",

        accelerator="gpu",

        instance_type="nvidia-h200",

        instance_size="x8",

        vendor="aws",

        region="us-east-2",

        endpoint_type="protected",

        min_replica=0,

        max_replica=1,

        custom_image={

            "health_route": "/health",

            "url": "ghcr.io/huggingface/text-generation-inference:3.3.1"

        }

    )

    # 使用自动重试逻辑进行部署

    deployment = huggingface_model_deployer_step(

        service_config=service_config,

        timeout=timeout,

    )

    return deployment

service_config 镜像了 SDK 参数（模型、硬件、区域、缩放和镜像），并成为端点规范的事实来源。将其视为基础设施即代码；在提升到生产环境前请检查差异。


##### 运行流水线

if __name__ == "__main__":

    deepseek_deployment_pipeline()

此步骤触发实际的端点创建（具有内置重试逻辑），并返回一个可以后续检查的服务句柄。下游步骤可以轮询该句柄以获取状态、URL 和健康状况，从而控制发布检查：

验证与首次推理

部署完成后，验证端点是否可访问并执行测试推理：


##### 测试已部署的端点

def test_endpoint(endpoint):

    """使用简单的提示验证端点功能。"""

    try:

        # (此处原文截断，应包含测试代码)

        pass

    except Exception as e:

        print(f"Test failed: {e}")

2. 监控与成本管理

缩放到零机制有所帮助，但请注意，在 Hugging Face Inference Endpoints 上，此功能是可选参与的，并且通常在闲置约 15 分钟后才会触发。请密切监控：


##### 检查端点状态和成本

def monitor_endpoint(endpoint_name: str):

    from huggingface_hub import get_inference_endpoint

    endpoint = get_inference_endpoint(endpoint_name)

    endpoint.fetch()  # 刷新状态

    print(f"状态: {endpoint.status}")

    print(f"URL: {endpoint.url}")

    print(f"创建时间: {endpoint.created_at}")

    print(f"更新时间: {endpoint.updated_at}")

    # 根据状态估算每日成本

    hourly_cost = 40  # 8x H200 副本为 $40/小时

    if endpoint.status == "running":

        print(f"运行成本: ${hourly_cost}/小时 (${hourly_cost * 24}/天)")

    elif endpoint.status == "scaledToZero":

        print("✓ 已缩放到零 - 无计算费用")

3. 拆卸与清理

当您完成后，请进行清理以避免意外账单：


#### 优雅关闭

endpoint.scale_to_zero()  # 立即缩容

# 或

endpoint.pause()  # 完全暂停，需要手动恢复

# 或

endpoint.delete()  # 永久删除 - 请谨慎！

部署模式总结

本次部署展示了若干生产模式：

硬件抽象：您指定了硬件（8x H200），而无需管理底层基础设施。
成本优化：缩容至零防止了空闲期间账单失控。
容器定制：TGI 容器提供了针对大模型服务的特定优化。
默认安全：受保护的端点要求身份验证，防止未授权访问。

权衡是明确的：按每小时 40 美元计算，Hugging Face 的托管基础设施与 AWS 容量块（约 36 美元/小时）具有竞争力，并且显著低于按需定价（约 65-70 美元/小时）。您获得了自动扩缩容、监控和更新，无需承担运维开销。

您应该了解的一些运维考虑因素如下：

冷启动很重要：缩容至零后，第一个请求需要 2-3 分钟，因为模型会加载到 GPU 内存中。请相应地设计您的应用程序。
区域可用性：H200 并非在所有区域都可用。在承诺使用特定地理区域之前，请检查可用性。
批处理以提高效率：使用如此昂贵的硬件时，请尽可能批处理您的请求。TGI 服务器能高效处理并发请求。
监控 Token 使用量：在此规模下，即使是高效的模型也可能迅速产生成本。在您的应用程序逻辑中实现 Token 限制。

这些示例能让您快速启动并运行。维持可靠性、性能和成本需要运维实践（监控、扩缩容和治理），我们将在接下来讨论。

生产运维与监控

模型已部署。第一批请求正在流入。现在是困难的部分：在规模上保持可靠、高效和安全地运行。本节提炼了区分概念验证与生产系统的运维智慧。

监控与可观测性

LLM 运行在昂贵的 GPU 上；可见性对于可靠性和开支控制至关重要。重要的指标分为四类：

Token 计数直接转化为成本。跟踪每个请求的提示（prompt）和补全（completion）Token。大多数提供商在响应元数据中返回此信息：兼容 OpenAI 的 API 公开 prompt_tokens 和 completion_tokens 用量；Amazon Bedrock Converse 报告 inputTokenCount/outputTokenCount；Vertex AI 包含 usageMetadata.promptTokenCount/outputTokenCount。对于自托管引擎（vLLM/TGI），从服务器公开使用量和 Prometheus 指标（用于提示/生成 Token 的计数器），并可选择通过模型的分词器在客户端验证计数。这不仅仅是为了计费；突然的飙升通常揭示低效的提示，或者更糟的提示注入尝试。当您看到 Token 飙升时，首先问一个简单问题：是提示变大了，还是答案变长了？提示 Token 的阶跃变化通常意味着模板或 RAG 发生了变化（检索到的段落更多、分块方式不同），而补全 Token 的激增通常指向越狱或“失控的”推理，模型持续思考并达到长度限制。如果每分钟总 Token 量飙升，而每个请求的 Token 在众多用户间看起来正常，您可能看到的是使用量激增；如果飙升来自一小部分重复请求的用户群体，那可能是个重试循环或客户端错误。将飙升视为一次差异分析：比较前后的提示，如果常见头部增长则启用前缀缓存，限制最大 Token 以驯服长答案，如果输入膨胀则收紧检索设置。
延迟分布告诉您用户如何体验您的服务。对于 DeepSeek-R1，首个 Token 延迟（TTFT）可能因模型决定推理的深度而剧烈波动，使得 p95 延迟成为比平均值更好的服务等级目标（SLO）指标。
GPU 利用率值得特别关注。当 vLLM 报告 KV 缓存使用率接近容量时，您即将撞上性能悬崖。模型不会优雅降级；它会直接坠崖。将吞吐量想象成高速公路，KV 缓存就是车道；随着缓存填满和碎片化，车辆仍然驶入，但一切同时减速——每秒 Token 数下降，队列变长，TTFT 和 p95 一起上升。那个同步的跃升就是悬崖：您已经跨过一个容量边界，逐出和更小的有效批处理导致了非线性崩溃。从悬崖边返回的最快方法是缩减每个请求的需求（降低最大上下文或总 Token 数），减少并发序列，或临时丢弃低优先级流量；持久的解决方案是更多裕量（更大的 VRAM 或缓存），更好的批处理，以及通过共享前缀或 KV 复用提高缓存命中率。

使用 vLLM 的 Prometheus 指标来跟踪缓存压力和吞吐量（例如，每秒 Token 数、vllm:num_requests_running、运行中与等待中的请求数），并在 KV 缓存利用率接近您的容量阈值（例如约 90%）之前告警。

第四类是成本跟踪。将您的 Token 指标与当前定价表关联，并实时计算每个请求的美元成本。当该数字每周环比跳升 50% 时，说明有事情发生了——可能是新的提示模板，或者是您的缓存层出现了退化。

对于生产部署，vLLM 的 /metrics 端点开箱即提供兼容 Prometheus 的可观测性。将这些指标导出到您选择的时序数据库。添加一个 Prometheus scrape_config（或 Kubernetes ServiceMonitor），其目标指向 http://YOUR_VLLM_HOST:8000/metrics，如果您使用外部后端，请启用 remote_write 到您的时序数据库（例如 Thanos、Cortex 或 VictoriaMetrics），以便您可以在 Grafana 中可视化这些指标。Grafana 仪表盘包含专门针对 vLLM 部署的预构建可视化。

最终，指标只能告诉您发生了什么，而非原因。为了进行根本原因分析，您需要请求级别的可见性。像 Langfuse 这样的工具在提供自动 个人身份信息（PII）脱敏的同时，保留了足够的上下文来调试那些不可避免的“模型说了什么？”事件。

扩缩容与性能

DeepSeek 模型提出了一个迷人的扩缩容挑战。与您可以简单启动更多副本的传统 Web 服务不同，LLM 扩缩容需要理解模型架构、硬件拓扑和请求模式之间的相互作用。

天真的方法（将您的模型服务视为任何其他无状态应用程序）会惨败。原因如下：每个请求都会建立可为类似提示复用的 KV 缓存状态。在副本间随机分发请求会破坏这种缓存局部性。实现基于用户或会话 ID 的会话粘性路由。这个简单的更改可以在对话工作负载中将吞吐量提高 40%。

对于 DeepSeek-V3 的 MoE（混合专家）模型，每个 Token 只有一小部分专家（通常每层两个）处于活动状态。实践中，许多服务栈会在服务池中保持所有专家常驻，以最小化跨节点路由；然而，这不是硬性要求。您可以跨 GPU/节点分片专家（专家并行），或将部分专家卸载到 CPU/NVMe，以额外的全互联通信和更高延迟为代价，换取更低的单节点内存。

因此，水平扩缩容可以有两种形式：复制完整的 671B 参数集以增加并发性的副本，或通过专家并行分片增长总专家池而不重复所有权重的容量扩展。

根据您的延迟预算和互联速度进行选择；在较慢的互联条件下，拥有 NVLink/NVSwitch 的更少、更大的节点通常优于许多小实例。运维人员可以手动执行此操作，通过配置更大的 NVLink/NVSwitch 节点并限制放置（例如，使用 Kubernetes nodeSelector/affinity 和污点/容忍的节点池，或在裸机上进行显式 GPU 绑定），同时限制该服务的水平自动扩缩容。

生产环境中的性能优化是寻找瓶颈并系统性地消除它们的实践。以下是您实现优化的方式：

从连续批处理开始：传统服务顺序处理请求；连续批处理动态地将请求分组以最大化 GPU 利用率。这里的一个关键参数是 --max-num-seqs，它控制 vLLM 可以并发处理的最大序列（请求数）。正确设置此值是在吞吐量和内存稳定性之间的关键平衡。设置过低，您会在流量高峰时留下性能潜力；设置过高，您将面临灾难性的 内存溢出（OOM）错误。

理想值直接取决于模型权重加载后的可用 VRAM。您可以使用以下逻辑估算一个安全的起点：

考虑两个实际场景：

场景 1：高吞吐量聊天机器人：您在单个 80 GB H100 GPU 上运行一个 33B 模型（量化为 INT4，约 19 GB）。您的工作负载由许多并发用户组成，对话轮次较短（平均序列长度 512 个 Token）：
- VRAM 分配：总共 80 GB - 模型占用 19 GB ≈ 61 GB 可用。预留 10% 作为开销，约剩 55 GB 用于 KV 缓存。
- 每序列的 KV 缓存：对于此大小的模型，缓存大约为 25 KB/Token。因此，512 Token * 25 KB/Token ≈ 12.8 MB/序列。
- 计算：55,000 MB / 12.8 MB ≈ 4,296。由于内存碎片，此理论最大值是不切实际的。
- 实际值：--max-num-seqs 的一个安全起始值是 256（一个常见的默认值）。在真实流量下进行负载测试和监控 KV 缓存使用情况后，您可能会谨慎地将其提高到 512 或 1024 以最大化吞吐量。
场景 2：长上下文文档分析：您在 4x A100-80 GB 节点（总共 320 GB VRAM）上服务一个 70B 模型（BF16，约 140 GB）。工作负载涉及较少的并发请求，但文档非常长（平均序列长度 16,384 个 Token）：
- VRAM 分配：总共 320 GB - 模型占用 140 GB ≈ 180 GB 可用。预留 10% 作为开销，约剩 162 GB 用于 KV 缓存。
- 每序列的 KV 缓存：对于 70B 模型，缓存大约为 80 KB/Token。因此，16,384 Token * 80 KB/Token ≈ 1.31 GB/序列。
- 计算：162 GB / 1.31 GB ≈ 123 个序列。
- 实际值：鉴于每个请求的高内存压力，从一个更保守的 --max-num-seqs 值 32 开始是谨慎的。根据测试期间的稳定性，您可能会将其提高到 64，但超过此值将显著增加此工作负载的 OOM 错误风险。

最佳实践是计算一个保守的基线，然后通过监控 GPU 内存和 vLLM Prometheus 指标端点的 KV 缓存使用情况，经验性地向上调整该值。

缓存是您的秘密武器：您可以实施两层策略。第一层，KV 缓存，是 Transformer 模型生成文本的基础部分，由现代推理引擎（如 vLLM）自动处理。没有“开启” KV 缓存的标志；它是防止模型在每次生成新 Token 时重新计算上下文窗口中每个 Token 的核心过程。您通过上下文长度等参数间接管理其内存影响。

第二层是显式的提示前缀缓存，您可以为具有重复起始文本的工作负载启用此功能。在 vLLM 中，您通过使用 --enable-prefix-caching 标志启动服务器来激活此功能。启用后，vLLM 会自动检测多个请求是否共享公共前缀（例如系统提示或一组 few-shot 示例）。它仅在第一个请求时计算该前缀的 KV 状态并存储。所有随后开始于完全相同前缀的请求都会重用缓存的 KV 状态，跳过昂贵的重新计算。这样，系统提示和模板可以无限期缓存，显著加快常见任务的预填充阶段。缓存命中率超过 60% 通常可将延迟降低三分之一，并按比例削减成本。

思考成本：对于 R1 模型，您面临一个独特的决策：多少思考才算过多？在延迟敏感的应用程序中，考虑实现推理截断。监控推理 Token 计数的分布，并根据您的 SLA 设置限制。是的，您可能会牺牲一些准确性，但用户更喜欢快速、足够好的回答，而非 arrives too late 的完美答案。

在扩缩容、缓存和思考边界都设定好后，下一步就是将这些转化为实际的美元或欧元金额：Token、缓存命中率和 GPU 利用率如何转化为每个请求的成本和每个用户的成本。

成本管理

LLM 成本有一种指数级增长的坏习惯。与成本随流量线性扩展的传统基础设施不同，LLM 成本可能基于提示设计、模型选择或用户行为模式而爆发式增长。单一提示模板的更改就可能在一夜之间使您的账单增加三倍。

这里是一种您可以尝试的务实成本管理方法：

从请求级预算开始。每个请求都应包含一个基于预期 Token 使用量的成本估算。在请求到达模型之前，拒绝超过每用户或每端点预算的请求。这不仅仅是为了防止滥用；更是为了在配置错误耗尽您的银行账户之前将其捕获。

为了应用此方法，在 API 网关或推理服务前的中间件层中实现检查。对于每个传入的请求，通过将 max_tokens 参数乘以每个输出 Token 的成本来计算 max_potential_cost，并加上输入 Token 的成本。将每用户或每密钥的每日/每月预算存储在快速访问数据库（如 Redis）中。然后中间件执行简单的检查：

if (user_current_spend + max_potential_cost > user_budget) { return 429_BUDGET_EXCEEDED; }

这个预计算步骤成本低廉，且能有效将您的模型与高成本请求隔离开来。

基础设施效率需要用不同的视角来看待。Kubecost 的 GPU 模块揭示了残酷的现实：您配置的 8 GPU 集群可能 40% 的时间都处于空闲状态。以每 H100 小时 $2.48 的价格计算，空闲时间就成了一种您负担不起的奢侈。但不要只看平均值——要绘制使用率随时间变化的图表。可预测的模式（夜晚和周末使用率低）表明您需要自动伸缩。随机的尖峰可能意味着批处理作业调度存在问题。
实际上，您可以通过使用 NVIDIA DCGM Exporter（github.com/NVIDIA/dcgm-exporter）将 GPU 指标暴露给 Prometheus，从而在 Kubernetes 上实现这一点。然后，创建一个 Grafana 仪表板，其中包含一个显示 avg(dcgm_gpu_utilization) by (pod) 的面板。对于自动伸缩，使用 Prometheus 伸缩器配置 KEDA。您的 ScaledObject 定义将针对诸如 sum(rate(vllm_requests_running[2m])) 之类的指标，允许您基于实际推理负载而非仅仅是 CPU 或内存（它们通常是 GPU 工作负载的糟糕代理）来伸缩副本。
最阴险的成本蔓延源于模型漂移。您部署了一个出色、高效的 330 亿参数模型。六个月后，您不知怎地就在生产环境中运行着 700 亿参数的变体，却没人记得为什么。请按模型版本跟踪每个输出令牌的成本，并在发生变化时发出警报。每周 20% 的增长值得调查；50% 的增长则需要立即采取行动。
要实施此方案，请确保每个推理日志都包含 model_id、input_tokens 和 completion_tokens。在您的日志平台（如 Datadog 或 ELK 技术栈）中，通过使用价格查找表（例如 deepseek-v3_input_cost = $0.27/M_tokens）来丰富这些日志，从而创建一个指标。然后，您可以构建一个仪表板，可视化 sum(request_cost) / count(requests) 并按 model_id 分组。设置一个自动监控器，将此指标的周平均值与上周进行比较，如果百分比变化超过定义的阈值，则触发警报。
预算警报应呈几何级数上升。在您的预算的 50%、80% 和 100% 处设置警报。50% 警报是信息性的。收到此警报时，您应检查增长是否符合预期。80% 警报会通知值班人员，表明需要关注某些问题。100% 警报应触发自动节流。宁愿降低服务质量，也不要在醒来时面对一笔六位数的云服务账单。
您可以使用云提供商的原生工具来设置此功能。例如，在 AWS 中，创建一个具有三个警报阈值的 AWS 预算。50% 的操作可以是一个发送到 Slack 频道以提高可见性的 SNS 通知。80% 的操作可以是一个触发 PagerDuty 或 Opsgenie webhook 的 SNS 通知。对于 100% 的操作，让 SNS 触发一个 Lambda 函数，该函数以编程方式将您的 API Gateway 使用计划修改为高度受限的节流限制（例如，每分钟一个请求），或更新 AWS AppConfig 中的功能标志以优雅地禁用服务。

设置了支出护栏（预算、警报和自动节流）之后，下一个问题是韧性：我们如何防止这些相同的控制被滥用、提示词注入或数据渗出所绕过？好消息是，您刚刚设置的基元（网关、使用计划和功能标志）同样可以作为安全执行点。

实践中的安全

LLM 带来了传统应用安全手册未曾涉及的安全挑战。您可能遇到的一些挑战如下：

提示词注入是真实发生且确实存在的。
基于请求次数的速率限制可以阻止用户发送 100k 令牌的提示。
标准身份验证可能无法区分用户请求俳句与请求长篇大论之间的成本差异。

网络安全仍然是基础但远远不够。因此，您需要将模型部署在私有子网中，用 API 网关作为前端，并在服务间使用 mTLS（即所有标准网络安全实践均适用）。但您还需要添加特定于 LLM 的控制措施：

输入验证不仅必须检查 SQL 注入，还必须检查提示词注入模式。首先根据允许列表进行验证：强制执行严格的模式（预期的 JSON 键/类型）、长度和字符集限制，并拒绝控制令牌。然后，将输入通过一个提示词注入过滤器，该过滤器结合了简单规则（正则表达式匹配诸如 ignore previous instructions、reveal system prompt、越狱标记和 Unicode 同形异义词等短语）和针对已知攻击样本的嵌入相似性检查；阻止或挑战高风险请求，并将标记的日志发送到您的集中式日志存储。
输出过滤不仅需要捕获 PII（个人身份信息），还需要捕获越狱尝试。
DeepSeek 模型的独特挑战在于其能力。一个足够聪明、能够处理复杂问题的模型，也足够聪明地被创造性地滥用。对提示词实施语义相似性检查，与已知的越狱模式进行比较。常见模式包括指令覆盖（ignore previous instructions、reveal/repeat the system prompt）、Do Anything Now（DAN）/开发者模式角色扮演、翻译/编码包装器（Base64/ROT13）、JSON/Markdown 围栏逃逸，以及旨在诱导禁止内容的 for research/simulation 请求。使用 OWASP 的 LLM Top 10 示例（owasp.org/www-project-top-10-for-large-language-model-applications/）、PromptBench 攻击集（github.com/microsoft/promptbench）以及来自 NVIDIA NeMo Guardrails 的样本越狱（github.com/NVIDIA/NeMo-Guardrails）等公共参考来填充您的相似性索引。但不要仅仅依赖黑名单；它们总是落后于一个创意提示。相反，实施纵深防御：按令牌数量而非仅仅按请求数进行速率限制。监控异常的令牌消耗模式并跟踪提示复杂度指标。
对于敏感部署，请考虑完整的攻击链。攻击者可能不会直接攻击您的模型，而是可能利用它来探测您的基础设施。监控询问系统配置、内部细节或错误消息的提示。通过将所有请求路由到记录提示词并运行轻量级分类器（正则表达式规则或本地 LLM）的 API Gateway 或中间件，实施服务器端提示检查，以标记“侦察”意图，例如询问系统提示、环境变量、堆栈跟踪、文件路径或 IP 地址。将这些标记的事件转发到您的集中式日志存储（例如 CloudWatch、Datadog 或 Splunk），并在高风险匹配或每个用户/会话的尖峰时发出警报，同时配合基于令牌的速率限制以及针对重复命中的自动挑战/拒绝规则。这些侦察尝试通常在更严重的攻击之前发生。

模型的 CI/CD

模型部署不是一次性事件——它是一个持续的过程。新版本承诺更好的性能，安全补丁要求立即部署，而经过微调的变体终于在测试中证明了其价值。没有适当的 CI/CD，模型更新就会变成每个人都想避免的高压事件。

以下是一个可帮助您入门的行动计划：

从版本管理纪律开始。即使 DeepSeek 发布了模型版本，您实际部署的是一个组合制品；它是基础模型修订版加上您的量化、分词器、适配器/LoRA、提示模板、解码限制、安全过滤器、推理引擎/容器和配置。进行版本控制是为了能够重现行为、安全回滚、满足审计要求以及进行 A/B 测试；仅供应商模型版本无法解释生产中的行为或成本差异。所有这些都说明：模型不仅仅是文件——它们是具有依赖项、配置和行为契约的制品。使用一个模型注册表（MLflow 或云原生选项）来跟踪不仅仅是权重，还包括整个部署上下文。当出现问题时，您需要确切知道改变了什么。

要付诸实践，请为您的组合制品采用一个具体的版本控制方案，该方案既要人类可读又要机器可解析——例如，[base-model-version]-[quantization]-[adapters-hash]-[config-version]，它可能看起来像 deepseek-v3-b1.0-int4-lora-abc123-v2。这里，abc123 可以是您训练 LoRA 的仓库的简短 Git 提交哈希，而 v2 指的是您的解码配置文件的版本。这为生产环境中运行的确切制品创建了一个明确的标识符。

模型注册表，例如 MLflow，是您将所有这些整合在一起的地方。您的 CI 流水线（由提交触发）应打包这些组件并将其记录到单个 MLflow 运行中。使用 mlflow.log_artifact() 存储您的分词器配置、提示模板文件和安全过滤器脚本。使用 mlflow.log_param() 记录键值元数据，例如 quantization_method: AWQ 或 base_model_id: "deepseek-ai/DeepSeek-V3"。
最后，将该整个运行注册为名为 "production-chat-endpoint" 的模型的新版本。现在，当您在 MLflow UI 中比较版本 1.2.0 和 1.3.0 时，您看到的不仅是新的模型文件，还有可能导致行为或成本变化的每个依赖项和配置更改的精确差异。
蓝绿部署对于模型来说效果很好。在旧版本旁边启动新版本，使用合成流量进行验证，然后逐渐将真实用户迁移过去。关键见解是，不要只测试功能正确性。在金丝雀阶段监控延迟、令牌使用量和成本指标。一个准确率提高 2% 但成本增加 50% 的模型可能算不上升级。

在 Kubernetes 环境中，您可以使用服务网格（如 Istio 或 Linkerd）或 API Gateway（如 Emissary-ingress）来实现这一点。该模式涉及创建两个 Kubernetes Deployment：一个用于旧模型版本（例如 deepseek-v1），一个用于新的金丝雀版本（deepseek-v2）。一个 Kubernetes Service 对象同时指向这两个部署。然后，您配置一个路由规则（例如 Istio VirtualService），最初将 100% 的流量定向到 v1：

您的 CI/CD 流水线（例如使用 GitHub Actions 或 GitLab CI）自动化了流量迁移。滚动部署作业的第一步是应用新的路由规则，仅将 1% 的流量发送到 v2。然后流水线暂停，并查询您的可观测性平台（例如 Prometheus）的关键指标，在 Grafana 仪表板上并排比较 v1 和 v2 部署。如果在设定的一段时间内（比如 15 分钟），v2 的 p95 延迟、令牌使用量和计算出的每请求成本保持在可接受的范围内，流水线将自动继续将流量权重增加到 10%，然后是 50%，最后是 100%，从而完成滚动部署。
回滚能力不是可选项。保持前一个模型版本处于热备状态并随时待命。当指标异常时，您需要一键式回滚，而不是手忙脚乱地重新部署过程。这意味着接受临时运行两个版本的基础设施成本。将其视为保险。

这个一键式回滚就是您部署流水线中的紧急制动。这不是一次新的部署；它是一个配置更改，可以立即将流量重新路由。如果您的金丝雀监控仪表板显示关键指标回归，例如有害内容生成激增或延迟突然翻倍，您就触发回滚。在您的 CI/CD 工具中，这应该是一个手动触发的作业，执行单个 kubectl apply 或向您的服务网格发送 API 调用，将流量路由规则更新为将 100% 的流量送回稳定、之前运行的版本。整个回滚过程应只需几秒钟，而不是几分钟。
7. 保持前一个版本热备意味着旧模型的 Kubernetes Deployment 在整个金丝雀阶段保持其完整的副本数。它正在主动运行，并准备随时接收 100% 的生产负载。保险成本是实实在在的：如果单个模型副本每小时花费您 $5，并且您运行两个副本以实现高可用性，那么在滚动部署期间，您的成本将暂时从每小时 $10 增加到每小时 $20。一旦新版本被完全提升并确认稳定（例如，24 小时后），您流水线的最后一步应自动将旧部署的副本数缩减为零，从而消除额外的成本。
8. 模型的测试金字塔与传统软件不同。单元测试验证提示格式和响应解析。集成测试确认模型可以加载和响应。但关键的层次是行为测试：此模型是否保持了前一个版本的安全边界？它是否一致地处理边缘情况？构建一个包含提示和预期行为的黄金数据集，并在回归测试通过后才允许部署。

实际上，这些测试应作为 CI 流水线中的阶段运行。单元测试是在每次提交时运行的快速检查。使用 pytest 之类的框架，您可以编写一个简单的函数来验证提示模板：assert "User:" in format_prompt(history, query)。另一个测试可以确认您的响应解析器处理预期的格式，并在 JSON 格式错误时引发特定错误：pytest.raises(MalformedJsonError): parse_response("{'invalid': 'json'}")。这些测试在模型加载之前就能捕获逻辑错误。
9. 集成测试在容器构建后运行。您的流水线启动模型容器，并使用 curl 之类的工具访问其健康检查端点，然后用一个简单的 "hello" 提示访问其推理端点。测试只需断言返回 200 OK 状态码和有效的响应结构。这可以捕获依赖项、模型加载或环境配置的问题。
10. 行为回归测试是部署前最后也是最重要的关口。您的流水线将新模型部署到临时环境，并针对其运行测试套件。该套件会遍历您的“黄金数据集”（例如，Git 仓库中的 test_cases.jsonl 文件）。对于每个提示，它断言特定的行为。对于安全性，您可能断言有害提示会导致拒绝响应：assert "cannot comply" in response.lower()。对于数据提取等关键任务的一致性，您可以将 LLM 的 JSON 输出与预定义的模式进行比较。对于更细微的质量（如语气或有用性），您甚至可以使用一个更小、更便宜的评判 LLM 来评估新模型的响应，并标记出与前一版本基线响应相比任何显著的负面变化。此阶段的失败应自动阻止生产环境的滚动部署。

有了监控、伸缩、成本控制、安全性和 CI/CD，您的端点就从脆弱的演示演变成了持久的服务。我们将以一个简洁的操作手册作为结束。

您的部署操作手册

您现在拥有了完整的部署栈：从本地开发代理到多 GPU 生产集群。决策框架可归结为三个约束：数据敏感性、令牌量和运维能力。

以下是您的操作手册：

常见场景速查：
- 原型和开发：使用 DeepSeek API，直到遇到速率限制或成本顾虑。对于离线工作，切换到使用较小模型的本地 Ollama。在验证用例之前，不要优化基础设施。
- 处理敏感数据的生产环境：如果托管服务（Bedrock/Vertex）满足合规要求，则从它们开始。仅在托管选项在法律或技术上不足时才自托管。为自托管学习曲线预留 2-3 个月。
- 高流量生产环境（>10 亿令牌/月）：从数学上讲自托管更有利，但前提是拥有专门的 DevOps 资源。进行成本分析，包括人员时间，而不仅仅是 GPU 小时。考虑混合部署：API 用于可变负载，自托管用于基线流量。
需要避免的常见陷阱：
- 在未进行容量规划的情况下部署完整的 DeepSeek-V3（有关规模指导，请参阅关于硬件和推理优化引擎的章节）。
- 忽略缩放到零的能力（空闲时 $40/小时与 $0/小时的差别）。
- 将 R1 的推理令牌视为免费（它们会计入成本和延迟）。
- 在未监控令牌使用量的情况下部署（账单可能因提示更改而一夜之间增加 10 倍）。
实施清单：
- 实施请求级别的成本跟踪。
- 在 50%、80% 和 100% 阈值处设置预算警报。
- 缓存系统提示和常见前缀（60% 以上的命中率是可实现的）。
- 对于对话式工作负载，使用粘性路由。
- 保持前一个模型版本热备，以便即时回滚。

生态系统发展迅速。今天的 8×H200 需求可能是明天的单 GPU 部署。专注于构建能够经受基础设施变化的抽象层：标准化 OpenAI 兼容 API，对所有部署进行版本控制，并保持从开发到生产的环境一致性。

以下是一些对工程师可能有用的提示。

对于初级工程师：从动手部署指南部分的示例 1（Ollama）开始。在处理云部署之前，请先熟悉请求/响应周期和令牌经济学。
对于高级工程师：直接跳到示例 3（Hugging Face Inference Endpoints）进行生产部署，但请查看监控部分。LLM 的可观测性与传统服务不同。
对于平台团队：专注于推理引擎比较和生产运维指导将加速您的基础设施决策。

请记住

部署是达到目的的手段。选择满足您需求的最简单方法，然后随着您对实际使用模式的了解而迭代。

总结

本章将部署从理论转变为在生产环境中运行全参数 DeepSeek-V3/R1 的务实操作手册。您通过控制/成本/复杂度视角绘制了全景图（API vs. 托管 vs. DIY），学习了如何配置硬件规模（VRAM、KV 缓存、TTFT/吞吐量），并有意识地选择了推理引擎（vLLM/TGI/TensorRT-LLM）以及量化选项（FP8、GPTQ 和 AWQ）以满足性能和预算要求。三个具体路径锚定了指导：本地开发代理（Ollama）用于迭代，托管服务（Bedrock/Vertex/Azure/Hugging Face）用于企业可靠性，DIY（vLLM 在 GPU/Kubernetes 上）用于最大控制力和规模经济效益。

然后，您实现了服务化：监控令牌/延迟/GPU 和 KV 压力，在活动生成时通过连续批处理和粘性路由进行伸缩，缓存共享前缀，并通过缩放到零和预算警报强制执行基于令牌的预算。安全性超越了网络基础知识，扩展到提示词注入防御和输出过滤。最后，您将模型视为版本化的组合制品，并通过蓝绿/金丝雀部署和即时回滚安全地发布。凭借这套工具包，您可以选择满足当前约束的最简单部署方案，并随着工作负载的增长而充满信心地演进它。

获取本书的 PDF 版本和独家附加内容

扫描二维码（或访问 packtpub.com/unlock）。按书名搜索本书，确认版本，然后按照页面上的步骤操作。 | |
| 注意：请保留好您的发票。直接从 Packt 购买无需发票。 |

9 后记

穿越本书的旅程，追溯了从 DeepSeek 的基础原理到将其部署到生产级系统的路径。我们从其颠覆性的市场进入开始，到大规模运行它的运营现实结束。本后记综合了这一历程，将核心技术创新与利用它们所需的实践技能联系起来。

DeepSeek 范式

DeepSeek 的发布标志着一个关键时刻，证明了一个开源模型可以在推理方面达到前沿性能，直接挑战了专有系统的主导地位。这不是一次渐进式改进，而是一次由架构和训练创新融合驱动的范式转变。本书首先探讨了这些技术基础。

其核心是 混合专家（MoE）架构，它实现了巨大的规模（超过 6000 亿参数），同时通过仅激活每个令牌模型的一部分来保持计算效率。这与 多头潜在注意力（MLA） 相结合，后者压缩了 KV 缓存，使长上下文理解变得实用且内存高效。

然而，最重要的突破是训练方法。通过优先考虑具有自动化、基于规则的奖励的强化学习——体现在 群体相对策略优化（GRPO） 等技术中——DeepSeek 学会在回答前生成结构化、逐步的推理，而无需大量的监督微调。这种 先思考再回答 的能力成为了该模型的标志性特征及其实际应用的核心主题。

从理论到实践

理解架构只是第一步。随后的章节从 DeepSeek 是什么 过渡到如何 有效使用它。这需要抛弃已确立的提示习惯。我们看到 DeepSeek 的推理模型行为不像聊天机器人，更像定理证明器，在这种场景下，极简、上下文丰富的提示优于有利于其他模型的详细、少样本示例。

掌握了这种新的思维模型后，我们构建了一个完整的应用程序，展示了从本地 CPU 推理到托管云服务的全谱系部署。这一实践练习突显了结构化输出对于创建可靠和灵活系统的重要性。然后，我们将此扩展到自主系统，探索将复杂任务分解、迭代精炼输出并与外部工具交互的智能体模式。

核心教训是：DeepSeek 是一个强大的推理引擎，当被正确驾驭时，可以自动化复杂的分析和决策。

通往生产的道路

最后的章节探讨了最终目标：在真实的业务环境中部署 DeepSeek。我们探讨了两种不同且强大的策略来实现这一目标。

首先，理性蒸馏 工作流展示了如何将 DeepSeek-R1 不作为最终产品，而是作为教师来使用。通过为法律合同分析等专业领域生成解释性原理，我们创建了一个增强的数据集来训练一个更小、更高效的学生模型。结果是一个可部署的模型，它不仅私密且经济高效，而且在其专业任务上比它大得多的教师模型要准确得多。

其次，对于需要完整模型全部能力的用例，我们详细介绍了 DeepSeek V3 和 R1 的部署操作手册。这涉及在 API、托管服务和自托管基础设施之间导航成本、控制和复杂性之间的关键权衡。从选择合适的 GPU 硬件和推理引擎，到实施稳健的监控和安全，我们建立了一个框架来做出合理的运营决策。

本书详述的原则和模式是持久的。DeepSeek 代表的不仅仅是一个模型系列；它提供了一个由强大、易用和可适应的推理定义的 AI 时代的蓝图。穿越其架构、应用和部署的旅程，已为您构建下一代智能系统提供了必要的技能。

附录

是时候了解我们使用 DeepSeek 模型的不同方式了。

如果您已经想清楚如何使用 DeepSeek，可以跳过本附录。

但如果没有，我们将为您展示一些方法。

我们将介绍一些（截至撰写时）运行中国 AI 实验室发布的一些主要模型的最流行方式。

我们将涵盖以下使用 DeepSeek 的方式：

入门使用官方 DeepSeek API
使用常见的第三方 API
使用 Cursor 的 IDE 操作 DeepSeek
自行运行或部署 DeepSeek
为 DeepSeek 构建自己的设置

技术要求

需要安装以下内容：

回忆一下 第 5 章，在使用 DeepSeek API 处理 Python 示例时，我们将使用 uv。如果您尚未安装，可以按照此处的说明安装 uv：docs.astral.sh/uv/getting-started/installation/。
要运行基于终端的示例，我们将使用 curl。要安装 curl，您可以访问官方网站 curl.se/；或者 everything.curl.dev/install/index.html 网站显示了所有主要平台的安装说明。
要跟随一些示例，您需要设置 API 密钥，我们将在每个示例中逐步介绍如何设置。
所有代码示例都已正确记录在此处：github.com/PacktPublishing/DeepSeek-in-Practice。

注意

虽然我们使用了 macOS，但所有代码示例和说明应适用于所有平台。对于 Windows 用户，我们建议使用 Windows Subsystem for Linux（WSL）（learn.microsoft.com/en-us/windows/wsl/install）。

入门使用官方 DeepSeek API

像大多数 AI 实验室和公司一样，DeepSeek 有一个类似 ChatGPT 的网络应用程序，托管在 chat.deepseek.com/，以及一个 API 平台，托管在 platform.deepseek.com/。这种设置对大多数公司（包括像 Qwen 这样的开源项目：chat.qwen.ai/）来说很常见。值得注意的是，在 2025 年 1 月发布 DeepSeek-R1 之后，该 API 平台突然遭到大规模网络攻击（www.cnbc.com/2025/01/27/deepseek-hit-with-large-scale-cyberattack-says-its-limiting-registrations.html）。此后，该平台已恢复运行，并且稳定，是那些想要开始使用 DeepSeek 构建的好选择。

本节的目标是向您简要介绍 DeepSeek 的官方平台和 API。请记住，DeepSeek API 是一个付费且（希望）持续发展的产品。因此，我们将根据其 2025 年 6 月左右的状态向您介绍 API 平台。这可能会随时间而变化。因此，如果看起来有所不同，请考虑阅读 API 文档 api-docs.deepseek.com/。这不是一个全面的指南——只是一个突出主要功能的概述。事不宜迟，让我们开始设置。

设置

在这里，我们将向您展示如何开始使用 DeepSeek API 进行构建。要进行设置，请遵循以下步骤：

访问平台网站 platform.deepseek.com/。
创建一个账户。也可以选择使用 Google 登录。
导航到左侧的 Top up。
使用您首选的支付方式为您的账户添加一些额度。
导航到 API keys。
点击 Create new API key。
保存您的 API 密钥（通常以 sk-XXXX 开头）。

现在我们有了 API 密钥，让我们运行一个小测试以确保一切正常。如果您在 Mac 或 Linux 上，打开终端并键入以下内容，将 <DeepSeek API Key> 替换为正确的值：

DeepSeek API 调用指南

1. 通过 `curl` 进行调用

你可以使用 curl 命令直接向 DeepSeek API 发送请求。


curl https://api.deepseek.com/chat/completions \

  -H "Content-Type: application/json" \

  -H "Authorization: Bearer <DeepSeek API Key>" \

  -d '{

        "model": "deepseek-chat",

        "messages": [

          {"role": "system", "content": "You are a helpful assistant."},

          {"role": "user", "content": "What is the capital of Portugal?"}

        ],

        "stream": false

      }'

你将会看到类似如下的输出：


{"id":"3dbb972a-80e1-4558-b310-deeaeac1968d","object":"chat.completion","created":1749810137,"model":"deepseek-chat","choices":[{"index":0,"message":{"role":"assistant","content":"The capital of Portugal is **Lisbon** (Portuguese: *Lisboa*). It is the largest city in the country and is known for its historic neighborhoods, vibrant culture, and scenic location along the Atlantic Ocean."},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":16,"completion_tokens":44,"total_tokens":60,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":16},"system_fingerprint":"fp_8802369eaa_prod0425fp8"}

如果没有成功，请确保你已遵循所有步骤，并正确设置了账单和 API 密钥。

2. 通过 Python 进行调用

要在 Python 中进行首次 API 调用，请遵循以下步骤：

创建一个名为 01-deepssek-api-call.py 的脚本。
通过运行 export DEEPSEEK_API_KEY=sk-XXXXXX 导出你的 DeepSeek API 密钥（或者使用某种环境管理器，例如 direnv (direnv.net/)）。
将以下内容复制到脚本文件中：


# /// script

# requires-python = ">=3.12"

# dependencies = [

#     "openai",

# ]

# ///

from openai import OpenAI

import os

API_KEY = os.getenv("DEEPSEEK_API_KEY")

assert API_KEY, "Please set the DEEPSEEK_API_KEY environment variable."

BASE_URL = "https://api.deepseek.com"

client = OpenAI(api_key=API_KEY, base_url=BASE_URL)

response = client.chat.completions.create(

    model="deepseek-chat",

    messages=[

        {"role": "system", "content": "You are a helpful assistant"},

        {"role": "user", "content": "What is the capital of Portugal?"},

    ],

    stream=False,

)

print(response)

# ChatCompletion(id='1bdf1ecf-a365-4f8b-9c26-fbebff421127', choices=[Choice(finish_reason='stop', index=0,

# logprobs=None, message=ChatCompletionMessage(content='The capital of Portugal is **Lisbon** ...

使用 uv 的好处在于，我们无需处理 Python 版本的安装和设置。现在，你只需运行脚本即可：


$ export DEEPSEEK_API_KEY=sk-XXXXXX

$ uv run 01-deepseek-api-call.py

你可以预期看到类似如下的输出：


Reading inline script metadata from `01-deepseek-api-call.py`

ChatCompletion(id='41164ddd-c648-4945-96c2-67f64e6d62a1', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='The capital of Portugal is **Lisbon** (Portuguese: *Lisboa*). It is the largest city in the country and is known for its historic neighborhoods, vibrant culture, and scenic location along the Atlantic Ocean and the Tagus River.', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None))], created=1749375653, model='deepseek-chat', object='chat.completion', service_tier=None, system_fingerprint='fp_8802369eaa_prod0425fp8', usage=CompletionUsage(completion_tokens=49, prompt_tokens=15, total_tokens=64, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0), prompt_cache_hit_tokens=0, prompt_cache_miss_tokens=15))

你刚刚使用 Python 完成了对 DeepSeek 的首次 API 调用！

3. 关于 API 兼容性与响应结构

与许多其他大语言模型（LLM）提供商一样，DeepSeek API 与 OpenAI API 格式兼容。这意味着你可以使用 OpenAI Python SDK (github.com/openai/openai-python)，只需更换 api-key 和 base-url 关键字参数，而无需关心其内部工作原理。这非常棒，因为它使用户能够相对容易地切换提供商——无需完全重构其代码库。但请注意：并非所有 API 功能都将对每个提供商可用。因此，请务必彻底测试。

现在，让我们看看响应结构。当你调用 DeepSeek API 时，你会收到一个遵循 OpenAI 兼容格式的 JSON 对象：


ChatCompletion(

  id='41164ddd-c648-4945-96c2-67f64e6d62a1',

  object='chat.completion',

  created=1749375653,

  model='deepseek-chat',

  system_fingerprint='fp_8802369eaa_prod0425fp8',

  choices=[

    Choice(

      index=0,

      finish_reason='stop',

      logprobs=None,

      message=ChatCompletionMessage(

        role='assistant',

        content='The capital of Portugal is **Lisbon** (Portuguese: *Lisboa*)...',

        refusal=None,

        annotations=None,

        audio=None,

        function_call=None,

        tool_calls=None

      )

    )

  ],

  usage=CompletionUsage(

    prompt_tokens=15,

    completion_tokens=49,

    total_tokens=64,

    completion_tokens_details=None,

    prompt_tokens_details=PromptTokensDetails(

      audio_tokens=None,

      cached_tokens=0

    ),

    prompt_cache_hit_tokens=0,

    prompt_cache_miss_tokens=15

  ),

  service_tier=None

)

响应对象的关键组件如下：

choices：一个聊天补全选项的列表，每个选项包含一个结束原因和一条消息。这些选项也可能包含工具调用，稍后会详细说明。
id：唯一的请求 ID。
model：所使用的模型——我们稍后会详细讨论这个。
system_fingerprint：模型运行时的后端配置。
usage：你可能关心的使用统计信息（补全 token 的数量、缓存统计信息等）。

既然我们已经对如何调用 DeepSeek API 有了更好的了解，让我们进一步了解有哪些可用的模型以及如何利用它们。

使用可用模型

DeepSeek AI 实验室已经发布了大量的模型。你可能已经从代码示例中注意到，我们最初的 API 调用是发送到一个名为deepseek-chat的模型。如果这个名字听起来不熟悉，那是因为它并不是实际模型的名称！在他们的 API 中，DeepSeek 只提供了两个端点和两个模型：

deepseek-chat：这是公司提供其非推理模型的端点。在撰写本文时，此端点下的模型是非思考模式下的 DeepSeek-V3.2-Exp。
deepseek-reasoner：这是公司发布其著名推理模型的端点。这些是经过推理训练或能够进行推理的模型。在撰写本文时，此端点指向思考模式下的 DeepSeek-V3.2-Exp。

值得指出的是，当你读到这篇文章时，公司可能已经发布了更多模型。因此，这两个端点的底层模型可能已经发生了变化。请务必查阅 DeepSeek API 文档（api-docs.deepseek.com/）以获取关于模型的最新信息。

现在是时候了解可能影响你使用 DeepSeek API 的因素了。

Temperature

temperature参数修改模型生成的下一个 token logits 上的 softmax 概率分布。具体来说，它在应用 softmax 函数之前对 logits 进行缩放。如果你对这个话题感兴趣，我推荐阅读 Luke Salamone 的这篇精彩博文（blog.lukesalamone.com/posts/what-is-temperature/）。因此，较低的温度使输出更确定，而较高的温度则使输出不确定性增加。例如，OpenAI 也提供了一个seed参数，以尝试使输出更具确定性。然而，如果上下文增长，即使我们将温度设置为 0（以获得非常确定的输出），我们仍可能看到变化。

DeepSeek 建议用户根据需要使用不同的温度值（api-docs.deepseek.com/quick_start/parameter_settings）。例如，对于更具创造性的任务，可以设置更高的温度。

我的建议是，如果你正在集成到一个更大的、理想情况下是确定性的系统中，请务必将温度设置为 0。

定价和速率限制

与大多数 LLM 提供商一样，DeepSeek 按 token 收费。这意味着用户为输入和输出的 token 付费。作为参考，一个英文字符≈0.3 个 token，而一个中文字符≈0.6 个 token。DeepSeek 使用自己的分词器，并提供了一个可用的包（api-docs.deepseek.com/quick_start/token_usage#calculate-token-usage-offline）来计算 token 数量。

DeepSeek API 以其相较于大型 AI 公司极具竞争力的定价而闻名，因此在选择提供商时，它绝对是一个值得考虑的选择。同样值得注意的是，deepseek-reasoner的价格大约是非推理模型的 2 倍——对于 LLM 来说这是标准做法。

还要注意，推理模型会输出更多的 token。由于按 token 收费，推理模型实际上更昂贵。同时也要注意，DeepSeek（与大多数提供商一样）还提供提示缓存功能——缓存部分提示以使请求更快、更便宜。

与其他提供商不同，DeepSeek不对用户的请求进行速率限制。这与大多数提供商非常不一致。LLM API 以不稳定著称，因此——由于高需求——提供商往往会严格地限制 API 调用。DeepSeek 的 API 文档中有如下说明：

“我们将尽力服务每个请求。但是，请注意，当我们的服务器承受高流量压力时，你的请求可能需要一些时间才能从服务器获得响应。”

与其他提供商相反，DeepSeek 不会抛出RateLimitError或类似错误，而是会简单地保持你的请求，直到它可以完成。对于非流式请求，它们会发送空行；对于流式请求，它们会持续发送: keep-alive事件。一个很好的替代方法是重试请求，以确保你的 API 调用可靠：


from tenacity import retry, stop_after_attempt, wait_fixed

@retry(stop=stop_after_attempt(3), wait=wait_fixed(2))

def make_request() -> str:

    response = client.chat.completions.create(

        model="deepseek-chat",

        messages=[

            {"role": "system", "content": "You are a helpful assistant"},

            {"role": "user", "content": "What is the capital of Portugal?"},

        ],

        stream=False,

        max_tokens=100,

    )

    response_text = response.choices[0].message.content

    if not response_text:

        raise ValueError("Received empty response from the API.")

    return response_text

上面的示例将在执行期间发生异常（例如，由于请求失败或空响应）时，重试make_request函数最多三次，每次尝试之间等待两秒。如果函数在未引发异常的情况下成功执行，它将返回 API 响应文本。如果所有尝试都失败，则会引发最后一个异常。

现在让我们将注意力转向 DeepSeek API 的功能特性。

API 功能

在关于 DeepSeek API 的最后一节中，我们将简要介绍它提供的一些最有趣（当前）的功能。我们不想深入探讨所有功能，但希望让你对每个功能有一个良好的初步了解。我们将讨论推理、流式传输、JSON 输出、函数调用和填充中间 (FIM)功能。

推理

deepseek-reasoner模型/端点具有推理特定主题并返回其推理痕迹的能力。这对于更复杂的主题非常有用，但对于其他主题可能有些大材小用。

以下是使用方法：


model = "deepseek-reasoner"

messages = [

    {"role": "user", "content": "What is the population of Copenhagen in 2030?"}

]

response = CLIENT.chat.completions.create(

    model=model, messages=messages, temperature=TEMPERATURE

)

reasoning_content = response.choices[0].message.reasoning_content

content = response.choices[0].message.content

print("Reasoning Content:")

print(reasoning_content)

# Okay, the user is asking about Copenhagen's population in 2030...

print("\nFinal Answer:")

print(content)

# The projected population for Copenhagen Municipality ...

流式传输

没有用户喜欢盯着空白屏幕等待计算机回答问题。流式响应是使你的 LLM 响应看起来快速且对用户响应灵敏的好方法，而无需实际加速它：


model = "deepseek-chat"

messages = [{"role": "user", "content": "What is the second largest city in Portugal?"}]

response = CLIENT.chat.completions.create(

    model=model,

    messages=messages,

    stream=True,  # this is important!

    temperature=TEMPERATURE

)

for chunk in response:

    if chunk.choices[0].delta.content:

        # 处理每个流式块的内容

        pass

print(chunk.choices[0].delta.content, end="", flush=True)

输出结果：葡萄牙第二大城市是 `Porto`。


{

    "model": "deepseek-chat",

    "messages": [

        {

            "role": "system",

            "content": "Extract a JSON response with the keys 'name', 'age', and 'city'.",

        },

        {"role": "user", "content": "Duarte is 31 years old and lives in Copenhagen."},

    ],

    "response_format": {"type": "json_object"},

    "temperature": TEMPERATURE

}


model = "deepseek-chat"

messages = [

    {

        "role": "system",

        "content": "Extract a JSON response with the keys 'name', 'age', and 'city'.",

    },

    {"role": "user", "content": "Duarte is 31 years old and lives in Copenhagen."},

]

response = CLIENT.chat.completions.create(

    model=model,

    messages=messages,

    response_format={"type": "json_object"},

    temperature=TEMPERATURE,

)

json_object = json.loads(response.choices[0].message.content) # 转换为 JSON 对象

print("JSON 响应:")

print(json_object)

# {'name': 'Duarte', 'age': 31, 'city': 'Copenhagen'}

print(json_object["name"])

# Duarte

json.loads() 方法可以帮助你将 LLM 响应结构化为一个 JSON 对象。

函数调用

当我们想让 LLM 具备与外部世界交互（调用 API、抓取网页或与其他系统交互）的能力时，函数调用是一个很好的方式。我们已经在 第 6 章（工具部分）涵盖了这个主题。

FIM

FIM 补全允许我们在模型响应中定义一个前缀和一个后缀。定义前缀和后缀使得模型可以为它们之间的空白生成特定内容。以代码为例，你可以提供一个函数、循环或方法，其中开头和结尾已定义好——实际上就是要求模型只填充中间的内容。这就是 GitHub Copilot (https://github.com/features/copilot) 自动补全的工作原理！让我们通过一个简单的例子来尝试让 DeepSeek 模型给出居住在里斯本的人数。我们可以提供一个输入和一个输出，让模型填充其余部分：


prompt = "The population of Lisbon is exactly "

suffix = " million people."

response = BETA_CLIENT.completions.create(

    model="deepseek-chat",

    prompt=prompt,

    suffix=suffix,

    max_tokens=5,

    temperature=TEMPERATURE,

)

final_text = prompt + response.choices[0].text + suffix

print(final_text)

# The population of Lisbon is exactly 2,000,000 million people.

这当然不仅限于代码。你可以让模型补全 JSON、XML 或任何其他内容，防止它们偏离主题或生成你可能不期望的文本。

请注意，在撰写本文时，FIM 仅在 Beta API 中可用（你需要设置 base_url=https://api.deepseek.com/beta）。

你也可以在第三方 API 中使用 DeepSeek 模型。让我们看看如何使用第三方提供商提供的 DeepSeek 模型。

使用常见的第三方 API

如前所述，DeepSeek 的所有模型都是开源的。因此，DeepSeek 已在许多不同的云提供商中普及：Google Cloud Platform、Azure、AWS、Fireworks、OpenRouter 和 Cloudflare 等。

谈论所有这些提供商会显得冗长，因此在本节中我们将重点介绍三个主要的：

Cloudflare (https://www.cloudflare.com/)：一个在全球范围内可用的大型提供商。熟悉前端技术的用户可能已经了解 Cloudflare。由于其规模，他们也是“中等规模”的提供商。
AWS (https://aws.amazon.com/)：许多用户熟悉 AWS 或在其公司中使用 AWS，因此为简洁起见，我们将重点介绍 AWS。Azure 和 Google Cloud Platform 是其他流行的选择。我们选择 AWS 的原因是其在企业用户中的受欢迎程度。
OpenRouter (https://openrouter.ai/)：OpenRouter 是一个相对较新的云提供商，但它极其专注于服务 LLM。初创公司或新项目可能对能够轻松切换模型的提供商感兴趣。

让我们从 Cloudflare 开始。

Cloudflare

Cloudflare 是一家美国公司，以其 内容分发网络 (CDN) 产品而闻名。目前，Cloudflare 通过其 Workers AI 产品使 LLM 变得易于访问。在开始之前，你需要了解一些关于 Cloudflare 的细节：

支持的模型：deepseek-r1-distill-qwen-32b，这是 2025 年 1 月发布的 DeepSeek-R1 的蒸馏版本 (https://arxiv.org/pdf/2501.12948)，以及 deepseek-math-7b-instruct，这是 2024 年 4 月发布的专门用于数学的指令微调模型 (https://arxiv.org/pdf/2402.03300)。
定价：deepseek-r1-distill-qwen-32b 的输入价格为每百万 token 0.50 美元；输出价格为每百万 token 4.88 美元。deepseek-math-7b 版本暂无定价信息。
区域：Cloudflare Workers AI 在全球边缘节点运行。
身份验证：可以通过 Cloudflare 认证令牌完成。
OpenAI 兼容端点：支持 /v1/chat/completions 和 /v1/embeddings。

以下是如何设置 Cloudflare Workers AI：

访问 cloudflare.com。
登录或创建一个新账户。
在左侧菜单中，单击 AI 并选择 Workers AI（图 A）。

图 A: Cloudflare 支持的 DeepSeek 模型

单击 {} Rest API。
单击 Create a Workers AI API Token。
将你的账户 ID 和令牌安全地保存起来。

让我们看一些 API 调用示例。

以下是使用 curl 发送请求的示例。我们首先导出环境变量，然后按以下格式发出请求：


$ export CLOUDFLARE_ACCOUNT_ID=******

$ export CLOUDFLARE_AUTH_TOKEN=****

$ curl https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/run/@cf/deepseek-ai/deepseek-r1-distill-qwen-32b -X POST -H "Authorization: Bearer $CLOUDFLARE_AUTH_TOKEN" -d '{ "messages": [{ "role": "system", "content": "You are a friendly assistant" }, { "role": "user", "content": "Why is pizza so good" }]}'

如果你想使用 Python，以下是使用 Cloudflare 提供的 OpenAI 兼容端点的 Python 调用示例：

我们首先设置密钥并定义我们的 OpenAI 客户端：


from openai import OpenAI

# 设置密钥和客户端

api_key = os.environ["CLOUDFLARE_AUTH_TOKEN"]

account_id = os.environ["CLOUDFLARE_ACCOUNT_ID"]

assert account_id, "Please set the CLOUDFLARE_ACCOUNT_ID environment variable."

assert api_key, "Please set the CLOUDFLARE_API_KEY environment variable."

model = "@cf/deepseek-ai/deepseek-r1-distill-qwen-32b"

client = OpenAI(

    base_url=f"https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1",

    api_key=api_key,

)

然后我们创建一个消息列表：


# 创建消息列表

messages = [

    {"role": "system", "content": "You are a helpful assistant."},

    {"role": "user", "content": "What is the most likely sky color in Copenhagen?"},

]

最后，我们向 @cf/deepseek-ai/deepseek-r1-distill-qwen-32b 模型发出请求：


# 发起 API 调用

response = client.chat.completions.create(

    model=model,

    messages=messages,

    max_tokens=1000,

    temperature=0.0,

)

AWS

如果你从事技术工作，很可能听说过 Amazon Web Services，即 AWS。目前，它是全球最大的云服务提供商（www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/）。AWS 以其云服务产品的快速扩展和新技术应用而闻名。他们的大部分 AI 产品都以 Bedrock 品牌提供。

以下是在 AWS 上使用 API 调用 DeepSeek 模型的一些快速须知：

支持的模型：
- 作为无服务器部署（无需自己部署即可运行），你可以使用 DeepSeek-R1 模型。此功能未来可能有所发展，请查阅 aws.amazon.com/bedrock/deepseek/ 以获取最新信息。
- 你也可以通过 Bedrock Marketplace 使用大多数 DeepSeek-R1 蒸馏模型和最新的 DeepSeek-R1-0528 模型。这意味着你需要自行管理部署——并为之付费（即使不使用时）。
定价（按需）：关于定价的最新信息，请咨询 AWS 文档 aws.amazon.com/bedrock/pricing/。请注意，无服务器模型通常更便宜——因为你只需为你对模型的请求付费。另一方面，从 Bedrock Marketplace 部署模型意味着你需要自己支付基础设施费用。我们建议从无服务器产品开始。
区域：DeepSeek-R1 通过无服务器产品在美东（弗吉尼亚北部）、美东（俄亥俄）和美西（俄勒冈）区域提供。Bedrock Marketplace 模型可以部署在任何 AWS 区域。
身份验证：你可以使用你的 AWS 账户进行基于 IAM 的身份验证和 Bedrock 服务集成（aws.amazon.com/iam/）。
兼容 OpenAI 的端点：开箱即不支持，但可以与 AWS 的一些开源项目配合使用（github.com/aws-samples/bedrock-access-gateway）。

图 B：AWS Bedrock 模型目录中可用的 DeepSeek 模型列表

让我们开始设置：

如果你还没有 AWS 账户，请创建或设置一个：signin.aws.amazon.com/signup。
在 AWS 控制台中，搜索 Billing and Cost Management 并打开第一个结果以设置计费。
确保你位于正确的区域——us-east-1 是一个很好的默认选择，因为大多数模型首先在该区域可用。
在顶部搜索栏中搜索 Amazon Bedrock 并转到 Amazon Bedrock 服务。
在右侧面板中，转到 Foundation models 下的 Model Catalog。
使用 Provider 筛选器并选择 DeepSeek。
单击你感兴趣的模型（你可能需要先在控制台中启用访问权限；通过单击控制台中的模型，然后单击 Modify Access，接着请求访问权限来完成此操作）。
使用此处指南安装 AWS CLI：docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html。
（推荐）配置 AWS CLI：docs.aws.amazon.com/cli/。

安装 AWS CLI 后，你可以通过运行以下命令向模型发送请求：


aws bedrock-runtime converse \

 --model-id us.deepseek.r1-v1:0 \

        --messages '[{"role": "user", "content": [{"text": "What is the population of Ancona?"}]}]' \

        --region us-east-1

我们也可以使用 Python 和 boto3 库向我们的模型发送一个简单的请求。默认情况下，AWS 会将文本和思考内容分开。因此，我们可以独立访问这两个字段。同时请注意，Bedrock 运行时 (brt) 的 API 格式略有不同：


import boto3

brt = boto3.client("bedrock-runtime")

model_id = "us.deepseek.r1-v1:0"

# create our messages

conversation = [

    {

        "role": "user",

        "content": [{"text": "What is the most likely sky color in Copenhagen?"}],

    }

]

# request

response = brt.converse(

    modelId=model_id,

    messages=conversation,

    inferenceConfig={"maxTokens": 5000, "temperature": 0.0},

)

response_text = response["output"]["message"]["content"][0]["text"]

repsonse_reasoning = response["output"]["message"]["content"][1]["reasoningContent"][

    "reasoningText"

]["text"]

print("=" * 10, "AWS Thinking", "=" * 10)

print(repsonse_reasoning)

# Okay, so I need to figure out the ..

print("=" * 10, "AWS Response", "=" * 10)

print(response_text)

# The most likely sky color in Copenhagen is **pale gray or overcast white**...

Cloudflare

请求的 content 参数中包含了模型的思考过程和最终响应。因此，我们使用 Python 提取思考部分，它位于 `` 标签之间：


# extract thinking content and final response text

raw_response = str(response.choices[0].message.content)

start = raw_response.index("") + len("")

end = raw_response.index("</think>")

thinking_content = raw_response[start:end]

response_text = raw_response[end + len("</think>") :].strip()

print("=" * 10, "Cloudflare Thinking", "=" * 10)

print(thinking_content)

# Okay, so I need to figure out what the most likely ...

print("=" * 10, "Cloudflare Response", "=" * 10)

print(response_text)

# The most likely sky color in Copenhagen is gray.

现在来说说使用 Cloudflare 时的一些特点和建议。

特点和建议

在我们测试的 Cloudflare 平台中，任何模型都不支持函数调用。
调用推理模型时，其格式不会将推理内容分解为 API 的不同 reasoning 组件。你会注意到思考步骤包含在 `` 令牌内的一个 xml 部分中。
Cloudflare 不支持官方 DeepSeek API 支持的任何模型。同样需要注意的是，Cloudflare 支持的任何模型在官方 DeepSeek API 中都不可用。

如果你正在寻找一个提供简单模型访问的直接提供商，Cloudflare 是一个不错的选择。如果你已经在使用 Cloudflare，那么这是一个很好的选择。但是，请注意模型的可用性，尤其是 DeepSeek 模型，是有限的。

现在让我们来看看 AWS 以及如何使用它。

奇闻与提示

AWS 的模型调用界面可能令人困惑。它提供了 invoke 和 converse 两个 API。你可在此阅读更多关于它们差异的信息：dgallitelli95.medium.com/amazon-bedrock-explained-with-memes-converse-api-and-tool-usage-w-anthropic-claude-3-001c341347ca。
文档非常详尽。以下推荐阅读材料可帮助你了解工具调用、提示缓存、计算使用量等更多内容：dgallitelli95.medium.com/amazon-bedrock-explained-with-memes-converse-api-and-tool-usage-w-anthropic-claude-3-001c341347ca。
如果你进行大量调用（用于分类或其他“批处理”用例），请考虑使用 AWS Batch 推理。这是一项很棒的服务，可以通过牺牲响应时间来以折扣价运行模型。
你可能需要调整在 AWS IAM 控制台中的权限，以便在 AWS 内调用模型。
并非所有模型在所有区域都可用——请检查 AWS 控制台以查看各区域的可用情况。
大多数编程式使用都通过 boto3 Python 库完成。对于 Bedrock，请注意 bedrock 和 bedrock-runtime API 之间的区别：
- boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock.html
- boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime.html

AWS 是一个不断发展的平台，它非常快速地添加对模型的支持——并为你提供了自行部署模型的可能性（当然，需要付费）。在控制台中导航并看到所有可能性可能会让人不知所措，但它是一项强大的工具，值得掌握。

接下来，让我们谈谈一个更简单、更新的服务提供商：OpenRouter。

OpenRouter

OpenRouter (openrouter.ai/) 是一个平台，它充当一个通用网关，通过一个简单的统一接口访问来自各种提供商的 LLM。更有趣的是，对于开源模型，OpenRouter 会将多个提供商聚合在同一个 API 下，允许用户查询性能最佳的提供商。例如，对于 DeepSeek-V3，他们支持 Deep Infra、Novita AI、Nebius AI Studio 和 Fireworks 作为提供商。

以下是关于 DeepSeek 的 OpenRouter API 你需要了解的一些事实：

支持的模型：在撰写本文时，支持 DeepSeek-R1、DeepSeek-R1-0528、DeepSeek-V2、DeepSeek-V3 以及额外的蒸馏或微调模型。

图 C：同一个 DeepSeek-V3 模型的多个提供商

定价（按需付费）：OpenRouter 会将你的请求定向到你所请求模型的“最佳可用提供商”。他们在你购买平台积分时向你收费（5% + 0.35 美元手续费）。他们还提供多种免费模型（例如 DeepSeek-R1-0528）。
区域：它本身不支持“区域”。它将你的请求直接路由到提供商。因此，你应该将其视为一个“全球”区域。
认证：它使用一个需要你通过 OpenRouter 控制台生成的 API 密钥。
兼容 OpenAI 的端点：OpenRouter 的所有端点都兼容 OpenAI 格式和 SDK。

以下是设置步骤：

访问 openrouter.ai。
登录或创建账户。
将鼠标悬停在右上角的汉堡菜单上，然后点击 Credits（积分）。
为你的账户充值积分。
再次将鼠标悬停在右上角的汉堡菜单上，然后点击 API Keys（API 密钥）。
生成一个 API 密钥并保存。

以下是如何使用 curl 进行 API 调用：


curl https://openrouter.ai/api/v1/chat/completions \

  -H "Content-Type: application/json" \

  -H "Authorization: Bearer $OPENROUTER_API_KEY" \

  -d '{

  "model": "deepseek/deepseek-r1-0528",

  "messages": [

    {

      "role": "user",

      "content": "What is the most likely sky color in Copenhagen?"

    }

  ]

}'

如你所见，我们定义了消息列表并将其发送到必要的端点。

使用 Python 时，该请求与对 OpenAI SDK 客户端的其他请求非常相似：


from openai import OpenAI

client = OpenAI(

    base_url="https://openrouter.ai/api/v1",

    api_key=os.environ["OPENROUTER_API_KEY"],

)

response = client.chat.completions.create(

    model="deepseek/deepseek-r1-0528",

    messages=[

        {"role": "user", "content": "What is the most likely sky color in Copenhagen?"}

    ],

)

response_reasoning = response.choices[0].message.reasoning

response_text = response.choices[0].message.content

print("=" * 10, "OpenRouter Thinking", "=" * 10)

print(response_reasoning)

# Okay, the user asked about the most likely sky col...

print("=" * 10, "OpenRouter Response", "=" * 10)

print(response_text)

你会注意到，我们需要做的只是覆盖 OpenAI 客户端的 base_url 参数。

你应该会得到一个类似于这样的输出：


# The **most statistically likely overall sky color in Copenha...

以下是使用 OpenRouter 时的一些提示和细节。

使用 OpenRouter

特性与技巧

免费模型: OpenRouter 提供大量免费模型供您测试。只需访问其模型仪表板（或 openrouter.ai/models?max_price=0）即可开始调用 API。（不建议在生产环境中使用。）
您可以在提供商名称后追加 :nitro 以访问吞吐量最佳的提供商，或追加 :floor 以访问价格最优的提供商。例如，要访问最快的 DeepSeek-R1，可以查询 deepseek/deepseek-r1-0528:nitro。
OpenRouter 支持您可能期望的所有主要功能（工具调用、图像与 PDF 处理、结构化输出，以及在某些提供商上的提示缓存）。具体实现细节请参阅其文档：openrouter.ai/docs/quickstart。
请注意，由于 OpenRouter 不控制实际模型运行的硬件，您可能会在不同提供商之间收到不同的结果。
OpenRouter 在 openrouter.ai/rankings 提供了排行榜，您可以看到哪些模型被用于哪些类别，并了解当前的热门趋势。例如，在撰写本文时，DeepSeek-R1-0528 在金融类别中排名第 2。

使用 Cursor IDE 进行 DeepSeek 开发

我们将使用 Cursor IDE 演示 DeepSeek 的功能，这是一个强大的 AI 增强型开发环境，可与 DeepSeek 模型无缝集成。然而，我们探索的原理和技术适用于任何支持 DeepSeek 集成的开发环境，无论您偏好传统 IDE、云平台还是本地 Web 界面。关键在于建立开发工作流与 DeepSeek-R1 先进推理能力之间的牢固连接。

确认 API 访问后，我们可以开始建立开发环境，将其作为探索 DeepSeek 实际应用的实验室。

设置开发环境

创建一个用于 AI 辅助开发的环境，需要仔细选择能够补充 DeepSeek 能力并支持完整开发生命周期的工具。我们推荐的工具包以 Cursor IDE 为核心，您可以从 cursor.com 下载。这个 AI 原生编辑器提供了与各种语言模型的无缝集成，包括 DeepSeek-R1，使其成为我们演示的理想选择。

我们开发技术栈的基础包括：

Python 3.11 或更高版本，由于其丰富的 AI 和数据处理库，它将作为我们后端实现的主要语言。
Docker Desktop，使我们能够对应用程序进行容器化，确保一致的部署环境和简化的分发。
Git，提供必要的版本控制功能，这在与 AI 生成的代码协作时尤为重要，因为跟踪更改和维护代码历史至关重要。
Node.js 18 或更高版本，支持我们在更全面的示例中将用到的各种前端组件和开发工具。

在 Cursor IDE 中配置 DeepSeek

Cursor IDE 与 DeepSeek-R1 的集成是我们开发方法的核心，它将传统的编码工作流转变为人类创造力与 AI 推理能力之间的协作伙伴关系。这个配置过程虽然简单，但为后续所有演示奠定了基础。

直接集成：将 Cursor 连接到 DeepSeek

首先，通过设置菜单访问 Cursor 的模型配置。在 macOS 上，您可以在 Cursor > 设置中找到它；在 Windows 和 Linux 上，可以在文件 > 设置中找到。在设置界面中，左侧边栏的模型部分包含管理 AI 模型集成的所有选项。

Cursor IDE 内置了对 DeepSeek 模型的支持，您可以通过滚动浏览可用模型列表并选择 deepseek-r1 来启用它。启用后，该模型将出现在您的可用选项中，准备好使用您的特定 API 凭据进行配置。

图 D：Cursor IDE 的 DeepSeek-R1 模型选择界面

配置过程需要输入您的 DeepSeek API 凭据以及相应的连接设置。您需要将 deepseek-r1 指定为模型名称，提供来自 DeepSeek 平台的 API 密钥，将基础 URL 设置为 https://api.deepseek.com/v1，并将提供商配置为 OpenAI 兼容 以确保正确的 API 通信协议。

要验证您的集成是否正常工作，请在 Cursor 中打开一个新文件，并使用 Cmd + K（macOS）或 Ctrl + K（Windows 和 Linux）调用 AI 命令面板。从模型下拉菜单中选择 DeepSeek-R1，并使用一个实质性的提示进行测试，例如“解释微服务和单体架构之间的区别”。成功的响应确认您的开发环境已准备好进行我们将要探索的高级用例。

替代方案：命令行集成

虽然 Cursor IDE 为 AI 辅助开发提供了出色的交互环境，但许多开发者更喜欢使用 CLI 进行自动化、脚本编写以及与现有工作流的集成。DeepSeek 的 API 遵循 OpenAI 兼容协议，使其易于集成到命令行工具和自动化脚本中。

建立 Python 客户端访问

设置编程访问需要安装 openai Python 库以及用于安全管理凭据的 python-dotenv：


pip install openai python-dotenv

安装依赖项后，您可以使用一个简单的 Python 脚本建立与 DeepSeek API 的连接。这种方法对于批量处理、自动化测试以及与更大系统的集成特别有价值：


import os

from openai import OpenAI

client = OpenAI(

    api_key=os.getenv("DEEPSEEK_API_KEY"),

    base_url="https://api.deepseek.com/v1"

)

response = client.chat.completions.create(

    model="deepseek-r1",

    messages=[{"role": "user", "content": "Hello DeepSeek!"}]

)

print(response.choices[0].message.content)

自行运行或部署 DeepSeek

DeepSeek 模型的一个有趣方面在于它们实际上是开放的，任何人都可以交互。在本节中，我们将介绍一些使用您自己的基础设施来交互和部署这些模型的方法。知道您的数据和应用程序完全由您控制会带来多种好处；隐私和合规性只是其中的两个。最后，值得注意的是，自己部署和管理这些模型（尤其是较大的变体）可能是一项相当艰巨的任务。

在这里，我们将介绍一些本地使用这些模型的方法，并向您介绍如何进一步操作并自己部署它们。我们将使用 llama.cpp 和 Ollama 来帮助您在本地使用 DeepSeek。

使用 llama.cpp

llama.cpp 是一个高性能的开源 C/C++ 推理库，旨在本地硬件上高效运行 LLM。它由 Georgi Gerganov 发起，拥有一个活跃的社区和代码库（github.com/ggml-org/llama.cpp）。它允许用户在本地执行模型，无需专用 GPU 或云基础设施。

要安装 llama.cpp，请按照 GitHub 代码库（github.com/ggml-org/llama.cpp/blob/master/docs/install.md）上的安装说明操作。如果您在 Mac 上，安装 llama.cpp 就像下面这样简单：


$ brew install llama.cpp

要验证您的安装是否正确，只需运行以下命令：


$ llama-cli --help

您应该会看到一些输出，解释如何使用 CLI 工具：


----- common params -----

-h, --help, --usage print usage and exit

--version show version and build info

--completion-bash print source-able bash completion script for llama.cpp

--verbose-prompt print a verbose prompt before generation (default: false)

...

llama.cpp 使用 GGUF 文件格式（您可以在 github.com/ggml-org/ggml/blob/master/docs/gguf.md 阅读更多相关信息）。GGUF 代表 GPT-Generated Unified Format（www.ibm.com/think/topics/gguf-versus-ggml）。为了使用该工具运行模型，您需要下载 GGUF 格式的模型，或者将模型从 PyTorch 格式转换为 GGUF 格式。幸运的是，Unsloth 的团队（unsloth.ai/）在 Hugging Face 上免费提供了 DeepSeek 模型的 GGUF 变体（huggingface.co/collections/unsloth/deepseek-r1-all-versions）。

要与原版 DeepSeek-R1 模型进行本地聊天会比较有挑战性。您可能需要多台配备多 GPU 的机器才能运行它。让我们选择一个更简单的选项，运行 DeepSeek-R1 的一个蒸馏版本。DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q4_K_M 是从 DeepSeek-R1 模型蒸馏而来的 Qwen 15 亿参数模型的一个版本。在本例中，我们运行的是量化版本 Q4_K_M，这意味着它是一个小模型，很可能可以在我们的 CPU 上运行：


$ llama-cli -hf unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q4_K_M

这将自动从 Hugging Face 仓库下载模型，并直接将您带入终端中的聊天界面。

图 E：通过 Unsloth 界面在本地使用 DeepSeek 模型

Hugging Face 还为您提供了一种便捷的方式来开始使用您浏览时看到的任何模型：点击 Use this model 按钮并按照说明操作即可。

如果您考虑集成到更广泛的应用中，您可能有兴趣使用 Python 或任何其他编程语言运行此模型。幸运的是，我们可以使用 llama-cpp-python（github.com/abetlen/llama-cpp-python），这是一个 Python 库，它提供了与 llama.cpp 相同 CLI 的 Python 绑定。以下是使用 Python 与该模型交互的示例。您可以看到，我们提供与通过 CLI 使用 llama.cpp 时相同的参数，但现在直接将其传入库中：


from llama_cpp import Llama

REPO_ID = "unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF"

MODEL_FILENAME = "DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf"

llm = Llama.from_pretrained(

    repo_id=REPO_ID,

    filename=MODEL_FILENAME,

    verbose=False,

)

response = llm.create_chat_completion(

    messages=[

        {

            "role": "user",

            "content": "What is the most likely sky color in Copenhagen? Think hard and answer in one word.",

        },

    ],

)

text_response = response["choices"][0]["message"]["content"]

print(f"{MODEL_FILENAME} response:\n{text_response}")

如果您有兴趣部署 llama.cpp 并将其用作兼容 OpenAI 的服务器，这是可以的！只需运行以下命令：


$ llama-server -hf unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q4_K_M

现在，您可以通过访问 localhost:8080 来访问聊天界面，它允许您使用一个完全本地的模型。

图 F：通过 llama.cpp 使用 DeepSeek

或者，您可以查询兼容 OpenAI 的 API 端点：localhost:8080/v1/chat/completions。使用前述命令启动 llama-server 后，打开另一个终端并发送一个 curl 请求，如下所示：


$ curl --request POST \

          --url http://localhost:8080/completion \

          --header "Content-Type: application/json" \

          --data '{"prompt": "What is the capital of Portugal?","n_predict": 120}'

这将输出直接来自您运行服务器所使用的模型的响应。

现在，让我们谈谈另一个有趣的建立在 llama.cpp 之上的工具，名为 Ollama。

Ollama

Ollama（ollama.com/）是一个库和 CLI 工具，首次发布于 2023 年。它的目标是允许您在本地机器上以最少的设置运行 LLMs。其后端由 llama.cpp 驱动，但它提供了更用户友好的体验。Ollama 拥有广泛的模型库，您可以通过访问 ollama.com/search 下载。这包括许多开源模型（当然也包括 DeepSeek 模型）。

要开始使用 Ollama，请前往他们的下载页面 ollama.com/download，并下载适用于您平台的二进制文件。完成后，您可以通过在终端中运行以下命令来验证安装是否正确：


$ ollama help

您应该会看到一条帮助信息，其中包含可用于运行 Ollama 的不同选项。

要了解可以运行哪些模型，请首先访问可用模型库 ollama.com/library。

图 G：Ollama.com 上的 DeepSeek-R1-1.5B

一旦您选择了希望在本地运行的模型（在我们的例子中是 deepseek-r1:1.5b），您只需像拉取 Docker 镜像一样将其拉取下来：


$ ollama run deepseek-r1:1.5b

现在，您可以在本地机器上开始与该模型聊天。一个可以使用的有趣技巧是，通过输入 /set nothink 或 /set think 来分别开启或关闭思考过程（这仅适用于推理模型）。以下是一个终端会话示例和动态设置这些模式：


➤ ollama run deepseek-r1:1.5b

>>> /set nothink

Set 'nothink' mode.

>>> What is the capital of the Le Marche Region in Italy?

The capital of the Le Marche Region in Italy is Rome

>>> /set think

Set 'think' mode.

>>> What is the capital of the Le Marche Region in Italy?

Thinking...

Alright, let's see. The user initially asked about the capital of the Le Marche region and I...

此外，Ollama 还提供了一个名为 ollama-python 的 Python SDK，可在 github.com/ollama/ollama-python 获取，您可以如下使用：


from ollama import chat

messages = [

    {

        "role": "user",

        "content": "What is the capital of Le Marche region in Italy?",

    },

]

response = chat(

    "deepseek-r1:1.5b", messages=messages, think=True, options={"temperature": 0.0}

)

print(f"Thinking:\n========\n\n{response.message.thinking}")

# Okay, so I need to figure out the capital of the Le Marche region in Italy...

print(f"\nResponse:\n========\n\n{response.message.content}")

# The capital of the Le Marche region in Italy is

最后，Ollama 还提供了一个兼容 OpenAI 的 API，您可以使用它。以下是与之前相同的请求，使用 API 和 curl：


$ curl --location 'http://localhost:11434/api/chat' \

      --header 'Content-Type: application/json' \

      --data '{

    "model": "deepseek-r1:1.5b",

    "options": { "temperature": 0.0 },

    "messages": [

      {

        "role": "user",

        "content": "What is the capital of Le Marche region in Italy?"

      }

    ],

    "think": true

  }'

您将在终端中看到模型的响应实时返回，正如流式传输所预期的那样：

Ollama 是一个用于本地使用甚至部署模型的优秀工具。它很好地补充了 llama.cpp 的功能。您可以自行选择更偏好哪一个。虽然 llama.cpp 提供了一些更强大的用户特性（例如设置采样率、聊天模板和分词器），但 Ollama 试图做到尽可能简洁，并与周围的世界集成。您可以尝试两者，然后决定最喜欢哪一个！

自行部署 DeepSeek

根据模型的规模，自行部署大语言模型并非易事。如果您部署的是一个可在 CPU 上运行的 1 GB 模型，那不成问题。但如果您要为多个用户提供 4050 亿参数的模型服务，并且有延迟和吞吐量要求，那么事情就会变得更加复杂。

让我们举个例子。假设您想要部署完整的 DeepSeek-R1 模型。在 huggingface.co/deepseek-ai/DeepSeek-R1 上的 DeepSeek-R1 模型总共有 6710 亿参数。以全精度运行，这意味着您的模型大小将约为 1.5 TB。请记住，这是 RAM 大小，而非磁盘大小。您将需要大约 16 个每个具有 80 GB RAM 的 A100 GPU。另一种选择是部署一个更小的模型（例如 7B 参数版本）。如果您决定部署一个更小的模型，AWS SageMaker 就是一个很好的起点。Hugging Face 有一篇很棒的教程，介绍如何将其仓库中的（几乎）任何模型部署到 AWS 上，链接为 huggingface.co/docs/sagemaker/en/inference。

如果您认为使用 API 或部署较小的模型对您来说不是可行的选择，那么我请您认真重新考虑一下。部署、管理并为自己的 GPU 集群付费，其成本实际上可能比使用 API 更高。

现在，让我们来讨论如何创建您自己的设置。

为 DeepSeek 构建您自己的设置

在本附录的最后一部分，我们将重点介绍一些在构建使用 DeepSeek 模型的应用时，可以添加到您武器库中的有趣库。我们发现这些库在基于模型构建时特别有用。这些库为您提供了更多的灵活性，与模型和提供商无关，并帮助您实现结构化输出。

请记住，这些库是不断发展的，因此请务必查看每个库的文档以获取最新信息。希望这些库的核心不会很快改变——这也是我们在此介绍它们的原因。

LiteLLM

LiteLLM (www.litellm.ai/) 是一个库，同时也是一个代理服务器和 SDK，专注于在构建应用时方便地使用大语言模型。在这里，我们将重点介绍 SDK，因为这是我们使用它进行模型构建的主要方式。它为常见问题提供了一些好的解决方案：

通用接口：允许您在构建使用模型的应用时使用相同的 API。您无需弄清楚如何使用 AWS、Azure、GCP、Cloudflare 或任何其他提供商的模型，借助 LiteLLM，您可以使用相同的补全函数。
重试与后备：大语言模型 API 可能“不稳定”。它们可能宕机、响应缓慢，甚至拒绝完成您的请求。LiteLLM 使您能够多次尝试给定的请求。此外，您可以设置多个模型作为“后备”，在主模型失败时使用它们。
缓存：缓存是提供和使用大语言模型的重要组成部分。通过缓存，您可以避免在不需要时重复调用大语言模型。这既能降低您的成本，也能使您的请求更快。

让我们看一个实现缓存的脚本：

我们将首先在磁盘上缓存响应，并确认存在 DeepSeek API 密钥：


from litellm import completion

from litellm.caching.caching import Cache

import litellm

import os

litellm.cache = Cache(type="disk")

env_var_name = "DEEPSEEK_API_KEY"

assert os.environ[env_var_name], f"Please set your {env_var_name} environment variable."

然后我们向 deepseek-reasoner 模型发送一个聊天补全请求：


response = completion(

    model="deepseek/deepseek-reasoner",

    messages=[

        {

            "role": "user",

            "content": "What is the capital of region of Le Marche in Italy?",

        }

    ],

它告诉 LiteLLM 最多重试调用两次，如果该模型失败，则回退到 deepseek-chat：


    num_retries=2,  # Number of retries

    fallbacks=["deepseek/deepseek-chat"],  # Fallback models

    caching=True,  # Enable caching

)

由于启用了缓存，后续相同的请求将从磁盘提供服务，而不是再次调用模型：


if hasattr(response.choices[0].message, "reasoning_content"):

    reasoning = response.choices[0].message.reasoning_content

    print(f"Reasoning:\n{reasoning}\n")

# Okay, the user is asking about the capital o

final_msg = response.choices[0].message.content

print(f"Final Answer:\n{final_msg}\n")

# The capital of the region of **Le Marche (...

现在是时候看看 LangChain 了。

LangChain

LangChain (python.langchain.com/docs/introduction/) 是最流行的开源框架之一，涵盖了使用大语言模型构建的大多数方面：与数量不断增加的提供商的数千种集成，用于构建 RAG 系统、智能体等的组件。它非常适合原型设计和快速推出产品。但是，当您的应用程序开始变大时，您可能会考虑自己实现某些组件以获得额外的控制。这些组件可能包括：

统一 API：对所有模型和提供商使用相同的代码和接口，包括 Ollama 和 llama.cpp 等本地模型。
RAG 助手：文档加载器帮助您快速加载外部数据并将其作为上下文传递给大语言模型。文档加载器、文本分割器以及检索和生成工作流使您的 RAG 管道非常容易实现。
结构化输出：集成使您能够从大语言模型中获取结构化数据。支持 JSON 模式、流式传输、少样本提示以及更多功能。

让我们看以下脚本，了解如何使用 LangChain：

首先创建一个来自 langchain_deepseek 的 ChatDeepSeek 大语言模型包装器：


from langchain_deepseek import ChatDeepSeek

from pydantic import BaseModel, Field

from typing import Optional

from enum import Enum

llm = ChatDeepSeek(

    model="deepseek-chat",

    temperature=0,

    max_tokens=None,

    timeout=None,

    max_retries=2,

)

然后我们创建一个 Pydantic 模型（SkyColor），定义期望的结构化输出：


class SkyColorEnum(str, Enum):

    BLUE = "blue"

    GRAY = "gray"

    WHITE = "white"

    CLOUDY = "cloudy"

    OVERCAST = "overcast"

    CLEAR = "clear"

class SkyColor(BaseModel):

    """Sky color prediction for Copenhagen."""

    color: SkyColorEnum = Field(description="The primary color of the sky")

    description: str = Field(description="Detailed description of the sky appearance")

    confidence: Optional[int] = Field(

        default=None, description="Confidence level of the prediction, from 1 to 10"

    )

API 响应示例

以下是两个从 deepseek-r1:1.5b 模型流式获取的思考过程响应片段：


{"model":"deepseek-r1:1.5b","created_at":"2025-06-13T14:29:01.074249Z","message":{"role":"assistant","content":"","thinking":"Okay"},"done":false}


{"model":"deepseek-r1:1.5b","created_at":"2025-06-13T14:29:01.084569Z","message":{"role":"assistant","content":"","thinking":","},"done":false}

指导器

指导器 (python.useinstructor.com/) 是最受欢迎的 Python 库，用于从大语言模型（LLMs）中提取结构化数据。它基于 Pydantic 构建，提供类型安全的数据提取，并支持自动验证、重试和流式处理。它支持大多数主流平台和模型。指导器 最出色的部分实际上是他们的文档，提供了出色的指南、实用案例和其他精彩内容！

结构化输出：将 Pydantic 模型作为响应模型传递，并返回一个 Pydantic 模型。你也可以使用开源或自托管模型，指导器 会支持它。
自动重试和自我修正：验证失败会触发重试，并将错误消息注入到模型上下文中。这效果很好，使模型能够修正自身答案。
流式验证块：指导器 还支持对部分验证对象进行流式处理。这意味着你不必等待完整的模型响应就可以向用户显示部分内容。在此处阅读更多信息：python.useinstructor.com/blog/2023/11/26/python-generators-and-llm-streaming/#stream-processing。

我们现在将在以下脚本中查看 指导器 库的使用。生成的对象是类型化、已验证且可直接使用的：

指导器 封装了与 OpenAI 兼容的 DeepSeek 端点：


import os

from openai import OpenAI

import instructor

from pydantic import BaseModel, Field

client = instructor.from_openai(

    OpenAI(api_key=os.getenv("DEEPSEEK_API_KEY"), base_url="https://api.deepseek.com"),

    mode=instructor.Mode.MD_JSON,  # because this is a reasoning model..

)

然后我们添加两个 Pydantic 类（City 和 CitiesResponse），它们描述了所需响应的结构：


class City(BaseModel):

    name: str = Field(description="The name of the city")

    population: int = Field(description="The population of the city")

    notable_landmarks: list[str] = Field(

        description="List of notable landmarks in the city"

    )

class CitiesResponse(BaseModel):

    cities: list[City] = Field(description="List of interesting cities")

然后我们调用 deepseek-reasoner，要求列出三个有趣的葡萄牙城市：


cities: CitiesResponse = client.chat.completions.create(

    model="deepseek-reasoner",

    messages=[

        {

            "role": "user",

            "content": "Top 3 most interesting cities in Portugal",

        },

    ],

    response_model=CitiesResponse,

    temperature=0.0,

)

指导器 将确保回复完全符合模式（基础模型），如果验证失败则重试，并将验证错误提供给大语言模型：


for city in cities.cities:

    print(f"City: {city.name}")

    print(f"Population: {city.population}")

    print(f"Notable Landmarks: {', '.join(city.notable_landmarks)}")

你将看到以下输出：


# City: Lisbon

# Population: 545796

# Notable Landmarks: Belém Tower, Jerónimos Monastery, São Jorge Castle, Alfama District

# City: Porto

# Population: 237584

# ...

既然我们已经介绍了一些基础库，我们还想提及其他一些可能在您探索旅程中有用的库和资源。

其他有趣的库和资源

llm (https://github.com/simonw/llm) 是一个命令行工具加 Python 库，可以与 OpenAI、Anthropic、Gemini、Llama 等模型交互，同时记录每一次提示/响应并将其嵌入 SQLite 以便后续分析。其插件系统和类 Unix 的管道机制让您能够自动化完成诸如总结网页、生成 shell 命令或在终端中直接比较模型等任务。
Pydantic AI (https://ai.pydantic.dev) 是 Pydantic 团队推出的一个类型安全的智能体框架，它按照 Pydantic 模式验证 LLM 输出，同时支持 OpenAI、Anthropic、Gemini 和 Groq 等提供商。它具有依赖注入、流式传输和基于图的控制流特性，并内置监控功能，有助于以最少的样板代码将生成式 AI 项目推向生产环境。
Outlines (https://dottxt-ai.github.io/outlines/latest) 能够引导模型输出匹配正则表达式、JSON 模式、语法或枚举的结果，使得下游代码可以轻松解析。它兼容闭源或开源后端，且几乎不增加延迟，因此许多团队依赖它在生产环境中实现可靠的结构化生成。

现在我们已经介绍了许多不同的库，您在构建 DeepSeek 应用时可以将其添加到您的工具集中。

通过使用 DeepSeek 模型，您有很多选择，可以有效地根据需求进行构建，无论是使用官方 API 还是第三方提供商，如 AWS 或 OpenRouter。如果控制权很重要，您也可以使用 llama.cpp 或 Ollama 在本地运行模型。我们还提供了一份很好的指南，说明如果需要部署在 GPU 集群上（如果您有能力的话），该怎么做。

我们还介绍了诸如 LiteLLM、LangChain 和 Instructor 等库，它们将帮助您尽可能充分地利用 DeepSeek 模型，同时保持您的逻辑和代码清晰可控。

获取本书的 PDF 版本和独家附加内容

扫描二维码（或访问 packtpub.com/unlock）。搜索本书书名，确认版本，然后按照页面上的步骤操作。


注意：请保留好您的发票。直接从 `Packt` 购买的图书不需要发票。

解锁您的独家福利

您购买的本书包含以下独家福利：

新一代 Packt 阅读器
无 DRM 的 PDF/ePub 下载

请按照以下指南解锁福利。该过程仅需几分钟，并且只需完成一次。

3 个简单步骤解锁本书的免费福利

第 1 步

准备好您的购买发票，用于 第 3 步。如果您有实体书，请用手机扫描并将其保存为 PDF、JPG 或 PNG 格式。

如需帮助查找您的发票，请访问 https://www.packtpub.com/unlock-benefits/help 。

注意：如果您直接从 Packt 购买了本书，则不需要发票。完成 第 2 步 后，您就可以立即访问您的独家内容。

第 2 步

扫描二维码或访问 packtpub.com/unlock。


白色背景上的二维码 AI 生成的内容可能不准确。

在打开的页面上（类似于桌面端的 图 11.1），搜索本书书名并选择正确的版本。

图 11.1：桌面端的 Packt 解锁着陆页

第 3 步

选择您的书后，登录您的 Packt 账户或免费创建一个。然后上传您的发票（PDF、PNG 或 JPG 格式，大小不超过 10 MB）。按照屏幕上的指示完成该过程。

需要帮助？

如果您遇到问题需要帮助，请访问 https://www.packtpub.com/unlock-benefits/help ，获取有关如何查找发票等问题的详细解答。此二维码将引导您至帮助页面。

注意：如果您仍然遇到问题，请联系 customercare@packt.com。

packtpub.com

订阅我们的在线数字图书馆，即可完全访问超过 7,000 本书籍和视频，以及行业领先的工具，帮助您规划个人发展并推进您的职业生涯。更多信息，请访问我们的网站。

为什么订阅？

通过超过 4,000 位行业专业人士提供的实用 eBook 和视频，减少学习时间，增加编码实践时间
通过专门为您构建的技能计划来提升学习效果
每月免费获得一本电子书或一个视频
完全可搜索，便于获取重要信息
支持复制和粘贴、打印以及收藏内容

在 www.packtpub.com，您还可以阅读免费技术文章合集，注册各种免费新闻通讯，并享受 Packt 图书和电子书的独家折扣和优惠。

您可能感兴趣的其他书籍

如果您喜欢本书，您可能对 Packt 的这些其他书籍感兴趣：

构建智能体 AI 系统

Anjanava Biswas, Wrick Talukdar

ISBN: 9781803238753

掌握生成式 AI 和智能体系统的核心原理
理解 AI 智能体在动态环境中如何运作、推理和适应
使 AI 智能体能够分析自身行为并即兴调整
实现 AI 智能体可以利用外部工具并规划复杂任务的系统
应用方法来增强 AI 的透明度、可解释性和可靠性
探索 AI 智能体在各行业中的真实世界应用

企业中的大语言模型

Ahmed Menshawy, Mahmoud Fahmy

ISBN: 9781836203070

应用设计模式将 LLM 集成到企业应用程序中，以提高效率和可扩展性
克服 LLM 扩展和部署中的常见挑战
使用微调技术和 RAG 方法来提升 LLM 效率
通过了解新兴趋势和进展（包括多模态）来保持领先优势
通过定制化上下文模型、高级推理引擎和评估模式来优化 LLM 性能
确保 AI 应用的公平性、透明度和可解释性

Packt 正在寻找像您这样的作者

如果您有兴趣成为 Packt 的作者，请访问 authors.packtpub.com 并立即申请。我们已经与数千名像您一样的开发者和科技专业人士合作，帮助他们与全球科技界分享他们的见解。您可以进行通用申请，申请我们正在招募作者的特定热门话题，或者提交您自己的想法。

分享您的想法

您已经读完了《实践中的 DeepSeek》，我们很想听听您的想法！扫描下方二维码，直接前往本书的亚马逊评论页面，分享您的反馈或在您购买该书的网站上留下评论。

packt.link/r/180602084X

您的评论对我们和科技社区都很重要，将帮助我们确保提供高质量的内容。

索引

准确度奖励 66

激活感知权重量化 (AWQ) 357

激活值 49

自适应注意力路由 15

亲和度得分 54

Agent2Agent (A2A) 协议 257

智能体应用程序 249

智能体系统 259, 260

arXiv 论文，总结示例 261-269

评估器-优化器工作流 260

协调器-工作者工作流 270

智能体 249, 250, 258

参考链接 260

与...合作 257-259

AI 辅助开发环境

配置管理 147

设置 144

工作区组织 146

AIME 70

对齐技术 14

Amazon Bedrock

在...上进行托管部署 363-366

参考链接 350

Amazon 资源名称 (ARN) 238

Amazon SageMaker 245

Amazon Web Services (AWS) 233, 406-410

使用...部署隔离模型服务 233-235

架构修改 15, 16

通用人工智能 (AGI) 82

人工智能 (AI) 3

arXiv.org

URL 261

增强型 LLM 258, 279

自动缓存 315

自回归生成 47

自回归词元生成 51

自回归 Transformer 47

AWS DLC 仓库

参考链接 237

Azure AI Foundry

参考链接 351

Bedrock API 233

Bedrock 自定义模型导入 233

基准测试系统

最佳实践 194, 195

评估对比 183-186

指标计算 189

为用例选择工具 193

基准测试工具

设置 148, 149

二元奖励 59

bioRxiv

URL 261

因果掩码 51

草稿链 (CoD) 107

思维链 (CoT) 12, 60, 90, 251

聊天模板 121

Claude Code

参考链接 249

Cloudflare 403-406

冷启动数据 12

冷启动监督训练数据 70-72

复合错误 257

一致性奖励 66, 72

约束生成 229

容器 220

内容分发网络 (CDN) 403

DeepSeek 的上下文感知响应

高级 RLHF 60

思维链推理 61

数据集选择 61

FP8 训练 57, 58

分组相对策略优化 (GRPO) 59

多词元预测 58

精度控制 57, 58

测试时缩放 62

上下文敏感路由 53

合同审查 301

合同理解 Atticus 数据集 (CUAD) 296, 302

蒸馏微调过程概述 303, 304

扩展 303

在...上微调 Gemma 3 318-323

控制流 258

卷积神经网络 (CNN) 79

CoT 追踪 77

CrewAI

参考链接 249

交叉熵损失 77

Cursor IDE

配置 DeepSeek 145, 146

用于 DeepSeek 的工作 414

每日健康摘要 201

解码器 40

解码器模块

内部机制 44

工作示例 50-52

仅解码器自回归 Transformer 42

Deep Java Library (DJL) 233

深度研究

参考链接 249

DeepSeek 3-7, 14, 39

高级能力 78

智能体推理 81

架构修改 15-18

在 Cursor IDE 中配置 145, 146

上下文长度和压缩上限 86

相关争议 33-36

数据集和文化范围差距 87

训练数据集与理念 23, 24

部署 417, 423

部署格局 336

部署特点 337

对全球 AI 生态系统的影响 30-32

推理延迟 86

可解释性与对齐风险 86

关键架构组件 40

使用 llama.cpp 417-420

思维模型与原则 91-98

MoE 架构 19

Ollama 421-423

局限性展望 87

提示技巧与建议 99-102

提示路由 40-42

实时交互权衡 86

强化学习 14

对提示的响应影响 103-109

扩展挑战 85

选择自部署 337-341

重要性 337-341

稀疏专业知识限制 85

技术突破 7

工具集成 81

训练过程 8-14

用于实现 MoE 架构 19-22

版本与演进 24-30

与领先的 LLM 对比 82-85

视觉能力 79

在 Cursor IDE 中与...协作 414

DeepSeek API

使用可用模型 397

功能 399-401

FIM 402

定价与速率限制 398, 399

设置 392-397

温度参数 397

工作 392

用于金融文档提取的 DeepSeek

提示 174, 175

响应评估 176-183

在 Cursor IDE 中的 DeepSeek

命令行集成 416

配置 414

直接集成 415, 416

DeepSeek 模型

最佳实践 246

使用 Transformers 和 XGrammar 进行基于 CPU 的推理 227-230

交互 222

LiteLLM 222-224

建议 246

重构以用于 CPU 本地生成 231, 232

使用 Ollama 在本地运行 224-227

DeepSeek 提示

故障排除 129-133

迁移指南 133-138

DeepSeek-R1 42-44

自回归 Transformer 47

复杂技术文档 150

提取错误 187

层归一化 49, 50

MoE 路由 52, 53

多头注意力 48, 49

提示设计 148, 149

参考链接 42

使用路由器的词元处理和生成 44-46

工具设置基准测试 148, 149

用于基准测试设置 171-174

用于比较文档解析工具 188-194

用于创建测试文档 171-174

DeepSeek-R1-Distill-LLaMA 12

DeepSeek-R1-Distill-Qwen 12

DeepSeek-R1-Zero 11

DeepSeek-V3 11

将 DeepSeek V3 部署到云端

使用 Hugging Face 推理端点部署 366-374

DeepSeek-VL-7B 79-81

部署 335

示例 359-374

DeepSeek 的部署策略

成本合理性检查 344-346

用于选择的决策框架 341-344

路径 346-354

开发环境

设置 414

直接偏好优化 (DPO) 17

蒸馏 77, 78, 293

变体 5

蒸馏与微调

法律文本中的多标签提取问题 301, 302

重要性 297-300

用例与数据集 301

DIY 部署

架构选择 353

在 IaaS 上 351-354

Docker

用于部署原型 220

Docker + vLLM 349

Docling

测试 188, 189

文档分析挑战，DeepSeek R1

后续代码生成请求 160

后续响应评估 166-169

提示 150

通过迭代提示重新校准 169, 170

响应评估 151-159

领域对象模型 (DOM) 89

动态路由 54

动态工作流 287

构建端到端原型 202

重构为 API 216-220

创建上下文 207-210

创建每日健康摘要 212-215

获取数据 202-206

使用 Docker 部署 220-222

定义结构化输出 211, 212

增强型 CUAD 数据集 296

蒸馏微调过程概述 303, 304

评估智能体 288

评估器 LLM 260

FastMCP

参考链接 255

前馈网络 (FFN) 43

少样本谬误 99

填充中间 (FIM) 126, 399, 402

微调 DeepSeek

需求 318

微调模型

评估与结果 324-330

微调 293, 296

指令格式 317, 318

可选的独立 Python 脚本 296, 297

微调 Gemma 3

在 CUAD 上 318-323

FP8 (8 位浮点) 57

无损音频编解码 (FLAC) 357

功能词元共振成像 (fTRI) 44

函数调用功能 112-114

门控函数 54

Gemini 深度研究

参考链接 249

Gemma 3

需求 318

通用矩阵乘法 (GEMM) 57

生成器 260

全球 AI 生态系统

DeepSeek 的影响 30-32

Google Vertex AI

参考链接 350

GPQA 70

GPT 量化 (GPTQ) 357

分组查询注意力 (GQA) 50

分组相对策略优化 (GRPO) 59, 97

分组相对奖励机制 65

加固模式 116

硬件

部署选择 354

Hugging Face 深度学习容器 (DLC) 233

Hugging Face 推理端点

参考链接 351

用于将 DeepSeek V3 部署到云端 366-374

人类偏好微调 12

混合验证策略 71

IAM 角色 238

推理引擎 235

使用 355-357

Instructor 428-430

通过...使用 Pydantic 进行类型强制生成 114

用于实现类型安全 114-116

使用 AWS 进行隔离模型服务部署 233, 234

使用 LMI 容器部署 DeepSeek 237-245

推理后端 235-237

为使用 Amazon SageMaker 端点更新服务 245

关键绩效指标 (KPI) 171

键值 (KV) 55, 318, 336

KL 散度

参考链接 60

知识数据 12

知识蒸馏 71

知识蒸馏损失 77

Kubernetes 事件驱动自动缩放 (KEDA) 353

KV 缓存 378

LangChain 426, 427

LangGraph

参考链接 249

大语言模型 (LLM) 3, 143, 395

大语言模型运维 (LLMOps) 工具

用于模型蒸馏 304-308

大语言模型 (LLM) 200, 293

层归一化 49, 50

法律 AI 训练

动态 323

法律原理蒸馏

两阶段工作流 304, 308-323

轻量级解码器 42

LiteLLM 222-224, 305, 424-426

参考链接 222

llama.cpp

使用 417-420

llm 430

LLM 架构

比较 17, 18

LMI 容器 233, 237

长期记忆 251

低秩适应 (LoRA) 321

Luke Salamone

参考链接 397

Marginalia 搜索引擎

参考链接 281

MarkItDown

测试 188, 189

MATH 70

记忆

长期记忆 251

短期记忆 251

并购 (M&A) 301

混合专家模型 (MoE) 19, 40, 52, 335

使用 DeepSeek 实现架构 19-22

在 DeepSeek 中路由 52, 53

模型上下文协议 (MCP) 254-257

模型蒸馏

LLMOps 工具 304-308

MoE 门控网络 54

多头潜在注意力 (MLA) 40, 55, 56, 124, 340

多头自注意力 48, 49

多词元预测 (MTP) 58, 86

优势 59

原生 JSON 模式 111, 112

自然语言处理 (NLP) 8

Ollama 225, 348, 422, 423

URL 226, 421

用于本地部署 361-363

OpenRouter 224, 412, 413

URL 411

协调器组件 270

协调器-工作者工作流 270

报告生成工作流示例 271-278

基于结果的评分 60

outlines 430

内存溢出 (OOM) 377

模式 259

个人身份信息 (PII) 376

逐位置网络 49

注意力后归一化 49

注意力前归一化 49

偏好排序模型 (PRM) 86

生产系统

模型的 CI/CD 382-385

成本管理 379-381

监控与可观测性 375, 376

扩展与性能 376-379

安全实践 381, 382

提示 DeepSeek

用于结构化输出的高级技术和工具 109-111

提示 DeepSeek，技术与工具

函数调用功能 112-114

设置原生 JSON 模式 111, 112

健壮性与特殊情况策略 116-121

使用 Pydantic 进行类型强制生成 114-116

提示前缀缓存 378

提示路由 40

Pydantic

通过 Instructor 用于类型强制生成 114

Pydantic AI 430

Pydantic 模型

参考链接 211

Python 客户端

建立访问 416

python-garminconnect 项目

参考链接 202

量化

重要性 357-359

量化后的 DeepSeek Coder 模型

用于本地部署 361-363

每秒查询数 (QPS) 312

推理 62, 63

强化学习 14

基于人类反馈的强化学习 (RLHF) 6

强化学习 (RL) 11

重排序模块 42

检索增强生成 (RAG) 29

检索模块 42

奖励模型 10

旋转位置编码 (RoPE) 43, 48

路由器 44

R 系列

与 V 系列模型对比 126-128

基于规则的奖励机制 10

基于规则的 RLHF 67

基于规则的验证 12

SageMaker JumpStart

参考链接 350

脚手架 259

模式 96

模式漂移 118

评分函数 66

自注意力机制 15

自我教学

涌现模式 78

顺序处理 51

DeepSeek 设置

构建 424

Instructor 428-430

LangChain 426, 427

库与资源 430

LiteLLM 424-426

短期记忆 251

推测解码 58

技术栈 295

最先进 (SOTA) 21

结构化推理 63

监督微调 (SFT) 10

合成器 LLM 270

系统提示 63, 100

任务 250

温度参数 230

TensorRT-LLM 356

测试时缩放 62, 97

文本生成推理 (TGI) 233, 356

DeepSeek 的思考 64

处理复杂场景 68, 69

使用 GRPO 评估响应质量 65, 66

基于规则的 RLHF 67

思考与回答块 64

第三方 API

AWS 406-410

Cloudflare 403-406

OpenRouter 411-413

使用 402

首词元时间 (TTFT) 336

逐词元门控机制 54

词元化 51

工具调用 113, 251

工具调用智能体 279

网络搜索智能体 280-287

工具 251-253

DeepSeek 的训练 69, 70

冷启动数据 70-72

R1 的训练流水线 73-76

训练过程 8-14

Transformer 架构 8

Transformers

基于 CPU 的推理 227-231

类型强制生成

通过 Instructor 使用 Pydantic 114

Unsloth 227, 305

冗长提示陷阱 100-102

DeepSeek 的视觉能力

用于视觉语言集成的架构变化 79-81

视觉 Transformer (ViT) 79

vLLM 237

参考链接 235

V 系列模型

提示技术 121

与 R 系列对比 126-128

V 系列模型提示技术

上下文窗口混淆 124, 125

格式化嘉年华 123, 124

模板探戈 121, 122

网络搜索智能体 280-287

工作者 270

工作流 259

XGrammar

基于 CPU 的推理 227-231

ZenML 305

流水线数据处理 314-317

用于创建本地环境 294

ZenML Cloud 账户

API 密钥和环境变量 295

创建 294, 295

DeepSeek in Practice

From basics to fine-tuning, distillation, agent design, and prompt engineering of open source LLM

Andy Peng

Alex Strick van Linschoten

Duarte O.Carmo

DeepSeek in Practice

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Portfolio Director: Gebin George

Relationship Lead: Sonia Chauhan

Project Manager: Prajakta Naik

Technical Editor: Aditya Bharadwaj and Rahul Limbachiya

Copy Editor: Safis Editing

Indexer: Manju Arasan

Proofreader: Safis Editing

Production Designer: Jyoti Kadam

First published: November 2025

Production reference: 1171125

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-80602-085-0

www.packtpub.com

Contributors

About the authors

Andy Peng is a Senior Engineer at Amazon. He specializes in large language model inference optimization and evaluation for models like DeepSeek, Qwen, and Claude. His work spans AWS Bedrock, SageMaker, Amazon S3, AWS Fargate, AWS App Runner, Alexa Health & Wellness, and fintech. A NeurIPS 2025 Chair and program committee member for ICML, ICLR, KDD, and NeurIPS, he contributes to CNCF and the Linux Foundation, mentors at the University of Washington, and serves as a Resident Expert at the AI2 Incubator.

I would like to express my gratitude to my family for their support and for understanding that I needed to work long hours on this book. My sincere thanks go to my manager, Rakesh Ramakrishnan, and my colleagues, Raj Vippagunta and Siddharth Shah, for their valuable input and support.

I would also like to thank Gebin George for reaching out with the opportunity to write this book, which has been a truly unique experience. Special thanks to my co-authors, Alex and Duarte, and to the entire Packt team—including Vandita Grover, Prajakta, Gebin, and everyone else—for their unwavering support throughout the writing process. I first connected with Packt in 2022, and it is a pleasure to see the successful delivery of our first new book.

Alex Strick van Linschoten is a Machine Learning Engineer at ZenML. His work focuses on bridging the gap between machine learning research and production deployment, particularly within the LLMOps space. He leads and maintains the LLMOps Database, a comprehensive collection of over 1,000 case studies examining LLMOps and GenAI implementations in production environments. He transitioned to software engineering after earning a PhD in History and spending 15 years living and working as a historian and researcher in Afghanistan. He has authored, edited, and translated several books based on his historical research and is currently based in Delft, the Netherlands.

I’d like to thank Saba, Aria, and Blupus for their patience as I took many weekends off to work on the chapters of this book. I’d also like to thank Hamza and the rest of the ZenML team for their support in thinking through how best to present the ideas introduced below. Of course, much appreciation goes to the Packt team as well for their support in getting this out into the world!

Duarte O. Carmo is a technologist from Lisbon, Portugal, now based in Copenhagen, Denmark. For the past decade, he’s worked at the intersection of machine learning, artificial intelligence, software, data, and people. He has helped solve problems for both global corporations and small startups across industries such as healthcare, finance, agriculture, and advertising. His approach to solving tough problems always starts with the same thing: people. For the past five years, he’s been running his one-man consulting company, working with clients of all sizes and across industries. He’s also a regular speaker in the Python and machine learning communities and an active writer.

I’d like to thank my family, who have always encouraged me to follow my passion. In particular, I want to thank Vittoria. Writing a book is no easy task. Following your passion is no easy task. Leaving the dinner table because a client has a problem is no easy task. Hiding in the attic to write about an open-source LLM while the rest of you are on holiday is no easy task. Your unconditional support and love inspire me every day to keep going. As you once told me: “There are a lot of fun things out there to do—go do them!”

About the reviewer

Franck Benichou is a Senior AI Engineer with over six years of experience in machine learning and large language model (LLM) engineering. He currently works at Carta, the leading platform for private-market equity and fund data management. Following Carta’s acquisition of Accelex, Franck drives advancements in AI-powered document intelligence, applying Generative AI to transform complex financial data into structured insights. Before joining Carta, Franck worked at Deloitte (2024–2025) as an in-house Generative AI Developer, creating enterprise AI solutions and contributing to the firm’s internal AI strategy. From 2022 to 2024, he led Generative AI initiatives at EY (Ernst & Young), developing retrieval-augmented and content automation systems. Earlier, at Intact Financial Corporation’s R&D Data Lab (2020–2022), he specialized in usage-based insurance modeling and analytics, supporting telematics-driven pricing innovation. Franck combines strong technical depth with a product-focused mindset, building scalable and interpretable AI systems that bring automation, intelligence, and measurable value to data-driven organizations.

Preface

The space of large language models (LLMs) is evolving at a ludicrous pace. As we write this, the DeepSeek team has just released a new paper showing how to compress an LLM’s context using computer vision techniques. The sheer level of activity in the field of artificial intelligence is impressive and shows no signs of slowing down.

As authors of this book, we thought, What can we possibly write about DeepSeek that wouldn’t make this book instantly outdated? Our answer is simple: we believe that the future of AI technology is open, and the pioneer of that movement is DeepSeek.

DeepSeek has not stopped surprising the world of artificial intelligence, from releasing incredibly powerful models to publishing breakthrough research focused on LLMs. Given these developments, it was clear to us that we needed to write a book about what was happening.

The goal of this book is to give you - dear reader - the necessary tools to master DeepSeek. We want it to serve as your guide to mastering open-source language models.

In this book, we cover nearly everything you might encounter while working with DeepSeek models - from understanding what sets DeepSeek models apart to how you can use them for practical applications. We also cover how to master prompting DeepSeek models, so you can become an expert at interacting with this family of models. Finally, we dive deep into the practical side of things, with examples of how to design, build, and deploy agentic and non-agentic applications powered by DeepSeek models.

In this book, we start with an introduction to DeepSeek, where we explore its foundations and understand what sets it apart from the rest (Chapters 1 and 2). Once you understand what DeepSeek can do, it’s time to learn how to use it effectively. We have an entire chapter dedicated to the art of prompting reasoning models, where we show you how less is more (Chapter 3).

We then move into practical applications, demonstrating how to use it to produce consultant-grade analysis of complex problems (Chapter 4). From there, we build a complete end-to-end application where you’ll go from a simple API all the way to a fully containerized service running on Amazon Web Services (Chapter 5).

In the final part of this book, we explore the edge of LLM and AI technologies. You’ll learn how to use DeepSeek models as backbones for agentic applications (Chapter 6). We also walk you through a more complex MLOps use case where you’ll use a more powerful DeepSeek model to distill knowledge into a smaller one (Chapter 7). In the last chapter of this book, we cover the deployment of DeepSeek models, with their many trade-offs, so you can choose the best deployment methodology for your use case (Chapter 8).

To conclude this book we have an Epilogue to walk you through the key takeaways. The book also provides an Appendix that guides you through various ways to use DeepSeek.

Who this book is for

This book is aimed at anyone who wants to learn how DeepSeek models work. Whether you are an AI Engineer, a Software Developer, a Machine Learning Engineer, or a Researcher - as long as you are familiar with Python - you’ll be able to benefit from this book. If you have experience with Data Science, APIs, and some Machine Learning concepts, you’ll be able to benefit even more!

What this book covers

Chapter 1, What is DeepSeek, introduces DeepSeek, an open-source large language model, exploring its breakthroughs, evolution, comparisons with other LLMs, and impact on the global AI landscape today.

Chapter 2, Deep Dive into DeepSeek, dives into DeepSeek’s architecture, reasoning mechanics, advanced capabilities, offering a practical understanding of its unique strengths and emerging controversies.

Chapter 3, Prompting DeepSeek, shows you how to effectively craft prompts for DeepSeek models, making sure to showcase the difference between how this should be done for the V- and R-series models.

Chapter 4, Using DeepSeek: Case Studies, showcases real-world DeepSeek case studies, revealing how industry leaders apply it to boost productivity, improve development, and implement AI-assisted workflows using practical methodologies.

Chapter 5, Building with DeepSeek, takes you through creating a real-world application backed by DeepSeek models, how to iteratively improve it, and how to deploy DeepSeek models to AWS as an isolated service.

Chapter 6, Agents with DeepSeek, shows you how to start building agents with DeepSeek models, the different agentic patterns that exist, and a quick introduction to MCP.

Chapter 7, DeepSeek-Driven Fine-Tuning of Gemma 3 for Legal Reasoning, covers the distillation of DeepSeek’s R1 down into a fine-tuned Gemma 3 model. We explore a legal use case and how this would be evaluated in production scenarios.

Chapter 8, Deploying DeepSeek Models, showcases how to deploy DeepSeek models for production use cases. We show you how to spin up the full-scale models on cloud infrastructure and guide you on how to manage these as part of your production stack.

Chapter 9, Epilogue, reflects on the journey you took to understand the internals of DeepSeek, how to use it effectively, from application to deployment.

Chapter 10, Appendix, contains a DeepSeek Cheat Sheet which shows you how to get started quickly.

To get the most out of this book

Hands-on experience with Python, APIs, and tools like Ollama or llama.cpp
Experience with tools and platforms like uv and Docker
A solid understanding of machine learning concepts

Declaration

The authors acknowledge the use of cutting-edge AI, such as ChatGPT, with the sole aim of enhancing the language and clarity within the book, thereby ensuring a smooth reading experience for readers.

Download the example code files

The code bundle for the book is hosted on GitHub at https://github.com/PacktPublishing/DeepSeek-in-Practice. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://packt.link/gbp/9781806020850.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. For example: “Execute the python-garminconnect command:”

A block of code is set as follows:


# We get the values as environmental variables

GARMIN_EMAIL = os.getenv("GARMIN_EMAIL") # your email

GARMIN_PASSWORD = os.getenv("GARMIN_PASSWORD") # your password

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:


tools = [{

    "type": "function",

    "function": {

        "name": "analyze_sentiment",

        "description": "Analyze the sentiment of text",

        "parameters": {

            "type": "object",

            "properties": {

                "text": {"type": "string"},

                "confidence": {"type": "number",

                               "minimum": 0, "maximum": 1}

            },

            "required": ["text", "confidence"]

        }

    }

}]

Any command-line input or output is written as follows:


git clone https://github.com/PacktPublishing/DeepSeek-in-Practice.git

cd Chapter03

pip install -r requirements.txt

Bold: Indicates a new term, an important word, or words that you see on the screen. For instance, words in menus or dialog boxes appear in the text like this. For example: “Access the Cursor model configuration through the Settings menu.”

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book or have any general feedback, please email us at customercare@packt.com and mention the book’s title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you reported this to us. Please visit http://www.packt.com/submit-errata, click Submit Errata, and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright@packt.com with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit http://authors.packt.com/.

Once you’ve read DeepSeek in Practice, we’d love to hear your thoughts! Scan the QR code below to go straight to the Amazon review page for this book and share your feedback.

https://packt.link/r/180602084X

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Free Benefits with Your Book

This book comes with free benefits to support your learning. Activate them now for instant access (see the “How to Unlock” section for instructions).

Here’s a quick overview of what you can instantly unlock with your purchase:

| PDF and ePub Copies | Next-Gen Web-Based Reader |
| | |
| | Access a DRM-free PDF copy of this book to read anywhere, on any device. | | Multi-device progress sync: Pick up where you left off, on any device. |
| | Use a DRM-free ePub version with your favorite e-reader. | | Highlighting and notetaking: Capture ideas and turn reading into lasting knowledge. |
| | | | Bookmarking: Save and revisit key sections whenever you need them. |
| | | | Dark mode: Reduce eye strain by switching to dark or sepia themes. |

How to Unlock

Scan the QR code (or go to packtpub.com/unlock). Search for this book by name, confirm the edition, and then follow the steps on the page. | |
| Note: Keep your invoice handy. Purchases made directly from Packt don’t require one. |

Stay tuned

To keep up with the latest developments in the fields of Generative AI and LLMs, subscribe to our weekly newsletter, AI_Distilled, at https://packt.link/8Oz6Y.

Join our communities on Discord and Reddit

Have questions about the book or want to contribute to discussions on Generative AI and LLMs?

Join our Discord server at https://packt.link/4Bbd9 and our Reddit channel at https://packt.link/wcYOQ to connect, share, and collaborate with like-minded enthusiasts.

| | |

Once you’ve read DeepSeek in Practice, we’d love to hear your thoughts! Scan the QR code below to go straight to the Amazon review page for this book and share your feedback.

https://packt.link/r/180602084X

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Part 1 Understanding and Exploring DeepSeek

In the first part of this book, we’ll build a strong foundation for understanding DeepSeek and its role in the rapidly evolving world of AI. We begin by introducing DeepSeek as an open-source large language model and exploring why it has gained global attention. Next, we take a deep dive into its internal architecture, reasoning mechanics, and advanced capabilities to uncover what truly sets it apart. We’ll also explore effective prompting strategies to help you get the most out of DeepSeek models.

By the end of this part, you’ll have the context, technical understanding, and practical insights needed to confidently leverage DeepSeek in modern AI workflows.

This part of the book includes the following chapters:

Chapter 1, What is DeepSeek
Chapter 2, Deep Dive into DeepSeek
Chapter 3, Prompting DeepSeek

Stay tuned

To keep up with the latest developments in the fields of Generative AI and LLMs, subscribe to our weekly newsletter, AI_Distilled, at https://packt.link/8Oz6Y.

Join our communities on Discord and Reddit

Have questions about the book or want to contribute to discussions on Generative AI and LLMs?

Join our Discord server at https://packt.link/4Bbd9 and our Reddit channel at https://packt.link/wcYOQ to connect, share, and collaborate with like-minded enthusiasts.

| | |

1 What Is DeepSeek?

Artificial intelligence (AI) is rapidly evolving, and with it comes a suite of tools that allow developers, researchers, and innovators to build smarter, more adaptive systems. One such emerging tool is DeepSeek: a powerful, open-source large language model (LLM) designed to rival the capabilities of major LLMs such as GPT-4 and LLaMA. But what exactly is DeepSeek, and why should you care?

In this chapter, we’re going to dive into what DeepSeek is, how it fits into the broader AI landscape, and why it’s generating interest across the tech industry. You’ll gain an understanding of DeepSeek’s unique features and how it compares to other models in terms of training data, efficiency, and performance benchmarks.

By the end of this chapter, you’ll be equipped to understand the development of DeepSeek and key contributors to its success.

In this chapter, we’re going to cover the following main topics:

Introducing DeepSeek
Understanding the technical breakthroughs of DeepSeek
Impact on the global AI ecosystem
Exploring the versions and evolution of DeepSeek

Free Benefits with Your Book

Your purchase includes a free PDF copy of this book along with other exclusive benefits. Check the Free Benefits with Your Book section in the Preface to unlock them instantly and maximize your learning experience.

Introducing DeepSeek

DeepSeek is an open-source language model that aims to democratize and make advanced AI accessible. The first version, DeepSeek-R1, appeared on 20 January 2025, just before the Chinese New Year. Instead of shipping only a closed binary, the team published the weights, training scripts, and inference code, so anyone can examine or rebuild the system.

Released under the MIT license, the model carries no usage fees or strict terms. Anyone may run it locally or adapt it for new tasks. This freedom drew developers, researchers, teachers, and small firms worldwide. They apply DeepSeek-R1 in support bots, classroom aids, lab studies, and writing tools.

Benchmarks show DeepSeek-R1, competing with OpenAI-o3 and Gemini-2.5-Pro. It handles math, code, many languages, and complex prompts. The results suggest strong models need not be closed and underscores China’s growing role in frontier AI. The release also revived debates on open access and safety, and improving global research cooperation. On September 17, 2025, another milestone was reached as the DeepSeek-AI team published their research on the model DeepSeek-R1 in Nature and made it to the cover of that issue (https://www.nature.com/articles/s41586-025-09422-z).

From an architectural standpoint, DeepSeek leaned heavily on innovations in transformer-based models, while adding its own spin in later versions (explored in depth in the section, Versions and evolution of DeepSeek). But what made it truly stand out was its usability. DeepSeek could be deployed in a wide range of environments, from cloud servers to edge devices, and even laptops using lightweight versions.

Figure 1.1: Benchmark performance of DeepSeek-R1 (0528) (source: https://api-docs.deepseek.com/news/news250528)

Let’s talk about the various factors that contributed to the sudden rise and popularity of DeepSeek:

Open source architecture and training details: DeepSeek-R1 was released with a detailed research paper (https://arxiv.org/abs/2501.12948) outlining its architecture and training approach, benchmark scores across reasoning, math, and programming tasks (https://artificialanalysis.ai/providers/deepseek). This release was supported by full model weights, configuration files, and training scripts, along with six smaller distilled variants suited for local or low-resource environments (https://api-docs.deepseek.com/news/news250120) and immediate API availability (https://api-docs.deepseek.com/guides/reasoning_model) for developers wanting hosted access.
Timing: Part of its popularity was due to timing, as global organizations, scientists, and developers began exploring the new release. Additionally, the MIT license provided complete freedom for commercial use – an increasingly rare trait among performant models. The release also sparked excitement because it was not a research-only artifact; it was practical. Developers were able to fine-tune it, deploy it in production environments, and integrate it into existing AI workflows. The combination of power and usability became an instant draw.
Initial technical highlights: The most notable aspects of DeepSeek-R1 at launch included the following:
- Reasoning: It outperformed or matched leading models on key benchmarks involving mathematics, code, and logical reasoning.
- Efficiency: It provided performance close to GPT-4-level systems at significantly lower inference costs.
- Reinforcement-first training: Unlike conventional fine-tuning workflows that depend on supervised human-annotated data, DeepSeek skipped straight to reinforcement learning from human feedback (RLHF) or similar paradigms – though with minimal human labeling. The change sped up reasoning scores, lowered human-labeling costs, and allowed the model to tackle diverse tasks, such as math problems, zero-shot code, and so on, without needing narrow task-specific instructions.
- Custom architecture: While based on the transformer framework, DeepSeek-R1 incorporated innovations optimized for training stability and long-context understanding.

These elements together enabled the model to punch well above its weight, especially in multi-step reasoning and problem solving.

Real-world readiness: DeepSeek demonstrated real-world readiness from the outset, standing apart from many state-of-the-art models that excel in benchmarks but struggle in deployment. Unlike others that require extensive setup or closed infrastructure, DeepSeek was immediately usable in practical settings. It offered production-ready access via its API, local deployment with open weights and inference code, and customization through LoRA fine-tuning or prompt engineering. Integration with enterprise platforms such as Trae and Windsurf further streamlined orchestration. These capabilities, rarely combined so seamlessly in other models at launch, underscored DeepSeek’s commitment to practical utility beyond academic performance.
Community interest: DeepSeek-R1’s release sparked intense community activity. GitHub quickly overflowed with plug-ins, adapters, and fine-tuned spin-offs, while thousands of Hugging Face forks powered tools for contract review, tutoring, research aid, summarization, and coding support. Online forums shared benchmarks and hardware guides, and universities adopted the model for courses and lab projects. Forums such as Reddit, Zhihu, and Stack Overflow buzzed with shared experiments, performance tests, and guides for fine-tuning DeepSeek on local hardware. The accessibility of the model turned casual enthusiasts into researchers and developers into entrepreneurs. Today, DeepSeek also fuels educational initiatives. Several MOOCs and university labs have begun teaching LLM theory and experimentation using DeepSeek as the base model due to its openness and clarity.
Philosophical vision: DeepSeek’s vision aligns with a broader movement to build AI not as a gatekept asset, but as a shared global resource. Much like how Linux reshaped the software industry, DeepSeek aims to reshape AI development by putting tools in the hands of anyone curious or capable enough to use them. Its strategy is not just to compete with OpenAI or Google but to focus on accessibility and collaborative innovation.

DeepSeek-R1’s success is attributed to three factors: RLHF training, open source commitment, and its competitive performance across benchmarks.

Together, these elements laid the groundwork for DeepSeek not just as a model, but as an ecosystem. The remainder of this chapter will explore the reception of DeepSeek-R1 and the motivations behind its open philosophy, what technical breakthroughs powered it, and how it is evolving into a full-scale ecosystem.

Understanding the technical breakthroughs of DeepSeek

What truly sets DeepSeek-R1 apart are the technical innovations embedded in its architecture and training process. These innovations enabled it to outperform many contemporary models and helped redefine how future LLMs might be built.

The training process

Before we begin with DeepSeek’s training process, let’s first take a look at the development process of leading LLMs, which usually consists of the following:

Pretraining on massive corpora using self-supervised learning: In this stage, models are exposed to large-scale, diverse datasets such as books, websites, and code. The goal is to learn general language patterns without explicit labels. Common pretraining strategies include the following:
1. Autoregressive modeling (e.g., GPT): The model predicts the next word in a sequence.
2. Masked language modeling (e.g., BERT): The model predicts missing (masked) words.
3. Permutation-based modeling (e.g., XLNet): The model learns over multiple possible word orders.
Transformer architecture: Most LLMs use the transformer architecture (Figure 1.2), known for its scalability and performance in Natural Language Processing (NLP) tasks. They usually employ self-attention to determine contextual relationships between words. Some of the variants are as follows:
1. Encoder-only (e.g., BERT) for classification or understanding.
2. Decoder-only (e.g., GPT) for generation.
3. Encoder-decoder (e.g., T5) for tasks such as translation and summarization.

Figure 1.2: Transformer architecture (Attention Is All You Need, Vaswani et al., https://arxiv.org/pdf/1706.03762)

Supervised fine-tuning on curated instruction-following datasets: After pretraining, the model is fine-tuned on high-quality, labeled datasets where it learns to follow specific instructions and perform useful tasks. These datasets typically consist of human-written prompts and ideal responses, helping the model learn how to interact more directly and purposefully.
RLHF: To better align model outputs with human preferences, RLHF is applied:
1. Human reviewers evaluate and rank multiple model responses.
2. A reward model is trained to predict these rankings.
3. The LLM is then fine-tuned using reinforcement learning algorithms (commonly Proximal Policy Optimization (PPO)) to generate outputs that maximize the reward model’s score, thus better aligning with human values and expectations.

DeepSeek-R1 broke this convention by bypassing supervised fine-tuning entirely. Instead, it jumped directly from pretraining to reinforcement learning, helping it learn strong reasoning skills on its own.

Avoiding supervised fine-tuning (SFT) eliminates the dependency on expensive, manually annotated datasets. Instead, DeepSeek adopts rule-based reward mechanisms, such as automatically validating correct answers or checking output formats. This approach is more scalable and cost-efficient, and helps overcome the limitations of human data curation. Additionally, the use of explicit, rule-driven rewards, such as verifying answers within designated structures or confirming code functionality, mitigates reward hacking, a common issue with less predictable neural reward models.

Figure 1.3 shows the workflow diagram of DeepSeek’s model training.

Figure 1.3: DeepSeek-R1 model training

The training pipeline for DeepSeek-R1 begins with DeepSeek-V3, a large 671B-parameter base model. This foundational model undergoes reinforcement learning (RL) using rewards focused on accuracy and output formatting, resulting in an intermediate model called DeepSeek-R1-Zero. This model serves as a crucial transition point, enabling more task-specific training in the following stages.

Next, DeepSeek-R1-Zero is fine-tuned using cold start data, which refers to a broad and diverse collection of instruction-following examples. This data is typically well-structured and curated to give the model a basic understanding of various task formats and domains, making it suitable for initial general-purpose instruction tuning.

Following this phase, additional rounds of SFT are applied using two specialized datasets. The first is chain-of-thought (CoT) data, which emphasizes multi-step reasoning. This data helps the model learn how to solve complex problems by breaking them down into intermediate steps – essential for mathematical reasoning, logical inference, and multi-hop question answering. The second set is knowledge data, which contains fact-rich, domain-specific content such as scientific literature, encyclopedic information, and technical manuals. This helps the model improve factual accuracy and domain coverage.

Once these fine-tuning stages are complete, the resulting model, DeepSeek-R1, is further enhanced through advanced RL techniques. It is trained with rewards not only for accuracy and formatting, but also for consistency, ensuring that its outputs are logically coherent and self-consistent. Furthermore, rule-based verification is applied to automatically validate responses in domains such as mathematics and code generation. Finally, human preference fine-tuning is incorporated to align the model’s behavior with human expectations and judgments of quality.

DeepSeek-R1 has been distilled into smaller, more efficient variants. These include DeepSeek-R1-Distill-Qwen, which uses Qwen 2.5 models ranging from 1.5B to 32B parameters, and DeepSeek-R1-Distill-LLaMA, which utilizes LLaMA 3 models in 8B and 70B sizes. These distilled versions retain much of the capability of the original R1 model but are optimized for different resource and latency constraints.

Overall, the DeepSeek-R1 pipeline represents a multi-phase strategy combining supervised learning, reward-based tuning, and targeted model distillation to deliver a family of instruction-following language models optimized for performance, generalization, and deployment flexibility.

As a result, the training process becomes more stable and efficient, benefiting from simplified, reliable reward signals that reduce noise and ambiguity.

DeepSeek’s training approach helped in the following aspects:

Reduced human labor cost: No need to manually annotate or rank thousands of instructions.
Faster development cycle: The training timeline was streamlined significantly.
Greater generalization: The model learned to generalize instruction-following through trial-and-error interactions rather than fixed templates.

Despite lacking traditional supervised instruction datasets, DeepSeek-R1 demonstrated robust instruction-following capabilities, competitive with models that underwent fine-tuning. This suggested that RL alone – when well-designed – can endow a model with a deep understanding of instructions and intent.

Let’s take a look at DeepSeek’s inference pipeline, as depicted in Figure 1.4.

Figure 1.4: DeepSeek-R1 model inference

The inference pipeline of DeepSeek-R1 is designed to prioritize structured, verifiable outputs through a combination of rule-aware decoding and prompt optimization. During inference, DeepSeek-R1 leverages formatting-aware generation mechanisms, where the model is encouraged, often via prompt design and internal alignment, to produce well-structured, interpretable responses, especially for tasks involving code, math, or CoT reasoning. It is optimized not only for fluency but also for factual and logical coherence, frequently incorporating intermediate steps (CoT) in its answers, even without explicit prompting. This enables DeepSeek-R1 to deliver step-by-step solutions and structured outputs in JSON, Markdown, or code blocks, increasing reliability for downstream applications.

What sets DeepSeek-R1 apart from other state-of-the-art LLMs is its rule-based, reward-aligned inference behavior. Many leading LLMs rely primarily on end-to-end training with human preference fine-tuning, whereas DeepSeek-R1 integrates rule-based verification techniques directly into the RL loop. This has downstream effects at inference time: DeepSeek-R1 is more likely to generate outputs that are compatible with formal validators or downstream evaluators (e.g., test cases for code, equations for math). As a result, it shows stronger performance in domains where precision, structure, and interpretability are critical, while slightly trading off open-ended conversational flexibility.

Next up is DeepSeek’s RL approach.

Reinforcement learning

DeepSeek-R1’s RL approach is notable for its distinctive design choices, which we will explore in depth in Chapter 2. Unlike many models that apply RL only in the final stages of training, DeepSeek-R1 introduced alignment techniques. Alignment techniques are the methods aimed at steering the model’s outputs to be more helpful, honest, and harmless – early in its training process. This early alignment contributes to more consistent and desirable behavior throughout the model’s development.

It also replaced large-scale human annotation with an automated reward model, significantly improving scalability and reducing reliance on manual labeling. Another defining feature was its use of self-play and iterative refinement, enabling the model to generate, evaluate, and improve its own outputs. This approach helped DeepSeek-R1 internalize advanced reasoning patterns and strategic decision-making, making it particularly effective at multi-turn reasoning, code explanation and completion, and solving complex mathematical problems.

Moreover, RL training helped mitigate hallucinations by reinforcing factual accuracy through self-generated success metrics.

For readers unfamiliar with alignment in the context of AI, it generally refers to techniques that ensure a model’s behavior aligns with human intent and values. A helpful introduction to these concepts can be found in OpenAI’s alignment overview (https://openai.com/index/our-approach-to-alignment-research/) or in the Alignment Newsletter (https://www.alignmentforum.org/s/dT7CKGXwq9vt76CeX).

Apart from this, DeepSeek made some modifications to the transformer architecture. Let’s find out.

Architecture modifications

DeepSeek-R1 builds on the widely adopted transformer architecture, which forms the foundation for most modern LLMs. At its core, the transformer uses a self-attention mechanism that allows each token in an input sequence to weigh the importance of every other token, regardless of position. This enables the model to capture long-range dependencies and contextual relationships more effectively than traditional recurrent models.

However, standard attention becomes computationally expensive as input length increases. To address this, DeepSeek-R1 introduces adaptive attention routing, a major architectural evolution. Unlike traditional transformers that apply fixed full attention across all tokens, this mechanism allows the model to selectively attend to the most relevant tokens based on learned relevance scores computed during training. These scores are typically derived from internal attention weights or auxiliary gating mechanisms, which prioritize tokens that contribute most to minimizing the training loss. By focusing computational resources on high-impact tokens, especially in long sequences, adaptive attention routing enables DeepSeek-R1 to handle inputs of up to 32,000 tokens more efficiently. This not only enhances the model’s ability to comprehend and summarize large documents but also reduces computational overhead by avoiding redundant attention over less informative tokens.

Figure 1.5 compares the traditional transformer attention and DeepSeek’s adaptive attention routing.

Figure 1.5: Traditional transformer attention versus DeepSeek-R1 adaptive attention routing

In addition, DeepSeek-R1 employs mixed-precision optimization, combining FP16 (half-precision) and INT8 (quantized) arithmetic to improve training and inference efficiency. This approach reduces memory usage and accelerates computation while maintaining competitive model performance in terms of accuracy and perplexity. Typically, FP16 is used throughout most of the model for general computation, while INT8 quantization is selectively applied to inference-time matrix multiplications, often in attention and feed-forward layers, where precision can be reduced without significantly impacting output quality. By carefully choosing which layers to quantize, DeepSeek-R1 achieves a favorable trade-off between efficiency and performance. This approach significantly accelerates inference and training while maintaining output quality, making it ideal for deployment at scale.

The model also benefits from efficient parallelization strategies. It leverages tensor parallelism and activation checkpointing to reduce memory usage during training, allowing it to be trained on multi-GPU systems more effectively.

Together, these enhancements make DeepSeek-R1 an evolution of the transformer, one that is not only more scalable and context-aware but also more cost-efficient in both training and inference.

Let’s now turn our focus to how the DeepSeek architecture compares to the architecture of other LLMs.

Comparison of major LLM architectures (2025)

As the field of LLMs evolves, different architectures and training paradigms have emerged. This comparison focuses (Table 1.1) on key differentiators, particularly the use of Mixture-of-Experts (MoE) architectures and RL among some of the most impactful models in recent years.

Model	MoE ?	Key architecture highlights	RLHF	Open source?
DeepSeek-R1	Yes. Sparse MoE	671 B params; MoE + Multi-Head Latent Attention; reasoning-centric	No RLHF, uses pure reinforcement learning (GRPO)	Yes (MIT)
Claude 4	No	Dense transformer; built with Anthropic’s Constitutional AI and Direct Preference Optimization (DPO)	Yes, advanced RLHF + DPO	No
Gemini 2.5 Pro	Yes. Sparse MoE	Multimodal sparse MoE transformer; 1 M token context (2 M soon)	Yes. RLHF + ongoing alignment	No
GPT-4.5	No	Released Feb 27, 2025; OpenAI’s largest non-CoT model (Orion)	Yes. RLHF + SFT	No
o3	Unknown (likely dense)	Optimized for personalized assistant tasks; improved grounding and memory modules	Yes, advanced RLHF	No
Grok 3.5	No evidence of MoE	Dense transformer; enhanced reasoning from Grok 3; advanced “Think” mode; still proprietary	Yes. RL-based training + RLHF fine-tuning	No
Gemma 3	No, dense	Lightweight MoE; instruction-tuned with long-context	Yes. RLHF	Yes (Apache 2.0)
LLaMA 4	No	Dense transformer; advanced memory and modular layers	Yes. RLHF and safety fine-tuning	Yes

Table 1.1: Comparison of LLM architectures

Leading language models are increasingly diverging in their architectural strategies, particularly around the use of MoE. Models such as Gemini 2.5 Pro and DeepSeek-R1 adopt sparse MoE architectures, enabling large parameter scales while maintaining efficient compute usage. GPT-4.1 is widely believed to incorporate some form of MoE or sparse expert routing, based on its strong performance and low-latency characteristics, though exact details remain undisclosed.

Sitting between these approaches, Grok 3.5 retains a dense transformer architecture, optimized for real-time responsiveness and integrated reasoning. It avoids MoE entirely, focusing instead on RL techniques and iterative refinement using live feedback data.

In contrast, Claude 4 and LLaMA 4 continue with fully dense designs, prioritizing simplicity, alignment stability, and predictable behavior over raw parameter scaling.

Now that we have introduced you to the DeepSeek’s technical innovation, we will dive deeper into the MoE design of DeepSeek.

MoE architecture

A major architectural breakthrough in DeepSeek-R1 is the integration of the MoE design.

MoE is a modular neural network design where only a subset of parameters (called experts) is activated for any given input. Rather than using the full parameter space for every prediction, MoE selectively activates a few experts dynamically.

How DeepSeek implements MoE

DeepSeek employs a sparse MoE architecture, in which a gating network dynamically selects two out of N expert networks at each layer to process a given input. Selecting two experts strikes a balance between computational efficiency and model expressiveness. It allows the model to leverage diverse expertise without incurring the full cost of activating all experts. This approach enables specialization across experts while keeping inference latency and resource usage manageable. These experts are not manually assigned to specific tasks such as math or code; instead, specialization emerges during training. The gating mechanism learns, through optimization, to route inputs to the most effective experts based on contextual cues. Over time, certain experts become more activated for specific domains (e.g., language, reasoning, and coding) as a result of this learned routing, effectively developing functional specialization.

Figure 1.6 provides an overview of this architecture.

Figure 1.6: Conceptual view of MoE architecture

Each expert processes the same type of input representations but may learn to emphasize different aspects depending on the patterns it receives. Their role is shaped by the data they are most often routed for, which, in turn, guides their parameter updates. This allows experts to take on distinct roles organically, without needing different input formats or encodings. The model contains a large pool of expert subnetworks (each a small feed-forward network), but only a small subset – typically two – is activated per input token. A gating network evaluates the context and dynamically decides which experts to activate, allowing the model to adaptively route information where it’s most effectively processed.

This structure yields several key benefits:

Scalability: Since only a few experts are active at any time, the model can maintain a large number of total parameters while consuming less compute per token than a dense model of equivalent size. This enables DeepSeek to scale up without linear increases in computational cost.
Modularity: Experts can be trained, frozen, updated, or even swapped independently. This modularity allows for efficient continual learning, domain adaptation, or task-specific fine-tuning without retraining the entire model.
Specialization: As the gating network learns to route different inputs to different experts, these subnetworks begin to specialize; some become more attuned to code, others to mathematical reasoning, natural language, or dialogue. This reduces the risk of overfitting and enhances the model’s ability to generalize across diverse tasks.

Through this architecture, DeepSeek-R1 essentially behaves like an ensemble of domain-specific models, but without duplicating resources or incurring the latency overhead typically associated with running multiple systems in parallel.

Apart from DeepSeek, many state-of-the-art (SOTA) LLMs also employ MoE architecture, the details of which are provided in the following table:

Model	Parameter count	Active experts per token	Total experts	Routing type	Use case strengths
DeepSeek-R1	~130B total / ~30B active	2	~64	Sparse + Gating	Math, reasoning, code, and multilingual understanding
Gemini 2.5 Pro	Estimated 1T+ (MoE config)	Unspecified (Likely 2–4)	Dozens	Proprietary Sparse	Multimodal apps, coding, and retrieval-augmented reasoning
Grok-3	Estimated 400B+ total (MoE)	2–4 (adaptive)	Unspecified (20+)	Advanced Dynamic MoE	Enhanced reasoning, DeepSearch, vision + code + chat, and long context
Grok-1.5	Estimated 300B total	2–4 (adaptive)	16+	Dynamic Routing	Real-time interaction, multimodal learning, and large-scale context tracking
Mixtral 8x7B	56B total / 12.9B active	2	8	Top-2 Gated MoE	General-purpose reasoning, fast inference, and multilingual
Switch Transformer	1.6T total / ~15B active	1	2,048	Top-1 Routing	Scalability benchmark; pioneered MoE at trillion-scale
GLaM	1.2T total / 93B active	2	64	Top-2 Routing	NLP understanding, code, and scientific tasks

Table 1.2: Comparison of MoE models

With MoE architecture gaining traction, Mixtral 8×7B showed how open source models could gain strong reasoning with limited compute, while newer systems such as Grok and Gemini 2.5 add adaptive or proprietary routing and multimodal pretraining. Google’s Switch Transformer and GLaM, both trillion-parameter prototypes, first confirmed that MoE could scale reliably. Together, these projects show how MoE lets very large models grow while keeping inference fast enough for real-time, high-performance tasks.

Another foundational aspect of building an LLM is the data on which it is trained. Let’s see how DeepSeek utilized its training dataset.

Training dataset and philosophy

In the paper DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (https://arxiv.org/abs/2501.12948), the authors state that DeepSeek-R1 was trained on a large, diverse, and domain-rich multilingual dataset intended to support reasoning, coding, and cross-lingual understanding.

While the precise composition of this dataset is not publicly disclosed, the paper outlines the use of carefully curated cold-start data and approximately 800,000 SFT samples, integrated into a multi-stage RL framework to develop the model.

The data selection strategy emphasized high-quality sources across several key domains:

The model was heavily trained on programming languages such as Python, JavaScript, and Rust, using curated code from repositories such as GitHub and developer Q&A platforms such as Stack Overflow.
To strengthen mathematical and logical reasoning, the dataset incorporated formal logic corpora, symbolic mathematics benchmarks, and collections of competitive math problems (e.g., MATH, GSM8K, and ProofWriter).
Scientific understanding was bolstered through pretraining on academic publications and technical manuals, drawing from sources such as arXiv, and open-access research datasets.
The multilingual component of the dataset was anchored in both Chinese and English, with additional coverage of major global languages sourced from datasets such as CCMatrix and OPUS.

This diverse training foundation reflects DeepSeek-R1’s objective: to combine high-level reasoning with robust multilingual and domain-specific capabilities.

In the same paper, the authors describe adopting a low-filtering strategy in curating their training data, contrasting with the more aggressive data filtering pipelines used by many other LLM developers. This design choice was made to preserve the natural complexity and diversity of language, enabling the model to better capture informal expressions, culturally specific idioms, emotionally charged language, and edge-case scenarios. According to the authors, such linguistic variety supports more expressive, creative, and contextually fluent model behavior. However, the paper does not specify exactly what types of content, if any, were filtered out. Without transparency into the dataset composition or filtering criteria, it remains unclear whether the training corpus included potentially harmful content such as hate speech, misinformation, or offensive material. While this low-filtering approach may enhance the model’s ability to generalize across diverse linguistic and cultural contexts, it also introduces the risk that undesirable content could be learned and reproduced. These trade-offs underscore the importance of downstream safeguards and responsible use, particularly when deploying the model in real-world settings.

DeepSeek-R1’s architecture and training strategy resulted in a model particularly well-suited for open-ended tasks. It demonstrates strong capabilities in brainstorming, idea generation, and exploratory dialogue – areas that benefit from flexibility and minimal preconception. The model also performs well in nuanced translation and multilingual reasoning, aided by its diverse language training. Additionally, DeepSeek-R1 is noted for its adaptability to user tone and conversational style, which many users find useful in creative and collaborative contexts.

Well, there have been several versions of DeepSeek, and we expect newer releases as the race toward artificial general intelligence heats up.

In the next section, we will talk about how the DeepSeek ecosystem has evolved since the release of R1.

Versions and evolution of DeepSeek

The story of DeepSeek is not just about a single model launch. It’s about a continuously evolving ecosystem. Each iteration of DeepSeek introduces significant upgrades in reasoning, usability, safety, and integration across platforms. Understanding the evolution of DeepSeek is critical to appreciating its long-term vision and potential.

DeepSeek’s evolving ecosystem

Since its debut, DeepSeek has made significant strides across multiple domains, including language understanding, mathematical reasoning, code generation, and multimodal capabilities. Each version introduces meaningful enhancements, underscoring DeepSeek’s goal of democratizing high-performance language models without compromising on quality.

Understanding DeepSeek’s evolution provides critical insight into its growth trajectory, vision, and how it continues to disrupt both proprietary and open AI ecosystems. In the following table, we chronologically explore the key milestones, model variants, and product layers that now form the DeepSeek suite.

Table 1.3: DeepSeek model evolution

Each of these models targets specific use cases, from general-purpose chatbot functions to highly focused coding and math tasks. Let’s talk about them in detail.

Deep dive: Feature comparison of each model

The DeepSeek ecosystem has rapidly evolved into a suite of specialized models, each designed to address different use cases in reasoning, coding, vision, and general-purpose AI. While all variants build on a shared architectural backbone and training philosophy, each model iteration introduces new capabilities, performance trade-offs, and domain optimizations. Here is an overview of the key models within the DeepSeek family and how they compare in terms of specialization and utility:

DeepSeek LLM (https://github.com/deepseek-ai/DeepSeek-LLM): The original backbone of the DeepSeek family, the DeepSeek LLM laid the foundation for all future iterations. While it lacked some specialized capabilities, it established multilingual competence and solid reasoning as core priorities.
DeepSeek Math (https://github.com/deepseek-ai/DeepSeek-Math): Tailored for students, researchers, and technical professionals, DeepSeek Math focuses on multi-step reasoning problems in algebra, calculus, geometry, and symbolic logic. It serves as a viable open source alternative to Wolfram Alpha-like reasoning systems.
DeepSeek Coder (https://github.com/deepseek-ai/DeepSeek-Coder) and Coder V2 (https://github.com/deepseek-ai/DeepSeek-Coder-V2): The first Coder model introduced competitive performance in Python and JavaScript, integrated with development environments such as VS Code and GitHub Copilot. Coder V2 (June 2025) significantly raised the bar, approaching Claude 3.5 in inline function synthesis, docstring generation, and type inference.
DeepSeek VL (https://github.com/deepseek-ai/DeepSeek-VL): A pivotal release for multimodal applications, VL supports both image and text inputs, opening the door for applications in visual question answering, optical character recognition, document summarization, and more. While it still lags behind GPT-4-V or Gemini 1.5 Pro in vision capabilities, it’s rapidly improving.
DeepSeek V2 (https://github.com/deepseek-ai/DeepSeek-V2) and V3 (https://github.com/deepseek-ai/DeepSeek-V3): The V2 update prioritized prompt alignment, minimizing hallucinations and expanding support for longer contexts. V3 followed up with better long-term memory support, faster inference, and internal planning modules that enabled early-stage agentic behavior.
DeepSeek-R1 (https://github.com/deepseek-ai/DeepSeek-R1): This was the launch version that started it all. Released on January 20, 2025, it quickly became the top-performing open source model across a wide array of benchmarks. Key highlights include a fully open source, MIT-licensed release that provides model weights, a tokenizer, and the entire training pipeline. DeepSeek-R1 delivers strong logical reasoning capabilities, surpassing most open models and rivaling some proprietary systems, along with robust multilingual performance in both English and Chinese. As already discussed, the release also features a set of smaller, distilled variants under 20B parameters, enabling efficient use in local or edge environments. Practical applications range from API-based chatbot integration and Copilot-style coding assistance to lightweight deployments via platforms such as Ollama and VS Code extensions.
DeepSeek-R1-0528 (https://github.com/marketplace/models/azureml-deepseek/DeepSeek-R1-0528): This update solidified DeepSeek’s position at the top of the open source pyramid. The May 28 version refined the core model, making it more aligned, accurate, and efficient. The new features and enhancements include the following:
- Substantial reduction in hallucinations: Notably in scientific and historical facts.
- Improved coding fluency: Achieved parity with GPT-4-turbo in many Python tasks.
- Mathematics performance: Enhanced accuracy in multi-step algebra, geometry, and logic problems.
- Updated prompt alignment: Better adherence to user instructions, even in ambiguous prompts.
- Multimodal readiness: Architecture adapted for future image/text fusion.

In benchmark evaluations, the May release of DeepSeek-R1-0528 showed a 7% improvement in mathematical reasoning tasks compared to the January version. Code generation performance, measured on HumanEval-style benchmarks, increased by 9%. Additionally, the model demonstrated effective long-context reasoning, handling inputs up to 32,000 tokens with minimal performance degradation.

The release of DeepSeek-R1-0528 underscored that the pace of DeepSeek’s development remained strong and consistent. Its improved performance and open accessibility led many developers to begin migrating entire workflows from GPT-based systems to DeepSeek APIs. This shift was further supported by a surge in ecosystem integrations, including Visual Studio Code plugins, Ollama compatibility, and Dockerized deployment options, signaling growing adoption across both individual and enterprise-level users.

DeepSeek-V3.1 (August 2025) (https://huggingface.co/deepseek-ai/DeepSeek-V3.1): DeepSeek V3.1 is a cutting-edge hybrid reasoning model featuring both thinking and non-thinking modes, advanced agent and tool use capabilities, a massive 685-billion-parameter architecture, and an extended 128,000-token context window for long-document understanding. The model is engineered for fast, structured, multi-step reasoning and supports code generation, search, and agentic workflows, with enhanced efficiency from MoE architecture and optimized inference, matching or surpassing previous DeepSeek benchmarks while maintaining low latency. DeepSeek V3.1 also boasts strong multilingual support, open source availability for research, and specialized training for reliable external tool integration and reduction of hallucinations, making it suitable for diverse enterprise and developer applications.

Apart from various models, DeepSeek has created many products for streamlined adoption and use. Let’s take a look.

DeepSeek product ecosystem

Beyond the models themselves, DeepSeek has developed a growing suite of user-facing products and developer tools that make adoption frictionless:

DeepSeek app: A mobile-first AI assistant app available for Android and iOS, offering real-time interaction with DeepSeek-R1 and Math/Coder variants. Key features include voice input, code cell execution, note-taking, and multilingual support.
DeepSeek web app: Accessible at deepseek.com/chat, this offers a clean and responsive interface for real-time interaction with various DeepSeek model variants. It includes conversation memory, allowing users to maintain context across multiple exchanges for more coherent dialogues. Prompt templates are available to streamline repetitive tasks or structured inputs, making it easier to prototype or test specific behaviors. Additionally, users can export entire chat sessions in Markdown or PDF formats, which is particularly useful for documentation, collaboration, or offline review.
DeepSeek developer platform (https://platform.deepseek.com/): Offers a flexible and open environment for building and deploying AI-powered applications. Developers can fine-tune models to create custom endpoints tailored to specific tasks or domains. The platform supports seamless model selection across the DeepSeek family, including general-purpose (R1), coding-focused (Coder), and multimodal (VL) variants. With support for context-aware API calls up to 32,000 tokens, it enables sophisticated multi-turn reasoning and long-form content processing. A beta feature for role and function calling is also available, allowing developers to define structured interactions and extend model capabilities for tools, agents, or workflow automation.

DeepSeek’s value proposition lies in its accessible model weights, strong reasoning capabilities, and low cost. This combination makes DeepSeek-R1 an attractive choice for educational use, research and development, and start-ups seeking advanced AI tools without restrictive licensing or high costs.

Now, let’s take a look at the integration and deployment support DeepSeek offers.

Platform integrations and deployment ecosystem

DeepSeek’s accessibility is one of its defining strengths, thanks to a wide range of integration and deployment options across local, cloud, and web environments. Here is a high-level overview of where and how DeepSeek can be used:

For local and edge deployment, DeepSeek runs seamlessly on platforms such as Ollama (https://ollama.com/library/deepseek-r1 – ollama run deepseek), Docker Hub (https://hub.docker.com/r/devlinrocha/ollama-deepseek-r1-7b/tags), and through VS Code extensions (https://github.com/enesbasbug/deepseek-vscode-extension) for code completion and inline assistance. It also supports private cloud and on-premises deployments via Kubernetes, Docker Compose, or direct GPU cluster setups.
In the cloud, DeepSeek is integrated with Amazon Bedrock (https://aws.amazon.com/bedrock/deepseek), Amazon SageMaker (https://aws.amazon.com/blogs/machine-learning/deploy-deepseek-r1-distilled-models-on-amazon-sagemaker-using-a-large-model-inference-container/), Azure ML (https://azure.microsoft.com/en-us/blog/deepseek-r1-is-now-available-on-azure-ai-foundry-and-github/), and Google Cloud Vertex AI (https://cloud.google.com/vertex-ai/generative-ai/docs/maas/deepseek), enabling scalable inference via Hugging Face or custom containers.
Hosted APIs are available via providers such as Fireworks.ai (https://fireworks.ai/), Together.ai (https://www.together.ai/), Replicate (https://replicate.com/), and Modal (https://modal.com/), offering both rapid prototyping and production-ready workflows.
For lightweight and browser-based access, users can try models on deepseek.com/chat, Hugging Face Spaces, or through ready-to-run notebooks on Google Colab and Kaggle.

If you wish to explore these deployment options, we have created an Appendix toward the end of the book for you to explore.

DeepSeek also integrates with popular orchestration frameworks such as LangChain, Haystack, and LlamaIndex for retrieval-augmented generation (RAG), as well as no-code tools such as Turing, Trae, and Windsurf for structured agent workflows.

DeepSeek’s upcoming roadmap outlines an ambitious expansion of its model capabilities and deployment strategies. Planned developments include multimodal training that incorporates image and audio understanding, as well as the creation of autonomous agents equipped with memory, planning, and tool-use functions for complex workflows. The team is also working on smaller, task-specific variants tailored to domains such as medical question answering, legal analysis, and STEM education. In addition, DeepSeek is investing in privacy-focused deployments through federated training methods and secure, on-premises LLMs. Following the success of the R1 series, anticipation is building for a potential DeepSeek-R2 release in late 2025. While unconfirmed, early reports suggest it may offer structured reasoning capabilities that rival GPT-5, while maintaining a fully open source framework.

But with all the hype comes skepticism, too. The next section will examine how DeepSeek’s choices have influenced the broader AI ecosystem, including pricing models, competition, and global policy dynamics. We will also talk about some concerns and risks that are being commonly discussed in the community.

DeepSeek’s impact on the global AI ecosystem

The release of DeepSeek-R1 wasn’t just a technical milestone; it was a strategic turning point for the global AI industry. Its unique combination of open source accessibility and top-tier performance sent shockwaves through research labs, start-ups, and policy circles alike. The implications have spanned economic competition, academic acceleration, ethical discourse, and geopolitical realignment.

Market disruption and price wars

Before DeepSeek-R1, frontier-level LLMs were typically expensive, restricted to API-only access, or bound by licenses that limited commercial use. The release of DeepSeek-R1, with freely available model weights, inference code, and permissive licensing, marked a significant shift, making high-performing language models far more accessible.

DeepSeek-R1 made a notable impact on the AI landscape by introducing free, full access to a high-performing reasoning model. It demonstrated competitive accuracy across domains such as mathematics, programming, and formal logic – areas traditionally dominated by proprietary systems. By making these capabilities openly available, DeepSeek offered a viable alternative to closed source platforms for both researchers and commercial developers.

Perhaps the most significant consequence of this release was the market pressure it generated. By lowering the economic barrier to advanced AI experimentation, DeepSeek-R1 challenged prevailing assumptions about access and affordability in the field. Its open availability pushed several proprietary labs to accelerate their own open source strategies in response. Moreover, the move sparked broader conversations about AI equity, competition, and global governance, highlighting the growing tension between innovation, accessibility, and responsible deployment. Its open release created ripples far beyond China, inspiring labs in Europe, India, and even Silicon Valley start-ups to explore more transparent models.

DeepSeek-R1’s release triggered a notable shift in AI pricing and market positioning. In response, OpenAI introduced more affordable options, such as discounted access to GPT-3.5 Turbo, while keeping premium GPT-4.1 plans priced at around USD 200 per month for those needing top-tier performance. Anthropic and Cohere also reacted by launching smaller, lower-cost chat models aimed at maintaining their appeal to budget-conscious users. Meanwhile, Meta reaffirmed its commitment to its LLaMA roadmap, signaling expanded licensing options and underscoring a pivot toward broader accessibility in model deployment.

Open source suddenly wasn’t a niche, with DeepSeek becoming a serious economic threat to closed-model business models. Enterprises, particularly cost-sensitive ones, began evaluating DeepSeek for customer support bots, embedded agents, and enterprise knowledge bases, replacing more expensive APIs.

Sparking the next wave of open source AI

DeepSeek’s open release emboldened the global open source movement. Developers and researchers who had previously felt locked out of meaningful LLM contributions found new momentum. DeepSeek demonstrated that you didn’t need a billion-dollar infrastructure to create something truly impactful.

This led to the following:

Academic labs fine-tuning DeepSeek for specialized domains (biomedicine, law, and STEM education).
Start-ups building SaaS tools using DeepSeek as a backend.
Governments investigating use cases for public-sector LLM deployment.

Open source development surged globally following DeepSeek-R1’s release, with new models emerging in regions such as India, South Korea, the EU, and Latin America. On Hugging Face, one notable Indian-based project, Deepdive404/Deepseek-fork, released distilled versions of R1 across multiple parameter sizes: 1.5B, 7B, 8B, 14B, 32B, and 70B. You can find it here: https://huggingface.co/Deepdive404/Deepseek-fork Meanwhile, the original deepseek-ai/DeepSeek-R1 repository provides the core R1 and R1-Zero models at https://huggingface.co/deepseek-ai/DeepSeek-R1, along with its distilled variants.

These community-driven forks incorporate local languages, regional usage patterns, and various export formats, evidenced by quantized and optimized versions such as gghfez/DeepSeek-R1-11446-Q2_K at https://huggingface.co/gghfez/DeepSeek-R1-11446-Q2_K, which is tailored for efficient GPU inference. Within just months, Hugging Face recorded hundreds of forks and integrations inspired by DeepSeek, spotlighting localized models, quantization, and community-led improvements.

The model also inspired collaboration across borders. Researchers began publishing cross-lab benchmark studies using DeepSeek as a baseline, and community-maintained evaluation leaderboards gave the model credibility well beyond its original launch hype (https://artificialanalysis.ai/models/deepseek-r1, https://www.statista.com/statistics/1552824/deepseek-performance-of-deepseek-r1-compared-to-open-ai-by-benchmark/, and https://pubmed.ncbi.nlm.nih.gov/40267969/).

A symbol of the technological maturity of China

DeepSeek-R1 marked a milestone as one of the first open source model from China to achieve competitive performance on globally recognized benchmarks. Its strong results in reasoning-intensive tasks and rapid international adoption challenged the perception of Western dominance in frontier AI. The model’s reception highlighted the growing importance of open collaboration and cross-border evaluation in legitimizing AI innovation worldwide.

Benchmarks from independent labs confirmed that DeepSeek-R1-0528 was rivaling OpenAI’s O3 and Google’s Gemini 2.5 Pro. For an open source model, this was unprecedented.

It proved two critical things:

DeepSeek was not a one-hit wonder – it was a growing ecosystem.
Open source development could keep pace with proprietary labs when supported by community collaboration and smart engineering.

The release of DeepSeek-R1-0528 on May 28, 2025, delivered enhanced performance in mathematical reasoning, code generation, and factual retrieval. It also demonstrated reduced hallucination rates in factual question-answering benchmarks and introduced improved long-context handling, now supporting input lengths of up to 32,000 tokens – thus, making it more effective for complex, multi-turn tasks and extended document analysis, and reaffirming continuous model improvement with this new release.

The release of DeepSeek-R1 has also prompted critical questions and ongoing debates. Let’s talk about some of the controversies surrounding DeepSeek.

Controversies surrounding DeepSeek

As with any major development in AI, DeepSeek has not emerged without controversy. While it is widely celebrated for its technical sophistication, open source stance, and trailblazing approach to reasoning, its ascent has stirred debate in areas ranging from research ethics and safety to geopolitical strategy and intellectual property. This section explores the multifaceted controversies that have accompanied DeepSeek’s rise, acknowledging the tension between technological progress and responsible innovation.

Potential for bias and misuse

One area of concern stems from DeepSeek’s decision to rely on RL alone for alignment, without the conventional SFT or RLHF. Critics argue this could leave the model vulnerable to alignment drift or unpredictable behaviors over time. Others have questioned whether the open-access nature of the model, while consistent with the team’s stated goals of transparency and collaboration, increases the potential for misuse. There have also been public discussions about whether the model suppresses or softens responses to politically sensitive topics, raising questions about transparency in content moderation and training data choices.

DeepSeek’s team has published several papers, such as DeepSeekR1 (https://arxiv.org/abs/2501.12948) and news docs (https://api-docs.deepseek.com/news/news250120) that describe evaluation methodologies for community, researchers, and critics alike. DeepSeek has also released smaller variants for deployment in sensitive, more conservative, or resource-constrained settings. Some of these models are DeepSeek-R1-Zero and several distilled 1.5B–70B models; these are available via the official GitHub repositories at deepseek-ai/DeepSeek-R1(https://github.com/deepseek-ai/DeepSeek-R1) and Hugging Face deepseek-ai/DeepSeek-R1(https://huggingface.co/deepseek-ai/DeepSeek-R1).

While these measures have helped reassure many in the developer and research communities, they do not fully resolve the tension between openness and control. Underscoring the reality, balancing powerful general-purpose AI with responsible deployment remains an unresolved and evolving challenge.

DeepSeek’s radical transparency reignited the AI community’s debates on access versus control:

Was it responsible to release a powerful model with minimal guardrails?
Could it be misused to generate disinformation or harmful content?
Was community red-teaming enough to ensure safety?

Supporters argued that this openness enabled academic research into LLM alignment and trustworthiness in ways no closed model could. Critics insisted more should be done to prevent misuse. Nonetheless, DeepSeek showed that the conversation around safety didn’t have to be confined to corporate hallways. It could be open, collaborative, and community-governed.

For the curious reader, a high-level assessment of the DeepSeek-R1 model’s trustworthiness, summarizing its risk levels across dimensions such as security, privacy, hallucination, toxicity, stereotypes, ethics, performance, and robustness, along with general recommendations for mitigating these risks, is available in Trust Report: Vijil DeepSeek-R1 Trust Report (https://www.vijil.ai/blog/deepseek-trust-report).

Openness and national security concerns

One of the most polarizing aspects of DeepSeek is its radical openness. While celebrated by academic and open-source communities, the model’s full weight releases, training recipes, and RL protocols have drawn criticism from national security analysts and policy think-tanks. Some argue that the sophistication of DeepSeek-V3 and R1 rivals or exceeds many proprietary models and could be misused in ways that amplify the capabilities of bad actors.

For example, intelligence analysts have expressed concern over DeepSeek’s potential application in automated propaganda generation, cyber-intrusion planning, and synthetic misinformation campaigns, especially given its RAG and code-writing prowess. The ability to simulate reasoning, explain complex concepts, and even manipulate dialogue patterns through CoT scaffolding makes DeepSeek powerful in the wrong hands.

This debate echoes broader tensions in AI: How much openness is too much? Should models capable of autonomous reasoning be released without constraints? While DeepSeek has implemented content filters and RL-based ethical instruction, the absence of API gatekeeping (as seen in OpenAI or Anthropic) means fewer restrictions on end use. The model’s utility for good is clear, but so too is the risk of abuse.

Still, supporters argue that censoring open science in the name of security may backfire. They contend that transparency enables greater research collaboration on alignment and safety, decentralizes innovation, and builds public trust. As of mid-2025, DeepSeek’s team has maintained their position that controlled openness – combined with robust documentation and red-teaming – remains the best path forward for responsible AI.

Cultural and political perceptions

Given DeepSeek’s origins in China and bilingual training in both English and Chinese, it has become a lightning rod for geopolitical scrutiny. Some Western commentators have accused DeepSeek of being a strategic tool for soft power projection – an effort to establish AI dominance not just through capability, but through influence over global open source ecosystems. While such claims are speculative and often politically charged, they reflect real anxieties about AI leadership in a multipolar world. Open source software, once considered neutral ground, is increasingly viewed as a geopolitical asset. DeepSeek’s success in outperforming Western models on key reasoning benchmarks has intensified this narrative.

However, within the global AI community, DeepSeek’s dual-language foundation is often viewed more charitably – as a sign of pluralism and linguistic inclusivity. By publishing its models and reasoning data in multiple languages, DeepSeek has arguably done more than most to support cross-cultural research collaboration. It has also enabled more diverse participation in AI benchmarking, training, and evaluation – key ingredients for scientific equity.

That said, critics have questioned the model’s internal content moderation standards. While the model includes guardrails against sensitive or violent outputs, regional norms and political sensitivities inevitably shape training data and reward functions. This raises questions about value alignment across cultures, whose norms are encoded in the model’s reasoning scaffolds? And should open models be required to disclose ideological biases explicitly?

Academic and industrial pushback

Another point of tension arises from DeepSeek’s disruption of traditional research hierarchies. In democratizing access to world-class LLMs, DeepSeek has inadvertently challenged the academic-industrial complex that has long governed AI progress. Some large labs and commercial players have criticized DeepSeek for eroding the competitive advantage that proprietary models enjoy.

Academics have also raised concerns about reproducibility, despite DeepSeek’s transparent releases. Some worry that without shared benchmark governance or citation conventions, the proliferation of DeepSeek derivatives could muddy scientific attribution and disrupt standardization efforts. Additionally, there’s unease that DeepSeek’s rapid publication cadence may incentivize volume over peer review, leading to a release-first, refine-later culture that undermines the rigor of academic research.

Others, however, see DeepSeek as a necessary counterweight to gatekeeping. Its model weights and training protocols offer students, independent researchers, and non-Western institutions a way to participate in top-tier research on their own terms. Rather than collapsing academic standards, they argue, DeepSeek offers a complementary path that integrates open engineering with scholarly goals.

Ethical debates around RL methods

DeepSeek’s R1-Zero approach, which uses sparse binary rewards for CoT scaffolding, has ignited debate in the alignment community. While celebrated for its simplicity and elegance, critics argue that it lacks nuance in ethical edge cases. Binary correctness may not capture subtleties in moral reasoning, social fairness, or user-centric alignment.

For example, by scoring only based on structural format and correctness, R1-Zero may overlook tone, ambiguity, or unintended implications in its output. Compared to multi-signal preference modeling in Claude or the nuanced style-weighted reward systems in GPT-4o, DeepSeek’s RL strategy may appear blunt.

However, defenders highlight that R1-Zero is not a final destination but a starting point. By showing that CoT reasoning can emerge from sparse signals, DeepSeek challenges the field to rethink how alignment can be achieved efficiently and scalably. It raises a compelling question: Is a think clearly, answer correctly paradigm sufficient for safe AI interaction? Or must alignment always involve intricate preference networks?

What’s clear is that DeepSeek’s alignment choices – both in what it rewards and what it omits – will shape how future models reason. The community’s engagement with these methods, through critique or replication, will help determine their evolution.

Discourse as a feature, not a flaw

DeepSeek’s controversies are not signs of dysfunction but indicators of relevance. They reflect the model’s central position in ongoing conversations about openness, safety, equity, and governance in AI. While no model is beyond reproach, DeepSeek’s willingness to publish, iterate, and engage in public discourse positions it uniquely in the ecosystem.

Whether one sees DeepSeek as a democratizing force or a disruptive wildcard, its impact is undeniable. By provoking new questions and challenging old assumptions, DeepSeek does not merely participate in the AI debate – it helps shape it.

Summary

In this chapter, we explored the foundations of DeepSeek, a powerful open source language model series that is shaping the next wave of AI innovation. We began by understanding what DeepSeek is and why it stands out in the growing field of LLMs. We took a closer look at the technical breakthroughs that power DeepSeek, from its training methodology and architecture to its impressive performance on key benchmarks. We also examined the broader impact DeepSeek is having on the global AI ecosystem, especially in terms of open source access, innovation democratization, and regional development. Finally, the last section provided an overview of the different versions of DeepSeek, illustrating its rapid evolution and ongoing improvements in capabilities. You should now be able to clearly articulate what DeepSeek is, what makes it technically significant, how it’s influencing the global AI landscape, and how its different versions reflect strategic design choices over time. These foundational insights will serve as your lens for evaluating and working with DeepSeek in practice.

In the next chapter, we’ll go deeper into the model’s inner workings, examining how DeepSeek reasons, its approach to multimodality, and the broader implications and limitations of the system. This exploration will equip you to critically assess not just what DeepSeek can do, but how and why it does it.

Get This Book’s PDF Version and Exclusive Extras

Scan the QR code (or go to packtpub.com/unlock). Search for this book by name, confirm the edition, and then follow the steps on the page. | |
| Note: Keep your invoice handy. Purchases made directly from Packt don’t require one. |

2 Deep Dive into DeepSeek

In this chapter, we will take a comprehensive look under the hood of DeepSeek. This chapter will equip you with a practical understanding of what makes DeepSeek unique and how it fits into the evolving landscape of AI models. We will begin with DeepSeek’s technical architecture, components, and training datasets. Then, we will walk you through how these components come together to form DeepSeek’s reasoning mechanisms. You will also learn about DeepSeek’s emerging multimodal features and where DeepSeek stands in the multimodal landscape.

In this chapter, we’re going to cover the following topics:

Key architectural components of DeepSeek
Understanding the reasoning mechanics of DeepSeek
Advanced capabilities of DeepSeek

Key architectural components of DeepSeek

As you may recall from Chapter 1, DeepSeek builds on the foundational transformer architecture introduced by Vaswani et al. (2017), using its core components as a baseline for subsequent innovations. We will first discuss the key architectural components.

The key architectural components of DeepSeek include the decoder, multi-head latent attention (MLA), and mixture-of-experts (MoE), as illustrated in Figure 2.1.

Figure 2.1: DeepSeek V3/R1 architecture

The architecture and training pipeline support modularity and specialization in ways that directly enhance reasoning. Instead of relying on a single, monolithic transformer, DeepSeek routes the prompt through specialized components.

Let us talk about prompt routing first.

Prompt routing in DeepSeek

Take a look at the following figure, which illustrates how DeepSeek routes the prompt through its components.

Figure 2.2: Prompt routing in DeepSeek

Here is what happens once the prompt is input and tokenized:

Retrieval modules fetch relevant knowledge.
Next, reranker modules evaluate and refine retrieved information to prioritize the most logically pertinent content.
The lightweight decoder then incrementally builds the response, leveraging the MoE to dynamically activate sub-networks specialized in mathematical reasoning and logical inference.

For example, let’s see what happens when processing the prompt Sum of the odd numbers is even:

DeepSeek first retrieves foundational principles about odd and even sums from its knowledge base (here odd and even numbers and arithmetic properties).
The reranker elevates explanations that correctly describe why an odd plus an odd equals an even number, discarding irrelevant or misleading facts.
The decoder then generates a stepwise explanation, activating experts that focus on arithmetic properties and logical deduction, ultimately producing a coherent justification before stating the final answer.

This modular process allows DeepSeek to adapt reasoning depth and pathway dynamically, ensuring precise, context-aware, and transparent outputs even for topics requiring layered understanding. This flexible infrastructure paves the way for much more nuanced reasoning than seen in traditional autoregressive models.

Let’s now talk about the decoder module.

Decoder

With DeepSeek-R1 (https://arxiv.org/abs/2501.12948), DeepSeek employs a decoder-only autoregressive transformer that generates rationales and explanations as part of its output. Figure 2.3 shows the block diagram of the DeepSeek-R1 decoder.

Figure 2.3: DeepSeek-R1 decoder block diagram

The decoder block of DeepSeek consists of the following:

Embedding layer: This converts input tokens into vector representations.
Layers 1–3: The first three layers use the following:
- MLA for efficient attention and rotary positional embedding (RoPE) for extended sequence handling.
- Standard dense feed-forward networks (FFNs).
Layers 4–61: These layers use the following:
- Each block contains MLA as previously.
- Feed-forward is replaced with a MoE layer: a routing mechanism selects a subset of specialized FFN experts for each token, improving both capacity and efficiency.
Prediction heads: Final output goes to one or more heads to support next-token and multi-token prediction, a technique for more reliable generation.

When the tokens enter the embedding layer, they are processed through a stack of 61 transformer decoder blocks. The initial blocks are fully dense, while blocks 4–61 employ a MoE architecture, each utilizing MLA to achieve a response from an activated expert. This output is then passed through prediction heads that generate the next token(s) in an autoregressive manner.

Okay, then, now you understand how the prompt and tokens flow through the decoder. And we know now you are itching to understand what is happening behind the scenes. Let’s dive right in.

Internal mechanics of the decoder module

DeepSeek-R1’s decoder module consists of 61 transformer layers, where the first three are dense, and the remaining 58 employ a MoE structure. Within each MoE layer, there is a combination of one shared expert, which is always active, and 256 routed experts for a total of 14,906 experts distributed throughout the model.

Let’s look at all the components that work through the tokens.

Token processing and generation with router

For every input or generated token, a lightweight router evaluates which specialized sub-networks (the routed experts) should process that token. Specifically, the router dynamically chooses 8 out of 256 routed experts in each layer, together with the always-on shared expert. This selection isn’t random; it’s determined for each token by considering the token’s context and matching its features to the affinity or specialization of each expert. Which experts are activated depends on the nature of the prompt and the type of token being generated. For example, when generating a rationale that involves a mathematical argument or code, the model will preferentially activate those experts that are specialized in mathematical reasoning or programming. Similarly, for reasoning or refusal behaviors, analyses using functional token resonance imaging (fTRI) have shown that certain identifiable experts are consistently activated, depending on the category of response required.

Finally, the retrieval candidates are generated.

In DeepSeek-R1, reranking plays a central role in transforming retrieval candidates into high-quality, human-readable justifications. Typically, the model follows the process as shown in the following figure.

Figure 2.4: Cross-attentive reranking module

Here is how it works for your prompt:

Top-k retrieval: Given a prompt (e.g., a user query or reasoning task), an initial retriever selects the k most relevant candidate passages or answers using fast, often vector-based similarity (such as cosine or dot-product in embedding space).
Cross-attentive reranking: These candidates, their embeddings, and the original prompt are input into a specialized transformer block. This module uses cross-attention – attending jointly to both the query and each candidate – to compute refined similarity scores and produce reranked outputs that inform the next generation step.
Final justification generation: The aligned, reranked candidates (plus intermediate scoring vectors) are then processed in the decoder, which generates a rationale or explanation as part of its output.

Consider, for example, the following prompt:


"What is the capital of France?"

Assuming k= 3, the top 3 retrieved passages are as follows:


["Paris is a major European city.", "Berlin is the capital of Germany.", "Paris has many museums."]

The cross-attentive reranker, via attention to both query and answer, scores Paris as a major European city, the highest, since Paris and capital align strongly.

After top-k retrieval and cross-attentive ranking, the model consumes the aligned embeddings and token sequences to generate human-readable justifications through its standard generation process.

Consequently, as DeepSeek-R1 generates tokens for a rationale or explanation step by step, the exact subset of experts activated can shift dynamically from one token to the next.

For each token, the following happens:

The router takes the hidden state corresponding to that token and rapidly computes similarity scores with each expert’s centroid representation.
It then selects the top-k (say k= 8) experts with the highest scores and routes that token to be processed only by those experts, alongside the always-active shared expert, for that layer.
Across a sequence of tokens and diverse prompt types, a wide variety of expert subsets become active, enabling DeepSeek-R1 to marshal highly specialized capabilities, such as complex logical reasoning or domain-specific knowledge, while maintaining computational efficiency by not activating all experts at once.

This MoE-based architecture ensures that DeepSeek-R1 can construct detailed, step-by-step explanations or rationales.

Each token in the generated output can benefit from both the context (previous tokens) and the unique expertise of a dynamically selected set of expert networks at every layer, allowing for nuanced, efficient, and highly specialized reasoning throughout the response.

Autoregressive transformer

DeepSeek-R1 uses the autoregressive transformer, where the model produces one token at a time, with each new token being generated based on all previously generated tokens. This design choice is called autoregressive because, at every generation step, the model’s next token prediction is conditioned explicitly on the entire sequence constructed so far, not on the future context, enabling the model to build coherent, contextually relevant outputs step by step.

Once the initial input or prompt is set, the following happens:

The sequence is processed through attentive and feed-forward components in the transformer stack.
The model then predicts the most likely next token, appends it to the sequence, and repeats the process for the following token – always conditioning exclusively on what has already been generated:
- Autoregressive generation works by generating only a single token per forward pass, not a batch or series of tokens simultaneously.
- However, through repeated forward passes, the model constructs a growing sequence, one token at a time, until a stopping condition is met (such as an end-of-sequence token).

This stepwise process is crucial for structured tasks such as logical explanations or code, where the meaning of each next token often tightly depends on the context set by all previous outputs.

As for the possible tokens generated at each step, the model considers its entire vocabulary, potentially tens of thousands of tokens.
For every new position, the decoder outputs a probability distribution over the full vocabulary. Probable candidates include syntactically and semantically plausible tokens based on the current context. For example, after the phrase The answer is, likely next tokens might be numbers, a colon, or words expressing a solution. If generating source code, probable tokens could include keywords such as def, data types, or variable names, guided by the prompt and semantic coherence.

Multi-head attention

DeepSeek-R1 also uses a multi-head self-attention design, specifically an optimized variant called MLA. This allows the model to analyze relationships between tokens effectively. DeepSeek-R1’s multi-head attention does not mean each head focuses on a single token. Instead, multiple heads operate in parallel, each learning to attend to different parts of the input sequence simultaneously.

Although the exact number of heads in DeepSeek-R1’s transformer layers isn’t explicitly stated in available sources, typical large transformer models use dozens of heads (e.g., 64 or more), balancing the total available heads with the number of tokens. It is indeed possible to have a trade-off – more heads can provide finer-grained attention at the cost of more computation, while fewer heads reduce complexity but may capture less diverse relationships.

The core function of multi-head attention is to capture different types of relationships between tokens in the input sequence. Each head computes queries, keys, and values (Q, K, and V) that interact to generate attention scores, which highlight how much one token should consider others when forming its internal representation.

Queries: In a transformer, the query is a vector representing what information or context the model is looking for in the input data.
Keys: The key is a vector representing the characteristics or features of each element in the input that the query is compared against to find relevant information.
Values: The value is a vector containing the content or information associated with each key, which is weighted and aggregated based on similarity to the query to produce the output.
More information can be found at https://en.wikipedia.org/wiki/Attention_(machine_learning).

Through this, attention identifies pairwise relationships not just between the existing input tokens but also takes into account previously generated tokens during autoregressive generation, since the model attends over the entire prefix.

To better handle the large sequence lengths DeepSeek-R1 supports (up to 128K tokens), it employs RoPE. RoPE injects positional information so the model knows the relative positions of tokens, which is crucial for meaningful attention over long contexts.

Suppose the input tokens so far include the phrase The answer is during autoregressive generation. Multi-head attention enables some heads to focus on the phrase The answer while others attend to the latest token, is, combining these perspectives to predict plausible next tokens, such as numbers or words, forming a solution. As the model generates new tokens, these tokens get appended to the sequence, and the attention mechanism extends over this growing sequence, dynamically adapting which tokens are most relevant, all processed in parallel by multiple heads.

Moreover, DeepSeek-R1 integrates an efficiency optimization with MLA. Here, instead of separately storing full key and value matrices for each head, the model factorizes them into latent vectors, reducing memory consumption without sacrificing the quality of attention.

Layer normalization

Layer normalization is applied to the hidden representations (also called activations) of tokens at specific points within each transformer layer to improve training stability, convergence speed, and overall model performance. More precisely, the normalization operates on the token embedding vectors; these vectors represent the tokens’ current states as they are processed through the network.

Activations in an LLM transformer refer to the specific scalar values in the hidden states of the model during its layer-by-layer computations; these activations represent the transformed input features at various stages and play a critical role in how the model processes and generates language.

The key transformations where layer normalization is applied include critical components inside each transformer layer: the multi-head self-attention module and the FFN (also called the position-wise network). Here is how this works:

Before entering the attention mechanism, the token embeddings are normalized by computing their mean and variance across the embedding dimensions and then scaling and shifting these values to have consistent statistical properties. This pre-attention normalization ensures that the inputs to the attention computations have stable distributions, which helps the model learn more effectively.
After the attention output is added (via residual connections) back to the tokens, another normalization step is often applied to keep the activations well-conditioned as they move into the FFN.
Similarly, pre- and post-normalization also occur around the feed-forward layers, ensuring that the transformations – which involve linear projection, nonlinear activations (such as GELU), and another linear projection – maintain stable activation statistics.

For example, consider the Calculate the sum prompt tokens:

As these tokens enter a transformer layer, their embeddings are normalized before self-attention calculates how much each token should attend to itself and others in the prompt.
The attention module produces updated embeddings, after which layer normalization again stabilizes these new vectors before passing them into the FFN.
This FFN then applies linear weights, a nonlinearity, and another linear transformation – key transformations where normalization supports smooth gradient flow and mitigates internal covariate shift.
Following this, a final normalization step prepares the output embeddings to enter the next transformer layer or the output head.
Layer normalization in DeepSeek-R1 normalizes the token embeddings before and after the key internal transformations – multi-head self-attention and feedforward layers – within each transformer block. This layered normalization keeps the token representations consistent and stable throughout the stepwise, autoregressive generation process, thereby enhancing training reliability and model accuracy over very long sequences.

The use of MoE and attention optimizations such as grouped-query attention (GQA) enables the model to scale without excessive resource demands.

GQA is an optimized attention mechanism in transformer models that divides query heads into groups, where each group shares a single key and value projection, balancing computational efficiency and model performance by interpolating between multi-head attention and multi-query attention.

Through its structured reasoning capabilities, DeepSeek-R1 can not only generate answers but also produce explanations, verifications, and arguments, critical abilities for domains such as theorem proving, law, and structured reasoning. The decoder module in DeepSeek-R1 refers to a stack of transformer decoder blocks (61 layers in total) that form the core of its autoregressive language modeling capabilities.

We will now show you how this works for our sample prompt.

Working example

Suppose the user inputs the following:


Prove that the sum of two even numbers is even.

This prompt goes through the following steps:

Tokenization: The input is split into tokens and embedded as vectors for processing. Typical tokens might be (in plain text) the following:

Prove, that, the, sum, of, two, even, numbers, is, even, .

Sequential processing: The embedded tokens flow through the decoder stack. In each layer of DeepSeek-R1’s decoder, self-attention enables each token to consider earlier tokens in the sequence using key (K), query (Q), and value (V) representations. For example, suppose token t₂ wants to attend to t₁ and t₀; it generates Q₂, compares it to K₁ and K₀, and uses the resulting attention scores to weight V₁ and V₀. This operation allows t₂ to aggregate relevant contextual information from preceding tokens.
Causal masking enforces the autoregressive property by blocking each token from attending to future tokens, ensuring that predictions only depend on previously seen inputs and preserving the left-to-right generation structure. As output is generated, the model selects the most relevant prior token activations according to the attention scores, building understanding with each forward step.

It is important to distinguish that this self-attention operates within a single sequence, the tokens of the input prompt or generated response, allowing the model to capture internal dependencies. In contrast, cross-attention is used in separate, specialized modules (such as cross-attentive reranking), where the model allows tokens from one sequence (e.g., the prompt) to attend to tokens from another sequence (e.g., retrieved candidates) to refine similarity scores before entering the decoder. Thus, the decoder itself relies on self-attention for in-sequence reasoning, while cross-attention is applied earlier during retrieval and ranking.

Autoregressive token generation: After each pass through the decoder, the model predicts the next token by scoring multiple possible candidates. For example, after generating the prefix <think> Let the two even numbers be 2a and 2b, where a and b are integers, the model considers several top candidates for the next token. Suppose the top-k options are as follows:
- Their (score: 0.34)
- The (score: 0.21)
- Sum (score: 0.18)
- Difference (score: 0.16)
- Product (score: 0.11)

Using cross ranking, Their receives the highest score, so it is selected as the next token:


<think> Let the two even numbers be 2a and 2b, where a and b are integers.

<think> Their sum is 2a + 2b = 2(a + b)

<answer> Since a + b is an integer, 2(a + b) is even.

<answer> Therefore, the sum of two even numbers is even.

At each generation step, the decoder weighs its top candidates and picks the token with the highest score, constructing the output token by token, guided by context and autoregressive constraints.

Phew! That was quite something. Well, there is more. Let’s now move to the next component, the MoE.

Mixture of experts

As outlined in Chapter 1, MoE routing improved parameter efficiency by selectively activating specialized submodules for each token. Throughout all stages, advancements in projection and pooling methods contributed to stabilizing embedding norms, which enhanced retrieval performance. Furthermore, metadata-aware cross-attention allowed DeepSeek’s encoders to incorporate contextual document features such as titles and categories.

In DeepSeek-R1, the MoE architecture selectively activates specialized expert sub-networks dynamically for each token during generation, enabling efficient use of vast model capacity. When processing a prompt, the model’s router evaluates the token’s contextual embedding and compares it with learned centroids representing each expert’s specialization. Based on this similarity, it activates a small subset of experts, typically 8 out of 256 routed experts plus one shared expert per MoE layer, tailoring computation to the token’s semantic and functional needs.

If you are wondering how this expert selection happens, keep reading.

How are experts initialized and configured?

DeepSeek-R1’s architecture defines hundreds of routed experts (e.g., a noun expert, number expert, web expert, etc.) for each MoE layer, distinguished by learnable centroid vectors. These centroids begin as random vectors and, throughout pretraining, are shaped to specialize in distinct linguistic or semantic domains – such as recognizing numbers, identifying named entities, or interpreting reasoning cues – depending on patterns encountered in the training data. Experts such as the number expert or web expert are not hardcoded; instead, these specializations emerge as different experts are repeatedly activated for certain token types or contexts during training.

For example, consider the following partial prompt token sequence: Prove that the sum of two even numbers is even. During autoregressive generation, when the model encounters tokens related to mathematical reasoning (such as even numbers, sum, or prove), experts specialized in mathematical operations and stepwise logic are strongly activated.

Figure 2.5: MoE routing in DeepSeek-R1

The router selects those experts whose specialization best matches the role of each token: mathematical experts handle numeric manipulations and proof steps, while reasoning experts are chosen for tokens that demand justification or logical explanation.

If the prompt shifts to programming, such as outputting def or specific variable names, the router would preferentially activate experts trained in code generation. In this way, context-sensitive routing ensures that the model leverages the knowledge of those experts most relevant to the current task.

How does gating compute which experts to activate?

When a prompt such as Prove that the sum of two even numbers is even is tokenized and fed into the model, each token’s embedding (its hidden state) is processed by the MoE gating network at every MoE layer it passes through. For each token at each layer, the following happens:

The token’s embedding is compared to every expert’s centroid vector (pre-learned) using a dot product, producing affinity scores that measure how closely the token’s features align with each expert’s specialization.
These affinity scores pass through a normalized gating function. DeepSeek-R1 typically uses a sigmoid-based mechanism with potential bias adjustments for balancing expert usage, rather than a simple softmax.
The gating function selects the top-K experts (e.g., K=8) with the highest affinity scores for that token, plus one shared expert that is always activated.
Only the selected experts process the token; their outputs are weighted by the gating scores and combined to form the output representation for that token at the layer.

How are experts activated?

We will now go back to our sample prompt to see how the experts get activated.

Consider the token even within the prompt Prove that the sum of two even numbers is even. At an MoE layer, the router compares the embedding of even to the centroids of all experts:

If even aligns closely with the mathematical reasoning expert’s centroid, that expert’s affinity score will be high, making it one of the top-K experts selected.
Tokens such as sum and prove, associated with logical reasoning and proof construction, may activate reasoning or proof-logic experts.
If the prompt context included programming syntax or variable names, programming experts would be preferentially routed instead, but for this mathematical proof, such experts are less relevant.
Even connective tokens such as that, of, or two undergo this gating independently, leading to a dynamic, token-specific expert routing pathway throughout the prompt.

This token-by-token gating mechanism generates a unique activation pathway through the expert network tailored to each input and generated token. It efficiently leverages specialized expert knowledge exactly when needed, minimizing computational overhead. This dynamic routing also supports DeepSeek-R1’s ability to produce clear, stepwise explanations and reasoned outputs by activating the most relevant experts precisely at the right time.

On the other hand, the traditional state-of-the-art (SOTA) LLMs (such as GPT-4o or Claude) mostly use dense architectures, meaning all parameters are used on every input, leading to high resource use. DeepSeek achieves SOTA-level performance but with much lower effective active parameter count, making it much more efficient. Additionally, metadata-aware cross-attention layers incorporate contextual features such as document titles or categories, enriching the representations that influence expert routing and enhancing retrieval and relevance. Instead of routing input through every neuron in every layer, each layer contains multiple expert subnetworks, but only a few specialists (e.g., 8 out of 64 experts) are activated per token. This drastically lowers computational load while maintaining capacity.

The following equation shows how the experts get activated:

where gating functions select which experts to use ( is the total number of experts; often only the top two are activated).

Next up, we will move our focus to MLA.

Multi-head latent attention: Memory-efficient attention for long contexts

MLA was first operationalized in DeepSeek-V2 and later refined in DeepSeek-V3 to reduce memory usage during inference without compromising output quality.

Standard transformers (and many SOTA models such as LlaMa 3 and GPT-4o) use multi-head attention with full key-value (KV) caches for each token, leading to high memory requirements for long contexts. DeepSeek-V2, V3, and R1 introduce MLA, compressing the KV cache into a compact latent for each head, reducing inference memory by over 90% while preserving or improving quality.

MLA replaces the standard multi-head attention mechanism by introducing a form of low-rank compression to the KV cache. Instead of storing a large set of key and value memory tensors for every token during inference, leading to immense memory usage, especially with long sequences, MLA absorbs these representations into a smaller set of learned latent vectors. At each layer, attention computations are performed not directly over the full cache, but through these compressed latents, which maintain sufficient contextual information for high-quality generation.

Additional enhancements include selective application of rotary positional embeddings and advanced weight absorption techniques, allowing this mechanism to match or exceed the model’s original reasoning quality while often reducing inference cache size by over 90%. This innovation is a key reason behind DeepSeek-R1’s ability to efficiently handle long-context tasks and to offer economical large language model inference at scale.

Let’s take a look at how this works.

In MLA, instead of storing keys/values (where N is sequence length), KV for each head is projected into a much smaller latent space:

During attention, queries attend to this latent rather than the full sequence – a major speed and memory gain.

LLaMa 3 and many GPT models use GQA to reduce KV cache size, but MLA empirically delivers better modeling performance and higher efficiency, as attested by benchmarks and publications.

For the prompt Prove that the sum of two even numbers is even, MLA enhances the model’s understanding by incorporating external context, such as document titles or category information, into the attention mechanism. This external context provides additional signals that help the model better interpret and specialize its response based on relevant metadata. Within one attention head, the key and value vectors are first computed from the input token embeddings. Each of these vectors, originally high-dimensional (e.g., 1,024 dimensions), is then projected into a smaller latent space (for example, 64 dimensions) using learned projection matrices. This dimensionality reduction allows for more efficient computation and focuses attention on the most relevant features. The projected keys represent the contextual features used to match against the query (Q), while the projected values carry the enriched information that is aggregated. MLA achieves this by attending to both the token sequence and the injected metadata simultaneously across multiple latent subspaces, allowing the model to weigh and integrate relevant external knowledge dynamically as it generates the proof. This process enables more accurate, context-aware reasoning for claims like the sum of two even numbers being even.

There are other key aspects that make DeepSeek one of the most competitive models. Hang in there.

DeepSeek features for efficient and context-aware responses

Apart from the architectural innovations mentioned, DeepSeek employed several other techniques to ensure efficiency and context-aware responses. Let’s talk about them.

FP8 training and precision control

DeepSeek-R1 is among the first LLMs to implement a true, large-scale FP8 (8-bit floating point) mixed-precision strategy for both pretraining and inference. FP8 is one of the key factors behind DeepSeek’s breakthrough in speed and memory efficiency over earlier formats such as FP16 or BF16. However, the successful deployment of FP8 required a deliberately engineered approach to address the well-known risks of numerical instability and degraded convergence that commonly arise with such low-precision arithmetic.

The careful design lies in DeepSeek’s mixed-precision training framework, where the majority of computation-heavy operations, such as linear transformations (matrix multiplications, or general matrix multiply (GEMM) operations) involved in both forward and backward passes, are executed in FP8 precision. For each linear operator (whether for the forward pass, gradient backpropagation, or weight update), model weights and activations are quantized and processed in FP8, doubling computational throughput relative to BF16 and slashing memory usage by half.

Critically, to preserve accuracy, the actual accumulation of products (sums) is performed in FP32. Activations and gradients are cached in BF16 for memory efficiency and stability.

What distinguishes DeepSeek-R1 is its selective application of FP8. Through a detailed empirical study, the team identified specific modules where further reduction in precision would undermine model stability or learning dynamics. For example, they retain higher (BF16 or FP32) precision for embedding layers, the sequence output head, MoE gating modules, layer normalization, and the attention operators, all of which are sensitive to quantization noise and demand more numerical robustness. Only the dense core (GEMM) and certain memory-intensive pathways are safely quantized to FP8. This selective approach ensures that while the vast bulk of the network benefits from the efficiency of FP8 (and thus enables scaling to much larger models within the same compute envelope), key computations always retain the precision needed for model convergence and high-fidelity output. The optimizer states, master weights, and weight gradients are all stored in BF16 or FP32, with sophisticated sharding and parallelism techniques (such as DualPipe pipeline parallelism) further reducing memory and communication overhead.

During DeepSeek-R1’s pretraining, matrix multiplications for token transformations and intermediate computations, such as those in mathematical prompts, for example, Prove that the sum of two even numbers is even, are performed using FP8 precision to maximize computational efficiency. Meanwhile, critical operations, including the MoE gating logic, layer normalization, and final output generation, utilize higher precision to ensure accuracy and stability as the sequence lengthens. This targeted, module-specific precision approach balances efficiency with reliability, enabling DeepSeek-R1 to train multi-trillion-parameter models at scale in a single run without compromising output quality or training stability. This explanation highlights the practical trade-offs and design decisions important for understanding how the model achieves efficient and stable training at unprecedented scale.

Multi-token prediction for strategic decoding

Most transformer language models are trained as autoregressive models, maximizing the likelihood of the next token given the sequence so far:

This means the model learns to predict just one token at a time, always conditioning on all previous tokens.

At inference time, the model generates text one token after another (token-by-token), sampling or greedily picking the most likely next token.

DeepSeek’s V3 model introduces multi-token prediction (MTP), inspired by speculative decoding. Here, the model learns to predict not just the next token, but several future tokens in parallel. Here is the MTP joint likelihood objective DeepSeek employs:

Instead of focusing only on the immediate next token, MTP jointly optimizes for k tokens ahead.

This aligns the training objective with more efficient decoding and better captures long-range dependencies.

Some of the benefits of MTP include the following:

Parallelizes training: By predicting multiple tokens together, training can be faster and more efficient.
Enhances context modeling: The model learns to anticipate broader patterns and plan ahead, encouraging globally coherent and consistent reasoning.
Accelerates inference: At generation time, the model can validate or propose multiple tokens per forward pass, drastically increasing throughput.

DeepSeek-R1’s group relative policy optimization (GRPO) represents a major step beyond standard RLHF by enabling much more precise and context-aware reward assignment during fine-tuning. Let’s explore this in detail.

GRPO and advanced RLHF

In the custom GRPO system of DeepSeek, during training, the policy model generates multiple responses per prompt – typically, a fixed number such as 4 to 8 samples, though this number can vary depending on the training phase or prompt complexity. Each sampled output is assigned a binary reward (acceptable or not) based on strict rule-based criteria. Even if some responses in the group are incorrect and receive zero reward, they are still included in the group to compute an average reward. This means that the presence of incorrect outputs lowers the group’s average reward baseline, effectively penalizing poorer quality responses relative to better ones. In practice, a scalar reward is computed from rule-based checks (e.g., correctness and a simple formatting constraint), which often reduces to an effectively binary signal; GRPO uses this scalar after centering by the group mean to push up higher-scoring samples and push down lower-scoring ones.

The model’s advantage for each response is then calculated by comparing its binary reward against this group average (and normalizing by the group’s standard deviation), guiding policy updates toward responses that outperform the group mean. This relative scoring mechanism encourages consistent improvement while accounting for varied output quality across all samples for the same prompt.

With GRPO, when the prompt Prove that the sum of two even numbers is even is presented, the model generates a group of candidate responses in parallel. For example, suppose the candidates and their rewards are as follows:

Candidate A: Let the two even numbers be 2a and 2b. Their sum is 2(a + b), which is even. (Reward: 0.85).
Candidate B: Two even numbers added together result in an even number because even plus even equals even. (Reward: 0.75).
Candidate C: Since even numbers are multiples of 2, their sum is divisible by 2, so it's even. (Reward: 0.80).

GRPO first calculates the mean reward of the group:

Then, for each candidate, it computes the advantage by subtracting this mean reward from its reward, and optionally normalizes by the group’s standard deviation to balance scale differences, as shown here:

Advantage A = 0.85 - 0.80 = +0.05
Advantage B = 0.75 - 0.80 = -0.05
Advantage C = 0.80 - 0.80 = 0.00

By optimizing based on these relative advantages rather than absolute rewards alone, the model learns to prefer responses that stand out positively within the realistic set of candidates, improving overall generation quality and relevance for complex prompts like this proof.

Rewards in DeepSeek-R1 can be based on multiple criteria: logical correctness, coherence, faithfulness to the prompt, completeness, or even adherence to mathematical principles or style – in short, any property that can be evaluated by a rule, heuristics, or through outcome-based scoring. For instance, when fine-tuning a reasoning task, rewards may be higher for outputs that provide a logical chain of thought (CoT) or precise intermediate steps, not merely for producing the final correct answer.

By using each group’s average reward as a context-specific baseline, GRPO reduces reward hacking and noise, encouraging the policy to favor relatively better responses in each situation. DeepSeek-R1 also adds a KL-divergence penalty (https://hanj.cs.illinois.edu/cs412/bk3/KL-divergence.pdf) to the loss to stabilize updates and maintain desired behavior. This granular, group-relative approach enables precise control over learning, improving clarity, factual accuracy, and alignment with user intent – especially for complex, multi-step reasoning tasks. GRPO underpins DeepSeek-R1’s strong performance in producing detailed, interpretable, and user-aligned outputs.

Chain-of-thought reasoning: CoT and long CoT

DeepSeek’s training, especially with specialized prompts (<think>, <answer>), is optimized for stepwise, interpretable reasoning. It is particularly strong at long CoT tasks – demonstrated by winning in the 2025 US Math Olympiad LLM challenges.

Consider this prompt, for example:


Explain why the sum of two odd numbers is even?

Here is DeepSeek’s response:


<think> Let the numbers be 2a+12a+1 and 2b+12b+1\. 

<think> Their sum is 2a+1+2b+1=2(a+b+1)2a+1+2b+1=2(a+b+1), which is even.

Other SOTA LLMs can produce CoT outputs, but DeepSeek excels due to its architectural and training bias toward structured explanations (more on this in the next section and Chapter 4), often outperforming in competitive benchmarks.

Choice of datasets

Recall from Chapter 1 that DeepSeek’s dataset curation spans a rich and diverse set of sources, thoughtfully selected to cover reasoning, coding, science, and general world knowledge. For programming tasks, DeepSeek leverages large-scale, open source code from repositories such as the official GitHub dataset, as well as curated forums and libraries found in data collections such as The Stack (https://huggingface.co/datasets/bigcode/the-stack) and CodeContests (https://huggingface.co/datasets/deepmind/code_contests). For mathematical and scientific reasoning, the model is trained on benchmark math datasets (such as GSM8K and MATH), scientific paper corpora (such as arXiv and PubMed Central), and collections of logic puzzles and textbook problems.

Crucially, DeepSeek’s training diets include explicit CoT datasets, notably using CoT-annotated examples from resources such as the CoT Collection (https://github.com/kaistAI/CoT-Collection), MathQA (https://math-qa.github.io/), and curated human rationales. Instructional and narrative diversity is provided via synthetic datasets inspired by prompts from Flan Collection (https://github.com/google-research/FLAN), Alpaca (https://github.com/gururise/AlpacaDataCleaned)), and datasets released for self-instruction or dialogue supervision. For robust multilingual and domain-general competence, DeepSeek includes substantial portions of Wikipedia (https://www.wikipedia.org/), Common Crawl – filtered for quality – (https://commoncrawl.org/), Project Gutenberg (https://www.gutenberg.org/), and domain-specific corpora ranging from legal documents (e.g., Pile-CC’s legal section at https://pile.eleuther.ai/) to medical literature, such as PMC-PubMed (https://pmc.ncbi.nlm.nih.gov/) and MedQA (https://huggingface.co/datasets/bigbio/med_qa).

Test-time scaling

DeepSeek-R1 applies test-time scaling by allocating extra computational resources during inference, enabling step-by-step deep reasoning over input prompts to produce higher-quality outputs. Instead of generating a single direct answer, the model performs chains of inference passes, similar to The Human System 2 (deliberate, effortful reasoning) logical reasoning, iteratively evaluating possible solutions before finalizing a response. This approach is supported by its MoE architecture, which routes each token to multiple experts in parallel and leverages advanced hardware acceleration to deliver rapid multi-expert evaluation.

At runtime, the model’s output quality improves as more compute is devoted to inference, especially for complex tasks in math, coding, and logical analysis. Real-time answers from DeepSeek-R1, which scales up to thousands of tokens per second on modern GPU clusters, require tightly connected high-performance infrastructure to maximize token throughput and expert communication. This scaling mechanism effectively extends the model’s ability to think longer, check and refine its reasoning, and boost benchmark performance on tasks demanding extended computation and precise logic.

Together, this broad, carefully curated mix equips DeepSeek with the breadth and depth needed to reason, code, answer multi-step questions, and generalize across languages and domains. For more detailed breakdowns or pointers to documentation of each dataset, you may refer to the associated repositories or academic dataset leaderboards.

Whoa! That’s too much to digest. We suggest you take a breather and come back to see how all these concepts come together to form the reasoning mechanics of DeepSeek.

Understanding the reasoning mechanics of DeepSeek

What makes DeepSeek more than just another transformer is its focus on reasoning. Reasoning in language models refers to the ability to solve multi-step problems, understand cause and effect, perform symbolic manipulation, or derive answers based on implicit context.

Rather than offering isolated performance gains, DeepSeek’s architectural innovations organize internal representations, maintain extended context within a prompt, and structure its responses in a more deliberate and context-aware manner – leading to faster, more reliable reasoning across diverse domains.

DeepSeek’s approach to prompt engineering, training design, and model alignment underlines a deliberate shift toward explicit, structured reasoning. Rather than relying solely on the model to infer a logical chain behind the scenes, DeepSeek encourages intermediate reasoning steps by introducing structured prompting formats, such as explicit <think> and <answer> tokens or instruction-tuned templates that guide the model to show its work. This is not the same as using a system prompt in the conversational sense. This approach relies on carefully designed, task-specific prompt formats that are explicitly embedded into the model’s training and fine-tuning datasets.

During data preparation, examples are structured with dedicated tokens such as <think> to mark intermediate reasoning steps and <answer> to signal the final conclusion. These tokens serve as clear semantic cues that instruct the model to generate detailed, stepwise explanations rather than immediate final answers.

This involves the following steps:

The training corpus includes large quantities of annotated data where human-written or model-augmented rationales accompany final responses, broken down into logical inference steps.
The model is then trained not only to predict the correct final token sequence but also to reproduce the intermediate reasoning steps within the output sequence.
Loss functions are applied over the entire token sequence, including these reasoning steps, to encourage faithful reproduction of structured thought.
Moreover, during fine-tuning and reinforcement learning phases (such as using GRPO), evaluation metrics explicitly reward outputs for clarity, coherence, factual alignment, and completeness of reasoning, not just answer correctness. This encourages the model to value interpretability and internal consistency of its chain of thought, reinforcing behaviors that make the reasoning process transparent and verifiable.

As a result, DeepSeek’s training pipeline instills an operational paradigm where structured prompts with embedded reasoning markers become integral to both the learning objective and output evaluation. This end-to-end design ensures that the model reliably shows its work through explicit intermediate steps, producing outputs that users can inspect for logical validity and trustworthiness rather than opaque final answers alone.

So, how does DeepSeek think? Let’s zoom in right away.

DeepSeek’s thinking

One notable aspect of DeepSeek’s development was observed during the training of DeepSeek-R1-Zero, a variant trained solely with RL and without any supervised fine-tuning data. During this process, the model began to demonstrate emergent reasoning abilities that were not explicitly programmed, highlighting new insights into how language models can develop reasoning skills through RL alone.

The training setup for R1-Zero was straightforward. Engineers at DeepSeek designed a fixed prompt structure during training and alignment that encourages the model to reason through problems explicitly before providing an answer.

There are three key factors at play here: the think and answer blocks, GRPO, and RLHF on training samples. Let’s talk about them.

The think and answer blocks

While the exact system prompt remains proprietary, available tokens and training artifacts suggest that prompts were consistently framed to include a <think> token preceding an <answer> token, guiding the model to generate structured, intermediate reasoning steps before arriving at a final response. This explicit delineation between thinking and answering – whether embedded directly in system instructions or demonstrated across supervised examples – acts like a latent system-level directive, conditioning the model to treat reasoning as an integral first-class step rather than an optional behavior.

DeepSeek models (particularly in the R1 series) show strong performance in multi-step tasks such as mathematical problem solving and legal argumentation, where breaking down the question into logical steps is essential for accuracy and transparency. Each model response was expected to follow a two-stage structure, consisting of a <think> block that contains intermediate reasoning steps and an <answer> block with the final conclusion. This format, introduced during training and evaluation, reinforces explicit reasoning before decision-making, especially for tasks that demand step-by-step logic. The boundary between the two is defined by special tokens, such as <think> and <answer>, which act as delimiter cues for both formatting and behavior.

This explicit prompt structure helps make the model’s reasoning transparent, verifiable, and extensible, particularly for complex domains such as mathematics, law, and multi-turn decision tasks. We will discuss prompt formats in detail in Chapter 3.

Consider the following prompt, for example:


"Is the sum of two even numbers always even? Explain why."

The model responds as follows:


<think>

Let the two even numbers be represented as 2a and 2b, where a and b are integers.

Their sum is 2a + 2b = 2(a + b), which is divisible by 2 and therefore even.

</think>

<answer>

Yes, the sum of two even numbers is always even.

</answer>

Let’s look at the structure of the response generated by the model. The <think> section is where the model is encouraged to lay out the step-by-step reasoning, while the <answer> section delivers the final response succinctly.

Now, the GRPO mechanism comes into play for the model response, which we discuss next.

Evaluating response quality with GRPO

During training and alignment with GRPO, the quality of the <think> section is evaluated through a group-relative reward mechanism that fosters granular control over intermediate reasoning steps. Instead of assigning an absolute reward to each model output, GRPO compares multiple candidate responses generated for the same prompt as a group:

For each response, including its <think> intermediate reasoning block, the reward assigned is relative to the performance of other candidates in that group. This relative advantage is computed by subtracting the group’s average reward and normalizing by its standard deviation. This setup allows explicit evaluation of the <think> section quality based on criteria such as logical coherence, consistency, factual correctness, formatting, and interpretability, independently of just the final answer correctness.
By rewarding outputs that contain clear, stepwise, and well-structured reasoning, the model learns not merely to guess answers but to construct them logically and transparently. Concretely, during GRPO fine-tuning, the model generates multiple candidate explanations per prompt.
A scoring function, typically a weighted combination of heuristics and learned reward models, assigns scores for attributes such as the clarity and correctness of the <think> block.
The learned reward models often include neural rankers or reward predictors trained on human preference data to evaluate reasoning quality, factual accuracy, and coherence within the generated explanations.

GRPO uses the scores (outlined in the preceding list) to compute group-relative advantages, which guide policy updates, aligning the model toward producing better intermediate reasoning aligned explicitly with human-aligned quality standards.

Here is how the reward functions work:

The accuracy reward remained a binary signal, assigned as 1 if the final answer was correct and properly formatted (e.g., including <think> and <answer> blocks), and 0 otherwise. This evaluation leveraged automatic checks and, where necessary, manual validation against ground-truth answers as established in earlier stages.
The consistency reward was designed to quantify how uniformly the model maintained a single language style and logical coherence throughout the reasoning steps, preventing mixing of languages or incoherent phrasing within a single response. Conceptually, this reward can be expressed as a scalar function.

The consistency reward, denoted as , is defined as a function applied to the model’s response:

The value of lies within a certain range (e.g., from 0 up to a maximum value of 10), where the function evaluates the internal linguistic uniformity and coherence of the response text. For instance, might perform the following:

Calculate the proportion of tokens or sentences in the dominant language compared to any other languages present.
Penalize abrupt changes in stylistic tone or phrasing.
Apply coherence metrics across the chain of reasoning to ensure logical flow.

The combined reward that guides policy optimization in RL is then computed as the sum of the accuracy reward and a weighted consistency reward:

Here, is a binary or scalar reward measuring the correctness of the final response, and λ is a weighting coefficient that balances the effect of the consistency reward relative to the accuracy reward.

This combined reward is used within the GRPO framework. GRPO calculates relative advantages by comparing the total reward of each sampled response to the group average reward, encouraging the model to generate outputs that are not only accurate and well-structured but also linguistically consistent and logically coherent throughout their reasoning process.

While this sampling and comparison process increases the compute workload during training – since several candidate completions must be generated and evaluated per prompt – the impact is confined strictly to the training stage and does not affect inference speed once deployment begins. During inference (serving real user requests), only a single model rollout per prompt is needed, just like any conventional LLM. As a result, DeepSeek achieves strong reasoning alignment and sample efficiency during training, but delivers responses at standard, highly efficient inference speeds in production, without incurring overhead from GRPO’s training-time multi-sampling methods. GRPO did not require auxiliary value networks or rollouts, making it fast, stable, and accessible.

Rule-based RLHF

The range of example questions used during training and evaluation spanned a variety of domains, from simple arithmetic and symbolic logic to code generation and physics, demonstrating the model’s versatility across different reasoning challenges.

DeepSeek’s rule-based RLHF signals were applied in a relatively straightforward manner, where each response or reasoning step was marked as acceptable or not based on fixed, predefined logical criteria rather than learned via a complex reward model. According to technical descriptions, this process was not entirely manual annotation in the traditional sense but relied on rule-based heuristics that automatically evaluated outputs against set logic for correctness and format. This approach reduced the need for time-consuming manual labeling of each sample or step by humans, making it more scalable and less resource-intensive than typical RLHF involving learned reward models or human rankings.

However, some initial manual effort was still required to define the precise logical rules and thresholds that constituted acceptable steps, which is a significant upfront design task. The simplicity of binary, rule-based feedback also means fewer annotators or less annotation effort per data point is needed during training, since the evaluation is automated and less subjective. This contrasts with standard RLHF workflows, where human annotators must spend significant time comparing multiple responses or assigning scalar reward scores, which can be costly and slow.

DeepSeek’s method balanced between manual and automated tasks: manual logic design upfront to create rule-based evaluators, combined with automated binary labeling of the model outputs during training, thus obtaining clear accept/reject signals without heavy manual annotation burden or the use of complex learned reward functions. This made the alignment process more efficient and scalable while preserving meaningful feedback for reinforcing correct reasoning steps. It also made the feedback pipeline much more scalable and efficient, even if it sometimes lacked the fine-grained alignment possible with advanced learned reward models. The system evaluated outputs for correctness and format compliance.

How DeepSeek handles complex scenarios

Consider a scenario where you want DeepSeek to identify a bug in your code.

In such a scenario, where a code snippet compiles correctly but produces an incorrect result, DeepSeek employs multiple training and alignment strategies to identify and rectify errors. For instance, consider the task of implementing a function to sum two numbers. The model might initially generate the following code, which compiles but returns the product instead of the sum:


def sum_two_numbers(a, b):

    # Incorrect implementation: returns product instead of sum

    return a * b

# Testing the function

result = sum_two_numbers(8, 6)

print("Initial output (incorrect):", result)  # Outputs 48 instead of 14

During training, DeepSeek’s feedback mechanisms detect the logical discrepancy between the expected output and the actual result. Leveraging these signals, the model iteratively refines its output, leading to the corrected implementation:


def sum_two_numbers(a, b):

    # Correct implementation: returns the sum of two numbers

    return a + b

# Testing the corrected function

corrected_result = sum_two_numbers(8, 6)

print("Corrected output:", corrected_result)  # Outputs 14 as expected

This correction process is supported by targeted reward models and error diagnostics that guide the model to recognize and amend mistakes. As a result, DeepSeek produces code that is not only syntactically valid but also functionally accurate and reliable.

The reasoning mechanics are also driven by how DeepSeek was trained. Let’s check it out.

DeepSeek training

DeepSeek’s sparse but well-defined rule-based feedback was applied selectively on training samples identified through a data-driven prioritization strategy. Samples were chosen based on criteria such as prompt complexity, presence of multi-step reasoning, factual or logical difficulty, or decision-critical context where alignment and correctness were paramount.

For instance, prompts involving multi-step mathematical proofs, code generation with correctness checks, or intricate legal reasoning were flagged for receiving explicit rule-based evaluation signals. In contrast, simpler or more routine prompts, such as straightforward fact retrieval or common conversational queries, were often skipped or assigned coarser, indirect rewards, reducing unnecessary computational load. Once these reward signals guide updates to the model’s parameters through policy optimization (e.g., via GRPO), the learning is encoded directly into the model weights, meaning the model generalizes what it learned across similar prompts, even if feedback wasn’t explicitly provided for every case.

Throughout training, as R1-Zero iterated through complex reasoning tasks, an emergent behavior was observed: the model increasingly generated longer and more detailed intermediate reasoning steps. This was not explicitly incentivized by rewards tied to response length; rather, the underlying GRPO framework naturally favored reasoning paths that improved correctness. Mathematically, longer stepwise reasoning sequences tended to receive higher relative rewards because they more reliably demonstrated logical coherence and correctness across intermediate steps. These rewards are influenced by weighted components within the reward model that prioritize factors such as accuracy, completeness, and clarity, assigning greater weight to sustained logical consistency throughout the reasoning chain.

In essence, by producing extended CoTs, the model maximized its expected reward signal computed under GRPO’s group-relative scoring function, defined as the normalized advantage:

Here, is the reward of candidate , is the mean reward across the candidate group, and is the standard deviation. This formal, optimization-based framework explains the increase in reasoning detail – DeepSeek effectively learned to elaborate its internal thought process when doing so increased the relative reward advantage and improved the likelihood of the correct final answer. Therefore, the model’s longer reasoning outputs emerge as a strategic response to the reward structure rather than a random or unexplained behavior.

The model did not receive explicit supervised CoT labels during training. Instead, its outputs were assessed based on both the correctness of the final answer and adherence to a prescribed structured format containing intermediate reasoning steps (e.g., <think> and <answer> blocks).

Since the model was exposed to examples where intermediate reasoning steps correlated causally with receiving a reward, it began favoring generation paths where thought preceded answer, leading to a form of self-organized internal reasoning strategy. Importantly, this behavior was not present in earlier DeepSeek models trained with standard supervised fine-tuning alone. Those versions might reach correct answers, but their outputs lacked explainability, struggled with multistep tasks, and did not generalize well to harder benchmarks such as AIME, MATH, and GPQA. In contrast, R1-Zero, without using any supervised CoT data, began outperforming these earlier models simply by being aligned through this reward-only approach.

This was the real A-ha moment that a model could discover interpretable and effective reasoning patterns, not because it mimicked labeled explanations, but because it was incentivized to generate structured outputs that worked consistently. It challenged the belief that hand-crafted rationales were necessary for learning to reason, and laid the foundation for the fully realized DeepSeek-R1 model.

Cold start data for training

DeepSeek’s team invested heavily in generating high-quality, structured reasoning data – not by manually labeling thousands of CoT examples, but by leveraging R1-Zero’s unsupervised reasoning ability. Once R1-Zero began generating outputs that adhered to the CoT format and were largely accurate, a large set of these model-generated responses was collected to serve as cold start supervised training data.

The correctness of these outputs was validated automatically whenever possible by comparing generated final answers against known ground truths using programmatic checks, test cases, or logical verification rules. In cases where automated validation was insufficient or ambiguous, human reviewers performed selective manual verification to ensure quality and correctness. While the manual checks required additional time and effort, the majority of validation leveraged automated procedures, making the overall process more efficient and scalable than fully manual annotation. This hybrid validation strategy enabled the researchers to efficiently bootstrap high-quality CoT-labeled data from the model’s own outputs without incurring prohibitive annotation costs. Though this data was model-generated, it effectively became labeled: each sample contained the original prompt (implicit question), a <think> block with reasoning steps, and an <answer> block with the final response.

Consider, for example, the following prompt:


"What is the square of the sum of 3 and 2?"

The model output is as follows:


<think>

3 + 2 = 5.

5² = 25.

</think>

<answer>

25

</answer>

This structured dataset seeded the next phase: using the R1-Zero-generated corpus to fine-tune a checkpoint of DeepSeek-V3, thereby transferring R1-Zero’s emergent reasoning into the larger backbone. Here, R1-Zero functioned as both a student and a teacher – initially learning through simple, rule-based reward signals during reinforcement learning, and subsequently guiding its own training by generating distilled examples. While this process shares similarities with GRPO’s reward-driven optimization, R1-Zero’s approach focuses specifically on reinforcing behaviors aligned with logical reasoning and correctness rather than just maximizing a group-relative advantage.

Unlike traditional knowledge distillation that uses supervised loss to match teacher outputs, DeepSeek leveraged reward signals to explicitly optimize for structured, stepwise reasoning quality and adherence to output format, which standard distillation losses alone struggle to capture. Employing reinforcement learning in this way helped the model dynamically explore and refine reasoning strategies, discovering outputs that maximize clarity and correctness. These outputs then served as reliable teacher examples for subsequent supervised fine-tuning, effectively bootstrapping improved performance while complementing GRPO’s group-relative reward framework.

Here is what happens during training:

DeepSeek-R1’s training begins with pretraining to establish foundational language understanding and expert specializations.
Following this, the model is further trained on the CoT data, where the RL process is enhanced by an additional consistency reward alongside the original binary correctness and format rewards.
While the original reward function primarily assigned binary accept/reject signals based on the correctness of the final answer and adherence to the <think> + <answer> output structure, the consistency reward encourages language consistency and coherence across the model’s generated responses. These two rewards are combined by simple summation to form the final reward used in policy optimization.
The consistency reward is computed per response and does not explicitly store or compare against previous responses generated for the same prompt. Instead, it evaluates the internal coherence of each individual output using a combination of rule-based heuristics and learned models, typically analyzing factors such as language uniformity and stylistic consistency to assign a consistency reward. For example, if the prompt is in English, responses are rewarded for maintaining English without introducing other languages or inconsistent phrasing.

This reward was introduced after observing issues with mixed-language outputs in earlier models (such as DeepSeek-R1-Zero). Ablation studies showed that while the consistency reward sometimes slightly reduced raw task performance, it greatly improved human-readability and user experience by encouraging outputs that are linguistically coherent.

This additional signal ensured that the model’s reasoning style remained coherent across all steps – not just logically sound, but also stylistically aligned and linguistically fluid.

The consistency reward function combines a binary signal with language-based metrics to evaluate internal coherence within each output. Formally, it can be expressed as follows:

Let’s break this down:

is a rule-based binary signal indicating the presence or absence of key consistency criteria (e.g., adherence to formatting rules or logical constraints), valued as 0 or 1
is a continuous score derived from language models measuring stylistic uniformity, fluency, and semantic coherence
α∈ [0,1] is a weighting factor balancing the contribution of the binary and language-based signals

By integrating these components, the model effectively rewards outputs that both meet explicit rule-based standards and demonstrate natural, consistent language flow, enhancing overall reasoning clarity and coherence.

This process wasn’t a one-off; it was iterative and self-reinforcing, running through multiple cycles (typically 3–5 iterations) where each iteration used the latest distilled examples and updated reward signals to progressively enhance the model’s reasoning ability and output quality, creating what we describe as a looped strategy. Improved reasoning led to higher-quality CoT examples, which in turn trained better models – which then produced even stronger outputs, fueling the next cycle of generation and refinement.

The final DeepSeek-R1 model resulted from a multi-stage process combining efficient architecture, adaptive training rewards, and a data-driven approach that encouraged clear reasoning and used top outputs to improve future training.

Now, let’s see how all that we have learnt comes together in the training pipeline.

The training pipeline of R1

The training pipeline of DeepSeek involved the following steps:

Figure 2.6: The training pipeline of DeepSeek-R1

Let’s walk you through these stages:

Cold start SFT with R1-Zero outputs: The cold start data served as a bootstrapping corpus for a new model checkpoint built upon the DeepSeek-V3 base.

This stage served two purposes:

First, it transferred the emerging reasoning behavior from R1-Zero into a new model through supervised learning
Second, it allowed the team to evaluate how well the DeepSeek-V3 base could generalize from R1-Zero’s reasoning traces.

The results were promising. Even without additional reinforcement learning at this point, the model trained with cold start supervised fine-tuning (SFT) data began showing measurable improvements in structured reasoning compared to V3 alone. This was reflected in evaluation metrics such as the accuracy on multi-step reasoning benchmarks, where the cold-start SFT model achieved an improvement of approximately 10–15% over V3 – for example, going from around 65% accuracy to 75–80% correctness on complex math and logic tasks. Additionally, metrics assessing reasoning coherence and adherence to the expected <think> + <answer> format showed significant gains, indicating that the model was not only more accurate but also produced better-organized and transparent reasoning paths.

Reinforcement learning with consistency reward: The next step in the pipeline involved training this new checkpoint via a second round of RL. This time, the training included not only the original reward signals based on accuracy of the response, which was measured by comparing each model output against the expected, known correct answers following the criteria established in the cold start phase, but also a newly introduced metric: linguistic consistency.
Final SFT with general knowledge CoT fusion: Although the original pipeline envisioned a distinct final SFT round to integrate common-sense and world knowledge into CoT-style reasoning, this stage was ultimately bypassed. Instead, training proceeded directly from large-scale pretraining to supervised fine-tuning using the distilled CoT datasets. This approach effectively combined the model’s broad general knowledge with structured reasoning examples within a single fine-tuning phase, without requiring an additional dedicated fusion round. The rationale was straightforward: a reasoning model cannot thrive in the real world if it can only solve equations and puzzles. It must also handle everyday queries, facts, and human-interpretable logic.

DeepSeek fuses CoT reasoning with broad pretrained knowledge through fine-tuning on a mixed dataset combining unsupervised text and supervised, stepwise CoT examples. This trains the model to generate coherent, fact-based explanations by integrating structured reasoning traces with its extensive knowledge base.

Multi-task objectives and consistency constraints further refine this integration, enabling DeepSeek to produce transparent, logically sound outputs across diverse domains while amplifying patterns linked to accurate, interpretable answers.

Final RLHF: The final stage in R1’s training involved RLHF, where the model’s outputs were optimized using explicit reward functions based on human preferences. This phase also involved preference comparison pairs, where human annotators ranked multiple responses to the same prompt. These signals were used to fine-tune a reward model that then guided a final RL loop.

To capture and enforce these preferences, the RL phase used explicit reward functions designed to evolve the model’s behavior iteratively:

User expectation definition: A panel of domain experts and annotators developed detailed guidelines describing desirable model responses. These guidelines emphasized avoiding ambiguity, minimizing unnecessary verbosity, maintaining precision in factual and logical statements, and prioritizing helpfulness in context.
Rule-based checks: Automated rule-based evaluation scripts parsed each generated response to check for compliance with these guidelines, as in these examples:
- Ambiguity was detected by scanning for vague words or contradictory statements.
- Verbosity was measured by comparing token counts against expected length ranges for similar prompt types.
- Helpfulness was partially evaluated through heuristic indicators such as the presence of relevant explanations, stepwise reasoning, and actionable conclusions.
- Precision was assessed by comparing model outputs against ground-truth answers or reference solutions, primarily using binary correctness labels.
Score computation: Each response was assigned a composite reward score combining multiple factors:

is 1 if the final answer is correct and corresponds to the known truth, else 0.
is a scalar reward based on the absence of ambiguous phrases.
penalizes overly long or redundant responses (lower values for excessive length).
rewards the presence of detailed, relevant reasoning and explanations.
The weights were tuned to balance these aspects according to human preference studies.

For the Prove that the sum of two even numbers is even prompt, DeepSeek’s reward functions combine multiple components weighted to balance logical correctness, clarity, and consistency.

Rule-based check heuristics: The following are performed:
- Verification that all key mathematical terms (e.g., even, sum, 2a, 2b) appear in the proof.
- Structural checking to ensure the proof follows an expected logical format: definition, substitution, and conclusion.
- Formatting rules, such as correct use of symbols and stepwise argument separation.
Weight choices for reward components: The reward components include the following:
- Logical correctness (rule-based checks) weighted heavily at around 0.5 due to its critical role.
- Clarity and language consistency (language model-derived signals) weighted at 0.3 to reward fluent, understandable explanations.
- Internal coherence and adherence to the reasoning format weighted at 0.2 to encourage stepwise structure.
Human preference data collection: Human annotators were presented with multiple candidate proofs generated for the prompt and asked to rank them based on correctness, clarity, and completeness. These rankings were used to train reward predictors that estimate human preferences, guiding the model to prioritize reasoning quality over superficial attributes. This data-driven supervision helped calibrate reward weights and shape heuristic rules, aligning automated scoring with human judgment.

Together, these methods ensured that DeepSeek produced mathematically sound, clear, and well-structured proofs, such as the example demonstrating that the sum of two even numbers is even.

After finalizing the main R1 model, DeepSeek performed distillation experiments to transfer its advanced reasoning abilities to smaller, more efficient models.

Distillation and distribution

Distillation is important because it enables compact models to learn complex behaviors by training on high-quality outputs from larger models, improving accessibility and deployment flexibility. In this process, the distribution of R1’s detailed CoT outputs serves as the training data distribution, guiding smaller models such as DeepSeek-R1-Distill-Qwen and DeepSeek-R1-Distill-Llama to generalize sophisticated reasoning even without RL.

The implications were significant in demonstrating that reasoning capabilities could be effectively transferred through knowledge distillation. Instead of retraining large models via costly RL cycles, the CoT traces generated by DeepSeek-R1 served as high-quality teacher data for training smaller, more efficient student models.

During distillation, a combined loss function was used to balance the student’s imitation of the teacher’s soft output distributions and adherence to ground-truth labels. Specifically, the loss was formulated as follows:

Let’s break this down:

is the knowledge distillation loss, computed as the Kullback-Leibler divergence between the softened probability distributions of the teacher and student models.
is the standard cross-entropy loss with the ground-truth labels.
is a weighting hyperparameter controlling the balance between distillation and supervised learning.
Temperature T is applied to soften the teacher’s logits before calculating , typically set to T=2 or T=4 to provide richer gradient signals.

DeepSeek’s distillation process balanced knowledge transfer and supervised learning by tuning hyperparameters such as (commonly set to 0.7) and temperature (often T=3) to optimize learning signals. These settings led to substantial reductions in model size and inference latency while closely matching the teacher model’s reasoning accuracy. For further details on distillation loss and hyperparameter choices, you can refer to the DeepSeek research paper at https://arxiv.org/abs/2501.12948.

This distillation strategy effectively compressed the large-scale reasoning competence of R1 into lighter models by leveraging CoT instructional material as a rich supervisory signal. It demonstrated that large-scale reasoning need not be restricted to hefty architectures or repeated RL training, enabling scalable deployment of strong, explainable reasoning assistants.

Emergent patterns in self-teaching

One of the most fascinating aspects of this pipeline was its self-reinforcing structure. R1-Zero created an iterative cycle:

Figure 2.7: DeepSeek’s self-teaching iterative cycle

This iterative cycle continued for a fixed number of rounds (typically 3–5 iterations) until performance gains plateaued and further improvements became marginal (please refer to Figure 2.7 for a quick recap).

This controlled loop allowed the model to progressively bootstrap stronger reasoning abilities without running indefinitely, balancing training efficiency with quality improvements. At each stage, the model was learning not just from external feedback, but from its own historical best efforts.

Well, well! Who said it was so easy to build an LLM!

Next, we turn to how DeepSeek’s engineering pushed the boundaries of efficiency without sacrificing intelligence.

Advanced capabilities of DeepSeek

DeepSeek’s evolution from a powerful text reasoning model into a general-purpose AI agent continues in its multimodal and real-world extensions. In this section, we explore DeepSeek’s foray into multimodal reasoning, its emerging agent framework, and how these innovations will pave the way for real-world deployment across education, science, enterprise, and daily life.

Vision capabilities of DeepSeek

DeepSeek’s architecture (originally designed for language tasks) has evolved to support multimodal capabilities, particularly through models such as DeepSeek-VL. These models integrate text and visual inputs, enabling the system to understand and reason about images jointly with language. While DeepSeek is currently multimodal in the sense that it supports text and vision, there is no official support yet for additional modalities such as audio or video in publicly released versions.

Architectural changes for vision-language integration

To enable image understanding, DeepSeek-VL extends its language foundation with a vision encoder, typically a modified vision transformer (ViT) or convolutional neural network (CNN).

The system is composed of three main components: a hybrid vision encoder, a vision-language adapter, and a language model.

The following figure shows how the vision-language adapter works:

Figure 2.8: DeepSeek’s vision adapter (source: https://arxiv.org/html/2403.05525v2)

DeepSeek utilizes a two-layer hybrid MLP to effectively connect the vision encoder with the LLM. Initially, separate single-layer MLPs are used to process high-resolution and low-resolution visual features independently. These processed features are then concatenated and passed through an additional MLP layer, which maps them into the input space of the LLM. Specifically, the DeepSeek-VL-1B model is built upon DeepSeek-LLM-1B, which was trained on approximately 500 billion text tokens. Similarly, the DeepSeek-VL-7B model is based on DeepSeek-LLM-7B, trained on roughly 2 trillion text tokens.

When a user includes an image in the input, the model first processes it using a vision encoder (typically a CNN or ViT) that converts the image into fixed-size patch embeddings, each represented as a vector in a shared latent space (commonly 768 or 1,024 dimensions). These patch-level embeddings capture local visual features and are then fed into cross-modal attention layers alongside tokenized text embeddings of the same dimension.

The model identifies relevant image regions by dividing the image into uniform patches (e.g., 16x16 pixels), which serve as the units of attention. Through cross-modal attention within each transformer block, text tokens (processed via a standard tokenizer) learn to focus selectively on these visual patches. For example, when processing the word graph, the model attends to specific patches containing visual elements of the graph, effectively aligning linguistic context with spatial features.

This multimodal fusion allows the model to build a joint representation incorporating both the semantic content of text and the spatial hierarchy of images, enabling coherent reasoning and generation across modalities. This explanation balances technical detail and clarity, making it accessible for our target audience while highlighting how patch-based attention drives visual grounding.

Suppose a user inputs the following, along with the image:


Refer to the attached chart and explain if the growth pattern in Q2 seems normal compared to Q1 and Q3.

Figure 2.9: An example image input for DeepSeek

Here is how DeepSeek-VL-7B processes this:

Visual encoding: The image (e.g., a chart) in the prompt is divided into patches and passed through a vision encoder. This step captures details such as bar heights, axis labels, and overall layout.
Text encoding: The question is tokenized and fed into the language backbone, which creates a representation of the user’s intent (for example, growth pattern in Q2).
Cross-modal alignment: The model uses attention to link words or phrases in the question to the most relevant visual regions. For instance, Q2 activates attention over the part of the chart labeled Q2, while growth links to the corresponding bar’s height.
Response generation: By combining these aligned representations, the model generates an answer, such as The growth in Q2 is noticeably higher than Q1 and Q3, suggesting a likely one-time event or seasonal spike.

The ability to reason simultaneously over text and imagery enables DeepSeek-VL-7B to solve problems that neither modality could address alone, such as interpreting a diagram with accompanying questions or reading and correcting handwritten equations. In education, this supports intelligent tutoring systems that can interpret scanned notebooks; in enterprise settings, it enables systems to extract structure and meaning from invoices, tables, or schematics.

DeepSeek extends its use of MoE and MLA into the multimodal regime. This means that even when handling visual data, the system activates only a subset of its parameters, ensuring efficient computation. Visual reasoning does not dramatically increase the inference cost thanks to this dynamic specialization approach.

Agentic reasoning and tool integration

Although this chapter primarily discusses reasoning mechanisms, it is important to acknowledge that DeepSeek also includes foundational components that support broader agentic capabilities. These include the ability to interact with external tools, perform multi-step task planning, and dynamically adapt its actions based on feedback. Some of these features are elaborated in detail in Chapter 6.

DeepSeek models, especially DeepSeek-R1, serve as powerful reasoning engines within AI agents by enabling them to break down complex problems into explicit, verifiable reasoning chains through CoT techniques. This structured reasoning underpins DeepSeek’s proficiency in multi-step problem solving, logical inference, and contextual decision-making.

In applied workflows, DeepSeek can act as an autonomous agent by doing the following:

Executing sequential actions informed by intermediate outputs and environmental feedback.
Integrating external tools and data sources dynamically to augment its decision-making process.
Handling dependencies and error recovery through context-aware planning.
Coordinating multi-agent workflows for complex task execution.

These agentic reasoning capabilities make DeepSeek suitable for real-world applications requiring adaptive, goal-directed behavior beyond static question-answering.

We will now focus on how DeepSeek compares with other LLMs and some of the shortcomings.

DeepSeek in the global LLM landscape

The global LLM landscape is bustling with new models and techniques, with a competitive race to achieve reasoning capabilities and, ultimately, artificial general intelligence (AGI). LLMs form the necessary first step toward AGI.

Take a look at how different LLMs compare against each other (as of mid-2025):

Parameter	DeepSeek (V3, R1, CoderV2/V3)	OpenAI GPT-4o	Anthropic Claude 4 Sonnet	Google Gemini 2.5 Pro
Technique used	Sparse MoE, FP8 low-precision, RL CoT, single-pass training	Dense transformer, speculative decoding, vision-augmented layers	Dense transformer, advanced RLHF, long-context, improved multimodal	Dense transformer, multimodal fusion, advanced RLHF, long-context, video capabilities
Architecture	670B total params: 37B active per inference, 2 experts/layer from 256 experts, MLA, MoE routing	~175B params (rumored ~1T planned), dense, multimodal vision layers	~400–600B dense params, all activated per input	~600–800B dense params, dense activation per input
Tasks and capabilities	State-of-the-art structured reasoning, math, symbolic logic, code-generation, retrieval-augmented Q&A, interpretable CoT output, scalable context	Leading Q&A, multimodal (vision-language), summarization, math, advanced code, high robustness	Excellent general Q&A, advanced summarization, strong math/programming, long-context reasoning, robust vision-language synergy	Cutting-edge multimodal, video understanding, long context, advanced reasoning, generative workflows
Cost (training)	~$5.6M (2,000 H800 GPUs), open source recipes, efficient via FP8 and MoE \| Rumored $100–300M, dense, proprietary	Estimated $100M+, proprietary, dense, high GPU requirements \| Estimated $150M+ scale, proprietary, extensive multimodal training costs
Performance	SOTA/better on math (AIME, MATH, GPQA), code (pass@1, pass@5), transparent stepwise reasoning	SOTA in vision, general AI, multimodal tasks, code/math competitive	SOTA on long-text, strong on math, code, and vision-to-text, robust reasoning, and summarization	Leading performance in multimodal video generation, long context reasoning
Parameters active	37B (MoE) per inference out of 670B total	100% model activated per step	100% model activated per step	100% model activated per step
Metrics	pass@1/pass@5 (HumanEval++, Codeforces, LeetCode), MATH/GPQA/AIME accuracy, interpretability, retrieval benchmarks	Similar standard benchmarks, detailed scores not public	Benchmark metrics (MATH, GPQA, code), limited public availability	Proprietary benchmark results, showing cutting-edge multimodal capabilities
Inference time/latency	Approximately 5x faster than dense models at a similar scale; typical latency ~100–150ms per token on optimized hardware for 32–64k token context	Moderate to high latency, around 500–700ms per token depending on input length and hardware; optimized for batch inference	Slightly higher latency than GPT-4o, approx. 700–900ms per token due to complex dense activations and long-context processing	Comparable or slightly higher latency than Anthropic, with additional overhead for multimodal and video processing
Reasoning traceability	Explicit “think/answer” format, public CoT tools/data, interpretable intermediate reasoning steps	Limited transparency; mostly black-box internal logic	Some interpretability emphasis, less exposed intermediate representations than DeepSeek	Some interpretability, but focus on multimodal fusion and certain black-box components
Transparency and openness	100% open source (weights, data, recipes, RL protocols)	Closed source, minimal model details released	Closed source, sparse technical disclosures	Closed source, released via commercial API and partnerships

Table 2.1: Comparative table of DeepSeek versus leading LLMs (mid-2025)

DeepSeek delivers strong cost-to-performance efficiency through sparse routing, interpretable reasoning outputs, and full open source transparency, excelling in math, code, and CoT tasks. GPT-4o stands out in multimodal and vision capabilities but comes with high computational costs and limited transparency. Claude 4 Sonnet offers a competitive proprietary alternative with improved efficiency and long-context handling over previous Claude models, performing well in long text, math, and code tasks, though it remains less open and constrained by dense inference bottlenecks.

DeepSeek’s limitations and how they compare to other models

While DeepSeek has pushed the envelope in reasoning, openness, and architectural innovation, it is by no means without limitations. Like all language models, it grapples with trade-offs between scale and interpretability, latency and capability, openness and performance.

This section aims to provide a balanced view of DeepSeek’s current boundaries, how they stack up against its peers such as GPT-4o, Claude 3.5/4, Gemini 1.5, and Mistral, and how these challenges might be addressed in future iterations.

Scaling challenges and sparse expertise limitations

DeepSeek-V3’s 670B parameters – while efficient via MoE – still make training and serving the model a non-trivial task. Even with sparse activation (37B active params per inference), deploying DeepSeek at scale requires high-throughput GPUs and bandwidth-optimized environments. This means that for many smaller institutions, full model usage remains aspirational, despite the release of distilled variants.

Moreover, MoE introduces its own set of constraints. Routing instability across experts, token-drop issues, and expert under-utilization remain areas of active research. DeepSeek’s use of shared experts and FFN pre-routing helps mitigate some of these issues, but performance degradation at tail token distributions still occurs. In contrast, dense models such as Claude 4 or GPT-4o, while more compute-heavy, maintain more stable token flow across all layers.

Sparse models also suffer from limited representation sharing across unrelated inputs, meaning generalization in certain zero-shot scenarios may lag behind dense counterparts. This trade-off between efficiency and universality is not unique to DeepSeek, but its manifestation is accentuated at DeepSeek’s unprecedented scale.

Inference latency and real-time interaction trade-offs

Although MoE and FP8 training help speed up inference per token, DeepSeek still faces latency challenges in real-time settings, especially when paired with search modules or multi-document rerankers. While CoT reasoning improves output quality, it often results in longer responses, which increases perceived latency.

Claude 4 and GPT-4o, with their highly optimized decoding and caching strategies, often outperform DeepSeek in real-time generation benchmarks. GPT-4o, in particular, benefits from tight model-server integration and aggressive speculative decoding, yielding response times comparable to GPT-3.5 with GPT-4 level reasoning.

DeepSeek’s multi-token prediction (MTP) and KV compression (via MLA) help alleviate this somewhat, but user-facing applications requiring millisecond-level responsiveness, such as chat agents or embedded devices, may find Claude or GPT variants better optimized today.

Interpretability and alignment risks

Ironically, while DeepSeek pioneered structured reasoning with think/answer formatting, this strength can also become a bottleneck. The enforcement of CoT outputs means models can hallucinate rationales that appear structured but are subtly flawed, leading to false confidence. Because the model is trained to produce coherent logic, even wrong answers may be wrapped in convincing language, creating alignment and safety risks.

This issue isn’t unique to DeepSeek. GPT-4 and Claude 4 also face similar concerns around hallucination in multi-step reasoning. However, their models often include adversarial preference modeling and multi-signal safety protocols – techniques still being developed for DeepSeek. The lack of a preference ranking model (PRM)-based training step in DeepSeek means alignment relies on simpler reward heuristics, potentially missing nuance in ethical or edge-case scenarios.

Context length and compression ceiling

Although DeepSeek uses MLA to reduce context footprint, it still trails Claude 4 and Gemini 1.5 in ultra-long context tasks. Claude’s 200k context window and Gemini’s sparse attention tricks make them more capable of absorbing entire books, legal documents, or multi-session dialogues. DeepSeek caps more realistically at 64k tokens for usable performance.

Additionally, while MLA compression helps, it requires trade-offs in retrieval fidelity, particularly in multi-document QA or hybrid retrieval/ranking setups. In contrast, Claude’s context prioritization and GPT-4o’s selective attention offer more intelligent context utilization at scale.

Dataset and cultural scope gaps

Despite its multilingual focus, DeepSeek remains comparatively undertrained on certain cultural datasets and niche domain corpora. GPT-4 and Claude 4, which benefit from broad corpus ingestion and fine-grained domain tuning, demonstrate superior nuance in fields such as finance, law, and niche scientific domains. DeepSeek, due to its cost-aware pretraining, takes a more generalist route, occasionally at the cost of narrow-domain depth.

However, this is also where DeepSeek’s open infrastructure shines: its modular retriever, reranker, and decoder setup allows researchers to fine-tune for missing knowledge areas, a flexibility that proprietary models do not offer.

Outlook on limitations

DeepSeek’s limitations are real but surmountable. As the community around it grows, many of these challenges – especially those related to routing, real-time decoding, safety tuning, and context optimization – are likely to be addressed by research contributions. Unlike proprietary models, DeepSeek’s transparent ecosystem allows for shared learning and iterative correction.

In many ways, the very act of exposing limitations becomes an invitation for others to refine, adapt, and challenge the model itself. That’s the promise of open science – not perfection, but participation.

Summary

In this chapter, we delved into the core research and technical evolution behind DeepSeek, tracing its trajectory. We examined how DeepSeek’s development splits into two powerful threads: the construction of foundational models and the advancement of reasoning capabilities. Each successive version of DeepSeek added a new layer of innovation, from scaling dense architectures to pioneering efficient MoE models, and, ultimately, refining multi-step reasoning through rule-based reinforcement learning.

We explored how the adoption of MoE techniques led to significant performance gains at lower compute costs, and how innovations such as MLA, MTP, and GRPO addressed key challenges in training stability, inference efficiency, and reasoning reliability. The chapter also highlighted DeepSeek’s specialized efforts in areas such as code intelligence, mathematics, and formal logic, illustrating its versatile application across domains.

The release of DeepSeek-R1 represents the culmination of these developments, demonstrating that a powerful, efficient, and reasoning-capable LLM can emerge from the integration of sparse architectures and minimal, rule-driven reinforcement learning. You should now have a clear understanding of how DeepSeek evolved through iterative innovation and the technical breakthroughs that drive its performance.

In the next chapter, we will discuss various techniques for prompting DeepSeek.

Get This Book’s PDF Version and Exclusive Extras

3 Prompting DeepSeek

If you’ve ever moved from writing imperative code, manually telling the Domain Object Model (DOM) how to update a web page step-by-step, to a declarative framework such as React, you know the required mental shift. You stop giving meticulous instructions and start describing the desired outcome, trusting the system to figure out the how. The same principle applies when moving from traditional LLMs such as Mistral 7B or Llama 3.3 8B to DeepSeek’s reasoning engines. The muscle memory you’ve developed for carefully guiding a model’s thought process might actually work against you with an engine that’s built to reason on its own.

This chapter is all about navigating those differences, from the headline-grabbing quirks (such as R1’s allergy to system prompts) to the subtle behavioral shifts that can make or break your results. Whether you’re migrating an existing prompt library or starting fresh with DeepSeek, consider this chapter as your field guide to a fundamentally different paradigm in language models.

It’s important to remember that we are in the early days of understanding these powerful new models. At this point, the DeepSeek models are almost a year old. The guidance in this chapter is a synthesis of the best information currently available: official documentation from the DeepSeek team, usage reports from industry pioneers, and the first wave of academic papers exploring these systems. Think of this as a living document: a snapshot of our collective understanding today. The best practices will undoubtedly evolve as the community learns more, but what follows is our best map of the terrain so far.

Most of this chapter focuses on the R1 series: DeepSeek’s reasoning-first models that flip conventional prompt engineering wisdom on its head. These models don’t just process differently; they were trained differently, using reinforcement learning techniques that created a model with its own internal Chain-of-Thought (CoT) process. It’s like the difference between giving turn-by-turn directions to a human driver versus simply telling a self-driving car the destination. The R1 models already know how to navigate the reasoning journey.

The implications ripple through every aspect of prompt design:

Few-shot examples? They’ll actually make your results worse (a 15% accuracy drop, as mentioned in the original DeepSeek paper (https://arxiv.org/html/2501.12948v1).

Think step -by step instructions? Redundant. R1 already does that internally.

Rich system prompts? The model might ignore them entirely.

Complex, verbose instructions? Less effective than clear, minimal directives.

Meanwhile, the V-series models behave more like the LLMs you’re used to, but even they have their quirks. For example, temperature mappings differ from OpenAI’s defaults, and there is a preference for structured formatting that goes beyond typical markdown.

This chapter dissects the DeepSeek prompting paradigm piece by piece. We’ll explore the core mental models that explain why these differences exist (hint: it’s all about that reinforcement learning), provide concrete dos and don’ts backed by official documentation and community testing, and equip you with templates and troubleshooting strategies for both R-series and V-series models.

By the end, you’ll understand not just what to do differently, but why DeepSeek’s unique training approach demands we rethink prompt engineering from first principles. Think of it as learning a new programming paradigm. Once you grasp the underlying philosophy, the specific techniques fall naturally into place.

In this chapter, we will cover the following main topics:

Core mental models and principles of DeepSeek
General tips and advice for prompting DeepSeek
Working with the structured output
Unique prompting techniques for the V series
Troubleshooting
Prompt migration guide

Technical requirements

You will need the following for this chapter:

Start with the project repository and install the dependencies:


git clone https://github.com/PacktPublishing/DeepSeek-in-Practice.git

cd Chapter03

pip install -r requirements.txt

The requirements file includes OpenAI and Instructor for sending prompts and getting structured output, and Pydantic for helping to create structures we can pass to our models.

API Keys and environment variables: You will need API keys for accessing OpenRouter models (like DeepSeek and any others you want).
OpenRouter provides access to DeepSeek models for generating baseline comparisons during evaluation. Sign up at openrouter.ai and obtain your API key from the dashboard. We will use a free model to access the base DeepSeek-R1 model, but you need an API key to access it (via litellm). If you do not have an OpenRouter API key, you can create one at https://openrouter.ai/docs/api-reference/api-keys/get-api-key.

Core mental models and principles of DeepSeek

Before we dive into the technical weeds, let’s recalibrate your mental model of what DeepSeek models, especially the R-series models, actually are. Forget everything you know about coaxing ChatGPT into the right response with elaborate system prompts.

Here are three things you need to be careful about when you structure your prompts for the DeepSeek-R1 series.

DeepSeek models work more like theorem provers and less like chatbots: Think of R1 less like a conversational partner and more like a mathematical proof assistant that happens to speak English. When you submit a prompt to R1, you’re not starting a dialogue. You’re submitting a formal problem specification to a reasoning engine.

This isn’t just rhetoric. The R1 models were trained with reinforcement learning to prioritize getting the right answer on verifiable tasks (e.g., math/coding), and they do substantial internal deliberation before replying. Concretely, the model first generates a hidden reasoning scratchpad (the block) and only then emits the final answer (see Chapter 2 for a refresher on hidden thinking tokens).

What this means in practice:

Your prompt is a problem statement, not a conversation starter.
Expect the model to internally sketch a plan/scratchpad and then produce a single solution. (By internal plan, we simply mean the model’s hidden CoT/scratchpad; there is no separate compilation phase.)
You usually get back a solution – not a chatty back-and-forth.
Structure your prompt like a research request: Imagine you’re commissioning a research paper from a brilliant but literal-minded graduate student. You wouldn’t give them a rambling email with examples of other papers. You’d provide a clear problem statement, constraints, and expected output format. Good DeepSeek prompt structure mirrors academic problem specification. In other words, if you create a prompt in this order and format, DeepSeek generally will give you better results. It’s not something specified as official guidance, though:
```
Context: [What background information is essential?]

Task: [What specific problem needs solving?]

Constraints: [What limitations or requirements exist?]

Output: [What format should the solution take?]
```

This isn’t about being formal or stuffy – it’s about clarity. The model performs best when it can parse your intent without ambiguity.

For example, if I were interested in understanding the philosophical implications of deploying a new triage model for an emergency department, I might use a prompt such as this with a traditional LLM like Mistral Medium 3.1:


Is it ethically OK for a hospital to use a black-box AI triage system in the ER if it's more accurate than doctors? Please discuss pros and cons and conclude with what you'd recommend.

You can see we get a response like this from a Mistral model (Medium 3.1), for example:

Figure 3.1: Response from a traditional LLM – Mistral Medium 3.1

You can check out eg1.py in the book’s GitHub repository for full responses (https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg1.py). But a structured prompt that follows the preceding specification might look like this (see eg2.py https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg2.py):


## Context:

You are analyzing an ethical policy choice in clinical triage. The model is more accurate on retrospective data than clinicians but its internal logic is not interpretable.

## Task:

Evaluate whether the hospital should deploy the model and under what conditions.

## Constraints:

- Evaluate explicitly under: (1) Utilitarianism, (2) Deontology (Kantian duties), (3) Virtue ethics, (4) Ethics of care.

- For each lens: state a verdict {permit|forbid|conditional} with a 1–2 sentence justification.

- Address: fairness, accountability, transparency, harm mitigation, informed consent/assent, oversight.

- Include one strong counterargument that could overturn your conclusion and how you'd test it.

- Do NOT reveal chain-of-thought; provide only final labeled reasoning.

- Keep each field ≤ 60 words.

## Output:

Return ONLY JSON with this shape:

{

  "per_lens": {

    "utilitarianism": {"verdict": "permit|forbid|conditional", "rationale": "..."},

    "deontology": {"verdict": "...", "rationale": "..."},

    "virtue_ethics": {"verdict": "...", "rationale": "..."},

    "care_ethics": {"verdict": "...", "rationale": "..."}

  },

  "deployment_recommendation": {"verdict": "permit|forbid|conditional", "conditions": ["...","..."]},

  "oversight_plan": ["metric_audits","bias_monitoring","appeals_process","A/B_safety_guardrails"],

  "counterargument": {"claim": "...", "test": "..."},

  "residual_risks": ["...","..."],

  "confidence": 0.0

}

This prompt resulted in a long thinking output by DeepSeek-R1 and then the following JSON response (as requested):


{

  "per_lens": {

    "utilitarianism": {"verdict": "permit", "rationale": "Superior accuracy maximizes overall patient outcomes and welfare, fulfilling the utilitarian goal of the greatest good for the greatest number."},

    "deontology": {"verdict": "forbid", "rationale": "The opaque logic treats patients as mere means to an end, violating the duty to respect persons by making unexplainable, non-autonomous decisions."},

    "virtue_ethics": {"verdict": "conditional", "rationale": "Permit only if it supports, not replaces, clinician prudence and compassion. Deployment must foster trust and collaboration, not undermine professional virtues."},

    "care_ethics": {"verdict": "conditional", "rationale": "Permit only as a supportive tool. The model lacks empathy and context; final decisions must remain with human caregivers to preserve relational care."}

  },

  "deployment_recommendation": {"verdict": "conditional", "conditions": ["Use only as a decision-support tool for clinicians, not autonomous triage", "Implement robust, continuous bias auditing on protected classes", "Secure informed consent/assent from patients disclosing AI involvement"]},

  "oversight_plan": ["metric_audits", "bias_monitoring", "appeals_process", "A/B_safety_guardrails"],

  "counterargument": {"claim": "The model's superior retrospective accuracy may not translate to real-world performance due to hidden confounding variables or data drift, causing net harm.", "test": "Run a rigorous prospective A/B trial comparing model-assisted vs. standard care, with an independent DSMB monitoring for significant outcome differences and safety."},

  "residual_risks": ["Automation bias leading clinicians to over-rely on the model", "Model performance degradation over time due to data drift"],

  "confidence": 0.8

}

The minimalist mindset (or how I learned to stop worrying and trust the model): Here’s where seasoned prompt engineers often struggle. With DeepSeek, less really is more. Those carefully crafted few-shot examples you’ve been collecting? They’re not just unnecessary. They’re actively harmful. That detailed step-by-step breakdown you usually provide? You’re micromanaging a system that already knows how to think. It’s like the difference between teaching someone to drive by explaining every micro-movement versus just telling them the destination. DeepSeek models, especially R1, already have the how to think part down. They just need to know what to think about.

Here are some guidelines to get you started with the minimalist prompts for DeepSeek:

Stating the problem clearly and getting out of the way.
Resisting the urge to over-explain or provide examples
Accepting that the model’s internal reasoning might be better than your external guidance.

This minimalist approach can feel uncomfortable at first, especially if you’re used to the prompt engineering as creative writing approach that works well with other models. But once you embrace it, you’ll find that DeepSeek’s responses are often more thorough and accurate precisely because you didn’t constrain its reasoning process.

In a practical sense, a minimalist prompt template derived from the preceding suggestion could look like this:


Task: [what to decide/compute; 1–2 sentences]

Constraints: [numbered rules or lenses; keep to essentials]

Output: [tight schema or bullet template; short fields only; no chain-of-thought]

See eg3.py in the repository (https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg3.py) for an example of this principle in action. eg3.py runs a controlled comparison of three prompting styles – zero-shot minimal, few-shot with examples, and verbose step-by-step – on the same ethical triage task, using DeepSeek via OpenRouter’s OpenAI-compatible client. It constructs message payloads for each style, requests a JSON-only answer with a tight schema, and validates the response by checking required keys, allowed verdicts, word limits, and types (with minor tolerance for capitalization). Each run is timed, token usage is captured when available, and outputs are printed with PASS/FAIL validation feedback. This example highlights how minimalist prompts often comply better and cost less.

Schema: Here, schema means the explicit specification of the model’s expected output – the shape and constraints of the answer. A schema defines field names, data types, allowed values/enums, required versus optional fields, nesting, and limits (for example, word or range constraints). We express it in the prompt or via API features (JSON mode, function/tool parameters, Pydantic) so the model knows what to return, and we can validate it reliably. In short: the prompt describes the task; the schema is the output contract the response must satisfy (e.g., {"verdict": "approve|deny|escalate", "rationale": "string, <=50 words"}).

Now that you have a broad understanding of the kind of prompts that work with DeepSeek, let’s find out why they work.

Why structured, minimal prompts “click” with the R series (what’s happening under the hood)

If the research brief → JSON (or single-shot) answer style feels unusually effective with R1, that’s because it is aligned to how the model was trained to win. R1 wasn’t optimized to produce eloquent chat; it was optimized to produce verifiable, format-compliant answers after hidden reasoning.

Recall from Chapter 2, that, during post-training, DeepSeek used reinforcement learning where each candidate response was graded by rule-based checkers: math answers were checked for the right value in a specified final-answer slot; code was compiled and tested; and a format reward enforced a separation between an internal reasoning region and a clean final answer region (originally expressed with <think> … </think> and <answer> … </answer> in the training template). Give the model a tight output contract, and it recognizes the game it was trained to play. That’s why a clear problem statement plus a strict schema routinely beats ornate role-play or example dumps.

R1 trains with Group Relative Policy Optimization (GRPO): for each prompt, the policy samples K responses (typically 4–8), deterministic rule based graders compute a single scalar reward per sample (usually correctness, plus a small formatting check), rewards are centered by the group average, and the policy is updated so above-average samples gain probability mass while below-average ones lose it. In R1-Zero, there is no learned reward model – only these deterministic graders.

During training the model also learns test-time scaling: for harder problems, it thinks longer internally before responding – so you don’t need to instruct think step-by-step. Your job is to define the target and the format; the model plans privately.

Test-time scaling: R1 adaptively allocates more internal reasoning to difficult inputs and less to easy ones at inference time. This emergent behavior lets accuracy improve by thinking longer about hard problems without any special step-by-step prompting.

This training recipe also explains two practical quirks you saw earlier:

Few-shot hurts: In R1 evaluations, adding demonstrations consistently degraded performance; the authors recommend zero-shot with an explicit output format. Examples bias the internal planner toward your exemplar trail rather than the strategy RL already rewarded.
System prompts: Early R1 runs effectively down-weighted system role instructions in practice, hence the put everything in the user message advice. With the R1-0528 refresh, a system prompt is now supported – but it is still best kept minimal (identity, date, global policy), with task-specific constraints and schemas living in the user content.

A natural question that may come to your mind at this point is, why don’t the traditional chat-based LLMs have these quirks? Let’s answer this for you, dear readers.

Why don’t other LLMs behave like this?

Most chat LLMs (e.g., V series, GPT-4o, Claude 3.x) were tuned primarily via supervised fine-tuning and preference/RLHF, so they often benefit from few-shot exemplars and CoT demonstrations – the classic GPT-3 and CoT results. The traditional LLMs weren’t broadly trained with rule-based verifiers that grade a single, rule-checked final answer per prompt, so schemas help for parsing, but demonstrations still tend to teach the distribution. Reasoning-first models (R1, OpenAI o1) flipped the objective: reward verifiable correctness and format after internal deliberation, so zero-shot and schema becomes the safer default.

Here are some advantages and trade-offs DeepSeek possesses as compared to traditional LLMs:

Pros: A higher ceiling on verifiable tasks (math, code, scientific QA); less reliance on demos; stronger reliability when you pin outputs to a single slot or strict JSON (what the RL loop reinforced).
Cons: More internal thinking means latency/cost trade-offs; over-prompting (few-shot CoT, role-play) can degrade results; on open-ended, style-heavy tasks without checkers, the advantage narrows and a classic chat model with good exemplars may equal or beat R1 at a lower cost.

So, keep in mind this mental model when you prompt DeepSeek: you’re not teaching a chatbot. You’re giving a verifier-graded specification to a planner. The more unambiguous the spec and the tighter the output contract, the more you let the RL-trained behaviors do their work.

Ready to see how this mental model translates into concrete techniques? Let’s get specific.

General tips and advice for prompting DeepSeek

With the foundational principles established, we will lay down some prompt guidelines to help you achieve accurate and reliable results with DeepSeek. As we discussed in the previous section, many of these recommendations directly contradict established practices for other leading models. This adjustment is a necessary consequence of DeepSeek’s unique architecture and training methodology.

First up, we will talk about how few-shot prompting, otherwise popular, may in fact be detrimental to DeepSeek’s responses.

The few-shot fallacy

The most significant departure from conventional prompt engineering is DeepSeek’s handling of few-shot examples. While this technique is a cornerstone for improving performance on models such as OpenAI’s GPT-4o, providing examples to DeepSeek-R1 is actively detrimental. The official documentation is unequivocal on this point: Do not provide examples in the prompt, as this consistently degrades model performance.

In controlled experiments, accuracy dropped from 79.8% to ~70% when few-shot examples were added (https://www.linkedin.com/pulse/zero-shot-few-rag-benchmaking-deep-seek-sandeep-k-gil1c/).

This performance degradation occurs because providing examples interferes with the model’s native reasoning process. R1 was trained using reinforcement learning to develop its own internal problem-solving methodologies, which manifest as a hidden CoT process. When examples are included in the prompt, they attempt to force a specific reasoning path, overriding the model’s more robust, internally optimized strategy.

Instead of examples, provide the following:

Clear task descriptions.
Explicit output format requirements.
Schema definitions (for structured output).

You may want to take a look at eg3.py (https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg3.py) to check out some zero-shot prompt samples.

Next, we will talk about why system prompts are not a popular approach to follow with DeepSeek.

System prompts

It’s common to use a system prompt to set an assistant’s role and global constraints. With the original DeepSeek-R1 releases, however, system prompts were largely ignored in practice; the official guidance was to avoid them and place all actionable instructions in the user message. This reflects how early R1 was trained and evaluated: the chat template emphasized a user→assistant exchange where the model first thinks internally and then answers, rather than attending strongly to a separate system role.

In May 2025, the R1-0528 refresh added support for system prompts. DeepSeek’s own apps use a deliberately minimal system message (identify the assistant; include the current date). Even so, the practical rule of thumb still stands: keep the system message for high-level, global constraints (identity, policy, formatting preferences) and put task-specific instructions in the user message. You no longer need to seed <think>; the model will reason without it.

For the V series (V3/V3.1), system prompts do work and the official chat templates include a system slot, but they aren’t essential for good results. In line with DeepSeek’s examples, prefer short, declarative system text and focus your effort on making the user instruction clear and specific. In short: with DeepSeek, simple, explicit user prompts do the heavy lifting.

With DeepSeek, simple and clear prompts do the job just right. Let’s talk about this in detail next.

The verbose prompt trap

If you’re used to writing prompts that read like Victorian novels, it’s time to channel your inner Hemingway. DeepSeek’s official guidance emphasizes: Complex, lengthy prompts often lead to less effective results. Focus on clarity over complexity.

This isn’t about being terse for its own sake. It’s about signal-to-noise ratio. Every extra word is another opportunity for the model to misinterpret your intent. Remember, R1 is going to generate potentially thousands of tokens of internal reasoning. It doesn’t need your help overthinking things.

Here is an example of a good prompt for R1:


Analyze the security vulnerabilities in this code snippet.

Focus on SQL injection and authentication bypass.

Code: [your code here]

One example of a bad prompt could be the following:


You are an expert security researcher with 20 years of experience.

I want you to carefully examine the following code, thinking about

all possible security issues. Consider things like SQL injection,

XSS, authentication problems, and any other vulnerabilities...

[several more paragraphs of instructions]

To see an illustration of this example, check out eg4.py (https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg4.py), which runs the same vulnerable Flask snippet through two prompts – a concise good prompt that focuses explicitly on SQL injection and authentication bypass.

Here is the response for the good prompt:


Security Vulnerabilities Analysis

1\. SQL Injection

Vulnerability: The code uses direct string interpolation for SQL queries:

query = f"SELECT id, password_hash FROM users WHERE username = '{username}'"

Risk: Attackers can inject malicious SQL by manipulating the username input. For example:

' OR 1=1 -- would return the first user in the database, bypassing authentication.

'; DROP TABLE users; -- could delete tables.

Why It's Critical: Direct string interpolation allows arbitrary SQL execution, compromising data confidentiality, integrity, and availability.

2\. Authentication Bypass via Weak Hash Comparison

…

3\. Admin Privilege Escalation

…

4\. Insecure Password Hashing

…

7\. Database Connection Management

…

Exploit Scenarios

SQL Injection + Auth Bypass:

…

Fix Recommendations

Prevent SQL Injection:

query = "SELECT id, password_hash FROM users WHERE username = ?"

…

Fixed Code Snippet

…

On the other hand, you will observe that a verbose, bad prompt invites topic drift.

Check out a snippet of the prompt response here:


Security Analysis of the Provided Flask Login Code

1\. SQL Injection (Critical – CVSS 9.8)

Issue

query = f"SELECT id, password_hash FROM users WHERE username = '{username}'"

…

Mitigation

Use parameterized queries:

query = "SELECT id, password_hash, is_admin FROM users WHERE username = ?"

cur.execute(query, (username,))

Reference: OWASP SQL Injection Prevention Cheat Sheet

2\. Weak Password Hashing (Critical – CVSS 9.8)

…

8\. Database Connection Management (Low – CVSS 3.7)

Issue: No connection pooling or error handling.

Mitigation: Use context managers (with sqlite3.connect()).

…

Broader Architectural Concerns

No rate limiting → susceptible to brute force.

Missing login attempt logging and monitoring.

Weak credential storage practices.

Database not isolated securely.

Dependencies not pinned → supply-chain risks.

Severity Summary

…

Corrected Code (Secure Example)

…

Additional Recommendations

References

…

Here, as you can see, the response could have been more focused.

Using the same DeepSeek model via OpenRouter at a low temperature, the script prints both outputs and computes simple focus heuristics (mentions of SQLi/auth bypass, off-topic hits like XSS/CSRF, a focus_score, sentence-level signal-to-noise ratio, and word count).

The eg4.py script also summarizes which prompt produced the tighter, more on-target answer:


Summary: focus_winner=Good shorter_output=Good | good_focus_score=17 bad_focus_score=9 | good_words=380 bad_words=872

Let’s now talk about additional factors that may impact DeepSeek’s response generation.

Other factors impacting DeepSeek’s response to prompts

Apart from prompting techniques, there are other factors that may impact how DeepSeek processes and responds to prompts. Let’s find out.

Temperature: The official docs (https://api-docs.deepseek.com/) land on temperature ≈ 0.6 as the sweet spot for R1. Too low, and you risk repetition loops (the model gets stuck in local optima). Too high, and the careful reasoning process starts to hallucinate.

For V3, there’s an additional quirk: the API maps temperature=1.0 to an effective 0.3, so you might want to set it even lower for factual tasks.

Recommended settings for R1 are temperature=0.6, top_p=0.95. For V3, use temperature=0.3-0.5 for factual tasks, up to 0.``7 for creative work.

Persona hints: While elaborate role-playing is out, subtle persona cues can dramatically improve results. Community testing shows that simple audience-level hints such as I'm new to finance cause R1 to adjust explanation depth without compromising accuracy.

This works because you’re providing context about the output requirements, not trying to change the model’s reasoning process. It’s the difference between saying pretend you're a teacher (role-play) and explain this for a high school student (output specification).

Here are some examples of effective persona hints:


I'm a software engineer familiar with Python.

Writing for a general audience blog post.

Technical documentation for API developers.

A reliable way to write persona hints is to keep them to one sentence that specifies audience, deliverable, and depth – nothing about acting as anything. Phrase it as an output requirement, not a role: who it’s for, what you want back, and how deep to go.

A useful template is For [audience], produce [format] that [purpose]; assume [background] and [depth constraints].

For example, consider this prompt:


For a non-technical CFO, produce a 150-word briefing that informs a buy/no-buy decision; assume no ML background and define acronyms, or For API engineers, provide a runnable Python 3.12 snippet with a docstring; avoid external deps and keep the explanation under 5 lines.

This prompt is targeted and specifies one audience, one output, and one depth. If the result is still off, iterate by adjusting those three dials rather than adding role-play.

There are a few more techniques worth applying for DeepSeek prompting:

Structured formatting: Use XML tags or markdown headers to organize complex prompts. R1 parses these beautifully. Here is how you can structure the prompt with XML tags, for example:
```
<context>Background information here</context>

<task>Specific problem to solve</task>

<constraints>Any limitations or requirements</constraints>

<output>Expected format</output>
```
You may recall from Chapter 1 that R1 exhibits formatting-aware generation (i.e., a consistent structure that acts as signposts the decoder can follow). In Chapter 2, we showed how structured prompts encourage private intermediate reasoning (e.g., implicit think → answer).

We can now operationalize these learnings: use a light, flat structure to specify inputs and outputs, then let the model do the planning internally. You’re not making R1 an XML parser; you’re giving it a stable layout it can align to.
Explicit output formats: For math problems, always specify an explicit output format for final answers. For code, specify the language and any style requirements upfront.
- You may remember from our previous discussions in Chapter 1 and Chapter 2 that R1 is reinforced with rule-based validators that check final answers and format conformance. Therefore, the safest way to boost reliability is to define an output contract that a machine could grade.
- In practice, give the model exactly one slot for the final result, specify units and precision, and forbid extra prose. Here are a few tips and tricks you could try:
For math, require a single <answer> field (or final answer: line) with explicit rounding rules and units – do not ask for step-by-step.
For code, require one fenced block with the language and version (e.g., Python 3.12), an entry-point signature, and any constraints (stdlib-only, no network, time/memory limits).
If you need metadata (confidence, method), put it in separate fields – never mix with the final answer.
- Keep the contract stable across tasks; validators reward consistency more than cleverness. For example, using tags for a math problem, you could use something like this:
```
### Output

Return only:

<answer units="m/s^2" rounding="3dp">…</answer>

Or for code you could use something like this:

### Output

Return exactly one fenced code block:

```python
  # Python 3.12, stdlib only
  def normalize(v: list[float]) -> list[float]:
      ...
  if __name__ == "__main__":
      ...
```
eg5.py (https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg5.py) demonstrates how to pair explicit output contracts with lightweight, machine-checkable validators. The script sends three prompts to DeepSeek: two math problems that require returning only a single <answer> tag with specified units and decimal precision, and one programming task that must be returned as exactly one fenced Python 3.12 code block with a particular entry-point signature and a main guard.

Some of the prompt excerpts used in eg5.py (truncated) are as follows:
```
Math prompt 1
### Task
A car accelerates from rest to 20 m/s in 8 s. Compute the constant acceleration.
### Constraints
- Do not show steps or intermediate numbers.
- Do not include any text outside the required tag.
### Output
Return only:
<answer units="m/s^2" rounding="3dp">…</answer>
Math prompt 2
### Task
Compute the area of a circle with radius r = 3.2 m. Use π ≈ 3.141592653589793.
### Constraints
- Do not show steps or intermediate numbers.
- Do not include any text outside the required tag.
### Output
Return only:
<answer units="m^2" rounding="2dp">…</answer>
Code prompt
### Task
Implement a vector normalization function.
### Requirements
- Language: Python 3.12
- Libraries: stdlib only (no third-party imports)
- Style: Type hints; PEP 8 friendly; include a minimal main guard demo
- Entry point signature must be exactly:
  def normalize(v: list[float]) -> list[float]:
### Output
Return exactly one fenced code block:
```python

# Python 3.12, stdlib only

def normalize(v: list[float]) -> list[float]:

    ...

if __name__ == "__main__":

    ...
```
You will notice the following from these prompts:
- There are no extra prose, explanations, or additional code fences: The validators check that math answers have exactly one <answer> tag with the correct units, rounding, and value; and that the code response is exactly one Python code fence with the header line, the normalize (v: list[float]) -> list[float] function, a main guard, and no non-stdlib imports.
- For each response, eg5.py prints the raw model output and runs simple validators: A tag parser verifies units, rounding, and numeric correctness for the math items, while a code checker ensures there is exactly one Python code fence, the Python 3.12, stdlib only header, the normalize (v: list[float]) -> list[float] function, a main guard, and no non-stdlib imports. This shows how clear contracts plus automatic checks can raise reliability without needing to inspect CoT or intermediate steps.
- You can check out the response here: https://app.warp.dev/block/SmWy7R2GE1nsLIGoFNYwUg.
Chain-of-Draft: For token efficiency, you can ask R1 to keep only a minimum draft for each thinking step. This reduces token usage by 80% while maintaining accuracy. Chain-of-Draft (CoD) is not the same as CoT. CoT asks the model to externalize full step-by-step reasoning (often verbose). CoD is a compression policy for intermediate notes: you tell the model to keep only minimal, checkpoint-style drafts per step (e.g., 3–7 words) and return a clean final answer field.
- With R1, those drafts typically live in the hidden <think> space – you’re not asking for visible reasoning, you’re constraining how much scratchpad it uses. Because R1 is rewarded on final-answer correctness and format adherence, compressing the draft often preserves accuracy while cutting tokens substantially (reports range from ~40% to ~80% fewer tokens, task-dependent). Use CoD when you care about latency/cost but still want robust reasoning; use CoT only when you truly need a readable derivation.
- Here’s an example of what that might look like in a prompt:
```
### Constraints

- Reasoning: think step by step, but keep only a minimal draft

  (≤5 words per step, ≤4 steps). If stuck, expand to 8 words.

- Do not print the draft; return only the final field below.

### Output

<answer rounding="2dp" units="m/s^2">…</answer>
```

Here are a few more tips for CoD prompting:

If answers degrade, raise the per-step word cap (5→8→12) or allow one extra step.
For proofs/long derivations, prefer a higher cap or skip CoD.
Keep the final-answer slot stable across prompts; validators reward consistency.

Let’s take a look at a quick checklist of dos and don’ts for prompting DeepSeek (Table 3.1).

Technique	Do	Don’t	Why
Examples	Describe the task clearly	Provide few-shot examples	Examples degrade R1 performance by 5-15%
System Prompts	Put everything in user message (R1)	Use system role for instructions	R1 ignores system prompts entirely
Prompt Length	Be concise and structured	Write verbose explanations	Clarity beats complexity for reasoning models
Temperature	Use 0.6 for R1, 0.3-0.5 for V3	Go below 0.5 or above 0.7 (R1)	Prevents repetition loops and hallucinations
Personas	Give audience-level hints	Heavy role-playing instructions	Context helps without constraining reasoning
Formatting	Use XML/markdown structure	Dump unstructured text	Clear parsing improves compliance
Output Specs	Define format explicitly	Assume model knows your needs	Ensures consistent, usable outputs
Reasoning	Trust internal process	Add think step by step	R1 already does CoT internally

Table 3.1: Dos and don’ts of prompting DeepSeek

Remember, these aren’t just arbitrary preferences. Each recommendation stems from how DeepSeek models were trained and what they’ve learned to expect.

Once you have a grip of the fundamentals to prompt DeepSeek, you’ll be ready for the model-specific techniques that we’ll explore next.

Advanced techniques and tooling for structured output

A primary challenge in productionizing language models is ensuring their outputs are in a reliable, machine-readable format. While many models require extensive post-processing to parse unstructured text, DeepSeek provides dedicated features for generating structured data like JSON. However, DeepSeek’s implementation has unique characteristics and requirements that differ from other models, and understanding them is key to successful integration.

The capabilities for structured output in DeepSeek models improved significantly with the R1-0528 release in May 2025 (https://huggingface.co/deepseek-ai/DeepSeek-R1-0528). This update introduced native JSON mode and function-calling capabilities, aligning DeepSeek more closely with the feature sets of other prominent models.

A notable characteristic of DeepSeek’s implementation, however, is that even when using the dedicated JSON mode, performance is more reliable when the prompt also explicitly requests a JSON output. This prompting requirement is a crucial detail for ensuring consistent results.

Here’s the why and when before we get into the how. Structured output (JSON, function calls, or typed models) isn’t just for building chat UIs; it’s for any place where model output will be consumed by code rather than eyeballs.

Imagine you need to implement the following for your application:

Routing a support ticket to the right queue.
Writing extracted entities into a database row.
Triggering a downstream tool with typed parameters.
Running evaluation harnesses that assert on fields.
Emitting telemetry that your observability stack can parse.

In these situations, you don’t want prose that a human can interpret – you want a predictable schema your program can validate, retry, and diff in tests.

In a typical product, end users almost never see JSON and don’t need to ask for it. The JSON is a behind-the-scenes contract between your backend and the model: the UI renders friendly text, while your server reads the machine-readable portion to decide what to do next (store, route, call a tool, show buttons, etc.).

Two common patterns for responses emerge:

Dual-channel responses where the model returns human-readable text for the UI plus a small JSON metadata object for the application.
Pure structured responses for batch/ETL jobs, agent loops, and evaluations, where no human prose is required at all.

So, use structured output whenever a downstream system needs determinism: when you must validate fields, run automated tests, support idempotent retries, or chain tool calls safely. Skip it when the output is purely human-facing and won’t be parsed by code. With DeepSeek specifically, you’ll get the best reliability by using its native structured modes (JSON or tools) and still stating the schema expectations in the prompt. For reasoning models, keep the machine-readable bits in the final answer channel only and never feed their hidden thinking back into the conversation; we’ll show exactly how to do that next.

There are two ways to get structure: make the model emit it, or have your client enforce it. We’ll cover four progressively stronger techniques to do this:

Native JSON mode asks the model to return a JSON object directly – simple, fast, and good for extractions and small schemas.
Function/tool calling lets you define a schema (parameters) the model must fill to call a tool, which is ideal when outputs should trigger code paths.
Type-enforced generation with Pydantic (via Instructor) validates the model’s reply against a Python type, giving you precise errors and automatic retries; we’ll show both Mode.TOOLS and Mode.MD_JSON (handy for reasoning models).
Production guardrails provide prompt-level schema hints, low temperature, max-token sizing, a retry-repair loop, and – when you need a hard guarantee – constrained generation on providers that support it.

All examples use the OpenAI-compatible chat-completions interface but target DeepSeek models/endpoints.

Let’s first look at how to set up the native JSON mode.

Native JSON mode

Setting up JSON mode looks deceptively familiar if you’re coming from OpenAI. Take a look at the following code snippet.


from openai import OpenAI

import json

client = OpenAI(

    api_key=os.getenv("DEEPSEEK_API_KEY"),

    base_url="https://api.deepseek.com"

)

response = client.chat.completions.create(

    model="deepseek-chat",

    messages=[

        {

            "role": "system",

            "content": "Extract user information as JSON. Example: {\"name\": \"Alice\", \"age\": 30}"

        },

        {

            "role": "user",

            "content": "My name is Bob and I'm 25 years old"

        }

    ],

    response_format={"type": "json_object"},

    temperature=0.1  # Keep it deterministic

)

data = json.loads(response.choices[0].message.content)

This snippet calls DeepSeek through its OpenAI-compatible API to extract structured data from a sentence. It creates a client (using DEEPSEEK_API_KEY and base_url), sends a chat request to the deepseek-chat model with a system instruction and a user message, and enables native JSON output via response_format={"type": "json_object"} while keeping randomness low (temperature=0.1). The model responds with a JSON string (e.g., {"name":"Bob","age":25}), which the final line parses with json.loads(...) into a Python dict (data) for programmatic use.

For optimal results when using response_format={"type": "json_object"}, adhere to the following guidelines:

Explicitly request JSON: The prompt should still contain a phrase like return the output as JSON to reinforce the instruction.
Provide a schema example: Including a simple JSON structure in the prompt (e.g., {"name": "string", "age": "number"}) significantly improves compliance.
Set max_tokens appropriately: Ensure the token limit is sufficient to accommodate the full JSON response to prevent truncation.
Use a low temperature: A temperature between 0.1 and 0.3 is recommended to minimize creative deviations from the requested schema.

It may happen that sometimes JSON is not enough. This is usually when there is complex logic or nested levels in your JSON object. At a certain point, you’ll start to get errors or unexpected outputs. In such cases, we can use function calling.

Function calling

For more complex structured data requirements, DeepSeek supports a function-calling feature that adheres directly to the OpenAI specification. This design choice simplifies integration significantly, as existing code bases built for OpenAI’s function-calling API can be adapted to use DeepSeek with minimal changes to the client configuration.


tools = [{

    "type": "function",

    "function": {

        "name": "analyze_sentiment",

        "description": "Analyze the sentiment of text",

        "parameters": {

            "type": "object",

            "properties": {

                "text": {"type": "string"},

                "confidence": {"type": "number", "minimum": 0, "maximum": 1}

            },

            "required": ["text", "confidence"]

        }

    }

}]

response = client.chat.completions.create(

    model="deepseek-chat",

    messages=[{"role": "user", "content": "I love this product!"}],

    tools=tools,

    tool_choice="auto"

)

In the OpenAI-compatible API that DeepSeek exposes, tool calling is the umbrella and function calling is the concrete tool type you define with a JSON-Schema parameters block. You register one or more tools (type function), the model decides which one to invoke, and it returns tool_call with a function name and JSON arguments. Your application then validates and executes that function. This is the same mental model we use for agents. We will touch on function calling and structured outputs in more depth, as well as how to combine them to create agents, in Chapter 6. Here, we stay focused on structured output: using a single function as a typed envelope to force predictable fields.

The function-calling example declares a schema that the model must fill, shifting structure from polite prose requests into a machine-checked contract. At run time, the model doesn’t emit free-form text; it emits a tool invocation with strictly typed arguments (numbers, enums, and nested objects), which you can validate, log, retry, and pass to real code. This reduces prompt drift, eliminates brittle regex parsing, and yields clean failure modes: when the output violates the schema, you can detect it deterministically and trigger a repair or a retry. The pattern is now industry-standard: OpenAI, Anthropic (tools), Google (function calling), Cohere (tool use), and Mistral all support schema-constrained calls. You should use JSON mode when you just need a blob of structured data; use function/tool calling when the output is an action with parameters, when schemas are deep/nested, or when you want the model to choose among multiple operations.

Instead of working with unstructured text responses, you can define clear Pydantic models that represent the desired output – such as objects, lists, or nested data – and Instructor ensures the model is followed. Let’s talk about it next.

Type-enforced generation with Pydantic (via Instructor)

As you may be aware, Instructor is a Python library (https://python.useinstructor.com/) that makes prompting large language models both structured and reliable by integrating directly with Pydantic. It automatically validates responses, retries on schema errors, and provides type safety, making prompts more robust and predictable. This approach not only streamlines how prompts are written but also transforms LLM outputs into dependable, strongly typed data structures that integrate seamlessly into real applications.

For DeepSeek, you can use Instructor to add a type-enforcement layer on top of either JSON mode or tool calling. You give it a Pydantic model; it handles prompting, parses the response, validates types/ranges/enums, and can auto-repair on failures – returning a real Python object rather than a raw JSON string. For reasoning models, Mode.MD_JSON keeps latency low (no tool round-trip) while still enforcing the schema; for tool-heavy flows, Mode.TOOLS plugs into function calling. Because Instructor is transport-agnostic, it can sit over both JSON and tools, and its value is validation, retries, and typed objects.

Let’s see how you can use Instructor for type safety.

Using Instructor for type safety

You can use Instructor to wrap DeepSeek’s API with Pydantic-powered validation. Here is an example to demonstrate this.


import instructor

from pydantic import BaseModel, Field

client = instructor.from_provider(

    "deepseek/deepseek-chat",

    base_url="https://api.deepseek.com",

    mode=instructor.Mode.TOOLS  # or Mode.MD_JSON for reasoning models

)

class ProductReview(BaseModel):

    rating: int = Field(ge=1, le=5, description="1-5 star rating")

    summary: str = Field(max_length=100)

    pros: list[str]

    cons: list[str]

# Magic happens here - guaranteed valid ProductReview or exception

review = client.chat.completions.create(

    messages=[{"role": "user",

         "content": "Review of iPhone 15: Great camera, battery life could be better..."}],

    response_model=ProductReview

)

Pro tip: Use Mode.MD_JSON with reasoning models to avoid tool-calling overhead. It’s like choosing the scenic route – slower (low latency) but more reliable.

That snippet wraps DeepSeek’s OpenAI-compatible API with Instructor, which enforces a Python Pydantic schema on the model’s output. from_provider(...) creates a client bound to deepseek-chat and chooses an enforcement mode (Mode.TOOLS in the example, which uses function/tool-calling under the hood; Mode.MD_JSON is the alternative that enforces a schema over plain JSON). The ProductReview Pydantic model is the contract: an integer rating constrained to 1–5, a summary capped at 100 chars, and two lists of strings. When you call client.chat.completions.create(..., response_model=ProductReview), Instructor does the following:

Presents the schema to the model (as a tool schema or JSON schema, depending on the mode).
Receives the model’s structured arguments.
Parses them.
Validates every field against the Pydantic constraints – returning a real ProductReview object on success or raising an error if the output is malformed.

In other words, you’re not asking for JSON; you’re guaranteeing typed data that your code can rely on.

Use this approach whenever downstream code needs strong guarantees or clean failure modes: writing to databases, triggering actions with parameters, running evals that assert on fields, or building agent steps that must be type-safe. Prefer Mode.MD_JSON with reasoning models when you want schema enforcement without the ceremony of tools; prefer Mode.TOOLS when you’re already in a tool/agent setting, when schemas are deep or nested, or when you want the model to choose among multiple tools with typed arguments. Either way, Instructor centralizes parsing and validation so your application logic stays simple and robust.

Reasoning-style DeepSeek models (e.g., deepseek-reasoner) emit two channels: hidden reasoning_content (the thinking) and the final content (the answer). Your validators and parsers must operate only on the answer – never echo or feed the model’s thinking back into the conversation or downstream calls, or you’ll trip schema checks and even 400s on the next request.

The next section covers these edge cases and hardening patterns – stripping the thinking channel, retry-repair loops, and other safeguards for production.

Strategies for robustness and special cases

Remember how we said R1 models think before they speak? That creates a unique challenge for structured output. The deepseek-reasoner model (https://api-docs.deepseek.com/guides/reasoning_model) outputs both reasoning_content (the thinking) and content (the answer). If you fail to handle this properly, you’ll get a nasty 400 error on your next API call.


# Assume `client` is the Instructor-wrapped DeepSeek client created above.

from pydantic import BaseModel

class DateInfo(BaseModel): date: str

class TimeInfo(BaseModel): time: str

# 1) First call  -  model "thinks" and answers; we keep only the answer (`content`)

completion, raw = client.chat.completions.create_with_completion(

    model="deepseek-reasoner",

    messages=[{"role": "user", "content": "Extract the date as JSON: The meeting is on March 15th."}],

    response_model=DateInfo,

    temperature=0

)

print(raw.choices[0].message.reasoning_content)  # optional: inspect the thinking

assistant_answer = raw.choices[0].message.content  # <-- keep ONLY this for history

# 2) Next call  -  what we actually feed the model (note: no `reasoning_content` anywhere)

next_messages = [

    {"role": "assistant", "content": assistant_answer},

    {"role": "user", "content": "Now extract the time as JSON: It starts at 14:30."}

]

_ = client.chat.completions.create(

    model="deepseek-reasoner",

    messages=next_messages,

    response_model=TimeInfo,

    temperature=0

)

The strip the thinking step is specific to R-series reasoning models because they return two channels: reasoning_content (hidden CoT) and content (the final answer).

If you naïvely replay the entire assistant message in your next request (e.g., by serializing the raw object), you end up sending a field the OpenAI-compatible API does not accept (reasoning_content). That violates the request schema, and the server responds with HTTP 400 Bad Request – it’s not an infinite loop, just an invalid payload. The remedy is simple: persist and replay only content, never reasoning_content. (Chat/coder models don’t emit reasoning_content, so this pitfall is unique to reasoning endpoints.)

We might encounter some issues while structuring output. Let’s talk about them.

The retry-repair loop: Even with all these safeguards, sometimes DeepSeek outputs invalid JSON. Instead of throwing up your hands, make it fix its own mess:


def extract_with_retry(prompt, model_class, max_retries=2):

    for attempt in range(max_retries):

        try:

            return client.chat.completions.create(

                messages=[{"role": "user", "content": prompt}],

                response_model=model_class

            )

        except Exception as e:

            if attempt < max_retries - 1:

                prompt = f"The previous JSON was invalid:\n{str(e)}\n\nPlease fix and return valid JSON for: {prompt}"

            else:

                raise

This helper implements a self-repair loop around a typed generation call. It asks the model to produce a value that conforms to a Pydantic schema (model_class) and immediately validates the reply; if validation (or JSON parsing) fails, the except block rewrites the prompt to include the exact error message and a plain instruction – the previous JSON was invalid… please fix and return valid JSON – then retries up to max_retries. In other words, you turn an unstructured failure into a new, highly constrained task for the model, which often succeeds on the second pass because it now sees what was wrong (e.g., missing required field, wrong type, trailing commas, truncated output).

Use this pattern anywhere you occasionally see schema drift despite good prompting: batch/ETL extractions from messy text, eval harnesses that assert on fields, agent steps that must hand off typed parameters, or R1 runs where the final answer sometimes leaks stray prose into the JSON. Keep retries small (2–3), log the exception, and pair the loop with low temperature, adequate max_tokens, and an inline schema/example; if failures persist, escalate (fallback to tool/function calling with stricter schemas or surface a human in the loop). Avoid pasting huge error dumps back to the model (sanitize and keep it short), and ensure that with reasoning models, you validate only the answer channel (content), never their hidden reasoning_content.

What do we mean by schema? There’s no server-side validator here. The model is conditioned on a schema you provide; your client code validates the result.

You can supply that schema in two common ways:

Native JSON mode: Ask for JSON and include a tiny example (or JSON-Schema-like hints) in the prompt – good for simple extractions.
Function calling: Pass an OpenAI-compatible JSON Schema under parameters. The model returns a tool_call whose arguments must match your declared properties (types, required fields, and nesting).

Use tool mode whenever downstream code needs typed parameters (e.g., routing, writing to a DB row, or triggering an action) or when your schema is nested.

The possessive apostrophe bug: In R1-0528, we sometimes see keys lose their apostrophes in JSON (e.g., user's_name → users_name). Property names that include apostrophes – especially the typographic U+2019 (') – are rare in training data and tokenized as punctuation, so the model often normalizes or drops them (e.g., emitting users_name instead of user's_name). That breaks strict validators and any code expecting an exact key match.

Fix it by doing either of the following:

Avoiding punctuation in property names (for example, prefer ASCII snake_case).
Showing explicit examples and accepting common variants during validation. Common variants can be captured via aliases and normalization:

Here is how you can do this:


# Tool schema with risky keys (curly apostrophes)  -  works but fragile:

tools = [{

  "type": "function",

  "function": {

    "name": "capture_profile",

    "parameters": {

      "type": "object",

      "properties": {

        "user's_name": {"type": "string"},

        "company's_address": {"type": "string"}

      },

      "required": ["user's_name", "company's_address"]

}}}]

# Robust parsing with Pydantic: accept curly, straight, or dropped apostrophes

from pydantic import BaseModel, Field, AliasChoices

class Profile(BaseModel):

    users_name: str = Field(validation_alias=AliasChoices("user's_name","user's_name","users_name"))

    company_address: str = Field(validation_alias=AliasChoices("company's_address","company's_address","company_address"))

    model_config = {"populate_by_name": True}  # allow using field names in your code

# Better yet: instruct the model to emit ASCII snake_case keys only:

# "Return JSON with keys: users_name, company_address (ASCII only, no punctuation)."

In practice, use tool/JSON schemas whenever you need typed, machine-checked fields and avoid punctuation in property names; if you can’t rename keys (legacy contracts), add alias choices and a light pre-normalization step (e.g., replace →) before validation. This keeps your pipeline resilient without hiding errors that matter.

Table 3.2 serves as a quick reference to structured output strategies you could employ.

Table 3.2: Structured output strategies for effective DeepSeek prompting

DeepSeek’s structured output capabilities have come a long way since the wild west days of pre-0528. While it’s not quite as polished as OpenAI’s implementation, it’s more than capable for production use, especially when armored with Instructor or similar libraries.

The key is understanding DeepSeek’s quirks: always provide examples, keep temperatures low, and remember that even with native JSON mode, the model appreciates (or even demands) a gentle reminder that you want JSON. It’s like working with a brilliant but slightly eccentric colleague. Once you learn their preferences, you can do amazing things together.

If DeepSeek can be finicky, why do teams still pick it? In practice, the trade-off is compelling:

Cost or performance is unusually strong: Per-million-token rates (with cache discounts) undercut many peers, so high-volume reasoning workloads are economically viable.
Reasoning quality is competitive on math/logic/code: So, you often need less prompt scaffolding to reach correct answers.
Low switching cost: The API is OpenAI-compatible (the same SDKs, function/tool calling, JSON mode), so existing backends slot it in with minimal changes.
Ecosystem support: Hosted options add constrained or grammar-based generation for stricter structure when you need it.
Optionality: R1(-0528) and its distilled variants are available on Hugging Face (and even vendor-quantized builds), which enables self-hosting and hybrid architectures. The quirks mostly translate to disciplined prompting (state the schema, keep temps low, strip reasoning channels), and for many teams, the savings plus throughput and optional self-hosting outweigh the extra care required.

Next up, how DeepSeek’s V-series models bring their own flavor of prompts.

V series: unique prompting techniques for V-series models

While the R-series models are optimized for complex reasoning tasks, the V-series models are designed as powerful, general-purpose conversational agents. They demonstrate high performance across a range of applications, but their behavior is governed by a distinct set of operational characteristics. To leverage their full capabilities and ensure predictable outcomes, it is essential to understand and adapt to these unique prompting requirements.

The template tango: Unicode characters and special tokens

Here’s where the V series gets properly weird. While the R series mostly ignores formatting niceties, V-series models are extremely particular about their chat templates – and they use Unicode characters that’ll make you question your sanity.

A chat template is the deterministic formatting layer that turns your structured messages =[...] into the exact byte sequence the model was trained to see. It inserts the model’s special markers (begin/end-of-sentence, role tags, turn separators), handles mode switches (e.g., V-series <think> versus </think>), and preserves required Unicode tokens (full-width |, low block _).

Using the official template keeps tokenization stable and quality high; hand-rolling strings (or mixing ASCII look-alikes) is brittle and can measurably degrade performance. In practice, let your provider SDK or transformer’s tokenizer.apply_chat_template(...) build the prompt, merge into one system message, and ensure assistant turns end with <|end_of_sentence|> so decoding stops cleanly.

Consider, for example, the following prompt template, as used when the thinking mode is on and for a single-turn prompt.


<｜begin_of_sentence｜>{system}<｜User｜>{query}<｜Assistant｜><think>

The characters in this template are not standard ASCII. The vertical bars (|) are full-width Unicode characters (U+FF5C), and what appears to be an underscore is a special block character (U+2581). Using the incorrect, visually similar ASCII equivalents (|, _) will result in tokenization mismatches and significantly degrade model performance. This distinction is crucial: though visually similar, the characters are functionally distinct to the model’s tokenizer.

Do I need to hand-roll these weird characters?

Usually, no. When you call the V series or R series through a chat API (OpenRouter, provider SDKs, etc.), the server applies the model’s chat template for you. You pass structured messages =[...] and (for V3.1) optionally a reasoning/think mode flag; the provider builds the correct prompt string with the full-width bars (|, U+FF5C) and the SentencePiece low block (_, U+2581) so you don’t accidentally swap them for the ASCII look-alikes.

If you’re running locally with Hugging Face Transformers, always use tokenizer.apply_chat_template(...) (and select thinking=True/False for V3.1) rather than concatenating strings yourself. This avoids subtle tokenization mismatches that can measurably hurt quality.

You can follow these V-series template rules while crafting our prompt templates (if you insist on setting these up manually):

Use the tokenizer’s apply_chat_template() method – don’t try to hand-craft this.
Always close assistant responses with <|end_of_sentence|> or risk infinite generation.
Have exactly one system message at the start (merging multiple system prompts degrades performance).

That’s how to speak to V series; now let’s manage how it speaks back – often with a little too much Markdown.

The formatting fiesta: When Markdown goes wild

V-series models, especially V3, have some quirks in terms of what they output.

Here is what you get by default:


"""

*Really* interesting point here.

**Another** observation.

A single sentence paragraph.

Yet *another* single sentence.

**Bold** conclusion with *emphasis*.

"""

# What you probably wanted:

"""

Here's an interesting point that flows naturally into the next thought.

The observation connects seamlessly with the previous context, creating

a cohesive narrative that doesn't feel like a TG from 1952.

The conclusion emphasizes key points without resorting to typographical

fireworks every other word.

"""

The fix? Be explicit about formatting preferences in your prompt:


system_prompt = """

Write in flowing, connected paragraphs. Avoid:

- Single-sentence paragraphs unless for emphasis

- Excessive use of *italics* or **bold** formatting

- Breaking thoughts into choppy segments

"""

eg7.py https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter03/eg7.py provides a compact A/B harness that isolates formatting as the only variable in a generation: it sends the identical topic prompt to the same DeepSeek V3 model twice – first with no style guidance, then with a short, system-level style spec – and prints the results back to back. Because the content, model, and topic are held constant, any differences are attributable to the role and specificity of the instruction, making the effect of a style policy immediately visible.

The constraints are phrased as crisp, testable rules rather than vague adjectives, and putting them in the system message applies them globally, which V3 tends to follow more reliably. The file even includes a captured transcript that highlights the markdown-heavy baseline versus smoother, connected prose, and you can swap the spec to test house styles like no lists, two paragraphs max, or executive tone.

The practical takeaway: Treat formatting as a controllable constraint you set up front, not a messy cleanup step after generation.

Now that the prose isn’t shouting, make sure the model can actually read it. The next failure mode is invisible: tiny default context windows and silent truncation.

The context window confusion

V-series models advertise massive context windows (163k+ tokens), but there’s a catch. The default configuration often shows 8k tokens, and many inference frameworks will silently truncate at that limit unless you explicitly set max_model_len like this:


# Wrong (silently truncates at 8k):

model = load_model("deepseek-v3")

# Right:

model = load_model("deepseek-v3", max_model_len=65536)

Pro tip: V series uses Multi-Head Latent Attention (MLA) to compress the KV-cache for efficient long-context inference. Independent of MLA, transformers tend to show a U-shaped positional bias in long prompts (lost in the middle), so put your most critical instructions at the beginning and end – don’t bury them mid-prompt.

Why do most runtimes cap V series to ~8k by default when the model advertises 100k+? Because context length is a budget, not a mandate. Long inputs are expensive in three ways:

Prefill latency grows roughly linearly with input length, so a 50–100k-token prompt can dominate end-to-end time even for short answers.
The KV cache grows linearly with length and eats GPU memory, which lowers throughput and hurts concurrency.
Tail-latency or timeout risk rises, especially in shared clusters. DeepSeek’s MLA makes this cheaper by aggressively compressing the KV cache (a reported ~93% reduction), but it doesn’t make the context length free – the cache still scales with L.

Providers and frameworks, therefore, ship conservative limits to keep fleets stable; if you actually need more, you must opt in by raising max_model_len, ideally with profiling to verify you aren’t blowing up latency or capacity.

Should you widen the window at all?

Only when those extra tokens are doing real work. More tokens does not mean better reasoning. A language model exhibits a U-shaped positional bias on long prompts (lost in the middle). The start and end of the prompt get disproportionately more attention. Stuffing 60k of undigested text can therefore hurt fidelity. Instead, you should opt for retrieval and summarization and keep instructions tight; expand the window when you truly must preserve long spans verbatim (contracts, long logs, multi-file code diffs) or sustain very long dialogues.

Pragmatically, ablate prompt length, measure accuracy versus latency/cost, and keep the smallest window that meets your quality bar.

With the window sized correctly and key instructions placed up front and at the end, you’re ready to use what the V series adds beyond raw reasoning.

It’s time to flip on the extras: FIM, native tool calls, and guaranteed JSON.

Hidden superpowers: Features the R series doesn’t have

While the R series focuses on reasoning, the V series picked up some tricks along the way:

Fill-in-the-Middle (FIM): FIM lets the model generate the missing middle between a given prefix and suffix, so you can edit code in-place rather than just append continuations. Here, hole simply means the placeholder middle span – the <``｜``fim_hole``｜``> region the model fills between the fixed prefix and suffix, leaving the surrounding code unchanged.
```
prompt = """<｜fim_begin｜>

def calculate_average(numbers):

<｜fim_hole｜>

    return sum(numbers) / len(numbers)

<｜fim_end｜>"""

# V-series will generate the missing validation logic
```

This prompt marks three regions: a prefix before the hole (the function signature), the hole where code should be produced, and a suffix after the hole (the return line). Given those anchors, the model infers what belongs in the middle – typically, input checks or guard clauses – so you’ll see it add validation such as handling empty lists, None, or non-numeric items before it returns the average. In other words, you’re asking the model to surgically fill the missing body while preserving the surrounding context verbatim.

Native tool calling: Native tool calling lets the model emit a structured function call (name + JSON args) instead of prose, so your runtime can execute a tool and feed the result back.


system = """## Tools

### Function: get_weather

Description: Get current weather

Parameters: {"location": "string"}

"""

# V-series will emit structured tool calls with special tokens

The system prompt declares a callable tool, get_weather, and its expected parameter schema. When the user asks a weather question, the model won’t answer in free text; it will produce a structured tool-call message (the schema varies by SDK) selecting get_weather with something like {"location": "Delft"}. Your application intercepts that call, runs the real tool, and returns the tool’s output as a tool-result message, after which the model composes the final user-visible answer. The key point: the model chooses when to call the function and surfaces well-formed arguments rather than guessing a URL or scraping text.

Guaranteed JSON mode: JSON-only/strict-JSON mode constrains decoding so the model returns syntactically valid JSON (and nothing else), which is ideal for programmatic consumption.
```
system_prompt += "\n## Response Format\nReply with JSON object ONLY."

# Forces valid JSON output without additional libraries
```

The added Response Format instruction tells the model to emit a single top-level JSON object with no commentary, code fences, or trailing text. In practice, you’ll pair this instruction with your client’s JSON/structured-output setting (if available) to hard-enforce validity; together, they reliably yield clean JSON that downstream code can parse directly. If you also provide a minimal schema or example, you further reduce drift and get stable keys and types.

The reasoning residue: Some V-series checkpoints occasionally surface internal thinking traces (e.g., <think>…</think>); these aren’t needed for correctness and are usually best suppressed. V2.5 inherited some reasoning behaviors from R1, occasionally producing unwanted thinking outputs:
```
User: What's 2+2?

Assistant: <think>This is a simple arithmetic problem...</think>

The answer is 4.
```

In the example, the assistant exposes an internal thought block before stating The answer is 4. That trace is a vestige of inherited reasoning behavior rather than an intentional feature.

In production, suppress it by instructing: Do not include <think> tags or intermediate reasoning; return only the final answer, and/or by post-filtering those tags from outputs. You still get the correct result, but without the verbose justification unless you explicitly request an audit trail.

V3 mostly eliminated this, but you might still see verbose justifications when you just wanted the answer. Unlike the R series, where reasoning is intentional, V-series reasoning traces are vestigial – suppress them unless you specifically need the audit trail.

Temperature and sampling: Lower is better. V series tends to have a narrower sampling sweet spot, so small temperature increases degrade format fidelity and factuality faster than with many other models. V-series models are more sensitive to temperature than most:

Table 3.3: Recommended temperatures for various V-series models

Compare this to OpenAI’s GPT-4o, where 0.7-1.0 is common, or the R series, which likes 0.6. V-series models seem to have a narrower sweet spot before outputs get weird. For structured or high-precision tasks, keep the temperature low (e.g., ~0.15–0.35) and adjust top_p modestly if you need variety; this keeps outputs stable and reduces the risk of verbose justifications or JSON breakage. If you later include a config snippet, you can note that the chosen values aim for near-deterministic decoding with just enough stochasticity to avoid repetition, and that raising the temperature should be done cautiously and incrementally.

Here is a quick recap of the different capabilities of the V series and R series of DeepSeek.

Table 3.4: Feature comparison between V-series and R-series DeepSeek models

The V-series models are powerful conversational agents that serve as a counterpart to the reasoning-focused R series.

Effective use of V-series models hinges on precise control over the prompt structure. This includes strict adherence to their specific chat template, which utilizes non-standard Unicode characters, and explicit configuration of parameters like max_model_len to enable their full context window.

Furthermore, developers must be prepared to manage the model’s distinct formatting tendencies and temperature sensitivity to ensure consistent and predictable outputs.

With the rails in place, you’re ready for the rare derailment. The next section maps symptoms to causes so you can correct course in a single pass.

Troubleshooting

Even the best-laid prompts can go sideways with DeepSeek. Here’s your field guide to the most common issues and their fixes – think of it as your emergency toolkit for when the model decides to be… creative.

Here are some of the common issues you can troubleshoot with easy fixes:

When R1 forgets to think:
- Symptom: R1 jumps straight to an answer without its signature reasoning phase, producing shallow or incorrect results.
- Fix: Force the thinking process by prepending the expected token:
```
# Nuclear option - literally start their response for them

messages.append({

    "role": "assistant",

    "content": "<think>\n"

})

# Then continue with normal generation
```

Here, messages is the same chat-history array you pass to your completion call. You append a synthetic assistant turn with content "<think>\n" as the final item in that array immediately before you invoke the model. Because the latest turn is from the assistant, the model treats it as the start of its own reply and continues generating right after <think>, which reliably kicks off a reasoning block. Don’t add another user turn after this prefill. If you surface outputs to end users, strip the <think>…</think> section (or use your provider’s option to hide reasoning) so they only see the final answer.

This technique effectively preconditions the model’s generation, ensuring that its internal reasoning process is initiated before it attempts to formulate a direct answer. The successful application of this method is confirmed by the presence of a reasoning trace in the model’s output preceding the final answer.

The multilingual mystery tour:
- Symptom: Your English prompt suddenly gets a response peppered with Chinese characters, especially during complex reasoning. Here’s an example:
```
The solution involves 计算 the derivative...
```
- Fix: Add explicit language constraints to your user prompt:
```
prompt += "\nRespond ONLY in English. Do not use any Chinese characters."
```

Why does this happen? DeepSeek was trained on multilingual data with a strong Chinese presence. Under cognitive load, it sometimes reverts to its “native” language. It’s like how bilingual speakers might switch languages when doing mental math. Note that this might not actually matter too much in terms of the final result or response you get back, though, if you are an English speaker, it might harm your ability to understand the reasoning trace.

Repetition loops, when DeepSeek gets stuck:
- Symptom: The model outputs the same phrase or pattern repeatedly:
```
Therefore... Therefore... Therefore... Therefore...
```
- Fix cocktail: You can follow these steps to fix repetitions. Add these extra config details when you are setting up your model to respond.
  1. Raise temperature slightly (if below 0.5): temperature=0.6.
  2. Add repetition penalty: frequency_penalty=0.3.
  3. Set reasonable max_tokens: Don’t let it ramble indefinitely.
  4. Use top_p sampling: top_p=0.95 helps avoid local optima.

Repetition usually means the model is stuck in a probability valley. You need just enough randomness to climb out without going full chaos mode.

Majority voting: When in doubt, democracy. For critical tasks where accuracy matters more than speed, use DeepSeek’s own benchmarking strategy:


from collections import Counter

def get_best_answer(prompt, n_samples=5):

    responses = []

    for _ in range(n_samples):

        response = client.chat.completions.create(

            model="deepseek-r1",

            messages=[{"role": "user", "content": prompt}],

            temperature=0.6

        )

        responses.append(extract_answer(response))

    # Return most common answer

    return Counter(responses).most_common(1)[0][0]

Where this code snippet lives: drop it in exactly where you currently make a single chat.completions.create(...) call – your inference layer (API handler, job, or eval harness). Instead of one call, you run n_samples independent calls with a non-zero temperature (e.g., 0.4–0.8) to create diversity, use extract_answer(...) to normalize each result (strip reasoning, trim whitespace, canonicalize JSON, or pull the value after Final Answer:), and then return the plurality winner via Counter(...).most_common(1)[0][0].

The loop in get_best_answer(...) does precisely that: collects n_samples answers, collapses them to comparable strings, and picks the most frequent. If two answers tie, Python’s Counter breaks ties by first-seen; in production, you may re-sample, raise n_samples, or apply a deterministic tie-breaker (e.g., shortest valid JSON or a schema validator score).

Expect latency and cost to scale roughly linearly with n_samples; to keep the wall-clock low, run the calls in parallel. Use this pattern for high-stakes queries where accuracy matters more than speed; if errors are systematic rather than random, fix the prompt/constraints first before relying on voting.

This isn’t just paranoia – DeepSeek’s own team used 64-sample majority voting (https://arxiv.org/pdf/2501.12948) for their benchmarks. For production, 3–5 samples usually suffice unless you’re doing competitive mathematics.

Format-specific gotchas: You may encounter some output related discrepancies with DeepSeek. Let’s talk about them:
- JSON mode dies silently: In some cases, when using JSON mode with very long or complex prompts, the API may return an empty response without an error. This can happen when the model struggles to generate a valid JSON object that fits within internal processing limits. The most reliable solution is to reduce the prompt length or to reset the conversation history, retrying the request with only the most recent message. The specifics of how you do this will depend on your use case or application, but the basic principle looks like this:
```
# Symptom: Empty response with response_format active

# Fix: Shorten prompt or reset conversation

if not response.content:

    # Start fresh conversation with shorter context

    messages = [messages[-1]]  # Keep only latest
```
- Truncated outputs: A common failure mode is receiving a syntactically incomplete JSON object because the generation was stopped by the max_tokens limit. The model does not account for this limit when generating its response, so it may be cut off abruptly. To prevent this, calculate an estimated response size and set max_tokens with a generous buffer, such as 20% larger than your estimate.
```
# Symptom: {"result": "answer", "confidence": 0.9

# Fix: Always pad max_tokens by 20%

estimated_tokens = len(prompt.split()) * 2

max_tokens = int(estimated_tokens * 1.2)
```

When DeepSeek misbehaves, run these checks in order:

Temperature: If the temperature is too low (≈0.3–0.5 or below), the model can repeat itself or collapse to generic text; if it’s too high (≈0.7 or above), outputs become erratic. For most production prompts, start in the 0.3–0.6 band and move deliberately.
Template integrity: The V series requires its exact chat template and the EOS marker <``｜``end_of_sentence``｜``>. A missing marker or ASCII look-alike characters will degrade quality or cause run-on decoding. Build prompts with the official template function rather than hand-concatenating strings.
Token limits: Mid-thought cut-offs usually mean max_tokens is too small; silent input truncation means your configured max_model_len is below the prompt length. Increase max_tokens, set max_model_len explicitly, and define a stop sequence so answers end cleanly.
Language control: If you need English-only output, state that explicitly at the top of the prompt and avoid multilingual examples in context. Mixed-language cues in few-shot or in-context invite leakage.
Reasoning engagement: If R1 skips the hidden reasoning phase, prefill the assistant with <think>\n or instruct: Think inside <think>…</think>; give the final answer after </think>. Strip or hide the reasoning before surfacing results to end users.
Context discipline: Very long contexts (≈100k+ tokens) reduce fidelity. Summarize or retrieve only what is necessary, and place critical instructions at the beginning and end rather than burying them in the middle.
If these checks don’t stabilize outputs, simplify the task: restate the goal, tighten the schema, and only then consider ensembling (e.g., majority vote).

A core principle to remember when debugging DeepSeek models is their high sensitivity to initial conditions. Unlike some other models, minor variations in prompt phrasing or a small adjustment to hyperparameters like temperature (e.g., by 0.1) can significantly alter the model’s output trajectory. Therefore, iterative, small-scale changes should be the primary method for resolving unexpected behavior.

A crucial diagnostic technique is to maintain a known-good baseline prompt. When encountering issues with a new or complex prompt, running this baseline allows you to determine whether the unexpected behavior stems from the prompt itself or a potential transient issue with the model or inference environment. This practice serves as a fundamental step in isolating the source of an error, analogous to a unit test for model behavior.

With a stable debug loop and a known-good baseline in place, it’s time to shift from firefighting to prevention: translating what you already have. The next section maps common patterns from GPT-5/Claude into DeepSeek-friendly idioms – what to keep, what to change, and what to drop.

Prompt migration guide

Moving your prompts from OpenAI’s GPT-4o, Claude, or other models to DeepSeek? What follows are some suggestions and ideas for how to migrate your prompts. These are intended as illustrative suggestions to inspire you rather than hard-and-fast rules.

Let’s say you have system prompts from other models:


messages = [

    {"role": "system", "content": "You are a helpful Python expert..."},

    {"role": "user", "content": "Explain decorators"}

]

Then, for the DeepSeek R series, the same prompt could be written as follows:


messages = [

    {"role": "user",

    "content": "You are a helpful Python expert. Explain decorators."}

]

For the DeepSeek V series, you need to specify minimal prompts:
```
# V-series accepts system prompts, but keep it minimal

messages = [

    {"role": "system", "content": "Python expert assistant"},

    {"role": "user", "content": "Explain decorators"}

]
```
Migration note: Merge your system instructions into the user prompt for the R series. For the V series, drastically simplify – think name tag, not biography.

Consider some few-shot examples from other models:


prompt = """

Example 1: Input: "happy" → Output: "positive"

Example 2: Input: "sad" → Output: "negative"

Example 3: Input: "angry" → Output: "negative"

Now classify: "excited"

"""

For DeepSeek (both series), these could be specified as follows:


prompt = """

Task: Classify the sentiment as 'positive' or 'negative'.

Expected format: Return only the classification word.

Classify: "excited"

"""

Migration note: Replace examples with clear task descriptions and format specifications. Your accuracy might actually improve – DeepSeek’s RL training means it often knows better than your examples.

If you have a high temperature for creativity, your settings may look as follows in other models:


# OpenAI's GPT-4o creative writing

temperature = 0.9

top_p = 0.95

For DeepSeek V and R series, you can specify these settings as follows:


# V-series creative tasks

temperature = 0.5  # Max recommended

top_p = 0.9

# R-series creative tasks (not recommended, but if you must)

temperature = 0.6

# Add explicit creativity instruction instead

prompt += "\nBe creative and explore unconventional ideas."

Migration note: DeepSeek gets drunk at high temperatures faster than other models. Compensate with explicit creativity instructions rather than randomness.

If you have very long prompts (>50k tokens) for other models, you can place your prompt as is:


# Just dump everything

context = load_100k_token_document()

prompt = f"Analyze this document:\n{context}\n\nQuestion: {query}"

But for DeepSeek, you need to position your prompt smartly:


# Structure and position strategically

prompt = f"""

<context>

{context[:30000]}  # Less critical info

</context>

<critical_context>

{context[30000:]}  # Most important parts

</critical_context>

<task>

{query}

</task>

"""

# Don't forget to set max length for V-series!

model_kwargs = {"max_model_len": 65536}

Migration note: DeepSeek’s attention mechanism (especially V series MLA) weighs later tokens more heavily. Put critical info near the end.

If you have CoT prompting, the prompt for other models may take the following format:


prompt = "Let's approach this step-by-step:\n" + problem

For the DeepSeek R series, you may simply state the problem for the R1 series:


# Just state the problem

prompt = problem

# R1 already thinks step-by-step internally

But for the DeepSeek V series, you may specify that it needs to follow the CoT approach:
```
# V-series benefits from CoT prompting

prompt = "Think through this systematically:\n" + problem
```
Migration note: R series has CoT built in; adding it is like wearing two pairs of glasses. V series still benefits from explicit reasoning requests.

If you have complex role-playing, specify the roles for other models:


system = """You are Shakespeare, the famous playwright.

You speak in iambic pentameter, use Elizabethan English,

make references to your plays..."""

For DeepSeek, you need to specify what you expect in the response:


# Simplify to output requirements

prompt = """Write a response in Shakespearean style:

- Use iambic pentameter where possible

- Include elizabethan vocabulary

- Reference classic plays if relevant

User query: {query}"""

Migration note: Focus on output characteristics rather than identity. DeepSeek performs better with “write like Shakespeare” than “you are Shakespeare.”

If you have JSON generation without native support for other models, you can specify the requirement as follows:


prompt = "Return a JSON object with name and age fields."

For DeepSeek (Pre-0528) models, you would need to specify the prompt, format, and an example to tell it how to process it:


prompt = """Return ONLY a valid JSON object.

Format: {"name": "string", "age": number}

Example: {"name": "Alice", "age": 30}

Extract from: {text}"""

For DeepSeek (Post-0528), you need to specify the format and provide a clear specification in the prompt:
```
# Use native JSON mode

response_format = {"type": "json_object"}

# But STILL mention JSON in the prompt!

prompt = "Extract as JSON: {text}"
```
Migration note: Even with native JSON support, DeepSeek likes verbal confirmation. It’s like a safety mechanism that needs both a key and a button.

Before you migrate, ask yourself the following:

Am I using few-shot examples? → Convert to task descriptions.
Is my temperature > 0.7? → Lower it and add creativity instructions.
Do I have system prompts? → Merge into user prompt (R1) or simplify (V3).
Am I prompting for CoT? → Remove for R1; keep for V3.
Is my prompt > 50k tokens? → Structure it with critical info last.
Do I use role-playing? → Convert to output specifications.

When in doubt, simplify. DeepSeek models – especially R-series models – are like expert consultants. Give them the problem clearly and get out of their way. Your elaborate prompting strategies from other models might actually be holding DeepSeek back.

Remember: Migration isn’t just find-and-replace. It’s an opportunity to rethink whether all that prompt engineering complexity was really necessary. Often, with DeepSeek, it isn’t.

Summary

DeepSeek prompting demands a mindset shift: stop scripting how to think and instead state the objective crisply, the constraints explicitly, and the output schema unambiguously – then trust the model’s internal reasoning. Techniques that help elsewhere can backfire here: few-shot demonstrations often reduce accuracy, whereas clear task statements with tight schemas consistently perform better.

Operationally, put instructions in the right place and respect the wiring. R-series models largely ignore the system role, so keep guidance in the user message; V-series models tolerate a single, high-level system prompt for global constraints. Follow each family’s native chat template exactly (including the non-standard Unicode separators), and tune sampling parameters deliberately: both lines have relatively narrow good bands for temperature and top-p. Internalize these habits and you’ll move from merely using DeepSeek to shipping reliable, production-grade systems – after a little unlearning, you’ll gain access to a markedly stronger class of automated reasoning.

In the next chapter, we will discuss some use cases where DeepSeek can be applied.

Get This Book’s PDF Version and Exclusive Extras

Part 2 Using DeepSeek

In the second part of this book, we move from theory to practice with real-world case studies that show how DeepSeek is already driving innovation across industries. Next, we get hands-on with a practical build: an API powered by DeepSeek. You’ll follow the end-to-end process - designing the application logic, running it locally, and deploying it to Amazon Web Services. In the final chapter of this part, we introduce agents and demonstrate how to build robust agents backed by DeepSeek models.

By the end of this part, you’ll be equipped to solve common business problems with DeepSeek, whether you’re creating a simple web application or a full-fledged agentic system.

This part of the book includes the following chapters:

Chapter 4, Using DeepSeek: Case Studies
Chapter 5, Building with DeepSeek
Chapter 6, Agents with DeepSeek

Stay tuned

To keep up with the latest developments in the fields of Generative AI and LLMs, subscribe to our weekly newsletter, AI_Distilled, at https://packt.link/8Oz6Y.

Join our communities on Discord and Reddit

Have questions about the book or want to contribute to discussions on Generative AI and LLMs?

Join our Discord server at https://packt.link/4Bbd9 and our Reddit channel at https://packt.link/wcYOQ to connect, share, and collaborate with like-minded enthusiasts.

| | |

4 Using DeepSeek: Case Studies

By now, you have a fair idea of how DeepSeek functions, its internals, and its role in the global large language model (LLM) landscape. In this chapter, we will examine how DeepSeek is already transforming industries with real-world applications. Major players such as Microsoft, GitHub, and Fortune 500 companies are adopting LLMs such as DeepSeek to boost productivity, cut costs, and drive innovation. Through our own experience building production-ready applications, we’ve seen firsthand how DeepSeek enhances development velocity, improves code quality, and empowers better strategic decisions.

Some of the notable use cases where DeepSeek can be employed are automating trading strategies in financial firms, accelerating contract analysis for the legal teams, and modernizing legacy systems for developers. In this chapter, we will focus on the document understanding and document extraction use case.

We’ll examine these comprehensive case studies to demonstrate DeepSeek’s practical applications and how you can apply the structured approach to implement AI-assisted workflows. The Cursor IDE is our choice of platform to configure and use DeepSeek. However, the techniques we cover apply to any platform that supports DeepSeek, from traditional IDEs to cloud tools or local interfaces. If you don’t know where to get started yet, we have created an Appendix for you that briefs you about various ways you can use DeepSeek.

For the case studies in this chapter, we follow a systematic methodology by defining clear problem statements, crafting effective prompts, evaluating responses against both qualitative and quantitative metrics, and iterating to achieve production-quality results.

In this chapter, we will cover the following main topics:

Benchmarking tools setup and prompt design
Use case study: Document understanding
Use case study: Financial document understanding and benchmarking

Technical requirements

In this chapter, we demonstrate DeepSeek’s capabilities using the Cursor IDE, an AI-powered development environment tightly integrated with DeepSeek’s models. You’ll first need access to DeepSeek’s API. Start by creating an account at https://platform.deepseek.com/ and generating your API keys. Once you have your credentials, it’s worth verifying your access with a simple test call to ensure everything is configured correctly:


curl -X POST "https://api.deepseek.com/v1/chat/completions" \

-H "Authorization: Bearer YOUR_API_KEY" \

-H "Content-Type: application/json" \

-d '{

  "model": "deepseek-reasoner",

  "messages": [{"role": "user", "content": "Hello, DeepSeek!"}]

}'

With API access confirmed, we can proceed to establish the development environment that will serve as our laboratory for exploring DeepSeek’s practical applications.

Setting up your development environment

To set up an effective AI-assisted development environment, we will use the following:

Python 3.11+ for backend AI and data tasks.
Docker Desktop (https://docs.docker.com/desktop/).
Git for version control (https://git-scm.com/).
Node.js 18+.
The Cursor IDE, which can be downloaded from https://cursor.com/.

Configuring DeepSeek in the Cursor IDE

Configuring DeepSeek with the Cursor IDE is a straightforward process. To integrate DeepSeek with Cursor, you need to follow these steps to get started:

Access the Cursor model configuration through the Settings menu:

Figure 4.1: Cursor IDE DeepSeek-R1 model selection interface

On macOS, you’ll find this under Cursor > Settings.
If you are a Windows or a Linux user, you can locate it under File > Settings.

Within the Settings interface, the Models section in the left sidebar contains all the options for managing AI model integrations.

The Cursor IDE comes with built-in support for DeepSeek models, which you can enable by scrolling through the available model list and selecting deepseek-r1-0528.
Once enabled, the model appears in your available options, ready for configuration with your specific API credentials.
The configuration process requires entering your DeepSeek API credentials along with the appropriate connection settings.
You’ll specify deepseek-r1-0528 as the model name, provide your API key from the DeepSeek platform, set the base URL to https://api.deepseek.com/v1, and configure the provider as OpenAI Compatible to ensure proper API communication protocols.
To verify that your integration works correctly, open a new file in Cursor and invoke the AI command palette using Cmd + K on macOS or Ctrl + K on Windows and Linux.
Select DeepSeek-R1 from the model dropdown and test with a substantive prompt such as Explain the difference between microservices and monolithic architecture. A successful response confirms that your development environment is ready for the advanced use cases we’ll explore.

For more details on using DeepSeek with Cursor and other options, please refer to the Appendix.

Organizing your development workspace

A clear workspace keeps you productive and organized as projects grow more complex. For this chapter, set up a Chapter04 folder with the required subfolders. For example, we have set up our folders to be called brainstorming-ideation, document-analysis, and code-generation, as follows:


# Navigate to the DeepSeek-in-Practice directory

cd /Users/<user-id>/workplace/DeepSeek-in-Practice-main

# Create Chapter04 directory and subdirectories

mkdir Chapter04

cd Chapter04

mkdir 01-brainstorming-ideation

mkdir 02-document-analysis

mkdir 03-code-generation

# Initialize Git repository for version control

git init

Each subdirectory will contain the complete artifacts from its respective case study, including source code, configuration files, documentation, and any supporting resources.

Configuration management

Proper credential management forms the foundation of any professional development workflow, particularly when integrating with external AI services. Create a dedicated environment configuration file in your Chapter04 directory to store sensitive information securely:


# Navigate to Chapter04 directory

cd /Users/<user-id>/workplace/ DeepSeek-in-Practice-main/Chapter04

# Create .env file

cat > .env << EOF

# DeepSeek Configuration

DEEPSEEK_API_KEY=your_api_key_here

DEEPSEEK_BASE_URL=https://api.deepseek.com/v1

# Database Configuration (we'll use these later)

DATABASE_URL=sqlite:///./development.db

# API Configuration

API_HOST=localhost

API_PORT=8000

# Logging

LOG_LEVEL=INFO

EOF

This configuration approach separates sensitive credentials from source code, enabling secure sharing and deployment practices. Remember that environment files should never be committed to version control systems; always add .env to your .gitignore file immediately after creation to prevent accidental exposure of sensitive information.

All benchmarking scripts, test documents, and evaluation frameworks are available in our GitHub repository for readers interested in reproducing the results or conducting their own experiments:

https://github.com/PacktPublishing/DeepSeek-in-Practice/tree/main/Chapter04/document-understanding.

The repository includes the following:

Ground truth datasets for accuracy measurement.
Automated benchmarking comparison tool.
Complete test scripts for all three tools:
- DeepSeek-R1: LLM-based semantic extraction via direct API calls https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter04/document-understanding/test_deepseek_parsing.py.
- Docling (IBM Research): Document structure analysis and table recognition https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter04/document-understanding/test_docling_parsing.py.
- MarkItDown (Microsoft): LLM-optimized document preprocessing https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter04/document-understanding/test_markitdown_parsing.py.
Detailed evaluation framework documentation.

Benchmarking tools setup and prompt design

For the quantitative evaluation and benchmarking exercises in this chapter, we will test DeepSeek-R1 against specialized document parsing tools to provide comprehensive performance comparisons. We’ve created automated testing scripts that measure accuracy, processing time, and cost metrics across different approaches.

These are the tools used for benchmarking:

DeepSeek-R1: LLM-based semantic extraction via direct API calls.
Docling (IBM Research): Document structure analysis and table recognition.
MarkItDown (Microsoft): LLM-optimized document preprocessing.

To set up the benchmarking environment:


# Install benchmarking dependencies

pip install requests tabulate pandas

# Optional: Install comparison tools

pip install docling markitdown

Before we begin with our case studies, we will share a few quick tips on how to design prompts for our case studies:

Provide rich business context: Frame prompts with clear details about the business environment, constraints, aspirations, and objectives. Avoid generic queries such as help us innovate.
Focus on actionable outcomes: Structure prompts to elicit feasible, strategic outputs rather than abstract ideas.
Leverage DeepSeek-R1’s reasoning strengths: Use structured prompts that require systematic analysis.
Provide clear and detailed information: Include industry data, competitive insights, and clear success metrics. Present specific challenges with defined parameters and timeline constraints.
Encourage trade-off evaluation: DeepSeek-R1 performs best when tasked with problems involving multiple variables, risks, and strategic trade-offs – ideal for generating comprehensive innovation strategies.

All company names, scenarios, and code examples in this chapter are fictional and created specifically for educational purposes. Any resemblance to real companies, systems, or proprietary code is purely coincidental. The legacy code examples are based on common patterns found in legacy systems but do not represent any actual production systems. All code is original content created for this book and is provided under open source principles for educational use.

Content source clarification: This chapter contains actual prompts and responses from our DeepSeek-R1 interactions, clearly marked with formatting indicators. Any potential follow-up or enhanced prompting suggestions are illustrative examples of how iterative prompting could work, not actual executed prompts. All real DeepSeek-R1 interactions are explicitly labeled as such.

Let’s now begin with our case study in document understanding.

Use case study: Document understanding

CloudTech Solutions, a growing enterprise software company, faces a critical challenge in implementing a modern serverless architecture. Their development team has been tasked with migrating from a legacy monolithic API server to Amazon API Gateway and AWS Lambda functions to support 10,000+ requests per second across 15+ geographic regions while maintaining 99.99% uptime.

The company’s current situation reflects a common enterprise challenge: they have comprehensive technical documentation (a 45-page Amazon API Gateway integration guide) but need to quickly extract actionable implementation strategies from complex technical specifications. The development team must analyze this documentation to create a structured implementation plan that addresses architecture design, security requirements, performance optimization, and operational procedures.

The business impact is significant. CloudTech Solutions must complete this migration within a tight timeline to meet growing customer demands and reduce infrastructure costs by 40%. The challenge extends beyond just understanding the documentation – they need to transform high-level architectural concepts into specific technical decisions, implementation priorities, and production-ready code that their development team can immediately execute.

Our exploration of this document analysis challenge demonstrates how DeepSeek-R1 can process complex technical documentation and provide structured implementation guidance. This use case showcases the model’s ability to extract key architectural components, prioritize implementation tasks, identify critical configuration decisions, and generate actionable technical requirements from comprehensive enterprise documentation.

Prompting DeepSeek

For technical document analysis tasks, DeepSeek-R1 requires structured prompts that clearly define the analysis objectives and expected deliverables. The key is providing comprehensive context about the document type, specific technical requirements to extract, and the format for actionable outputs. Unlike general document review, technical analysis prompts should focus on implementation-ready insights and structured decision frameworks.

When working with enterprise technical documentation, it’s crucial to specify architectural considerations, implementation priorities, security requirements, and operational constraints. DeepSeek-R1’s reasoning capabilities excel when asked to not just extract information, but also synthesize complex technical concepts into actionable implementation strategies that development teams can immediately execute.

Response evaluation

Let me demonstrate with CloudTech Solutions’ Amazon API Gateway documentation analysis challenge using the actual prompt and response from our case study.

Note

The following analysis is based on a comprehensive 45-page Amazon API Gateway integration guide (source: Chapter04/document-understanding/amazon_api_gateway_integration_guide.md)

Here is our prompt to DeepSeek-R1 for the technical document analysis:


I need you to analyze this Amazon API Gateway Integration Guide and provide a comprehensive implementation strategy. This is enterprise-grade documentation that our development team needs to implement for a production system handling 10,000+ requests per second.

Please provide:

1\. **Architecture Summary**: Extract the key components and their relationships from the document. What are the core services and how do they interact?

2\. **Implementation Priority Matrix**: Based on the document, what should be implemented first, second, third? Consider dependencies, risk, and business value.

3\. **Critical Configuration Analysis**: Identify the most important configuration decisions that could impact security, performance, or cost. Extract specific values and settings.

4\. **Code Generation Plan**: Which Lambda functions, API configurations, and infrastructure components need to be created? Provide a structured implementation plan.

5\. **Security & Compliance Requirements**: Extract all security requirements, authentication flows, and compliance considerations. How should these be implemented?

6\. **Production Readiness Assessment**: What monitoring, error handling, and operational procedures are specified? Create a production deployment checklist.

Focus on extracting actionable technical requirements that a development team can immediately implement. Convert high-level concepts into specific technical decisions.

[Attached: amazon_api_gateway_integration_guide.md - 45 pages of technical documentation]

DeepSeek’s response to this prompt is as follows:


Based on my comprehensive analysis of the 45-page Amazon API Gateway Integration Guide, here's the structured implementation strategy for your production system:

1\. Architecture Summary

The system implements a serverless architecture using Amazon API Gateway and Lambda functions with the following core components:

Core Services: - API Gateway: Fronts the entire system, handles routing, authentication, throttling, and request/response transformation - Lambda Functions: Serve as the backend for different services (User, Order, Payment, etc.) - Data Stores: DynamoDB (Users, Payments), RDS PostgreSQL (Orders), SQS (async processing), S3 (analytics) - Authentication: AWS Cognito for OAuth 2.0/OIDC, with multi-factor authentication - Caching: Redis (via ElastiCache) and CloudFront for CDN caching - Monitoring: CloudWatch, X-Ray, and custom metrics - Security: AWS WAF, security headers, and encryption at rest and in transit

System Interaction Flow:

Clients → CloudFront → AWS WAF → API Gateway → Lambda Authorizer (Cognito)

→ Lambda Functions (User, Order, Payment) → Databases (DynamoDB, RDS, etc.)

Async flows: API Gateway → SQS → Async Lambda Processor

Monitoring: API Gateway and Lambda → CloudWatch logs and X-Ray traces

2\. Implementation Priority Matrix


Priority


Component


Reason

|
| --- | --- | --- |
|


Core Infrastructure


Set up VPC, IAM roles, security groups, and basic networking

|
|


Authentication & AuthZ


Without auth, no secure access. Cognito setup and Lambda Authorizer

|
|


User Service


Basic user management required for other services (e.g., orders)

|
|


Order Service


Core business functionality

|
|


Payment Service


Critical for transactions, but depends on User and Order

|
|


Async Processing


For non-critical background tasks

|
|


Monitoring & Logging


Essential for production, but can be built alongside

|
|


Caching & Optimization


Performance improvements after core functionality


3\. Critical Configuration Analysis

Authentication Configuration: - Cognito User Pool: MFA required, password policy (12+ chars, mixed case, numbers, symbols) - JWT Token: Must include custom claims (tenant_id, role) for multi-tenancy - Lambda Authorizer: Must validate JWT and apply tenant-specific and role-based access control

API Gateway Configuration: - Use HTTP API for new services (cost-effective) and REST API for legacy compatibility - Stages: Development (throttling: 100 burst, 50 rate) and Production (5000 burst, 2000 rate) - Custom domains: api.example.com with TLS 1.2+ and ACM certificates - Request/Response: Validation schemas for inputs, standardized response templates

Security Configuration: - WAF rules: SQL injection, XSS, rate limiting (100 req/min per IP), geo-blocking - Headers: Strict-Transport-Security, X-Content-Type-Options, X-Frame-Options - Encryption: KMS for DynamoDB, RDS, and S3; TLS 1.2+ for transit

Rate Limiting: - Global: 10,000 RPS (burst) in production - Per client: Usage plans (Free, Premium, Enterprise) with tiered limits

4\. Code Generation Plan & Implementation Strategy

Lambda Functions (each as a separate service): - User Service: CRUD operations, profile management with DynamoDB integration - Order Service: Order lifecycle management with RDS PostgreSQL backend - Payment Service: Payment processing, refunds with secure tokenization - Async Processor: Handles SQS messages for background jobs and notifications

Infrastructure as Code (CloudFormation): - API Gateway (HTTP API and REST API configurations) - Lambda functions with environment variables, VPC config, and security layers - DynamoDB tables with KMS encryption and backup policies - RDS instance (PostgreSQL) with backup, multi-AZ, and read replicas - SQS queues (main and dead letter queues) - Cognito User Pool and Client with MFA configuration - WAF rules and association with API Gateway - CloudWatch Alarms, Dashboards, and X-Ray configuration

5\. Security & Compliance Implementation

Authentication Flow:

Client Request → API Gateway → Lambda Authorizer → Cognito Validation

→ JWT Claims Extraction → Role-Based Access Control → Service Authorization

Compliance Requirements: - SOC2 Type II: Automated security controls and audit logging - PCI DSS: Secure payment processing with tokenization - GDPR: Data residency controls and user consent management - Multi-tenancy: Tenant isolation at database and application layers

6\. Production Readiness Checklist

Monitoring & Observability: - CloudWatch metrics for API Gateway, Lambda, and database performance - X-Ray distributed tracing for request flow analysis - Structured logging with correlation IDs for debugging - Custom business metrics for SLA monitoring

Error Handling & Resilience: - Circuit breaker patterns for external service calls - Exponential backoff and retry logic for transient failures - Dead letter queues for failed message processing - Graceful degradation for non-critical service failures

Operational Procedures: - Blue/Green deployment using Lambda aliases and weighted traffic - Automated rollback triggers based on error rate thresholds - Capacity planning based on request patterns and growth projections - Disaster recovery procedures with RTO/RPO targets

DeepSeek-R1 transformed a 45-page Amazon API Gateway specification into structured implementation guidance, demonstrating sophisticated technical analysis capabilities that enable development teams to execute complex enterprise projects efficiently.

Evaluation methodology for technical analysis

For technical document analysis, evaluation focuses on implementation accuracy and operational readiness:

Qualitative metrics: Emphasize architectural understanding, implementation feasibility, and production readiness considerations critical for enterprise system development.
Quantitative metrics: Measure technical specification completeness, configuration detail accuracy, and implementation artifact coverage.

To provide an objective, repeatable assessment of DeepSeek-R1’s document analysis capabilities, we have developed a systematic evaluation framework that measures both qualitative understanding and quantitative completeness. This section explains our scoring methodology in detail, demonstrating how we assigned scores and performed analysis for the Amazon API Gateway case study.

Let’s talk about this in detail:

Our evaluation process began with ground truth establishment, where we created the AmazonAPI Gateway integration guide spanning 45 pages and 1,743 lines as our source document containing specific architectural components, configuration values, and implementation requirements. This document served as the authoritative reference against which we measured DeepSeek-R1’s analysis accuracy.
Before evaluating DeepSeek-R1’s response, we manually cataloged all critical elements that a complete analysis should identify. This catalog included eight major serverless components, including API Gateway; Lambda functions for user, order, payment, and async processing services; DynamoDB; RDS PostgreSQL; Amazon Cognito; CloudWatch with X-Ray for monitoring; SQS for message queuing; and AWS WAF for security.
We also identified 23 specific configuration parameters, including throttling limits with burst rates of 5,000 for production and 100 for development; rate limits of 2,000 and 50, respectively; MFA requirements; password policy rules requiring 12 or more characters with mixed casing, numbers, and symbols; TLS version requirements of 1.2 or higher; rate limiting of 100 requests per minute per IP address; and custom domain settings.
Additionally, we cataloged four compliance requirements covering SOC2 Type II, PCI DSS, GDPR, and multi-tenancy isolation, along with six implementation deliverables encompassing CloudFormation templates, Lambda functions, IAM policies, monitoring dashboards, security configurations, and deployment procedures.
After establishing our ground truth, we performed manual comparison and scoring by evaluating DeepSeek-R1’s response against our reference element by element. For each evaluation criterion, we counted how many expected elements were correctly identified, understood, and explained.

Our qualitative metrics employ a 1 to 10 scale, as follows:

Scores of 9–10 indicate excellent performance with near-complete understanding and 90–100% of expected elements correctly identified and explained with appropriate context.
Scores of 7–8 represent good performance with a strong understanding covering 70–89% of elements and only minor gaps in depth or specificity.
Acceptable performance receives scores of 5–6, indicating adequate understanding with 50–69% coverage, but missing important details.
Below average performance, scoring 3–4, demonstrates partial understanding with 30–49% coverage and significant gaps.
Poor performance with scores of 1–2 shows minimal understanding, covering less than 30% of expected elements.

To demonstrate our scoring process concretely, we’ll examine how we evaluated architecture comprehension, which received a score of 9 out of 10:

DeepSeek-R1 correctly identified all eight major serverless components, including API Gateway, Lambda functions, DynamoDB, RDS, Cognito, CloudWatch with X-Ray, WAF, and SQS.
The model accurately described the system interaction flow where clients connect through CloudFront, pass through WAF security filtering, reach API Gateway, undergo authentication via the Lambda authorizer, and finally, invoke appropriate Lambda functions that interact with databases.
DeepSeek-R1 correctly explained the asynchronous processing path where API Gateway sends messages to SQS, which triggers the async Lambda processor, and properly identified the monitoring flow where services send data to CloudWatch logs and X-Ray traces.

We measured document understanding through several dimensions:

For component relationships, DeepSeek-R1 demonstrated a clear understanding that API Gateway fronts the entire system and intelligently routes requests to appropriate Lambda functions based on service type, whether User, Order, or Payment services.
Regarding integration patterns, the model correctly identified that Cognito handles initial authentication before the Lambda authorizer validates JWTs to enforce tenant isolation in the multi-tenant architecture.
For data flow logic, DeepSeek-R1 understood the architectural decision that different services use different data stores, with DynamoDB serving User and Payment services due to their key-value access patterns, while RDS PostgreSQL serves the Order service, requiring relational data structures and complex queries. The score was not a perfect 10 due to one minor gap where the model did not explicitly mention the ElastiCache/Redis layer documented in the source material, although the caching strategy was discussed in a different section of the response.

Here is a qualitative analysis of DeepSeek’s response.

Criteria	Analysis	Score (1–10)
Architecture comprehension	Correctly identified all major serverless components, their relationships, and interaction patterns. Demonstrated a clear understanding of API Gateway, Lambda, and supporting services integration.	9
Implementation practicality	Provided specific configuration values, security settings, and deployment parameters that development teams can directly implement without additional research.	9
Security awareness	Comprehensive coverage of authentication flows, compliance requirements (SOC2, PCI DSS), and security best practices with specific implementation guidance.	8
Production readiness	Addressed operational concerns, including monitoring, error handling, and disaster recovery procedures often overlooked in initial implementations.	8
Priority sequencing	Logical implementation ordering considering dependencies, risks, and business value with a clear rationale for each phase.	8

Table 4.1: Qualitative analysis of DeepSeek response

For quantitative assessment, we employ objective counting and percentage-based scoring using a straightforward formula where the score equals the number of elements correctly identified divided by the total expected elements, multiplied by 10, then rounded to the nearest whole number.

For example, correctly identifying all 23 parameters yields a perfect score of 10 out of 10, while identifying 20 out of 23 parameters produces a score of 9 out of 10, representing 87% accuracy.

We demonstrated this quantitative methodology through the configuration parameters evaluation, which scored 9 out of 10. We identified 23 specific configuration parameters in the source documentation and systematically checked whether DeepSeek-R1’s response included each one.

The model successfully identified the following:

Throttling burst limits of 100 for development and 5,000 for production.
Throttling rate limits of 50 and 2,000, respectively.
MFA requirement set to ON, password minimum length of 12 or more characters, and password complexity requirements for mixed case letters, numbers, and symbols.
JWT custom claims for tenant_id and role, TLS version requirement of 1.2 or higher, WAF rate limiting of 100 requests per minute per IP address, and global RPS limit of 10,000 burst capacity in production environments, among others.

In total, DeepSeek-R1 correctly identified 21 of the 23 expected parameters. The 2 missing parameters were specific timeout values for Lambda functions with a maximum of 30 seconds, and CloudFront TTL settings for the caching layer. The calculation of 21 correctly identified out of 23 expected yields 91.3% accuracy, which rounds to a score of 9 out of 10.

This detailed scoring methodology ensures that our evaluation is transparent, repeatable, and objective. For subsequent evaluations in this chapter, we apply the same framework but provide summarized results rather than explaining the methodology again.

Summing up the scores assigned in the Score column, the qualitative score is computed to be 42/50 (84%).

The response demonstrates strong technical architecture understanding with immediately actionable implementation guidance. High scores in comprehension and practicality reflect the analysis quality that enables direct development team execution.

The quantitative analysis is shown in the following table:

Table 4.2: Quantitative analysis of DeepSeek response

The quantitative score can be computed as 45/50 (90%), based on which we can infer that DeepSeek demonstrated the following:

Exceptional technical specification coverage with comprehensive implementation details.
Strong performance across all technical delivery categories demonstrates enterprise-grade analysis depth.

The combined score (qualitative and quantitative) is computed as 87/100 (87%).

DeepSeek-R1 delivers enterprise-grade technical analysis that transforms complex documentation into actionable implementation roadmaps. The response successfully balances architectural understanding with practical implementation guidance, providing development teams with immediately executable technical specifications.

DeepSeek also demonstrates implementation efficiency. While the traditional analysis may require 36 hours of specialized expertise ($6,000–$9,000), DeepSeek completed it in minutes, representing a 99%+ time reduction while maintaining professional technical standards.

Follow-up code generation request

After receiving the comprehensive analysis, CloudTech Solutions requested the actual implementation code. Here’s the follow-up prompt and DeepSeek-R1’s response:


Generate production-ready implementation code for the top 3 priority components from your analysis:

This generated the following response from DeepSeek:


Based on your analysis, generate production-ready code for the top 3 priority components:

1\. **Core Infrastructure (CloudFormation Template)**:

   - VPC, security groups, IAM roles

   - API Gateway with proper throttling and security

   - Cognito User Pool with MFA and custom attributes

2\. **User Service Lambda Function**:

   - Full CRUD operations with DynamoDB

   - Proper error handling and logging

   - Multi-tenant support with tenant isolation

3\. **Lambda Authorizer**:

   - JWT validation with Cognito

   - Custom claims extraction (tenant_id, role)

   - Enforces tenant isolation

   - Returns proper IAM policies

For each component, provide:

- Complete, deployable code

- Configuration parameters based on the documentation

- Security best practices implementation

- Error handling and resilience patterns

- Monitoring and logging integration

Focus on production-ready code that follows the security, performance, and compliance requirements from the documentation analysis.

Let’s take a look at DeepSeek-R1’s code implementation response.

DeepSeek-R1 provided three complete, production-ready implementations:

Core Infrastructure (CloudFormation Template)


AWSTemplateFormatVersion: '2010-09-09'

Transform: AWS::Serverless-2016-10-31

Description: Core infrastructure for API Gateway implementation

Parameters:

  Environment:

    Type: String

    Default: dev

    AllowedValues: [dev, staging, prod]

  SystemName:

    Type: String

    Default: enterprise-api-system

Resources:

  # API Gateway with security and throttling

  ApiGateway:

    Type: AWS::Serverless::HttpApi

    Properties:

      StageName: !Ref Environment

      DefaultRouteSettings:

        ThrottlingBurstLimit: !If [IsProd, 5000, 100]

        ThrottlingRateLimit: !If [IsProd, 2000, 50]

      Auth:

        Authorizers:

          CognitoAuthorizer:

            JwtConfiguration:

              issuer: !Sub "https://cognito-idp.${AWS::Region}.amazonaws.com/${UserPool}"

              audience: [!Ref UserPoolClient]

  # Cognito User Pool with security settings

  UserPool:

    Type: AWS::Cognito::UserPool

    Properties:

      UserPoolName: !Sub "${SystemName}-user-pool"

      Policies:

        PasswordPolicy:

          MinimumLength: 12

          RequireLowercase: true

          RequireUppercase: true

          RequireNumbers: true

          RequireSymbols: true

      MfaConfiguration: "ON"

      Schema:

        - Name: custom:tenant_id

          AttributeDataType: String

          Required: true

          Mutable: false

        - Name: custom:role

          AttributeDataType: String

          Required: true

          Mutable: true

User Service Lambda Function


import json

import boto3

import logging

from datetime import datetime

from aws_xray_sdk.core import xray_recorder

from botocore.exceptions import ClientError

# Initialize AWS services

dynamodb = boto3.resource('dynamodb')

table = dynamodb.Table('Users')

# Configure structured logging

logger = logging.getLogger()

logger.setLevel(logging.INFO)

@xray_recorder.capture('lambda_handler')

def lambda_handler(event, context):

    """

    User Service Lambda - handles CRUD operations with multi-tenant support

    """

    try:

        # Extract tenant context from authorizer

        tenant_id = event['requestContext']['authorizer']['tenant_id']

        user_role = event['requestContext']['authorizer']['role']

        # Route based on HTTP method

        http_method = event['httpMethod']

        if http_method == 'GET':

            return get_user(event, tenant_id)

        elif http_method == 'POST':

            return create_user(event, tenant_id)

        elif http_method == 'PUT':

            return update_user(event, tenant_id, user_role)

        elif http_method == 'DELETE':

            return delete_user(event, tenant_id, user_role)

        else:

            return error_response(405, "Method not allowed")

    except Exception as e:

        logger.error(f"Unhandled error: {str(e)}")

        return error_response(500, "Internal server error")

@xray_recorder.capture('get_user')

def get_user(event, tenant_id):

    """Get user with tenant isolation"""

    try:

        user_id = event['pathParameters']['user_id']

        response = table.get_item(

            Key={

                'user_id': user_id,

                'tenant_id': tenant_id  # Ensures tenant isolation

            }

        )

        if 'Item' not in response:

            return error_response(404, "User not found")

        return success_response(response['Item'])

    except ClientError as e:

        logger.error(f"DynamoDB error: {e.response['Error']['Message']}")

        return error_response(500, "Database error")

def success_response(data):

    return {

        'statusCode': 200,

        'headers': {'Content-Type': 'application/json'},

        'body': json.dumps(data, default=str)

    }

Lambda Authorizer


import json

import os

import jwt

from jwt import PyJWKClient

from jwt.exceptions import ExpiredSignatureError, InvalidTokenError

# Configuration

USER_POOL_ID = os.environ['USER_POOL_ID']

REGION = os.environ['AWS_REGION']

JWKS_URL = f"https://cognito-idp.{REGION}.amazonaws.com/{USER_POOL_ID}/.well-known/jwks.json"

# Initialize JWKS client

jwks_client = PyJWKClient(JWKS_URL)

def lambda_handler(event, context):

    """

    Lambda Authorizer with JWT validation and tenant isolation

    """

    try:

        # Extract token from Authorization header

        token = extract_token(event)

        if not token:

            raise Exception("No token provided")

        # Validate JWT token

        decoded_token = validate_jwt_token(token)

        # Extract custom claims

        tenant_id = decoded_token.get('custom:tenant_id')

        role = decoded_token.get('custom:role', 'user')

        if not tenant_id:

            raise Exception("Missing tenant_id in token")

        # Generate IAM policy

        policy = generate_policy(

            decoded_token['sub'],

            'Allow',

            event['methodArn'],

            tenant_id,

            role

        )

        return policy

    except Exception as e:

        print(f"Authorization failed: {str(e)}")

        raise Exception('Unauthorized')

def validate_jwt_token(token):

    """Validate JWT token with Cognito JWKS"""

    try:

        # Get signing key

        signing_key = jwks_client.get_signing_key_from_jwt(token)

        # Decode and validate token

        decoded_token = jwt.decode(

            token,

            signing_key.key,

            algorithms=["RS256"],

            options={"verify_exp": True}

        )

        return decoded_token

    except ExpiredSignatureError:

        raise Exception("Token has expired")

    except InvalidTokenError:

        raise Exception("Invalid token")

Follow-up response evaluation

DeepSeek-R1’s response demonstrates exceptional technical documentation analysis capabilities comparable to senior solution architects. The analysis transformed a complex 45-page technical guide into immediately actionable implementation strategies with specific code examples, configuration details, and production deployment procedures.

Evaluation methodology and metrics selection

For technical document analysis, we evaluate responses across criteria that reflect enterprise development requirements:

Qualitative metrics: Focus on technical accuracy, implementation clarity, and architectural insight quality that development teams require for production deployments.
Quantitative metrics: Measure response completeness, technical specificity, and structural organization that enables immediate development execution.

Let’s begin with the qualitative analysis (Table 4.3):

| Criteria | Analysis | Score (1–10) |
| Technical accuracy | Response demonstrates deep understanding of AWS architecture patterns, providing production-grade configurations and security implementations. Correctly identifies service relationships, data flows, and integration points. | 9 |
| Implementation clarity | Delivers specific code examples, configuration parameters, and deployment procedures. Technical recommendations include precise settings (rate limits, timeout values, and security headers) immediately usable by development teams. | 9 |
| Architectural insight | Provides comprehensive system design guidance, including microservice boundaries, priority matrices, and dependency analysis. Addresses both technical and business considerations for enterprise deployment. | 8 |
| Security depth | Identifies critical security requirements, including WAF rules, encryption standards, and compliance considerations. Provides specific implementation guidance for authentication flows and access control. | 9 |
| Production readiness | Covers monitoring, error handling, deployment strategies, and operational procedures. Includes specific metrics, alerting configurations, and disaster recovery considerations. | 8 |

Table 4.3: Qualitative analysis of the follow-up response

The computed qualitative score is 43/50 (86%).

The technical depth matches experienced solution architect capabilities, with comprehensive coverage of enterprise deployment requirements, and minor gaps in advanced monitoring configurations and regional compliance specifics.

Let’s take a look at the quantitative analysis:

| Criteria | Count/Metric | Analysis | Score (1–10) |
| Requirement coverage | 6/6 major analysis areas addressed | Complete coverage of architecture summary, implementation priorities, configuration analysis, code generation, security requirements, and production readiness | 10 |
| Technical specifications | 38 specific configuration parameters provided | Extensive technical detail, including rate limits (100 burst, 50 rate), timeout values, security headers, encryption standards, and deployment procedures | 9 |
| Code examples | 12 production-ready code samples | Comprehensive Lambda functions, API configurations, and infrastructure components with immediate implementation capability | 9 |
| Implementation steps | 8-phase structured deployment plan | Detailed priority matrix with dependencies, risk assessment, and business value considerations for systematic implementation | 9 |
| Response structure | 6 organized analysis sections with logical flow | Clear progression from high-level architecture through specific implementation details to operational procedures | 8 |

Table 4.4: Quantitative analysis of the follow-up response

The quantitative score is computed as 45/50 (90%).

Exceptional technical specificity with production-ready implementations is observed. Response provides comprehensive coverage, enabling immediate development execution.

Here is our overall assessment:

The combined score is computed as 88/100 (88%).
DeepSeek-R1 delivered solution architect-level technical analysis with immediate implementation capability. The response quality matches expensive technical consulting while providing specific code examples and deployment procedures that development teams can execute immediately.
While a traditional solution architecture engagement would require 3–4 weeks and $30–50K investment, DeepSeek-R1 achieved equivalent technical depth in minutes, representing 99%+ time reduction while maintaining enterprise-grade quality.

But there is always scope for improvement and extracting more knowledge. Here is how we can recalibrate our prompts for enhanced document understanding.

Recalibration through iterative prompting

The initial analysis provided comprehensive technical guidance but revealed opportunities for enhanced regional deployment specifics and advanced monitoring configurations.

The following areas require enhancement:

Regional compliance specifics (gap score: 7/10): We need specific regulatory requirements for EU/APAC deployments, which aren’t presently offered in the DeepSeek response.
Advanced monitoring (gap score: 7/10): Detailed observability patterns and custom metrics implementation are required for advanced monitoring.
Cost optimization (gap score: 6/10): Specific strategies for reducing API Gateway and Lambda costs at scale are essential for this solution architecture.

You can utilize the following iterative prompt examples to bridge these gaps:

Follow-up prompt 1 – Regional deployment:


Based on your AWS architecture analysis, create region-specific deployment configurations for EU (GDPR compliance) and APAC markets. Include:

- Specific data residency requirements and implementation

- Regional compliance certifications needed

- Cross-region disaster recovery procedures

- Regional cost optimization strategies

Provide configuration examples for each region.

Follow-up prompt 2 – Advanced monitoring:


Expand the monitoring strategy for the Amazon API Gateway system to include:

- Custom CloudWatch metrics for business KPIs

- Advanced X-Ray tracing configurations for microservices

- Automated anomaly detection and alerting procedures

- Performance optimization based on monitoring insights

Include specific implementation code and configuration examples.

Follow-up prompt 3 – Cost optimization:


Design a comprehensive cost optimization strategy for the Amazon API Gateway and Lambda architecture to reduce monthly operational expenses by 40% while maintaining performance and reliability. Include:

- API Gateway cost reduction strategies (HTTP API vs REST API selection, request optimization, caching)

- Lambda function cost optimization (memory configuration analysis, execution duration reduction, provisioned concurrency vs on-demand)

- Data transfer cost minimization (CloudFront caching, response payload optimization, compression strategies)

- Request batching and connection pooling implementations

- Reserved capacity analysis and purchasing recommendations for predictable workloads

- Cost monitoring and alerting setup with budget thresholds

Provide specific configuration examples, cost calculations, and projected monthly savings for each optimization strategy.

Practice exercise

Apply the evaluation framework to analyze the iterative prompting responses, focusing on how enhanced specificity improves implementation readiness for enterprise environments.

An interesting application of document analysis is in finance. In the next case study, we will prompt DeepSeek to analyze a financial document and compare its performance against benchmark tools.

Use case study: Financial document analysis and benchmarking

We will first identify the business context of this case example.

Financial institutions and enterprises regularly process complex financial documents containing structured tables, multi-quarter comparisons, and critical numerical data. Accurate extraction of this information is essential for automated financial analysis, regulatory compliance, and decision-making systems.

To comprehensively evaluate DeepSeek-R1’s document understanding capabilities, we created a rigorous benchmarking study comparing three different approaches:

DeepSeek-R1: Direct semantic extraction using advanced reasoning.
Docling (IBM Research): Specialized document structure and table parsing.
MarkItDown (Microsoft): LLM-optimized document preprocessing.

Test document creation and benchmarking setup

To rigorously evaluate document understanding capabilities, we created a comprehensive test dataset that would challenge all aspects of document parsing. Our fictional quarterly financial report for FruitStand Innovation Inc. spans 192 lines and contains the kind of complex structures commonly found in real financial statements. The document includes six major tables designed to test different parsing challenges: a multi-column consolidated balance sheet comparing three quarters (Q3 2024, Q2 2024, and Q3 2023), a detailed income statement with product versus service revenue breakdowns, geographic revenue analysis covering four global regions (North America, EMEA, APAC, and Latin America), product line performance metrics for four business segments (cloud platform, mobile solutions, enterprise services, and developer tools), eight key performance indicators (KPIs) tracked against quarterly targets, and forward-looking guidance ranges projecting Q4 2024 performance.

The balance sheet demonstrates the parsing complexity we wanted to test. Each financial line item appears across multiple quarters with corresponding percentage changes, requiring tools to maintain proper row-column associations while extracting values with exact numerical precision. Here’s an excerpt from our test document showing the Current Assets section:


| **Assets** | **Q3 2024** | **Q2 2024** | **Q3 2023** | **Change (%)** |

|-----------|-------------|-------------|-------------|----------------|

| **Current Assets** | | | | |

| Cash and cash equivalents | $234,678 | $198,543 | $187,432 | +25.2% |

| Short-term investments | $89,234 | $76,890 | $45,678 | +95.3% |

| Accounts receivable (net) | $142,567 | $134,289 | $118,945 | +19.9% |

| Inventory | $67,890 | $72,345 | $81,234 | -16.4% |

| Prepaid expenses | $23,456 | $21,234 | $19,876 | +18.0% |

| **Total Current Assets** | **$557,825** | **$503,301** | **$453,165** | **+23.1%** |

This structure tests whether tools can handle multi-column layouts, distinguish between regular line items and bolded totals, correctly parse both positive and negative percentage changes, and maintain the association between each value and its corresponding quarter. A tool that extracts $234,678 but associates it with Q2 2024 instead of Q3 2024 would fail our accuracy requirements, even though it captured the correct numerical value.

After creating the test document, we manually constructed a comprehensive ground truth dataset stored in ground_truth_data.json. This JSON file contains 150+ distinct data fields representing every piece of information we expect a perfect document parser to extract. The ground truth structure mirrors the document’s organization, with nested objects for different sections. Here’s how the ground truth represents the document metadata and key financial highlights:


{

  "document_metadata": {

    "company_name": "FruitStand Innovation Inc.",

    "report_type": "Quarterly Financial Report",

    "quarter": "Q3",

    "year": 2024,

    "report_date": "2024-09-30",

    "fiscal_year": 2024,

    "document_id": "FSI-FIN-2024-Q3-001",

    "classification": "Internal Use Only",

    "nasdaq_symbol": "FSTI"

  },

  "key_financial_highlights": {

    "total_revenue": {

      "value": 487.3,

      "unit": "million",

      "yoy_change": "+23%"

    },

    "net_income": {

      "value": 89.2,

      "unit": "million",

      "yoy_change": "+31%"

    },

    "operating_margin": {

      "value": 24.8,

      "unit": "percent",

      "comparison": "vs 21.4% in Q3 2023"

    }

  }

}

The ground truth format captures not just raw values but also their associated units, percentage changes, and contextual comparisons. This level of detail ensures that our benchmarking measures true semantic understanding rather than simple text extraction. When we evaluate whether a tool correctly extracted total_revenue, we verify that it captured the value (487.3), understood the unit (millions), and recognized the year-over-year change context (+23%).

Our benchmarking implementation uses these test files to automatically evaluate each tool’s performance. The test_deepseek_parsing.py Python script demonstrates how we prompt DeepSeek-R1 to extract this structured data. Here’s the actual prompt we used:


prompt = f"""Analyze this financial report and extract ALL key information in structured JSON format.

The document is a Q3 2024 financial report. Extract the following with EXACT values from the document:

1\. **Document Metadata**: company_name, report_type, quarter, year, report_date, document_id, nasdaq_symbol

2\. **Financial Highlights**: total_revenue, net_income, operating_margin, cash_and_equivalents, total_assets (with values and units)

3\. **Balance Sheet - Current Assets** (Q3 2024, Q2 2024, Q3 2023):

   - cash_and_cash_equivalents, short_term_investments, accounts_receivable_net,

     inventory, prepaid_expenses, total_current_assets

4\. **Balance Sheet - Non-Current Assets** (Q3 2024):

   - property_plant_equipment, accumulated_depreciation, net_ppe, intangible_assets,

     goodwill, long_term_investments, total_non_current_assets

...

CRITICAL: Extract EXACT numerical values, preserve decimals, include negative numbers

with minus sign. Return valid JSON only.

"""

This prompt structure proved effective because it specifies exactly what fields to extract, requests a structured JSON output format, and emphasizes the importance of numerical precision. The phrases EXACT values and preserve decimals help DeepSeek-R1 understand that approximate values or rounded numbers are insufficient for financial document analysis.

Complete test files, including the full financial statement, the comprehensive ground truth dataset with all 150+ fields, automated benchmarking scripts for all three tools (DeepSeek-R1, Docling, and MarkItDown), and evaluation framework documentation are available in our GitHub repository at https://github.com/PacktPublishing/DeepSeek-in-Practice/tree/main/Chapter04/document-understanding. You can clone the repository and reproduce our benchmarking results using the provided run_benchmark_comparison.py script.

Prompting DeepSeek for financial document extraction

Financial document extraction requires fundamentally different prompt design strategies than general document analysis. The critical challenge lies in achieving numerical precision across hundreds of data points while preserving complex relationships between values, their units, temporal contexts, and hierarchical structures. When prompting DeepSeek-R1 for financial documents, specificity becomes paramount: vague requests such as extract the revenue data will produce incomplete results, while precise instructions specifying extract total_revenue with its numerical value, unit (millions/thousands), currency, and year-over-year percentage change guide the model toward comprehensive extraction that captures all relevant attributes.

Three design principles proved essential in our financial document prompting approach:

First, explicitly request structured output formats (JSON) with defined schemas rather than accepting free-form text responses, as structured formats enable automated validation and downstream processing.
Second, emphasize numerical precision requirements by explicitly stating EXACT values, preserve decimal places, and include sign indicators for negative numbers and percentage changes, because financial analysis tolerates a zero error margin – a revenue figure of $487.3 million differs meaningfully from $487 million or $487.30 million, depending on reporting precision.
Third, specify the scope exhaustively by enumerating every category of data to extract (document metadata, financial highlights, balance sheet components, income statement sections, geographic breakdowns, product metrics, KPIs, and guidance), as financial documents contain numerous sections, and omitting categories from the prompt often results in those sections being overlooked during extraction.

DeepSeek-R1’s reasoning capabilities particularly shine when prompts establish clear validation criteria. Rather than simply requesting data extraction, our prompt includes phrases such as maintain quarter-to-quarter associations in multi-column tables and preserve parent-child relationships for revenue breakdowns, which activate DeepSeek’s contextual understanding. This approach leverages the model’s ability to reason about document structure rather than treating extraction as simple pattern matching. The following prompt demonstrates these principles applied to our FruitStand Innovation Inc. financial report:


Analyze this Q3 2024 financial report for FruitStand Innovation Inc.

and extract ALL financial data in structured JSON format.

Extract the following with EXACT values:

1\. Document Metadata: company_name, report_date, document_id, fiscal_year

2\. Financial Highlights: total_revenue, net_income, operating_margin,

   cash_equivalents, total_assets (with units and YoY changes)

3\. Balance Sheet (Q3 2024, Q2 2024, Q3 2023):

   - All current assets line items

   - All non-current assets line items

   - All liabilities and equity items

4\. Income Statement (all 3 quarters):

   - Revenue breakdown (product vs. service)

   - All cost categories

   - Operating metrics (margins, EPS)

5\. Revenue by Geographic Region (all 4 regions with %, growth, margins)

6\. Revenue by Product Line (all 4 products with %, growth, margins)

7\. All 8 Key Performance Indicators with exact values

8\. Management team names and titles

9\. Q4 2024 Guidance ranges

CRITICAL: Extract EXACT numerical values, preserve decimals,

include negative numbers with minus sign. Return valid JSON only.

[Attached: 192-line financial report document]

It is now time for us to look at how we will evaluate DeepSeek’s response in this case example.

Response evaluation

To objectively compare DeepSeek-R1’s performance against specialized document parsing tools Docling and MarkItDown, we established six quantitative metrics that measure different aspects of document understanding capability. Each metric was carefully designed to be objectively measurable through automated comparison against our ground truth dataset. The following sections explain what each metric measures, why it matters for document understanding, and how we compute it.

Metric 1: Field extraction accuracy

The first metric measures how accurately a tool can identify and extract semantic field-value pairs from document text. In document understanding, key-value pairs represent discrete pieces of information where a descriptive key (field name) is associated with a specific value.

Consider the phrase Total Revenue: $487.3 million from our test document. A document parser must recognize that total_revenue is the semantic field name, 487.3 is the numerical value, million represents the unit, and the currency is USD. Simple text extraction would only capture $487.3 million without understanding the semantic relationship to revenue.

Our test document contains key-value pairs ranging from simple metadata to complex nested financial metrics. Simple examples include company name (FruitStand Innovation Inc.) and report date (2024-09-30), while complex examples include multi-attribute financial metrics such as total revenue, which requires capturing the value (487.3), unit (million), and year-over-year change context (+23%) as a structured object. Operating margin represents another complex case where the value (24.8) must be associated with its unit (percent) and comparison baseline (versus 21.4% in Q3 2023). These compound values test whether tools understand document semantics or merely extract isolated text fragments.

To compute field extraction accuracy, we implemented an automated comparison script that loads the ground truth JSON and compares it field-by-field against each tool’s output. A field is marked correct only when both the key and value match our stringent criteria: numerical values must match to the exact decimal place (487.3, not 487 or 487.30), strings must match including, capitalization (FruitStand Innovation Inc., not fruitstand innovation inc.), dates must follow ISO format (2024-09-30), and compound values must include all required attributes.

The formula is straightforward: divide the number of correctly extracted fields by the total expected fields (150 in our test document) and multiply by 100. When DeepSeek-R1 correctly extracted 137 out of 150 fields, this yielded 91.3% accuracy.

Metric 2: Table parsing accuracy

Table parsing accuracy measures a tool’s ability to correctly extract individual cell values from tabular data while preserving critical row-column relationships. This metric proves particularly challenging because it requires understanding table structure, not just reading text sequentially. Our consolidated balance sheet exemplifies this complexity with its multi-column layout comparing three quarters of financial data. Consider the Cash and cash equivalents line item, which contains four distinct values across the row: $234,678 for Q3 2024, $198,543 for Q2 2024, $187,432 for Q3 2023, and a calculated change of +25.2%. Cell-level accuracy demands that all four values be extracted with exact numerical precision and correctly associated with their respective time periods.

The challenge becomes apparent when we consider potential parsing errors. A tool might successfully extract the value of $234,678 but incorrectly associate it with Q2 2024 instead of Q3 2024. While the numerical extraction succeeded, the semantic understanding failed because the quarter-value relationship was misidentified. This type of error would render the extracted data useless for financial analysis, even though superficially the tool found the right numbers. Our ground truth dataset addresses this by encoding not just cell values but also their positional context, requiring tools to demonstrate they understand table structure rather than merely scanning for numbers.

We enumerate all 245 cells across the 6 major tables in our ground truth dataset, identifying each by table name, row label, and column header. The automated comparison script checks whether each extracted cell matches both the expected value and position. Percentage values must include their signs (+23% not 23%), currency values must preserve dollar signs and comma separators where present, and calculated fields such as percentage changes must match exactly rather than being recomputed. This stringency ensures that we measure precise extraction capability rather than approximate understanding. DeepSeek-R1’s result of 232 correctly parsed cells out of 245 total cells translates to 94.7% table parsing accuracy.

Metric 3: Entity recognition accuracy

Named entity recognition measures how well a tool identifies specific real-world objects embedded in document text. Our financial report contains 42 distinct named entities spanning multiple categories, each serving a different purpose in document comprehension. Company information entities provide organizational context: the company name (FruitStand Innovation Inc.), its NASDAQ trading symbol (FSTI), and the unique document identifier (FSI-FIN-2024-Q3-001) all represent distinct entities that must be correctly identified and categorized. Executive names represent another critical entity class, with our test document mentioning Sarah Chen (Chief Executive Officer), Michael Rodriguez (Chief Financial Officer), Jennifer Wu (Chief Technology Officer), and David Park (Chief Operating Officer). Each of these names must be recognized as person entities associated with their respective leadership roles.

Geographic entities add another layer of complexity. Our revenue analysis section references four distinct regions: North America, EMEA (Europe, the Middle East, and Africa), APAC (Asia-Pacific), and Latin America. These aren’t merely location names but semantic categories that structure our revenue breakdown. Similarly, our four product lines – Cloud Platform, Mobile Solutions, Enterprise Services, and Developer Tools – represent product entities that organize financial performance metrics. Temporal entities round out our entity catalog with specific dates, such as September 30, 2024, serving as the report date, alongside quarterly designators (Q3 2024, Q4 2024) that establish the reporting and projection timeframes.

Entity recognition accuracy is computed by checking whether each of the 42 expected entities appears in the tool’s extracted output with correct spelling, appropriate context, and proper entity classification. A tool that extracts Sarah Chen but fails to recognize it as a person name, or that identifies North America but doesn’t understand it as a geographic region, would fail our entity recognition requirements. DeepSeek-R1 achieved perfect performance on this metric, correctly identifying all 42 out of 42 entities, yielding 100% entity recognition accuracy.

Metric 4: Structure preservation score

Structure preservation evaluates how well a tool maintains the document’s logical organization, hierarchical relationships, and reading order during extraction. This metric matters because financial documents communicate meaning through structure as much as through content. Consider how our financial report follows the conventional sequence: the executive summary presents key highlights, followed by the detailed balance sheet, then the income statement with segment breakdowns, concluding with forward-looking guidance. This ordering isn’t arbitrary but reflects increasing levels of detail and temporal progression from historical results to future projections.

Hierarchical relationships carry semantic weight throughout the document. When our report presents Revenue by Geographic Region as a section header followed by four subsections (North America, EMEA, APAC, and Latin America), this structure conveys that the four regions are components of a comprehensive geographic analysis. A tool that extracts these region names but loses the parent-child relationship would miss the organizational logic. Similarly, visual formatting provides crucial cues: bold text typically denotes totals or section headers, indented line items indicate detail breakdowns supporting bolded totals, and the Total Revenue entry appearing in bold, followed by Product Revenue and Service Revenue indented beneath it, signals a sum-detail relationship that aids comprehension.

We evaluate structure preservation using a hybrid quantitative-qualitative approach on a 0–100 point scale. This could cover the following aspects:

Section ordering contributes 25 points (did the extraction maintain the executive summary → balance sheet → income statement → analysis sequence?)
Table structure integrity accounts for 30 points (were row-column relationships and header associations preserved?)
Header data associations provide 25 points (are section headers correctly linked to their content?), and hierarchical nesting levels comprise the final 20 points (are parent-child relationships and indentation patterns maintained?)

This scoring combines automated checks, such as verifying table column counts and section ordering, with human evaluation of whether the extracted output remains logically organized and usable.

Metric 5: Processing time

Processing speed becomes critical for production systems handling thousands of documents. We measure end-to-end processing time by recording timestamps immediately before submitting the document and immediately after receiving the complete output. For API-based tools such as DeepSeek-R1, we use Python’s time.time() function to capture these boundaries, measuring the total elapsed time, including network latency, server queue time, model inference, and response formatting. This real-world timing reflects what production systems would experience. DeepSeek-R1 processed our 192-line financial document in 4,250 milliseconds (4.25 seconds), fast enough for interactive applications where users can wait a few seconds for results, but potentially challenging for batch processing systems needing to handle thousands of documents hourly.

Metric 6: Cost per document

Economic viability determines whether document understanding solutions can scale to production volumes. For DeepSeek-R1, we calculate per-document cost by tracking token consumption and applying published API pricing. Our test document required 12,847 total tokens (combining input and output), which, at DeepSeek’s pricing of $0.14 per million input tokens and $0.28 per million output tokens, translates to approximately $0.0027 per document. At this rate, processing 10,000 documents would cost $27, making it economically viable for most business applications. Open source tools such as Docling and MarkItDown incur $0.00 direct API costs, though self-hosting introduces infrastructure expenses (compute resources, maintenance, and updates) that we didn’t quantify in this benchmark.

Test data analysis

Having defined our evaluation metrics and prompt strategy, we can now examine how DeepSeek-R1 performed when processing our actual test document. Our 192-line financial report for FruitStand Innovation Inc. serves as the input for all three tools in our benchmark comparison. The document begins with executive summary information that tests entity recognition and simple field extraction, then progresses through increasingly complex table structures that challenge parser capabilities. Here’s how the test document opens:


# FRUITSTAND INNOVATION INC.

## Quarterly Financial Report Q3 2024

**Report Date:** September 30, 2024 

**Fiscal Year:** 2024 

**Document ID:** FSI-FIN-2024-Q3-001 

**Classification:** Internal Use Only 

---

## Executive Summary

FruitStand Innovation Inc. (NASDAQ: FSTI) reports strong financial performance for Q3 2024,

with revenue growth of 23% year-over-year and operating margin expansion of 340 basis points.

**Key Financial Highlights:**

- Total Revenue: $487.3 million (up 23% YoY)

- Net Income: $89.2 million (up 31% YoY)

- Operating Margin: 24.8% (vs 21.4% in Q3 2023)

- Cash and Equivalents: $234.7 million

- Total Assets: $1,247.8 million

This opening section tests whether tools can extract structured metadata (company name, document ID, and NASDAQ symbol) from the header while simultaneously capturing key financial metrics from the bullet-pointed highlights section. The metrics appear in a semi-structured format mixing absolute values ($487.3 million), percentage changes (up 23% YoY), and comparative contexts (vs 21.4% in Q3 2023), requiring tools to parse compound information rather than isolated numbers.

The document’s primary challenge lies in its consolidated balance sheet, which presents a multi-column structure comparing three time periods simultaneously. Each financial line item spans four data columns plus descriptive text, demanding that parsers maintain precise associations between values and their temporal contexts:


## Consolidated Balance Sheet

### As of September 30, 2024

**(In thousands, except share data)**

| **Assets** | **Q3 2024** | **Q2 2024** | **Q3 2023** | **Change (%)** |

|-----------|-------------|-------------|-------------|----------------|

| **Current Assets** | | | | |

| Cash and cash equivalents | $234,678 | $198,543 | $187,432 | +25.2% |

| Short-term investments | $89,234 | $76,890 | $45,678 | +95.3% |

| Accounts receivable (net) | $142,567 | $134,289 | $118,945 | +19.9% |

| Inventory | $67,890 | $72,345 | $81,234 | -16.4% |

| Prepaid expenses | $23,456 | $21,234 | $19,876 | +18.0% |

| **Total Current Assets** | **$557,825** | **$503,301** | **$453,165** | **+23.1%** |

This table structure reveals several parsing complexities. Empty cells in the Current Assets header row must be recognized as structural elements rather than missing data. Bold formatting distinguishes category headers and totals from detailed line items, requiring tools to preserve formatting semantics. Negative inventory values (indicated by decreasing quarter-over-quarter figures) test sign handling, while percentage changes in the rightmost column require extracting both magnitude and direction indicators (+/- signs). The accumulated depreciation line later in the document shows ($123,456) in parentheses, representing negative values in accounting convention, which parsers must interpret correctly rather than treating parentheses as mere formatting.

The income statement section introduces additional complexity through nested calculations and percentage-based metrics that span multiple quarters:


## Consolidated Income Statement

### For the Three Months Ended September 30, 2024

**(In thousands, except per share data)**

| **Revenue** | **Q3 2024** | **Q2 2024** | **Q3 2023** | **Change (%)** |

|------------|-------------|-------------|-------------|----------------|

| Product revenue | $312,456 | $287,654 | $245,678 | +27.2% |

| Service revenue | $174,892 | $168,543 | $150,234 | +16.4% |

| **Total Revenue** | **$487,348** | **$456,197** | **$395,912** | **+23.1%** |

| | | | | |

| **Cost of Revenue** | | | | |

| Cost of product revenue | $143,567 | $139,876 | $128,945 | +11.3% |

| Cost of service revenue | $76,543 | $74,321 | $70,234 | +9.0% |

| **Total Cost of Revenue** | **$220,110** | **$214,197** | **$199,179** | **+10.5%** |

| | | | | |

| **Gross Profit** | **$267,238** | **$242,000** | **$196,733** | **+35.8%** |

| **Gross Margin** | **54.8%** | **53.0%** | **49.7%** | **+5.1 pts** |

Notice how this section alternates between absolute dollar amounts and percentage-based metrics. For example, the field Gross Margin, requires parsers to handle different data types within the same table structure. The +5.1 pts notation for margin improvement uses a domain-specific abbreviation (pts for basis points) that semantic parsers must interpret correctly.

Computing the final score

The overall performance score combines all six metrics using a weighted average that reflects their relative importance in production document understanding systems. We assigned weights based on what matters most for real-world deployments: field extraction accuracy receives 35% weight as the primary objective of document parsing, table parsing accuracy gets 30% weight given how prevalent tabular data is in business documents, entity recognition accounts for 15% weight as entities represent key document actors and objects, structure preservation contributes 10% weight for maintaining document organization, while processing time and cost each receive 5% weight as practical constraints rather than primary goals.

The formula multiplies each metric score by its weight and sums the results:

Let’s break this down:

Field extraction contributes 31.96 points.
Table parsing adds 28.41 points.
Perfect entity recognition provides 15.00 points.
Structure preservation contributes 8.50 points.
Processing time adds 4.50 points.
Cost efficiency provides 4.75 points.

This yields a total of 93.12 points, which we round to 93/100.

We established performance tiers to interpret these scores. Scores from 95 to 100 indicate excellent performance suitable for production deployment with minimal human validation, representing tools that consistently deliver accurate results requiring only spot-checking. Scores from 85 to 94 indicate good performance appropriate for production with targeted validation workflows, where automated extraction handles the bulk of work but critical fields undergo verification. Scores from 75 to 84 represent acceptable performance, requiring substantial validation and quality assurance, suitable for semi-automated workflows where human review remains extensive. Scores below 75 indicate poor performance, unsuitable for production deployment without significant improvements, typically requiring more development before practical use.

Benchmark comparison: From document to accuracy scores

Our benchmarking system evaluates each tool through a four-stage automated pipeline implemented in run_benchmark_comparison.py.

Let’s describe this pipeline in detail:

The process begins by running each tool’s test script (test_deepseek_parsing.py, test_docling_parsing.py, and test_markitdown_parsing.py) sequentially, capturing their outputs and measuring processing time.
Each tool receives the identical input document and must produce a structured output representing the extracted information.
For DeepSeek-R1, we measure end-to-end processing time from the moment we submit the HTTP request to the DeepSeek API until we receive the complete JSON response.
The API call includes the full document text as input along with our extraction prompt, and DeepSeek-R1 returns structured JSON containing all identified fields.
We save this output to deepseek_test_results.json for subsequent accuracy analysis. The processing time measurement uses Python’s time.time() function to capture high-precision timestamps before and after the API call, giving us the real-world latency that production applications would experience.
Once all three tools complete their extractions, our comparison script loads the ground truth dataset from ground_truth_data.json and performs field-by-field validation. The comparison algorithm iterates through every field defined in the ground truth, checking whether each tool’s output contains that field with the correct value. For simple fields such as company name or report date, we verify exact string matches. For numerical fields, we compare values with appropriate precision tolerances. For compound fields such as total_revenue, which includes value, unit, and year-over-year change, we verify that all attributes match the expected structure.

This automated comparison generates accuracy percentages for field extraction, table parsing (cell-by-cell comparison), and entity recognition (checking that all 42 expected entities appear correctly).

The benchmark script then compiles results into comparison tables showing side-by-side performance across all metrics. These tables display not just accuracy percentages but also processing times, token usage (for API-based tools), and cost per document, providing a comprehensive view of each tool’s production readiness. The entire benchmarking process runs automatically via the python run_benchmark_comparison.py command, making it reproducible for readers who want to verify our results or test with different documents.

DeepSeek-R1 successfully extracted the financial data with the following output structure:


{

  "document_metadata": {

    "company_name": "FruitStand Innovation Inc.",

    "report_date": "2024-09-30",

    "document_id": "FSI-FIN-2024-Q3-001",

    "nasdaq_symbol": "FSTI"

  },

  "key_financial_highlights": {

    "total_revenue": {"value": 487.3, "unit": "million", "yoy_change": "+23%"},

    "net_income": {"value": 89.2, "unit": "million", "yoy_change": "+31%"},

    "operating_margin": {"value": 24.8, "unit": "percent"}

  },

  "balance_sheet_current_assets_q3_2024": {

    "cash_and_cash_equivalents": 234678,

    "short_term_investments": 89234,

    "accounts_receivable_net": 142567,

    "inventory": 67890,

    "prepaid_expenses": 23456,

    "total_current_assets": 557825

  },

  // ... [145 more fields extracted correctly]

}

Here are the quantitative results of the DeepSeek-R1 response:

Table 4.5: DeepSeek-R1 quantitative results for finance document understanding

Here is our analysis of DeepSeek-R1’s performance.

DeepSeek-R1 demonstrated excellent semantic understanding of financial terminology throughout the document extraction process. Here is how the results looked:

When encountering the executive summary’s key financial highlights, the model correctly interpreted complex phrases such as operating margin expansion of 340 basis points and properly extracted both the numerical value (24.8%) and its comparative context (versus 21.4% in Q3 2023).
The model successfully distinguished between different revenue types in the income statement, accurately categorizing $312,456 as product revenue and $174,892 as service revenue while maintaining their quarterly associations across the multi-column structure.
The tool’s handling of nested table structures proved particularly strong. In the consolidated balance sheet, DeepSeek-R1 correctly parsed the hierarchical relationship between Total Current Assets ($557,825) and its component line items, understanding that cash and cash equivalents ($234,678), short-term investments ($89,234), accounts receivable ($142,567), inventory ($67,890), and prepaid expenses ($23,456) sum to the total.
The model also successfully navigated the income statement’s nested calculations, recognizing that gross profit ($267,238) equals total revenue ($487,348) minus total cost of revenue ($220,110), demonstrating comprehension of accounting relationships rather than merely extracting isolated numbers.
Entity recognition achieved perfect accuracy across all 42 named entities in the document. DeepSeek-R1 correctly identified executive names, including Sarah Chen (Chief Executive Officer), Michael Rodriguez (Chief Financial Officer), Jennifer Wu (Chief Technology Officer), and David Park (Chief Operating Officer), properly associating each name with its corresponding title. Geographic entities were accurately extracted with their complete designations: North America, EMEA (explicitly recognized as Europe, the Middle East, and Africa rather than treating it as an undefined acronym), APAC (Asia-Pacific), and Latin America. Product line entities – Cloud Platform, Mobile Solutions, Enterprise Services, and Developer Tools – were correctly identified and associated with their respective revenue figures and performance metrics.

However, DeepSeek-R1 encountered 13 extraction errors, primarily in deeply nested multi-quarter comparison cells:

Eight errors occurred in the balance sheet’s quarterly comparison columns, where cells contained multiple related values. For example, in the Accounts Payable row, the model correctly extracted the Q3 2024 value ($87,654) and the year-over-year percentage change (+14.5%), but misassociated the Q2 2024 value, placing $82,345 in a Q3 2023 column instead of Q2 2024.
Three formatting ambiguity errors involved numbers with accounting conventions: the accumulated depreciation value displayed as ($123,456) in parentheses was initially extracted without the negative sign, requiring the model to infer that parenthetical values in the Liabilities section represent negative amounts.
Two errors involved abbreviation expansion, where PP&E (Property, Plant & Equipment) in table headers was extracted as the abbreviated form rather than the expanded terminology expected by the ground truth dataset.

Despite these minor errors, DeepSeek-R1 exhibited strong handling of percentage calculations and currency formatting throughout the document. The model correctly preserved positive and negative signs in percentage changes (+25.2% for cash growth versus -16.4% for inventory decline), maintained decimal precision in financial figures ($234,678 not rounded to $234,700), and properly interpreted the +5.1 pts notation for basis point changes in gross margin. Currency symbols and comma separators were consistently preserved in the extracted output, demonstrating attention to formatting details that matter for financial data integrity.

Understanding DeepSeek-R1’s extraction errors

Before comparing DeepSeek-R1 against alternative tools, we should examine the specific errors it encountered to understand its limitations. The 13 extraction errors fell into 3 distinct categories, each revealing different aspects of document understanding challenges. Eight errors occurred in nested cell confusion scenarios where multi-quarter comparison tables presented ambiguous cell boundaries. In the consolidated balance sheet’s Accounts Payable row, DeepSeek-R1 correctly extracted the Q3 2024 value ($87,654) and the year-over-year percentage change (+14.5%), but misassociated the Q2 2024 value by placing $82,345 in the Q3 2023 column instead of its proper Q2 2024 position. This type of error suggests the model sometimes struggles with maintaining precise column associations when multiple time periods appear in adjacent cells, particularly when percentage change calculations create additional cognitive complexity.

Three formatting ambiguity errors involved numbers presented in accounting conventions that require domain knowledge to interpret correctly. The accumulated depreciation line item appeared as ($123,456) with the value enclosed in parentheses, following standard accounting practice where parentheses indicate negative amounts. DeepSeek-R1 initially extracted this value as a positive 123,456, missing the semantic meaning of the parenthetical notation. While the model eventually identified these as liabilities and inferred negative values from context, the initial extraction demonstrated that implicit formatting conventions can challenge even sophisticated language models. Similarly, when encountering the accumulated other comprehensive loss value of ($7,520), the model correctly interpreted the negative sign but experienced minor decimal placement uncertainty in the initial extraction.

Two abbreviation expansion errors revealed challenges in technical terminology recognition. Table headers in the balance sheet used PP&E as shorthand for Property, Plant & Equipment, a standard abbreviation in financial reporting. DeepSeek-R1 extracted PP&E verbatim rather than expanding it to the full term expected by our ground truth dataset. While the abbreviated form contains the same semantic meaning, automated validation systems comparing against ground truth require exact string matches, highlighting how domain-specific abbreviations can create discrepancies between functionally correct extractions and technically incorrect results according to strict validation criteria. This suggests that prompts for financial documents might benefit from including glossaries of common abbreviations or explicitly requesting expanded terminology.

Comparing the three approaches

To provide a comprehensive context and understand how DeepSeek-R1’s performance compares to specialized document parsing tools, we tested the identical financial document with two alternative approaches: Docling, an IBM Research toolkit specializing in document structure analysis and table recognition, and MarkItDown, a Microsoft tool designed for LLM-optimized document preprocessing. Each tool was evaluated using the same test document and ground truth dataset, measured against the same six quantitative metrics we defined earlier: field extraction accuracy, table parsing accuracy, entity recognition accuracy, structure preservation score, processing time, and cost per document.

Testing Docling and MarkItDown

Docling operates fundamentally differently from DeepSeek-R1 by focusing on document structure extraction rather than semantic understanding. We installed Docling using pip install docling and created a Python script (test_docling_parsing.py) that processes our financial statement Markdown file. The basic usage pattern involves loading the document, running Docling’s parser, and extracting table structures:


from docling import DocumentConverter

converter = DocumentConverter()

result = converter.convert("test_financial_statement.md")

# Extract tables

tables = result.document.tables

for table in tables:

    print(f"Table: {table.num_rows} rows x {table.num_cols} columns")

    # Process table cells...

Docling excels at recognizing table boundaries, preserving row-column relationships, and maintaining document structure, but produces structural representations rather than semantic field extractions. For our financial document, Docling correctly identified all 6 tables and preserved 97.2% of cell structures (238 out of 245 cells), outperforming DeepSeek-R1’s table parsing accuracy. However, Docling cannot directly extract semantic fields such as total_revenue with associated units and year-over-year changes, which is why we show N/A for its field extraction accuracy. To obtain semantic information from Docling’s output requires additional LLM processing.

MarkItDown follows a similar philosophy, optimizing document preprocessing for LLM consumption. Installation requires pip install markitdown, and the tool converts various document formats to clean Markdown suitable for LLM input:


from markitdown import MarkItDown

converter = MarkItDown()

markdown_output = converter.convert("test_financial_statement.md")

print(markdown_output.text)

MarkItDown processed our document in just 890 milliseconds, significantly faster than both DeepSeek-R1 (4,250 ms) and Docling (2,100 ms), but achieved only 78.5% table parsing accuracy. The tool prioritizes processing speed and markdown cleanliness over perfect structural preservation, making it suitable for documents where some structural ambiguity is acceptable. Like Docling, MarkItDown requires subsequent LLM processing for semantic field extraction.

Complete usage examples, configuration details, and output samples for both Docling and MarkItDown are documented in our GitHub repository at https://github.com/PacktPublishing/DeepSeek-in-Practice/tree/main/Chapter04/document-understanding. The repository includes docling_usage_guide.md with step-by-step instructions, markitdown_usage_guide.md with implementation examples, and tool_comparison_results.md showing complete outputs from all three tools for direct comparison.

Computing benchmark metrics

Our benchmark comparison requires objective measurements of field accuracy, table accuracy, processing time, and cost across all three tools. Field extraction accuracy was computed by running each tool against the test document, capturing its output, and comparing extracted fields against our ground truth dataset using automated Python scripts. For each of the 150 defined fields in ground_truth_data.json, we checked whether the tool’s output contained that field with the correct value, considering numerical precision, string case sensitivity, and date formatting. DeepSeek-R1’s 137 correctly extracted fields divided by 150 total expected fields yielded 91.3% accuracy. Docling and MarkItDown received N/A ratings for field extraction because their structural outputs don’t directly produce semantic field extractions without additional LLM processing.

Table parsing accuracy measurement required cell-by-cell comparison of the extracted table data against the 245 expected cells in our 6 major tables. We identified each cell by its table name, row index, and column index in the ground truth, then verified whether the tool correctly extracted that cell’s value with proper formatting and positional association. Docling achieved 97.2% accuracy (238/245 correct cells) through its specialized table structure recognition algorithms. DeepSeek-R1 achieved 94.7% accuracy (232/245 correct cells) using semantic understanding, while MarkItDown’s lightweight preprocessing yielded 78.5% accuracy (192/245 correct cells). The cell-level comparison reveals that specialized table parsers such as Docling excel at structure recognition, even though they lack semantic comprehension.

Processing time measurements used Python’s time.time() function to record high-precision timestamps immediately before submitting the document to each tool and immediately after receiving complete output. For DeepSeek-R1, we measured end-to-end API latency, including network transmission, server queue time, model inference, and response formatting, yielding 4,250 ms for our 192-line document. Docling processed the same document in 2,100 ms, running locally on a standard development machine (MacBook Pro M1), while MarkItDown completed processing in just 890 ms. These measurements reflect real-world performance that production systems would experience, though actual times vary based on network conditions, server load, and hardware specifications.

Cost per document calculation differed significantly between API-based and local tools. DeepSeek-R1’s cost stems from token consumption multiplied by API pricing: our test document consumed 12,847 total tokens (input plus output), which, at $0.14 per million input tokens and $0.28 per million output tokens, resulted in approximately $0.0027 per document. At this rate, processing one million documents would cost $2,700. Docling and MarkItDown, being open source tools running locally, incur zero direct API costs. However, local deployment introduces infrastructure expenses, including server hardware, maintenance, updates, and operational overhead, that we did not quantify in this benchmark. Organizations must weigh direct API costs against infrastructure investment when choosing between cloud-based and self-hosted solutions.

Quantitative comparison across six metrics

The following table presents the complete benchmark results measuring all three tools against the six quantitative metrics we defined earlier: field extraction accuracy, table parsing accuracy, entity recognition accuracy, structure preservation score, processing time, and cost per document:

Metric	DeepSeek-R1	Docling	MarkItDown
Field extraction accuracy	91.3% (137/150)	N/A*	N/A*
Table parsing accuracy	94.7% (232/245)	97.2% (238/245)	78.5% (192/245)
Entity recognition accuracy	100% (42/42)	N/A*	N/A*
Structure preservation score	88/100	95/100	72/100
Processing time	4,250 ms	2,100 ms	890 ms
Cost per document	$0.0027 \| $0.00	$0.00

Table 4.6: Overall performance comparison

*Docling and MarkItDown focus on structure extraction and require additional LLM processing for semantic field extraction.

Interpreting the results

Each tool demonstrates distinct strengths aligned with its design philosophy. DeepSeek-R1 achieves the highest field extraction accuracy at 91.3%, successfully identifying 137 out of 150 semantic fields such as quarterly_revenue, operating_margin_expansion, and year_over_year_revenue_growth directly from the document. This semantic extraction capability sets it apart from structural parsers. The model’s perfect entity recognition score (42/42 entities) demonstrates robust named entity identification across executives, geographic regions, and product lines. However, DeepSeek-R1’s table parsing accuracy of 94.7% trails Docling’s industry-leading 97.2%, revealing that specialized table recognition algorithms still outperform general-purpose language models for pure structural extraction.

Docling achieves the highest table parsing accuracy and structure preservation score, reflecting its specialized design for document layout analysis. The tool correctly identified 238 out of 245 table cells with precise row-column associations, outperforming DeepSeek-R1 by 2.5 percentage points. Its structure preservation score of 95/100 indicates near-perfect maintenance of hierarchical relationships, nested table structures, and document reading order. Docling’s faster processing time (2,100 ms versus DeepSeek-R1’s 4,250 ms) and zero API cost make it attractive for high-volume batch processing scenarios. The N/A ratings for field extraction and entity recognition reflect architectural choice rather than limitation: Docling delivers structured representations that require downstream LLM processing to extract semantic meaning.

MarkItDown prioritizes speed and simplicity over comprehensive accuracy, completing document processing in just 890 ms – less than one-fifth of DeepSeek-R1’s processing time. This performance comes at the cost of table parsing accuracy (78.5%) and structure preservation (72/100), as the tool sometimes simplifies complex nested tables or merges cells to create cleaner Markdown output. For use cases where perfect structural fidelity is less critical than rapid preprocessing for LLM consumption, MarkItDown’s lightweight approach offers practical advantages. Like Docling, it requires subsequent LLM processing for semantic extraction.

Understanding tool capabilities

Beyond quantitative metrics, each tool possesses specialized capabilities that influence its suitability for different document understanding scenarios. Direct field extraction capability refers to a tool’s ability to identify and extract semantic fields (such as total_revenue, ceo_name, or quarterly_growth_rate) without additional processing steps. DeepSeek-R1 excels at this through natural language understanding, while Docling and MarkItDown produce intermediate structural representations requiring LLM interpretation.

Table structure recognition measures how well a tool identifies table boundaries, preserves row-column relationships, handles merged cells, and maintains hierarchical nesting in complex tables. Docling achieves excellence here through specialized computer vision and layout analysis algorithms trained specifically for table detection. DeepSeek-R1 demonstrates very good performance by understanding table semantics even when the visual structure is ambiguous, while MarkItDown provides moderate capability suitable for simpler tables.

OCR integration capability determines whether a tool can process scanned documents or images containing text. Docling integrates with OCR engines to handle PDF files created from scanned images, a critical capability for legacy document digitization projects. Neither DeepSeek-R1 nor MarkItDown includes native OCR, requiring preprocessed text-based inputs. Semantic reasoning capability measures a tool’s ability to understand meaning, context, and relationships between extracted information. DeepSeek-R1 demonstrates excellent semantic reasoning, recognizing that operating margin expansion of 340 basis points means the current 24.8% margin increased from a previous 21.4%. Docling and MarkItDown lack this capability, focusing instead on structural extraction.

Cost efficiency considerations extend beyond direct API pricing to include infrastructure, maintenance, and operational overhead. DeepSeek-R1’s $0.0027 per document API cost translates to $2,700 per million documents with zero infrastructure investment but ongoing usage fees. Docling and MarkItDown eliminate API costs but require server infrastructure, software updates, dependency management, and technical expertise for self-hosted deployment. Setup complexity varies accordingly: DeepSeek-R1 requires only API credentials and simple HTTP requests, MarkItDown installs via pip with minimal configuration, while Docling involves more complex dependency management and configuration for optimal performance.

Choosing the right tool for your use case

Our benchmark results reveal that different document understanding scenarios call for different tool selections based on specific requirements and constraints. When analyzing financial statements or business documents where semantic extraction of specific fields is the primary goal, DeepSeek-R1 offers the most direct path to actionable data. Its 91.3% field extraction accuracy and perfect entity recognition enable production deployments with validation workflows to catch the remaining 8.7% of errors. Organizations requiring immediate API-based deployment without infrastructure investment will find DeepSeek-R1’s simple integration and pay-per-use pricing attractive for moderate document volumes.

For scenarios involving PDF documents with complex visual layouts, a hybrid approach combining Docling’s structure extraction with DeepSeek-R1’s semantic understanding delivers best-in-class results. Docling preprocesses the PDF to extract table structures with 97.2% accuracy, then DeepSeek-R1 interprets those structures to extract semantic fields. This pipeline leverages each tool’s core strength: Docling’s specialized layout analysis and DeepSeek-R1’s language understanding. The combined approach costs $0.0027 per document (only DeepSeek-R1’s API fee) while achieving higher overall accuracy than either tool alone.

Processing scanned documents or images containing text requires OCR capabilities that only Docling provides in our benchmark comparison. Legacy document digitization projects, historical financial record analysis, or any scenario involving physical documents converted to images necessitates Docling’s OCR integration. The typical workflow involves Docling performing OCR and structure extraction, then feeding the resulting text to DeepSeek-R1 for semantic analysis. Real-time API integration scenarios where sub-second response latency is critical might favor MarkItDown’s 890-ms processing time for initial preprocessing, though DeepSeek-R1’s 4.25-second end-to-end latency remains acceptable for most interactive applications.

Batch processing of large document volumes (millions of documents) requires careful cost-benefit analysis. Docling’s zero API cost makes it economically attractive for massive scale, but organizations must factor in infrastructure investment estimated at $50,000–200,000 annually for servers, storage, networking, and operational staff. At one million documents per month, DeepSeek-R1 costs $2,700 monthly ($32,400 annually), substantially less than self-hosted infrastructure until reaching approximately 5–10 million documents monthly, depending on infrastructure efficiency. Multi-language documents leverage DeepSeek-R1’s strong multilingual capabilities, accurately processing financial reports in English, Spanish, Mandarin, French, German, and other languages without requiring language-specific configurations.

Hybrid approach: Best practices

Based on our comprehensive benchmarking results, the optimal approach for processing complex financial documents leverages the complementary strengths of multiple tools rather than relying on a single solution. Organizations can achieve higher accuracy, lower costs, and greater flexibility by architecting hybrid pipelines that combine structural extraction with semantic understanding.

Recommended workflow architecture

The ideal workflow begins with document preprocessing for any PDF or DOCX format inputs. At this initial stage, we recommend using Docling for structure extraction because its specialized algorithms excel at identifying table boundaries, preserving row-column relationships, and maintaining proper reading order even in complex multi-column layouts. Docling’s OCR integration proves particularly valuable when processing scanned documents or images containing financial data, as it can extract text from legacy paper documents that have been digitized. This preprocessing stage converts diverse input formats into clean, structured representations that downstream tools can reliably process.

The second stage involves semantic extraction, where DeepSeek-R1 interprets the structured output from Docling to identify and extract specific fields with contextual understanding. Rather than processing raw PDF files, DeepSeek-R1 receives prestructured Markdown or JSON from Docling, allowing it to focus computational resources on semantic interpretation rather than layout analysis. During this stage, the model extracts named fields such as quarterly_revenue, operating_margin_expansion, and year_over_year_growth_rate while understanding their contextual relationships. DeepSeek-R1 also validates numerical calculations by checking whether computed values, such as gross_profit, correctly equal total_revenue minus cost_of_revenue, catching potential OCR errors or structure recognition mistakes from the preprocessing stage.

The final quality assurance stage implements validation workflows that cross-check critical fields using business rules and statistical anomaly detection. Financial extractions undergo threshold-based review, where fields exceeding expected value ranges (such as revenue changes greater than 50% quarter-over-quarter) trigger human verification. Organizations should implement automated tracking of accuracy metrics over time, measuring field extraction accuracy, entity recognition, and table parsing performance against curated ground truth datasets. As accuracy improves through prompt refinement and error analysis, confidence thresholds can gradually increase to reduce manual review overhead while maintaining quality standards.

Cost-benefit analysis and economic justification

The economic advantages of hybrid automation become compelling at scale. Combining Docling’s zero-cost local processing with DeepSeek-R1’s $0.0027 per document API fee yields a total cost of $2.70 per thousand documents processed. In contrast, traditional manual extraction by trained financial analysts costs between $50 and $100 per document, depending on complexity and required accuracy. For an organization processing 10,000 financial documents annually, manual extraction costs $500,000 to $1,000,000 per year, while the automated hybrid pipeline costs just $27 annually in API fees plus infrastructure expenses for running Docling locally.

This dramatic cost reduction of over 99% enables previously infeasible use cases such as comprehensive analysis of all supplier invoices, automated extraction from historical financial archives, or real-time processing of competitor financial disclosures. The per-document cost at scale ($0.0027) means processing one million documents costs only $2,700, making document understanding economically viable for applications that were prohibitively expensive with manual methods. Organizations must factor in infrastructure costs for self-hosting Docling (estimated at $5,000–15,000 annually for modest volumes) and engineering resources for pipeline development and maintenance (approximately $50,000–100,000 annually depending on complexity), but even with these additional costs, total expense remains under $120,000 annually for processing millions of documents.

If you wish to explore more case studies, such as Brainstorming and ideation and Code generation and understanding, please visit the GitHub repo of this chapter:

Brainstorming and ideation: https://github.com/PacktPublishing/DeepSeek-in-Practice/tree/main/Chapter04/brainstorming-ideation
Code generation: https://github.com/PacktPublishing/DeepSeek-in-Practice/tree/main/Chapter04/code-assistant

Summary

This chapter demonstrated DeepSeek-R1’s document understanding capabilities through two detailed case studies: Amazon API Gateway integration documentation and financial statement analysis. In the Amazon case study, DeepSeek-R1 analyzed a 45-page technical integration guide and achieved an 87% overall evaluation score by successfully extracting serverless architecture components, identifying security configurations, and generating production-ready CloudFormation templates. The model demonstrated strong comprehension of complex technical documentation, including API Gateway throttling policies, Lambda authorization patterns, JWT validation logic, and distributed tracing configurations.

The financial document analysis case study evaluated DeepSeek-R1 against specialized document parsing tools using a fictional quarterly report for FruitStand Innovation Inc. containing 192 lines, 6 complex tables, and 150 semantic fields. DeepSeek-R1 achieved 91.3% field extraction accuracy (137/150 fields), 94.7% table parsing accuracy (232/245 cells), and perfect entity recognition (42/42 entities) with a processing time of 4,250 milliseconds at $0.0027 per document. We compared DeepSeek-R1 against Docling (IBM Research) and MarkItDown (Microsoft), finding that Docling achieved higher table parsing accuracy (97.2%) through specialized structure recognition, MarkItDown provided the fastest processing (890 ms) with moderate accuracy (78.5%), and DeepSeek-R1 uniquely delivered semantic understanding with direct field extraction capabilities. Hybrid pipelines combining Docling’s structure extraction with DeepSeek-R1’s semantic interpretation achieved the best overall results.

Throughout both case studies, we applied consistent evaluation metrics, including qualitative assessment of technical accuracy and implementation readiness, and quantitative measurement of extraction accuracy, processing time, and cost efficiency. DeepSeek-R1 consistently performed at a professional consultant level with 88–92% accuracy across complex documents at $0.0027 per document compared to $50–100 for manual analysis, demonstrating 99%+ cost reduction for automated document understanding workflows. Organizations implementing DeepSeek-R1 for document understanding should establish clear evaluation metrics before deployment, defining both qualitative criteria, such as technical accuracy and completeness, alongside quantitative measures, including field extraction accuracy, entity recognition, processing time, and cost per document appropriate to their specific document types. Financial documents typically require 95%+ accuracy thresholds due to regulatory and compliance requirements, while general technical documentation may accept 85%+ accuracy with human validation for critical sections.

Successful implementation requires careful prompt engineering with explicit instructions about output format, required fields, numerical precision requirements, and domain-specific terminology. Our financial document analysis demonstrated that providing glossaries for common abbreviations, such as PP&E for Property, Plant & Equipment, and specifying whether to expand acronyms, significantly reduces extraction errors. For complex multi-column tables, explicitly instructing the model about column associations and time period comparisons minimizes nested cell confusion. Iterative prompting proves valuable when initial extractions miss critical fields, as follow-up prompts targeting specific gaps consistently improve completeness without requiring complete document reprocessing. Organizations should also consider hybrid architectures that combine tool strengths based on document characteristics, such as preprocessing PDF files with complex visual layouts using Docling to extract table structures at 97.2% accuracy, then using DeepSeek-R1 for semantic field extraction and entity recognition at a combined cost of only $0.0027 per document while achieving higher accuracy than either tool alone.

Next up, we will show how to create a working service using DeepSeek API. Stay tuned.

Get This Book’s PDF Version and Exclusive Extras

5 Building with DeepSeek

In the previous chapters we discussed the inner-workings of DeepSeek, the best strategies to effectively prompt DeepSeek models, and how to use DeepSeek. In this chapter, it’s time to get practical. We’ll tackle a real-world problem, and we’ll show you how you can leverage DeepSeek models to solve it.

DeepSeek models are open-source, which is both powerful and challenging. You get endless options: large models via APIs, or small, distilled models that you can run locally, which is powerful but comes with trade-offs. There is no single right way to use DeepSeek models; the best choice depends on your situation and requirements for your use case. Should you go for a local option? Should you go for a cloud option? Should you go for a small model, a big model, or a reasoning model?

Our goal in this chapter is twofold. We want to show you how to solve a particular use case (end-to-end) using DeepSeek models and demonstrate the main ways you can interact with or use DeepSeek models. During the chapter, you’ll learn all the different ways you can use models: local, through an API, in the cloud, and on your own machine.

But before we start, we need a use case. Let’s consider what Duarte has to stay about his day-to-day struggles with his smartwatch.

My wife and I have been runners for a long time. I won’t lie, I’m big into marathons. Running is one of those rare occasions when I actually physically get out from behind the desk and go outside. I love everything about it. But this is probably not the place to talk too much about it. We both use Garmin smartwatches; we love them!

Every day at exactly 8:45 p.m., my wife and I look at our watches, and at that same time, the watch sends us a small summary of our day (Stressful day, Workout day, or Rest day). Here is an example:

Workout day

Your intense workout today and periods of rest are good for your physical and mental health. Now take time to relax and focus on getting a good night’s sleep.

Figure 5.1: An example of the daily Garmin notification

For the first weeks of having my watch, they were great! It gives me yet another indicator to track and understand my body. But after a couple of weeks, you start realizing that you always receive one of five prewritten messages. From one perspective, this is cool; the watch understands my body! From another perspective, it’s annoying, since the watch is only sending me one of the five prewritten messages it has stored in memory. Also, the messages are just too general. It almost feels like the watch is not telling me anything I didn’t know already. After a while, it just feels boring.

We hear you Duarte!

When we three authors began brainstorming on this book, we realized that in the age of large language models (LLMs) such as DeepSeek, Garmin’s notifications can surely be better - much better. A more dynamic, personalized message - one that can take some sort of user input. You know what we mean. There are many ways we can make this better. So, here we are going to do exactly this.

In this chapter, we’ll build a program that can access my smartwatch data, read it, and provide me with an interesting and dynamic summary every day. No more five predefined messages; every time we receive the summary, the message will be different. We’ll leverage DeepSeek to analyze the data and build a nice summary for us. We’ll call it the Daily Health Summary.

A couple of disclaimers before we get started:

This Daily Health Summary will be generated by an LLM. As you likely know, LLMs can sometimes hallucinate or produce inaccurate information. We won’t be discussing guardrails or guaranteeing safe model outputs in this chapter. However, if you’re interested in learning more, here’s a helpful library to get you started: https://github.com/guardrails-ai/guardrails.
This chapter will focus on extracting data from your Garmin watch. We will use the Garth library (https://github.com/matin/garth) for authentication. While widely known and popular, it’s important to remember that it’s not an official Garmin library. Consequently, there are no guarantees that Garmin won’t change its authentication methods in the future. If you don’t have a Garmin device, you can still extract data from Strava or another service you use; however, you will need to handle the data extraction process yourself using those alternative sources.

All right, let’s get this started.

Here is what we will cover in this chapter:

Building our first prototype
Interacting with DeepSeek models
Deploying an isolated model service with AWS
Best practices and recommendations

Technical requirements

All the code for this chapter is in the book’s GitHub repository over at https://github.com/PacktPublishing/DeepSeek-in-Practice.

You should have the following installed on your machine:

uv: See the following link for instructions: https://docs.astral.sh/uv/getting-started/installation/.
Docker: We will be building Docker images for the API we will design. Check out this link to get started with the installation: https://docs.docker.com/engine/install/.

This chapter’s code is in the Chapter 05 directory; running uv sync inside the chapter folder will install all the dependencies.

This chapter assumes you have a DeepSeek API key, which you can acquire over at https://platform.deepseek.com/.

You might have to install other libraries/dependencies, but we will explicitly call them out during the chapter.

There are several ways to use DeepSeek, for example, the DeepSeek API, third-party APIs, or using some libraries like litellm to utilize DeepSeek models. We have covered a few of these options in the Appendix of this book. Go check them out.

Building the first prototype

In this first section, we’ll build an end-to-end prototype for the Daily Health Summary, following five high-level steps:

First, we’ll fetch data from the Garmin API and learn how to query it.
Next, we’ll shape that raw health data into a format that DeepSeek can understand and reason about.
Then, we’ll define the structured output: what we want DeepSeek to return and what it should look like.
Before wrapping up, we’ll refactor everything into an API you can query and use.
Finally, we will walk through deploying it all with Docker.

Let’s begin with fetching the data from Garmin.

Fetching our data

The first order of business is to query Garmin for the data it has about us. This is all about programmatically retrieving the health data our Garmin watch has about us (heart rate, sleep hours, stress, etc.). In order to get this done, we will leverage the python-garminconnect project (https://github.com/cyberjunky/python-garminconnect).

In order for this to work, you should have both your Garmin email and the password you use to log in to your Garmin account. Both of those should work with https://connect.garmin.com/signin.

Note

Don’t worry if you don’t have a Garmin account. The GitHub repo includes instructions for loading sample data, so you can run the APIs and examples without any issues.

Once you have those at hand, we can log in to Garmin with the following function:


# We get the values as environmental variables

GARMIN_EMAIL = os.getenv("GARMIN_EMAIL") # your email

GARMIN_PASSWORD = os.getenv("GARMIN_PASSWORD") # your password

# and we define our function

def start_garmin() -> Garmin:

    """Initialize Garmin connection."""

    try:

        GARMIN = Garmin(

            email=GARMIN_EMAIL,

            password=GARMIN_PASSWORD,

            is_cn=False, # if you are in China, set to True

        )

        GARMIN.login()

        print("You are now logged in.")

        return GARMIN

    except Exception as e:

        print(f"Could not login with email and password: {e}")

        raise

If everything goes well, you can now run the following:

garmin = start_garmin()

# prints: You are now logged in.

This function now gives us a garmin instance we can work with and query the API. Now that we know how to instantiate the Garmin API, let’s go ahead and query all the information we need for a given day. To do this, we create a get_daily_health_summary function. This function will receive an API instance, a start date, and an end date. The goal is then to return health information for that date range.

First, we’ll create a couple of auxiliary functions to convert a date to ISO format (as Garmin requires) and a function that receives two dates and returns a range we can iterate over:


def get_daily_health_summary(

    api: Any, start: datetime.date, end: datetime.date

) -> list[dict[str, Any]]:

    def dstr(d: datetime.date) -> str:

        return d.isoformat()

    def daterange(a: datetime.date, b: datetime.date):

        for i in range((b - a).days + 1):

            yield a + datetime.timedelta(days=i)

We now iterate through each day, and we can retrieve summary information from Garmin by using the get_user_summary method. This method returns a dictionary containing the fields we want for context, such as resting heart rate, steps, exercise minutes, and sleep duration:


 for day in daterange(start, end):

        s = dstr(day)

        day_of_week = day.strftime("%A")

        summary = api.get_user_summary(s) or {}

        rhr = summary.get("restingHeartRate")

        steps = summary.get("totalSteps")

        stress_level = summary.get("averageStressLevel")

        body_battery_final = summary.get("bodyBatteryMostRecentValue") or summary.get(

            "mostRecentBodyBattery"

        )

        exercise_minutes = (summary.get("moderateIntensityMinutes") or 0) + (

            summary.get("vigorousIntensityMinutes") or 0

        )

        sleep_seconds = summary.get("sleepingSeconds")

        sleep_hours = round(sleep_seconds / 3600, 2)

        body_battery_start = summary.get("bodyBatteryAtWakeTime")

        total_distance_meters = summary.get("totalDistanceMeters")

Finally, once we have extracted all the fields we’re interested in, we can now append them to the array that our function will return:


 out.append(

            {

                "date": s,

                "day_of_week": day_of_week,

                "resting_heart_rate": rhr,

                "exercise_minutes": exercise_minutes,

                "stress_level": stress_level,

                "sleep_hours": sleep_hours,

                "steps": steps,

                "total_distance_meters": total_distance_meters,

                "body_battery_start_day": body_battery_start,

                "body_battery_end_day": body_battery_final,

            }

        )

    return out

With this function in place, we can now call it for any day we’re interested in. For example, if we would like to get the data for yesterday, we can get today’s date, remove a day (using timedelta), and call the get_daily_health_summary method:


yesterday = datetime.date.today() - datetime.timedelta(days=1)

summary_for_yesterday = get_daily_health_summary(

    garmin, yesterday, yesterday)

summary_for_yesterday

After calling this function, we can see that on August 13th, I had a resting heart rate of 62 and slept for a total of 9 hours(!). I’m not sure what happened there, but that was certainly a lot of sleep! As they say, sleep is an athlete’s best friend:


[{'date': '2025-08-13', 'day_of_week': 'Wednesday', 'resting_heart_rate': 62, 'exercise_minutes': 0, 'stress_level': 38, 'sleep_hours': 9.09, 'steps': 1866, 'total_distance_meters': 1387, 'body_battery_start_day': 74, 'body_battery_end_day': 17}]

Keep in mind that we can also call our function for a date range, which will be helpful, as in this example:


past_7_days_start = yesterday - datetime.timedelta(days=7)

past_7_days_end = yesterday - datetime.timedelta(days=1)

summary_for_past_7_days = get_daily_health_summary(

    garmin, past_7_days_start, past_7_days_end

)

summary_for_past_7_days

This will return a list of the data for each day between past_7_days_start and past_7_days_end If this does not make a whole lot of sense, don’t worry just yet! Everything will come together soon:


[

    {

        'date': '2025-08-06',

        'day_of_week': 'Wednesday',

        'resting_heart_rate': 58,

        'exercise_minutes': 3,

        'stress_level': 32,

        'sleep_hours': 8.17,

        'steps': 5320,

        'total_distance_meters': 3953,

        'body_battery_start_day': 97,

        'body_battery_end_day': 20

    },

    {

        'date': '2025-08-07',

        'day_of_week': 'Thursday',

        # ...

    },

    {

        'date': '2025-08-08',

        # ...

Creating the context

We can now look up health data for specific dates or date ranges. Next, we need to get the data ready for our DeepSeek model. There are many ways to do this, but we want to keep it simple. We will create a prompt that includes a short Markdown section for each measurement. For each measurement (such as stress level), we want to show the LLM the following:

Today’s value of the metric.
The average from the past seven days (the baseline), not including today.
The percentage change from today compared to the baseline.
The trend in the baseline period.
Whether a lower value is better for this measurement.

For example, in the case of sleep hours, we want the following:


## Sleep Hours

- Today's value (2025-08-13): 9.09

- 7-day baseline average (excluding today): 8.21

- Percent change vs. baseline: +10.7%

- Trend over previous 7 days: up ↑

- Better is lower: False

This will be replicated for every other metric we retrieve.

With this information, we are now giving the model a metric under a certain context, instead of giving a metric in isolation. This will ensure the model hallucinates less and understands the wider context of the metric.

To identify trends, we need a heuristic. Here’s a simple function that accepts a list of numbers and returns up, down, or flat. It compares the mean of the first and last three values in the list. If these means are within 5%, the function returns flat, since there was no major change in the metric. It’s not a perfect heuristic, but it serves our purpose:


def detect_trend(values, pct_threshold=5):

    recent, earlier = np.mean(values[-3:]), np.mean(values[:3])

    return (

        "up"

        if recent > earlier * (1 + pct_threshold / 100)

        else ("down" if recent < earlier * (1 - pct_threshold / 100) else "flat")

    )

With our heuristic in place, we can now create a function to create the context; build_llm_context_md receives a summary for today and a summary for the past seven days. These are the same lists of dictionaries that carry information about health metrics in the current day and the previous seven days.

We then create a list of strings, where each string represents a line of context for the LLM. Finally, we create a better_is_lower array to store metrics in which lower scores are preferable, such as stress level:


def build_llm_context_md(

    summary_for_today: list[dict], summary_for_past_7_days: list[dict]

) -> str:

    assert len(summary_for_today) == 1, "Expected 1 day of summary"

    today = summary_for_today[0]

    metrics = [key for key in today.keys() if key not in ["date", "day_of_week"]]

    lines = [

        f"# Daily Metrics Summary for {today['date']} ({today['day_of_week']})",

        "_Note: All comparisons use the **previous 7 days only**, excluding today._",

        "",

    ]

    better_is_lower = [

        "resting_heart_rate",

        "stress_level",

    ]

After that, we iterate through our metrics. For each, we create a small list of values. The current value of the metric, the mean for the baseline period, is better if lower. After iterating, we then apply a join in all our lines to build up our context for DeepSeek:


 for metric in metrics:

        today_val = today[metric]

        if not today_val:

            print(f"Skipped {metric} since no values were found")

        past_vals = [

            day[metric] for day in summary_for_past_7_days 

            if day[metric] is not None

        ]

        avg_7d = sum(past_vals) / len(past_vals) if past_vals else 0

        delta_pct = ((today_val - avg_7d) / avg_7d * 100) if avg_7d else 0

        trend_dir = detect_trend(past_vals)

        arrow = "↑" if trend_dir == "up" else ("↓" if trend_dir == "down" else "→")

        lines.append(

            f"## {metric.replace('_', ' ').title()}\n"

            f"- Today's value ({today['date']}): {today_val}\n"

            f"- 7-day baseline average (excluding today): {avg_7d:.2f}\n"

            f"- Percent change vs. baseline: {delta_pct:+.1f}%\n"

            f"- Trend over previous 7 days: {trend_dir} {arrow}\n"

            f"- Better is lower: {metric in better_is_lower}\n"

        )

    return "\n".join(lines)

We can now call our function by doing the following:


# start client and define date

garmin = start_garmin()

date = "2025-08-13"

# get data for date of summary

date_for_summary = datetime.datetime.strptime(date, "%Y-%m-%d").date()

summary_in_date = get_daily_health_summary(

        garmin, date_for_summary, date_for_summary

)

# get data for 7 days before

past_period_start = date_for_summary - datetime.timedelta(days=7)

past_period_end = date_for_summary - datetime.timedelta(days=1)

summary_in_past_period = get_daily_health_summary(

        garmin, past_period_start, past_period_end

)

# build prompt

prompt =  build_llm_context_md(summary_in_date, summary_in_past_period)

print(prompt)

This outputs a nice summary that we can then feed to our LLM:


# Daily Metrics Summary for 2025-08-13 (Wednesday)

_Note: All comparisons use the **previous 7 days only**, excluding today._

## Resting Heart Rate

- Today's value (2025-08-13): 62

- 7-day baseline average (excluding today): 55.86

- Percent change vs. baseline: +11.0%

- Trend over previous 7 days: up ↑

- Better is lower: True

## Exercise Minutes

- Today's value (2025-08-13): 0

- 7-day baseline average (excluding today): 3.00

- Percent change vs. baseline: -100.0%

- Trend over previous 7 days: up ↑

- Better is lower: False

## Stress Level

- ....

## Sleep Hours

- ...

... rest of metrics hidden for brevity

Great! We now have all the information we would like to feed our DeepSeek model. But how exactly would we like it to respond?

Defining the structured output

It’s now time to define how we would like our Daily Health Summary to look. As with most LLM-backed applications, it’s usually a good practice to make the models return some sort of structured output(JSON format) (we covered these in Chapters 2 and 3). To represent our Daily Health Summary, we create two Pydantic (https://docs.pydantic.dev/) models:

DayType: Classifies our day into one of five types.
DailySummary: Creates an object that includes a day type, a title, an emoji, an observation, and a recommendation.

Here is the code:


from pydantic import BaseModel, Field

class DayType(str, Enum):

    TRAINING = "training"

    ACTIVE_RECOVERY = "active_recovery"

    REST = "rest"

    HIGH_STRESS = "high_stress"

    BALANCED = "balanced"

class DailySummary(BaseModel):

    day_type: DayType = Field(

        ...,

        description="Classification of the day based on activity and recovery metrics",

    )

    title: str = Field(..., description="One sentence summary of the day")

    emoji: str = Field(..., description="Emoji to represent the day type")

    observation: str = Field(

        ...,

        description="Two sentence observation about key metrics and patterns",

    )

    recommendation: str = Field(

        ...,

        description="Two sentence actionable recommendation for tomorrow",

    )

Now, we have all the elements to make our first Daily Health Summary. Let’s create the summary.

Creating the Daily Health Summary

We start by first creating a simple function that calls the DeepSeek API using type: json_object to enforce a JSON structure from the model. Nothing surprising here; it returns the JSON the model created and the reasoning content, if there is any (in the case of deepseek-reasoner, for example):


def llm(

    messages: list[dict], model: str, response_format: dict | None = None

) -> tuple[dict, str | None]:

    client = OpenAI(

        api_key=os.environ["DEEPSEEK_API_KEY"],

        base_url="https://api.deepseek.com",

    )

    response = client.chat.completions.create(

        model=model, messages=messages, response_format=response_format, temperature=0.0

    )

    message = response.choices[0].message

    if hasattr(message, "reasoning_content"):

        reasoning_content = message.reasoning_content

    else:

        reasoning_content = None

    return json.loads(message.content), reasoning_content

To ensure that our model behaves well, we create two Daily Health Summary examples and create our system prompt as well:


# Example 1: Training day

training_day_example = DailySummary(

    day_type=DayType.TRAINING,

    title="Strong training day with elevated activity across all metrics.",

    emoji="💪",

    observation="Exercise minutes doubled your baseline with 95 minutes of activity, supported by 18,500 steps. Despite the high training load, body battery started at a solid 85, indicating good recovery from yesterday.",

    recommendation="Consider an active recovery or rest day tomorrow to allow adaptation from today's effort. Prioritize sleep tonight to maintain your body battery levels and support muscle recovery.",

)

# Example 2: High Stress Day

high_stress_day_example = DailySummary(

    day_type=DayType.HIGH_STRESS,

    emoji="😫",

    title="Elevated stress and poor recovery despite minimal physical activity.",

    observation="Stress levels jumped 46% above baseline while sleep dropped to just 6.1 hours, resulting in a low body battery start of 45\. Exercise and movement were minimal, suggesting stress is from non-physical sources.",

    recommendation="Focus on stress management techniques and aim for 8+ hours of sleep tonight. Consider light exercise like walking or yoga tomorrow, as gentle movement can help regulate stress levels.",

)

examples = [training_day_example, high_stress_day_example]

examples_str = "\n\n\n".join([example.model_dump_json() for example in examples])

The system prompt has three components - the general instructions to give the model context and guidance, the JSON schema we expect from it, and some example JSON outputs as well:


system_prompt = f"""

Instructions:

* You will be given a summary of the user's health and fitness data for today, in comparison to the past 7 days.

* Your goal is to generate a summary that will be shown in the user's smart watch.

* Keep things short, but also interesting to the user.

* Your summary should include a type of day, a title, some observations and recommendations for the user.

* Your summary should be in JSON format. Only output the JSON, no other text.

---JSON SCHEMA---

{DailySummary.model_json_schema()}

---END JSON SCHEMA---

---EXAMPLE JSON OUTPUTS---

{examples_str}

---END EXAMPLE JSON OUTPUTS---

"""

Now that we have everything in place, we can create one last function. The role of this function is simply to tie everything that we’ve built up until now together. It does the following:

Receive a Garmin API instance and a date in string format.
Fetch data from Garmin for that date and for the seven days before.
Build up our prompt and list of messages, including our system prompt.
Call the model and validate its response against our DailySummary Pydantic model.


def get_daily_summary(

    garmin: Garmin,

    date: str,

    model: Literal["deepseek-chat", "deepseek-reasoner"],

    verbose: bool = False,

) -> DailySummary:

        # convert to datetime

    date_for_summary = datetime.datetime.strptime(date, "%Y-%m-%d").date()

    # get information for date

    summary_in_date = get_daily_health_summary(

        garmin, date_for_summary, date_for_summary

    )

        # get information for baseline period

    past_period_start = date_for_summary - datetime.timedelta(days=7)

    past_period_end = date_for_summary - datetime.timedelta(days=1)

    summary_in_past_period = get_daily_health_summary(

        garmin, past_period_start, past_period_end

    )

        # create prompt with trends and metrics

    prompt = build_llm_context_md(summary_in_date, summary_in_past_period)

        # create messages list

    messages = [

        {"role": "system", "content": system_prompt},

        {"role": "user", "content": prompt},

    ]

        # call LLM and validate response

    response, reasoning = llm(messages, model, {"type": "json_object"})

    health_summary = DailySummary.model_validate(response)

    return health_summary

From now on, all we need to do to get a Daily Health Summary is to call the get_daily_summary function with a date and the model we would like to use:


daily_summary = get_daily_summary(garmin, "2025-08-19", "deepseek-chat")

print(daily_summary.model_dump_json())

This then returns the Daily Health Summary we built:


{

   "day_type":"balanced",

   "title":"A well-balanced day with improved recovery and lower stress.",

   "emoji":"⚖️",

   "observation":"Resting heart rate decreased by 9.3% to 39, and stress level dropped significantly by 58.8%, indicating excellent recovery. Body battery started high at 82 and ended at 72, showing efficient energy use despite slightly reduced sleep and activity.",

   "recommendation":"Maintain this balance by continuing with moderate exercise and ensuring adequate sleep. Focus on stress management to keep levels low and support ongoing recovery."

}

As you can see, this works pretty well. Looking at the example, I can tell you it’s absolutely right. This was my first day of vacation in August, which makes the summary particularly accurate. We now have our first prototype. But building an application does not stop there. We want to build a full service that can compute these daily summaries for anyone who is interested in them. And for that, we need to build an API.

Refactoring into an API

Now that our system works as a simple function, it’s time we build an API for it. The goal is to create a service that can receive a request from a user, as well as their details to log in to Garmin’s API, and return a Daily Health Summary of their health metrics. To accomplish this, we will use FastAPI.

Security note

The API service uses your authentication credentials, including your password, to call the Garmin Connect API. This approach is acceptable for a prototype, but in production, users typically won’t want to share plaintext passwords with an unfamiliar service. For a real application, consider implementing OAuth or using the official Garmin Connect Developer Program: https://developer.garmin.com/gc-developer-program/activity-api.

Our API includes two endpoints. The first is a small quality-of-life improvement: when users visit / (the root), they are automatically redirected to /docs. This makes it easier for developers (especially those new to the project) to quickly find and explore the available endpoints and understand the expected outputs:


from fastapi import FastAPI, Header, HTTPException

from fastapi.responses import RedirectResponse

app = FastAPI(title="Garmin Health Summary API")

@app.get("/")

async def root():

    return RedirectResponse(url="/docs")

Once the app is instantiated, we create HealthSummaryRequest, which is the request we expect from the users. As you can see in the following, we expect them to provide a date, which will default to today’s date if not provided. If they want, the user can also specify one of the two models (deepseek-chat or deepseek-reasoner) to create the summary:


class Model(str, Enum):

    chat = "deepseek-chat"

    reasoner = "deepseek-reasoner"

class HealthSummaryRequest(BaseModel):

    date: str = Field(

        default_factory=lambda: datetime.date.today().isoformat(),

        description="Date in YYYY-MM-DD format, defaults to today",

        example=datetime.date.today().isoformat(),

    )

    model: Model = Field(

        default=Model.chat,

        description="Model to use for the summary",

    )

To authenticate, the user must provide the Garmin credentials as a header to the request. We return the same DailySummary request we’ve introduced previously:


@app.post("/health-summary", response_model=DailySummary)

async def get_health_summary(

    request: HealthSummaryRequest,

    garmin_email: str = Header(..., description="Garmin email address"),

    garmin_password: str = Header(..., description="Garmin password"),

) -> DailySummary:

The core of the get_health_summary endpoint runs the exact same logic we’ve seen before, and additionally adds some error handling in case things go wrong:


 """Get Daily Health Summary for a specific date.

    Args:

        request: Health summary request with date and model

        garmin_email: Garmin account email from header

        garmin_password: Garmin account password from header

    Returns:

        Daily Health Summary with AI-generated insights

    """

    try:

        garmin = get_garmin_client(garmin_email, garmin_password)

        summary = get_daily_summary(garmin, request.date, request.model)

        logger.info(

            f"Daily Health Summary API request completed successfully for {request.date}"

        )

        return summary

    except ValueError as e:

        logger.error(f"Invalid date format provided: {request.date}")

        raise HTTPException(

            status_code=400, detail=f"Invalid date format: {e}"

        )

    except Exception as e:

        logger.error(

            f"Failed to generate Daily Health Summary for {request.date}: {e}"

        )

        raise HTTPException(

            status_code=500, detail=f"Error generating summary: {e}"

        )

With all of this in place, we can now run our API with the following command:


$ uv run fastapi run 02-api.py

This will start a production server on port 8000 of your machine. When you go to http://0.0.0.0:8000, you’ll see the API page ready for you to interact with:

Figure 5.2: Garmin Health Summary API page

You can call the API through the interface or through the command line. If you have curl (https://curl.se/) installed, here’s an example of how to call it:


$ curl -X 'POST' \

        'http://localhost:8000/health-summary' \

        -H 'accept: application/json' \

        -H 'garmin-email: your.email@example.com' \

        -H 'garmin-password: your-password' \

        -H 'Content-Type: application/json' \

        -d '{

    "date": "2025-08-19"

  }'

This returns the model’s response in JSON format:


{

   "day_type":"rest",

   "title":"Rest day with minimal activity and good recovery indicators.",

   "emoji":"😴",

   "observation":"Exercise minutes and steps were significantly below your baseline, suggesting a deliberate rest. Body battery started high and stress levels were low, indicating effective recovery from previous days.",

   "recommendation":"Continue with light activity or rest tomorrow to maintain this positive recovery state. Ensure adequate sleep to keep your body battery levels optimal for future training."

}

The final aspect of our service is deployment. You might connect via SSH and run the fastapi run command, or even run it locally. However, when working with live production systems, you’ll likely use Docker.

Deploying with Docker

Now, your API runs. That’s great. Unfortunately, we can’t ship your computer to serve our users. If our application will run in production, it needs to be deployed somewhere in the cloud. This means that our application needs to run on another computer. How can we make sure we have everything we need installed on that computer as well? For the past decade, the answer has been containers.

Docker is a well-established tool in the industry that uses containers to declaratively define dependencies and keep applications isolated when deploying. In short, Docker lets you package your application and everything it needs – code, libraries, and tools – into a single, lightweight container. This ensures that it runs the same way on any machine, from your laptop to production, without environment issues. Even though we would love to write a long book about the amazing and less amazing things about Docker, this is not the book for that. If you are not familiar with it and want to learn the basics, we advise you to start here: https://www.docker.com/101-tutorial.

Now, here’s our heavily commented Dockerfile that packages all the dependencies of our Daily Health Summary service. If you are interested in diving deeper into some of the choices here, we recommend you dive deeper into the uv Docker tutorial/walkthrough, as our Dockerfile is greatly inspired by it (https://docs.astral.sh/uv/guides/integration/docker/):


# First, build the application in the `/app` directory

FROM ghcr.io/astral-sh/uv:bookworm-slim AS builder

ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy

# Configure the Python directory so it is consistent

ENV UV_PYTHON_INSTALL_DIR=/python

# Only use the managed Python version

ENV UV_PYTHON_PREFERENCE=only-managed

# Install Python before the project for caching

RUN uv python install 3.12

WORKDIR /app

RUN --mount=type=cache,target=/root/.cache/uv \

    --mount=type=bind,source=uv.lock,target=uv.lock \

    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \

    uv sync --locked --no-install-project --no-dev --no-group local

COPY . /app

RUN --mount=type=cache,target=/root/.cache/uv \

    uv sync --locked --no-dev --no-group local

# Then, use a final image without uv

FROM debian:bookworm-slim

# Copy the Python version

COPY --from=builder --chown=python:python /python /python

# Copy the application from the builder

COPY --from=builder --chown=app:app /app /app

# Place executables in the environment at the front of the path

ENV PATH="/app/.venv/bin:$PATH"

# Expose the port

EXPOSE 8000

# Run the FastAPI application

CMD ["fastapi", "run", "app/02-api.py", "--host", "0.0.0.0", "--port", "8000"]

We can run the application by building the container and running it on our own machine with the docker build and docker run commands. The first builds an image called API using 02-api.Dockerfile as the Dockerfile, and the second starts a container from that image, exposing port 8000 and passing the DEEPSEEK_API_KEY environment variable:


$ docker build -t api -f 02-api.Dockerfile .

$ docker run -it -e DEEPSEEK_API_KEY=sk-... -p 8000:8000 -t api

The application should again spin up on your local URL: http://localhost:8000.

That was a lot! But you should be happy. You created a service that receives a user’s Garmin credentials and returns a DeepSeek-powered Daily Health Summary of their current day. Congratulations! Now, even though we would love to show you how to run this on a server, we will not do it in this book. The beautiful thing about containers is that you can deploy them pretty much anywhere. Almost all cloud providers have at least 5 ways of deploying containers on their clouds, although AWS has 17 different ones: https://www.lastweekinaws.com/blog/the-17-ways-to-run-containers-on-aws/.

After a deserved break, you’ll begin to think that while the system functions well, it isn’t specific to DeepSeek. We are simply calling the DeepSeek API. We could be calling any other API (OpenAI, Anthropic, etc.). However, DeepSeek’s true power lies in its open source nature. This opens some doors for us. We will explore them next.

Interacting with DeepSeek models

Our Daily Health Summary service works well, but we would like to start abstracting away the fact that we are calling the DeepSeek API. The first service we should talk about is LiteLLM (https://docs.litellm.ai/).

LiteLLM

LiteLLM is a library that allows us to interact easily with different LLMs and providers using a single unified format.

The advantage of LiteLLM is that when you change providers, the only thing you really need to change is the model string that you are feeding into the completion function.

In the following example, we demonstrate how to use DeepSeek with three different providers by leveraging the LiteLLM library:

deepseek/deepseek-chat: Uses the DeepSeek chat model available in the DeepSeek API
bedrock/us.deepseek.r1-v1:0: Uses Amazon Bedrock (https://aws.amazon.com/bedrock/) to run inference of the DeepSeek-R1 reasoning model.
openrouter/deepseek/deepseek-r1-distill-qwen-14b: Uses OpenRouter (https://openrouter.ai/) to run the Qwen 14B distilled model
Warning

You might have to set different API keys and authentication methods for using different providers with LiteLLM. Make sure to check out their documentation for more information. It’s extensive!

Here is how you can utilize litellm:


from litellm import completion

def litellm(messages: list[dict]) -> tuple[str, str | None]:

    response = completion(

        # model="deepseek/deepseek-chat", # Uses DeepSeek API

        # model="bedrock/us.deepseek.r1-v1:0", # Uses AWS bedrock for inference

        model="openrouter/deepseek/deepseek-r1-distill-qwen-14b", # Uses OpenRouter for inference

        messages=messages,

        temperature=0.0,

    )

    message = response.choices[0].message

        # if there is reasoning content, extract it

    if hasattr(message, "reasoning_content"):

        reasoning_content = message.reasoning_content

    else:

        reasoning_content = None

    return message.content, reasoning_content

# usage example:

messages = [

    {"role": "system", "content": "You are a helpful assistant."},

    {"role": "user", "content": "What is the capital of France?"},

]

result, reasoning = llm(messages)

print(f"Result: {result}")

# Result: The capital of France is Paris.

print(f"Reasoning: {reasoning}")

# Reasoning: Okay, so I need to figure out the capital ...

LiteLLM is a powerful tool, and in combination with OpenRouter, it means you can call any API and model, or even local models! Yes, we said local... But let’s take this one step at a time.

Running locally with Ollama

The release of DeepSeek-R1 in January 2025 shocked the world, and it did so for many reasons. Not only did the flagship R1 model beat benchmarks and prove to be competitive with the state of the art of the moment, but the Chinese lab went one step further. You may recall from our discussion in Chapters 1 and 2 that, together with the flagship model, they also released six different distilled models.

Each model was fine-tuned from a base open source model using samples generated by the DeepSeek-R1 model. The original model contains 685 billion parameters, totaling 641.33 GB of weights. Roughly, running the model would require approximately 1,540 GB of VRAM to run it at full precision. Now, you might have that type of compute in your backyard, but the reality is that very few companies have that sort of compute power, since it requires a multi-GPU setup.

But the distilled versions tell a different story. In Table 5.1, you can see the distilled models, their base models (from which they were fine-tuned), and the VRAM requirements to run them at 4-bit quantization. This opens doors for those of us without a GPU farm in the backyard. Some of these can run on very small GPUs. In fact, we can even run them on a single GPU!

Table 5.1: DeepSeek models’ VRAM requirements

There are two well-established tools for running LLMs locally. The first is llama.cpp (https://github.com/ggml-org/llama.cpp), and the second, which uses the first as its inference engine, is Ollama (https://ollama.com/).

Running a local model with Ollama is simple. After installing Ollama (https://ollama.com/download), just enter the following command in your terminal:


$ ollama run deepseek-r1:1.5b

>>> Hi there!

Hello! How can I assist you today? 😊

This command will download the distilled 1.5B model if you haven’t run it before and start a chat session. Depending on your system, it may run warm, but the 1.5B parameter model should run on most modern computers.

We can use the Ollama Python SDK (https://github.com/ollama/ollama-python) to generate our daily health summaries. We just need to adapt the get_daily_summary function, and leverage the 1.5B parameter distilled model we just downloaded:


from ollama import chat

def get_daily_summary(

    garmin: Garmin,

    date: str,

) -> DailySummary:

    prompt = get_daily_summary_prompt(garmin, date)

    messages = [

        {"role": "system", "content": SYSTEM_PROMPT},

        {"role": "user", "content": prompt},

    ]

    response = chat(

        messages=messages,

        model="deepseek-r1:1.5b",

        format=DailySummary.model_json_schema(),

        options={

            "temperature": 0.0,

        },

    )

    return DailySummary.model_validate_json(response.message.content)

We can specify the return format with Ollama by setting format=DailySummary.model_json_schema(). This passes the Pydantic model’s JSON schema to the model as well. We can call the function as follows:


garmin = get_garmin_client(

        email=os.environ["GARMIN_EMAIL"],

        password=os.environ["GARMIN_PASSWORD"],

    )

summary = get_daily_summary(garmin, "2025-08-14")

print(summary)

This produces the following:


day_type=<DayType.HIGH_STRESS: 'high_stress'> title='Elevated stress and poor recovery despite minimal physical activity.' emoji='😫' observation='Stress levels jumped 46% above baseline while sleep dropped to just 6.1 hours, resulting in a low body battery start of 45\. Exercise and movement were minimal, suggesting stress is from non-physical sources.' recommendation='Focus on stress management techniques and aim for 8+ hours of sleep tonight. Consider light exercise like walking or yoga tomorrow, as gentle movement can help regulate stress levels.'

Yes, today wasn’t a great day. But we just connected a local 1.5B parameter model to our Daily Health Summary service! Before we move forward, remember that you can also use local models with LiteLLM as we’ve shown you before, by setting the model to, for example, ollama_chat/deepseek-r1:1.5``b.

CPU-based inference with Transformers and XGrammar

Ollama is fantastic for quickly testing and experimenting with local models. But to truly build out our Daily Health Summary service, we need finer control over the device, the prompt structure, tokenization, and the generation process itself. This is where we move closer to the metal and lean on the well-established Transformers library (https://github.com/huggingface/transformers).

To run our 1.5B parameter model on a CPU, let’s run the Unsloth version of the 1.5B distilled model, which you can find on Hugging Face as well. Unsloth (https://docs.unsloth.ai/) is another interesting library worth exploring if you are into fine-tuning and running models yourself.

We first load the model with the following code:


from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

model_name = "unsloth/DeepSeek-R1-Distill-Qwen-1.5B"

device = "cpu"

question = "What is the capital of Le Marche, Italy?"

max_tokens = 1200

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(model_name).to(device)

We use apply_chat_template on the messages to transform them into model inputs:

messages =

[

    {"role": "user", "content": question},

]

inputs = tokenizer.apply_chat_template(

    messages,

    add_generation_prompt=True,

    tokenize=True,

    return_dict=True,

    return_tensors="pt",

).to(model.device)

With that in place, we can now stream tokens out of our CPU directly:


streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=False)

model.generate(

    **inputs,

    max_new_tokens=max_tokens,

    streamer=streamer,

)

# prints Okay, so I need to figure out the capital of Le Marche, Italy. Hmm, I'm not super familiar with Italy, but I know it's one of the largest and most populous countries in Europe....

That runs our model directly on the CPU using the transformers library. This works great, but if we want to integrate this model with the rest of our Daily Health Summary service, we need to ensure that the model respects our DailySummary format effectively. And that’s where the real magic of open source models shines.

Remember, the model runs locally, so we have full control over the tokens it generates. We could write a function to restrict each token to those that comply with our defined JSON schema. But this is exactly what libraries such as outlines (https://dottxt-ai.github.io/outlines), xgrammar (https://github.com/mlc-ai/xgrammar), and guidance (https://github.com/guidance-ai/guidance) do out of the box. They perform constrained generation. Instead of simply asking the model to generate JSON, they explicitly limit the tokens to those that conform to the JSON structure.

From our experience, xgrammar has produced the best results. The code to use it is somewhat convoluted, but if you have questions, we encourage you to consult the library’s documentation. Still, we’ll explain it at a high level.

The first part of the code simply imports the model, places it on the CPU, and sets the desired seed (for reproducible outputs):


import xgrammar as xgr

from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

MODEL_NAME = "unsloth/DeepSeek-R1-Distill-Qwen-1.5B"

MODEL = AutoModelForCausalLM.from_pretrained(

    MODEL_NAME,

    torch_dtype=torch.float32,

    device_map="cpu",

)

TOKENIZER = AutoTokenizer.from_pretrained(MODEL_NAME)

CONFIG = AutoConfig.from_pretrained(MODEL_NAME)

MAX_NEW_TOKENS = 1024

transformers.set_seed(42)

Now, we’ll adapt our get_daily_summary function so that we can adapt our API. We take the messages, but now run them through apply_chat_template as well:


def get_daily_summary(

    garmin: Garmin,

    date: str,

) -> DailySummary:

    prompt = get_daily_summary_prompt(garmin, date)

    messages = [

        {"role": "system", "content": SYSTEM_PROMPT},

        {"role": "user", "content": prompt},

    ]

    texts = TOKENIZER.apply_chat_template(

        messages, tokenize=False, add_generation_prompt=True

    )

At first glance, the code may look intimidating, but it’s really just doing four things:

It begins by extracting tokenizer metadata with TokenizerInfo.from_huggingface, providing the basis for interpreting tokens correctly.
During generation, it applies an xgrammar logits processor that forces the model to only produce tokens valid under the DailySummary schema.
The generation step itself runs with a low temperature and sampling enabled, striking a balance between consistency and variety.
Finally, the output is decoded and validated with DailySummary.model_validate_json, returning a clean, structured Pydantic object.
Temperature

The temperature parameter controls the randomness of model outputs. Lower values make responses more deterministic, and higher values provide more creative answers. In reality, temperature manipulates the probabilities of each token being generated by DeepSeek.

Patrick von Platen has an outstanding tutorial that dives deep into text generation with transformers (including parameters such as temperature): https://huggingface.co/blog/how-to-generate.

Together, these pieces ensure that we always get exactly what we need – clean and valid JSON every time:


 model_inputs = TOKENIZER(texts, return_tensors="pt").to(MODEL.device)

  tokenizer_info = xgr.TokenizerInfo.from_huggingface(

      TOKENIZER, vocab_size=CONFIG.vocab_size

  )

  grammar_compiler = xgr.GrammarCompiler(tokenizer_info)

  compiled_grammar = grammar_compiler.compile_json_schema(DailySummary)

  xgr_logits_processor = xgr.contrib.hf.LogitsProcessor(compiled_grammar)

  generated_ids = MODEL.generate(

      **model_inputs,

      max_new_tokens=MAX_NEW_TOKENS,

      logits_processor=[xgr_logits_processor],

      do_sample=True,

      temperature=0.01,

      top_p=0.95,

      top_k=50,

  )

  generated_ids = generated_ids[0][len(model_inputs.input_ids[0]) :]

  model_response = TOKENIZER.decode(generated_ids, skip_special_tokens=True)

  return DailySummary.model_validate_json(model_response)

We set temperature to 0.01 for deterministic outputs, favoring high-probability tokens. top_k=50 limits choices to the 50 most likely tokens per step, while top_p=0.95 restricts generation to tokens covering 95% of the probability mass, trimming the unlikely long tail. These choices aren’t strict, so feel free to experiment yourself.

With xgrammar in place, we can now refactor our Daily Health Summary service to run entirely locally. No API calls required, no JSON malformations.

Refactoring for local generation on the CPU

Now that we’ve replaced get_daily_summary, we can create a completely new API that runs locally. We’ve included a 05-api-cpu-xgrammar.py file in Chapter05 of the repository for this book. This looks exactly like the first model service we designed, but runs 100% locally.

To run it, we use FastAPI again with the following:


$ uv run fastapi run 05-api-cpu-xgrammar.py

In another terminal, we can call the API with the command line:


$ curl -X 'POST' \

        'http://localhost:8000/health-summary' \

        -H 'accept: application/json' \

        -H 'garmin-email: your.email@example.com' \

        -H 'garmin-password: your-password' \

        -H 'Content-Type: application/json' \

        -d '{

    "date": "2025-08-19"

  }'

This again returns well-formatted JSON produced from DeepSeek-R1-Distill-Qwen-1.5B, running 100% on your own machine:


{

   "day_type":"high_stress",

   "title":"Elevated stress and poor recovery despite minimal physical activity.",

   "emoji":"😫",

   "observation":"Stress levels jumped 46% above baseline while sleep dropped to just 6.1 hours, resulting in a low body battery start of 45\. Exercise and movement were minimal, suggesting stress is from non-physical sources.",

   "recommendation":"Focus on stress management techniques and aim for 8+ hours of sleep tonight. Consider light exercise like walking or yoga tomorrow, as gentle movement can help regulate stress levels."

}

The Dockerfile is 99% similar except for a couple of dependencies, so we’ll skip it for brevity.

Phew! Well done! You just created a fully local Daily Health Summary service. You learned how to run DeepSeek models on your CPU, ensuring that they produce reliable JSON, and gained a clear understanding of how to deploy and integrate them with the service we designed. That’s a lot to take in.

However, it’s important to note that what we designed for the 1.5B parameter model can also be applied to any other DeepSeek model, with one major caveat: you need specialized hardware. You probably won’t be able to run a model larger than the 1.5B version on a CPU. For anything bigger, we’ll need a GPU. Some of you may have powerful computers, but the reality is that larger models tend to perform better than smaller ones.

Let’s say we’d like to run our Daily Health Summary with the 70B parameter distilled version of DeepSeek-R1; we would need at least 46 GB of VRAM. I have a pretty high-end MacBook with 64 GB of unified memory, but the reality is that I’ve never managed to run anything larger than a 30B parameter model comfortably. And we’re not going to ship my MacBook.

You get the point. We’ll need to run this on someone else’s computer – that is, in the cloud. However, we still want to the advantages of a local model, where we can tightly control generation. In the next section, we’ll explore how to deploy one of the larger distilled models on the cloud. Buckle up, this is going to get interesting!

Deploying an isolated model service with AWS

Amazon Web Services (AWS) is one of the most popular cloud computing platforms in the world. We’ll be focusing on it to deploy a larger DeepSeek model in order to integrate it with our Daily Health Summary service. As with most cloud computing platforms, there are just short of 25 different ways of achieving the same thing. And AWS is no different.

To use/deploy a DeepSeek model on AWS, you could do the following:

Use the Bedrock API, where they host foundational models (including DeepSeek). Here, you can call and consume models just like we did with the DeepSeek API. You won’t deploy; you’ll simply consume.
Use Bedrock Custom Model Import, a service where AWS allows you to import a trained model into Bedrock, and then use it via the Bedrock API. The model should be in Hugging Face format. Keep in mind that only some architectures are supported (no support for Gemma models, for example).
Hugging Face Deep Learning Containers (DLC) can also be used to deploy models on AWS. For inference, they include either PyTorch or Text Generation Inference (TGI, https://github.com/huggingface/text-generation-inference) as supported frameworks. An interesting option if you’d like to use those frameworks. You can also use DLC for training models. These will be done through AWS’s SageMaker service.
Rolling your own: By this we mean that you purchase a VM in the cloud (i.e., an EC2 instance) with a GPU, install whatever packages you need to install, manage dependencies, access, upgrades, and so on – maximum control, but likely a good headache as well.
Finally, large model inference (LMI) containers through SageMaker allow you to serve LLMs with a number of available inference backends. This is made for models with larger sizes, where you need full control, but don’t want to manage the infrastructure – close to the metal, but without many headaches. LMI containers are powered by the Deep Java Library (DJL) framework. Think of DJL as the engine that powers those containers.

This is not an exhaustive list. AWS offers more options, some of which will persist for years, while others likely won’t. There is no single right way to deploy a model on AWS; however, you should consider the trade-offs before choosing. We have compiled a table (Table 5.2) of the pros and cons of each option and when to choose which:

No infrastructure to manage
Access to many frontier models (including Anthropic, Mistral, and DeepSeek)
Authentication integrated
Automatic scaling and SLAs

You don’t own the model
Limited customization
Costlier per token at scale

| You just need to consume models via API and don’t want to host anything |
| Bedrock custom model import |

Deploy your fine-tuned model without managing infrastructure
Integrates with Bedrock APIs, agents, and knowledge bases
Benefits from Bedrock security and logging

Limited architecture support
Must convert model to Hugging Face format
More setup than hosted FMs

| You have your own trained model, but want to leverage Bedrock’s managed environment |
| Hugging Face DLC (SageMaker) |

Managed training and inference
TGI included for optimized LLM inference
Easy integration with Hugging Face Hub
Auto-scaling endpoints

Limited to the Hugging Face, TGI, and PyTorch stack
Less optimized than DJL for very large models
Still pay for SageMaker infrastructure

| You want a Hugging Face-native path for training or inference on SageMaker |
| Hand-rolled on EC2 |

Maximum control
Pick any serving stack, such as vLLM, TGI, and Triton
No SageMaker or Bedrock overhead

You manage everything, including scaling, security, and networking
Harder to operate reliably
Higher operational burden

| You need full control or want to experiment with custom stacks and kernels |
| Large model inference (LMI/DJL) |

Purpose-built for large models
Supports vLLM, TensorRT-LLM, and NeuronX for Inf2
Manages infrastructure, scaling, and sharding
Best throughput on big models

Slightly more setup than Hugging Face DLC
Requires SageMaker knowledge
Less Hugging Face magic out of the box

| You need maximum performance for large or production-scale models without managing infrastructure |

Table 5.2: Comparison of deployment options with AWS

We recommend almost always starting with an API call and avoiding complex self-hosting unless you know your product truly needs it. So, if you’re building products, start with an API call. Once you have a million users, it’s time to start thinking about deploying. We eat our own dog food in this book, which is why we started with consuming DeepSeek models via an API. And now we’ll venture into deploying DeepSeek using LMI containers. This is because we believe vLLM is an incredibly powerful inference engine (https://github.com/vllm-project/vllm).

You are probably already asking yourself: What is an inference engine anyway? Good question.

Inference backends

An inference engine is a framework that simplifies the process of deploying an LLM. It provides packaging, optimization, and serving capabilities. This is so you don’t have to think about GPU optimizations, request management, memory optimization, LLM architectures, streaming, or support for structured generation. An inference backend provides all of these, so you can focus on using your models instead. We’ve compiled another (non-exhaustive) list of inference engines in Table 5.3:

Table 5.3: Comparison of popular inference engines

Even though each provides pros and cons, vLLM has gradually established itself as an established engine. It also supports a lot of features we are interested in for our Daily Health Summary service: structured generation, OpenAI-compatible APIs, quantization, first-class support for DeepSeek models, and even support for reasoning outputs!

Deploying DeepSeek with LMI containers

SageMaker supports a wide variety of container images for training and inference jobs. You can see an exhaustive list over at the AWS DLC repository in GitHub: https://github.com/aws/deep-learning-containers.

Within this ecosystem, we will focus on the LMI containers specifically, because of their first-class support for vLLM. LMI containers are hosted and documented as part of the DJL: an open source Java framework for deep learning. Confusing, we know.

The point here is that if we are interested in the documentation for the containers we will deploy, it’s over at https://docs.djl.ai/.

The first thing we need to figure out is the total amount of memory we will need for our model. We’ll focus on deploying DeepSeek-R1 Distill Qwen 14B for our use case. To do so, we’ll use an ml.g6.12xlarge AWS instance. That instance will give us around 96 GB of GPU VRAM, which should be enough to run our model.

Warning

In this deployment, we will use an ml.g6.12xlarge machine. Depending on the region, it costs around 5 USD per hour. Forget to turn it off, and you’ll be charged 3,720 USD per month! ALWAYS make sure you turn off resources after testing. You have been warned.

In Table 5.4, we provide some guidance regarding which machines to use for different popular open source models:

| Parameters | Instance types | GPU VRAM | Example models |
| ~7B | ml.g5.4xlarge, ml.g6.4xlarge | 24 GB | Llama 2 7B, Mistral 7B, DeepSeek-LLM 7B, DeepSeek-R1 Distill Qwen 7B |
| ~13B | ml.g5.12xlarge, ml.g6.12xlarge | 96 GB | Llama 2 13B, Code Llama 13B, Qwen2.5 14B, DeepSeek-R1 Distill Qwen 14B |
| ~20B | ml.g5.12xlarge, ml.g6.12xlarge | 96 GB | GPT-OSS 20B, Mistral Small 24B, Solar Pro 22B |
| ~35B | ml.g5.48xlarge, ml.g6.48xlarge | 192 GB | Code Llama 34B, Falcon 40B, Yi 34B, DeepSeek-R1 Distill Qwen 32B |
| ~70B | ml.g5.48xlarge, ml.g6.48xlarge | 192 GB | Llama 2 70B, Code Llama 70B, DeepSeek-R1 Distill Llama 70B, DeepSeek-R1 Distill Qwen 70B |
| ~70B | ml.p4d.24xlarge | 320 GB | - |
| ~180B | ml.p4de.24xlarge, ml.p5.48xlarge | 640 GB | Falcon 180B, GPT-OSS 120B, DeepSeek V2 236B (MoE), DeepSeek V3 671B (37B active) |

Table 5.4: Instance types options for popular open source models

To deploy our model, we will take advantage of SageMaker’s Python SDK (https://sagemaker.readthedocs.io/). The following script creates a general configuration to deploy the DeepSeek model. You can replace the model with the one you are interested in deploying, and adapt the machine configuration according to the preceding cheat sheet (keeping the costs in mind). Before you run it, you should ensure the following:

You have an AWS account with billing enabled and the AWS CLI installed. If you don’t have the CLI yet, you can grab it here: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html.

You’ve created an IAM role with the AmazonSageMakerFullAccess policy attached, and set up the following trust relationship so SageMaker can assume the role (make sure to grab the Amazon Resource Name (ARN)of the role after creating it):


{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Principal": {

                "Service": "sagemaker.amazonaws.com"

            },

            "Action": "sts:AssumeRole"

        }

    ]

}

The SageMaker Python SDK is already included in this project’s dependencies, so you don’t need to install it separately.
You might have to request a quota increase for the instance type that you are using. For the following script, you’ll need a service quota of at least one ml.g6.12xlarge instance. There is some good documentation on how to do that in the AWS documentation: https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html.

Here is how to deploy your model:


import boto3

import sagemaker

import os

import time

# setup session

boto_session = boto3.session.Session()

region = boto_session.region_name

sess = sagemaker.Session()

# create our configuration

CONFIG = {

    "INSTANCE_TYPE": "ml.g6.12xlarge", # the instance type

    "ENV": {

        "HF_MODEL_ID": "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",  # hugginface model id

        "TENSOR_PARALLEL_DEGREE": "max",  # set max to use all GPUs

        "OPTION_ENABLE_REASONING": "true", # enable reasoning parsing

        "OPTION_REASONING_PARSER": "deepseek_r1", # set reasoning parser

    },

    "IMAGE_NAME": "djl-inference:0.33.0-lmi15.0.0-cu128", # image

    "BEDROCK_ROLE_ARN": "arn:aws:iam::ACCOUNT_NUMBER:role/replace-with-yours", # role

}

# get base model name

ts = time.time()

hf_model_id = CONFIG["ENV"]["HF_MODEL_ID"]

model_name = hf_model_id.split("/")[-1]

base_model_name = sagemaker.utils.name_from_base(model_name)

# create SageMaker model

current_region = os.environ["AWS_DEFAULT_REGION"]

image_name = CONFIG["IMAGE_NAME"]

inference_image_uri = (

    f"763104351884.dkr.ecr.{current_region}.amazonaws.com/{image_name}"

)

lmi_model = sagemaker.Model(

    image_uri=inference_image_uri,

    env=CONFIG["ENV"],

    role=CONFIG["BEDROCK_ROLE_ARN"],

    name=base_model_name,

)

endpoint_name = f"{base_model_name}-endpoint"

# print details

print(f"{base_model_name=}")

print(f"{current_region=}")

print(f"{inference_image_uri=}")

print(f"{endpoint_name=}")

# deploy model (this can take up to 15 mins!)

lmi_model.deploy(

    initial_instance_count=1,

    instance_type=CONFIG["INSTANCE_TYPE"],

    container_startup_health_check_timeout=900,

    endpoint_name=endpoint_name,

)

print(f"Model has been deployed to endpoint {endpoint_name}")

The deployment of the model (depending on your region and instance availability) might take up to 15 minutes. Once everything is done, when you navigate to the AWS console, you’ll notice that a new model and endpoint have just been created:

Figure 5.3: The model we created on AWS

Figure 5.4: The dashboard for the SageMaker endpoint

Once the model is deployed, we can navigate to the Amazon SageMaker AI console. When you click the Inference section, you can navigate to Inference > Models, and you’ll see our newly deployed DeepSeek model. Navigating to Inference > Endpoints will show you our model’s endpoint, as well as a set of metrics for you to monitor the status of your endpoint (CPU utilization, memory utilization, GPU memory, GPU utilization, and invocation/request metrics).

Danger

Once you are done with your model endpoint, remember to shut it down, or else you’ll continue to be charged for the time the instance is turned on! For this, you can delete the endpoint on the AWS console or use the SageMaker SDK:


import sagemaker

sess = sagemaker.Session()

sess.delete_endpoint(endpoint_name)

sess.delete_endpoint_config(endpoint_name)

lmi_model.delete_model()

LMI containers support three different API schemas: OpenAI’s Chat Completions format, the OpenAI Completions format, and the TGI format. If we’d like to make predictions, the following three examples are effectively equivalent:


llm = sagemaker.Predictor(

    endpoint_name=endpoint_name,

    sagemaker_session=sess,

    serializer=sagemaker.serializers.JSONSerializer(),

    deserializer=sagemaker.deserializers.JSONDeserializer(),

)

TEMPERATURE = 0.01

MAX_TOKENS = 1024

USER_MESSAGE = "What is the capital of Le Marche, Italy?"

# 1\. OpenAI chat completions

llm.predict(

    {

        "messages": [

            {

                "role": "user",

                "content": user_message,

            }

        ],

        "temperature": TEMPERATURE,

        "max_tokens": MAX_TOKENS,

    }

)

# 2\. OpenAI Completions format

llm.predict(

    {

        "prompt": user_message,

        "temperature": TEMPERATURE,

        "max_tokens": MAX_TOKENS,

    }

)

# 3\. TGI format

llm.predict(

    {

        "inputs": user_message,

        "parameters": {

            "max_new_tokens": MAX_TOKENS,

            "temperature": TEMPERATURE,

        },

    }

)

However, as we mentioned before, one of the big reasons to choose LMI containers is the fact that it supports vLLM as an inference engine and various options for structured outputs. In fact, LMI containers support the Chat Completions format because LMI containers use vLLM as an inference engine. One of the options for using structured generation with vLLM (https://docs.vllm.ai/en/v0.8.1/features/structured_outputs.html) is xgrammar.

Next up is an example of a function called llm that will use our SageMaker endpoint with structured outputs to return a prediction that respects the response_model that we pass to it. As you can see, vLLM supports response_format and an extra_body parameter where we can specify the Pydantic model we would like it to respect, as well as the decoding backend we wish to use:


def llm(

    messages: list[dict],

    endpoint_name: str,

    response_model: BaseModel,

) -> BaseModel:

    client = get_aws_llm(endpoint_name=endpoint_name)

    response = client.predict(

        {

            "messages": messages,

            "temperature": 0.01,

            "max_tokens": 1024,

            "response_format": {

                "type": "json_schema",

                "json_schema": {

                    "name": response_model.__name__,

                    "schema": response_model.model_json_schema(),

                    "strict": True,

                },

            },

            "extra_body": {"guided_decoding_backend": "xgrammar"},

        }

    )

    return response_model.model_validate_json(

        response["choices"][0]["message"]["reasoning_content"]

    )

We can then use this function by providing a Pydantic model and a list of messages, as in this example:


class Capital(BaseModel):

        capital: str

        population: int

messages = [

        {

                "role": "user",

                "content": "What is the capital of France?",

        }

]

response = llm(

        messages,

        "DeepSeek-R1-Distill-Qwen-14B-2025-08-24-08-52-31-391-endpoint",

        response_model=Capital,

)

print(response)

# capital='Paris' population=2161512

So, now we have our large DeepSeek model deployed in AWS with an endpoint that supports any feature that the vLLM inference engine does. The only step left is to integrate it again with our Daily Health Summary service. Since we are making an API call to AWS, this should be pretty straightforward.

Updating our service to use Amazon SageMaker endpoints

To update our service to use our model deployed in AWS, we only need to replace the llm function we built earlier. In the chapter materials, we have included a 07-api-deepseek-sagemaker.py API, which can be run with the following:


$ uv run fastapi run 07-api-deepseek-sagemaker.py

Our API’s backend is now Amazon SageMaker, but everything else remains the same: the request format, the commands, and the user experience. This demonstrates the advantage of structured outputs. You can switch the backend to another service without disrupting users’ workflows or breaking the API. In fact, we could also have an API that serves three different endpoints with three different backends – DeepSeek API, CPU inference, and Amazon SageMaker – at the exact same time! We’ll leave that as an exercise for you.

Best practices and recommendations

In this chapter, while building our Daily Health Summary service, we explored three main ways of using DeepSeek models: calling the API, running distilled models locally on a CPU, and deploying larger models with AWS, LMI containers, and vLLM. These are just the tip of the iceberg. The beauty of open source is that you have even more options if you need them.

That said, deploying DeepSeek models yourself is rarely needed. Self-hosting gives you control and privacy but comes with extra work and extra costs. You’ll have to manage infrastructure, monitor performance, and ensure reliability. For most cases, especially when testing an idea or validating a product, starting with the API is almost always best. Keep it simple until you know you need more.

When deciding whether to host models yourself, use this quick checklist:

Cost: Compare API usage costs with self-deployment. How do they compare?
Latency: How fast do responses need to be? Do your users tolerate that?
Scaling: Can your setup handle peak usage? What if your users explode overnight? Can you serve them?
Robustness: How robust does your setup need to be? Do you have a team responsible for infrastructure?
Flexibility: Can you easily evolve your application from API calling to self-hosting and vice versa?

We’ve shown how to start with an API, move to CPU inference, and eventually deploy larger models. Since our API format stayed consistent, users would not notice the change. This is a proven way to evolve without breaking things.

Summary

In this chapter, we took you from an initial idea to a fully working service that replaces boring smartwatch summaries with dynamic insights powered by DeepSeek. We began by showing how to quickly build a simple prototype using the DeepSeek API to meet our initial requirements. We then explored running models locally with distilled versions and built a lightweight CPU-based service that runs without GPUs.

Next, we focused on flexibility, demonstrating how libraries such as LiteLLM let you switch backend providers seamlessly without touching your application code. Finally, we scaled up by deploying LLMs on AWS using LMI containers and vLLM, so we can deploy larger models while staying in control.

By now, you have the toolkit to take any DeepSeek-powered application to production, whether you want to keep things small and efficient, run via an API, or deploy larger models in the cloud.

But many successful applications today are powered not by a single LLM, but by multiple models working together, often described as agents. That’s exactly what we’ll explore in the next chapter.

Get This Book’s PDF Version and Exclusive Extras

6 Agents with DeepSeek

In Chapter 4, we discussed how to build simple applications using DeepSeek. In this chapter, we will discuss how to build LLM-powered agents with DeepSeek. At the time of writing, agents are a very popular topic. Many companies and engineers are rushing to build so-called agentic applications. A large number of agentic products have come out in recent years. Some popular examples are OpenAI’s deep research (https://openai.com/index/introducing-deep-research/) and Google’s Gemini Deep Research products (https://gemini.google/overview/deep-research), where LLMs will dynamically search the web for you and come up with a report that visits multiple websites and resources. If you work with software, you might have used some agentic systems: Claude Code (https://docs.anthropic.com/en/docs/claude-code/overview) and Cursor (https://www.cursor.com/) are good examples of agent-powered products.

Besides this, the libraries and frameworks used to build agents keep multiplying. Libraries such as CrewAI (https://www.crewai.com/), OpenAI’s swarm (https://github.com/openai/swarm), and LangChain’s LangGraph (https://langchain-ai.github.io/langgraph/) are evolving every day. The libraries and concepts around agents are incredibly new and constantly evolving at the time of writing. With the idea of making this book stand the test of time, we will focus on the key principles and definitions surrounding agents that likely won’t change. We will also show you how to put those building blocks in place while leveraging DeepSeek models specifically.

In this chapter, we will cover the following main topics:

A gentle introduction to agents
Tools
Understanding the Model Context Protocol
Working with agents and workflows
Exploring various agentic systems

Technical requirements

The following chapter is made up of three different Jupyter notebooks (https://docs.jupyter.org/en/latest/). The only software you will need to have installed is uv. To install uv, follow the instructions here: https://docs.astral.sh/uv/getting-started/installation/.

A gentle introduction to agents

Before we dive deeper into building agents with DeepSeek, we should try and define what an agent is. There have been many attempts to define what an agent is. OpenAI says that agents are “systems that independently accomplish tasks on your behalf”.

At its most fundamental level, an LLM-based agent is an autonomous system that can take a general task and operate independently to accomplish it. A task can be anything: create a report about a topic, create a pull request that solves this GitHub issue, or play chess against this opponent. It’s important to note that agents don’t operate in a vacuum. Agents are built to interact with an environment. That environment could be a chessboard, the internet (if an agent must search it), or a GitHub repo, in a case where an agent is going to submit a patch to a code base.

But agents need a way to interact and perform actions within an environment. For example, a chess-playing agent needs to be able to read/understand the chessboard, a research agent needs to be able to search the internet, and a Pokémon-playing agent (https://www.theverge.com/news/619482/anthropics-claude-ai-is-playing-pokemon) needs to be able to interact with the game.

To interact with an environment, agents use what we call tools. Tools allow agents to accomplish actions in the environment. For example, a coding agent needs to be able to read a code base and submit patches to it. In this case, you can imagine the agent will have access to tools like read_code_base and submit_patch tool. DeepSeek, for example, is able to call tools through its function-calling capability (https://api-docs.deepseek.com/guides/function_calling).

In summary, there are three main components that make up an agent:

Task: The goal that the agent wishes to accomplish for the user.
Environment: The space where the agent operates, in order to accomplish the task.
Tools: The set of actions that the agent can take within the environment in order to accomplish the goal.

For example, let’s say we are going to build an agent that can submit improvements to a code base. The task might be Solve issue #45634 in my GitHub repository, the environment is the GitHub repository with the code, and the tools could be read_file, search_repository, and submit_patch. Another example is a customer support agent. The task could be to support customers with whatever their problems are, the environment could be the conversation with the user (likely a chat interface), and the tools might be search_knowledge_base, escalate_ticket, or even mark_issue_as_resolved.

One important characteristic of agents versus normal LLM-based applications is that agents are made to act independently. This means that agents can asynchronously go and interact with the environment without requiring explicit interaction with the user who assigned a task. However, we should note that, agents can be especially powerful when paired up with humans. For example, one of the tools available to an agent could be the capability to ask for guidance from a human expert.

Other characteristics you will certainly have heard of are memory and planning. Memory can relate to two things.

The first is short-term memory, which is normally related to the history or the conversation you are having with an agent directly (or the sequence of messages). For example, you might tell an agent Always respond in Portuguese, and the agent will/should remember that for the rest of the conversation.
The second type of memory is long-term memory. Long-term memory might relate to concepts we want the agent to keep track of between tasks. We could build something very complicated for this last one. But we can also simply create two tools: read_memory and update_memory. This would allow the agent to retrieve past memories and update new ones. There’s no need to complicate things.

As for planning, you can think of it as similar to chain-of-thought: asking the agent to outline a plan before solving a problem. It’s an interesting concept, but it’s not the main focus here.

Both memory and planning – although interesting – are not critical when building agents. Instead, let’s move on to one of the most important aspects of agents: tools.

Tools

Tools are one of the most critical parts of building agents. Tool calling is what allows agents to interact and take actions in their environment. However, it’s important to note that we didn’t always have tool calling. In the early stages of LLMs (which is not that long ago), we used simple prompts to allow the LLMs to interact with the outside world.

Here’s an example:


Question: What is the capital of France?

Thought: I should look this up on Wikipedia.

Action: wikipedia: France

PAUSE

Observation: France is a country. The capital is Paris.

Answer: The capital of France is Paris.

In the preceding example, the LLM outputs an action (e.g., wikipedia: France). The system (us) sees that, runs the search, and feeds the results back to the LLM. It was crude, but it worked. This was the core mechanism behind popular papers such as the ReAct paper from Google (https://react-lm.github.io/).

As things evolved, this approach seemed brittle, and LLM models started having the native capability to call functions (or tools).

We will use the terms function and tool interchangeably in this chapter.

In the next generation of LLMs, we could give a more detailed description of the tools the model has access to and what arguments that specific tool can take in. The models can then respond directly with a tool call. This is very similar to the crude approach before, but since models were specifically trained for it (with techniques such as reinforcement learning), it’s more robust and reduces the likelihood of errors.

For example, we can specify a tool such as get_weather by providing a JSON schema to the LLM:


tools = [

    {

        "type": "function",

        "function": {

            "name": "get_weather",

            "description": "Get the weather for a given location",

            "parameters": {

                "type": "object",

                "properties": {

                    "location": {"type": "string"}

                },

                "required": ["location"]

            },

        }

    },

]

messages = [{"role": "user", "content": "How's the weather in Hangzhou?"}]

response = client.chat.completions.create(

    model="deepseek-chat",

    messages=messages,

    tools=tools,

)

The model responds with the function it wants to call and the arguments it intends to pass to that function it wants to call and the arguments it intends to pass to that function.


{

  "tool_calls": [

    {

      "function": "get_weather",

      "arguments": {

        "location": "Hangzhou"

      }

    }

  ]

}

It’s important to note that the model does not execute tools – you, as an engineer, do. The model just tells us what it would like us to execute. You can see how this approach is slightly less brittle. We can define complex tools by leveraging JSON schema.

Tool calling is very popular nowadays. There are models explicitly trained for tool calling, such as Salesforce’s xLAM family of models, which are trained specifically for tool use. The Berkeley Function-Calling Leaderboard (https://gorilla.cs.berkeley.edu/leaderboard.html) ranks the ability of different models to call different tools. DeepSeek V3 and DeepSeek-R1 rank in the twenty-second and twenty-seventh positions, respectively, at the time of writing, not making them the best models for function calling, but making them pretty good contestants for building with tools.

Before we move on to building your agents and workflows, let’s talk about something that has been getting more and more popular: the Model Context Protocol.

Understanding the Model Context Protocol

In November 2024, Anthropic released the Model Context Protocol (MCP). While many were starting to build applications with LLMs. LLMs were still operating in somewhat of a vacuum. Many companies were building their own tools to plug into LLMs, often leading to re-implementing the same logic again and again.

For example, let’s say you are a company building an integration of an LLM-based application with Gmail. This would lead to many companies creating their own versions of the read_inbox and compose_email functions. If we must develop a new tool every time we wish to connect our LLMs to new systems, this makes things very difficult to scale. MCP addresses this challenge.

MCP is an open standard that looks to simplify this process. The protocol aims to establish a pattern by which LLMs can connect to the outside world. It defines MCP clients and MCP servers.

Clients are applications that consume resources in an MCP server. An MCP server exposes three types of resources: (1) tools, (2) prompts, and (3) resources – each illustrated in the following diagram.

Figure 6.1: Illustration of MCP server (inspired by Anthropichttps://www.youtube.com/watch?v=kQmXtrmQ5Zg)

What does this mean in practice? It means that if you wish to connect your LLM to Google Drive, you could leverage a preexisting Google Drive MCP that someone else built (possibly Google), instead of having to build your own.

If it all sounded a bit abstract up till now, let’s make it more concrete.

Let’s create our own small MCP server. In the following example, we use the FastMCP (https://gofastmcp.com) framework to create an MCP server. This server serves two things: (1) a tool to get the weather from a city, and (2) a resource that returns the status of a given user:


from fastmcp import FastMCP

mcp = FastMCP(name="MyAssistantServer")

@mcp.tool

def get_weather(city: str) -> str:

    """Multiplies two numbers."""

   return f"The weather in {city} is sunny with a high of 25°C."

@mcp.resource("users://{user_id}/profile")

def get_profile(user_id: int):

    return {"name": f"User {user_id}", "status": "active"}

if __name__ == "__main__":

    mcp.run()

With our small MCP server defined, we can save it as mcp_server.py. How can we plug these MCP tools into our DeepSeek LLM? We can adapt an example from the documentation of DeepSeek. In the official function calling documentation (https://api-docs.deepseek.com/guides/function_calling), in order to give the model access to tools, we need to first define its JSON schema, as follows:


tools = [

    {

        "type": "function",

        "function": {

            "name": "get_weather",

            "description": "Get weather of an location, the user should supply a location first",

            "parameters": {

                "type": "object",

                "properties": {

                    "location": {

                        "type": "string",

                        "description": "The city and state, e.g. San Francisco, CA",

                    }

                },

                "required": ["location"],

            },

        },

    },

]

Alternatively, we can fetch these tools using MCP. Let me walk through how to go about it. First, we define a server configuration:


# Alternatively, we can use the MCP server to fetch tools

mcp_server_config = {

    "mcpServers": {

        "assistant": {

            "command": "uv", "args": ["run", "./mcp_server.py"]}}

}

Next, we create a function that uses the FastMCP Client to connect to our MCP server and retrieve the list of tools:


async def fetch_async_tools() -> list:

    async with Client(mcp_server_config) as mcp_client:

        return await mcp_client.list_tools()

Finally, we can wrap all of this logic in a get_tools function, where we also adapt our dictionary so that it conforms to what DeepSeek is expecting.


def get_tools() -> list:

    """Fetch and format the list of tools from the MCP server."""

    mcp_tools = asyncio.run(fetch_async_tools())

    return [

        {

            "type": "function",

            "function": {

                "name": tool.name,

                "description": tool.description,

                "parameters": tool.inputSchema,

            },

        }

        for tool in mcp_tools

    ]

As you can see, giving access to tools to an LLM can be done by defining those tools yourself or by connecting to an existing MCP server. For applications that serve a clear purpose, I would argue that you want to define the tools yourself. Often, if we are using too many tools, the models tend to get confused as to what to use and why. Defining the tools yourself gives you much more control over what you’d like the model to accomplish.

Many agentic frameworks, such as the OpenAI Agents SDK, LangGraph, and Pydantic AI, support connecting your LLMs to an MCP server and spare you from writing the boilerplate I’ve just demonstrated. However, in this chapter, we prefer to give you fine-grained control and understanding of the different tools (or mechanisms) under the hood; therefore, we won’t use those frameworks.

MCP has increased in popularity tremendously and even sparked other protocols, such as Google’s Agent2Agent (A2A) protocol (https://github.com/a2aproject/A2A). The real popularity of MCP has mainly been in consumer-facing applications. For example, you can connect your Claude desktop application to your Gmail MCP or connect your VSCode IDE to your company’s MCP to read documentation.

At the time of writing, MCP and A2A are very new protocols. Only the test of time will tell whether they become real standards or get replaced with something else. We do think LLMs and tool calling will stay around for a long time. Therefore, in an effort to explain to you how these mechanisms work at the lowest level, we’ll now dive deeper into agents and workflows.

Working with agents and workflows

Before we start building agents, I have some sad news for you, dear reader. Most agents don’t work quite yet. There are two reasons why.

The first is compounding errors. Imagine we have a challenging task for an agent to solve. That task will require careful orchestration of a series of steps. Let’s say it takes five careful steps to solve a task. If every step has a 90% chance of succeeding, that means the probability of completing all of them successfully drops to a meager 60%. Imagine you had a function that worked 60% of the time. Would you put it into production?

The second reason why agents – and, more broadly, LLMs – often fail is the context-rot problem. LLMs have a limited context window. Every tool and instruction we add to them fills up their context. Research has shown that LLM performance drops the more we fill up the context (https://research.trychroma.com/context-rot). This means, the more tools and instructions we give agents, the worse they eventually perform. These two factors combined make agents brittle. The solution is a more controlled approach.

In December 2024, Anthropic released a very interesting blog post called Building effective agents (https://www.anthropic.com/engineering/building-effective-agents). In it, they make an important distinction between agents and workflows. With so many different definitions of agents evolving every day, Anthropic’s distinction has worked well in our experience when thinking about building agent-backed applications. The distinction between agents and workflows and the structure of the workflows directly inspired how we approached the concepts in this chapter.

An agent (inspired by the original ReAct agent) has a goal, an environment, and a set of tools. In the following diagram, you can see an example of an augmented LLM, which is an LLM with tools. A tool can be the ability to search the web, to write a certain memory, and so on. It also has an environment it interacts with.

Figure 6.2: The augmented LLM with tools

In this scenario, we let the LLM automatically select the correct tools to reach its goal. Without guiding it. This means that the control flow is dynamic. In other words, we expect the agent to make the right decisions to reach its objective. This can prove to be brittle. The agent might get stuck in a loop, it might never reach its objective, and so on.

An alternative to an agent is a workflow. In a workflow, the control flow and direction of the overall system are predefined. Workflows are patterns where we control (to a certain extent) the actions and tools that the LLM will take ahead of time. You can think of a workflow as a pattern, or some sort of scaffolding around an augmented LLM, where we preemptively determine the actions, or flow, that we would like the LLM to take. This gives us back control of what actions we want the agent to take and can be a great choice when the set of actions is relatively simple.

Table 6.1 highlights some of the differences between agents and workflows.

Table 6.1: Differences between workflows and agents (adapted from Anthropic’s Building effective agents blog post)

It’s important to point out that both agents and workflows are agentic systems, both with varying degrees of autonomy. In a way, you can think of an agent as a pure self-driving car, without anyone inside, nothing to control it, and a workflow as a self-driving car with a driver (you) making sure it’s doing the right thing. We might use a lot of agents in the future, but when we can, we should choose control and go with workflows.

In this chapter, we will design three different agent systems powered by DeepSeek’s models. Two will be workflows, and one will be a pure agent. We will introduce each one in its specific sections. The goal of this section is for you to understand how to build agent systems using DeepSeek and be able to expand that to your own use case. We will not focus on particular libraries or tools. We believe that although some of these libraries are incredibly powerful, they will often overlook important aspects of agent systems – especially when learning.

Here are some interesting resources for you to read up on agents:

Let’s now build some example workflows and agents.

Exploring various agentic systems

In this section, we will talk about three different agent systems. We will start with an evaluator-optimizer workflow, where we will show an example by summarizing scientific papers. The second workflow is an orchestrator-worker workflow, where we will show how to generate detailed reports about a topic. Finally, we will introduce a pure tool calling agent, where you will see a dynamic workflow in action.

Workflow: Evaluator-optimizer

The first agentic workflow we will introduce is the evaluator-optimizer workflow. The idea of this workflow is to have a first LLM generate a response (i.e., the generator), and a second LLM evaluate the responses from the first (i.e., the evaluator). The evaluator LLM assesses the response, continuously providing feedback to the first, so that it can improve its generation, iteratively.

The key idea is to have a controlled loop, where we can iteratively refine the generation of a given target. This workflow is ideal when you have a general idea of where to start, and you also have a good idea of the criteria that make a good result. For example, you might want to generate a cover letter for a job application but would also like it to follow strict criteria (e.g., a certain number of paragraphs, a certain tone of voice). Another example is code generation. You can start by defining a function and establishing clear criteria (number of lines, time complexity, unit tests) that you would like the final result to respect.

Figure 6.3: The evaluator-optimizer workflow

Although it may sound simple, this workflow is particularly effective in the case of reasoning LLMs – such as deepseek-reasoner. This is because we not only provide feedback to the generator LLM, but also the reasoning behind that feedback. If this sounds abstract, don’t worry. We’ll build an example of this pattern next.

An example: Summarizing arXiv papers

arXiv.org (https://arxiv.org/) is a popular repository for publishing and uploading scientific papers. It has gained popularity in the field of machine learning. Especially nowadays, you’ll see hundreds of papers published every day. arXiv lets anyone publish and read papers that are meant for open access. It has gained increasing popularity and inspired similar concepts in adjacent fields, such as bioRxiv (https://www.biorxiv.org/) for biology.

In this example, we will build an agent that is able to produce summaries of scientific papers from arXiv, in the exact format that we need. You could, of course, download a paper and upload it to ChatGPT – but this system allows us to not only establish clear criteria for the type of summary we want, but also to illustrate this pattern in a simple way.

As a first step, we need to be able to download a given paper from arXiv and convert it into Markdown. To do this, we define a get_text_from_axiv_paper function. We can leverage the markitdown library from Microsoft to get this done:


from markitdown import MarkItDown

def get_text_from_arxiv_paper(url: str) -> str:

    md = MarkItDown(enable_plugins=True)

    result = md.convert(url)

    return result.text_content

print(get_text_from_arxiv_paper(url="https://arxiv.org/pdf/1706.03762"))

# prints Attention Is All You Need...

We now need a function to call our DeepSeek LLM. The llm_call function sends a prompt to the DeepSeek API and retrieves both the final answer and the reasoning from deepseek-reasoner. When with_json_output is set to True, it prepends a system instruction and sets the response format to ensure the model replies with a JSON object. The function then returns the parsed response as a Python dictionary along with the reasoning. If with_json_output is False, it simply returns the raw string response and reasoning.


import json

from openai import OpenAI

API_KEY = os.environ["DEEPSEEK_API_KEY"]

BASE_URL = "https://api.deepseek.com"

MODEL = "deepseek-reasoner"

def llm_call(

    prompt: str, with_json_output: bool = False

) -> tuple[str | dict, str]:

    client = OpenAI(api_key=API_KEY, base_url=BASE_URL)

    args = {

        "model": MODEL,

        "messages": [],

    }

    if with_json_output is True:

        json_prompt = """

        Output your response in JSON format with the keys specified in the prompt.

        Do not include any other text such as ```json or ```.
        The response should be directly parseable by json.loads.
        """.strip()
        args["messages"].append(
            {"role": "system", "content": json_prompt}
        )
        args["response_format"] = {"type": "json_object"}
    args["messages"].append({"role": "user", "content": prompt})
    response = client.chat.completions.create(**args)
    reasoning = response.choices[0].message.reasoning_content
    final_response = response.choices[0].message.content
    if with_json_output is True:
        return json.loads(final_response), reasoning
    return final_response, reasoning
final_response, reasoning = llm_call(
    prompt="What is the capital of Portugal?", with_json_output=True
)
print(f"Final Response: {final_response}")
# prints Final Response: {'capital': 'Lisbon'}
print(f"Reasoning: {reasoning}")
# prints Reasoning: 'First, the user asked ...'

Now we can start working on our evaluator-optimizer pattern. The generate function sends a target task to the LLM along with an optional context and a base prompt. It then calls llm_call to get the model’s response and reasoning. It also expects the response to have a <RESPONSE>...</RESPONSE> block, so it extracts the content inside of it. For debugging purposes, we also add some verbose printing so we can see what is going on.

def generate(
    prompt: str, task: str, context: str = ""
) -> tuple[str, str]:
    """Generate and improve a solution based on feedback."""
    full_prompt = (
        f"{prompt}\n{context}\nTask: {task}"
        if context
        else f"{prompt}\nTask: {task}"
    )
    response, thoughts = llm_call(full_prompt)
    result = re.search(r"<RESPONSE>(.*?)</RESPONSE>",
        response, re.DOTALL).group(1)
    print("\n=== GENERATION START ===")
    print("\n*** THOUGHTS START ***")
    print(thoughts)
    print("\n*** THOUGHTS END ***")
    print("\n*** RESULT START ***")
    print(result)
    print("\n*** RESULT END ***")
    print("=== GENERATION END ===\n")
    return thoughts, result

With our generator in place, we need our evaluator. The evaluator’s goal is to provide an evaluation – think How good was your try?, and feedback – think Pass, Fail. For this, we define an evaluate function. This builds a prompt and uses JSON outputs to retrieve both the evaluation and feedback. We also add some prints. These are optional, of course.

def evaluate(prompt: str, content: str, task: str) -> tuple[str, str]:
    full_prompt = f"{prompt}\nOriginal task: {task}\nContent to evaluate: {content}"
    response, thoughts = llm_call(full_prompt, with_json_output=True)
    evaluation = response.get("evaluation")
    feedback = response.get("feedback")
    print("=== EVALUATION START ===")
    print("\n*** THOUGHTS START ***")
    print(thoughts)
    print("\n*** THOUGHTS END ***")
    print("\n*** STATUS START ***")
    print(f"Status: {evaluation}")
    print("\n*** STATUS END ***")
    print("\n*** FEEDBACK START ***")
    print(feedback)
    print("\n*** FEEDBACK END ***")
    print("=== EVALUATION END ===\n")
    return evaluation, feedback

Now that we have both our generate and evaluate functions in place, we can create our evaluation-optimization loop. The loop function continuously generates solutions and evaluates them until a passing result is found. It takes in task, evaluator_prompt, and generator_prompt. It starts by calling the generate function and storing the initial thoughts and result in a chain_of_thought list. The goal of the chain_of_thought list is to only keep track of the different thoughts that the generator went through. We create this so that we can inspect the different thoughts that have occurred during the process.

Then it enters an infinite loop, beginning each iteration by passing the latest result to the evaluator. If the evaluation returns PASS, this means that our generator deemed that the paper summary respected all the criteria we defined (don’t worry – we will define them in a bit), and so the loop ends.

If the result does not pass (which means our evaluator was not satisfied), the function builds a new context that includes all previous attempts and the most recent evaluator feedback. This context is passed back into the generator, prompting it to refine the output. The result is a feedback loop that incrementally improves each generation. In theory, our infinite loop is limited by the total context of DeepSeek. But in production, especially, you should make sure to limit this loop! Never let an LLM run indefinitely! This will break your bank!

def loop(
    task: str, evaluator_prompt: str, generator_prompt: str
) -> tuple[str, list[dict]]:
    """Keep generating and evaluating until requirements are met."""
    memory = [] # keeps track of previous results
    chain_of_thought = [] # keeps track the chain of thought
    thoughts, result = generate(generator_prompt, task)
    memory.append(result)
    chain_of_thought.append({"thoughts": thoughts, "result": result})
  # enter infinite loop
    while True:
        evaluation, feedback = evaluate(evaluator_prompt, result, task)
    # everything is approved and done
        if evaluation == "PASS":
            return result, chain_of_thought
    # add to the context from the memory
        context = "\n".join(
            [
                "Previous attempts:",
                *[f"- {m}" for m in memory],
                f"\nFeedback: {feedback}",
            ]
        )
    # generate again and append to memory
        thoughts, result = generate(generator_prompt, task, context)
        memory.append(result)
        chain_of_thought.append({"thoughts": thoughts, "result": result})

With everything in place, we can now create our paper summarization workflow. The first step is to create our prompts. We create two prompts: one for the evaluator, outlining what makes a good summary, and one for the generator, which tells it that it needs to complete a task and output its response in the <RESPONSE> block:

evaluator_prompt = """
Evaluate the following summary. A good summary should:
1\. Be understandable by an undergraduate student
2\. Formatted in markdown, with proper headings and subheadings
3\. Have a title and a clear structure
4\. Have at least 500 words
5\. Grammar and spelling should be correct
You should be evaluating only and not attempting to solve the task.
Only output "PASS" if all criteria are met and you have no further suggestions for improvements.
Output your evaluation concisely in the following format:
EXAMPLE JSON OUTPUT:
{
    "evaluation": "PASS, NEEDS_IMPROVEMENT, or FAIL",
    "feedback": "What needs improvement and why."
}
"""
generator_prompt = """
Your goal is to complete the task based on <user input>. If there are feedback
from your previous generations, you should reflect on them to improve your solution
Output your answer concisely in the following format:
<RESPONSE>
Content of the response
</RESPONSE>
"""

With these prompts in place, our final step is to create a function that acts as the main entry point for our agentic loop. We’ll call this one paper_summary_for. It fetches the text from the paper, defines a task, which in our case is a summary, and then initiates the loop function. Once done, it returns our summary as well as the chain of thought.

def paper_summary_for(paper_url: str) -> str:
    web_page_text = get_text_from_arxiv_paper(paper_url)
    task = f"""
    <user input>
    Write a summary of the following article:
    <article>
    {web_page_text}
    </article>
    </user input>
    """
    result, cot = loop(task, evaluator_prompt, generator_prompt)
    return result, cot

As an example, we will ask our agent to summarize a recent paper called MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention (https://arxiv.org/pdf/2506.13585). When we run our main function, we can see that the process kicks off and goes for about five rounds. We can see that the evaluation fails multiple times – if the summary is too short, uses technical jargon, or does not format Markdown properly.

result, cot = generate_agentic_summary_for(
    "https://arxiv.org/pdf/2506.13585")
# prints
# === GENERATION START ===
# *** THOUGHTS START ***
# Summarizing the MiniMax-M1 article…
# *** THOUGHTS END ***
# *** RESULT START ***
# MiniMax-M1 is an open-weight reasoning model using a hybrid MoE architecture…
# *** RESULT END ***
# === GENERATION END ===
#
# === EVALUATION START ===
# *** STATUS START ***
# Status: FAIL
# *** STATUS END ***
# *** FEEDBACK START ***
# Too short (189 words) and missing a markdown title.
# *** FEEDBACK END ***
# === EVALUATION END ===
#
# ... Loops hidden for conciseness.
#
# === GENERATION START ===
# *** RESULT START ***
# **MiniMax-M1: Efficient Long-Context Reasoning**
# MiniMax-M1 is a large-scale model supporting 1M-token context and trained with the CISPO RL algorithm…
# *** RESULT END ***
# === GENERATION END ===
#
# === EVALUATION START ===
# *** STATUS START ***
# Status: PASS
# *** STATUS END ***
# === EVALUATION END ===

Eventually, at the end of the process, the evaluator gives a PASS score to the process, and the loop finishes. Our summary is nicely formatted, includes all the sections, and is understandable by a graduate student (at least the judgments of our evaluator!). Here’s the introduction section.

**MiniMax-M1: Revolutionizing Efficient Long-Context Reasoning in Large Language Models**
### Introduction
MiniMax-M1 represents a groundbreaking advancement in open-weight large language models (LLMs), engineered to tackle complex real-world problems requiring extensive reasoning and long-context processing. Developed by MiniMax AI, it pioneers efficient scaling of "test-time compute" (computational resources used during task execution) through a hybrid architecture and novel training techniques. The model supports context windows of **1 million tokens**—8 times larger than predecessors like DeepSeek-R1—and generates outputs up to **80K tokens**, enabling unprecedented capabilities in domains like software engineering, scientific research, and multi-step agent workflows.

You can visit the full notebook with all the code in the GitHub repository for this book and chapter.

As you can see, evaluator-optimizer is a powerful pattern where tasks can be incrementally improved toward an objective. It’s worth mentioning that we could be stuck in an infinite loop, so we advise you to limit the optimizer loop to a fixed set of iterations; otherwise, things might get expensive! Additionally, if your responses are time sensitive (i.e., you need generations fast), this pattern might not be the most appropriate one, since depending on the complexity of the task, things might be pretty slow!

Now that we’ve talked about our first workflow, let’s discuss another important one, the orchestrator-workers workflow.

Workflow: Orchestrator-workers

It’s time to complicate things a little bit. In this second example, we will introduce the orchestrator-workers workflow. If the name of this workflow sounds scary, don’t be scared, dear reader. This workflow is much simpler than it sounds!

The idea behind the orchestrator-workers workflow is that you can separate a given task into multiple tasks that can be run in parallel. Firstly, we have an orchestrator component. The job of this LLM is to receive a certain task and divide it into N independent tasks. Each worker LLM takes one of these tasks and performs whatever it needs to perform. The final component here is a synthesizer LLM, whose job is to gather all the outputs of the worker LLMs and transform them into a single coherent output for the process.

Figure 6.4: The orchestrator-workers workflow pattern

You might have seen this workflow in the wild. In fact, it powers systems such as ChatGPT Deep Research, a system that conducts deep research for a user by searching the web. In it, after creating a search report, several LLMs then act as workers and independently search the web for relevant information about a sub-topic. To clarify, we might be using the same type of LLM (DeepSeek V3, for example), but we then spark different instances of that LLM, in parallel, to research different topics. Finally, an orchestrator LLM synthesizes all the information and returns a coherent final report to the user.

Of course, this is all speculation – the official Deep Research product is actually closed source, so all we can do is guess. But a recent talk from the creators (https://www.youtube.com/watch?v=eJOjdjO45Sc) actually points in the multiple LLM calls direction. Also, if a single LLM had to browse 70 websites one at a time, the user would quickly lose their patience!

Still, it is a particularly useful pattern when the task at hand can be decomposed into smaller pieces and parallelized.

An example: A report-generating workflow

We’ve talked about Deep Research as an interesting use case for the orchestrator-workers workflow. Let’s build something similar. The goal will be to build a report about a certain task. We will let the orchestrator LLM decide on which section makes sense to write about. Once it decides, we will then run all our workers in parallel. As soon as the last one is done, we will then feed all inputs to our final building block, the synthesizer LLM, which will generate the final report. If you’re feeling scared, don’t – we’ll build it together step by step.

We start by defining two data classes using Python’s @dataclass decorator. Task is the input for the worker. Each task has a section title and the rationale for writing it. We also define WorkerResult, which is the output of each worker. It contains the same elements, as well as a result string. We could use the same class, but let’s keep things separated for now. It’s good practice.

from dataclasses import dataclass
@dataclass
class Task:
    section_title: str
    rationale: str
@dataclass
class WorkerResult:
    section_title: str
    rationale: str
    result: str | None = None

We now start defining our OrchestratorWorker class. Nothing fancy. Just an __init__ method that takes in a prompt for the orchestrator, a prompt for each worker, and a prompt for the final synthesis. Those prompts will be made into templates using the jinja2 package.

from jinja2 import Template
class OrchestratorWorker:
    """Break down tasks and run them in parallel using worker LLMs."""
    def __init__(
        self,
        orchestrator_prompt: str,
        worker_prompt: str,
        synthesis_prompt: str,
        task: str,
    ):
        self.orchestrator_template = Template(orchestrator_prompt)
        self.worker_template = Template(worker_prompt)
        self.synthesis_template = Template(synthesis_prompt)
        self.task = task

We also define a run_single_worker method that will take a main task, as well as a task, render the worker prompt template, run it through the LLM, and output a worker result. Notice we use with_json_output set to True, as we expect the worker result to be a dictionary with a response key. More on this in a bit.

 def run_single_worker(
        self, main_task: str, task: Task
    ) -> WorkerResult:
        """Run a single worker and return the result."""
        worker_input = self.worker_template.render(
            original_task=main_task,
            section_title=task.section_title,
            rationale=task.rationale,
        )
        print(f"\n[WORKER INPUT] Section: {task.section_title}\n{worker_input}\n")
        worker_response = llm_call(worker_input, with_json_output=True)
        result = worker_response["response"]
        print(f"\n[WORKER OUTPUT] Section: {task.section_title}\n{result}\n")
        return WorkerResult(
            section_title=task.section_title,
            rationale=task.rationale,
            result=result,
        )

Here’s the core loop of the logic. We receive the orchestrator input and receive a certain number of tasks from it. Each task is parsed as a Task type and mapped to the run_single_worker function, in parallel. Since we are parallelizing an IO-bound task (an API call to our LLM), we use Python’s built-in ThreadPoolExecutor.

 def process(self, task: str) -> list[WorkerResult]:
        """Run orchestrator and execute all workers in parallel."""
        input_str = self.orchestrator_template.render(task=task)
        print(f"\n[ORCHESTRATOR INPUT]\n{input_str}\n")
        orchestrator_output = llm_call(input_str, with_json_output=True)
        print(f"[ORCHESTRATOR OUTPUT]\nAnalysis:\n{orchestrator_output['analysis']}")
        print(f"\nTasks:\n{orchestrator_output['tasks']}\n")
        worker_tasks = [Task(**t) for t in orchestrator_output["tasks"]]
        print(f"[INFO] Total tasks: {len(worker_tasks)}\n")
        with ThreadPoolExecutor() as executor:
            results = list(
                tqdm(
                    executor.map(
                        lambda t: self.run_single_worker(task, t),
                        worker_tasks
                    ),
                    total=len(worker_tasks),
                    desc="Running workers",
                )
            )
        return results

A crucial step to describe here is the synthesize method. It’s relatively simple, as its main task is to take a list of WorkerResult and use synthesis_template to render the final result to the user.

 def synthesize(self, worker_results: list[WorkerResult]) -> str:
        """Generate final report by synthesizing all worker results."""
        synthesis_input = self.synthesis_template.render(
            original_task=self.task,
            worker_results=worker_results,
        )
        print(f"\n[SYNTHESIS INPUT]\n{synthesis_input}\n")
        response = llm_call(synthesis_input, with_json_output=True)
        return response["response"]

Finally, we implement the run method, which will serve as the entry point for our class. It calls the self.process method and feeds the result into the synthesize method. At the end, it returns the result of the synthesis back to the user.

 def run(self) -> str:
        """Run the orchestrator and return the final report."""
        worker_results = self.process(self.task)
        synthesis_result = self.synthesize(worker_results)
        return synthesis_result

Now we’re ready to run our orchestrator-workers workflow. First, we define the prompt for our three components:

Orchestrator prompt: Receives the main task and breaks it down into 3–5 distinct tasks. Here is how you can define it.

ORCHESTRATOR_PROMPT = """
You're tasked with creating a report on a specific topic.
Break this down into 3-5 distinct sections that could each be written with a different focus.
Report topic: {{ task }}
Return your response in this JSON format:
{
  "analysis": "Summarize your understanding of the reporting goal and explain why breaking it into multiple sections with different focuses would be valuable. Highlight how each section emphasizes a different priority (e.g., technical depth, accessibility, narrative).",
  "tasks": [
    {
      "section_title": "Technical Overview",
      "rationale": "Explain why this section is important and what it will cover.",
    },
    {
      "section_title": "Practical Implications",
      "rationale": "Explain why this section is important and what it will cover.",
    }
  ]
}
"""

Worker prompt: Receives the main topic, an isolated task (a section title in this case), and a rationale for that task. Let’s define the worker prompt.

WORKER_PROMPT = """
You're generating a section of a report.
Topic: {{ original_task }}
Section title: {{ section_title }}
Rationale: {{ rationale }}
Write the content for this section using **Markdown formatting** (bold, italics, lists, code blocks, etc.) where appropriate, but **do not include a header/title** — just the body content.
Return your response in this JSON format:
{
  "response": "Your report content here, using Markdown formatting but without any section headers."
}
"""

Synthesis prompt: Receives the input from all the workers and synthesizes the response into a final one. Here is the synthesis prompt.

SYNTHESIS_PROMPT = """
You're synthesizing multiple report sections into one cohesive final report.
Original topic: {{ original_task }}
These are the completed sections (in order):
{% for result in worker_results %}
Section title: {{ result.section_title }}
---
{{ result.result }}
{% endfor %}
Your task is to merge the sections into a well-flowing report. Use **Markdown formatting** where helpful, but **do not repeat the section titles**.
Ensure:
- The tone and style feel consistent throughout.
- Transitions between sections are smooth.
- Any overlap is minimized.
- The final output reads like a single unified piece.
Return your response in this JSON format:
{
  "response": "Your final merged report here, in Markdown."
}
"""

Now that everything is defined, we can fire off our workflow with a question. I chose one that is a bit personal to me. But you can choose anything!

task = "Make an in-depth report that explains why biking has not been adopted in the City of Lisbon, and what can be done to encourage it."
orchestrator = OrchestratorWorker(
    orchestrator_prompt=ORCHESTRATOR_PROMPT,
    worker_prompt=WORKER_PROMPT,
    synthesis_prompt=SYNTHESIS_PROMPT,
    task=task,
)
final_report = orchestrator.run()

You can see the GitHub repo for the full output of the workflow. But I’ll outline a few interesting parts here.

First, the orchestrator divides this question into five different tasks, each with a rationale, and each one can be individually explored. We also provide the rationale so that the worker has more context on why the particular section is important.

[
    {'section_title': 'Current Infrastructure and Challenges',
    'rationale': '...'
    },
    {'section_title': 'Cultural and Social Attitudes Towards Biking',
    'rationale': '...'
    },
    {'section_title': 'Policy and Governance',
    'rationale': 'T...'
    },
    {'section_title': 'Successful Case Studies and Potential Solutions',
    'rationale': '...'
    },
    {'section_title': 'Community Engagement and Awareness Campaigns',
    'rationale': '...'
    }
]

I will not show all the input from all the workers. You can check the GitHub repo for that, but here’s an example of a worker in action:

[WORKER INPUT] Section: Current Infrastructure and Challenges
You're generating a section of a report...
#...
[WORKER OUTPUT] Section: Current Infrastructure and Challenges
The City of Lisbon faces several infrastructure and logistical challenges
that hinder the widespread adoption of biking as a mode of transportation.
Below is an in-depth analysis of these barriers:
...

As you can see, this worker independently researches and outputs a report section on this topic. Finally, the synthesizer takes all these outputs and condenses them into a single coherent report.

The City of Lisbon faces significant challenges in adopting biking as a mainstream mode of transportation, primarily due to infrastructure limitations, cultural attitudes, and policy gaps. However, by addressing these issues through targeted strategies, Lisbon can foster a more bike-friendly environment.
**Current Infrastructure and Challenges**
Lisbon's urban landscape, characterized by steep hills and a lack of dedicated bike lanes, poses considerable barriers to biking. The absence of secure parking and poor integration with public transport further discourages potential cyclists. To overcome these obstacles, expanding the bike lane network, introducing e-bike sharing programs, and improving parking facilities are essential steps.
**Cultural and Social Attitudes Towards Biking**
Cultural perceptions in Lisbon often view biking as a leisure activity rather than a practical transportation option, compounded by safety concerns and the status associated with car ownership. Shifting these attitudes requires community engagement programs, infrastructure improvements, and incentives to highlight biking's benefits.

This is an interesting pattern, where the task can be broken down into individual sections/tasks that can be run in parallel. You can take this example and apply it to other tasks – for example, grading different essays, analyzing resumes, conducting different web searches on a topic, or editing files in a code base. Almost anything can be broken down into smaller, independent tasks! Not everything, though! Some tasks need to be sequential. So, think before throwing this pattern at a problem.

Now that we’ve covered the orchestrator-workers pattern, we will talk about a final, but very important, building block of building agents (especially with DeepSeek), which is the tool-calling agent.

Agent: Tool-calling agent

We’ve built two agentic workflows until now. But we are yet to build a full agent. No workflows, no guardrails – just let the agent make all decisions. However, one of the most powerful things you can add to any LLM is tools. Recall that tools are the functions that your LLM can call. For example, you might have a search_web tool. When the LLM decides to call it, it will tell you that it wants to use the search_web tool with the deepseek argument if it wants to search for that term.

As we mentioned before, in this case, we will let the LLM make the decisions. DeepSeek will decide which tools to call and in what sequence to call them. We can also talk about this as an augmented LLM, which is an LLM powered with some tools. Modern versions of this pattern might include something such as memory for an agent, or more sophisticated techniques. Some other definitions of agent tend to be simplified: An agent is an LLM with tools in a loop. Although simplistic, this covers the core of our final agentic system. You can see the tool-calling agent illustrated here.

Figure 6.5: A tool-calling agent

When building a tool-calling agent, we should be mindful of a few things. Which tools does our agent have access to, and do they help in solving the overall task? Can the loop of the LLM choosing different tools go on forever? Can the agent reach out to a human for help? All of these are important when building a tool-calling agent. For example, giving access to a repo-reading tool when our agent has no business reading a code base will simply confuse it. Giving a customer support agent access to do web searches will probably result in it citing something wrong from the internet. We should be extremely careful and considerate when creating the tools for our agents to use. This is, as we say in Portugal, how the sausage gets made (i.e., the secret sauce that makes things work well!).

Enough with the explanation, let’s build a tool-calling agent powered by DeepSeek.

An example: A web search agent

In this example, we will create a web search agent that can receive a question, and go off into the internet and search for the best response possible to that question. We will equip this agent with tools to accomplish the task and implement a tool-calling loop from scratch. Since our agent needs to be able to search the internet, we will give it access to two tools: search_web (to conduct web searches) and read_website (to read any website on the internet). We could also add a third tool to ask us for support or clarification for a task. But we’ll leave that as an exercise for you.

We start with our usual LLM calling function, but we add two important aspects: a cache backed by diskcache to avoid redundant model calls. And for reasoning agents (such as deepseek-reasoner), we strip the reasoning content from the message chain (unlike other models, DeepSeek documentation conveys that we must remove the reasoning content from the messages chain).

from diskcache import Cache
cache = Cache("./cache")
@cache.memoize(expire=3600)
def llm_call(
    messages: list[dict[str, str]], tools: list[dict[str, Any]], model: str
) -> str:
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        tools=tools if tools else None,
        temperature=0.0,
        tool_choice="auto",
    )
    message = response.choices[0].message
    if hasattr(message, "reasoning_content"):
        del message.reasoning_content
    return message

Let’s now define our tools. The first is a search_web tool that can be used to search the web using the Marginalia search engine (https://www.marginalia.nu/). The main reason for choosing this one is that it’s free and has a pretty simple API. But in production, you might evaluate or use other search engines (Google, Brave, or others).

When we search for something, it will return a list of links, titles, and a couple of other details.

from pydantic import BaseModel
import httpx
class SearchWeb(BaseModel):
    """Search the web for a given query."""
    query: str
    max_results: int = 10
def search_web(data: SearchWeb) -> str:
    base_url = "https://api.marginalia.nu/{key}/search/{query}"
    url = base_url.format(key="public", query=data.query)
    url += f"?count={data.max_results}"
    rsp = httpx.get(url)
    rsp.raise_for_status()
    results = rsp.json()["results"]
    return str(results)
# print(search_web(SearchWeb(query="Paris")))
# [
# {'url': 'https://en.wikipedia.org/wiki/Paris_Las_Vegas', 'title': 'Paris ..', 'description': '...', 'quality': 5.037784569660175, 'format': 'html', 'details': [[]]},
# {'url': 'https://en.wikipedia.org/wiki/Paris_Observatory', 'title': 'Paris Observatory', 'description': 'The Paris ...', 'quality': 5.161455267002603, 'format': 'html', 'details': [[]]}
# ]

The second tool that we will equip our agent with is the read_website tool. It uses markdownify to convert the content of a website into Markdown, so that the LLM can better understand it.

from markdownify import markdownify as md
class ReadWebsite(BaseModel):
    """Read the content of a website and return it as text. (useful for further research)"""
    url: str
def read_website(data: ReadWebsite) -> str:
    html_content = httpx.get(data.url).text
    return md(html_content)
# print(read_website(data=ReadWebsite(url="https://www.theguardian.com/europe")))
# prints "The Latest news, sport and opinion from the Guardian.."

We also define a small helper function that converts plain functions into the required tool-call format from DeepSeek. This is required since DeepSeek’s API expects the JSON format for calling tools.

def to_openai_tool(func: callable) -> dict[str, Any]:
    sig = inspect.signature(func)
    if sig.parameters:
        param = next(iter(sig.parameters.values()))
        param_type = param.annotation
        schema = param_type.model_json_schema()
    else:
        param_type = None
        schema = {"type": "object", "properties": {}, "required": []}
    return {
        "type": "function",
        "function": {
            "name": func.__name__,
            "description": (param_type.__doc__ if param_type else func.__doc__ or ""),
            "parameters": schema,
        },
    }

Finally, we create our tools by calling to_openai_tool on both our functions. We do this since we are using the OpenAI SDK, which requires a particular tool format. We also create a FUNC_TYPES dictionary that maps each function to the correct Pydantic input format:

TOOLS = [
    to_openai_tool(search_web),
    to_openai_tool(read_website),
]
# {'type': 'function', 'function': {'name': 'search_web', 'description': ...
FUNC_TYPES = {
    search_web: SearchWeb,
    read_website: ReadWebsite,
}

Now that the tools and boilerplate are in place, let’s look at how the agent works. It’s a simple class. On init, it sets up the model, tools, message list, and a max iteration limit. The main logic is in __call__. It adds the user prompt to the message history, then enters a loop:

If the model responds directly, we return it.
If it wants to call a tool, we find the right function, run it, append the result, and continue.sw

The _send method just passes messages to the model and returns the response – nothing fancy. Overall, the agent keeps looping: asking the model what to do, calling tools if needed, and feeding results back – until it’s done or hits the iteration cap.

class Agent:
    def __init__(
        self,
        model: str,
        system: str,
        tools: list[dict[str, Any]] | None = None,
        max_iters: int = 10,
    ):
        self.model = model
        self.tools = tools or []
        self.messages: list[dict[str, str]] = []
        if system:
            self.messages.append({"role": "system", "content": system})
        self.max_iters = max_iters
    def __call__(self, content: str, verbose: bool) -> str:
        self.messages.append({"role": "user", "content": content})
        iterations = 0
        while True:
            iterations += 1
            message = self._send()
            self.messages.append(message)
            tool_calls = getattr(message, "tool_calls", None)
            if not tool_calls:
                return message.content
            for call in tool_calls:
                func = globals()[call.function.name]
                args_model = FUNC_TYPES[func]
                args = (
                    args_model.model_validate_json(call.function.arguments)
                    if call.function.arguments != "{}"
                    else None
                )
                print(f"** CALLING FUNCTION {func.__name__} **")
                print(f"*** ARGS ***\n{args}\n")
                result = func(args) if args else func()
                if verbose is True:
                    print(f"*** RESULT ***\n{result}\n***")
                self.messages.append(
                    {
                        "role": "tool",
                        "tool_call_id": call.id,
                        "content": result,
                    }
                )
            if iterations >= self.max_iters:
                raise Exception("Max iterations reached")
    def _send(self):
        return llm_call(self.messages, self.tools, self.model)

Now we define our system prompts. We tell the LLM that it should follow this thought-action observation loop, and we kick it off with a simple question about the population of the city of Ancona in 2025.

system_prompt = """
You are a helpful assistant who can answer multistep questions by sequentially calling functions.
Follow a pattern of:
- THOUGHT (reason step-by-step about which function to call next)
- ACTION (call a function as a next step towards the final answer)
- OBSERVATION (output of the function)
Reason step by step which actions to take to get to the answer.
Only call functions with arguments coming verbatim from the user or the output of other functions.
"""
question = "What is the population of the city of Ancona as of 2025?"

To call the agent, we attach the system prompt, tools, and start with the deepseek-chat model.

bot = Agent(system=system_prompt, tools=TOOLS, model="deepseek-chat")
response = bot(question, verbose=False)
print("Response:", response)

Every time the agent calls a function, we log it. We can see that in this first try, the agent called the search web function with the query population of Ancona Italy in 2025. Then it read the website https://population.city/italy/ancona/, where it finally encountered a number. It then took that number and extrapolated it from 2014 to 2025. Not sure we can count this as correct, but it’s certainly interesting to see the model call different tools for itself. In any case, we could specify in our prompt that it should only use estimates and not extrapolate. But we’ll leave this as an exercise for you, our dear reader.

** CALLING FUNCTION search_web **
*** ARGS ***
query='population of Ancona Italy 2025' max_results=10
** CALLING FUNCTION read_website **
*** ARGS ***
url='https://population.city/italy/ancona/'
Response: The last known population of Ancona, Italy, was approximately **101,500** in 2014\. If the population growth rate remained consistent at **0.34% per year** (as observed between 2011-2014), the estimated population of Ancona in 2025 would be **105,351**.
This is an unofficial projection based on historical data. For the most accurate and up-to-date figures, official census data or reports from the Italian National Institute of Statistics (ISTAT) would be required.

Now let’s try the deepseek-reasoner model for a potentially more intelligent loop.

bot = Agent(system=system_prompt, tools=TOOLS, model="deepseek-reasoner")
response = bot(question, verbose=False)
print("Response:", response)

We can see the reasoner-powered agent calls more functions and decides (by itself) to look for official estimates. After some digging, it eventually finds an estimate from the official institute of statistics of around 99K people. This sounds more reasonable, especially since the model also provided us with a reference!

** CALLING FUNCTION search_web **
*** ARGS ***
query='Ancona population 2025 projection' max_results=10
** CALLING FUNCTION search_web **
*** ARGS ***
query='Ancona Italy population 2025 official estimate' max_results=10
** CALLING FUNCTION read_website **
*** ARGS ***
url='https://www.citypopulation.de/en/italy/cities/marche/'
Response: Based on the official population estimates from the Istituto Nazionale di Statistica Italia (Italy's National Institute of Statistics), the population of Ancona as of January 1, 2025 is **99,469**.
This information comes directly from the detailed city and commune population tables on citypopulation.de, which sources its data from Italy's official statistics agency. The table shows:
- 2021 census: 98,402
- 2025 estimate: 99,469
The population estimate represents the projected population for the city proper of Ancona as of the beginning of 2025.

It’s worth pausing here for a note. Both runs used the same tool-calling agent, the same tools (search_web and read_website), and the same system prompt. The only difference was the model: deepseek-chat versus deepseek-reasoner. And yet, the outcomes were very different. DeepSeek-Chat found a number on the page and chose to extrapolate it forward to 2025, even though the page (or prompt) itself didn’t suggest doing so. DeepSeek-Reasoner made a more cautious choice: it reformulated the query, searched for an official estimate, and correctly returned 99,469, as reported by the Italian statistics office. This divergence highlights a critical feature of dynamic workflows: even when the structure and tooling are constant, the model’s internal reasoning governs the control flow. DeepSeek-Chat followed a shorter but flawed path. DeepSeek-Reasoner added steps and reached the right answer. This is the promise and the pitfall of dynamic agentic systems – they’re flexible and capable of adaptation, but they’re also inherently unpredictable, because the flow is driven by model behavior and not hardcoded logic.

The augmented LLM is a pretty powerful building block of LLM-backed applications. In this example, we used it as a single block to solve a specific problem. However, you can think of an augmented LLM as a Lego building block of any of the other workflows.

For example, in the evaluator-optimizer workflow, the evaluator can be equipped with tools (for example, to test code or for validating SQL), and the generator as well. In the orchestrator-workers workflow, each worker can be equipped with tools to research a certain topic or to get a certain task done. You are not limited by a certain workflow or a certain set of tools. Tools can have state. In modern IDEs, for example, such as VSCode, your chat probably has access to tools to search across your code base, edit a certain number of files, or read and write a set of rules you’d like it to remember.

An important note about evaluating agents

Evaluating agents – and LLM applications – is critical. Shipping a product without measuring success means you’ll struggle to improve it. While this book is not about evaluations, nor will we design an evaluation system for the agents we built, we feel compelled to point you in the right direction.

You’ll find plenty of frameworks promising 4-lines-of-code evaluations with metrics such as answer relevancy or faithfulness. Those numbers might look good – great, my relevance score is 0.87! – but in many cases, they don’t matter to your product’s success. If you’re building an agent to answer What’s the population of a small city in Italy?, the only success metric is whether it returns the correct number. Everything else is secondary.

From our experience, the most effective way to evaluate an LLM-based application is through a simple, iterative process:

Gather a dataset of 1,000 different cities and their population from a trusted source.
Run your agent on all the cities: What is the population of X?
Measure your accuracy: How many answers were correct? Which ones were wrong? Why?
Change the system based on learnings: switch search engines, change prompts, add planning steps.
Repeat from step 2.

By repeating this process multiple times, you’ll systematically improve your system. It’s more complicated than a 4-lines-of-code framework that gives you a shiny number, but trust us: you’ll learn far more in the process.

Summary

At the time of writing, agents are all the rage. Every company’s executive leadership and start-up leaders are raging about all the different problems agents can solve. Now, we don’t know whether agents will solve scientific mysteries or make scientific discoveries, and we certainly have doubts that they can do so in a short lifespan. However, knowing how to leverage DeepSeek to build agents is a superpower to have under your belt.

In this chapter, we covered the essential workflows and building blocks of agentic applications and patterns. As this technology moves rapidly, we decided to focus on the principles and build each one from scratch.

In the next chapter, we will continue exploring the ins and outs of DeepSeek and understand how we can fine-tune it for different applications.

Now that you’ve learned how to leverage DeepSeek to build agentic systems, we will move our focus to fine-tuning with DeepSeek in the next chapter.

Get This Book’s PDF Version and Exclusive Extras

Scan the QR code (or go to packtpub.com/unlock). Search for this book by name, confirm the edition, and then follow the steps on the page. | |

| Note: Keep your invoice handy. Purchases made directly from Packt don’t require one. |

Part 3 Distilling and Deploying DeepSeek

In this third part of the book, we move from understanding and using DeepSeek to transforming it for production environments. We begin by exploring rationale distillation - leveraging DeepSeek’s advanced reasoning capabilities to create specialized, efficient models that can run on modest hardware while maintaining professional-grade performance. Next, we address the practical challenges of deploying full-scale DeepSeek models, examining the trade-offs between APIs, managed services, and self-hosted solutions. Finally, we focus on building production-ready operations, covering monitoring, scaling, cost management, and security practices that turn experimental systems into reliable, enterprise-grade services.

By the end of this part, you’ll understand how to distill domain-specific models, make informed deployment decisions based on your requirements and constraints, and operate DeepSeek models at scale with confidence and reliability.

This part of the book includes the following chapters:

Chapter 7, DeepSeek-Driven Fine-Tuning of Gemma 3 for Legal Reasoning
Chapter 8, Deploying DeepSeek Models
Chapter 9, Epilogue
Appendix

Stay tuned

To keep up with the latest developments in the fields of Generative AI and LLMs, subscribe to our weekly newsletter, AI_Distilled, at https://packt.link/8Oz6Y.

Join our communities on Discord and Reddit

Have questions about the book or want to contribute to discussions on Generative AI and LLMs?

Join our Discord server at https://packt.link/4Bbd9 and our Reddit channel at https://packt.link/wcYOQ to connect, share, and collaborate with like-minded enthusiasts.

| | |

7 DeepSeek-Driven Fine-Tuning of Gemma 3 for Legal Reasoning

Imagine you’re tasked with building a high-frequency trading system. You wouldn’t reach for a general-purpose scripting language and a consumer-grade laptop – you’d reach for a low-latency, highly optimized stack, tuned for the precise demands of your domain. In engineering, we know that the right tool, properly configured, is the difference between it works and it works at scale, reliably, and with guarantees.

This is the core challenge with general-purpose Large Language Models (LLMs). Models like OpenAI’s GPT-5 or Claude 4 Sonnet are astonishingly capable generalists, but they’re the equivalent of a robust, all-in-one framework: flexible, but not optimized for the edge cases and performance requirements of specialized domains. When you’re parsing thousands of legal contracts for subtle risk factors, or automating compliance checks in a regulated industry, close enough is not sufficient – precision, reliability, and explainability are paramount. You also probably care about how much it costs to use these models.

In this chapter, we will perform a two-stage workflow comprising rationale distillation with DeepSeek-R1 and fine-tuning Gemma 3 in the rationale-augmented data.

In our terminology here, distillation means data generation by the teacher (no teacher weights are updated); that is, DeepSeek-R1 serves as a teacher model to generate the training data. Fine-tuning means updating the student’s parameters (via LoRA/Unsloth) to learn the teacher’s behavior. For our use case, we’ll fine-tune Gemma 3, a smaller model that is more suitable for deployment. We will also evaluate the student versus the teacher on held-out data. DeepSeek-R1 itself is not fine-tuned in this process; this reflects industry practice, where distillation is much more common.

In this chapter, we will cover the following main topics:

Understanding the importance of distillation and fine-tuning
Introducing CUAD: a structured benchmark for legal clause classification
LLMOps tools for model distillation
The two-stage workflow for legal rationale distillation
Evaluation and results
Key takeaways

Technical requirements

You will need the following for this chapter the project repository at https://github.com/PacktPublishing/DeepSeek-in-Practice

Dependencies

Start with the project repository and install the required packages:

git clone https://github.com/PacktPublishing/DeepSeek-in-Practice.git
cd Chapter07
pip install -r requirements.txt

The requirements file includes ZenML for orchestration, Unsloth for efficient training, transformers for model handling, and various utilities for data processing and evaluation.

Creating your local environment with ZenML

We use ZenML in code samples to demonstrate good MLOps hygiene (pipelines, caching, and lineage). You can substitute other platforms that provide the same capabilities without changing the learning objectives (for example, MLflow, Weights & Biases, Flyte, Metaflow, Kubeflow, or Airflow).

Creating your ZenML Cloud account

Create a free managed account at cloud.zenml.io. This serves as your experiment tracking and pipeline orchestration hub, monitoring training progress and managing artifacts. The free tier provides sufficient resources for our legal classification project.

Connect your local environment and set your project name after registration:

# Log in and select your workspace
zenml login
# Activate your project
zenml project set default # usually the default project is fine

Cloud stack with GPU: Model training requires GPU compute with memory requirements that scale by model size:
- 1B model: 8 GB+ GPU memory.
- 4B model: 16 GB+ GPU memory.
- 12B model: 24 GB+ GPU memory.

For this, you need to set up a cloud stack following the ZenML cloud deployment guide (https://docs.zenml.io/stacks/deployment/deploy-a-cloud-stack). A stack in ZenML represents the infrastructure where your pipelines run. You can easily switch from a local setup to cloud infrastructure just by switching your stack. The fastest way to create a cloud stack is through the Infrastructure-as-Code option. This uses Terraform to deploy cloud resources and register them as a ZenML stack. You’ll need the following:

Terraform (https://www.terraform.io/downloads.html) version 1.9+ installed locally.
Authentication configured for your preferred cloud provider (AWS, GCP, or Azure).
Appropriate permissions to create resources in your cloud account.

The deployment wizard will guide you through each step. The deployment will take a few minutes.

Once deployment is complete, activate your new stack:
```
zenml stack set <NAME_OF_YOUR_NEW_STACK>
```

API keys and environment variables

You will need API keys for accessing OpenRouter and Hugging Face models:

OpenRouter provides access to DeepSeek models for generating baseline comparisons during evaluation. Sign up at openrouter.ai and obtain your API key from the dashboard. We will use a free model to access the base DeepSeek-R1 model, but you need an API key to access it (via litellm). If you do not have an OpenRouter API key, you can create one at https://openrouter.ai/docs/api-reference/api-keys/get-key.
Hugging Face is used to upload the models you train. Create a token at huggingface.co/settings/tokens with write permissions. Create your API key by using https://huggingface.co/join.

Configure the required API credentials:

export OPENROUTER_API_KEY="your_openrouter_api_key"
export HF_API_KEY="your_huggingface_api_key"  # Optional

Enhanced CUAD dataset

Enhanced CUAD is a rationale-augmented version of the Contract Understanding Atticus Dataset (CUAD), adding DeepSeek-generated step-by-step explanations (rationales) to the original expert labels for 41 clause types across 510 real-world contracts. We’ll use the Hugging Face release zenml/cuad-deepseek (https://huggingface.co/datasets/zenml/cuad-deepseek).

Fine-tuning without ZenML (standalone script)

If you just want to fine-tune a model without installing ZenML, we provide a standalone Python script that reproduces the chapter’s training loop.

Optional standalone script to fine-tune without ZenML

The standalone script is placed here: https://github.com/PacktPublishing/DeepSeek-in-Practice/blob/main/Chapter07/standalone_training.py. The script follows this process:

Load the enhanced CUAD dataset (zenml/cuad-deepseek) from Hugging Face and ensure train/validation/test splits.

Enhanced CUAD is split into train/validation/test using an 80/10/10 stratified split with seed=42. We fine-tune the student on the train split, monitor and tune on the validation split, and report all metrics on the held-out test split only. No test examples – or their rationales – are used during training.

Format examples into Gemma 3 chat messages (user JSON prompt; assistant JSON with rationale and label).
Attach LoRA adapters to Gemma 3 (1B, 4B, or 12B) via Unsloth and train with TRL’s SFTTrainer on assistant responses only.
Save adapters and the tokenizer locally and optionally push to the Hugging Face Hub.
Run a quick post-train sanity evaluation on the test split.

To get started with the script, do the following:

Install dependencies (CUDA-ready PyTorch recommended):

pip install unsloth transformers trl datasets peft accelerate torch

Run a minimal training command (12B requires ~24 GB GPU RAM; try --model-size 4b if you have less):
```
python standalone_training.py --model-size 12b --filter-none-labels --eval-after-train
```
See all options with the following:
```
python standalone_training.py --help
```

Tip: To push the trained adapters to the Hugging Face Hub, add --push-to-hub and run this first so you’re logged in:

huggingface-cli login

Understanding the importance of distillation and fine-tuning

Fine-tuning and distillation are two core techniques for adapting LLMs to specialized domains. Each addresses a different aspect of the problem, and together they provide a practical path from general-purpose models to production-ready systems.

Before we get started, let’s do a quick round-up of fine-tuning and distillation.

Fine-tuning adapts an already pre-trained model to your domain by updating (some of) its parameters on targeted data. Think of it as taking a strong general system and tuning it to your workload – terminology, formats, and decision patterns – so it performs reliably in your context.

Distillation, on the other hand, compresses the behavior of a large teacher model into a smaller student. There are many flavors (logit, response, or rationale distillation).

Why do we need specialized models?

Using general-purpose models via external APIs can introduce hidden costs, especially at scale, when compared to specialized, self-hosted options. Some of these are listed as follows:

Token costs:

Even with lower prices and batch processing, handling a single 300-page merger and acquisition agreement (about 600,000 tokens each way) can become expensive with external APIs. Organizations processing many such documents each month may face high bills, while a distilled in-house model running on two A100 GPUs (about $3,000 monthly amortized cost) can process the same workload for much less.

Regulatory and compliance risks:

There are regulatory risks with external API use. For example, in December 2024, Italy’s privacy regulator fined OpenAI €15 million for GDPR violations (https://www.reuters.com/technology/italy-fines-openai-15-million-euros-over-privacy-rules-breach-2024-12-20/) related to ChatGPT data processing. This highlights the liability when sensitive data leaves an organization’s security perimeter. Some industries now require sovereign LLM deployments to avoid such penalties and maintain data residency.

Performance and availability:

Internal benchmarks from financial institutions (https://ctomagazine.com/jp-morgan-chase-accelerates-ai-adoption/) show that 7-billion parameter distilled models hosted on-premises return responses in 70-120 ms. In contrast, public APIs can have 600-1,200 ms P95 latency during peak load. This difference can affect service-level agreements for high-frequency review workflows and is important for real-time applications.

Flexibility and vendor lock-in:

External APIs cannot be customized for organization-specific terminology, formatting, or business rules. General-purpose models may lack the domain knowledge of fine-tuned alternatives. Relying on APIs can also create vendor lock-in, reducing strategic flexibility and increasing long-term risk.

Here are some of the important considerations that drive the use of domain-specific models:

Accuracy: Consider, for example, that a legal LLM fine-tuned on contract language can routinely achieve 95%+ accuracy on document classification, compared to about 70% for general models. This is not a marginal improvement. It’s the difference between a tool that occasionally assists and one that reliably automates core workflows.
Lower risk: Accuracy, however, is only one axis. General-purpose models are prone to hallucination, confidently producing incorrect outputs. In high-stakes domains such as healthcare or finance, this is unacceptable. Specialized models, trained on curated datasets, are better at recognizing the limits of their knowledge and are more likely to express uncertainty rather than fabricate plausible-sounding nonsense.
Efficiency is another key factor. Generalist LLMs are computationally expensive, sifting through vast, irrelevant knowledge for every query. Specialized models are leaner, smaller, faster, and focused. This translates to lower latency and reduced hardware costs.
Data privacy and control: Regulated industries cannot risk sending sensitive data to external APIs. On-premises, domain-specific models allow organizations to leverage LLM capabilities without compromising confidentiality or compliance.

In short, specialization yields higher accuracy, lower risk, improved efficiency, and greater control. These are outcomes that engineers value in any production system.

The impact of specialized models is evident in real deployments by major organizations. These cases show improvements in efficiency, accuracy, and cost savings:

Healthcare documentation:

Stanford Health Care implemented Nuance DAX Copilot, an AI system fine-tuned on clinician-patient dialogues, to automate clinical documentation (https://hitconsultant.net/2024/03/11/stanford-deploys-nuance-ai-powered-clinical-documentation/). The system captures conversations during patient visits and generates draft summaries, reducing administrative workload. Physicians in the pilot reported reduced documentation time (from 90 minutes to under 30 minutes each day), allowing more focus on patient care (https://www.dugganletter.com/p/from-burnout-to-breakthrough-how).

Financial services:

JPMorgan Chase’s COIN (Contract Intelligence) platform (https://ctomagazine.com/jp-morgan-chase-accelerates-ai-adoption/) uses LLM distillation for legal document processing. The system reviews commercial loan agreements that previously required about 360,000 human hours annually (https://medium.com/@ishan_dhodu/how-jpmorgan-chases-coin-is-revolutionizing-financial-operations-with-ai-120a2938dab7). Since its introduction, COIN has reduced review time and error rates.

Edge computing:

Google’s Gemini Nano is a distilled model for mobile devices (https://assets.publishing.service.gov.uk/media/661e5a4c7469198185bd3d62/AI_Foundation_Models_technical_update_report.pdf). This 1.6-billion-parameter model, derived from Gemini Ultra, runs on Pixel phones to provide smart-reply, summarization, and captioning in under 300 ms without network calls, while maintaining most of the parent model’s quality in less than 1 GB of RAM.

Apart from several advantages and applications in a myriad of use cases, the true value of fine-tuning and distillation lies in combining them.

How do distillation and fine-tuning help in creating specialized models?

When you combine distillation and fine-tuning, you can create models that are both highly specialized and efficient. Fine-tuning imparts domain expertise, while distillation ensures that expertise is accessible in real-world, resource-constrained settings. This approach allows you to move beyond generic solutions and deliver models that are tailored, performant, and ready for production.

Fine-tuning imparts domain expertise to the student; distillation ensures that expertise is portable, efficient, and deployable under real-world constraints (latency, privacy, cost).

We will use this approach in this chapter to create a specialized model for the legal domain and use it for a multi-label classification task.

In the next section, we will examine the use case and dataset in detail that we will use for distillation and fine-tuning with DeepSeek.

Use case and dataset

To learn how to create specialized models using fine-tuning and distillation, we will focus on a legal use case. But before we begin, we will outline the approach we will follow in this chapter for rationale distillation to power a multi-label classification problem on legal texts.

Conventions used in this chapter:

Teacher model: DeepSeek-R1 (API/inference only).
Student model: Gemma 3 (the model we actually fine-tune).
Distillation: Data-generation step.
Fine-tuning: Parameter-update step on the student.

Why Gemma (not DeepSeek)?

Fine-tuning DeepSeek directly is impractical and unnecessary for our goal; the industry-standard pattern is to distill a large, capable teacher into an efficient open student you can run locally.

The multi-label extraction problem in legal texts

Contract review is a prototypical example of a high-stakes, high-complexity information extraction problem. Legal professionals routinely spend around 50% of their time reviewing contracts, with billable rates at major firms ranging from $500 to $900 per hour. For a single Mergers and Acquisitions (M&A) transaction, this can translate to hundreds of thousands of dollars in legal fees. The technical challenge is not just the volume of text, but the need to accurately identify and interpret 41 distinct clause types – each with its own legal and business implications. This is a multi-label, multi-class classification problem, complicated by the fact that the relevant information is often buried in dense, heterogeneous, and sometimes ambiguous language.

The analogy of finding needles in a haystack is often used to describe contract review, but it doesn’t fully capture the complexity. In reality, the task is more like finding 41 different types of needles, each with unique features and varying degrees of risk. For example, consider the following Change of Control clause:

“In the event that Contractor undergoes a change in control where 50% or more of its voting securities are transferred to a new beneficial owner, Client may terminate this Agreement upon thirty days written notice.”

To a layperson, this might seem like standard legal boilerplate. To a legal expert, it encodes a specific mechanism that could determine whether a planned acquisition proceeds smoothly or faces unexpected contractual complications. Multiply this complexity across hundreds of pages and dozens of contract types, and the scale of the problem becomes clear. The challenge for AI is not just to extract these clauses, but to classify them correctly and, ideally, to provide a rationale for each decision.

The business case for automating contract review is compelling: an AI-assisted system could reduce legal review costs by 60-70%, improve accuracy, and provide complete audit trails. More importantly, it could prevent catastrophic scenarios where missed clauses lead to deal failures or unexpected liabilities.

From a workflow perspective, automating the identification and explanation of risky clauses allows junior staff to focus on higher-value analysis and gives senior lawyers confidence in the system’s outputs. The result is a more efficient, consistent, and defensible review process.

Up next, we will detail the technical approach: using DeepSeek-R1 to generate rationales, and then distilling this expertise into smaller, deployable models – demonstrating how to operationalize legal reasoning in production environments without sacrificing transparency or accuracy.

Introducing CUAD: A structured benchmark for legal clause classification

The Contract Understanding Atticus Dataset (CUAD) was created to address this challenge. Developed by The Atticus Project through a year-long collaboration involving law students, practicing lawyers, and machine learning researchers, CUAD contains over 13,000 expert annotations across 510 real-world commercial contracts. These contracts are sourced from SEC filings and represent the kinds of documents that legal professionals review in practice. You may explore the dataset here: https://www.atticusprojectai.org/cuad.

CUAD’s value lies in its systematic taxonomy: it defines 41 clause types that lawyers consistently flag during corporate transactions. Examples include anti-assignment clauses (which restrict how agreements can be transferred), liability caps (which limit exposure to damages), governing law provisions (which determine jurisdiction for disputes), and competitive restriction clauses (which might prevent post-acquisition business activities). Each category represents a specific type of legal risk or obligation that experienced attorneys have learned to identify through years of practice. For engineers, CUAD provides a well-defined, multi-class classification benchmark grounded in real-world legal practice.

Extending CUAD: Why we add rationales

CUAD ships with high-quality labels but no per-example reasons. To make models auditable and easier to fine-tune, we attach teacher-generated rationales and refer to this augmented corpus as enhanced CUAD (details appear later in the Data preprocessing and the enhanced CUAD dataset section.

For example, an anti-assignment clause might include a rationale such as the following:

This clause is classified as Anti-Assignment because it explicitly restricts the Distributor’s ability to assign the agreement. The key elements are: (1) it specifically prohibits assignment, (2) it applies to both the entire agreement and partial rights, (3) it requires explicit consent from the counterparty…

You can use enhanced CUAD directly in the fine-tuning steps in this chapter (https://huggingface.co/datasets/zenml/cuad-deepseek).

Our technical objective is to distil the legal reasoning capabilities of a large model (DeepSeek-R1) into a smaller, resource-efficient model suitable for on-premises deployment. Most law firms cannot use external APIs due to confidentiality and compliance constraints, so the final model must be compact enough to run locally, yet accurate and explainable enough for professional use.

Specifically, we are targeting a 12-billion parameter model that can classify all 41 CUAD categories (plus a NONE class for out-of-scope clauses) with >90% accuracy, and generate clear, stepwise rationales for each decision. This is a classic knowledge distillation scenario, with the added requirement of explainability. The goal is to create a system that fits on a laptop (or perhaps a small server) but reasons like a senior associate – combining the broad knowledge of an LLM with the specialized expertise that comes from processing thousands of real-world contracts.

Overview of the distillation fine-tuning process with CUAD and enhanced CUAD datasets

To solve the aforementioned multi-label classification, we will follow the two-stage workflow shown in Figure 7.1.

Figure 7.1: Two-stage workflow for legal rationale distillation

Stage 1 (distillation): We ask DeepSeek-R1 to produce step-by-step explanations for CUAD examples and attach those rationales to the data. The teacher is not updated; it only generates enriched training signals, resulting in enhanced CUAD (data). (Note that while we explain and show how to create the enhanced CUAD dataset in this chapter, we also provide a Hugging Face dataset with the full processed data so that you don’t have to spend $100+ just to reproduce the distillation part yourself.)
Stage 2 (fine-tuning): We will then train Gemma 3 with LoRA/Unsloth on enhanced CUAD so the student learns both the labels and the explanation structure.

The outcome is a smaller, faster model you can run locally – with accuracy and explanations aligned to legal review needs.

So, here is what we will do:

DeepSeek-R1 (teacher) → generate rationales over CUAD → enhanced CUAD (rationale-augmented) → fine-tune Gemma 3 (student) with Unsloth/LoRA → evaluate the student versus the teacher.

It’s time to get started! We will begin with the tools we will use for this task.

LLMOps tools for model distillation

Building production-ready AI systems depends on having the right infrastructure to develop, deploy, and maintain them reliably. In the context of LLM fine-tuning, this foundation is provided by Large Language Model Operations (LLMOps) tools. These tools help bring structure and repeatability to what could otherwise be a disorganized process of experimentation and deployment.

In our distillation pipeline (Figure 7.2), we use three main tools: ZenML for orchestration and observability, Unsloth for efficient training, and LiteLLM for unified model access. Each tool plays a distinct role in moving our legal contract classification project from a research experiment to a system ready for production.

Figure 7.2: Architecture of the distillation and fine-tuning process

For our legal use case, we need to track exactly which CUAD examples were enhanced, which model/revision trained on them, and how each run performed – so results can be reproduced and audited. We use pipelines to freeze the sequence (load → distil → format → fine-tune → evaluate → compare) and to cache expensive steps.

Here is an overview of each tool’s role in this chapter’s workflow:

ZenML (orchestration and lineage): Runs the end-to-end pipeline (load_data → finetune_model → evaluate_model → compare_models), caches expensive steps, and records inputs, outputs, and config for reproducibility (https://docs.zenml.io).
LiteLLM (teacher API access): Provides a single call interface to DeepSeek-R1 via OpenRouter during the distillation and teacher-baseline evaluation steps (model string and API key configured in environment) (https://github.com/BerriAI/litellm) (https://openrouter.ai).

For more detailed instructions on the setup using LiteLLM, you can check out the appendix towards the end of the book.

Unsloth (efficient LoRA fine-tuning): Attaches LoRA adapters and trains Gemma 3 with optimized kernels so we can fine-tune a 12B student on a single high-end GPU (https://unsloth.ai).

This section focuses on how these tools fit the workflow.

Let’s look at the code to understand where each tool appears in this chapter’s code:

Distillation and teacher model inference are performed via LiteLLM calls (used in Stage 1 training and teacher-baseline evaluation), following the standard LiteLLM completion call pattern.

# Tool: LiteLLM + OpenRouter
import os, json
from typing import Dict
from litellm import completion
DEEPSEEK_MODEL = os.getenv("DEEPSEEK_MODEL", "openrouter/deepseek-r1")
OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY")
def generate_rationale(clause: str, label: str) -> str:
    system = "You are a senior contract lawyer. Explain why the clause matches the CUAD label. Return concise, stepwise rationale."
    messages = [
        {"role": "system", "content": system},
        {"role": "user", "content": json.dumps({"clause": clause, "label": label})},
    ]
    resp = completion(model=DEEPSEEK_MODEL, messages=messages, max_tokens=400, api_key=OPENROUTER_API_KEY)
    return resp["choices"][0]["message"]["content"].strip()

Fine-tuning Gemma 3 with Unsloth LoRA (student training in stage 2):

# Tool: Unsloth
from typing import Any
from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig
def finetune_gemma(
    train_ds: Any, val_ds: Any,
    base_id: str = "unsloth/gemma-3-12b-it", out_dir: str = "ckpt"
) -> str:
    model, tokenizer = FastLanguageModel.from_pretrained(
        base_id, load_in_4bit=True)
    model = FastLanguageModel.get_peft_model(
        model, r=16, lora_alpha=32, lora_dropout=0,
        target_modules="all-linear",
        use_gradient_checkpointing="unsloth",
        random_state=3407, use_rslora=False,
    )
    trainer = SFTTrainer(
        model=model, tokenizer=tokenizer, train_dataset=train_ds,
        eval_dataset=val_ds,
        args=SFTConfig(
            per_device_train_batch_size=1,
            gradient_accumulation_steps=8,
            num_train_epochs=2, learning_rate=1e-4,
            lr_scheduler_type="cosine",
            warmup_steps=100, logging_steps=20),
    )
    trainer.train()
    model.save_pretrained(out_dir); tokenizer.save_pretrained(out_dir)
    return out_dir

ZenML wiring that orchestrates the steps:

# Tool: ZenML
from typing import Tuple
from zenml import step, pipeline
@step(enable_cache=True)
def load_data(
    max_samples: int | None = None, filter_none: bool = True
) -> Tuple[list, list]:
    # loads HF dataset, generates/attaches rationales if needed, formats to Gemma-3 chat
    # returns train_ds, val_ds (already tokenized/ready)
    ...
@step
def finetune_model(train_ds: list, val_ds: list, model_size: str = "12b") -> str:
    # calls finetune_gemma(...) above (Unsloth)
    ...
@step
def evaluate_student_vs_teacher(...) -> dict:
    # runs local student on test split; calls DeepSeek-R1 via LiteLLM for teacher baseline; computes metrics
    ...
@pipeline
def distill_finetuning(max_samples: int | None = None, model_size: str = "12b"):
    train_ds, val_ds = load_data(max_samples=max_samples, filter_none=True)
    ckpt_path = finetune_model(train_ds, val_ds, model_size=model_size)
    _ = evaluate_student_vs_teacher(...)

With our tooling foundation established, we can now dive into the practical implementation of our distillation pipeline. The first stage involves enhancing the CUAD dataset itself – using our teacher model to generate the explanatory rationales that will make our student model more interpretable.

The two-stage workflow for legal rationale distillation

Well, we know dear reader, you are itching to get started. Without much ado, let’s jump straight to distillation.

Stage 1: Distillation

Let’s begin with the process of creating the enhanced CUAD dataset. The goal is to preserve the original human labels while adding clear, auditable why explanations from a teacher model (DeepSeek-R1). You can find the full source code for this process in the Chapter07 folder in the code repository under the filename standalone_synthetic_generation.py.

Data splits at a glance

Split: 80% train/10% validation/10% test (stratified by label, seed=42).
Training: train split only (rationale-augmented).
Validation: validation split only.
Evaluation: test split only (held-out; no leakage).
NONE label: filtered by default for this chapter’s training runs.

The process is as follows:

Download CUAD (once, idempotent): We fetch and unpack the official CUAD zip (Zenodo) into a stable folder so paths stay predictable (https://zenodo.org/records/4595826/files/CUAD_v1.zip?download=1).

def download_and_extract_cuad(url, data_dir="data", force=False) -> str:
    # stream download (with progress) → data/CUAD_v1.zip
    # unzip to data/CUAD_v1/ so CUAD_v1.json is at a fixed path
    ...
    return os.path.join(data_dir, "CUAD_v1")

What this does/why it matters: This step makes the process repeatable: a single function that either reuses an existing local copy or re-downloads cleanly when force=True. Reproducibility later depends on stable file locations.

Convert SQuAD JSON to clause records: CUAD ships in SQuAD format. (The specific format isn’t particularly important for our purposes. Often, there will be datasets in a variety of formats, and it will be our job to somehow wrangle them into whatever shape is useful for our work.) We normalize the CUAD dataset to compact records: the exact clause, a context window around it, the label, and contract metadata.

def extract_clauses_from_squad(
    squad: dict, context_window: int = 150
) -> list[dict]:
    items = []
    for doc in squad["data"]:
        title = doc.get("title", "")
        ctype = (title.split("-")[-1].strip()
            if "-" in title else "Unknown")
        for para in doc.get("paragraphs", []):
            ctx = para.get("context", "")
            for qa in para.get("qas", []):
                if qa.get("is_impossible"): continue
                label = (qa.get("id","").split("__")[-1] or "UNKNOWN")
                for ans in qa.get("answers", []):
                    start, text = ans["answer_start"], ans["text"]
                    end = start + len(text)
                    span = ctx[
                        max(0, start-context_window): min(len(ctx),
                        end+context_window)]
                    items.append({
                        "clause": text,
                        "clause_with_context": span,
                        "label": label,
                        "contract_name": title,
                        "contract_type": ctype,
                    })
    return items

What this does/why it matters: We keep the gold label and add a bounded context to help the teacher and (later) the student model reason. Using character windows avoids tokenizer coupling; you can swap to token-based windows later if desired.

Add NONE negatives from unlabeled gaps (Optional): To improve robustness, we can mine negatives by sampling from gaps (text regions not covered by any labeled span).

def add_negative_none_examples(
    items: list[dict], squad: dict,
    ratio: float = 3.0, neg_len: int = 100,
    context_window: int = 150
):
    # 1) record labeled (start,end) spans per paragraph
    # 2) compute gaps between labeled spans
    # 3) sample fixed-length snippets from gaps → label="NONE"
    # 4) cap per-gap samples (≤3) so negatives don't dominate
    ...
    return items_with_negatives

What this does/why it matters: This step adds coverage for everything else. For the benefits, risks, and default choice we use in this chapter, see the dedicated The NONE label decision section later in the chapter.

Create stratified splits (with graceful fallback): We split our data into train, validation, and test sets. This ensures that we aren’t getting a false sense of confidence when we evaluate how well our model is doing and it is used during the training process, as well as afterwards.

def split_train_val_test(items, train=0.8, val=0.1, seed=42):
    # stratify by label for train vs temp
    train_set, temp = train_test_split(
        items, train_size=train,
        stratify=[x["label"] for x in items], random_state=seed)
    # compute relative val size; stratify if each class has ≥2 samples, else plain split
    ...
    return train_set, val_set, test_set

What this does/why it matters: Stratified splits into train and test sets, preserves label proportions where possible, gracefully falling back if a minority class is tiny. We use an 80/10/10 stratified split with seed=42.

Leakage rule: We never train on the test split (or its rationales).

Prompt the teacher for rationales (rationale distillation): We ask for an explanation of the existing label (not for a new label). This is deliberate: we are distilling reasoning for a known classification, not re-labeling CUAD.

def create_prompt(sample: dict) -> str:
    return f"""[[TASK]]
Explain why this clause fits the label.
[[CONTEXT]]
CLAUSE: {sample["clause"]}
CLAUSE WITH CONTEXT: {sample["clause_with_context"]}
CONTRACT TYPE: {sample["contract_type"]}
CLASSIFICATION: {sample["label"]}
[[INSTRUCTIONS]]
Identify the clause language that justifies the label. Be concise, stepwise, and use Markdown."""

What this does/why it matters: Conditioning on the gold label keeps outputs on-task and cheaper. An alternative label-and-rationale mode can catch mislabels, but costs more and risks disagreement you must adjudicate.

Orchestrate generation (parallel, rate-limited, resumable): We make sure that the LLM generation happens in parallel and that any errors or rate limiting are caught so we can automatically retry.

@backoff.on_exception(
    backoff.expo, (Exception,), max_tries=5, max_time=300)
def generate_once(
    client, prompt, model="deepseek/deepseek-r1", max_tokens=1500,
    temperature=0.2, top_p=0.95
):
    # OpenRouter OpenAI-compatible chat.completions.create(...)
    return content, reasoning_trace_or_none
class RateLimiter:
    def __init__(self, qps: float): ...
    def wait(self): ...
def process_one(idx, sample, client, limiter, out_path, lock):
    limiter.wait()
    rationale, trace = generate_once(client, create_prompt(sample), ...)
    rec = {**sample, "rationale": rationale,
        "reasoning_trace": trace, "index": idx, ...}
    with lock:
        append_jsonl(out_path, rec)

What this does/why it matters: Parallel workers and a global queries per second (QPS) limiter give throughput without throttling errors; exponential backoff handles transient failures; thread-safe appends prevent interleaved writes. In practice, this made the full pass reliable and cheaper (OpenRouter docs: https://openrouter.ai/docs).

Resume safely (skip completed indices): Since this is a process that costs real money, takes a long time to execute and could potentially be interrupted, we keep track of which items we’ve completed so that if we want to pause or resume at some point, then we are able to do so.
```
def get_processed_indices(path) -> set[int]:
    # read output JSONL; collect "index" fields → allows resume
    ...
```

What this does/why it matters: You can stop and resume at any time. This is critical when batching costs or rotating API keys.

Package and publish: We store JSONL files for each split and optionally push DatasetDict to the HuggingFace Hub (https://huggingface.co).
```
def to_hf_dataset(jsonl_path: str) -> Dataset: ...
def push_to_hub(
    dataset_or_dict, repo_id: str, private=False
) -> None: ...
```

What this does/why it matters: A consistent, documented artifact others can reuse without re-generating synthetic data.

Cost and throughput note

In our reference runs, generating rationales for one split with modest length limits and low temperature cost roughly $100. Generating rationales across all splits scales proportionally (typically ~2–3× depending on token limits and coverage).

Lightweight quality checks (cheap but effective): Before we accept a rationale, we run simple filters:

def accept(sample: dict, rationale: str) -> bool:
    if rationale is None or len(rationale.strip()) < 50: return False
    if rationale.lower().count("this clause") >= 4: return False  # generic filler
    if sample["label"].lower() not in rationale.lower()[:300]: return False  # label not grounded early
    return True

What this does/why it matters: These heuristics are intentionally simple: they reject low-effort text and encourage label-grounded explanations without paying for an extra judge model. For higher stakes, replace this with a small rationale-judge prompt that scores specificity and cites key phrase matches.

Once these steps are complete, it’s time to move on to ZenML pipeline data processing.

ZenML pipeline data processing

The pipeline layer manages data preprocessing through a dedicated load_data step and records each run’s inputs and outputs for reproducibility. Each execution captures which dataset revision was used, the preprocessing configuration, and the resulting artifacts.

Here is how the pipeline is implemented:

@step(enable_cache=True)
def load_data(
    model_size: str = "12b",
    dataset_id: str = "zenml/cuad-deepseek",
    max_samples: Optional[int] = None,
    filter_none_labels: bool = True,
) -> Tuple[pl.DataFrame, pl.DataFrame, HTMLString]:
    """Load and preprocess the Enhanced CUAD dataset."""

    # Get model configuration and create tokenizer
    config = get_model_config(model_size)
    tokenizer = create_tokenizer(config)

    # Download and prepare the dataset
    download_and_save_dataset(
        dataset_id=dataset_id, output_dir=output_dir)
    train_dataset, val_dataset = load_datasets(
        max_samples, tokenizer, tokenize=False, filter_none_labels
    )

    return train_dataset.to_polars(), val_dataset.to_polars(), viz

When this step runs, it does the following:

Resolves model_size to a known configuration and builds the matching tokenizer so any downstream formatting uses the correct chat template and sequence limits.
Fetches or reuses the requested enhanced CUAD dataset (dataset_id) in a stable directory output_dir so the exact revision is materialized locally.
Prepares the training/validation splits via load_datasets(...), which applies your switches, max_samples for quick experiments, and filter_none_labels to include/exclude the open-ended NONE class, while deliberately deferring tokenization unless you enable it (we keep examples as strings here to pack or trim consistently during training).
Converts the resulting splits to compact Polars DataFrames so they serialize quickly as artifacts and are easy to profile.
Emits a small HTML visualization (viz) that records what was loaded (counts, filter policy) and becomes part of the run’s lineage.

Please note that since we have set @step(enable_cache=True), ZenML computes a cache key from the step code, parameters, and upstream artifacts; re-running with the same inputs reuses the materialized training or validation DataFrames and viz, while changing any input (e.g., model size, dataset ID, filter policy, or sample cap) invalidates the cache and recomputes. The net effect is reproducible data prep with fast, parameter-aware iteration.

A practical benefit is automatic caching: once a step has run with the same inputs, the orchestrator can reuse its outputs. That reduces iteration time without changing any training logic. In our examples, we use ZenML to provide these behaviors, but any orchestrator with step caching and run metadata will work similarly.

A key preprocessing consideration for this dataset is how to handle the NONE classification examples. Let’s talk about them.

The NONE label decision

These are clauses that do not match any of the 41 predefined CUAD categories. In effect, they serve as a catch-all for content outside the main schema. Including NONE examples can provide useful negative training data, but it also introduces additional complexity to the classification task.

The two practical trade-offs negatives and windows matter the most:

Negative ratio: We default to 3:1 negatives:positives when enabled; increasing this can improve precision in the wild, but slows training and can bias towards NONE. Start at 1–3× and tune by validation F1.
Context windowing: We use a character window (e.g., 150 each side) for speed and tokenizer independence. If you observe truncated tokens or sentence breaks that confuse the teacher/student, switch to a token-aware window (e.g., via a tokenizer) and keep total sequence lengths within your training budget.

The NONE category was not part of the original CUAD dataset. The initial focus was on identifying specific clause types that legal professionals care about during contract review. We introduced the NONE examples during our enhancement process to help balance the dataset, but this addition changes the nature of the classification problem.

Classifying NONE examples is fundamentally different from classifying among specific legal categories. The model must learn to recognize the absence of any defined legal pattern, which is a more abstract and open-ended task than matching a clause to a known type. For example, it is similar to asking a model to identify not-cats rather than distinguishing between breeds of dogs.

For the purposes of this chapter, we exclude NONE examples by default using the filter_none_labels=True parameter. This allows us to focus on the core challenge of distinguishing between meaningful legal categories without the added complexity of the everything else class. In a production setting, you might choose to include NONE examples to improve robustness, but for learning and demonstration, a more focused approach is often preferable.

# The data preparation includes smart filtering
def prepare_dataset_from_file(
    jsonl_path,
    max_samples=None,
    tokenizer=None,
    filter_none_labels=False,
):
    """Process JSONL data with enhanced instructional format."""
    data = []
    none_count = 0

    with open(jsonl_path, "r") as f:
        for line in f:
            item = json.loads(line)

            # Skip NONE examples if filtering is enabled
            if filter_none_labels and item.get("label", "").strip().upper() == "NONE":
                none_count += 1
                continue

            # Process valid examples...

Once we’ve downloaded the data and filtered it, we then need to format it so it’s ready for fine-tuning.

Next up, we will provide an instructional format for fine-tuning our student model.

Instructional format for fine-tuning

The data preprocessing also handles the crucial task of formatting examples for instruction-tuned models. (By instruction-tuned, we mean language models that have been specifically trained to follow user instructions and provide helpful responses, rather than just predicting the next word in a sequence like base models do.) Rather than simple input-output pairs, we structure each example in the dataset as a detailed conversation that teaches the model both what to classify and how to reason about the classification. Here’s an example:

{
    "clause": "access is reasonably required by the other, including without limitation, for audit, accounting and litigation purposes.",
    "clause_with_context": "ther (other than data and information subject to any attorney/client or other privilege), insofar as such\n\n
                                       27\n\naccess is reasonably required by the other, including without limitation, for audit, accounting and litigation purposes.\n\n     (c) Notwithstanding the foregoing, either party may destroy or otherwise dispose of any information at any time in accordance with the corporat",
    "label": "Audit Rights",
    "contract_name": "FIDELITYNATIONALINFORMATIONSERVICES,INC_08_05_2009-EX-10.3-INTELLECTUAL PROPERTY AGREEMENT",
    "contract_type": "INTELLECTUAL PROPERTY AGREEMENT"
}

Each training example becomes a JSON-formatted instruction that includes the task description, classification schema, valid label options, and the input clause with context. The model learns to respond with structured reasoning followed by the final classification – exactly the behavior we want in our deployed system.

This instructional approach helps provide the reasoning that legal professionals can evaluate and trust, and not just the output label. The preprocessing pipeline handles this formatting automatically, ensuring consistency across all training examples while maintaining the flexibility to experiment with different instructional prompts and response formats. If you don’t pay much attention to the data format for fine-tuning, your model might not even work at all.

We will now train the student (Gemma 3) on the rationale-augmented enhanced CUAD that we have generated at this stage.

Stage 2: Fine-tuning Gemma 3 on CUAD

With our enriched dataset prepared and our reasoning foundation established, we will now move on to the core challenge: distilling this legal expertise into smaller, deployable models that can operate efficiently within the constraints of an on-premises legal environment. This is where model selection becomes crucial – we need an architecture that balances capability with practicality.

Why Gemma 3?

Gemma 3 (https://ai.google.dev/gemma/docs/core) is a modern, resource-efficient language model architecture, inheriting core innovations from Google’s Gemini 2.0 family but specifically engineered for single-GPU deployment. For technical teams building legal document classifiers, Gemma 3 offers a compelling balance of performance and deployability.

Unlike most LLMs that require multi-GPU clusters or distributed inference, Gemma 3 is designed to run inference and even training for the 12B parameter variant on a single high-end GPU. This is achieved through architectural optimizations, most notably a 5:1 ratio of local to global attention layers. This design choice significantly reduces key-value (KV) cache memory requirements during long-context inference. For example, where conventional transformer architectures might see memory overheads increase by 60% or more as context length grows, Gemma 3 keeps this under 15%. This enables practical processing of long legal documents without the need for specialized hardware or distributed systems.

Why not fine-tune DeepSeek directly?

For this use case, it’s operationally better to distil DeepSeek’s behavior into an open, efficient student you can fine-tune and run locally. This avoids heavyweight compute requirements and simplifies deployment/governance while retaining most of the teacher’s legal reasoning performance.

For our use case – contract clause classification with context windows spanning thousands of tokens – this means we can deploy a 12B parameter model on standard workstation hardware, without sacrificing the reasoning depth or accuracy demonstrated by larger, less efficient models like DeepSeek-R1. The result is a production-ready legal AI system that is both performant and operationally feasible for on-premises deployments, even in environments with strict data privacy requirements.

The model size decision: Why we focus on 12B?

While our pipeline supports three model sizes (1B, 4B, and 12B parameters), our experiments with the enhanced CUAD dataset revealed a clear reality: for legal contract classification, you need substantial model capacity to achieve professional-grade accuracy.

Why smaller models fall short: The 1B and 4B Gemma 3 models, while computationally efficient, simply don’t deliver the accuracy levels required for serious legal work. The 1B model struggles to maintain consistent reasoning across complex legal language, especially while ensuring JSON output, while the 4B model, though better, still makes too many classification errors to be reliable in high-stakes contract review scenarios.
The 12B model: This is our primary focus. Hence, we focus primarily on the 12B model in our examples and analysis. While it requires 24 GB+ of GPU memory during training, it represents the minimum viable size for professional legal classification tasks of this kind. Even then, the 12B model isn’t perfect – legal language is nuanced and context-dependent in ways that challenge even sophisticated models.
Deployment trade-offs: You’re welcome to experiment with the smaller models using our pipeline (simply change the model_size parameter), and they may be suitable for preliminary document screening or resource-constrained environments. However, for the accuracy levels demanded in legal practice, the computational overhead of the 12B model is generally justified by its superior performance on complex contract language.

Let’s get started with the fine-tuning process.

The fine-tuning process

The following code defines the distil_finetuning pipeline.

@pipeline
def distill_finetuning(
    max_samples: Optional[int] = None,
    model_size: str = "4b",
    filter_none_labels: bool = True,
):
    """Pipeline to load the dataset and finetune the model.

    Args:
        max_samples: Maximum number of samples to use for training
        model_size: Model size to use ('1b', '4b', or '12b')
        filter_none_labels: Whether to filter out 'NONE' classifications
    """
    # Step 1: Load and prepare data with intelligent caching
    train_dataset, val_dataset, data_viz = load_data(
        model_size=model_size,
        filter_none_labels=filter_none_labels,
        max_samples=max_samples
    )

    # Step 2: Fine-tune with automatic resource management
    finetune_model(
        train_dataset,
        val_dataset,
        model_size=model_size
    )

Fine-tuning legal classification models requires hyperparameters that balance learning efficiency with stability. Our configuration reflects lessons learned from our experimentation while working on this chapter.

# Optimized training configuration for 12B legal model
TRAINING_CONFIG = {
    "per_device_train_batch_size": 1,
    "gradient_accumulation_steps": 8,
    "warmup_steps": 100,
    "num_train_epochs": 2,
    "learning_rate": 1e-4,
    "logging_steps": 20,
    "optim": "adamw_8bit",
    "weight_decay": 0.01,
    "lr_scheduler_type": "cosine",
    "seed": 3407,
}

Here is a rationale for why we chose these parameters:

Learning rate: The 1e-4 learning rate is more conservative than typical fine-tuning rates, reflecting the need for stability when working with large models on specialized legal text. The 12B model’s substantial parameter space requires careful learning rate selection to avoid destabilizing pre-trained knowledge while still developing legal expertise.
Memory-optimized batch configuration: The reduced batch size (1) accommodates the 12B model’s memory requirements, while increased gradient accumulation (8) maintains an effective batch size of 8. This configuration balances training stability with the practical constraints of single-GPU training for large models.
Extended warmup for stability: The 100-step warmup period provides additional stability for the large model, allowing gradual adaptation to the legal domain rather than aggressive early updates that might disrupt learned representations.
The cosine learning rate schedule: The cosine scheduler provides smooth learning rate decay, which works particularly well with larger models by maintaining stable training dynamics throughout the process.

With the optimizer and schedule fixed, the next decision is which parameters to update. Full fine-tuning changes every weight, but that’s memory-heavy and unnecessary; instead, we use a parameter-efficient method that learns small, targeted updates while keeping the base model frozen.

LoRA configuration for parameter updates: Low-Rank Adaptation (LoRA) is a way to teach a large model new skills without retraining the whole thing. Think of the base model as a high-end instrument: rather than rebuilding it, you clip on a tiny attachment that slightly adjusts how it plays. During training, only these small attachments learn; the original model stays untouched. This makes fine-tuning faster, cheaper, and reversible, and it works well even when the base model is loaded in 4-/8-bit (https://arxiv.org/abs/2106.09685).

Our LoRA configuration targets the specific adaptations needed for legal language understanding. By specific adaptations, we mean that we choose where to place those small attachments and how big to make them so the extra learning capacity goes where it matters.

For legal text, that typically means adding adapters to attention and MLP blocks so the model better handles long cross-references, formal clause language, headings and definitions, and the structured outputs we expect (rationale and label). In short, we focus the limited trainable capacity on the patterns that improve accuracy in this domain. Here’s how:

# LoRA configuration optimized for 12B legal model
LORA_RANK = 16
LORA_ALPHA = 32
LORA_DROPOUT = 0
LORA_BIAS = "none"

Here is a sample loss curve for our training run.

Figure 7.3: Loss curve for a sample training run made during development

Let’s take a look at what these optimizations mean:

Increased rank for larger model: The rank 16 configuration provides additional capacity needed for the 12B model to effectively capture legal language patterns. Larger models benefit from higher LoRA ranks as they have more parameter space to adapt while still maintaining efficiency compared to full fine-tuning.
Alpha = 2 × rank convention: Following the established convention, we set alpha to 32 (2 × rank), which provides appropriate scaling for the LoRA adaptation. This ratio has proven effective across different model sizes and domains.
Zero dropout and comprehensive targeting: Legal language leaves little room for ambiguity, so we preserve all learned associations. By fine-tuning both attention and MLP modules across all language layers, we ensure legal expertise permeates the entire model architecture – both syntactic parsing and semantic reasoning components. Concretely, the code sets LORA_DROPOUT=0 and enables finetune_language_layers, finetune_attention_modules, and finetune_mlp_modules in constants.py, then applies these flags via Unsloth’s FastModel.get_peft_model in steps/trainer.py. This attaches LoRA adapters across all attention and MLP modules in the language stack with no adapter dropout, ensuring domain updates propagate end to end.

Now that we have looked at the parameter settings, we will understand the specifics of model training for legal data.

Training dynamics of legal AI learning

Legal AI training follows predictable patterns that reveal how models internalize domain expertise. During the first 100 training steps, models typically show rapid loss reduction as they adapt to legal vocabulary patterns. The loss curve demonstrates characteristic behavior: an initial steep drop followed by gradual improvement as the model masters clause classification logic.

Training memory usage scales predictably with model size – the 1B model peaks at 8-10 GB, while the 12B model requires 24-28 GB, depending on sequence lengths. These kinds of classification loss curves typically show three phases: rapid initial learning (steps 1-50), steady improvement (steps 50-200), and gradual refinement beyond step 200. Note that this often applies to much of machine learning.

We will now compare the fine-tuned student (Gemma 3) against the DeepSeek-R1 teacher on a held-out test set to validate that rationale distillation and fine-tuning achieved the intended gains.

Once you have fine-tuned your model, it is essential to validate whether it is performing as per your expectations. Let’s evaluate our fine-tuned model next.

Evaluation and results

To demonstrate (with evidence) that our fine-tuned legal classifier actually performs better than the massive DeepSeek model we used as our teacher, think of this as a final exam where our 12B parameter student must demonstrate it learned legal reasoning effectively enough to compete with its 600+ billion parameter professor.

Split policy reminder: We evaluate on the enhanced CUAD test split (10% of the dataset), which is strictly held out from training and hyperparameter tuning.

The evaluation pipeline runs two parallel inference processes: one using our local fine-tuned model, and another querying DeepSeek-R1 through OpenRouter’s free API. Both models analyze the same legal clauses from our enhanced CUAD test set – data that was never seen during training.

@pipeline
def evaluation_pipeline(
    num_samples: Optional[int] = 50,
    exclude_none: bool = True,
    model_size: str = "4b",
    use_local_model: bool = True,
    deepseek_max_workers: int = 5,
):
    """Compare fine-tuned model against DeepSeek base model."""
    # Load held-out test data
    test_dataset = load_test_data(num_samples, exclude_none)

    # Evaluate our fine-tuned model
    finetuned_results, finetuned_viz = evaluate_model(
        test_dataset, model_size, use_local_model
    )

    # Evaluate DeepSeek base model via API
    deepseek_results, deepseek_viz = evaluate_deepseek_base(
        test_dataset, max_workers=deepseek_max_workers
    )

    # Generate comprehensive comparison
    comparison_results, comparison_viz = compare_models(
        finetuned_results, deepseek_results
    )

An important aspect to measure model performance is the choice of metrics. While there are many popular metrics, such as accuracy and F1 score, that are often used in multi-label classification tasks, these metrics might not be sufficient for all use cases across domains.

Let’s explore the performance metrics we will use to evaluate our fine-tuned model.

Performance metrics

Legal AI evaluation requires more nuanced metrics than typical classification tasks. A model that misses critical cap on liability clauses (one of the possible labels in the dataset) poses different risks than one that occasionally misclassifies less crucial provisions. Our evaluation framework captures this complexity through multiple performance dimensions.

Overall accuracy: Raw classification performance across all 41 legal clause types provides the baseline metric. Our fine-tuned 12B model consistently achieves accuracy in the high 80% range – remarkable performance for a model 150x smaller than DeepSeek.
Non-NONE accuracy: This metric focuses on actual clause identification, excluding the everything else category. Since legal professionals primarily care about finding specific clause types, this often represents the most practically relevant metric.
Per-class analysis: Legal clauses vary dramatically in complexity and importance. Governing law provisions follow predictable patterns, while change of control clauses can be subtly buried in complex language. Our evaluation breaks down performance by clause type, revealing where fine-tuning provides the most value.
Precision, recall, and F1: These metrics help understand the trade-offs between catching all relevant clauses (recall) versus avoiding false positives (precision) – both critical in legal applications, where missed clauses can be costly.

Let’s take a look at the results now.

Evaluation results

Our comparative evaluation (Figure 7.4) reveals surprising results that validate the distillation approach for legal applications. The fine-tuned models do not just compete with DeepSeek; they substantially outperform it on legal classification tasks.

Figure 7.4: Performance comparison showing fine-tuned Gemma 3 models achieving 84.5% accuracy compared to DeepSeek’s 41.5%, with significantly reduced computational requirements

The results demonstrate that specialized training on high-quality synthetic data from the enhanced CUAD dataset can create models that dramatically exceed the performance of much larger general-purpose systems.

Error analysis: Understanding model limitations

Both models struggle with certain types of legal clauses, but their failure patterns differ in instructive ways. The fine-tuned model shows a concentrated error pattern with only 11 unique ways it failed, compared to DeepSeek’s 38. This suggests more systematic and potentially addressable failure modes.

Figure 7.5: Confusion matrices revealing the fine-tuned model’s 169 correct classifications out of 200 samples, with notably cleaner diagonal patterns than the teacher model

Both models struggle with the following:

Agreement date disambiguation: The fine-tuned model’s most common error (6 instances) involves distinguishing between Effective Date and Agreement Date.
License grant nuances: Several error types (Exclusivity, Non-Transferable License, Affiliate License) converge to License Grant classifications.
Clause boundary detection: Both models occasionally struggle with overlapping or compound clauses.

But the fine-tuned model is able to achieve improvement in several aspects when compared to the base DeepSeek-R1 model:

Elimination of parsing failures: Unlike DeepSeek’s 12 parsing errors, the fine-tuned model produces consistently parseable outputs.
Reduced error diversity: 71% fewer unique error types (11 versus 38), indicating more predictable behavior.
Improved F1 score: 75.3% compared to 69.8%, balancing precision and recall more effectively.

Our evaluation writes metrics and artifacts to an experiment-tracking dashboard, so results persist beyond a single run. In our reference implementation, this is a ZenML dashboard (https://docs.zenml.io), but MLflow or Weights & Biases (W&B) could serve the same purpose. What matters is that you log the following:

Model/checkpoint identifiers and lineage.
Dataset hash/split spec and preprocessing version.
Per-class metrics, confusion matrices, and error slices.

Figure 7.6: Screenshot of our evaluation results and the training pipeline on the ZenML dashboard

The implementation for this chapter highlights the following:

Quantified improvements: Clear metrics show the 2x performance gain from distillation.
Error pattern analysis: Detailed breakdown of the 31 remaining errors guides targeted improvements.
Reproducible evaluations: Complete artifact lineage from the unsloth/gemma-3-12b-it base through fine-tuning.

Of course, there will always be scope to improve performance. Let’s see how you can do it.

Performance optimization potential

Our current results, achieved with just 2 epochs of training, represent a strong foundation rather than a performance ceiling. The dramatic improvement from 41.5% to 84.5% accuracy suggests several optimization strategies could push performance even higher.

Targeted error remediation: With only 31 errors concentrated in 11 patterns, focused synthetic data generation for these specific cases (particularly the 6 Effective Date/Agreement Date confusions) could yield immediate gains. In other words, we could generate example data that focuses on the patterns our fine-tuned model found hardest to recognize. By boosting the sample strength for those patterns, we’d hope our (re-)fine-tuned model would perform better.
Precision-recall balancing: The slight dip in recall (–2.8%) alongside a solid precision gain (+11.4%) means you can move the operating point without retraining the whole model. Start with inference-time tweaks:
1. Pick per-class decision thresholds from a validation set so the model is stricter on high-cost mistakes and more permissive where misses are worse.
2. Introduce an abstain/needs review or NONE path for low-confidence cases.
3. Calibrate confidence scores using a held-out set so probabilities reflect reality (most libraries offer simple calibrators). If that isn’t enough, do small, targeted training changes: add class weights to address imbalance, try a loss that focuses more on hard examples, or lightly over-/under-sample difficult labels.

Because each fine-tuning run costs time and money, prefer thresholding and calibration first, then a very small, pre-planned sweep of training options.

Multi-stage training: Given the complete elimination of parsing errors, a curriculum learning approach that starts with output structure before advancing to nuanced classification could further improve the already strong 84.5% accuracy. You might use a staged schedule that teaches easy skills first, then adds complexity.
1. Stage 0: Format first: Short contexts and high-support labels to stabilize structured outputs (valid JSON, label tokens).
2. Stage 1: Full enhanced CUAD minus rare/ambiguous cases.
3. Stage 2: Introduce long contexts, compound clauses, and (optionally) the NONE class.
4. Stage 3: Hard-negative mining and near-misses focused on the top error slices (e.g., Effective Date versus Agreement Date). Then, gradually increase the sequence length and difficulty; keep adapters fixed while you scale difficulty to avoid destabilizing earlier gains.
  - Ensemble approaches: The distinct error patterns between models (38 unique types for DeepSeek versus 11 for the fine-tuned model) suggest ensemble methods could leverage complementary strengths. Start simple:
5. Seed/snapshot ensemble: Train the same 12B LoRA model three times with different random seeds or data shuffles and take a majority vote.
6. Mixed-capacity ensemble: Combine a 12B and a 4B student; when they disagree, send the case to human review or the teacher model.
7. Confidence-based routing: If the student’s confidence is low, fall back to DeepSeek-R1; otherwise, accept the student’s answer.
8. Test-time augmentation: Run two views of the same clause (e.g., different context windows) and vote on the result. To control cost, apply ensembling only to the hardest 10–20% of examples, as identified by low confidence or prior error slices.

These results validate that distilled legal models can meet professional requirements while operating within practical constraints. The accuracy score (more than 80%) represents a significant improvement over manual review consistency. The detailed reasoning capabilities enable legal professionals to verify and trust the system’s analysis.

For law firms considering AI adoption, these results demonstrate that sophisticated legal reasoning capabilities don’t require massive infrastructure investments. A single high-end workstation can deploy models that compete with the largest available systems while maintaining complete control over sensitive client data.

Up next, a quick roundup of key learnings and takeaways from this chapter.

Key takeaways

We’ve taken a remarkable journey from massive 600+ billion parameter models to efficient 12B parameter legal specialists that can run on a single GPU while delivering competitive performance. This transformation – from Swiss Army knife to surgical scalpel – demonstrates that, with the right approach, specialized AI can be both powerful and practical.

When does distillation and fine-tuning make sense?

The legal contract classification challenge taught us that distillation delivers exceptional value when three conditions align:

Specialized domain requirements: Legal language operates by different rules than general text. Phrases such as time is of the essence carry specific legal weight that general models struggle to understand consistently. When your domain has specialized vocabulary, reasoning patterns, or regulatory requirements, fine-tuning becomes essential rather than optional.
Quality training data availability: Our enhanced CUAD dataset with DeepSeek-generated rationales provided the rich learning signals that made distillation successful. The combination of expert classifications and detailed reasoning created training examples that captured both the what and why of legal analysis.
Deployment constraints: Law firms can’t send confidential client documents to external APIs. This constraint – common across regulated industries – makes distillation not just helpful but necessary. The ability to deploy sophisticated legal reasoning on-premises transforms distillation from an optimization technique into a business enabler.

Distillation also helps solve multiple challenges simultaneously in ways that traditional approaches can’t match:

Infrastructure democratization: Converting DeepSeek’s legal expertise into a 12B parameter model means sophisticated legal AI runs on hardware that law firms already own – no specialized infrastructure, no ongoing API costs, no external dependencies.
Privacy preservation: Client confidentiality remains intact when the entire AI system operates within the organization’s security perimeter. This isn’t just a technical advantage – it’s often a legal requirement.
Customization potential: Fine-tuned models can adapt to specific law firm writing styles, jurisdiction requirements, or client needs. Unlike API-based solutions, these models evolve with your organization.

Another important consideration is to ensure governance and reproducible results. Throughout this chapter, we rely on an experiment-tracking/orchestration layer to turn ad hoc scripts into repeatable workflows. In our example, this layer is ZenML, but any comparable system that captures runs, artifacts, and lineage is fine. The key capabilities we need are compliance documentation (where runs produce an immutable log of data, code, config, and metrics), reproducible experiments (where preprocessing/training/evaluation can be re-executed exactly), and model governance, where versions of models and datasets are promoted with the same rigor as software.

Summary

In this chapter, you transformed CUAD into an enhanced CUAD dataset via rationale distillation. You normalized SQuAD annotations to clause-level records, optionally mined NONE negatives, created stratified splits, and prompted DeepSeek-R1 (via OpenRouter) for concise, label-grounded rationales. You executed rate-limited, backoff-hardened parallel generation with a safe resume and simple quality checks, then packaged the result and formatted examples for instruction-tuned chat training.

You then fine-tuned Gemma 3 with LoRA using Unsloth, selected stable training/LoRA hyperparameters, and evaluated the student against the teacher on held-out data with accuracy, F1, and per-class analysis. You also learned how an orchestration/tracking layer can capture lineage and caching for reproducibility – and how to run the same workflow from a single-file script if you prefer no orchestrator. These skills generalize beyond law to any domain that benefits from explanation-augmented supervision.

Next, Chapter 8 moves from training to deployment. We’ll deploy both full DeepSeek models (V3/R1), compare self-hosted and managed options and the inference settings that balance latency, cost, privacy, and reliability.

Get This Book’s PDF Version and Exclusive Extras

Scan the QR code (or go to packtpub.com/unlock). Search for this book by name, confirm the edition, and then follow the steps on the page. | |

| Note: Keep your invoice handy. Purchases made directly from Packt don’t require one. |

8 Deploying DeepSeek Models

In the previous chapter, we distilled and fine-tuned smaller, domain-specific models that you could run on modest hardware and within strict privacy boundaries. That work is optimized for efficiency and control at a smaller scale. This chapter takes the complementary step of deploying full-parameter DeepSeek models (V3 and R1) as dependable production services.

Deployment is the bridge from research to production. It forces concrete choices about memory footprint, throughput, and operational risk. DeepSeek’s architectures magnify these trade-offs: V3’s Mixture-of-Experts (MoE) stresses VRAM placement; R1’s extended reasoning inflates token counts and time-to-first-token. The right path depends on your constraints.

As of August 2025, any pricing and throughput figures in this chapter are directional. Verify current provider sheets and validate on your workload before hardcoding assumptions.

This chapter provides a practical playbook for deploying full-parameter DeepSeek models across real-world scenarios. We will focus on deploying DeepSeek-V3 (671B) and DeepSeek-R1 at full scale; smaller models are mentioned only as development proxies.

We’ll cover the following main topics in this chapter:

The DeepSeek deployment landscape
A decision-making framework for choosing your deployment strategy
Hardware and inference optimization requirements
Hands-on deployment guides
Production operations and monitoring
Your deployment playbook

Note on scope

This chapter does not cover deploying the smaller fine-tuned/student model from Chapter 7. Any mention of smaller models is strictly for development or proxy workflows, not production deployments of the full models.

Technical requirements

In this chapter, we will explore various methods to deploy DeepSeek. For each method, the technical requirements will be mentioned in the respective sections. You can utilize any method that best suits your application and your access to the tools and platforms.

The DeepSeek deployment landscape

In production, you focus on three things: monitoring, scaling, and cost. You monitor tokens per request, time to first token (TTFT), p95/p99 latency, GPU, and key-value (KV) cache usage, and error/timeout rates. You scale based on how many tokens the system is actively generating, not just how many HTTP requests arrive. And you control cost by limiting tokens, batching where possible, caching common prefixes, and pausing or right-sizing replicas when idle.

For this, you need to keep a tab on the following:

Memory footprint: This is the VRAM you need at runtime. It includes model weights plus overhead such as the KV cache and activations. Track peak VRAM and KV-cache bytes per token; these numbers decide your safe batch size and context length, and whether you need multiple GPUs or some weights moved to CPU/NVMe.
Throughput: This is how many tokens per second your service can produce at your target concurrency. For planning, also report requests per second at a fixed prompt/response size, and always pair it with TTFT and p95/p99 latency, so the number reflects real user experience.
Operational risk: This is the chance you miss SLOs or blow your budget because of instability or bad settings. Watch error and timeout rates, Out of Memory (OOM) events, queue growth, and how safely you roll out changes or scale up and down.

DeepSeek adds some extra wrinkles. V3 uses an MoE design that runs best when all experts stay in GPU memory, which pushes VRAM needs even with quantization; moving weights to CPU or NVMe helps fit but slows things down. R1 thinks a lot, often generating thousands of internal tokens before the final answer, which raises TTFT, increases memory use, and makes latency less predictable. Because of this, clear sizing, careful batching, and basic dashboards are must-haves before you choose API, managed, or self-hosted deployment.

DeepSeek’s deployment quirks

DeepSeek models bring specific challenges:

Model size: Even quantized versions push hardware limits. DeepSeek-V3’s 671B total parameters need substantial VRAM, regardless of its MoE efficiency.
Reasoning overhead: R1 models generate extensive internal reasoning traces. What looks like a simple query might produce thousands of intermediate tokens.
Architecture considerations: V3’s MoE design typically keeps all experts resident in GPU memory for performance; however, this is not a hard requirement. Modern stacks can offload/partition (CPU/NVMe, tensor-parallel across nodes) at the cost of significant latency/throughput.

These challenges require specific tools and techniques for large-model deployment. Let’s start by examining why you might deploy DeepSeek yourself rather than using the official API.

Why self-deploy and what makes DeepSeek unique?

Often, you will encounter scenarios where you will choose to self-deploy DeepSeek or any other SOTA LLM. Consider, for example, a scenario where you’re handling sensitive legal documents such as merger agreements – the official DeepSeek API may not be an option. Not because it’s inadequate, but because client contracts often forbid sending privileged documents to third-party services. This scenario illustrates why self-deployment isn’t just a technical exercise. It’s often a business necessity.

Let’s talk about some of the key reasons you might want to self-deploy:

Control and customization: Control isn’t a nice-to-have; it’s how you turn a generic model into a system that fits your domain, risk posture, and product constraints. Self-deploying lets you touch the layers that APIs won’t expose (tokenization, routing, adapters, system prompts, and safety policies) so the model reflects your requirements instead of a vendor’s defaults. When Bloomberg built BloombergGPT, they didn’t just want a financial language model; they needed to modify the tokenizer to handle financial terminology and keep half their training corpus proprietary. This level of control is impossible with API-based services.

For DeepSeek deployments, control means the following:

Adjusting generation parameters beyond API limits.
Merging domain-specific adapters (e.g., LoRA) or specialized adapters.
Implementing custom safety filters appropriate for your domain.
Modifying system prompts without vendor restrictions.
Cost at scale: Cost is not just a price list; it’s a function of utilization, batching, and caching. If your traffic is steady or high-volume, owning the serving stack lets you amortize hardware and push utilization, often dropping effective per-token costs far below metered APIs. The economics flip dramatically at scale. For a legal document workflow processing hundreds of contracts daily, use the following directional costs, as of August 2025 (verify before hardcoding):
- API costs: ~$30–50 per million tokens
- Self-hosted (amortized): ~$2–5 per million tokens
- Break-even point: Typically around 10–20M tokens/month
Privacy and data residency: For many organizations, privacy isn’t optional; it’s mandated by contracts and regulation. Self-deployment keeps sensitive tokens within your perimeter, enforces data residency and retention policies, and provides auditability over where data flows. For instance, University Hospital Bonn deployed on-premises LLMs for chest X-ray reports specifically to comply with German data locality rules. Local inference handles all sensitive medical data without cloud egress or third-party processing.

In the legal domain, the requirements are even stricter:

Attorney-client privilege prohibits most external processing.
Financial documents may trigger SOX compliance issues.
EU data residency requirements under GDPR.
Industry-specific regulations (such as HIPAA and PCI-DSS).
Some legal teams pilot local LLMs to avoid content filters that could flag privileged material as inappropriate.
Performance: Latency and throughput are product features that shape user experience, SLOs, and even unit economics. Running the stack yourself removes network round-trips and lets you tailor the serving engine to your hardware and workload, unlocking lower TTFT and higher tokens/sec than one-size-fits-all APIs. As of August 2025, vLLM benchmarks report 2–4× throughput gains (https://arxiv.org/abs/2309.06180) versus prior SOTA serving systems, and up to 24× versus Hugging Face Transformers (https://blog.vllm.ai/2023/06/20/vllm.html) with continuous batching.

Performance gains from self-hosting are as follows:

Eliminate API round-trip latency (often 50–200ms).
Implement continuous batching for higher throughput.
Optimize for your specific hardware configuration.
Cache common prompt prefixes locally.
Specialized use cases: Major banks such as Westpac have partnered with Kasisto (https://www.westpac.com.au/about-westpac/media/media-releases/2022/23-august/) to deploy sector-specific LLMs for banking workflows; these deployments typically run inside the bank’s cloud perimeter.

While the motivations for self-deployment apply broadly, DeepSeek’s architecture creates unique challenges worth understanding before you provision that GPU cluster.

Some DeepSeek-specific deployment considerations for various models that you should be aware of are listed as follows:

DeepSeek-V3 memory requirements and placement: V3’s MoE design is elegant in theory: 671B total parameters, but only 37B active per token. In practice, it’s a memory management puzzle. For best performance, teams keep all experts resident in GPU memory. Offloading or weight streaming to CPU/NVMe or sharding across nodes is possible, but it incurs substantial latency and throughput penalties. For consolidated VRAM and GPU sizing, including INT4 footprint and H100 counts, see the section on hardware and inference optimization engines
Reports of 4-GPU deployments typically target smaller variants or rely on heavy offload/weight streaming to CPU/NVMe, which introduces large slowdowns.
For consolidated VRAM and GPU sizing including INT4 footprint and H100 counts, capacity planning and headroom guidance, see the section on hardware and inference optimization engines.
DeepSeek-R1 reasoning-heavy generation: R1’s reasoning chains present a different challenge. Those <think> tokens that make R1 so capable also explode memory usage and latency (long think phases, spiky memory). A simple question might generate thousands of reasoning tokens before the actual answer.

So what does this imply in real-world settings? For instance, with DeepSeek-V3’s multi-head latent attention (MLA), the KV-cache footprint is about 70 KB/token (≈0.7 GB per 10k tokens), far lower than conventional attention KV sizes but still first-order for capacity planning. R1’s public docs don’t publish a per-token KV size; memory and latency will depend on reasoning depth:

TTFT can be noticeably higher due to the thinking phase; specific TTFT depends on the workload and serving stack (no widely cited >30s benchmark).
Memory usage is unpredictable; it depends on the reasoning complexity.
Throughput is usually dominated by the thinking phase.

As of August 2025, public H200 numbers for full R1 vary by stack and workload. NVIDIA reported ~3,872 tokens/sec for 8× H200 in early NVIDIA’s Inference Microservices (NIM) previews; later Blackwell (8×) marketing claims reach ~30k tokens/sec. For your own use case, be sure to treat vendor numbers as marketing until validated on your workload, and verify current figures before hardcoding capacity assumptions. Either way, a single request might need 5,000+ tokens just for internal reasoning.

These challenges aren’t meant to discourage self-deployment. They’re meant to prepare you for it. This chapter focuses on deploying the full DeepSeek-V3 (671B) and DeepSeek-R1 model; Although smaller distilled or fine-tuned variants are useful for prototyping, they are not the focus here. The bottom line is that this does not automatically mean you should self-deploy to save money. Self-hosting pays off primarily when compliance requires it or when you have sustained, high token volumes with strong GPU utilization; otherwise, the API or a managed service is usually cheaper and simpler until your measured workload crosses your validated break-even.

The key is matching your deployment strategy to your actual needs:

Need to process sensitive documents? Self-deployment is likely mandatory.
Processing millions of tokens daily? The economics favor self-hosting.
Require sub-100ms latency? Local deployment with optimized serving is best.
Just experimenting? Start with the API and migrate when needed.

Let’s walk you through a framework to make these decisions systematically, turning deployment from an overwhelming challenge into a series of manageable trade-offs.

A decision-making framework for choosing your deployment strategy

The deployment choice primarily depends on the key drivers we discussed in the previous section: control, cost, and complexity. You will find yourself juggling and finding the right trade-off among these three.

Here are three deployment strategies you could choose from based on your use case:

The API approach minimizes complexity but comes with variable costs and less control.
A managed service provides moderate complexity with enterprise features, at the expense of higher unit costs.
A DIY (self-hosted) deployment maximizes control and achieves the best unit costs at scale, while imposing the highest operational burden.

Let’s build a framework to help you choose wisely, with a decision table (Table 8.1) keyed by compliance constraints, sustained token rate, and target SLOs.

| API | Low | Variable, pay per token | Low | Prototyping; spiky load | TTFT, p95 latency (network); provider rate limits |

Table 8.1: Choosing deployment options for DeepSeek models

Every deployment decision boils down to these three competing forces (control, cost, and complexity). In my experience, you can optimize for two, but rarely all three:

Control means the ability to tune every parameter, merge models, implement custom safety filters, or run uncensored versions for legitimate research. A company might migrate from Vertex AI to a self-hosted vLLM, which gives them the control they need, but then they have to staff a dedicated LLM SRE squad to maintain sub-300ms latency.
Cost is where the math gets interesting. DeepSeek’s API is billed per input/output token (your effective blended rate depends on the mix). Self-hosting on 4× A100 GPUs? That’s about $1.24 per million tokens at 1,600 tokens/second, but that’s assuming you keep those GPUs busy. On infra-only numbers, self-hosting can already be cheaper; a true break-even analysis only emerges once you add fixed staff/operations overhead explicitly for your organization.
Complexity is the hidden tax. Running your own inference infrastructure means hiring specialists, writing playbooks, and getting paged when your model server crashes at 3 AM. AWS’s reference architecture for DeepSeek-R1 on EKS requires deep Kubernetes expertise and GPU scheduling knowledge.

The sweet spot? Most organizations find it’s not choosing one path exclusively, but knowing when to use each. You can imagine keeping low-volume features on Vertex AI while migrating high-volume reconciliation jobs to an in-house vLLM cluster, cutting costs from 1.4¢ to 0.3¢ per token.

You can use the following decision graph(Figure 8.1) as a starting point to explore which option works best for you. You can choose the deployment path that best fits your application. We’ll go over all of them in detail shortly.

Figure 8.1: Decision graph for choosing how to deploy DeepSeek

Note

The Local/Ollama path in the decision tree is suitable only as a development proxy and does not apply to full DeepSeek-V3 (671B) or R1 deployments, which require multi-GPU or managed infrastructure.

Taken together, this framework maps each use case to the API, managed, or DIY strategy by balancing control, cost, and operational complexity. To pressure-test the option you’ve picked, run a quick cost sanity check using your own $/hour quotes and measured tokens/sec on your workload so you’re deciding from measured economics, not marketing numbers.

Cost sanity check

As of August 2025, treat $/h and tokens/sec as variable, and verify current rates before hardcoding.

For example, if your 8× H200 endpoint is quoted at ~$40/h and sustains ~3,000 tokens/sec in your measurements, infra costs ≈ $0.0037 per 1k tokens (excluding staff/ops). Compare that to provider token prices and your observed prompt/response distribution.

Unless explicitly marked as blended or amortized, per-token costs are shown as split input/output rates. Any single number is a 1:1 blended assumption or an amortized infra estimate, and is labeled accordingly. Verify current provider sheets before hardcoding prices.

Let’s put real numbers on these trade-offs:

| --- | --- | --- | --- | --- | --- | --- |

| Managed Service (Bedrock/Vertex) | $0 on-demand. $2,000+/mo commits | DeepSeekR1 on Vertex: $1.35/M input + $5.40/M output (region/model dependent) | Medium | Indicative; varies by model/region/batching.Example: ~3,872 tokens/sec on 8× H200 for R1 (NVIDIA preview). | In-region, SOC2 | Auto-scaling, global |

Table 8.2: Comparison matrix of pricing of various deployment options of DeepSeek, as of August 2025

Use a simple planning formula to model the cost for your workload:

For example, let’s plug in the numbers for the 4× A100 DIY setup from the matrix. At roughly $7.16 per hour (4 GPUs × $1.79/h) and a measured 1,600 tokens/second, the formula simplifies to an hourly calculation: $7.16 ÷ (1,600 × 3600) ≈ $1.24 per million tokens. You can now compare this directly to an API. This self-hosted cost is slightly lower than the DeepSeek-R1 API’s blended rate of ~$1.37/M tokens (at 1:1 input/output). The decision then comes down to utilization: if your workload is consistent enough to keep the GPUs busy, self-hosting provides savings; if it’s intermittent, the pay-as-you-go API will be cheaper.

So how do you make the choice?

When the API wins: You’re experimenting, have variable load, or process less than 100M tokens monthly. Zero operational overhead is worth the premium.
When local shines: You have development, testing, or sensitive data that absolutely cannot leave your premises. For example, assuming ~65 tokens/sec on a 4090 GPU, a 1:1 I/O ratio, and amortizing a $1,599 GPU over 24 months at 50% utilization, the GPU-only cost is roughly $0.8 per million tokens (excluding power and operations). Use the cost formula to adjust for your workload.
When managed services make sense: You need enterprise features (audit logs, IAM integration, and global deployment) without the operational burden. The higher per-token cost buys peace of mind.
When DIY delivers: You are processing billions of tokens monthly with a predictable load. A 4× A100 cluster processing 3B tokens/month saves ~$3,000 versus the API – enough to hire that SRE.

But the decision tree is not the holy grail. It doesn’t tell you that most successful deployments are hybrid. Recall the legal scenario we discussed at the beginning of this chapter. They might require a deployment option. You may come across three paths or choices for DeepSeek deployment:

Use the API for low-volume experimental features, or for synthetic data generation on private data.
Deploy the fine-tuned legal classifier on local GPUs for document processing.
Leverage Bedrock for client-facing applications needing five nines uptime.

The framework isn’t about picking one path forever – it’s about knowing which path serves each use case best.

Are you ready to see what each path actually looks like in practice? Let’s dive into the implementation details.

Three paths to deployment

This section details each deployment path we identified in Figure 8.1 and focuses on the concrete steps you can undertake for deployment.

Path 0: The baseline (official DeepSeek API)

The API approach offers the lowest operational overhead. Everything just works, someone else handles the maintenance, and you can focus on what you came to do. For many teams, this remains the right choice, even after considering all the alternatives.

The DeepSeek API offers two primary models: deepseek-chat (DeepSeek-V3) and deepseek-reasoner (DeepSeek-R1). As of August 2025, both expose a 128K context window. Maximum output tokens depend on API parameters (e.g., max_tokens) and provider settings. Pricing matches the public sheet. V3 runs $0.27 per million input tokens and $1.10 per million output tokens; R1 runs $0.55 per million input and $2.19 per million output.

Getting started requires minimal ceremony, as you’ve already seen in Chapter 5:

import deepseek
deepseek.api_key = "YOUR_KEY"
resp = deepseek.ChatCompletion.create(
    model="deepseek-reasoner",
    messages=[{"role": "user", "content": "Prove the Pythagorean theorem."}],
    stream=True,   # always stream – reduces perceived latency on R1
    # WHY: Stream to reduce perceived latency (TTFT) on R1; also caps client-side buffer growth.
)
for chunk in resp:
    print(chunk.choices[0].delta.content, end="", flush=True)
# WHY: Protect client from long "thinking" phases on R1; tune to your SLA (e.g., 60–120s).

This code is a simple example of using the API to get a response and to stream the results directly.

The operational reality of the API brings both pleasant surprises and hidden complexities. As of August 2025, requests can run until a ~30-minute timeout. Verify current limits before relying on them. Though not shown in the preceding code example, you might want to keep an eye on your spending for your API keys to protect budgets, especially given R1’s propensity for verbose reasoning.

One supply chain SaaS company documented its migration from OpenAI’s GPT-4o to DeepSeek, motivated by API pricing that was nearly 25× lower (https://medium.com/tracified/deepseek-vs-openai-aec06fbee43a). However, the savings came with a trade-off: the migration was not a simple drop-in and required a complete overhaul of their prompt engineering strategy. The team found that DeepSeek demanded far more explicit, detailed instructions to match the extraction accuracy they were getting from the more flexible OpenAI models.

Path 1: Local and on-premises (for dev and specialized cases)

Local deployment is where things get interesting. This path is primarily for development and prototyping; it is not applicable to full-parameter DeepSeek-V3 (671B) or DeepSeek-R1 deployments, which require multi-GPU or managed infrastructure. You trade the API’s convenience for complete control – perfect for that legal classifier handling privileged documents or the research lab exploring uncensored model behaviors.

Let’s begin with Ollama first for local deployment.

Ollama: The lowest-friction local path

Ollama has become the de facto standard for getting models running locally with minimal fuss. Installation requires one command on Linux or macOS, as follows:

curl -fsSL https://ollama.com/install.sh | sh

An MSI installer is available for Windows users, which can be downloaded at https://ollama.com/download/OllamaSetup.exe.

ollama pull deepseek-r1:14b-q4_K_M  # 14B distilled version, fits in 16GB VRAM
ollama run deepseek-r1:14b

In the preceding code snippet, we are using the 14B distilled model. The 14B model represents an interesting middle ground, balancing the ability to run it locally with the power of the responses it gives back. At Q4_K_M quantization, it occupies ~8.5 GB of VRAM and, as of August 2025, produced ~65–72 tokens/sec in informal local tests on an RTX 4090 (non-scientific; workloads vary). Not blazing fast, but entirely serviceable for development or small-scale production use. The full model catalog ranges from aggressive 3-bit quantization (~7.5 GB) for memory-constrained systems to 8-bit versions (~16.7 GB) that preserve more of the original model’s capabilities.

Docker + vLLM: Production-grade local serving

When Ollama’s simplicity hits its limits, a containerized vLLM (i.e., a vLLM run from a Docker container) offers a more sophisticated approach. This isn’t your standard Flask or FastAPI-wrapped model – vLLM brings industrial-strength optimizations such as paged attention, achieving 24× throughput improvements over naive implementations. The trade-off is complexity versus workload profile. That 24× gain reflects throughput under high concurrent load, which is a game-changer for production but often overkill for a developer running single requests locally. For low-concurrency or development workflows, Ollama’s simplicity is hard to beat; the vLLM justifies its setup overhead when you need to serve multiple users or applications from a single, shared GPU.

PagedAttention

PagedAttention is a memory management algorithm that stores the KV cache in non-contiguous memory blocks, much like an operating system uses virtual memory and paging. This solves internal memory fragmentation, enabling much higher batch sizes and ultimately delivering the throughput gains the vLLM is known for.

Launch the following script with docker run --gpus all -p 8000:8000 -v models:/models deepseek-vllm:latest. You'll get an OpenAI-compatible endpoint that speaks the same protocol as the official DeepSeek API.

FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04
RUN pip install vllm==0.8.5
ENV HF_TOKEN=<your-token>
CMD vllm serve \
     --model "deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct" \
     --dtype "bfloat16" \
     # WHY: Cap context to bound KV cache growth; raise only if you can afford memory.
     --max-model-len "8192" \
     # WHY: Push VRAM use for throughput; >0.95 risks OOM under burst. Safe range: 0.85–0.95.
     --gpu-memory-utilization "0.95"

This compatibility becomes crucial when you need to switch between local and cloud deployments without rewriting your application code.

For teams handling sensitive data, projects such as OnPrem.LLM (https://arxiv.org/html/2505.07672v2) can build complete RAG pipelines on top of these local serving layers, ensuring that proprietary documents never leave your infrastructure.

Path 2: Managed inference services (the balanced approach)

Managed services occupy the sweet spot between control and convenience. You get enterprise features (audit logs, IAM integration, and global deployment) without hiring a dedicated MLOps team. Imagine a fast-growing B2B SaaS company whose core feature is an AI-powered analytics dashboard. While the public API was perfect for their MVP, their new enterprise customers demand guaranteed uptime, lower latency, and processing within a private network. Building their own Kubernetes cluster would take months and require hiring specialized MLOps engineers. Instead, they turn to a managed service such as Amazon Bedrock or Vertex AI. This move allows them to get the required enterprise features, such as private endpoints and provisioned throughput, with a predictable cost model and minimal operational lift, letting them meet enterprise demands without derailing their product roadmap.

Each major cloud provider has embraced DeepSeek, though their approaches differ significantly. Let’s talk about them:

Amazon Bedrock (https://aws.amazon.com/bedrock) makes DeepSeek-R1 available via Bedrock (including Marketplace) with serverless invocation. Provisioned Throughput exists in Bedrock generally, but availability can depend on the specific third-party model and region – verify for your account. The serverless option works well for variable loads, while Provisioned Throughput (where available) guarantees consistent performance once you exceed sustained token rates. Their built-in guardrails service adds content filtering without custom code, which is crucial for customer-facing applications.
SageMaker JumpStart (https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html) takes a different approach, focusing on the distilled versions (8B and 32B) that fit comfortably on single-GPU instances. A recent AWS blog demonstrated serving the 8B model on ml.g5.xlarge instances. Pricing varies by region and commitment, so verify rates in your account.
Google Vertex AI (https://cloud.google.com/vertex-ai/generative-ai/docs/maas/deepseek) added DeepSeek-R1 to their Model Garden with one-click deployment. As of August 2025, representative pricing is ~$6.75 per million tokens in us-central1. Rates are region/model dependent, so verify them in your account. It’s pricier than the official API but includes enterprise SLAs and private endpoint support.
Azure AI Foundry (https://azure.microsoft.com/en-us/products/ai-foundry) rounds out the big three with both serverless and dedicated cluster options. Their responsible AI scorecard provides automated bias detection and content safety analysis – features that can save months of custom development for regulated industries.
Hugging Face Inference Endpoints (https://huggingface.co/inference-endpoints/) deserves special mention for transparency. Their deployment wizard shows line-item costs upfront, but rates vary by region/plan/provider. As of August 2025, treat ~$40/h for 8× H200 as a market reference (e.g., Together AI’s public quote is ≈$0.67/min). For Hugging Face Endpoints, automatic scale-to-zero is opt-in and, by default, triggers after ~15 minutes of idle time (historically also documented as ~1 hr). Programmatic controls exist to manually scale an endpoint to 0; do not assume a 60-second idle timeout unless your provider explicitly supports it. Verify availability and semantics for your chosen region/plan.

A practical hybrid pattern is to keep latency-sensitive, customer-facing inference on a managed platform while shifting large, non-interactive batch jobs to a dedicated endpoint to lower cost. Cost case studies on SageMaker report substantial savings; for example, Forethought cites up to 80% lower inference costs after migrating to SageMaker (https://aws.amazon.com/solutions/case-studies/forethought-technologies-case-study/).

Path 3: DIY on IaaS (for maximum control)

DIY deployment is where engineering meets economics. You’re not just running a model; you’re building infrastructure. This path demands expertise but delivers unmatched flexibility and potential cost savings. Here are some typical use cases with examples of each:

Regulated and air-gapped workloads: You must keep data inside a specific network boundary or country and cannot use third-party endpoints. For example, a hospital deploys DeepSeek behind a private API in its own VPC, with no internet egress.
Steady, high-volume traffic where cost per token matters: Your workload has predictable daily/weekly batches and keeps GPUs busy. For example, an e-commerce team runs nightly catalog enrichment and translation on a vLLM cluster, cutting unit costs versus per-token APIs and controlling batch windows.
Ultra-low latency or network locality: You need p95 latency targets that are hard to hit over public APIs. For example, a trading desk colocates an inference node next to its market-data stream, uses compiled kernels (TensorRT-LLM) and KV-cache warmers, and avoids cross-region hops.
Custom model or runtime control: You need capabilities that managed endpoints don’t expose. For example, a legal-tech platform merges domain LoRAs, enforces custom redaction filters, adjusts R1 thinking budgets, and tunes continuous batching parameters in the vLLM to match its traffic.
Multi-tenant platforms with strict isolation and budgeting: You must guarantee per-tenant data isolation and enforce token budgets across thousands of tenants. For example, a developer platform runs separate namespaces per tenant, sticky routes conversations for cache locality, and enforces token-bucket limits at the gateway.
On-premises, edge, or unreliable WAN: You can’t rely on consistent connectivity. For example, a factory runs DeepSeek on a small on-premises GPU fleet with periodic weight updates, ensuring that local inference continues if the wireless network link drops.
Vendor independence and capacity control: You want to lock in pricing/availability ahead of demand spikes. For example, a media company reserves GPUs across two regions, runs DIY for baseline load, and bursts to managed endpoints only during launches.

These patterns play to DIY’s strengths: full control over memory placement (important for V3’s MoE), the ability to cap or shape R1 reasoning tokens, predictable unit economics at scale, and deep integration with your own security and SRE practices.

Here are some of the approaches that you can try out:

Single VM or bare metal: The simplest DIY approach mirrors local deployment but with beefier hardware. A typical configuration might include 4× A100-80GB GPUs running vLLM with DeepSeek-V3 quantized to INT4. As of August 2025, measured throughput around 1,600 tokens per second yields an infra cost per million tokens below ~$2 at high utilization; validate with your workload before committing.

The math gets even more interesting with newer hardware. NVIDIA reported ~3,872 tokens/sec for R1 on 8× H200 in early NIM preview conditions; later Blackwell (8×) marketing claims reach ~30k tokens/sec ( https://blogs.nvidia.com/blog/deepseek-r1-nim-microservice/ and https://developer.nvidia.com/blog/nvidia-blackwell-delivers-world-record-deepseek-r1-inference-performance/ ). For organizations processing billions of tokens daily, the ROI can materialize quickly.

Kubernetes orchestration: Kubernetes deployment elevates DIY from running a server to operating a service. The modern approach in 2025 centers on KServe v0.15 as your serving layer, but the real story is how you think about scaling and optimization.
Instead of traditional web service patterns, LLM deployments require different mental models. You’re not scaling on requests per second; you’re scaling on actual work being done: how many requests are actively generating tokens versus waiting in the queue. Tools such as Kubernetes Event-driven Autoscaling (KEDA) enable this smarter scaling by watching real LLM metrics rather than naive request counts (https://keda.sh/).

These are the architecture choices that matter most for DIY deployment:

Smart autoscaling: Scale on meaningful metrics such as active generation load, not HTTP request rates. R1’s long reasoning phases make this especially critical.
Token budget controls: Unlike traditional rate limiting, you need to cap spending at the token level. Envoy AI Gateway and similar tools provide this control without rewriting your clients.
KV cache sharing: In conversational workloads, sharing cache across replicas dramatically improves throughput. Technologies such as LMCache make this practical without application changes.
Multi-node considerations: When models exceed single-node memory, you enter a different complexity realm. Pipeline and tensor parallelism work, but require careful planning, and they don’t currently support autoscaling.

Note that even INT4-quantized V3 needs multiple GPUs. R1’s thinking phases demand generous timeouts. For the largest models, fewer powerful nodes with fast interconnects typically outperform many smaller instances. These aren’t plug-and-play solutions, though. Each optimization requires understanding your specific workload patterns. Teams often discover that their chat workload is actually 80% single-turn questions, making conversation-optimized caching worthless, or they implement complex autoscaling only to find that their load is perfectly predictable.

Each path we’ve explored serves legitimate use cases. The API excels for prototypes and variable loads. Local deployment protects sensitive data while enabling experimentation on smaller models; full DeepSeek-V3 or R1 requires multi-GPU or managed serving. Managed services balance capability with operational simplicity. DIY infrastructure delivers ultimate control and economics at scale.

Your choice depends less on technical superiority and more on organizational reality. Do you have the expertise to run Kubernetes? Can your data leave your premises? Will you process enough tokens to justify infrastructure investment? Answer these questions honestly, and the right path becomes clear.

Next, we’ll dig into the foundational technologies that make any of these paths possible, because whether you’re calling an API or running your own cluster, understanding the underlying machinery helps you make better decisions.

The next section surveys the hardware and serving engines that underpin these choices.

Hardware and inference optimization engines for deployment

Before we start deploying models, we need to talk about the infrastructure that makes deployment possible. Let’s first discuss whether you have the right hardware for deployment and how to choose the right configuration.

Choosing your hardware

VRAM capacity determines feasibility; other components mainly affect speed. When it comes to LLM deployment, your GPU’s memory determines what’s possible, while everything else just determines how fast it happens.

The current GPU hierarchy for DeepSeek deployment looks like this: The H100 with 80 GB HBM3 sits at the apex, delivering 3.3 TB/s of memory bandwidth and native FP8 support through its Transformer Engine. The A100 (40 GB or 80 GB variants) remains the production workhorse –reliable, widely available, and still fast enough for most use cases. The L40S offers an interesting middle ground with 48 GB GDDR6 – less memory bandwidth but often easier to procure.

A 33B-parameter model in BF16 (bfloat16) precision – a 16-bit floating-point format used on modern GPUs/TPUs that stores 2 bytes per parameter and preserves FP32’s numeric range with fewer mantissa (i.e., precision) bits – requires the following:

Base memory = 33B parameters × 2 bytes/parameter = 66 GB.
Runtime overhead (~10%) = 6.6 GB.
Total VRAM needed = ~73 GB.

That’s before you add KV cache for context, activation memory for batching, or safety margins. Your 33B model that barely squeezes onto a single H100 will demand multiple GPUs once you enable production features.

And what if you opt to deploy DeepSeek-V3 with 671B parameters? Even with aggressive INT4 quantization, you’re looking at 335 GB minimum. In this case, a multi-GPU cluster isn’t optional; it’s mandatory.

Another important aspect is to utilize an inference engine. Here is why.

Inference engines

With DeepSeek, it’s not enough to write a quick model.generate() function within a simple web server. We quickly find performance bottlenecks that inference engines address.

Naïve per-request generation handles one request at a time and steps each sequence token by token before accepting a new request. Between tokens, the GPU often waits on Python overhead and kernel launches, and the KV cache is allocated in large contiguous chunks that fragment as sequences finish at different times. For example, a simple Flask or FastAPI server that calls model.generate() per HTTP request with batch_size=1 will run one user stream to completion while other users queue, leaving the GPU underutilized. Another common case is fixed batching that waits for all sequences in the batch to finish; short prompts sit idle while the longest prompt stalls the entire batch.

Moving to vLLM or TGI with continuous batching avoids these gaps. They reuse paged KV memory and insert new requests as others finish, which keeps the device saturated. In practice, this often yields 2–4× higher throughput at similar latency. Public benchmarks repeatedly show large gains versus Hugging Face Transformers servers under load (https://arxiv.org/abs/2309.06180 and

https://blog.vllm.ai/2023/06/20/vllm.html).

The naive approach treats each request in isolation. Your expensive GPU sits idle while waiting for I/O, memory fragments, and shorter sequences to finish, while longer ones block the entire batch.

Enter inference engines. Inference systems help address the wastage of GPU time, increase throughput, and reduce latency. Let’s talk about the various options you have:

vLLM (https://github.com/vllm-project/vllm): LLM introduced PagedAttention, borrowing virtual memory concepts from operating systems to manage KV cache efficiently. Instead of pre-allocating massive contiguous memory blocks, vLLM pages cache entries like a modern OS handles RAM. This meant a 2–4x throughput improvement over naive serving, with some workloads seeing up to 23x gains
The real magic of vLLM happens with continuous batching. While traditional serving waits for all sequences in a batch to complete before processing new requests, vLLM dynamically inserts new requests as others finish, keeping your GPU saturated. It’s the difference between a taxi that waits for all passengers to reach their destinations versus one that picks up new fares along the way.

Here is how you can achieve batching:

# Traditional batching pseudocode
def serve_batch(requests):
    outputs = model.generate(requests)  # All must finish
    return outputs
# Continuous batching concept
def serve_continuous():
    while True:
        if completed := get_finished_sequences():
            yield completed
            add_new_requests_to_batch()
        step_active_sequences()

Text Generation Inference (TGI) (https://github.com/huggingface/text-generation-inference): Hugging Face’s TGI powers enterprise deployments at IBM Watson and Grammarly. Its secret sauce combines Flash Attention kernels, tensor parallelism, and safetensors for near-instant model loading. Where vLLM optimizes for throughput, TGI balances throughput with consistent latency – crucial for user-facing applications.

TGI’s strengths include continuous batching, Flash Attention, PagedAttention, quantization, and tensor parallelism. It uses safetensors and optimized loaders to speed cold starts, but weights must be fully loaded before serving requests.

TensorRT-LLM (https://github.com/NVIDIA/TensorRT-LLM): NVIDIA’s TensorRT-LLM represents the closer to metal approach. It compiles your model into optimized CUDA kernels, fuses operations, and leverages hardware-specific features such as the H100’s Transformer Engine. The compilation step adds complexity, but the payoff is substantial – 4.6x higher throughput when moving from A100 FP16 to H100 FP8.

The catch is that TensorRT-LLM is pickier about model architectures and hardware. It shines brightest on NVIDIA’s latest GPUs with models it knows well. For DeepSeek’s MoE architecture, you might need to wait for official support or dive into custom plugins.

So, how do you choose your inference engine?

The choice isn’t about which engine is best; it’s about matching strengths to requirements. Here is a quick guide to help you make this decision:

Use vLLM when you need maximum flexibility and throughput.
Opt for TGI when you need production stability and consistent latency.
Go with TensorRT-LLM when you have modern NVIDIA hardware and need every last token per second.

Most teams start with vLLM or TGI for their ease of use, then evaluate TensorRT-LLM once they hit scale. The beauty? All three expose OpenAI-compatible APIs, so switching engines is often a configuration change rather than a code rewrite.

Quantization is another technique that can help you speed up your inference time and costs.

The power of quantization

Recall from Chapter 2 that quantization reduces precision to shrink memory and improve throughput, usually with a small quality loss. The quantization landscape offers several techniques, each with its own trade-offs:

FP8 represents the gentlest compression, halving memory usage while maintaining near-perfect accuracy. It requires newer hardware (H100/L40S) but delivers the best quality-to-compression ratio. Think of it as moving from lossless Free Lossless Audio Codec (FLAC) to high-bitrate MP3: technically lossy, practically indistinguishable.
GPT quantization (GPTQ) takes a post-training approach, analyzing your model to determine which weights can be squeezed to 4 bits without catastrophic forgetting. It’s fast, requiring only a calibration dataset of a few hundred examples, and typically loses less than one percentage point of accuracy. (To learn more about GPTQ, check out the Hugging Face blog here: https://huggingface.co/blog/gptq-integration.)
Activation-aware weight quantization (AWQ) goes further, identifying salient weights that disproportionately affect outputs and keeping them at higher precision. For DeepSeek’s MoE models, this activation awareness is particularly valuable – expert routing depends on precise weight values, and AWQ preserves them. (The Hugging Face documentation for AWQ is a good place to learn more and find more resources for understanding how it works: https://huggingface.co/docs/transformers/en/quantization/awq.)

A simple way to budget GPU memory is to think in three parts: the model weights, a small runtime headroom, and the KV cache:

Weights: Start with the weight tensor itself. Multiply the number of parameters by the bytes per parameter for your chosen precision. For a 33B model, BF16 stores about 2 bytes per parameter, so the weights alone are roughly 66 GB. FP8 halves that to about 33 GB. With 4-bit quantization, you store about half a byte per parameter. This means, in practice, GPTQ-4b checkpoints for 33B land around 16–17 GB each, while AWQ-4b checkpoints are often closer to 18–19 GB because they keep a few critical values at higher precision. AWQ frequently preserves slightly more accuracy than GPTQ at the same bit width.
Runtime headroom: Add about 10–20% on top of the weight size for temporary workspaces, CUDA kernel buffers, allocator fragmentation, and (for MoE) routing overhead. Using the same 33B example, a BF16 deployment might need an extra 6–13 GB beyond the 66 GB for weights before you consider sequence memory.
KV cache: Finally, budget for the KV cache, which grows with context length, generated tokens, and batch size. This is separate from the weight size. For DeepSeek-V3’s MLA, the KV cache is about 70 KB per token (roughly 0.7 GB per 10k cached tokens). Other architectures can be larger. Multiply by the number of concurrent sequences to estimate your total KV footprint.

In short, for a 33B model, expect roughly 66 GB (BF16) or ~33 GB (FP8) for the weights, or ~16–17 GB (GPTQ-4b) / ~18–19 GB (AWQ-4b), plus 10–20% runtime headroom, plus KV cache for the tokens you serve.

One important gotcha: Make sure your serving stack actually runs low-bit kernels natively. Some systems store 4-bit weights on disk but upcast them to BF16/FP16 at load or during matrix multiplies. That silently increases VRAM back toward full-precision numbers and erases the performance and cost savings. Verify that your engine (for example, vLLM, TGI, or TensorRT-LLM) is using the intended low-bit execution path by checking logs and observed VRAM after load.

The practical impact? That 33B model requiring an expensive H100 in full precision runs comfortably on a single L40S after 4-bit quantization. Your inference costs drop by 75% for a 1–2% accuracy hit – a trade most production systems gladly make.

Not all models quantize equally. Smaller models suffer more from quantization; they have less redundancy to sacrifice. Task complexity matters too: summarization might tolerate aggressive quantization while legal document analysis demands higher precision.

Always validate quantized models on your specific use case. That 0.5% average accuracy drop might hide a 10% degradation on rare but critical inputs. And remember: quantization is just virtual VRAM. You still need an inference engine that understands low-bit formats, or you’ll dequantize on the fly and lose all benefits.

Hardware provides the raw capability. Inference engines unlock that capability’s potential. Quantization stretches your hardware budget. Together, they transform DeepSeek from an impressive demo into a production-ready system.

The key insight? These aren’t independent choices. An H100’s FP8 support only matters if your inference engine can leverage it. AWQ only helps if your engine handles mixed-precision execution efficiently. Your deployment strategy must consider the full stack, not just individual components.

With the hardware and serving engines in mind, the next section demonstrates three concrete deployment scenarios.

Hands-on deployment guides

These three deployment examples progress from laptop experimentation to cloud production, each building on lessons from the previous. Follow along with the one that matches your current needs, but read all three to understand the full deployment spectrum.

Let’s first begin with the cost analysis for managed endpoints. An 8× H200 deployment costs approximately $40 per hour on managed providers, as of August 2025. Together AI publicly quotes $0.67/min (https://www.together.ai/deepseek). These costs add up quickly:

Daily cost if left running: $960.
Monthly cost at 24/7: $28,800.
Deployment spin-up time: ~10-15 minutes ≈ $7-$10.

So, what are your options to manage your costs? Here are some of the smart cost management options:

Use scale-to-zero where supported: On Hugging Face Inference Endpoints, it’s opt-in and typically triggers after ~15 minutes of idle time (though you can configure it as per your needs). Per-endpoint 60-second idle timeouts generally aren’t supported; instead, use programmatic manual scale-to-zero/pause for aggressive cost control. Check your provider’s capabilities. Since it usually takes a few minutes to spin up the model and provision infrastructure, there will usually be a bit of a delay in scaling up or down, regardless of what value you set.
Set billing alerts: Multiple thresholds are your friend. Most major providers expose budget and usage alerts (AWS Budgets/CloudWatch, Google Cloud Budgets and Alerting, Azure Cost Management, and usage notifications on managed endpoints); if yours doesn’t, enforce token-level caps in your API gateway and alert from your observability stack. As a default, set alerts at 50%, 80%, and 100% of the monthly budget (auto-throttle or pause at 100%) and add a daily speed bump alert (e.g., 10% of the monthly budget in a day) to catch runaway bursts early.
Consider your use case: Bursty workloads? Perfect for scale-to-zero. Sustained traffic? Maybe negotiate capacity blocks.
Do the math: At $40/hour, you need to be processing enough value to justify the compute. For example, if your endpoint sustains ≈3,000 tokens/sec, the infra cost is ≈$0.0037 per 1k tokens, so a 5M-token batch runs ≈$18.5 (excluding staff/ops). Conversely, if you only trickle ~100k tokens/hour, you’re effectively paying ≈$0.40 per 1k tokens (~$400/M), which is often above API rates, so pause/scale-to-zero or use serverless for that load profile.

This pricing reflects the cost of operating a fully managed 671B-parameter model with enterprise-grade infrastructure. Compare this to:

Building your own 8x H200 cluster: $500K+ upfront.
AWS capacity blocks: ~$36/hour (but you manage everything).
Smaller cloud providers: Often unavailable or unreliable.

Pro tip

For experimentation, start with shorter inference windows. A 2-hour deep dive costs $80, a manageable expense for targeted experimentation. Execute endpoint.scale_to_zero() after completion to avoid ongoing charges.

These numbers might seem intimidating, but they underscore the importance of matching the tool to the task. Personally, my workflow almost always starts with a local Ollama instance for rapid iteration, only graduating to a managed endpoint when I need to test performance at scale. This approach keeps development fast and initial costs at zero.

Example 1: Local deployment with Ollama and a quantized DeepSeek Coder model

Local deployment with Ollama represents the path of least resistance. It’s the Hello World of LLM deployment – simple enough to complete over coffee, yet sophisticated enough to power real development workflows. We’ll use the 16B parameter version quantized to 4-bit precision, which delivers near-full-model performance while fitting comfortably in consumer hardware. This example is a development proxy and does not deploy the full DeepSeek-V3 (671B) or R1 models.

For this section, our goal is to get DeepSeek Coder V2 running on your local machine in under 10 minutes, perfect for development and experimentation without cloud dependencies or API costs.

Prerequisites and performance expectations

The quantized model (DeepSeek Coder V2) weighs approximately ~9 GB. On consumer hardware, you should expect tens of tokens/second on 4-bit Coder models and a TTFT typically <1 second locally. The throughput varies by quantization, drivers, and workload. Treat any single-machine numbers in this section as directional, not SLA targets.

Let’s get started with the step-by-step deployment steps:

Installation requires just one command on Unix-like systems:
```
curl -fsSL https://ollama.com/install.sh | sh
```

Windows users get a proper MSI installer from the Ollama website (https://ollama.com/download/OllamaSetup.exe). Once installed, pulling the model is straightforward:

ollama pull deepseek-coder-v2:16b

This downloads the default Q4_0 quantization. The download shows progress in real time, and Ollama handles all the complexity of model format conversion and optimization for your hardware.

To test interactively, start an interactive session:
```
ollama run deepseek-coder-v2
```

This drops you into a REPL where you can start coding immediately. But for integration with development tools, you’ll want the API server:

ollama serve

This exposes an OpenAI-compatible REST endpoint at `http://localhost:11434`. The compatibility layer is nearly perfect; most tools expecting OpenAI’s API work with a simple base URL change.

Next is verification and testing. Confirm your deployment with a quick curl test:

curl http://localhost:11434/api/generate \
  -d '{"model":"deepseek-coder-v2","prompt":"// bubble sort in Go"}'

For Python applications, the OpenAI client library works seamlessly. Note that the OpenAI library has become a sort of standard SDK that people use to make inference requests for LLMs. As you can see in the following code sample, we pass in a configuration that specifies we want to make our requests against the local Ollama model and not the cloud-hosted OpenAI models:
```
import openai
openai.base_url = 'http://localhost:11434/v1'
openai.api_key = 'ollama'  # Any non-empty string works
response = openai.chat.completions.create(
    model='deepseek-coder-v2',
    messages=[
        {
            "role": "user", 
            "content": "Write a Rust macro for logging."
        }
    ]
)
print(response.choices[0].message.content)
```
Next, we troubleshoot common issues. The most frequent problem is running out of memory. If you see `cudaMalloc failed: out of memory`, you have three options: use a more aggressive quantization (try `deepseek-coder-v2:16b-q3_K_M`), enable CPU offloading with `OLLAMA_NUMA=1`, or close other GPU-hungry applications.
Download interruptions leave partial files that can cause cryptic errors. If you encounter `EOF` errors or `model manifest not found`, remove and repull:
```
ollama rm deepseek-coder-v2
ollama pull deepseek-coder-v2:16b
```
Disk space is another gotcha. The download process temporarily requires about 2x the model size. If your pull fails partway, you might need to run `ollama prune` to reclaim space from orphaned blob files.

Cleaning up is as simple as deployment:

ollama stop deepseek-coder-v2  # Stop running model
ollama rm deepseek-coder-v2     # Remove model files
ollama prune                    # Clean up unused data

What this teaches

This example demonstrates the fundamentals of model lifecycle management: pulling, running, serving, and cleaning up. You’ve seen how quantization makes large models practical on consumer hardware, and that Q4 quantization trades a mere 1–2 percentage points of accuracy for a 75% reduction in memory requirements.

More importantly, you now have a fully functional LLM development environment. VS Code with the Continue extension can connect to your local Ollama instance, giving you AI-powered coding assistance without sending your code to external services. It’s the perfect setup for experimentation, development, and scenarios where data privacy is paramount.

Next, we’ll scale up to managed cloud deployment, where we trade some of this simplicity for enterprise features and production reliability.

Example 2: Managed deployment on Amazon Bedrock

AWS offers multiple paths for DeepSeek deployment. As of March 2025, DeepSeek-R1 is available as a fully managed serverless model in Amazon Bedrock (https://aws.amazon.com/blogs/aws/deepseek-r1-now-available-as-a-fully-managed-serverless-model-in-amazon-bedrock/), providing the simplest deployment option with automatic scaling and pay-per-token pricing.

The goal here is to deploy any of the supported DeepSeek models (for example, R1 or V3) on Amazon Bedrock for enterprise-grade managed inference with AWS-native integration, security, and compliance features.

AWS provides three main approaches for running DeepSeek models:

Bedrock Serverless (simplest): Fully managed, no infrastructure to configure. You pay per token with no idle costs. Available for DeepSeek-R1 via the cross-region inference profile.
Bedrock Provisioned Throughput: Reserved capacity for predictable workloads. Provides guaranteed performance but requires upfront commitment. Check regional availability, as not all models support this mode.
AWS Marketplace + SageMaker: Deploy the model yourself with full control. This path requires requesting quota increases for large GPU instances (such as ml.p5en.48xlarge with H100s). You manage the infrastructure but gain maximum flexibility.

Rather than providing step-by-step instructions that will inevitably become outdated, navigate to the Amazon Bedrock console in your AWS account. The console provides the most current deployment wizards and will guide you through the following:

Model access: Some models require explicit opt-in via the Model access section.
Quota verification: Check service quotas for either Bedrock (requests per minute) or SageMaker (GPU instances), depending on your deployment path.
Guardrails configuration: Optional content filtering for production deployments.
IAM setup: Role-based access control for your applications.

For the Marketplace option, be prepared to request significant quota increases; DeepSeek-V3 and R1 require multiple high-end GPUs even when quantized.

Regardless of your deployment path, you’ll need to make several important decisions when deploying with AWS:

API choice: AWS offers multiple inference APIs. The Converse API provides a consistent interface across models with built-in token usage metrics. The InvokeModel API uses model-specific schemas but may offer lower latency. Hence, the choice depends on your application’s need for standardization versus performance.
Timeout configuration: DeepSeek-R1’s reasoning can be extensive. Thus, you need to configure client timeouts to at least 300 seconds.
Cost controls: Always set maxTokens limits. R1 can generate very long reasoning chains, potentially consuming thousands of tokens per request.
Private networking: For sensitive workloads, configure VPC endpoints to keep traffic within AWS. Even with AWS’s managed services, you’ll encounter some common challenges:
- Model access denied: Check the Bedrock console under Model access. Some models require explicit opt-in even with correct IAM permissions. Approval times vary from instant to 48 hours, depending on your account standing.
- Throttling errors: Bedrock enforces per-model rate limits (requests and tokens per minute). Implement exponential backoff with jitter.
- Quota issues: For Marketplace deployments, you’ll need substantial GPU quotas. A single `ml.p5en.48xlarge` instance costs thousands per month. Request increases well in advance; AWS may take days to approve large GPU allocations.
- Long inference times: R1’s reasoning depth varies dramatically. Simple questions might return in seconds while complex problems take minutes. Implement streaming responses for a better user experience.
Cost surprises: Without proper limits, a single runaway request can generate thousands of dollars in charges. Monitor CloudWatch metrics and set billing alerts at multiple thresholds (50%, 80%, and 100% of budget). Before production deployment, you need to validate that your chosen path meets your SLOs:
- TTFT: Critical for user experience. R1 can have a high TTFT due to reasoning phases.
- Tokens per second: Varies significantly based on deployment type and instance size.
- p95 latency: More meaningful than average latency for capacity planning
Concurrent request handling: Test at expected load levels; performance degrades non-linearly. Consider running load tests at different times of day, as shared infrastructure can have variable performance.

What this teaches

Amazon Bedrock demonstrates the cloud provider approach to LLM deployment: deep integration with existing services (IAM, CloudWatch, and VPC) at the cost of vendor lock-in and premium pricing. The serverless option eliminates operational overhead entirely – no instances to manage, no scale-to-zero to configure, just API calls and bills.

The key trade-off is control versus convenience. Bedrock serverless gives you zero control over the underlying infrastructure, but also zero responsibility. The Marketplace path gives you full control but requires expertise in GPU instance management, model optimization, and capacity planning.

For teams already invested in AWS, Bedrock’s integration advantages often outweigh the premium pricing. Native CloudWatch metrics, IAM policies that your security team already understands, and VPC endpoints that work with existing network configurations reduce the hidden costs of adoption.

The rapid evolution of AWS’s AI services means that specific features and pricing will change, but the fundamental pattern remains: AWS will continue to offer both fully managed and self-managed options, with price points that reflect the operational complexity you’re willing to accept.

Next, we’ll explore Hugging Face Inference Endpoints, which offer a middle ground between full self-management and complete abstraction.

Example 3: Deployment of DeepSeek V3 to the cloud using Hugging Face Inference Endpoints

Hugging Face Inference Endpoints allows you to control choosing your hardware and region, but Hugging Face handles the orchestration complexity. For DeepSeek-V3, this abstraction is particularly valuable; the model requires 8x NVIDIA H200 GPUs just to load, let alone serve efficiently.

Before we dive in, let’s talk numbers.

If you deploy DeepSeek yourself, be prepared to spend approximately ~$40 per hour for an 8× H200 setup on common managed providers (Together AI publicly quotes ≈$0.67/min ≈ $40.2/h); Hugging Face Inference Endpoints pricing varies by region/plan/provider. That’s about $960 per day if you leave the inference endpoint running continuously – still substantial.

Therefore, always go for a scale-to-zero feature (where supported) to manage your costs during idle periods.

To deploy via a HuggingFace API, you’ll need the following:

A Hugging Face account with API access.
An API token with write permission.
Budget approval (double-check account spending limits.)

The cleanest approach uses the Hugging Face Python SDK. Here’s the complete deployment script:

from huggingface_hub import create_inference_endpoint
# Create the endpoint with production-ready configuration
endpoint = create_inference_endpoint(
    name="deepseek-v3-prod",
    repository="deepseek-ai/DeepSeek-V3",
    framework="pytorch",
    task="text-generation",
    accelerator="gpu",
    vendor="aws",
    region="us-east-2",
    type="protected",  # Requires auth for access
    instance_size="x8",
    instance_type="nvidia-h200",

This first section defines the endpoint’s identity, model, and infrastructure: we target DeepSeek-V3 on 8x H200 GPUs in us-east-2 with a protected (auth required) endpoint. These base settings establish where the model runs and who can access it before we layer on scaling and serving details.

Auto-scaling settings control cost and availability: min_replica=0 enables scale-to-zero when idle (where supported), while max_replica=1 caps spend during initial testing. Increase these later if you need higher concurrency:

# Auto-scaling configuration
    min_replica=0,  # Enable automatic scaletozero when idle (if supported)
    max_replica=1,  # Single replica for cost control
    # Note: HF Endpoints typically trigger scaletozero after ~15 min of idle; perendpoint 60 s timeouts are not generally supported.

We pin a TGI container and tune serving limits (prefill, input, and total tokens) to keep large model requests within memory while maximizing throughput. MODEL_ID points the container at the mounted repository for weight loading:

# Custom TGI container for optimized serving
    custom_image={
        "health_route": "/health",
        "url": "ghcr.io/huggingface/text-generation-inference:3.3.1",
        "env": {
            # TGI optimizations for large models
            "MAX_BATCH_PREFILL_TOKENS": "16384",
            "MAX_INPUT_LENGTH": "32768",
            "MAX_TOTAL_TOKENS": "65536",
            "MODEL_ID": "/repository"
        }
    }
)

Deployment typically takes 10–15 minutes as images pull and weights load into GPUs; wait() provides a clean blocking flow with a bounded timeout. On failure, fetch() helps you inspect partial states for quick debugging.

To monitor deployment progress, you can do the following:

print(f"Initial status: {endpoint.status}")
print("Waiting for deployment (this typically takes 10-15 minutes)...")
try:
    endpoint.wait(timeout=1200)  # 20 minute timeout
    print(f"✓ Deployment successful!")
    print(f"Endpoint URL: {endpoint.url}")
    print(f"Status: {endpoint.status}")
except Exception as e:
    print(f"✓ Deployment failed: {e}")
    # Check if partial deployment occurred
    endpoint.fetch()
    print(f"Current status: {endpoint.status}")

For CI/CD pipelines or environments where Python isn’t available, the REST API works equally well to help you monitor your deployment :

#!/bin/bash
# Configuration
HF_TOKEN="${HF_TOKEN:-your-token-here}"
HF_USERNAME="${HF_USERNAME:-your-username}"
ENDPOINT_NAME="deepseek-v3-prod"

These variables supply your organization’s context and credentials to the Hugging Face Endpoints API. Ensure that HF_TOKEN has write scope and matches the target account/space.

This request creates the endpoint and mirrors the Python SDK fields: model, serving image, token limits, hardware, and scaling. Some fields are provider-specific (e.g., scaling semantics), so omit or adjust them if your region or plan doesn’t support a given option:

# Deploy via curl with proper error handling
# Note: some scaling fields (e.g., "scaleToZeroTimeout") are provider-specific; omit them if unsupported.
RESPONSE=$(curl -s -w "\n%{http_code}" \
  "https://api.endpoints.huggingface.cloud/v2/endpoint/${HF_USERNAME}" \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${HF_TOKEN}" \
  -d '{
    "name": "'"${ENDPOINT_NAME}"'",
    "type": "protected",

    "model": {
      "repository": "deepseek-ai/DeepSeek-V3",
      "framework": "pytorch",
      "task": "text-generation",
      "fromCatalog": false,

      "image": {
        "tgi": {
          "url": "ghcr.io/huggingface/text-generation-inference:3.3.1",
          "healthRoute": "/health"
        }
      },

      "env": {
        "MAX_BATCH_PREFILL_TOKENS": "16384",
        "MAX_INPUT_LENGTH": "32768",
        "MAX_TOTAL_TOKENS": "65536"
      }
    },

    "compute": {
      "accelerator": "gpu",
      "instanceType": "nvidia-h200",
      "instanceSize": "x8",

      "scaling": {
        "minReplica": 0,
        "maxReplica": 1,
        "metric": "hardwareUsage",
        "measure": {
          "hardwareUsage": 80
        }
      }
    },

    "provider": {
      "vendor": "aws",
      "region": "us-east-2"
    }
  }')

We parse the HTTP status and JSON body to surface a clear success/failure signal in CI logs. A 200 code indicates the deployment was accepted; otherwise, the error payload helps pinpoint missing access, quota, or invalid field issues:

# Parse response
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
BODY=$(echo "$RESPONSE" | sed '$d')
if [ "$HTTP_CODE" -eq 200 ]; then
    echo "✓ Deployment initiated successfully"
    echo "$BODY" | jq -r '.status'
else
    echo "✗ Deployment failed with HTTP $HTTP_CODE"
    echo "$BODY" | jq -r '.error // .'
fi

For teams using ZenML for MLOps orchestration, the deployment integrates seamlessly into existing pipelines:

from zenml import pipeline
from zenml.config import DockerSettings
from zenml.integrations.constants import HUGGINGFACE
from zenml.integrations.huggingface.services import HuggingFaceServiceConfig
from zenml.integrations.huggingface.steps import (
    huggingface_model_deployer_step,
)

ZenML packages the Hugging Face integration and container settings so your deployment runs reproducibly in pipelines. This keeps endpoint creation, retries, and metadata tracking under a single, versioned workflow:

docker_settings = DockerSettings(
    required_integrations=[HUGGINGFACE],
)
@pipeline(enable_cache=True, settings={"docker": docker_settings})
def deepseek_deployment_pipeline(
    model_name: str = "deepseek-v3",
    timeout: int = 1200,
):
    # Configure with production settings
    service_config = HuggingFaceServiceConfig(
        model_name=model_name,
        endpoint_name="deepseek-v3-prod",  # Gets zenml- prefix + UUID
        repository="deepseek-ai/DeepSeek-V3",
        framework="pytorch",
        task="text-generation",
        accelerator="gpu",
        instance_type="nvidia-h200",
        instance_size="x8",
        vendor="aws",
        region="us-east-2",
        endpoint_type="protected",
        min_replica=0,
        max_replica=1,
        custom_image={
            "health_route": "/health",
            "url": "ghcr.io/huggingface/text-generation-inference:3.3.1"
        }
    )
    # Deploy with automatic retry logic
    deployment = huggingface_model_deployer_step(
        service_config=service_config,
        timeout=timeout,
    )

    return deployment

The service_config mirrors the SDK parameters (model, hardware, region, scaling, and image) and becomes the source of truth for the endpoint spec. Treat it like infrastructure as code; review diffs before promoting to production.

# Run the pipeline
if __name__ == "__main__":
    deepseek_deployment_pipeline()

This step triggers the actual endpoint creation with built-in retry and returns a service handle you can inspect later. Downstream steps can poll the handle for status, URL, and health to gate release checks:

Verification and first inference: Once deployed, verify that the endpoint is accessible and perform a test inference:

# Test the deployed endpoint
def test_endpoint(endpoint):
    """Verify endpoint functionality with a simple prompt."""
    try:
        response = endpoint.client.text_generation(
            "Explain containerization in three sentences.",
            max_new_tokens=100,
            temperature=0.7
        )
        print(f"✓ Inference successful: {response[:50]}...")
        return True
    except Exception as e:
        print(f"✗ Inference failed: {e}")
        return False
# For production use
def create_production_client(endpoint_name: str):
    """Create a client for production inference."""
    from huggingface_hub import get_inference_endpoint

    endpoint = get_inference_endpoint(endpoint_name)
    if endpoint.status == "scaledToZero":
        print("Endpoint is scaled to zero, waking up...")
        # First request will trigger wake-up

    return endpoint.client

Monitoring and cost management: The scale-to-zero mechanism helps, but note that on Hugging Face Inference Endpoints, it’s opt-in and typically kicks in after ~15 minutes of idle time. Monitor it closely:

# Check endpoint status and costs
def monitor_endpoint(endpoint_name: str):
    from huggingface_hub import get_inference_endpoint

    endpoint = get_inference_endpoint(endpoint_name)
    endpoint.fetch()  # Refresh status

    print(f"Status: {endpoint.status}")
    print(f"URL: {endpoint.url}")
    print(f"Created: {endpoint.created_at}")
    print(f"Updated: {endpoint.updated_at}")

    # Estimate daily cost based on status
    hourly_cost = 40  # $40/hour for 8x H200 replica
    if endpoint.status == "running":
        print(f"Running cost: ${hourly_cost}/hour (${hourly_cost * 24}/day)")
    elif endpoint.status == "scaledToZero":
        print("✓ Scaled to zero - no compute charges")

Teardown and cleanup: When you’re done, clean up to avoid surprise bills:

# Graceful shutdown
endpoint.scale_to_zero()  # Immediate scale down
# or
endpoint.pause()  # Complete pause, requires manual resume
# or
endpoint.delete()  # Permanent deletion - careful!

What this teaches

This deployment showcases several production patterns:

Hardware abstraction: You specified the hardware (8x H200) without managing the underlying infrastructure.
Cost optimization: Scale-to-zero prevents runaway bills during idle periods.
Container customization: The TGI container provides optimizations specific to large model serving.
Security by default: Protected endpoints require authentication, preventing unauthorized access.

The trade-offs are clear: at $40/hour, Hugging Face’s managed infrastructure is competitive with AWS capacity blocks (~$36/hour) and significantly cheaper than on-demand pricing (~$65-70/hour). You’re getting automatic scaling, monitoring, and updates without the DevOps overhead.

Some of the operational considerations you should be aware of are the following:

Cold starts matter: After scaling to zero, the first request takes 2–3 minutes as the model loads into GPU memory. Design your application accordingly.
Regional availability: H200s aren’t available in all regions. Check availability before committing to a specific geography.
Batch for efficiency: With such expensive hardware, batch your requests when possible. The TGI server handles concurrent requests efficiently.
Monitor token usage: At this scale, even efficient models can generate costs quickly. Implement token limits in your application logic.

These examples get you up and running. Sustaining reliability, performance, and cost requires operational practices (monitoring, scaling, and governance), which we will address next.

Production operations and monitoring

The model is deployed. The first requests are flowing. Now comes the hard part: keeping it running reliably, efficiently, and securely at scale. This section distills the operational wisdom that separates proofs-of-concept from production systems.

Monitoring and observability

LLMs run on costly GPUs; visibility is essential for reliability and spend control. The metrics that matter fall into four categories:

Token counts directly translate to costs. Track both prompt and completion tokens per request. Most providers return this in response metadata: OpenAI-compatible APIs expose prompt_tokens and completion_tokens usage; Amazon Bedrock Converse reports inputTokenCount/outputTokenCount; and Vertex AI includes usageMetadata.promptTokenCount/outputTokenCount. For self-hosted engines (vLLM/TGI), expose usage and Prometheus metrics from the server (token counters for prompt/generated tokens), and optionally validate by counting with the model’s tokenizer on the client side. This isn’t just for billing; sudden spikes often reveal inefficient prompts or, worse, prompt injection attempts. When you see a token spike, first ask a simple question: Did the prompt get bigger, or did the answer get longer? A step change in prompt tokens usually means a template or RAG change (more retrieved passages, different chunking), whereas a surge in completion tokens often points to jailbreaks or runaway reasoning where the model keeps thinking and hits length limits. If total tokens per minute soar while tokens per request look normal across many users, you’re probably seeing a usage surge; if the spike comes from a small cohort with duplicate requests and more, it’s a retry loop or client bug. Treat the spike like a diff: compare prompts before and after, enable prefix caching if common headers grew, cap max tokens to tame long answers, and tighten retrieval settings if inputs ballooned.
Latency distributions tell you how users experience your service. For DeepSeek-R1, TTFT can vary wildly depending on how deeply the model decides to reason, making p95 latency a better SLO metric than averages.
GPU utilization deserves special attention. When vLLM reports that KV cache usage is approaching capacity, you’re about to hit a performance cliff. The model doesn’t gracefully degrade; it falls off a cliff. Think of throughput as a highway and the KV cache as your lanes; as the cache fills and fragments, cars still enter, but everything slows at once – tokens per second drop, queues lengthen, and both TTFT and p95 rise together. That synchronized jump is the cliff: you’ve crossed a capacity edge where evictions and smaller effective batches cause non-linear collapse. The fastest way back from the brink is to shrink what each request demands (lower max context or total tokens), reduce concurrent sequences, or temporarily shed low-priority traffic; the durable fix is more headroom (larger VRAM or cache), better batching, and higher cache hit rates via shared prefixes or KV reuse.

Use vLLM’s Prometheus metrics to track cache pressure and throughput (e.g., tokens/sec, vllm:num_requests_running, running vs. waiting requests) and alert before KV cache utilization approaches your capacity threshold (e.g., ~90%).

The fourth category is cost tracking. Join your token metrics with current pricing tables and calculate dollars per request in real time. When that number jumps 50% week-over-week, something changed – maybe a new prompt template, or maybe a regression in your caching layer.

For production deployments, vLLM’s /metrics endpoint provides Prometheus-compatible observability out of the box. Export these to your time-series database of choice. Add a Prometheus scrape_config (or a Kubernetes ServiceMonitor) that targets http://YOUR_VLLM_HOST:8000/metrics and, if you use an external backend, enable remote_write to your time-series database (e.g., Thanos, Cortex, or VictoriaMetrics) so you can visualize the metrics in Grafana. Grafana dashboards include pre-built visualizations specifically for vLLM deployments.

Ultimately, metrics only tell you what happened, not why. For root cause analysis, you need request-level visibility. Tools such as Langfuse provide automatic Personally Identifiable Information (PII) redaction while maintaining enough context for debugging those inevitable the model said what? incidents.

Scaling and performance

DeepSeek models present a fascinating scaling challenge. Unlike traditional web services, where you can just spin up more replicas, LLM scaling requires understanding the interplay between model architecture, hardware topology, and request patterns.

The naive approach (treating your model service like any other stateless application) fails spectacularly. Here’s why: every request builds up KV cache state that can be reused for similar prompts. Randomly distributing requests across replicas destroys this cache locality. Implement sticky session routing based on user or conversation ID. This simple change can improve throughput by 40% in conversational workloads.

For DeepSeek-V3’s MoE, only a small subset of experts (typically two per layer) is active per token. In practice, many serving stacks keep all experts resident across the serving pool to minimize cross-node routing; however, this is not a hard requirement. You can shard experts across GPUs/nodes (expert parallelism) or offload some experts to CPU/NVMe, trading additional all-to-all communication and higher latency for lower per-node memory.

Consequently, horizontal scaling can take two forms: replicas that duplicate the full 671B parameter set to increase concurrency, or capacity scaling via expert-parallel sharding that grows the total expert pool without duplicating all weights on every machine.

Choose based on your latency budget and interconnect; with slower interconnects, fewer, larger nodes with NVLink/NVSwitch often outperform many small instances. Operators can enforce this manually by provisioning larger NVLink/NVSwitch nodes and constraining placement (e.g., Kubernetes node pools with nodeSelector/affinity and taints/tolerations, or explicit GPU pinning on bare metal) while limiting horizontal autoscaling for that service.

Performance optimization in production is an exercise in finding bottlenecks and eliminating them systematically. Here is how you can achieve optimization:

Start with continuous batching: Traditional serving processes requests sequentially; continuous batching dynamically groups requests to maximize GPU utilization. A key parameter here is --max-num-seqs, which controls the maximum number of sequences (requests) vLLM can process concurrently. Setting this value correctly is a critical balancing act between throughput and memory stability. Too low, and you leave performance on the table during traffic spikes; too high, and you risk catastrophic out-of-memory (OOM) errors.

The ideal value depends directly on your available VRAM after the model weights have been loaded. You can estimate a safe starting point with the following logic:

Let’s consider two practical scenarios:

Scenario 1: High-throughput chatbot: You’re running a 33B model (quantized to INT4, ~19 GB) on a single 80 GB H100 GPU. Your workload consists of many concurrent users with short conversation turns (average sequence length of 512 tokens):
- VRAM allocation: 80 GB total – 19 GB for the model ≈ 61 GB available. Reserving 10% for overhead leaves ~55 GB for the KV cache.
- KV cache per sequence: For a model of this size, the cache is roughly 25 KB/token. So, 512 tokens * 25 KB/token ≈ 12.8 MB per sequence.
- Calculation: 55,000 MB / 12.8 MB ≈ 4,296. This theoretical maximum is unrealistic due to memory fragmentation.
- Practical value: A safe starting value for --max-num-seqs would be 256 (a common default). After load testing and monitoring KV cache usage under real traffic, you might carefully increase it toward 512 or 1024 to maximize throughput.
Scenario 2: Long-context document analysis: You’re serving a 70B model (BF16, ~140 GB) on a 4x A100-80 GB node (320 GB total VRAM). The workload involves fewer concurrent requests but with very long documents (average sequence length of 16,384 tokens):
- VRAM allocation: 320 GB total – 140 GB for the model ≈ 180 GB available. Reserving 10% for overhead leaves ~162 GB for the KV cache.
- KV cache per sequence: For a 70B model, the cache is roughly 80 KB/token. So, 16,384 tokens * 80 KB/token ≈ 1.31 GB per sequence.
- Calculation: 162 GB / 1.31 GB ≈ 123 sequences.
- Practical value: Given the high memory pressure per request, starting with a much more conservative --max-num-seqs value of 32 is prudent. Based on stability during testing, you might increase it to 64, but exceeding this would significantly raise the risk of OOM errors for this workload.

The best practice is to calculate a conservative baseline, then empirically tune the value upward while monitoring your GPU memory and KV cache usage via vLLM’s Prometheus metrics endpoint.

Caching is your secret weapon: You can implement a two-tier strategy. The first tier, KV caching, is a fundamental part of how Transformer models generate text and is automatically handled by modern inference engines such as vLLM. There isn’t a flag to turn on KV caching; it’s the core process that prevents the model from recomputing calculations for every token in the context window each time a new token is generated. You manage its memory impact indirectly through parameters such as context length.

The second tier is explicit prompt prefix caching, which you can enable for workloads with repetitive starting text. In vLLM, you activate this feature by launching the server with the --enable-prefix-caching flag. When enabled, vLLM automatically detects when multiple requests share a common prefix (such as a system prompt or a set of few-shot examples). It computes the KV state for that prefix only once, on the first request, and then stores it. All subsequent requests that begin with that exact same prefix reuse the cached state, skipping the expensive recomputation. This is how system prompts and templates can be cached indefinitely, significantly speeding up the prefill stage for common tasks. Cache hit rates above 60% typically reduce latency by a third and can cut costs proportionally.

Thinking costs: For R1 models, you face a unique decision: how much thinking is too much? In latency-sensitive applications, consider implementing reasoning truncation. Monitor the distribution of reasoning token counts and set limits based on your SLAs. Yes, you might sacrifice some accuracy, but users prefer fast, good-enough responses over perfect answers that arrive too late.

With scaling, caching, and thinking bounded, the next step is to make these legible in actual dollar or euro amounts: how tokens, cache hit rate, and GPU utilization translate into cost per request and cost per user.

Cost management

LLM costs have a nasty habit of exponential growth. Unlike traditional infrastructure, where costs scale linearly with traffic, LLM costs can explode based on prompt design, model selection, or user behavior patterns. A single prompt template change can triple your bills overnight.

Here is a pragmatic cost management approach you can try:

Start with request-level budgeting. Every request should carry a cost estimate based on expected token usage. Reject requests exceeding per-user or per-endpoint budgets before they hit the model. This isn’t just about preventing abuse; it’s about catching configuration errors before they drain your bank account.

To apply this, implement the check in your API gateway or in a middleware layer before the request hits your inference service. For each incoming request, calculate max_potential_cost by multiplying the max_tokens parameter by the cost per output token, and add the cost of the input tokens. Store per-user or per-key daily/monthly budgets in a fast-access database such as Redis. The middleware then performs a simple check:

if
(user_current_spend + max_potential_cost > user_budget) { return 429_BUDGET_EXCEEDED; }

This precomputation step is cheap and effectively ring-fences your model from costly requests.

Infrastructure efficiency requires a different lens. Kubecost’s GPU module reveals the brutal truth: that 8-GPU cluster you provisioned might be sitting 40% idle. At $2.48 per H100-hour, idle time becomes a luxury you can’t afford. But don’t just look at averages – graph utilization over time. Predictable patterns (low usage nights and weekends) suggest you need autoscaling. Random spikes might indicate batch job scheduling issues.
Practically, you can achieve this on Kubernetes by using the NVIDIA DCGM Exporter (https://github.com/NVIDIA/dcgm-exporter) to expose GPU metrics to Prometheus. Then, create a Grafana dashboard with a panel showing avg(dcgm_gpu_utilization) by (pod). For autoscaling, configure KEDA with a Prometheus scaler. Your ScaledObject definition would target a metric such as sum(rate(vllm_requests_running[2m])), allowing you to scale your replicas based on actual inference load rather than just CPU or memory, which are often poor proxies for GPU work.
The most insidious cost creep comes from model drift. You deploy a nice, efficient 33B parameter model. Six months later, you’re somehow running the 70B variant in production, and nobody remembers why. Track cost-per-output-token by model version and alert on changes. A 20% week-over-week increase deserves investigation; 50% demands immediate action.
To implement this, ensure that every inference log includes model_id, input_tokens, and completion_tokens. In your logging platform (such as Datadog or the ELK stack), create a metric by enriching these logs with a price lookup table (e.g., deepseek-v3_input_cost = $0.27/M_tokens). You can then build a dashboard that visualizes sum(request_cost) / count(requests) grouped by model_id. Set up an automated monitor that compares the weekly average of this metric to the previous week and triggers an alert if the percentage change exceeds a defined threshold.
Budget alerts should escalate geometrically. Set them at 50%, 80%, and 100% of your budget. The 50% alert is informational. With this alert, its time for you to check whether growth is expected. The 80% alert pages on-call is an indication that something needs attention. The 100% alert should trigger automatic throttling. Better to degrade service than wake up to a six-figure cloud bill.
You can set this up using your cloud provider’s native tools. In AWS, for example, create an AWS budget with three alert thresholds. The 50% action can be an SNS notification to a Slack channel for visibility. The 80% action can be an SNS notification that triggers a PagerDuty or Opsgenie webhook. For the 100% action, have SNS trigger a Lambda function that programmatically modifies your API Gateway usage plan to a highly restricted throttle limit (e.g., one request per minute) or updates a feature flag in AWS AppConfig to disable the service gracefully.

With spend guardrails in place (budgets, alerts, and automated throttles), the next question is resilience: how do we prevent those same controls from being bypassed by abuse, prompt injection, or data exfiltration? The good news is that the primitives you just set up (gateways, usage plans, and feature flags) double as enforcement points for security.

Security in practice

LLMs introduce security challenges that traditional application security playbooks don’t address. Some of the challenges you may encounter are as follows:

Prompt injection is a real thing that actually happens.
Rate limiting by request count prevents the user from sending 100k-token prompts.
Standard authentication might not account for the cost differential between users asking for haikus versus dissertations.

Network security remains foundational but insufficient. Therefore, you need to deploy models in private subnets, front them with API gateways, and use mTLS between services (i.e., all the standard practices of network security apply). But you need to add LLM-specific controls too:

Input validation must check not just for SQL injection but also for prompt injection patterns. Validate against an allowlist first: enforce strict schemas (expected JSON keys/types), length and charset caps, and reject control tokens. Then, run inputs through a prompt injection filter that combines simple rules (regex for phrases such as ignore previous instructions, reveal system prompt, jailbreak markers, and Unicode homoglyphs) with an embedding similarity check against known attack examples; block or challenge high-risk requests and emit tagged logs to your centralized log store.
Output filtering needs to catch not just PII but also jailbreak attempts.
The unique challenge with DeepSeek models is their capability. A model smart enough to reason through complex problems is smart enough to be creatively misused. Implement semantic similarity checks on prompts, comparing against known jailbreak patterns. Common patterns include instruction overrides (ignore previous instructions, reveal/repeat the system prompt), Do Anything Now (DAN)/developer mode role play, translation/encoding wrappers (Base64/ROT13), JSON/Markdown fence escapes, and for research/simulation requests that elicit disallowed content. Seed your similarity index with public references such as OWASP’s LLM Top 10 examples (https://owasp.org/www-project-top-10-for-large-language-model-applications/), PromptBench attack sets (https://github.com/microsoft/promptbench), and sample jailbreaks from NVIDIA NeMo Guardrails (https://github.com/NVIDIA/NeMo-Guardrails). But don’t rely solely on blocklists; they’re always one creative prompt behind. Instead, implement defense in depth: rate limit by token count, not just request count. Monitor for unusual token consumption patterns and track prompt complexity metrics.
For sensitive deployments, consider the full kill chain. An attacker might not directly compromise your model but could use it to probe your infrastructure. Monitor for prompts asking about system configuration, internal details, or error messages. Implement server-side prompt inspection by routing all requests through an API gateway or middleware that logs prompts and runs a lightweight classifier (regex rules or a local LLM) to flag “recon” intents such as asking for system prompts, environment variables, stack traces, file paths, or IPs. Forward these tagged events to your centralized log store (e.g., CloudWatch, Datadog, or Splunk) with alerts on high-risk matches or spikes per user/session, and pair with token-based rate limits plus automatic challenge/deny rules for repeated hits. These reconnaissance attempts often precede more serious attacks.

CI/CD for models

Model deployment isn’t a one-time event – it’s an ongoing process. New versions promise better performance, security patches demand immediate deployment, and that fine-tuned variant finally proves its worth in testing. Without proper CI/CD, model updates become high-stress events that everyone avoids.

Here is an action plan that can get you started:

Start with versioning discipline. Even though DeepSeek publishes model versions, what you actually ship is a composite artifact; it’s the base model revision plus your quantization, tokenizer, adapters/LoRAs, prompt templates, decoding limits, safety filters, inference engine/container, and config. Version so you can reproduce behavior, roll back safely, satisfy audits, and A/B test changes; the vendor model version alone will not explain behavioral or cost deltas in production. All of which is to say: models aren’t just files – they’re artifacts with dependencies, configurations, and behavioral contracts. Use a model registry (MLflow or cloud-native options) that tracks not just the weights but the entire deployment context. When something breaks, you need to know exactly what changed.

To put this into practice, adopt a concrete versioning scheme for your composite artifact that is both human-readable and machine-parseable – for example, [base-model-version]-[quantization]-[adapters-hash]-[config-version], which might look like deepseek-v3-b1.0-int4-lora-abc123-v2. Here, abc123 could be the short Git commit hash from the repository where your LoRA was trained, and v2 refers to the version of your decoding configuration file. This creates an unambiguous identifier for the exact artifact running in production.

A model registry, such as MLflow, is where you bring this all together. Your CI pipeline, triggered by a commit, should package these components and log them to a single MLflow run. Use mlflow.log_artifact() to store your tokenizer configuration, prompt template files, and safety filter scripts. Use mlflow.log_param() to record KV metadata such as quantization_method: AWQ or base_model_id: "deepseek-ai/DeepSeek-V3".
Finally, register that entire run as a new version of a model named, for instance, "production-chat-endpoint". Now, when you compare version 1.2.0 to 1.3.0 in the MLflow UI, you see not just a new model file but a precise diff of every dependency and configuration change that could explain a shift in behavior or cost.
Blue-green deployments work beautifully for models. Spin up the new version alongside the old, validate with synthetic traffic, then gradually shift real users. The key insight is, don’t just test functional correctness. Monitor latency, token usage, and cost metrics during the canary phase. A model that’s 2% more accurate but 50% more expensive might not be an upgrade.

In a Kubernetes environment, you can implement this with a service mesh such as Istio or Linkerd, or an API gateway such as Emissary-ingress. The pattern involves creating two Kubernetes Deployments: one for the old model version (e.g., deepseek-v1) and one for the new canary version (deepseek-v2). A single Kubernetes Service object targets both deployments. You then configure a routing rule (such as an Istio VirtualService) to initially direct 100% of traffic to v1:

Your CI/CD pipeline (e.g., using GitHub Actions or GitLab CI) automates the traffic shift. The first step in the rollout job applies a new routing rule that sends just 1% of traffic to v2. The pipeline then pauses and queries your observability platform (e.g., Prometheus) for key metrics, comparing the v1 and v2 deployments side by side on a Grafana dashboard. If the p95 latency, token usage, and calculated cost-per-request for v2 remain within acceptable limits for a set period (say, 15 minutes), the pipeline automatically proceeds to increase the traffic weight to 10%, then 50%, and finally 100%, completing the rollout.
Rollback capability isn’t optional. Keep the previous model version warm and ready. When metrics go sideways, you need one-click rollback, not a frantic redeployment process. This means accepting the infrastructure cost of running two versions temporarily. Consider it insurance.

This one-click rollback is the emergency brake in your deployment pipeline. It’s not a new deployment; it’s a configuration change that instantly reroutes traffic. If your canary monitoring dashboard shows a critical metric regression, such as a spike in harmful content generation or a sudden doubling of latency, you trigger the rollback. In your CI/CD tool, this should be a manually triggered job that executes a single kubectl apply or API call to your service mesh, updating the traffic routing rule to send 100% of traffic back to the stable, previously running version. The entire rollback process should take seconds, not minutes.

Keeping the previous version warm means the Kubernetes Deployment for the old model maintains its full replica count throughout the canary phase. It’s actively running and ready to instantly absorb 100% of the production load. The insurance cost is tangible: if a single model replica costs you $5 per hour and you run two replicas for high availability, your cost temporarily increases from $10/hour to $20/hour during the rollout. Once the new version is fully promoted and deemed stable (e.g., after 24 hours), your pipeline’s final step should automatically scale the old deployment’s replica count down to zero, eliminating the extra cost.
The testing pyramid for models looks different from traditional software. Unit tests verify prompt formatting and response parsing. Integration tests confirm that the model loads and responds. But the critical layer is behavioral testing: does this model maintain the safety boundaries of the previous version? Does it handle edge cases consistently? Build a golden dataset of prompts and expected behaviors, and gate deployments on regression tests.

Practically, these tests should run as stages in your CI pipeline. Unit tests are fast checks run on every commit. Using a framework such as pytest, you can write a simple function to verify a prompt template: assert "User:" in format_prompt(history, query). Another test can confirm that your response parser handles expected formats and raises specific errors for malformed JSON: pytest.raises(MalformedJsonError): parse_response("{'invalid': 'json'}"). These tests catch logic errors before a model is even loaded.

Integration tests run after the container is built. Your pipeline spins up the model container and uses a tool such as curl to hit its health check endpoint and then its inference endpoint with a simple "hello" prompt. The test just needs to assert a 200 OK status and a valid response structure. This catches issues with dependencies, model loading, or environment configuration.
Behavioral regression testing is the final and most important gate before deployment. Your pipeline deploys the new model to a staging environment and runs a test suite against it. This suite iterates through your “golden dataset” (e.g., a test_cases.jsonl file in your Git repo). For each prompt, it asserts specific behaviors. For safety, you might assert that a harmful prompt results in a refusal: assert "cannot comply" in response.lower(). For consistency on critical tasks such as data extraction, you can compare the LLM’s JSON output against a predefined schema. For more nuanced qualities such as tone or helpfulness, you can even use a smaller, cheaper judge LLM to evaluate the new model’s response and flag any significant negative changes from the previous version’s baseline response. A failure at this stage should automatically block the production rollout.

With monitoring, scaling, cost control, security, and CI/CD in place, your endpoint transitions from fragile demo to durable service. We will conclude with a compact playbook.

Your deployment playbook

You now have the complete deployment stack: from local development proxies to multi-GPU production clusters. The decision framework boils down to three constraints: data sensitivity, token volume, and operational capacity.

Here is your playbook:

Quick reference for common scenarios:
- Prototyping and development: Use the DeepSeek API until you hit rate limits or cost concerns. Switch to local Ollama with smaller models for offline work. Don’t optimize infrastructure before validating your use case.
- Production with sensitive data: Start with managed services (Bedrock/Vertex) if they meet compliance requirements. Self-host only when managed options are legally or technically insufficient. Budget 2–3 months for the self-hosting learning curve.
- High-volume production (>1B tokens/month): The math favors self-hosting, but only with dedicated DevOps resources. Run cost analysis, including staff time, not just GPU hours. Consider hybrid deployments: API for variable loads, self-hosted for baseline traffic.
Common pitfalls to avoid:
- Deploying full DeepSeek-V3 without capacity planning (see the section on hardware and inference optimization engines for sizing guidance).
- Ignoring scale-to-zero capabilities (difference between $40/hour and $0/hour when idle).
- Treating R1’s reasoning tokens as free (they count toward cost and latency).
- Deploying without monitoring token usage (bills can increase 10x overnight from a prompt change).
Implementation checklist:
- Implement request-level cost tracking.
- Set budget alerts at 50%, 80%, and 100% thresholds
- Cache system prompts and common prefixes (60%+ hit rate is achievable).
- Use sticky routing for conversational workloads.
- Keep the previous model version warm for instant rollback.

The ecosystem evolves rapidly. Today’s 8×H200 requirement may be tomorrow’s single-GPU deployment. Focus on building abstractions that survive infrastructure changes: standardize on OpenAI-compatible APIs, version all deployments, and maintain environment parity from development to production.

Here are some tips for engineers that may come in handy.

For junior engineers: Start with Example 1 (Ollama) in the Hands-on deployment guides section. Get comfortable with the request/response cycle and token economics before tackling cloud deployments.
For senior engineers: Jump to Example 3 (Hugging Face Inference Endpoints) for production deployment, but review the monitoring section. LLM observability differs from traditional services.
For platform teams: Focus on the inference engine comparison and production operations guidance will accelerate your infrastructure decisions.

Remember

Deployment is a means to an end. Pick the simplest approach that meets your requirements, then iterate as you learn more about your actual usage patterns.

Summary

This chapter turned deployment from theory into a pragmatic playbook for running full-parameter DeepSeek-V3/R1 in production. You mapped the landscape (API vs. managed vs. DIY) with a control/cost/complexity lens, learned how to size hardware (VRAM, KV cache, TTFT/throughput), and chose inference engines deliberately (vLLM/TGI/TensorRT-LLM) with quantization options (FP8, GPTQ, and AWQ) to fit performance and budget. Three concrete paths anchored the guidance: local dev proxies (Ollama) for iteration, managed services (Bedrock/Vertex/Azure/Hugging Face) for enterprise reliability, and DIY (vLLM on GPUs/Kubernetes) for maximum control and economics at scale.

You then operationalized the service: monitor tokens/latency/GPU and KV pressure, scale on active generation with continuous batching and sticky routing, cache shared prefixes, and enforce token-based budgets with scale-to-zero and budget alerts. Security moved beyond network basics to prompt injection defenses and output filtering. Finally, you treated models as versioned, composite artifacts and shipped safely with blue-green/canary rollouts and instant rollback. With this toolkit, you can pick the simplest deployment that meets your constraints today and evolve it confidently as your workload grows.

Get This Book’s PDF Version and Exclusive Extras

Scan the QR code (or go to packtpub.com/unlock). Search for this book by name, confirm the edition, and then follow the steps on the page. | |

| Note: Keep your invoice handy. Purchases made directly from Packt don’t require one. |

9 Epilogue

The journey through this book traced the path from DeepSeek’s foundational principles to its deployment in production-grade systems. We began with its disruptive market entry and concluded with the operational realities of running it at scale. This epilogue synthesizes that journey, connecting the core technical innovations to the practical skills required to leverage them.

The DeepSeek paradigm

DeepSeek’s release marked a pivotal moment, demonstrating that an open-source model could achieve frontier performance in reasoning, directly challenging the dominance of proprietary systems. This was not an incremental improvement but a paradigm shift driven by a convergence of architectural and training innovations. The book first explored these technical foundations.

At its core, the Mixture-of-Experts (MoE) architecture enabled massive scale (over 600 billion parameters) while maintaining computational efficiency by activating only a fraction of the model per token. This was paired with Multi-Head Latent Attention (MLA), which compresses the KV cache to make long-context understanding practical and memory-efficient.

The most significant breakthrough, however, was the training methodology. By prioritizing reinforcement learning with automated, rule-based rewards - encapsulated in techniques like Group Relative Policy Optimization (GRPO) - DeepSeek learned to generate structured, step-by-step reasoning without extensive supervised fine-tuning. This ability to think before answering became the model’s defining characteristic and the central theme of its practical application.

From theory to practice

Understanding the architecture was only the first step. The subsequent chapters transitioned from what DeepSeek is to how to use it effectively. This required unlearning established prompting habits. We saw that DeepSeek’s reasoning models behave less like chatbots and more like theorem provers, where minimalist, context-rich prompts outperform the detailed, few-shot examples that benefit other models.

Armed with this new mental model, we built a complete application, demonstrating the full spectrum of deployment from local CPU inference to managed cloud services. This practical exercise highlighted the critical importance of structured outputs for creating reliable and flexible systems. We then extended this to autonomous systems, exploring agentic patterns that decompose complex tasks, iteratively refine outputs, and interact with external tools.

The core lesson - DeepSeek is a powerful reasoning engine that, when properly harnessed, can automate complex analysis and decision-making.

The path to production

The final chapters addressed the ultimate goal: deploying DeepSeek in real-world business environments. We explored two distinct and powerful strategies for achieving this.

First, the rationale distillation workflow showed how to use DeepSeek-R1 not as the final product, but as a teacher. By generating explanatory rationales for a specialized domain like legal contract analysis, we created an enhanced dataset to train a smaller, more efficient student model. The result was a deployable model that was not only private and cost-effective but also significantly more accurate on its specialized task than its far larger teacher.

Second, for use cases requiring the full power of the complete models, we detailed the deployment playbook for DeepSeek V3 and R1. This involved navigating the critical trade-offs between cost, control, and complexity across APIs, managed services, and self-hosted infrastructure. From selecting the right GPU hardware and inference engine to implementing robust monitoring and security, we established a framework for making sound operational decisions.

The principles and patterns detailed in this book are durable. DeepSeek represents more than a single family of models; it provides a blueprint for an era of AI defined by powerful, accessible, and adaptable reasoning. The journey through its architecture, application, and deployment has equipped you with the essential skills to build the next generation of intelligent systems.

Appendix

It’s time to understand the different ways we can use the DeepSeek models.

This appendix is skippable if you have already figured out how you want to use DeepSeek.

But if not, we’ll show you some ways.

We will go over some of the (at the time of writing) most popular ways of running some of the main models released by the Chinese AI lab.

We will cover the following ways to use DeepSeek:

Getting started with the official DeepSeek API
Using common third-party APIs
Working with Cursor’s IDE for DeepSeek
Running or deploying DeepSeek yourself
Building your own setup for DeepSeek

Technical requirements

The following will need to be installed:

Recall from Chapter 5 that we will be using uv for Python examples when working with the DeepSeek API. If you don’t have it installed yet, you can install uv by following the instructions here: https://docs.astral.sh/uv/getting-started/installation/.
To run the terminal-based examples, we will use curl. To install curl, you can check the official website at https://curl.se/; alternatively, the https://everything.curl.dev/install/index.html website shows installation instructions for all major platforms.
You will need to set up API keys to follow some of the examples, but we will walk through how to set them up in each example.
All the code examples are properly documented here: https://github.com/PacktPublishing/DeepSeek-in-Practice.

Note

While we have used macOS, all of the code examples and instructions should work on all platforms. For Windows users, we advise using Windows Subsystem for Linux (https://learn.microsoft.com/en-us/windows/wsl/install).

Getting started with the official DeepSeek API

Like most AI labs and companies, DeepSeek has both a ChatGPT-like web application, hosted at https://chat.deepseek.com/, and an API platform, hosted at https://platform.deepseek.com/. This setup is common for most companies (including open source projects such as Qwen: https://chat.qwen.ai/). Notably, after the release of DeepSeek-R1 in January 2025, the API platform was suddenly hit by a massive cyberattack (https://www.cnbc.com/2025/01/27/deepseek-hit-with-large-scale-cyberattack-says-its-limiting-registrations.html). The platform has since gotten back up and running and is stable and a good option for those wanting to get started with building with DeepSeek.

The goal of this section is to give you a short introduction to DeepSeek’s official platform and API. Keep in mind that the DeepSeek API is a paid and (hopefully) continuously developing product. Therefore, we will give you a small introduction to the API platform as it stands around June 2025. This will likely change over time. So, if something looks different, consider reading the API documentation at https://api-docs.deepseek.com/. This is not meant to be a comprehensive guide - just an overview that highlights the main features. Without further ado, let’s get set up.

Setting up

Here, we’ll show you how to get started with building with the DeepSeek API. To get set up, here are the steps you need to follow:

Go to the platform website over at https://platform.deepseek.com/.
Create an account. Optionally, log in with Google.
Navigate to Top up on the left panel.
Add some credits to your account using your preferred payment method.
Navigate to API keys.
Click Create new API key.
Save your API key (it will normally start with sk-XXXX).

Now that we have an API key, let’s run a small test to ensure everything is working correctly. If you are on Mac or Linux, open a terminal and type the following, replacing <DeepSeek API Key> with the correct value:

curl https://api.deepseek.com/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <DeepSeek API Key>" \
  -d '{
        "model": "deepseek-chat",
        "messages": [
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "What is the capital of Portugal?"}
        ],
        "stream": false
      }'

You should see an output similar to the following:

{"id":"3dbb972a-80e1-4558-b310-deeaeac1968d","object":"chat.completion","created":1749810137,"model":"deepseek-chat","choices":[{"index":0,"message":{"role":"assistant","content":"The capital of Portugal is **Lisbon** (Portuguese: *Lisboa*). It is the largest city in the country and is known for its historic neighborhoods, vibrant culture, and scenic location along the Atlantic Ocean."},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":16,"completion_tokens":44,"total_tokens":60,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":16},"system_fingerprint":"fp_8802369eaa_prod0425fp8"}

If you don’t, then make sure you have followed all the steps and set up your billing and API key correctly.

To make our first API call in Python, follow these steps:

Create a script called 01-deepssek-api-call.py.
Export your DeepSeek API key by running export DEEPSEEK_API_KEY=sk-XXXXXX (or use some sort of environment manager, such as direnv (https://direnv.net/)).
Copy the following contents to the script file:

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "openai",
# ]
# ///
from openai import OpenAI
import os
API_KEY = os.getenv("DEEPSEEK_API_KEY")
assert API_KEY, "Please set the DEEPSEEK_API_KEY environment variable."
BASE_URL = "https://api.deepseek.com"
client = OpenAI(api_key=API_KEY, base_url=BASE_URL)
response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "What is the capital of Portugal?"},
    ],
    stream=False,
)
print(response)
# ChatCompletion(id='1bdf1ecf-a365-4f8b-9c26-fbebff421127', choices=[Choice(finish_reason='stop', index=0,
# logprobs=None, message=ChatCompletionMessage(content='The capital of Portugal is **Lisbon** ...

The nice thing about using uv is that we don’t need to deal with installing and setting up Python versions. All you need to do now is run the script:

$ export DEEPSEEK_API_KEY=sk-XXXXXX
$ uv run 01-deepseek-api-call.py

You can expect an output similar to the following:

Reading inline script metadata from `01-deepseek-api-call.py`
ChatCompletion(id='41164ddd-c648-4945-96c2-67f64e6d62a1', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='The capital of Portugal is **Lisbon** (Portuguese: *Lisboa*). It is the largest city in the country and is known for its historic neighborhoods, vibrant culture, and scenic location along the Atlantic Ocean and the Tagus River.', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None))], created=1749375653, model='deepseek-chat', object='chat.completion', service_tier=None, system_fingerprint='fp_8802369eaa_prod0425fp8', usage=CompletionUsage(completion_tokens=49, prompt_tokens=15, total_tokens=64, completion_tokens_details=None, prompt_tokens_details=PromptTokensDetails(audio_tokens=None, cached_tokens=0), prompt_cache_hit_tokens=0, prompt_cache_miss_tokens=15))

You just made your first API call to DeepSeek using Python!

Like many other large language model (LLM) providers, the DeepSeek API is compatible with the OpenAI API format. This means that you can use the OpenAI Python SDK (https://github.com/openai/openai-python), for example, and simply switch the api-key and base-url keyword arguments, and you don’t need to care about the rest of the inner workings. This is great because it gives users the ability to switch providers relatively easily - without having to completely refactor their code bases. Beware, though: not all features of the API will be available with every provider. So, make sure to test things thoroughly.

Let’s now look at the response structure. When you make a call to the DeepSeek API, you’ll receive a JSON object in return, which follows the OpenAI-compatible format:

ChatCompletion(
  id='41164ddd-c648-4945-96c2-67f64e6d62a1',
  object='chat.completion',
  created=1749375653,
  model='deepseek-chat',
  system_fingerprint='fp_8802369eaa_prod0425fp8',
  choices=[
    Choice(
      index=0,
      finish_reason='stop',
      logprobs=None,
      message=ChatCompletionMessage(
        role='assistant',
        content='The capital of Portugal is **Lisbon** (Portuguese: *Lisboa*)...',
        refusal=None,
        annotations=None,
        audio=None,
        function_call=None,
        tool_calls=None
      )
    )
  ],
  usage=CompletionUsage(
    prompt_tokens=15,
    completion_tokens=49,
    total_tokens=64,
    completion_tokens_details=None,
    prompt_tokens_details=PromptTokensDetails(
      audio_tokens=None,
      cached_tokens=0
    ),
    prompt_cache_hit_tokens=0,
    prompt_cache_miss_tokens=15
  ),
  service_tier=None
)

The key components of the response object here are the following:

choices: A list of chat completion choices, each with a finish reason and a message. These can also contain tool calls; more on this later.
id: The unique request ID.
model: The model that was used - we will get into this later.
system_fingerprint: The backend configuration that the model runs with.
usage: Usage statistics you might care about (the number of completion tokens, caching statistics, etc.).

Now that we have a better idea of how to call the DeepSeek API, let’s understand a bit more about which models are available to us and how to leverage them.

Using the available models

The DeepSeek AI lab has released a significant number of models. You might have already noticed from the code examples that our initial API calls were made to a model called deepseek-chat. If this doesn’t ring a bell, it’s because that’s not the name of the actual model! In their API, DeepSeek only makes available two endpoints and two models:

deepseek-chat: This is where the company makes their non-reasoning models available. At the time of this writing, the model under this endpoint is DeepSeek-V3.2-Exp in non-thinking mode.
deepseek-reasoner: This is where the company releases their notable reasoning models. These are models that have been trained on or are capable of reasoning. At the time of this writing, this endpoint points to DeepSeek-V3.2-Exp in thinking mode.

It’s worth pointing out that by the time you are reading this, the company might have released more models. As a consequence, the underlying models of these two endpoints might have changed. Always check the DeepSeek API documentation (https://api-docs.deepseek.com/) for the most up-to-date information about the models.

It is now time to look at the factors that may impact your DeepSeek API usage.

Temperature

The temperature parameter modifies the distribution of softmax probabilities over the next-token logits produced by the models. Specifically, it scales logits before applying a softmax function. If you are curious about the topic, I recommend this great blog post by Luke Salamone (https://blog.lukesalamone.com/posts/what-is-temperature/). As a result, lower temperatures are more deterministic, and higher temperatures less so. OpenAI, for example, also makes available a seed parameter to try to make outputs more deterministic. However, if context grows, even if we set the temperature to 0 (for very deterministic outputs), we are likely to see changes.

DeepSeek recommends that users use different temperature values according to their needs (https://api-docs.deepseek.com/quick_start/parameter_settings). For example, a higher temperature may be set for more creative tasks.

My advice is that if you are integrating into a larger - ideally deterministic - system, always set the temperature to 0.

Pricing and rate limits

Like most LLM providers, DeepSeek charges on a per-token basis. This means that users pay for token inputs and outputs. For reference, one character in English ≈ 0.3 tokens, while a Chinese character ≈ 0.6 tokens. DeepSeek uses their own tokenizer and also makes a package available (https://api-docs.deepseek.com/quick_start/token_usage#calculate-token-usage-offline) that can be used to count tokens.

The DeepSeek API is well known to have very competitive pricing compared to larger AI companies, so it’s definitely an option to consider when selecting a provider. It’s also worth noting that deepseek-reasoner is approximately 2x the price of its non-reasoning counterpart - this is standard for LLMs.

Also note that reasoning models output many more tokens. As you get charged on a per-token basis, reasoning models are effectively more expensive. Also note that DeepSeek (like most providers) also gives you access to prompt caching - caching a part of the prompt to make requests faster and cheaper.

Unlike other providers out there, DeepSeek does not rate limit users’ requests. This is very out of line with most providers. LLM APIs are notoriously unstable and, therefore - due to high demand - providers tend to rate limit API calls pretty strongly. The following is noted in DeepSeek’s API documentation:

“We will try out best to serve every request. However, please note that when our servers are under high traffic pressure, your requests may take some time to receive a response from the server.”

Contrary to other providers, rather than throwing RateLimitError, or something similar, DeepSeek will simply hold your request until it can be completed. For non-streaming requests, they will send empty lines, and for streaming requests, they will continuously send : keep-alive events. A good alternative is to retry the request to ensure your API call is reliable:

from tenacity import retry, stop_after_attempt, wait_fixed
@retry(stop=stop_after_attempt(3), wait=wait_fixed(2))
def make_request() -> str:
    response = client.chat.completions.create(
        model="deepseek-chat",
        messages=[
            {"role": "system", "content": "You are a helpful assistant"},
            {"role": "user", "content": "What is the capital of Portugal?"},
        ],
        stream=False,
        max_tokens=100,
    )
    response_text = response.choices[0].message.content
    if not response_text:
        raise ValueError("Received empty response from the API.")
    return response_text

The preceding example will retry the make_request function up to three times, waiting two seconds between each attempt, if an exception is raised during execution (e.g., due to a failed request or an empty response). If the function succeeds without raising an exception, it returns the API response text. If all attempts fail, the last exception is raised.

Let’s now shift our focus to the features of the DeepSeek API.

API features

In this final section about the DeepSeek API, we’ll briefly cover some of the most interesting (current) features that it offers. We don’t want to deep dive specifically into all of them, but we would like to give you a good initial grasp of each one. We will talk about reasoning, streaming, JSON output, function calling, and fill-in-the-middle (FIM) features.

Reasoning

The deepseek-reasoner model/endpoint has the capability to reason about a certain topic and return its reasoning traces. This can be very useful for more complicated topics, but it might also be overkill for others.

Here’s how you can use it:

model = "deepseek-reasoner"
messages = [
    {"role": "user", "content": "What is the population of Copenhagen in 2030?"}
]
response = CLIENT.chat.completions.create(
    model=model, messages=messages, temperature=TEMPERATURE
)
reasoning_content = response.choices[0].message.reasoning_content
content = response.choices[0].message.content
print("Reasoning Content:")
print(reasoning_content)
# Okay, the user is asking about Copenhagen's population in 2030...
print("\nFinal Answer:")
print(content)
# The projected population for **Copenhagen Municipality ...

Streaming

No user likes to wait around with a blank screen for the computer to answer a question. Streaming responses is a great way to make your LLM response seem snappy and responsive to the user, without actually accelerating it:

model = "deepseek-chat"
messages = [{"role": "user", "content": "What is the second largest city in Portugal?"}]
response = CLIENT.chat.completions.create(
    model=model,
    messages=messages,
    stream=True,  # this is important!
    temperature=TEMPERATURE
)
for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
# prints: The second largest city in Portugal is \*\*Porto\*\* ..

Now the user will see the responses as they come.

JSON output

JSON structured output is a great way to enforce structured output on an LLM and ensure it integrates well with the rest of an application. This is especially useful since LLMs don’t work in silos and traditionally should integrate with other applications.

Here’s how to ask DeepSeek for a structured JSON response:

model = "deepseek-chat"
messages = [
    {
        "role": "system",
        "content": "Extract a JSON response with the keys 'name', 'age', and 'city'.",
    },
    {"role": "user", "content": "Duarte is 31 years old and lives in Copenhagen."},
]
response = CLIENT.chat.completions.create(
    model=model,
    messages=messages,
    response_format={"type": "json_object"},
    temperature=TEMPERATURE,
)
json_object = json.loads(response.choices[0].message.content) # convert to json object
print("JSON Response:")
print(json_object)
# {'name': 'Duarte', 'age': 31, 'city': 'Copenhagen'}
print(json_object["name"])
# Duarte

The json.loads() method can help you structure the LLM response as a JSON object.

Function calling

When we want to give the capability to the LLM to interact with the outside world (call an API, scrape a web page, or interact with another system), function calls are a great way to do that. We have already covered this topic in Chapter 6 (in the Tools section).

FIM

FIM completion allows us to define a prefix and a suffix on the model responses. Defining a prefix and a suffix allows the model to generate content specifically for the gap between them. For code, for example, you can provide a function, a loop, or a method where the beginning and end are already defined - effectively asking the model to fill only the content in the middle. This is how automated autocompletions with GitHub Copilot (https://github.com/features/copilot) effectively work! For a simple example, let’s try to make the DeepSeek model give us the number of people who live in Lisbon. We can provide an input and an output and let the model fill in the rest:

prompt = "The population of Lisbon is exactly "
suffix = " million people."
response = BETA_CLIENT.completions.create(
    model="deepseek-chat",
    prompt=prompt,
    suffix=suffix,
    max_tokens=5,
    temperature=TEMPERATURE,
)
final_text = prompt + response.choices[0].text + suffix
print(final_text)
# The population of Lisbon is exactly 2,000,000 million people.

This is, of course, not limited to code. You can have models complete JSON, XML, or anything else, preventing them from going off track and generating text you might not be expecting.

Note that at the time of writing, FIM is only available in the Beta API (you need to set base_url=https://api.deepseek.com/beta).

You can also use DeepSeek models with third-party APIs. Let’s take a look at how to use DeepSeek models with third-party providers.

Using common third-party APIs

As previously mentioned, all of DeepSeek’s models are open source. Due to this, DeepSeek has proliferated across many different cloud providers: Google Cloud Platform, Azure, AWS, Fireworks, OpenRouter, and Cloudflare, to name a few.

Talking about all of them would be overkill, so we will highlight the three main ones in this section:

Cloudflare (https://www.cloudflare.com/): A large provider that is globally available. Users familiar with frontend technologies might already know about Cloudflare. They are also a “middle-of-the-range” provider due to their size.
AWS (https://aws.amazon.com/): Many users will be familiar with AWS or work in companies already leveraging it, hence we will focus on AWS for the sake of brevity. Azure and Google Cloud Platform are other popular options. The reason we selected AWS is due to its popularity with enterprise users.
OpenRouter (https://openrouter.ai/): OpenRouter is a relatively new cloud provider, but it is extremely focused on serving LLMs. Start-ups or new projects might be interested in a provider that can also give them the flexibility to easily switch models as they get released.

Let’s start with Cloudflare.

Cloudflare

Cloudflare is an American company famous for their content-delivery network (CDN) products. Currently, Cloudflare makes access to LLMs easy through their Workers AI product. Here are a few details you should know about Cloudflare before we get started:

Supported models: deepseek-r1-distill-qwen-32b, a distilled version of DeepSeek-R1 released in January 2025 (https://arxiv.org/pdf/2501.12948), and deepseek-math-7b-instruct, an instruction-tuned model specialized in math released in April 2024 (https://arxiv.org/pdf/2402.03300).
Pricing: $0.50 per million input tokens; $4.88 per million output tokens for the Qwen 32B distilled version. No pricing information for the math-7b version.
Region: Cloudflare Workers AI runs globally at the edge.
Authentication: Can be done through a Cloudflare auth token.
OpenAI-compatible endpoints: Supported at /v1/chat/completions and /v1/embeddings.

Here’s how you can set up Cloudflare Workers AI:

Go to cloudflare.com.
Log in or create an account.
On the left menu, click on AI and select Workers AI (Figure A).

Figure A: The DeepSeek models supported by Cloudflare

Click on {} Rest API.
Click on Create a Workers AI API Token.
Save your account ID and token in a safe place.

Let’s look at some sample API calls.

Here is an example of sending a request with curl. We start by exporting our environment variables and then issuing a request in the following format:

$ export CLOUDFLARE_ACCOUNT_ID=******
$ export CLOUDFLARE_AUTH_TOKEN=****
$ curl https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/run/@cf/deepseek-ai/deepseek-r1-distill-qwen-32b -X POST -H "Authorization: Bearer $CLOUDFLARE_AUTH_TOKEN" -d '{ "messages": [{ "role": "system", "content": "You are a friendly assistant" }, { "role": "user", "content": "Why is pizza so good" }]}'

If you want to use Python, here’s an example Python call using the OpenAI-compatible endpoint provided by Cloudflare:

We start by setting our keys and defining our OpenAI client:

from openai import OpenAI
# setup keys and client
api_key = os.environ["CLOUDFLARE_AUTH_TOKEN"]
account_id = os.environ["CLOUDFLARE_ACCOUNT_ID"]
assert account_id, "Please set the CLOUDFLARE_ACCOUNT_ID environment variable."
assert api_key, "Please set the CLOUDFLARE_API_KEY environment variable."
model = "@cf/deepseek-ai/deepseek-r1-distill-qwen-32b"
client = OpenAI(
    base_url=f"https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1",
    api_key=api_key,
)

We then create a list of messages:

# create messages
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the most likely sky color in Copenhagen?"},
]

Finally, we issue a request to the @cf/deepseek-ai/deepseek-r1-distill-qwen-32b model:

# make api call
response = client.chat.completions.create(
    model=model,
    messages=messages,
    max_tokens=1000,
    temperature=0.0,
)

The request includes both the thinking and the final response of the model in the content parameter. So, we use Python to extract the thinking portion, which is between the <think> tags:

# extract thinking content and final response text
raw_response = str(response.choices[0].message.content)
start = raw_response.index("<think>") + len("<think>")
end = raw_response.index("</think>")
thinking_content = raw_response[start:end]
response_text = raw_response[end + len("</think>") :].strip()
print("=" * 10, "Cloudflare Thinking", "=" * 10)
print(thinking_content)
# Okay, so I need to figure out what the most likely ...
print("=" * 10, "Cloudflare Response", "=" * 10)
print(response_text)
# The most likely sky color in Copenhagen is gray.

Now it’s time for some quirks and tips while using Cloudflare.

Quirks and tips

Function calling is not supported by any of the models that we tested in the Cloudflare platform.
The format when calling the reasoning model does not break the reasoning content into a different reasoning component of the API. You’ll notice that the thinking steps are included in an xml section inside two <think> tokens.
Cloudflare does not support any of the models supported by the official DeepSeek API. Also important to note is that none of the models supported by Cloudflare are available in the official DeepSeek API.

Cloudflare is a good choice if you are looking for a straightforward provider that offers simple access to models. If you are already using Cloudflare, then it’s a great choice. However, do note that the model availability, especially DeepSeek models, is limited.

Let’s now look at AWS and how to use it.

AWS

If you’ve worked in technology, chances are that you’ve heard of Amazon Web Services, or AWS. They are currently the largest cloud provider (https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/) in the world. AWS is known for quickly expanding and adopting new technologies in their cloud offering. Most of their AI products are offered under the Bedrock brand.

Here are some quick facts about using the AWS API for DeepSeek models in AWS:

Supported models:
- As a serverless deployment (runs without having to deploy it yourself), you can use the DeepSeek-R1 model. This might evolve in the future, so check https://aws.amazon.com/bedrock/deepseek/ for up-to-date information.
- You can also use most of the DeepSeek-R1 distilled models and the newest DeepSeek-R1-0528 model through Bedrock Marketplace. This means that you will need to manage the deployment yourself - and pay for it (even when not using it).
Pricing (on-demand): Please consult the AWS documentation at https://aws.amazon.com/bedrock/pricing/ for up-to-date information on pricing. Keep in mind that serverless models are normally cheaper - since you only pay for the requests you make to the model. On the other hand, deploying models from Bedrock Marketplace means that you need to pay for the infrastructure yourself. We recommend going with the serverless offering as a starting point.
Region: DeepSeek-R1 is available via a serverless offering in US East (N. Virginia), US East (Ohio), and US West (Oregon). Bedrock Marketplace models can be deployed in any AWS Region.
Authentication: You can use your AWS account for IAM-based authentication and Bedrock service integration (https://aws.amazon.com/iam/).
OpenAI-compatible endpoints: Not available out of the box, but it can be used with some open source projects from AWS (http://github.com/aws-samples/bedrock-access-gateway).

Figure B: List of available DeepSeek models in the AWS Bedrock model catalog

Let’s get set up:

Create or set up an AWS account if you don’t have one: https://signin.aws.amazon.com/signup.
In the AWS console, search for Billing and Cost Management and open the first result to set up billing.
Make sure you are in the correct Region - us-east-1 is a good default choice since most models are available there first.
Search for Amazon Bedrock in the search bar at the top and go to the Amazon Bedrock service.
In the right-hand panel, go to Model Catalog under Foundation models.
Use the Provider filter and select DeepSeek.
Click on the model you’re interested in (you may need to enable access in the console first; do this by clicking on a model in the console and then Modify Access, and then request access).
Install the AWS CLI using the guide here: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html.
(Recommended) Configure the AWS CLI: https://docs.aws.amazon.com/cli/.

Once you’ve installed the AWS CLI, you can make a request to a model by running the following:

 --model-id us.deepseek.r1-v1:0 \
        --messages '[{"role": "user", "content": [{"text": "What is the population of Ancona?"}]}]' \
        --region us-east-1

We can also use Python and the boto3 library to make a simple request to our model. The text and thinking content are separated by default from AWS. So, we can access both fields independently. Also notice that the Bedrock runtime (brt) has a slightly different API format:

import boto3
brt = boto3.client("bedrock-runtime")
model_id = "us.deepseek.r1-v1:0"
# create our messages
conversation = [
    {
        "role": "user",
        "content": [{"text": "What is the most likely sky color in Copenhagen?"}],
    }
]
# request
response = brt.converse(
    modelId=model_id,
    messages=conversation,
    inferenceConfig={"maxTokens": 5000, "temperature": 0.0},
)
response_text = response["output"]["message"]["content"][0]["text"]
repsonse_reasoning = response["output"]["message"]["content"][1]["reasoningContent"][
    "reasoningText"
]["text"]
print("=" * 10, "AWS Thinking", "=" * 10)
print(repsonse_reasoning)
# Okay, so I need to figure out the ..
print("=" * 10, "AWS Response", "=" * 10)
print(response_text)
# The most likely sky color in Copenhagen is **pale gray or overcast white**...

Quirks and tips

The AWS interface for invoking models can be confusing. There is an invoke and a converse API. Read more about their differences here: https://dgallitelli95.medium.com/amazon-bedrock-explained-with-memes-converse-api-and-tool-usage-w-anthropic-claude-3-001c341347ca.
The documentation is very extensive. The following is a recommended read to learn about tool calling, prompt caching, compute usage, and more: https://dgallitelli95.medium.com/amazon-bedrock-explained-with-memes-converse-api-and-tool-usage-w-anthropic-claude-3-001c341347ca.
If you are making many calls (for classification or other “batch” use cases), consider using AWS Batch inference. It’s a great service to run models at a discounted price by trading off response time.
You might need to tweak your permissions in the AWS IAM console to invoke models within AWS.
Not all models are available in all Regions - check the AWS console to see what’s available where.
Most programmatic usage is done with the boto3 Python library. For Bedrock, note the distinction between the bedrock and bedrock-runtime APIs:
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock.html
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime.html

AWS is an ever-evolving platform, adding support for models very quickly - and giving you possibilities to deploy models yourself (at a cost, of course). It can be overwhelming navigating the console and seeing all the possibilities, but it’s a powerful tool to have under your belt.

Let’s talk about a simpler, newer provider, OpenRouter, next.

OpenRouter

OpenRouter (https://openrouter.ai/) is a platform that acts as a universal gateway for accessing LLMs from various providers using a simple unified interface. More interestingly, for the case of open source models, OpenRouter will aggregate multiple providers under the same API, allowing users to query the best-performing provider. For example, for DeepSeek-V3, they support Deep Infra, Novita AI, Nebius AI Studio, and Fireworks as providers.

Here are some facts you should know about the OpenRouter API for DeepSeek:

Supported models: At the time of writing, DeepSeek-R1, DeepSeek-R1-0528, DeepSeek-V2, DeepSeek-V3, and additional distilled or fine-tuned models.

Figure C: Multiple providers for the same DeepSeek-V3 model

Pricing (on demand): OpenRouter directs your request to the “best available providers” of the model you are requesting. They charge you when you purchase credits on the platform (5% + 0.35 USD fee). They also provide a wide range of free models (such as DeepSeek-R1-0528).
Region: It does not support “regions” per se. It routes your request to the provider directly. Therefore, you should consider it a “global” region.
Authentication: It uses an API key that you need to generate through the OpenRouter console.
OpenAI-compatible endpoints: All endpoints from OpenRouter are compatible with the OpenAI format and SDK.

Here’s how to get set up:

Go to https://openrouter.ai.
Log in or create an account.
Hover over the hamburger menu on the top right, and click Credits.
Add credits to your account.
Hover over the hamburger menu on the top right again and click API Keys.
Generate an API key and save it.

Here’s how you can make an API call using curl:

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -d '{
  "model": "deepseek/deepseek-r1-0528",
  "messages": [
    {
      "role": "user",
      "content": "What is the most likely sky color in Copenhagen?"
    }
  ]

}'

As you can see, we define our list of messages and send them to the necessary endpoint.

Using Python, the request is very similar to other requests to the OpenAI SDK client:

from openai import OpenAI
client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)
response = client.chat.completions.create(
    model="deepseek/deepseek-r1-0528",
    messages=[
        {"role": "user", "content": "What is the most likely sky color in Copenhagen?"}
    ],
)
response_reasoning = response.choices[0].message.reasoning
response_text = response.choices[0].message.content
print("=" * 10, "OpenRouter Thinking", "=" * 10)
print(response_reasoning)
# Okay, the user asked about the most likely sky col...
print("=" * 10, "OpenRouter Response", "=" * 10)
print(response_text)

You will notice that all we need to do is override the base_url parameter of the OpenAI client.

You should get an output similar to this:

# The **most statistically likely overall sky color in Copenha...

Here are some tips and specifics when you use OpenRouter.

Quirks and tips

Free models: OpenRouter gives access to a large suite of models completely free of charge for you to test out. Simply visit their model dashboard (or https://openrouter.ai/models?max_price=0) and start making API calls! (This is not advised for production.)
You can append :nitro to a provider to access the provider with the best throughput, or :floor for the provider with the best price. For example, to access the fastest DeepSeek-R1, query deepseek/deepseek-r1-0528:nitro.
OpenRouter supports all the main features you might expect (tool calling, images and PDFs, structured outputs, and prompt caching on some providers). Visit their docs for implementation specifics: https://openrouter.ai/docs/quickstart.
Keep in mind that you might receive different results between providers since OpenRouter does not control the hardware where the actual models run.
OpenRouter provides a ranking page over at https://openrouter.ai/rankings where you can see which models are being used for which categories, and get a sense of what is trending. For example, at the time of writing, DeepSeek-R1-0528 was ranked #2 in the Finance category.

Working with Cursor’s IDE for DeepSeek

We will demonstrate DeepSeek’s capabilities using Cursor’s IDE, a powerful AI-enhanced development environment that seamlessly integrates with DeepSeek’s models. However, the principles and techniques we explore are adaptable to any development environment that supports DeepSeek integration, whether you prefer traditional IDEs, cloud-based platforms, or local web interfaces. The essential element is establishing a robust connection between your development workflow and DeepSeek-R1’s advanced reasoning capabilities.

With the API access confirmed, we can proceed to establish the development environment that will serve as our laboratory for exploring DeepSeek’s practical applications.

Setting up your development environment

Creating an environment for AI-assisted development requires carefully selecting tools that complement DeepSeek’s capabilities while supporting the full development life cycle. Our recommended toolkit centers around Cursor’s IDE, which you can download from cursor.com. This AI-native editor provides seamless integration with various language models, including DeepSeek-R1, making it an ideal choice for our demonstrations.

The foundation of our development stack includes Python 3.11 or higher, which serves as the primary language for our backend implementations due to its extensive AI and data processing libraries. Docker Desktop enables us to containerize our applications, ensuring consistent deployment environments and simplified distribution. Git provides essential version control capabilities, which is particularly important when collaborating with AI-generated code, where tracking changes and maintaining code history becomes crucial. Finally, Node.js 18 and higher support various frontend components and development tooling that we’ll leverage in our more comprehensive examples.

Configuring DeepSeek in Cursor’s IDE

The integration between Cursor’s IDE and DeepSeek-R1 represents the cornerstone of our development approach, transforming traditional coding workflows into collaborative partnerships between human creativity and AI reasoning capabilities. This configuration process, while straightforward, establishes the foundation for all our subsequent demonstrations.

Direct integration: Connecting Cursor to DeepSeek

Begin by accessing Cursor’s model configuration through the Settings menu. On macOS, you’ll find this under Cursor | Settings, while Windows and Linux users can find it under File | Settings. Within the settings interface, the Models section in the left sidebar contains all the options for managing AI model integrations.

Cursor’s IDE comes with built-in support for DeepSeek models, which you can enable by scrolling through the available models list and selecting deepseek-r1. Once enabled, the model appears in your available options, ready for configuration with your specific API credentials.

Figure D: Cursor IDE DeepSeek-R1 model selection interface

The configuration process requires entering your DeepSeek API credentials along with the appropriate connection settings. You’ll specify deepseek-r1 as the model name, provide your API key from the DeepSeek platform, set the base URL to https://api.deepseek.com/v1, and configure the provider as OpenAI Compatible to ensure proper API communication protocols.

To verify that your integration works correctly, open a new file in Cursor and invoke the AI command palette using Cmd + K on macOS or Ctrl + K on Windows and Linux. Select DeepSeek-R1 from the model dropdown and test with a substantive prompt, such as “Explain the difference between microservices and monolithic architecture.” A successful response confirms that your development environment is ready for the advanced use cases we’ll explore.

Alternative: Command-line integration

While Cursor’s IDE provides an excellent interactive environment for AI-assisted development, many developers prefer CLIs for automation, scripting, and integration with existing workflows. DeepSeek’s API follows OpenAI-compatible protocols, making it straightforward to integrate into command-line tools and automation scripts.

Establishing Python client access

Setting up programmatic access requires installing the OpenAI Python library along with python-dotenv for secure credential management:

pip install openai python-dotenv

Once the dependencies are installed, you can establish a connection to DeepSeek’s API using a simple Python script. This approach proves particularly valuable for batch processing, automated testing, and integration with larger systems:

import os
from openai import OpenAI
client = OpenAI(
 api_key=os.getenv("DEEPSEEK_API_KEY"),
 base_url="https://api.deepseek.com/v1"
)
response = client.chat.completions.create(
 model="deepseek-r1",
 messages=[{"role": "user", "content": "Hello DeepSeek!"}]
)
print(response.choices[0].message.content)

Running or deploying DeepSeek yourself

One interesting aspect of the DeepSeek models is that they are, in fact, open for anyone to interact with. In this section, we will go over some methods to interact with and deploy these models using your own infrastructure. Knowing that your data and application are completely controlled by you brings a variety of benefits; privacy and compliance are just two of them. Finally, it’s worth noting that deploying and managing these models yourself (especially the larger variants) can be quite the undertaking.

Here, we’ll cover some methods to use these models locally and also introduce you to ways of taking it further and deploying them yourself. We will be working with llama.cpp and Ollama to help you use DeepSeek locally.

Using llama.cpp

llama.cpp is a high-performance, open source C/C++ inference library designed to run LLMs efficiently on local hardware. It was started by Georgi Gerganov, and has a vibrant community and repo (https://github.com/ggml-org/llama.cpp). It allows users to execute models locally, without needing specialized GPUs or cloud infrastructure.

To install llama.cpp, please follow the installation instructions on the GitHub repo (https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md). If you are on a Mac, installing llama.cpp is as easy as the following:

$ brew install llama.cpp

To verify your installation worked correctly, just run the following:

$ llama-cli --help

You should see some output explaining how to use the CLI tool:

----- common params -----
-h, --help, --usage print usage and exit --version show version and build info --completion-bash print source-able bash completion script for llama.cpp --verbose-prompt print a verbose prompt before generation (default: false)
...

llama.cpp uses the GGUF file format (read more about it at https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). GGUF stands for GPT-Generated Unified Format (https://www.ibm.com/think/topics/gguf-versus-ggml). In order to use models with the tool, you need to download the models in GGUF format or convert them from PyTorch into that format. Thankfully, the folks at Unsloth (https://unsloth.ai/) provide GGUF variants of the DeepSeek models for free on Hugging Face (https://huggingface.co/collections/unsloth/deepseek-r1-all-versions).

Chatting locally with the original DeepSeek-R1 model would be challenging. You would need several machines running with multiple GPUs in order to run it. Let’s choose a simpler option and run one of the distilled versions of DeepSeek-R1. DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q4_K_M is a version of the Qwen 1.5 billion parameters model that was distilled from the DeepSeek-R1 model. In this case, we are running the quantized version, Q4_K_M, which means it’s a small model and can probably run on our CPU:

$ llama-cli -hf unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q4_K_M

This will automatically download the model from the Hugging Face repository and throw you into a chat interface right in your terminal.

Figure E: Use the DeepSeek model locally with the Unsloth interface

Hugging Face also provides a convenient way for you to get started with any model you see while navigating by clicking the Use this model button and following the instructions.

If you are thinking of integrating with a wider application, you might be interested in running this model using Python or any other programming language. Fortunately, we can use llama-cpp-python (https://github.com/abetlen/llama-cpp-python), a Python library that offers Python bindings to the exact same CLI. Here’s an example of using the model with Python. As you can see, we provide the same parameters as if we were using llama.cpp through the CLI, but now passing it into the library directly:

from llama_cpp import Llama
REPO_ID = "unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF"
MODEL_FILENAME = "DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf"
llm = Llama.from_pretrained(
    repo_id=REPO_ID,
    filename=MODEL_FILENAME,
    verbose=False,
)
response = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": "What is the most likely sky color in Copenhagen? Think hard and answer in one word.",
        },
    ],
)
text_response = response["choices"][0]["message"]["content"]
print(f"{MODEL_FILENAME} response:\n{text_response}")

If you are interested in deploying llama.cpp and using it as an OpenAI-compatible server, you can! Simply run the following:

$ llama-server -hf unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF:Q4_K_M

Now you can access the chat interface by visiting http://localhost:8080, which allows you to play with a completely local model.

Figure F: Using DeepSeek through llama.cpp

Alternatively, you can query the OpenAI-compatible API at http://localhost:8080/v1/chat/completions. After starting your llama-server with the preceding command, open another terminal and send a curl request like so:

$ curl --request POST \
          --url http://localhost:8080/completion \
          --header "Content-Type: application/json" \
          --data '{"prompt": "What is the capital of Portugal?","n_predict": 120}'

This will output a response straight from the model you are running the server with.

Now let’s talk about another interesting tool that builds on top of llama.cpp, called Ollama.

Ollama

Ollama (https://ollama.com/) is a library and CLI tool first released in 2023. Its goal is to allow you to run LLMs locally on your machine, with minimal setup. Its backend is powered by llama.cpp, but it provides a more user-friendly experience. Ollama has a wide range of models that you can download by visiting https://ollama.com/search. This includes many open source models (including, of course, DeepSeek models).

To get up and running with Ollama, head to their downloads page at https://ollama.com/download and download the binary for your relevant platform. Once you’re done, you can verify the installation worked correctly by running the following in your terminal:

$ ollama help

You should see a help message with the different options you can use to run Ollama.

To know what models you can run, start by visiting the library of models available at https://ollama.com/library.

Figure G: DeepSeek-R1-1.5B on Ollama.com

Once you have selected the model (in our case, deepseek-r1:1.5b) you wish to run locally, you can simply pull it, as you would with a Docker image:

$ ollama run deepseek-r1:1.5b

You can now start chatting with the model on your local machine. An interesting tip you can use is turning thinking on and off (this only applies to reasoning models) by typing /set nothink or /set think, respectively. Here’s an example of a terminal session and setting those modes dynamically:

➤ ollama run deepseek-r1:1.5b
>>> /set nothink
Set 'nothink' mode.
>>> What is the capital of the Le Marche Region in Italy?
The capital of the Le Marche Region in Italy is Rome
>>> /set think
Set 'think' mode.
>>> What is the capital of the Le Marche Region in Italy?
Thinking...
Alright, let's see. The user initially asked about the capital of the Le Marche region and I...

Additionally, Ollama also provides a Python SDK called ollama-python, available at https://github.com/ollama/ollama-python, which you can use as follows:

from ollama import chat
messages = [
    {
        "role": "user",
        "content": "What is the capital of Le Marche region in Italy?",
    },
]
response = chat(
    "deepseek-r1:1.5b", messages=messages, think=True, options={"temperature": 0.0}
)
print(f"Thinking:\n========\n\n{response.message.thinking}")
# Okay, so I need to figure out the capital of the Le Marche region in Italy...
print(f"\nResponse:\n========\n\n{response.message.content}")
# The capital of the Le Marche region in Italy is

Finally, Ollama also provides an OpenAI-compatible API that you can use. Here’s the same request as previously, using the API with curl:

$ curl --location 'http://localhost:11434/api/chat' \
      --header 'Content-Type: application/json' \
      --data '{
    "model": "deepseek-r1:1.5b",
    "options": { "temperature": 0.0 },
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of Le Marche region in Italy?"
      }
    ],
    "think": true
  }'

You’ll see the model response in your terminal as it comes in, as it is supposed to with streaming:

{"model":"deepseek-r1:1.5b","created_at":"2025-06-13T14:29:01.074249Z","message":{"role":"assistant","content":"","thinking":"Okay"},"done":false}...
{"model":"deepseek-r1:1.5b","created_at":"2025-06-13T14:29:01.084569Z","message":{"role":"assistant","content":"","thinking":","},"done":false}...
...

Ollama is a great tool for using and even deploying models locally. It’s a very nice addition to what llama.cpp has to offer. It’s really up to you which one you prefer to use. While llama.cpp offers a bit more power user features, such as setting sampling rates, chat templates, and tokenizers, Ollama tries to be as simple as possible and integrate with the world around it. Test both and decide which one you like best!

Deploying DeepSeek yourself

Depending on the size of the model, deploying an LLM yourself is no small feat. If you are deploying a 1 Gigabyte model that runs on a CPU, it’s not a problem. If you are serving a 405 billion parameter model to multiple users, and have latency and throughput requirements, then things start to get more complicated.

Let’s take an example. You would like to deploy the full DeepSeek-R1 model. The DeepSeek-R1 model at https://huggingface.co/deepseek-ai/DeepSeek-R1 has a total of 671 billion parameters. In full precision, this would mean that your model will be approximately 1.5 TB in size. Remember, this is RAM size, not disk size. You would need approximately 16 A100 GPUs with 80 GB of RAM each. An alternative is to deploy a smaller model (the 7B parameter version, for example). If you decide to deploy a smaller model, a great place to start is AWS SageMaker, for example. Hugging Face has a great tutorial for deploying (pretty much) any model in their repo onto AWS over at https://huggingface.co/docs/sagemaker/en/inference.

If you think using an API or deploying a smaller model is not an option for you, then I ask you to seriously reconsider this. It’s likely that deploying, managing, and paying for your own GPU cluster might actually be a bigger cost than using an API.

Let’s now talk about how to create your own setup.

Building your own setup for DeepSeek

In the final section of this appendix, we will highlight some interesting libraries that you could add to your arsenal when building with DeepSeek models. We’ve found these to be particularly useful when building on top of models. These libraries offer you more flexibility, are agnostic to models and providers, and help you do structured outputs.

Remember that these libraries are forever evolving, so make sure to visit the documentation for each one for up-to-date information. Hopefully, the core of many of these isn’t going to change anytime soon - which is the reason why we cover them here.

LiteLLM

LiteLLM (https://www.litellm.ai/) is a library and both a proxy server and an SDK focused on facilitating the use of LLMs when building with them. Here, we’ll focus on the SDK, since that’s what we use most for building with these models. It provides some good solutions to common problems:

Common interface: Allows you to use the same API when building with models. Rather than figuring out how to use models with AWS, Azure, GCP, Cloudflare, or any other provider, with LiteLLM, you can use the same completion function.
Retries and fallbacks: LLM APIs can be “flaky.” They can be down, slow, or even refuse to complete your request. LiteLLM enables you to try a given request multiple times. Additionally, you can set a number of models to act as “fallbacks,” using them instead if your primary model fails
Caching: Caching is an important part of both serving and using LLMs. With caching, you can avoid calling an LLM twice when it’s not needed. This makes both your costs go down and your requests much faster.

Let’s take a look at a script to achieve caching:

We will first cache responses on disk and confirm that a DeepSeek API key is present:

from litellm import completion
from litellm.caching.caching import Cache
import litellm
import os
litellm.cache = Cache(type="disk")
env_var_name = "DEEPSEEK_API_KEY"
assert os.environ[env_var_name], f"Please set your {env_var_name} environment variable."

Then we send a chat-completion request to the deepseek-reasoner model:

response = completion(
    model="deepseek/deepseek-reasoner",
    messages=[
        {
            "role": "user",
            "content": "What is the capital of region of Le Marche in Italy?",
        }
    ],

It tells LiteLLM to retry the call up to two times and, if that model fails, fall back to deepseek-chat:

 num_retries=2,  # Number of retries
    fallbacks=["deepseek/deepseek-chat"],  # Fallback models
    caching=True,  # Enable caching
)

Since caching is enabled, an identical request will later be served from disk instead of hitting the model again:

if hasattr(response.choices[0].message, "reasoning_content"):
    reasoning = response.choices[0].message.reasoning_content
    print(f"Reasoning:\n{reasoning}\n")
# Okay, the user is asking about the capital o
final_msg = response.choices[0].message.content
print(f"Final Answer:\n{final_msg}\n")
# The capital of the region of \*\*Le Marche (...

It is now time to look at LangChain.

LangChain

LangChain (https://python.langchain.com/docs/introduction/) is one of the most popular open source frameworks that covers most aspects of building with LLMs: thousands of integrations with an increasing number of providers, components for building RAG systems, agents, and more. It’s great for prototyping and to get something out there. However, when your application starts getting bigger, you might consider implementing some of the components yourself for added control. They could include:

Unified API: Use the same code and interface for all the models and providers, including local models such as Ollama and llama.cpp.
RAG helpers: Document loaders help you quickly load external data and pass that as context to the LLM. Document loaders, text splitters, and retrieval and generation workflows help make your RAG pipelines very easy to implement.
Structured outputs: Integrations for you to get structured data out of the LLMs. Support JSON schemas, streaming, few-shot prompting, and many more features.

Let’s look at the following script to see how we can use LangChain:

A ChatDeepSeek LLM wrapper from langchain_deepseek is created first:

from langchain_deepseek import ChatDeepSeek
from pydantic import BaseModel, Field
from typing import Optional
from enum import Enum
llm = ChatDeepSeek(
    model="deepseek-chat",
    temperature=0,
    max_tokens=None,
    timeout=None,
    max_retries=2,
)

Then we create a Pydantic model (SkyColor), which defines the expected structured output:

class SkyColorEnum(str, Enum):
    BLUE = "blue"
    GRAY = "gray"
    WHITE = "white"
    CLOUDY = "cloudy"
    OVERCAST = "overcast"
    CLEAR = "clear"
class SkyColor(BaseModel):
    """Sky color prediction for Copenhagen."""
    color: SkyColorEnum = Field(description="The primary color of the sky")
    description: str = Field(description="Detailed description of the sky appearance")
    confidence: Optional[int] = Field(
        default=None, description="Confidence level of the prediction, from 1 to 10"
    )

The with_structured_output method wraps the LLM so that any reply is automatically parsed and validated against that schema. The code calls the model with a prompt and receives a fully validated SkyColor object:

structured_llm = llm.with_structured_output(SkyColor)
result: SkyColor = structured_llm.invoke(
    "What color is the sky likely to be in Copenhagen today?"
)
print(result.color)
# prints SkyColorEnum.GRAY

Instructor

Instructor (https://python.useinstructor.com/) is the most popular Python library for extracting structured data from LLMs. It’s built on top of Pydantic and provides type-safe data extraction with automatic validation, retries, and streaming support. It supports most major platforms and models. The best thing about Instructor is actually their documentation, which provides great guides, cookbooks, and other great gems!

Structured outputs: Pass a Pydantic model as a response model and get a Pydantic model back. You can also use open source or self-hosted models, and Instructor will support it.
Automatic retries and self-correction: Failed validations trigger retries that inject the error message into the model context. This works great so that the model patches its own answer.
Streaming validated chunks: Instructor also supports streaming of partially validated objects. That means that you don’t have to wait for the full model response to show something to users. Read more about it here: https://python.useinstructor.com/blog/2023/11/26/python-generators-and-llm-streaming/#stream-processing.

We will now look at the use of the Instructor library in the following script. The resulting object is typed, validated, and ready for use:

Instructor wraps the OpenAI-compatible DeepSeek endpoint:

import os
from openai import OpenAI
import instructor
from pydantic import BaseModel, Field
client = instructor.from_openai(
    OpenAI(api_key=os.getenv("DEEPSEEK_API_KEY"), base_url="https://api.deepseek.com"),
    mode=instructor.Mode.MD_JSON,  # because this is a reasoning model..
)

We then add two Pydantic classes (City and CitiesResponse), which describe the structure of the desired response:

class City(BaseModel):
    name: str = Field(description="The name of the city")
    population: int = Field(description="The population of the city")
    notable_landmarks: list[str] = Field(
        description="List of notable landmarks in the city"
    )
class CitiesResponse(BaseModel):
    cities: list[City] = Field(description="List of interesting cities")

We then make a call to deepseek-reasoner, asking for three interesting Portuguese cities:

cities: CitiesResponse = client.chat.completions.create(
    model="deepseek-reasoner",
    messages=[
        {
            "role": "user",
            "content": "Top 3 most interesting cities in Portugal",
        },
    ],
    response_model=CitiesResponse,
    temperature=0.0,
)

Instructor will ensure the reply exactly matches the schema (the base model), retrying if the validation fails, giving the validation error to the LLM:

for city in cities.cities:
    print(f"City: {city.name}")
    print(f"Population: {city.population}")
    print(f"Notable Landmarks: {', '.join(city.notable_landmarks)}")

You will observe the following output:

# City: Lisbon
# Population: 545796
# Notable Landmarks: Belém Tower, Jerónimos Monastery, São Jorge Castle, Alfama District
# City: Porto
# Population: 237584
# ...

Now that we have covered a few essential libraries, we would like to mention some more that might be useful in your journey.

Other interesting libraries and resources

llm (https://github.com/simonw/llm) is a CLI + Python library that speaks to OpenAI, Anthropic, Gemini, Llama, and other models while logging every prompt/response and embedding to SQLite for later analysis. Its plugin system and Unix-style piping let you automate tasks such as summarizing web pages, generating shell commands, or comparing models straight from the terminal.
Pydantic AI (https://ai.pydantic.dev) is a type-safe agent framework from the Pydantic team that validates LLM outputs against Pydantic schemas while supporting providers such as OpenAI, Anthropic, Gemini, and Groq. It features dependency injection, streaming, and graph-based control flow and adds built-in monitoring to move GenAI projects into production with minimal boilerplate.
Outlines (https://dottxt-ai.github.io/outlines/latest) guides a model to emit output that matches a regex, JSON schema, grammar, or enum so downstream code can parse it without fuss. It runs with proprietary or open source backends and adds almost no latency, which is why many teams rely on it for reliable structured generation in production.

We’ve now covered a lot of different libraries that you can add to your toolbelt when building DeepSeek.

By using DeepSeek models, you have a lot of options and can effectively build where it suits you ranging from the official API or third-party providers, such as AWS or OpenRouter. If control matters, you can also run models locally with llama.cpp or Ollama. We also provided a good guide on where to go if you need to deploy in GPU clusters (if you have the means).

We also covered libraries such as LiteLLM, LangChain, and Instructor, which will help you take the most juice possible out of DeepSeek models, while keeping your logic and code sane.

Get This Book’s PDF Version and Exclusive Extras

Scan the QR code (or go to packtpub.com/unlock). Search for this book by name, confirm the edition, and then follow the steps on the page. | |

| Note: Keep your invoice handy. Purchases made directly from Packt don’t require one. |

11 Unlock Your Exclusive Benefits

Your copy of this book includes the following exclusive benefits:

Next-gen Packt Reader
DRM-free PDF/ePub downloads

Follow the guide below to unlock them. The process takes only a few minutes and needs to be completed once.

Unlock this Book’s Free Benefits in 3 Easy Steps

Step 1

Keep your purchase invoice ready for Step 3. If you have a physical copy, scan it using your phone and save it as a PDF, JPG, or PNG.

For more help on finding your invoice, visit https://www.packtpub.com/unlock-benefits/help.

Note: If you bought this book directly from Packt, no invoice is required. After Step 2, you can access your exclusive content right away.

Step 2

Scan the QR code or go to packtpub.com/unlock. | |

On the page that opens (similar to Figure 11.1 on desktop), search for this book by name and select the correct edition.

Figure 11.1: Packt unlock landing page on desktop

Step 3

After selecting your book, sign in to your Packt account or create one for free. Then upload your invoice (PDF, PNG, or JPG, up to 10 MB). Follow the on-screen instructions to finish the process.

Need help?

If you get stuck and need help, visit https://www.packtpub.com/unlock-benefits/help for a detailed FAQ on how to find your invoices and more. This QR code will take you to the help page. | |

Note: If you are still facing issues, reach out to customercare@packt.com.

packtpub.com

Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Fully searchable for easy access to vital information
Copy and paste, print, and bookmark content

At www.packtpub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Other Books You May Enjoy

If you enjoyed this book, you may be interested in these other books by Packt:

Building Agentic AI Systems

Anjanava Biswas, Wrick Talukdar

ISBN: 9781803238753

Master the core principles of GenAI and agentic systems
Understand how AI agents operate, reason, and adapt in dynamic environments
Enable AI agents to analyze their own actions and improvise
Implement systems where AI agents can leverage external tools and plan complex tasks
Apply methods to enhance transparency, accountability, and reliability in AI
Explore real-world implementations of AI agents across industries

LLMs in Enterprise

Ahmed Menshawy, Mahmoud Fahmy

ISBN: 9781836203070

Apply design patterns to integrate LLMs into enterprise applications for efficiency and scalability
Overcome common challenges in scaling and deploying LLMs
Use fine-tuning techniques and RAG approaches to enhance LLM efficiency
Stay ahead of the curve with insights into emerging trends and advancements, including multimodality
Optimize LLM performance through customized contextual models, advanced inferencing engines, and evaluation patterns
Ensure fairness, transparency, and accountability in AI applications

Packt is searching for authors like you

If you’re interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their sinsight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Now you’ve finished DeepSeek in Practice, we’d love to hear your thoughts! Scan the QR code below to go straight to the Amazon review page for this book and share your feedback or leave a review on the site that you purchased it from.

https://packt.link/r/180602084X

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

https://packt.link/r/180602084X

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Index

accuracy reward 66

activation-aware weight quantization (AWQ) 357

activations 49

adaptive attention routing 15

affinity scores 54

Agent2Agent (A2A) protocol 257

agentic applications 249

agentic systems 259, 260

arXiv papers, summarizing example 261-269

evaluator-optimizer workflow 260

orchestrator-workers workflow 270

agents 249, 250, 258

reference link 260

working with 257-259

AI-assisted development environment

management, configuring 147

setting up 144

workspace, organizing 146

AIME 70

alignment techniques 14

Amazon Bedrock

managed deployment on 363-366

reference link 350

Amazon Resource Name (ARN) 238

Amazon SageMaker 245

Amazon Web Services (AWS) 233, 406-410

isolated model service, deploying with 233-235

architecture modifications 15, 16

artificial general intelligence (AGI) 82

Artificial intelligence (AI) 3

arXiv.org

URL 261

augmented LLM 258, 279

automatic caching 315

autoregressive generation 47

autoregressive token generation 51

autoregressive transformer 47

AWS DLC repository

reference link 237

Azure AI Foundry

reference link 351

Bedrock API 233

Bedrock Custom Model Import 233

benchmarking system

best practices 194, 195

evaluation, comparing 183-186

metrics, computing 189

tools, selecting for use case 193

benchmarking tools

setup 148, 149

binary reward 59

bioRxiv

URL 261

causal masking 51

Chain-of-Draft (CoD) 107

Chain-of-Thought (CoT) 12, 60, 90, 251

chat template 121

Claude Code

reference link 249

Cloudflare 403-406

cold start data 12

cold start supervised training data 70-72

compounding errors 257

consistency reward 66, 72

constrained generation 229

containers 220

content-delivery network (CDN) 403

context-aware responses, DeepSeek

advanced RLHF 60

chain-of-thought reasoning 61

choice of datasets 61

FP8 training 57, 58

group relative policy optimization (GRPO) 59

multi-token prediction 58

precision control 57, 58

test-time scaling 62

context-sensitive routing 53

contract review 301

Contract Understanding Atticus Dataset (CUAD) 296, 302

distillation fine-tuning process, overview with 303, 304

extending 303

fine-tuning Gemma 3 on 318-323

control flow 258

convolutional neural network (CNN) 79

CoT traces 77

CrewAI

reference link 249

cross-entropy loss 77

Cursor IDE

DeepSeek, configuring 145, 146

working with, for DeepSeek 414

Daily Health Summary 201

decoder 40

decoder module

internal mechanics 44

working example 50-52

decoder-only autoregressive transformer 42

Deep Java Library (DJL) 233

deep research

reference link 249

DeepSeek 3-7, 14, 39

advanced capabilities 78

agentic reasoning 81

architecture modifications 15-18

configuring, in Cursor IDE 145, 146

context length and compression ceiling 86

controversies, surrounding 33-36

dataset and cultural scope gaps 87

dataset and philosophy, training 23, 24

deploying 417, 423

deployment landscape 336

deployment quirks 337

impact, on global AI ecosystem 30-32

inference latency 86

interpretability and alignment risks 86

key architectural components 40

llama.cpp, using 417-420

mental models and principles 91-98

MoE architecture 19

Ollama 421-423

Outlook on limitations 87

prompting, tips and advice 99-102

prompt routing 40-42

real-time interaction trade-offs 86

reinforcement learning 14

response, impacting to prompts 103-109

scaling challenges 85

self-deploy, selecting 337-341

significance 337-341

sparse expertise limitations 85

technical breakthroughs 7

tool integration 81

training process 8-14

used, for implementing MoE architecture 19-22

versions and evolution 24-30

versus leading LLMs 82-85

vision capabilities 79

working, with Cursor IDE for 414

DeepSeek API

available models, using 397

features 399-401

FIM 402

pricing and rate limits 398, 399

setting up 392-397

temperature parameter 397

working with 392

DeepSeek, for financial document extraction

prompting 174, 175

prompting, response evaluation 176-183

DeepSeek, in Cursor IDE

command-line integration 416

configuring 414

direct integration 415, 416

DeepSeek models

best practices 246

CPU-based inference, with Transformers and XGrammar 227-230

interacting with 222

LiteLLM 222-224

recommendations 246

refactoring, for local generation on CPU 231, 232

running locally, with Ollama 224-227

DeepSeek, prompts

issues, troubleshooting 129-133

migration guide 133-138

DeepSeek-R1 42-44

autoregressive transformer 47

complex technical documentation 150

extraction errors 187

layer normalization 49, 50

MoE routing 52, 53

multi-head attention 48, 49

prompt designing 148, 149

reference link 42

token processing and generation, with router 44-46

tools setup, benchmarking 148, 149

used, for benchmarking setup 171-174

used, for comparing document parsing tools 188-194

used, for test document creation 171-174

DeepSeek-R1-Distill-LLaMA 12

DeepSeek-R1-Distill-Qwen 12

DeepSeek-R1-Zero 11

DeepSeek-V3 11

DeepSeek V3, to cloud

deploying, with Hugging Face Inference Endpoints 366-374

DeepSeek-VL-7B 79-81

deployment 335

examples 359-374

deployment strategy, DeepSeek

cost sanity check 344-346

decision-making framework, for selecting 341-344

paths 346-354

development environment

setting up 414

Direct Preference Optimization (DPO) 17

distillation 77, 78, 293

variants 5

distillation and fine-tuning

multi-label extraction problem, in legal texts 301, 302

significance 297-300

use case and dataset 301

DIY deployment

architecture choices 353

on IaaS 351-354

Docker

used, for deploying prototype 220

Docker + vLLM 349

Docling

testing 188, 189

document analysis challenge, DeepSeek R1

follow-up code generation request 160

follow-up response evaluation 166-169

prompting 150

recalibration, through iterative prompting 169, 170

response evaluation 151-159

Domain Object Model (DOM) 89

dynamic routing 54

dynamic workflows 287

end-to-end prototype, building 202

API, refactoring into 216-220

context, creating 207-210

Daily Health Summary, creating 212-215

data, fetching 202-206

deploying, with Docker 220-222

structured output, defining 211, 212

enhanced CUAD dataset 296

distillation fine-tuning process, overview with 303, 304

evaluating agents 288

evaluator LLM 260

FastMCP

reference link 255

feed-forward networks (FFNs) 43

few-shot fallacy 99

Fill-in-the-Middle (FIM) 126, 399, 402

fine-tune DeepSeek

need for 318

fine-tuned model

evaluation and results 324-330

fine-tuning 293, 296

instructional format 317, 318

optional standalone Python script 296, 297

fine-tuning Gemma 3

on CUAD 318-323

FP8 (8-bit floating point) 57

Free Lossless Audio Codec (FLAC) 357

functional token resonance imaging (fTRI) 44

function-calling feature 112-114

gating function 54

Gemini Deep Research

reference link 249

Gemma 3

need for 318

general matrix multiply (GEMM) 57

generator 260

global AI ecosystem

DeepSeek, impact on 30-32

Google Vertex AI

reference link 350

GPQA 70

GPT quantization (GPTQ) 357

grouped-query attention (GQA) 50

Group Relative Policy Optimization (GRPO) 59, 97

group-relative reward mechanism 65

hardening patterns 116

hardware

selecting, of deployment 354

Hugging Face Deep Learning Containers (DLC) 233

Hugging Face Inference Endpoints

reference link 351

used, for deployment of DeepSeek V3 to cloud 366-374

human preference fine-tuning 12

hybrid validation strategy 71

IAM role 238

inference engine 235

utilizing 355-357

instructor 428-430

Pydantic, used for type-enforced generation via 114

using, for type safety 114-116

isolated model service deployment, with AWS 233, 234

DeepSeek, deploying with LMI containers 237-245

inference backends 235-237

service, updating for using Amazon SageMaker endpoints 245

key performance indicators (KPIs) 171

key-value (KV) 55, 318, 336

KL-divergence

reference link 60

knowledge data 12

knowledge distillation 71

knowledge distillation loss 77

Kubernetes Event-driven Autoscaling (KEDA) 353

KV caching 378

LangChain 426, 427

LangGraph

reference link 249

large language model (LLM) 3, 143, 395

Large Language Model Operations (LLMOps) tools

for model distillation 304-308

large language models (LLMs) 200, 293

layer normalization 49, 50

legal AI training

dynamics 323

legal rationale distillation

two-stage workflow 304, 308-323

lightweight decoder 42

LiteLLM 222-224, 305, 424-426

reference link 222

llama.cpp

using 417-420

llm 430

LLM architectures

comparison 17, 18

LMI containers 233, 237

long-term memory 251

Low-Rank Adaptation (LoRA) 321

Luke Salamone

reference link 397

Marginalia search engine

reference link 281

MarkItDown

testing 188, 189

MATH 70

memory

long-term memory 251

short-term memory 251

Mergers and Acquisitions (M&A) 301

Mixture-of-Experts (MoE) 19, 40, 52, 335

architectures, implementing with DeepSeek 19-22

routing, in DeepSeek 52, 53

Model Context Protocol (MCP) 254-257

model distillation

LLMOps tools 304-308

MoE gating network 54

Multi-Head Latent Attention (MLA) 40, 55, 56, 124, 340

multi-head self-attention 48, 49

multi-token prediction (MTP) 58, 86

benefits 59

native JSON mode 111, 112

Natural Language Processing (NLP) 8

Ollama 225, 348, 422, 423

URL 226, 421

used, for local deployment 361-363

OpenRouter 224, 412, 413

URL 411

orchestrator component 270

orchestrator-workers workflow 270

report-generating workflow example 271-278

outcome-based scoring 60

outlines 430

out-of-memory (OOM) 377

pattern 259

Personally Identifiable Information (PII) 376

position-wise network 49

post-attention normalization 49

pre-attention normalization 49

preference ranking model (PRM) 86

production systems

CI/CD, for models 382-385

cost management 379-381

monitoring and observability 375, 376

scaling and performance 376-379

security, in practice 381, 382

prompt DeepSeek

advanced techniques and tooling, for structured output 109-111

prompt DeepSeek, techniques and tooling

function-calling feature 112-114

native JSON mode, setting up 111, 112

robustness and special cases, strategies 116-121

type-enforced generation, with Pydantic 114-116

prompt prefix caching 378

prompt routing 40

Pydantic

used, for type-enforced generation via Instructor 114

Pydantic AI 430

Pydantic models

reference link 211

Python client

access, establishing 416

python-garminconnect project

reference link 202

quantization

significance 357-359

quantized DeepSeek Coder model

used, for local deployment 361-363

queries per second (QPS) 312

reasoning 62, 63

reinforcement learning 14

reinforcement learning from human feedback (RLHF) 6

reinforcement learning (RL) 11

reranker modules 42

retrieval-augmented generation (RAG) 29

retrieval modules 42

reward model 10

rotary positional embedding (RoPE) 43, 48

router 44

R series

versus V-series models 126-128

rule-based reward mechanisms 10

rule-based RLHF 67

rule-based verification 12

SageMaker JumpStart

reference link 350

scaffolding 259

schema 96

schema drift 118

scoring function 66

self-attention mechanism 15

self-teaching

emergent patterns 78

sequential processing 51

setup, for DeepSeek

building 424

instructor 428-430

LangChain 426, 427

libraries and resources 430

LiteLLM 424-426

short-term memory 251

speculative decoding 58

stack 295

state-of-the-art (SOTA) 21

structured reasoning 63

supervised fine-tuning (SFT) 10

synthesizer LLM 270

system prompts 63, 100

task 250

temperature parameter 230

TensorRT-LLM 356

test-time scaling 62, 97

Text Generation Inference (TGI) 233, 356

thinking, DeepSeek 64

complex scenarios, handling 68, 69

response quality, evaluating with GRPO 65, 66

rule-based RLHF 67

think and answer blocks 64

third-party APIs

AWS 406-410

Cloudflare 403-406

OpenRouter 411-413

using 402

time to first token (TTFT) 336

token-by-token gating mechanism 54

tokenization 51

tool calling 113, 251

tool-calling agent 279

web search agent 280-287

tools 251-253

training, DeepSeek 69, 70

cold start data 70-72

training pipeline, of R1 73-76

training process 8-14

transformer architecture 8

Transformers

CPU-based inference with 227-231

type-enforced generation

with Pydantic, via Instructor 114

Unsloth 227, 305

verbose prompt trap 100-102

vision capabilities, DeepSeek

architectural changes, for vision-language integration 79-81

vision transformer (ViT) 79

vLLM 237

reference link 235

V-series models

prompting techniques 121

versus R series 126-128

V-series models, prompting techniques

context window confusion 124, 125

formatting fiesta 123, 124

template tango 121, 122

web search agent 280-287

worker 270

workflow 259

XGrammar

CPU-based inference with 227-231

ZenML 305

pipeline data processing 314-317

used, for creating local environment 294

ZenML Cloud account

API keys and environment variables 295

creating 294, 295

posted @ 2026-07-27 16:23 绝不原创的飞龙阅读(100) 评论(0) 收藏举报

刷新页面返回顶部

龙哥盟

掠夺·扩张·投机·博弈

DeepSeek-实战-全-

DeepSeek 实战（全）

前言

DeepSeek 实践指南

概述

本书的目标读者

本书涵盖的内容

如何充分利用本书

声明

下载示例代码文件

下载彩色图片

本书中的约定

联系我们

分享您的想法

书籍附带的免费福利

如何解锁

敬请关注

加入我们的 Discord 和 Reddit 社区

第一部分

理解与探索 DeepSeek

1 什么是 DeepSeek？

免费福利

介绍 DeepSeek

理解 DeepSeek 的技术突破

训练过程

仅编码器、仅解码器与编码器-解码器

监督微调与 RLHF

DeepSeek-R1 训练流程

DeepSeek-R1 推理流程

强化学习

架构修改

主流大语言模型架构对比（2025 年）

MoE 架构

DeepSeek 如何实现 MoE

训练数据集与理念

DeepSeek 的版本与演进

DeepSeek 不断演进的生态系统

深入解析：每个模型的功能比较

DeepSeek 模型与产品生态

DeepSeek 产品生态

平台集成与部署生态

DeepSeek 对全球 AI 生态系统的影响

市场颠覆与价格战

催生下一波开源 AI 浪潮

中国技术成熟的象征

围绕 DeepSeek 的争议

存在偏见和被滥用的可能性

开放性与国家安全关切

文化与政治认知

学术界与产业界的抵制

关于RL方法的伦理辩论

话语特性：非缺陷而是标志

总结

获取本书 PDF 版本与独家资源

2. 深入解析 DeepSeek

DeepSeek的核心架构组件

DeepSeek中的提示词路由

解码器

解码器模块的内部机制

通过路由器进行的 token 处理与生成

自回归 Transformer

Transformer 中的自回归生成

多头注意力

层归一化

工作示例

混合专家模型

专家是如何初始化和配置的？

门控如何计算激活哪些专家？

专家是如何被激活的？

我们现在将回到之前的示例提示，观察专家是如何被激活的。

多头潜在注意力：面向长上下文的高效内存注意力

DeepSeek 实现高效且上下文感知响应的特性

FP8 训练与精度控制

DeepSeek-R1 的独特之处在于其对 FP8 的选择性应用

用于策略解码的多令牌预测

GRPO 与高级 RLHF

思维链推理：CoT 与长 CoT

理解 DeepSeek 的推理机制

关于`RL`方法的伦理辩论

`DeepSeek`的核心架构组件

`DeepSeek`中的提示词路由

`DeepSeek`如何处理复杂场景

`DeepSeek`训练