随笔分类 -  7 大模型Infra

基础知识
摘要：分布式通信原语 Broadcast: 将一张XPU卡数据复制同步到其它所有XPU卡上 Scatter: 将一张XPU卡数据切片分发到其它所有XPU卡上 Reduce：接收其它所有XPU卡上数据，通过某种操作(Sum/Mean/Max)之后，最终放到某个XPU卡上 Gather: 接受其它所有XPU卡 阅读全文
posted @ 2025-07-02 20:21 fariver 阅读(16) 评论(0) 推荐(0)
[思考] LLM训练工程优化
摘要：背景 大语言模型（LLM）参数量已突破万亿，单次训练计算量达千亿亿次浮点运算（ExaFLOPs）。单卡GPU显存上限仅80GB（A100），算力峰值312 TFLOPS，显存墙与通信墙成为千卡/万卡分布式训练的核心瓶颈。 前置知识 1. DDP训练过程 ​​数据切片​​：全局Batch拆分为子Bat 阅读全文
posted @ 2025-07-02 20:19 fariver 阅读(141) 评论(0) 推荐(0)
[PaperReading] AWQ: ACTIVATION-AWARE WEIGHT QUANTIZATION FOR ON-DEVICE LLM COMPRESSION AND ACCELERATION
摘要：目录AWQ: ACTIVATION-AWARE WEIGHT QUANTIZATION FOR ON-DEVICE LLM COMPRESSION AND ACCELERATIONTL;DRMethodStory为什么用weights quantized-only的方案？如何挑选import wei 阅读全文
posted @ 2025-06-21 17:17 fariver 阅读(138) 评论(0) 推荐(0)
[PaperReading] ZeRO Memory Optimizations Toward Training Trillion Parameter Models
摘要：目录名称TL;DRMethodZeRO-DPZeRO-R背景优化策略小结Experiment效果可视化总结与思考相关链接 名称 link 时间：19.10 单位：microsoft 作者相关工作：https://i.cnblogs.com/posts/edit;postId=18916963 dee 阅读全文
posted @ 2025-06-21 14:36 fariver 阅读(26) 评论(0) 推荐(0)
[基础] GPU体系结构
摘要：整体架构 物理模块 包含关系为：GPC > TPC > SM > CORE GPC(Graphics Processing Clusters 图形处理簇)：GPC负责处理图形渲染和计算任务。每个GPC包含多个TPC，以及与其相关的专用硬件单元和缓存。 TPC(Texture Processing C 阅读全文
posted @ 2025-06-19 21:29 fariver 阅读(189) 评论(0) 推荐(0)
[Paper Reading] MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
摘要：目录TL;DR大规模高效训练算法优化通信重叠优化其他优化容错设计(Fault Tolerance)数据收集与分析诊断测试Fast Checkpointing and RecoveryExperiment相关链接 MegaScale: Scaling Large Language Model Trai 阅读全文
posted @ 2025-06-19 20:34 fariver 阅读(51) 评论(0) 推荐(0)
[PaperReading] Efficient Memory Management for Large Language Model Serving with PagedAttention
摘要：目录Efficient Memory Management for Large Language Model Serving with PagedAttentionTL;DRMotivation现状：GPU显存是瓶颈具体浪费情况MethodvLLM Framework调度与抢占其它TrickExpe 阅读全文
posted @ 2025-06-12 22:06 fariver 阅读(86) 评论(0) 推荐(0)
[PaperReading] PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
摘要：目录PyTorch FSDP: Experiences on Scaling Fully Sharded Data ParallelTL;DRMethodSystem DesignModel InitializationSharding Strategies（分片策略）​​Full Sharding 阅读全文
posted @ 2025-06-07 18:44 fariver 阅读(109) 评论(0) 推荐(0)
[PaperReading] DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
摘要：目录DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented ScaleTL;DR推理优化方法针对Transformer Kernel优化DeepFusionSBI-GeMM针对D 阅读全文
posted @ 2025-06-06 21:54 fariver 阅读(69) 评论(0) 推荐(0)
[PaperReading] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
摘要：目录FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTL;DRMethodFlashAttention算法详解Sparse FlashAttentionExperimentQ&A总结与思考相关链接 阅读全文
posted @ 2025-06-06 21:53 fariver 阅读(55) 评论(0) 推荐(0)
[PaperReading] Megatron: Reducing Activation Recomputation in Large Transformer Models
摘要：目录Megatron: Reducing Activation Recomputation in Large Transformer ModelsTL;DRMethodSP(Sequence Parallelism、序列并行)Selective RecomputationCode && Implem 阅读全文
posted @ 2025-06-03 21:24 fariver 阅读(52) 评论(0) 推荐(0)