<<Python高性能编程>>阅读记录

0. https://book.douban.com/subject/36500831/
1. a/b返回结果为float，float内部有方法为is_integer()
2. timeit快速剖析程序功能python -m timeit -n 5 -r 1 -s "print('Hello world')"
3. shell内置的time和/usr/bin/time不一样，后者功能更多

/usr/bin/time -p python3 hello.py
Hello, world!
real 0.01
user 0.00
sys 0.00

/usr/bin/time --verbose python3 hello.py
Hello, world!
        Command being timed: "python3 hello.py"
        User time (seconds): 0.00
        System time (seconds): 0.00
        Percent of CPU this job got: 93%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.01
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 8960
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 1008
        Voluntary context switches: 1
        Involuntary context switches: 1
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

4. cProfile

python3 -m cProfile -s cumulative hello.py
Hello, world!
         4 function calls in 0.000 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 hello.py:1(<module>)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

写入统计文件，再使用Python对其进行分析

python3 -m cProfile -o profile.stats hello.py
Hello, world!
python3
Python 3.8.10 (default, Mar 18 2025, 20:04:55)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pstats
>>> p = pstats.Stats("profile.stats")
>>> p.sort_stats("cumulative")
<pstats.Stats object at 0x751ed2788df0>
>>> p.print_stats()
Mon Jun 16 11:48:03 2025    profile.stats

         4 function calls in 0.000 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 hello.py:1(<module>)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}


<pstats.Stats object at 0x751ed2788df0>
>>> p.print_callers()
   Ordered by: cumulative time

Function                                          was called by...
                                                      ncalls  tottime  cumtime
{built-in method builtins.exec}                   <-
hello.py:1(<module>)                              <-       1    0.000    0.000  {built-in method builtins.exec}
{built-in method builtins.print}                  <-       1    0.000    0.000  hello.py:1(<module>)
{method 'disable' of '_lsprof.Profiler' objects}  <-


<pstats.Stats object at 0x751ed2788df0>

使用SnakeViz可视化cProfile的输出
5. 使用line_profiler逐行剖析，提供装饰器@profile剖析函数

pip install line_profiler
sudo apt install python3-line-profiler
kernprof -l -v sumN.py
sumN(10000) = 49995000
Wrote profile results to sumN.py.lprof
Timer unit: 1e-06 s

Total time: 0.002708 s
File: sumN.py
Function: sumN at line 1

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           @profile
     2                                           def sumN(n):
     3         1          2.0      2.0      0.1      res = 0
     4     10001       1281.0      0.1     47.3      for i in range(n):
     5     10000       1424.0      0.1     52.6          res += i
     6         1          1.0      1.0      0.0      return res

6. 使用memory_profiler诊断内存占用情况

python3 -m memory_profiler sumN.py
sumN(10000) = 49995000
Filename: sumN.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     1   37.004 MiB   37.004 MiB           1   @profile
     2                                         def sumN(n):
     3   37.004 MiB    0.000 MiB           1       res = 0
     4   37.004 MiB    0.000 MiB       10001       for i in range(n):
     5   37.004 MiB    0.000 MiB       10000           res += i
     6   37.004 MiB    0.000 MiB           1       return res

7. 使用PySpy查看既有进程，尤其适合长时间运行的进程或安装很复杂的生产环境中

sudo env "PATH=$PATH" py-spy --pid <pid>

8. 使用dis模块查看CPython字节码
9. 使用pytest进行单元测试
10. %%timeit, %%memit
11. 可散列类型是指实现了magic function __hash__, __eq__或__cmd__的类型
12. 名称查找，先locals()，后globals()，最后__builtin__
13. 使用perf进行性能分析

perf stat -e cycles,instructions,cache-references,cache-misses,branches,branch-misses,task-clock,faults,minor-faults,cs,migrations python3 sumN.py

 Performance counter stats for 'python3 sumN.py':

   <not supported>      cycles:u
   <not supported>      instructions:u
   <not supported>      cache-references:u
   <not supported>      cache-misses:u
   <not supported>      branches:u
   <not supported>      branch-misses:u
             12.10 msec task-clock:u                     #    0.960 CPUs utilized
               732      faults:u                         #   60.473 K/sec
               732      minor-faults:u                   #   60.473 K/sec
                 0      cs:u                             #    0.000 /sec
                 0      migrations:u                     #    0.000 /sec

       0.012611533 seconds time elapsed

       0.008342000 seconds user
       0.004171000 seconds sys

14. ctypes，cffi
15. aiohttp

import asyncio
import random
import string

import aiohttp

def generate_urls(base_url, num_urls):
    for i in range(num_urls):
        yield base_url + "".join(random.sample(string.ascii_lowercase, 10))

def chunked_http_client(num_chunks):
    semaphore = asyncio.Semaphore(num_chunks)

    async def http_get(url, client_session):
        nonlocal semaphore
        async with semaphore:
            async with client_session.request("GET", url) as response:
                return await response.content.read()
    return http_get

async def run_experiment(base_url, num_iter=1000):
    urls = generate_urls(base_url, num_iter)
    http_client = chunked_http_client(100)
    responses_sum = 0
    async with aiohttp.ClientSession() as client_session:
        tasks = [http_client(url, client_session) for url in urls]
        for future in asyncio.as_completed(tasks):
            data = await future
            responses_sum += len(data)
    return responses_sum

if __name__ == "__main__":
    import time

    loop = asyncio.get_event_loop()

    delay = 100
    num_iter = 1000

    start = time.time()
    result = loop.run_until_complete(
        run_experiment(
            f"****://127.0.0.1:8080/add?name=asyncio&delay={delay}", num_iter
        )
    )
    end = time.time()
    print(f"Result: {result}, Time: {end - start}")

16. multiprocessing完成的典型任务

• 使用Process或Pool对象来并行地执行CPU密集任务
• 使用模块dummy在线程池中并行地执行I/O密集型任务
• 通过Queue来共享序列化对象
• 在工作线程之间共享状态