为 Paddle2ONNX 搭建 Github Actions 自动发包机制

1 简介

Paddle2ONNX 此前一直使用手动编译所有版本的 Python 源码包再手动上传到 PyPI 的方式来分发发行版。很显然，这是一种极其低效的办法，本文介绍如何为 Paddle2ONNX 添加自动发包机制。

2 实现过程

Paddle2ONNX 的编译流程参考 onnx 的编译流程实现，因此在自动发包机制的设计上我也参考了 ONNX。

2.1 新建编译 protobuf 脚本

首先新建一个脚本用于编译 protobuf

#!/bin/bash

# Copyright (c) ONNX Project Contributors
#
# SPDX-License-Identifier: Apache-2.0

export CORE_NUMBER=$1

if [[ -z "$CORE_NUMBER" ]]; then
   export CORE_NUMBER=1
fi

# Build protobuf from source with -fPIC on Unix-like system
ORIGINAL_PATH=$(pwd)
cd ..
git clone https://github.com/protocolbuffers/protobuf.git -b v3.16.0
cd protobuf
mkdir build_source && cd build_source
cmake ../cmake -DCMAKE_INSTALL_PREFIX=${PWD}/installed_protobuf_lib -Dprotobuf_BUILD_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -Dprotobuf_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE=Release
make -j$CORE_NUMBER && make install
export PATH=${PWD}/installed_protobuf_lib/bin:${PATH}
cd $ORIGINAL_PATH

2.2 新建编译 Paddle2ONNX 脚本

然后新建一个脚本传入参数编译 Paddle2ONNX

#!/bin/bash

# Copyright (c) ONNX Project Contributors
#
# SPDX-License-Identifier: Apache-2.0

set -e -x

# CLI arguments
PY_VERSION=$1
PLAT=$2
GITHUB_EVENT_NAME=$3

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib

# Compile wheels
# Need to be updated if there is a new Python Version
if [ "$(uname -m)" == "aarch64" ]; then
 PIP_INSTALL_COMMAND="$PY_VERSION -m pip install --no-cache-dir -q"
 PYTHON_COMMAND="$PY_VERSION"
else
 declare -A python_map=( ["3.8"]="cp38-cp38" ["3.9"]="cp39-cp39" ["3.10"]="cp310-cp310" ["3.11"]="cp311-cp311" ["3.12"]="cp312-cp312")
 PY_VER=${python_map[$PY_VERSION]}
 PIP_INSTALL_COMMAND="/opt/python/${PY_VER}/bin/pip install --no-cache-dir -q"
 PYTHON_COMMAND="/opt/python/${PY_VER}/bin/python"
fi

# Update pip and install cmake
$PIP_INSTALL_COMMAND --upgrade pip
$PIP_INSTALL_COMMAND cmake

# Build protobuf from source
source .github/workflows/scripts/build_protobuf_unix.sh "$(nproc)"

# Build Paddle2ONNX wheels
$PYTHON_COMMAND -m build --wheel || { echo "Building wheels failed."; exit 1; }

# Bundle external shared libraries into the wheels
# find -exec does not preserve failed exit codes, so use an output file for failures
failed_wheels=$PWD/failed-wheels
rm -f "$failed_wheels"
find . -type f -iname "*-linux*.whl" -exec sh -c "auditwheel repair '{}' -w \$(dirname '{}') --plat '${PLAT}' || { echo 'Repairing wheels failed.'; auditwheel show '{}' >> '$failed_wheels'; }" \;

if [[ -f "$failed_wheels" ]]; then
    echo "Repairing wheels failed:"
    cat failed-wheels
    exit 1
fi

# Remove useless *-linux*.whl; only keep manylinux*.whl
rm -f dist/*-linux*.whl

echo "Successfully build wheels:"
find . -type f -iname "*manylinux*.whl"

2.3 新建发包工作流

我们使用 Github Actions 搭建一个自动发包工作流，这里先适配一下 Linux 版本，其他版本后期再适配。

name: Build Package

on:
  release:
    types: [published]

permissions:
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [ '3.8', '3.9', '3.10', '3.11', '3.12']
        architecture: [ 'x64' ]

    # For the sake of simplicity in testing, the Paddle2ONNX packaging program will temporarily only package executable files for Python 3.8.
    # In the future, we will need to extend support to cover Python 3.8 through Python 3.10.
    steps:
      # Checkout the latest branch of Paddle2ONNX.
      - name: Checkout Paddle2ONNX
        uses: actions/checkout@v4
        with:
          submodules: true

      - name: Build on manylinux2014_x86_64
        uses: docker://quay.io/pypa/manylinux2014_x86_64:latest
        with:
          entrypoint: bash
          args: .github/workflows/scripts/entrypoint.sh ${{ matrix.python-version }} manylinux2014_x86_64

      - uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
        with:
          name: wheels
          path: dist

      - name: Publish package
        uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
        with:
          user: __token__
          password: ${{ secrets.ZHENG_BICHENG_PYPI_TOKEN }}

创建工作流后，当你创建 Release 时，Github Actions 将自动为你编译最新的 Paddle2ONNX 并发布最新版本。

3 参考链接

posted @ 2024-11-29 17:24  Zheng-Bicheng  阅读(31)  评论(0)    收藏  举报