Compile openYuanrong from Source Code

Compile openYuanrong from Source Code#

Overview#

This chapter introduces how to compile openYuanrong from source code.

Environment Preparation#

A host with the following configuration:

  • Hardware

    Item

    Requirement

    CPU

    4 cores or more, 16 cores recommended

    Memory

    10GB or more, 32GB recommended

    Disk Space

    50GB or more

    OS Platform

    Linux x64/Linux ARM

  • Operating System

    Operating System

    Version

    openEuler (x86_64)

    openEuler 22.03 LTS or above

    openEuler (aarch64)

    openEuler 22.03 LTS or above

Install the following tools used for compilation on the host and keep the versions consistent:

Tool Name

Tool Type

Function

x86_64 Version

aarch64 Version

AdoptOpenJDK

Open Source Third Party

Build

8

8

Apache Maven

Open Source Third Party

Build

3.9.11

3.9.11

Golang

Open Source Third Party

Build

1.24.1

1.24.1

CMake

Open Source Third Party

Build

3.22.0

3.22.0

Python

Open Source Third Party

Build

3.9/3.10/3.11

3.9/3.10/3.11

ninja-build/ninja

Open Source Third Party

Build

1.13.1

1.13.1

bazelbuild/bazel

Open Source Third Party

Build

6.5.0

6.5.0

Node.js

Open Source Third Party

Build

20.19.0

20.19.0

Tip

We provide scripts for use on the openeuler_22.03_LTS_sp4 environment to facilitate quick installation of compilation tools. Refer to the following commands to download and use.

yum install -y wget
wget https://openyuanrong.obs.cn-southwest-2.myhuaweicloud.com/build_tools/openeuler_22.03_LTS/check_tools.sh
wget https://openyuanrong.obs.cn-southwest-2.myhuaweicloud.com/build_tools/openeuler_22.03_LTS/install_tools.sh
bash install_tools.sh
source /etc/profile.d/buildtools.sh

Compilation#

openYuanrong is divided into four code repositories:

  • yuanrong: Runtime and dashboard repository, where runtime compilation depends on release packages from yuanrong-functionsystem and yuanrong-datasystem repositories.

  • yuanrong-functionsystem: Function system repository, compilation depends on release packages from yuanrong-datasystem repository.

  • yuanrong-datasystem: Data system repository, can be compiled independently.

  • ray-adapter: ray adapter repository, can be compiled independently.

To use the multi-language function programming interface, you need to compile the yuanrong repository; to use the data system interface alone, you need to compile the yuanrong-datasystem repository; to use the ray adapter interface, you need to compile the ray-adapter repository.

Compile yuanrong-datasystem#

First download the source code. Create a code directory, for example mkdir -p /opt/openyuanrong/, and execute the following command in the directory.

# Here downloads the master branch of yuanrong-datasystem repository, replace with your forked personal repository and branch as needed.
git clone -b master https://atomgit.com/openeuler/yuanrong-datasystem.git

Execute the following script to compile.

# Learn more compilation options through bash build.sh -h
cd /opt/openyuanrong/yuanrong-datasystem
bash build.sh -X off

Note

You can set compilation concurrency through the compilation option -j. With the compilation host idle CPU and memory 1:2 configuration (e.g., 16U32G), it is recommended to set to: number of CPU cores + 1, to avoid compilation failure due to insufficient memory during the linking phase.

Compilation artifacts are generated in the yuanrong-datasystem/output directory, containing the following two files:

  • yr-datasystem-vx.x.x.tar.gz: Contains SDK and binary files.

  • yr_datasystem-x.x.x.whl: whl installation package, contains command-line tools and SDK.

Compile yuanrong-functionsystem#

Step1: Source Code Download

Download the function system source code through Git or other methods in the compilation directory. The following example will download the function system source code as the yuanrong-functionsystem folder in the /opt/openyuanrong folder.

cd /opt/openyuanrong
git clone -b master https://atomgit.com/openeuler/yuanrong-functionsystem.git

Step2: Get Dependencies

According to the architecture design, the function system compilation process depends on the data system SDK, so you need to copy the data system compilation artifacts or release artifacts to the function system build dependencies. The function system build script will automatically download or introduce dependencies at startup according to the configuration of vendor/VendorList.csv file. For data system compilation dependencies, it will be introduced by default through local files. That is, use the path descriptor file://localhost/vendor/src/yr-datasystem.tar.gz to describe the data system source and will not verify the dependency Hash value.

Assuming the data system compilation artifact path is /opt/openyuanrong/yuanrong-datasystem/output/yr-datasystem-v0.0.0.tar.gz, you can copy the data system compilation artifact to the default path configured in VendorList.csv through the following command.

cd /opt/openyuanrong
mkdir -p yuanrong-functionsystem/vendor/src
cp -a yuanrong-datasystem/output/yr-datasystem-v0.0.0.tar.gz yuanrong-functionsystem/vendor/src/yr-datasystem.tar.gz

Step3: Source Code Compilation

In the function system source code root directory, we can use the built-in compilation script’s build command to achieve source code compilation. The compilation process will be divided into three stages: third-party component download compilation, second-party component compilation, system source code compilation. The entire process will be completed automatically, and components that have already been compiled will be automatically skipped.

cd /opt/openyuanrong/yuanrong-functionsystem
./run.sh build

build command key parameters:

  • JOB_NUM, -j: Compilation concurrency, sets the number of CPU cores used when compiling the program. Default value: OS number of cores

  • VERSION, -v: Compilation version number, sets the version number recorded in the program when compiling. Default value: 0.0.0

For more commands, refer to: ./run.sh --help, ./run.sh build --help

Note

It is recommended to set compilation concurrency through the compilation option -j to avoid compilation failure due to insufficient memory during the compilation linking phase. The peak memory occupancy (GB) is about 1.5~1.8 times the concurrency, i.e., MEM(GB) ≈ JOB(CPU) * 1.8(GB/CPU), it is recommended to set safe concurrency: number of CPU cores / 2

Step4: Artifact Packaging

The main compilation artifacts of the function system are multiple binary programs and public dynamic libraries, and will be accompanied by some dependencies and runtime configurations. Therefore, you can use the pack command to complete the packaging of function system build artifacts. Packaging artifacts and packaging content will be uniformly stored in the output folder.

cd /opt/openyuanrong/yuanrong-functionsystem
./run.sh pack

pack command key parameters:

  • VERSION, -v: Compilation version number, sets the version number recorded in the program when compiling. Default value: 0.0.0

For more commands, refer to: ./run.sh --help, ./run.sh pack --help

The directory and key artifacts are as follows:

tree output/
├── functionsystem
│   ├── bin          # Function system compilation artifacts
│   ├── config       # Function system runtime configuration
│   ├── deploy       # Function system deployment configuration
│   ├── lib          # Function system dynamic libraries
│   └── sym          # Function system symbol table
├── metrics
│   ├── config       # Observability configuration
│   ├── include      # Observability interface
│   └── lib          # Observability dynamic libraries
├── metrics.tar.gz   # Observability packaging artifacts
└── yr-functionsystem-v0.0.0.tar.gz # Function system packaging artifacts

Compile Runtime and Dashboard#

First download the source code. Create a code directory, for example mkdir -p /opt/openyuanrong/, and execute the following command in the directory.

# Here downloads the master branch of yuanrong repository, replace with your forked personal repository as needed.
git clone -b master https://atomgit.com/openeuler/yuanrong.git

Compile Runtime#

Runtime compilation depends on data system and function system release packages:

  • Create directory /opt/openyuanrong/yuanrong/datasystem/output, copy the compiled data system release package yr-datasystem-vx.x.x.tar.gz to this directory and extract it through tar -zxf yr-datasystem-vx.x.x.tar.gz --strip-components=1 command. Extract the compiled function system release package yr-functionsystem-vx.x.x.tar.gz, copy the metrics folder in the package to the /opt/openyuanrong/yuanrong/ directory.

Execute the following script to compile.

# Learn more compilation options through bash build.sh -h
cd /opt/openyuanrong/yuanrong
bash build.sh

If dependency download failure error occurs, you can try deleting the /opt/openyuanrong/yuanrong/thirdparty/runtime_deps directory and recompiling

Note

You can set compilation concurrency through the compilation option -j. With the compilation host idle CPU and memory 1:2 configuration (e.g., 16U32G), it is recommended to set to: number of CPU cores + 1, to avoid compilation failure due to insufficient memory during the linking phase.

Compilation artifacts are generated in the yuanrong/output directory, containing the following files:

  • yr-runtime-vx.x.x.tar.gz: Runtime release package.

Compile Dashboard#

Dashboard is an optional component, execute the following script to compile when needed.

# Learn more compilation options through bash build.sh -h
cd /opt/openyuanrong/yuanrong/go
bash build.sh

Compilation artifacts are generated in the yuanrong/output directory, containing the following files:

  • yr-dashboard-vx.x.x.tar.gz: Dashboard release package.

Compile ray-adapter#

First download the source code. Create a code directory, for example mkdir -p /opt/openyuanrong/, and execute the following command in the directory.

# Here downloads the master branch of ray-adapter repository, replace with your forked personal repository as needed.
git clone -b master https://atomgit.com/openeuler/ray-adapter.git

Execute the following script to compile.

# Learn more compilation options through bash build.sh -h
cd /opt/openyuanrong/ray-adapter
bash build.sh

Compilation artifacts are generated in the ray-adapter/output directory, containing the following files:

  • ray_adapter-x.x.x.whl: whl installation package.