Deploy openYuanrong Job Applications

Deploy openYuanrong Job Applications#

This chapter introduces how to build and deploy job applications developed using single-machine program distributed parallelization interfaces in openYuanrong cluster.

Environment Dependencies#

Your application may contain other dependencies besides runtime, such as:

  • Packages that Python and Java programs depend on for running, dynamic libraries that C++ programs depend on.

  • Environment variables that need to be read in openYuanrong functions.

  • Data files that need to be read in openYuanrong functions, tools used, etc.

openYuanrong functions may run on any node in the cluster, so these dependencies need to exist and remain consistent on each openYuanrong node. In production environments, it is recommended to pre-install relevant dependencies.

Run Job Applications Locally#

You can directly run Driver programs in job applications on openYuanrong cluster nodes, convenient for viewing application output in real time.

Python Applications#

Python is an interpreted language and does not require a compilation process. When you deploy Python applications in openYuanrong cluster, you need to ensure that all nodes in the cluster have installed dependencies required by the application.

C++ Applications#

C++ applications need to compile a binary Driver program and a dynamic library containing all openYuanrong functions. During deployment, copy the dynamic library to the same path on all nodes of the openYuanrong cluster, and choose one of the following ways to configure the path.

  • When running Driver program, use --codePath parameter to specify the absolute path of the dynamic library.

  • When calling YR::Init(const Config &conf) interface in Driver program, configure Config’s loadPaths parameter.

Taking CMake build as an example, a simple application project directory is as follows. src directory stores source code, CMakeLists.txt is the configuration file used by CMake build system, build directory is used to store binary and dynamic library files generated by build. Execute commands cmake .. and make in sequence under build directory to complete the build.

yr-cpp-demo
├── src
│   └── demo.cpp
├── CMakeLists.txt
└── build

CMakeLists.txt file reference:

cmake_minimum_required(VERSION 3.16.1)
# Specify project name, for example: yr-cpp-demo
project(yr-cpp-demo LANGUAGES C CXX)
set(CMAKE_CXX_STANDARD 17)

# Specify compilation output file path in build directory
set(SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})
set(BINARY_DIR ${SOURCE_DIR}/build)
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${BINARY_DIR})

set(CMAKE_CXX_FLAGS "-pthread")
set(BUILD_SHARED_LIBS ON)

# Replace YR_INSTALL_PATH value with openYuanrong actual installation path
set(YR_INSTALL_PATH "/usr/local/lib/python3.9/site-packages/yr")
link_directories(${YR_INSTALL_PATH}/cpp/lib)
include_directories(
    ${YR_INSTALL_PATH}/cpp/include
)

# Generate executable file cpp-demo, modify demo.cpp to your corresponding source code file
add_executable(cpp-demo src/demo.cpp)
target_link_libraries(cpp-demo yr-api)

# Generate dynamic library file cpp-demo-dll, modify demo.cpp to your corresponding source code file
add_library(cpp-demo-dll SHARED src/demo.cpp)
target_link_libraries(cpp-demo-dll yr-api)

After successful build, binary file cpp-demo and dynamic library file libcpp-demo-dll.so will be generated. Taking /opt/openyuanrong/function/demo as code path example, you need to copy libcpp-demo-dll.so file to that path on all nodes in the cluster. Execute command ./cpp-demo --codePath=/opt/openyuanrong/function/demo under build directory to run the application.

Java Applications#

Jar package built by Java applications can choose whether to include dependencies according to needs. During deployment, copy the jar package and dependencies to the same path on all nodes of the openYuanrong cluster, and choose one of the following ways to specify the path.

  • When running application, use -Dyr.codePath parameter to specify the absolute path of the jar package.

  • When calling init(Config conf) interface in application program, configure Config’s loadPaths parameter.

Taking Maven build as an example, a simple application project directory is as follows. demo directory stores source code, pom.xml is the configuration file used by Maven build system. Execute command mvn clean package under yr-java-demo directory to complete the build.

yr-java-demo
├── pom.xml
└── src
    └── main
        └── java
            └── com
                └── demo
                    └── Main.java

pom.xml file reference:

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.demo</groupId>
    <artifactId>main</artifactId>
    <version>1.0.0</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

    <dependencies>
        <dependency>
            <!-- Modify version number to your actual version -->
            <groupId>org.yuanrong</groupId>
            <artifactId>yr-api-sdk</artifactId>
            <version>1.0.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>com.demo.Main</mainClass>
                        </manifest>
                    </archive>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <appendAssemblyId>false</appendAssemblyId>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

After successful build, jar package main-1.0.0.jar will be generated in yr-java-demo/target directory. Taking /opt/openyuanrong/function/demo as code path example, you need to copy main-1.0.0.jar file to that path on all nodes in the cluster. Execute command java -Dyr.codePath=/opt/openyuanrong/function/demo -cp ./main-1.0.0.jar com.demo.Main under target directory to run the application.

Submit Jobs Using REST API#

You can also use REST API to submit a job to openYuanrong cluster. The difference from local running is that Driver program will also run in the cluster as an openYuanrong function.

After job submission, it will run asynchronously until completion or failure. To retry or run job with different parameters, you need to submit the job again. Job is bound to cluster lifecycle, when cluster crashes, all running jobs will be terminated.

Refer to Job Management API to learn how to use.