Fault Tolerance

openYuanrong provides an automatic fault tolerance mechanism for stateless function execution:

• When a function instance is evicted due to priority or other reasons, the caller will automatically initiate rescheduling of the stateless function.

• When a function instance experiences network failures, the caller will automatically retry.

Additionally, openYuanrong supports configuring retries when stateless function instances fail due to other reasons such as core dump.

Configuring Retries

Python

import yr

yr.init()

@yr.invoke
def add(n):
    print("run add")
    raise Exception('my test')

# Set retry count to 3
opt = yr.InvokeOptions(retry_times=3)
result = add.options(opt).invoke(1)
try:
    print(yr.get(result))
except Exception:
    pass

yr.finalize()

Running the program, in the function’s standard output log file, you will see four identical lines of output run add, indicating that the function executed once and then retried 3 times after failure.

C++

For C++ language, we provide two configurations: retry count and retry judgment callback function retryChecker, supporting user-defined retries. You need to configure the retryTimes parameter greater than 0 for the retry judgment callback function to take effect.

The callback function prototype is as follows:

bool (*retryChecker)(const Exception &e) noexcept = nullptr;

#include <iostream>
#include <yr/yr.h>

int Add(int n)
{
    std::cout << "run Add" << std::endl;
    throw std::runtime_error("connection failed");
}
YR_INVOKE(Add)

bool RetryForConnection(const YR::Exception &e) noexcept
{
    // Trigger retry when error code is 2002 (code has core dump or exception) and error message contains "connection failed"
    if (e.Code() == YR::ErrorCode::ERR_USER_FUNCTION_EXCEPTION) {
        std::string msg = e.what();
        if (msg.find("connection failed") != std::string::npos) {
            return true;
        }
    }
    return false;
}

int main(int argc, char *argv[])
{
    YR::Init(YR::Config{}, argc, argv);

    // Set retry count to 3 and configure callback function
    YR::InvokeOptions opt;
    opt.retryTimes = 3;
    opt.retryChecker = RetryForConnection;

    auto ref = YR::Function(Add).Options(opt).Invoke(1);
    std::cout << *YR::Get(ref) << std::endl;

    YR::Finalize();
    return 0;
}

Running the program, in the function’s standard output log file, you will see four identical lines of output run Add, indicating that the function executed once and then retried 3 times after failure.

Java

This feature is not currently supported.