Streaming Response

In large language model inference scenarios, streaming returns can gradually send results to the client while the model generates text, achieving real-time interactive experience.

Function services support streaming response data. openYuanrong provides two streaming response REST APIs: Subscribe to Stream Service and Invoke Service with streaming returns.

Subscribe to Stream Service

Streams produced using the data stream API in the standalone program distributed parallelization interface can be consumed by calling the Subscribe to Stream Service REST API. When using this interface, note the following:

• Stream producers and subscribers must maintain the same stream name.

• Ensure in business logic that streams are subscribed before producing data to avoid produced data not being fully consumed.

• It is recommended that stream producers and subscribers negotiate specific terminators to express stream data transmission completion, facilitating subscribers to actively close after consuming data.

View the complete usage example in Using Streams in Function Services.

Streaming Returns When Invoking Services

Streaming returns when invoking openYuanrong services are implemented based on the SSE (Server-Send Events) protocol. In function services, associate streams through the context.getStream() interface in the context interface to send stream data. An example of a function service with streaming returns is as follows.

Python

import json

def handler(event, context):
    data = {"event":"this is a stream"}
    context.get_stream().write(json.dumps(data))
    return 'ok'

Java

package org.yuanrong.demo;

import org.yuanrong.services.runtime.Context;
import org.yuanrong.services.runtime.action.Stream;
import com.google.gson.JsonObject;

public class Demo {
    public String handler(JsonObject jsonObject, Context context) throws Exception {
        try {
            Stream stream = context.getStream();
            JsonObject obj = new JsonObject();
            obj.addProperty("event","this is a stream");
            stream.write(obj);
        } catch (Exception e) {
            System.out.println("executor exception occurred");
        }
        return "ok";
    }
}

When invoking a function service, you need to add the Accept: text/event-stream field to the request header.

Note

Streaming response requires deploying openYuanrong with function service support and configuring frontend.args.enableEvent=true.