LLM Application 0.3+

Chat Request Filters

The LLM Application supports filters to modify both the incoming chat completion request and the response. The filter system that is used for the context is designed to be extensible by extension. The filter system is similar to Wiki Components, though there is only a single real component for every model and filters are stored inside that component. A filter consists of at minimum three parts:

• An XClass to store its configuration
• A sheet to display and edit that configuration
• A component implementing org.xwiki.contrib.llm.ChatRequestFilterBuilder that provides references to the XClass and the sheet and has a method to construct a list of no, one, or several org.xwiki.contrib.llm.ChatRequestFilter objects from a BaseObject (of the XClass of the configuration)

All available filters are always enabled, it is up to the filter to ensure that it has a sensible default state and to provide a way to disable it if that makes sense. When the filter is disabled, the ChatRequestFilterBuilder can return an empty list to not insert any filter into the filter chain.

Filters use a chain of responsibility design pattern to forward a chat completion request between a chain of filters until it reaches the component that is responsible for making the actual request to the LLM. This provides filters a lot of flexibility, filters can:

• Modify the incoming chat completion request, e.g., adding context the context filter does.
• Decide to stop propagating the request, e.g., based on a rate limit or a content filter, and reply with an error message instead. This could also be used to implement a caching system or to log all requests.
• Intercept the response and modify it, e.g., to add additional information or to filter results based on some criteria or to log responses.

Both request and response are available at the same time, allowing to easily pass information between them or also, e.g., log both at the same time.

The priority of the ChatRequestFilterBuilder component decides the order of the filters. Filters with a lower priority are called first. To simplify the implementation of filters, an abstract class org.xwiki.contrib.llm.AbstractChatRequestFilter is provided that implements the forwarding to the next filter. There are two possible request modes that are supported by the LLM Application: streaming and non-streaming. For streaming request, the LLM provides a stream of tokens that are forwarded to the client one by one. In the LLM Application, this is implemented with a callback that is called for every chat completion chunk. If any modification of the result shall be performed, the callback that is forwarded can be replaced by a custom callback. It could also be imagined to cache a number of tokens and only then forward them as whole paragraph if the paragraph passes the content filtering guidelines. It is also possible to add a prefix to the output by calling the callback before forwarding the call in the filter chain. For non-streaming requests, the full result is returned by the filter chain and can thus be examined modified as a whole. In both cases, the request can be modified before forwarding it and in both cases forwarding the request by calling the parent method is optional. The callback for streaming requests may throw an IOException, this happens in particular when the connection is closed by the client because the user cancelled the chat completion request. Both streaming and non-streaming requests are fully synchronous at the moment. This might be changed in particular for streaming requests in case it should turn out to be a problem in the future.

public class MyChatRequestFilter extends AbstractChatRequestFilter
{
    @Override
    public void processStreaming(ChatCompletionRequest request,
        FailableConsumer<ChatCompletionChunk, IOException> consumer) throws IOException, RequestError
    {
        // Modify the request or replace/wrap the consumer if desired.
        super.processStreaming(request, consumer);
    }

    @Override
    public ChatCompletionResult process(ChatCompletionRequest request) throws IOException, RequestError
    {
        // Modify the request or the result if desired.
        return super.process(modifiedRequest);
    }
}

For the sheet, when using editable properties, it is important to set data-object-policy="updateOrCreate" as the XObject won't be present on models that were created before the extension has been installed. The sheet for models takes care of adding the XObject in regular view and edit mode, so in these cases the presence of the XObject can be assumed.