LLM Application (BETA)

Last modified by Michael Hamann on 2024/05/27 00:06

cogLLM Artificial Intelligence for search, content generation and editing in XWiki.
TypeXAR
CategoryApplication
Developed by

Matéo Munoz, Ludovic Dubost, Michael Hamann, Paul Pantiru

Active Installs0
Rating
3 Votes
LicenseGNU Lesser General Public License 2.1
Compatibility

14.10 and above, 16.2.0 and above starting with version 0.3

Installable with the Extension Manager

Description

This extension is still in BETA-release, some unexpected behavior could happen.

The LLM Application extension for XWiki is a tool that enriches the functionalities of XWiki, using LLM AI to provide a chat with an LLM assistant. In combination with the Index for the LLM Application, it can also provide answers based on content of the wiki or other applications.

A dedicated chat UI can also be embedded in other applications that can authenticate with the help of the token-based authentication for the LLM Application.

 

Using the extension

To start using the interface, the first step is to add at least one working configuration (detailed here). Once that is done, there are two ways too launch the chat:

  • LLM Application 0.3+ By opening the "LLM Application" entry in the Applications Panel.
  • By editing any page in the wiki, and clicking on this button:

toolbarLLM.png
(Yes that is the same button icon as the "paste" function in the toolbar, since the LLM application has no button icon dedicated to this day. If you already have the "paste" button activated on the toolbar, LLM application will still work).
The interface should appear with an output that looks like the example image below, and you should be able to work with it. If the button appears to be disabled, please go to the Configuration part.
StartingInterface.png

Features

The LLM Application bundle the following features:

Chat interface

The interface is organized to facilitate the use of the platform. First, It consists of two drop-down menus, one allows the selection of LLM models and the other to choose a specific prompt that corresponds to specific tasks.
modelMenu.png
As you can see on the image above, model are described as follows: modelname (configTheyComeFrom). So you can get two models with the same name coming from two different configuration.

promptMenu.png
Here, the prompt selector just displays the labels of the prompt (that should describe the action it does). More detail on the prompts use in the Quick task section. 

There are three buttons with specific functions: start a new conversation, submit a request, and stop an ongoing request if needed. The exchanges that occur afterward are visible in a specific window, where the user's messages and the model's responses appear in the form of discussion bubbles for better visualization.

LLM Application 0.2+ There is no more button except the submit button, other one have been simplified in a dropdown menu like this:
BurgerView.png
This menu has a "New Conversation" button, that just reset everything in the window (erase the previous message + the context in the background). Then at the bottom there is one new feature : Advanced Settings. Advanced Settings allow you to do 2 things, change temperature (i.e, the level of creativity of the models) and edit system prompt on the go. You can see the interface right below.
settingsView.png

Change models

When you change models, the conversation you had with the previous model(s) are kept for the next exchanges. So you can use one model and switch to another without any consequences. 

Quick task

This feature saves time and streamlines the user's experience, making the process more efficient and straightforward. Quick task eliminates unnecessary back-and-forth dialogue to deliver direct results, that correspond to the chosen. Simply put, the user choose the task they want, and the system produces the desired output without further interaction. This enhanced user interface simplifies complex processes into a quick, one-step task. When you use one, the interface will show the corresponding prompt that has been used in the background above your message (see the image below).
quicktaskexpl.png
LLM Application <0.2 There is only a defined list of quick task (Summarize, Completion, Translate, Auto-tagging).
LLM Application 0.2+ Since 0.2, the extension provide a database that you can feed with as many tasks as you want. They are then retrieved in the menu, and you can use it.
Keep in mind that this feature is more efficient with powerful LLM models.

Interaction with the XWiki page

To interact with the edited page, you have two options.
interactionbtnexpl.png
First, you can use the include button located on the assistant response. This will include the message in the edited page in place of the cursor.
It is a good use to select some lines you want to work on, make the model output what you want and click "include" when you are done. It will instantly replace the lines you first selected with the new text.
One of the limitation of the include button for now is the incapacity to write to the source of XWiki, that means if you want the model to write a text in XWiki syntax, the only choice you have is to copy / paste it in the source editor. That is why we implemented a copy button, so you can use any response in the way you want.
The copy button will copy the message to clipboard, what you do with it is up to you.

LLM Application 0.2+ Prompt Database

The user and system prompt database represents a structured manner of managing and organizing prompts for enhanced interaction and functionality.
PromptDB.png
Here, you can see each entry has 7 fields:

  • Title: define the name of the prompt. That is the value displayed in the Quick Task menu. 
  • Description: provide a description of what the prompt is about.
  • LLM Application 0.2.1+  Active: if checked the prompt will be displayed in the Quick Task menu. Else the prompt is not displayed in the Quick Task.
  • System prompt: The system prompt is the initial set of instructions that sets the boundaries for an AI conversation. What rules the assistant should follow, what topics to avoid, how the assistant should format responses, and more.
  • User prompt: A text or instruction provided to guide the model's response generation. It sets the context and provides guidelines for producing relevant output.
  • Temperature: Number between 0 and 2 describing the creativity of the model for its answer(0 being deterministic and 2 most creative).
  • LLM Application 0.2.1+  Default: if checked the prompt will be used as default.
  • Database list: not displayed in the database view, this has no purpose for user.

Configuration

To configure the extension correctly, you will have to follow these steps.
ConfigRelease0_2.png
As you can see in this example, you can have as many configurations as you want. In your case at the start the configuration should be empty.

Server name

The server name stand for the name you want to give to the config. There are no rules on the name you give, just keep in mind to not give the same name to two different configuration, else the first one (in the order of declaration) will not be considered anymore.

URL prefix

The URL prefix is the place where you will define the prefix URL to reach the final URI (/models and /chat/completions). It's really important to have a URL respecting the following rules: 

  • It should respect this template: "http://.../" (or "https://.../"), where "..." has to be replaced with your real address.
    • To use the OpenAI models you should use the URL prefix "https://api.openai.com/v1/" for example.
  • It has to end with a "/", for request to be valid.
  • The endpoint "http://.../models" and "http://.../chat/completions" has to exist, else it can't work. 

LLM Application <0.3

Configuration file

The configuration file has to be filled with one model ID at least, and the models has to be compatible with the /chat/completions endpoint! For example with the OpenAI configuration, you can see that the 'Configuration File' field is filled with "gpt-4,gpt-3.5-turbo". In the end, The interface will display these two models in the selection menu.

Token

The token field is used in case you need one to authenticate yourself when doing a request. So it is not always necessary to fill it. On the example image above you can see the OpenAI configuration has a token, but the LocalAI one (a private server with LLM models stocked locally) does not need any, so its token field is let empty.

Can Stream box

The 'Can Stream' option is here in case you want to use models that could not use a streaming API to answer. In that case, you will have to make a configuration with the specified models and to let the 'Can Stream' box unchecked. You will then have to wait for their response to be finalized before receiving it all at once.

LLM Application 0.2+ Group Allowed

This field allows you to choose which groups of users are allowed of use for each configuration. If a configuration has no group defined, nobody will be able to use it.

LLM Application 0.3+

Model Configuration

The model and group list aren't part of the server configuration anymore. Instead, you now need to configure each model separately. This is to provide more control for each individual model and to allow providing different versions of a model with different context collections configured. You can access the model configuration below the chat interface in the LLM application.

1715703438419-888.png

Every model that shall be available for users or for embedding in the Index for the LLM Application needs to be configured. The name that you set when creating the model configuration is the name that is displayed to the user when selecting the model. Make sure to choose a descriptive name that is understood by the users like "Fast, with context", "Slow, no context", etc. For each model, you can configure the following properties:

  • Server name: the server that provides the model, as configured in the general configuration.
  • Type: The type of model, for chat models, select "Large language model", for embedding models "Embeddings model".
  • Model ID: This is the ID of the model on the server. This doesn't influence the name that is displayed to users of the chat interface.
  • Number of Dimensions: For embedding models, this is the number of dimensions that the embedding has, e.g., 768. You can leave this empty for chat models.
  • Context size: The maximum context size in terms of tokens. Used to limit the chat history to this context size.
  • Allow guests to access the model: Whether guest users can use this model. Use this carefully as this might incur high costs if you don't pay attention.
  • Group: The user groups that are allowed to use this model. Note that for embedding models it is important to allow both the users that are creating documents in the Index for the LLM Application and the users who perform queries by using the chat interface.
  • Filters: Provides configuration for various filters that can be provided by extensions, extensions can add further filters as documented in the LLM Models API documentation.
    • Context: when the Index for the LLM Application is installed, the context can be configured. The current chat message is embedded using the embedding model configured for the selected collections and the most similar content chunks from these collections are provided to the LLM as context. This allows the LLM to more accurately answer user requests based on relevant content from the collections. While this can reduce hallucinations and wrong responses, the LLM could still make up facts including references to context that doesn't actually exist, and even when the provided context is relevant it might ignore or misunderstand it and could still provide wrong answers.
      • Collections: the collections that shall be queried, empty if the request shouldn't be augmented with context.
      • Search results limit: how many results shall be considered. Note that a result is just a single chunk, not a whole document. There could be several chunks of the same document.
      • Context prompt: The prompt that is used to instruct the model to use the context. The text {{search_results}} is replaced by the actual search results.

1715703582292-762.png

Error

If you have an empty selection menu, or if you have a System message stating that an error happened when trying to get the model list, look carefully to your configuration and make sure you did every step described above correctly.

LLM Application 0.3+ The list of LLMs is unfortunately empty whenever an extension has been installed, upgraded or uninstalled until the next restart of XWiki. This is a known bug in XWiki. The only known workaround is to restart the wiki after the extension operation.

 

Prerequisites & Installation Instructions

We recommend using the Extension Manager to install this extension (Make sure that the text "Installable with the Extension Manager" is displayed at the top right location on this page to know if this extension can be installed with the Extension Manager). Note that installing Extensions when being offline is currently not supported and you'd need to use some complex manual method.

You can also use the following manual method, which is useful if this extension cannot be installed with the Extension Manager or if you're using an old version of XWiki that doesn't have the Extension Manager:

  1. Log in the wiki with a user having Administration rights
  2. Go to the Administration page and select the Import category
  3. Follow the on-screen instructions to upload the downloaded XAR
  4. Click on the uploaded XAR and follow the instructions
  5. You'll also need to install all dependent Extensions that are not already installed in your wiki

Dependencies

Dependencies for this extension (org.xwiki.contrib.llm:application-ai-llm-models-ui 0.3.1):

Get Connected