Knowledge index providing search capabilities to the LLM Application.

Description

The knowledge index provides a way to configure and manage an index of collections of documents that can provide additional context to chats in the LLM Application. A technique called Retrieval Augmented Generation (RAG) is used to provide context to the chat completion. Context chunks that are semantically similar to the chat message are retrieved from the index and provided to the completion model. This allows the completion model to use knowledge from the index to generate more factual completions.

There are two ways to manage collections and the documents in them: through the UI that is provided in this extension or through the REST API. The extension supports indexing both content that has been added specifically for this index inside the application/through the REST API and content from existing spaces in your wiki (since version 0.5). Spaces that shall be indexed need to be configured explicitly.

Managing Collections

The AI application provides a simple interface for managing collections and their documents. When opening the AI application as admin, a link is provided at the bottom to open the Collections overview.

Here you can add a new collection or edit or delete an existing one. LLM 0.4+ At the bottom, you can see the number of documents that are currently in the indexing queue, waiting to be embedded and indexed.

In the view of a collection, you can edit all properties, see the documents in the collection and add new documents. The name of the page of a document is a random value that is derived from the name of the collection. It doesn't use the name directly in order to support long document names, e.g., coming from an external application that might not be valid document names in XWiki. Renaming documents isn't supported at the moment as the primary use case is indexing external content. In a future version, it will be possible to index existing content of the wiki instead of managing the content separately.

The following properties can be edited:

• Title: The display name of the collection
• Embedding Model: The reference to the embedding model that can be configured in the models section of the AI application. This model is used to generate embeddings of parts (chunks) of the documents in the collection in order to support similarity search. By embedding the chat message in the same way, the similarity of the chat message to the documents in the collection can be calculated and the most relevant documents can be found.
• Chunking Method: The method used to split the documents into chunks. Currently, only the maxChars method is supported, which splits the document into chunks of a maximum size with an overlap.
• Chunking Max Size: The maximum size of a chunk in characters. LLM 0.5+ The size of a chunk must be at least 10 characters. The chunk size may be reduced to half of that size if this allows for a better splitting. The splitting prefers ending a chunk before a heading, an empty line, a line break, a sentence, or a space character. The chunking searches for these splitting points in that order and splits at the first option that yields a chunk of the configured size. For example, if the chunk size is 2000 characters, but there is a heading at position 800, the chunking continues with searching for an empty line. If there is then an empty line at position 1500, the chunk will be split at position 1500.
• Chunking Overlap Offset: The overlap of chunks in characters. LLM 0.5+ The overlap size must not be negative and must be smaller than half of the maximum chunk size. Similarly to the maximum chunk size, the overlap may be reduced to half of the configured overlap to find a better splitting point using the same procedure as explained there, just searching from the start instead of the end of the possible overlap region.
• Allow Guests: If guests can query this collection (as part of a chat completion)
• Query Groups: The list of groups that are allowed to query this collection (as part of a chat completion). This is only used when guests aren't allowed to query the collection.
• Rights Check Method: The method used for checking access rights of individual documents during queries. Supported values: public, external. See also the Authorization section below.
• LLM 0.5+ Document store: the way the documents are stored. There are currently two stores available, the internal store stores documents as special documents below the collection. The XWiki store allows defining a list of spaces in the wiki that are indexed for this collection.
• LLM 0.5+ Document spaces: When the XWiki store is selected, the spaces that shall be indexed. Otherwise, this property is ignored.
• URL of the external rights check method: The URL that is used to check rights for all found documents when the external rights check method is selected. This property is only displayed when the external rights check method is selected.

LLM 0.5+ A collection can be re-indexed by clicking on the respective button. This is necessary after changing the embedding model or when there were errors during indexing and thus indexing should be repeated. When chunking parameters were changed, re-indexing can be used to immediately apply those changes, otherwise they are only applied the next time a document is updated.

Managing Documents

In the view of a collection, you can see the documents in the collection and add new documents.

You can configure the following properties of documents:

• Title: The title of the document
• Language: The language of the document (currently not used)
• URL: The URL of the document, used to display a link to the original resource when the document is used as context in a chat
• Mime Type: The mime type of the document, currently not used, could be used in the future to use a chunking method that is specific to the mime type
• Content: The content of the document that is indexed

Additionally, you can attach files to a document and they will also be indexed if the content can be extracted by Apache Tika.

Authorization

At query time, regular XWiki access rights aren't checked. Instead, access to the indexed can be controlled at two levels:

• The collection: It can be controlled which groups can query a collection in the configuration of the collection. A collection can also be allowed for guests, in this case no check is performed.
• The document: On every collection, a method for checking rights for individual documents can be configured. After retrieving relevant chunks of context information, this rights checking method is asked for authorization for every retrieved document.

By default, two rights check methods are provided:

• Public: this method just allows access to all documents. It is best suited when all users who have access to the collection should also be allowed to access all documents in it.
• External: this method queries an external API via HTTP to check which documents can be accessed. It supports configuring a URL that is contacted on every query.

External Authorization Checks

On every query, a POST request with the following contents is submitted to the configured URL:

• an array document_ids of document IDs as strings
• a string that contains the user's name in XWiki in the form wiki:XWiki.Username
• an object ldap_user with properties uid and dn if the user is connected to an LDAP account from the LDAP Authenticator
• an array jwt_users with objects with issuer and subject fields if the user is connected to one or several authorized applications configured in the Token-based authentication for the LLM Application (BETA)
• an array oidc_users if the user is connected to one or several OpenID Connect providers.

If you need to pass additional parameters like the collection name you can specify them as URL parameters.

An example request for a single document for the standard admin user could look like this:

The response needs to be an object mapping document IDs to true or false, meaning that the user has or hasn't access to the respective document. To grant access to the "Performance" document in the example, the endpoint would need to reply {"Performance":true}. Any missing values or errors are interpreted as no access.

Prerequisites & Installation Instructions

We recommend using the Extension Manager to install this extension (Make sure that the text "Installable with the Extension Manager" is displayed at the top right location on this page to know if this extension can be installed with the Extension Manager). Note that installing Extensions when being offline is currently not supported and you'd need to use some complex manual method.

You can also use the following manual method, which is useful if this extension cannot be installed with the Extension Manager or if you're using an old version of XWiki that doesn't have the Extension Manager:

1. Log in the wiki with a user having Administration rights
2. Go to the Administration page and select the Import category
3. Follow the on-screen instructions to upload the downloaded XAR
4. Click on the uploaded XAR and follow the instructions
5. You'll also need to install all dependent Extensions that are not already installed in your wiki

Dependencies

Dependencies for this extension (org.xwiki.contrib.llm:application-ai-llm-index-ui 0.6.2):

• org.xwiki.contrib.llm:application-ai-llm-index-api 0.6.2
• org.xwiki.contrib.llm:application-ai-llm-models-ui 0.6.2