Allow replicating pages and other data between different XWiki instances

Description

The aim of this project is to make easy to share data between different XWiki instances and also expose as a demonstration the replication of wiki pages.

You can contribute translations for this application on https://l10n.xwiki.org/projects/xwiki-contrib/application-replication-default/.

Linking instances

Communication between instances is done through the HTTP protocol, and the first thing to do before replicating any data is to tell each instance how to access the other(s). The linking is done in the main wiki administration -> Other -> Replication.

An instance will start by requesting another one using the URL leading to XWiki (for example https://mydomain.com/xwiki). This will make the current instance appear in the target instance administration as a "Requesting instance" which can then be accepted (or declined).

Linked instances also exchange keys to verify signed messages to make sure that the message really is sent by the instance it's claiming to come from.

Page replication

The main use case covered by the application is Page Replication.

Implement a new type of replication

The replication framework distribute "replication messages". Those messages are composed of the following:

• a unique identifier
• the date/time at which it was created
• the instance which is the initial source of the message
• a message type associated with the data
• custom binary data
• custom map of metadata

There are two main entry points for custom replication implementations:

• org.xwiki.contrib.replication.ReplicationSender: this component allows sending messages to specific (or all) registered instances
• org.xwiki.contrib.replication.ReplicationReceiver: a component role to implement in order to receive all messages associated with the message type indicate as component hint

Limitations

The application only replicate wiki pages (and a page likes) by default currently, and not everything is stored in wiki pages in XWiki. So using it as a backup solution is not recommended.

It's also not a good idea to replicate the XWiki top level space as it contains things which will break the other instance if replicated (like the URL setup of the instance and, more importantly, the global replication setup of the instance which has to be different in each instance).

Notifications replication

Notifications and notification preferences are not directly replicated currently.

But there are cases where you will get the same notifications on all instances (they are not technically replicated, but all instances appear to have created the same notifications on their own). In general, the instance in which a document is replicated behave as if the page was saved locally (except rare cases like, of course, not replicating back a page that was just received) so any page creation/modification/deleting which produce a notification locally will produce the same notification in the other instances too (provided the conditions are the same).

Here are such examples for page related notifications:

• pages you explicitly follow on several instances: notifications preferences control for which page you want to receive notifications, so you will get notifications if an instance has preferences indicating that you follow a given page even if that page came from a replication message
• auto watch: by default you automatically follow a page on which you made a major modification and same as for the previous point this will produce notifications preferences for a page exactly as if it was saved locally

Subwiki

It's possible to install the replication extension in subwiki but with important limitations: it's only possible to chose the replication instances from the main wiki administration. Moreover, the replication instance where to replicate the pages should have a subwiki with the exact same name for the replication to work properly.

See Logging.

Instance behind NGINX

By default, NGINX is breaking most replication messages because it's dropping all headers which contains an underscore (despite the fact that underscore is a perfectly valid character in HTTP headers). It's hard to fix this as it means introducing a breaking change to the protocol.

In the meantime, you can use the following workaround in your NGINX configuration:

underscores_in_headers on;

Generic architecture

See Architecture.

Troubleshooting

Restoring a previous backup

If you restore an instance to a previous state in time, there is a good chance that it's outdated from replication point of view (it won't contain modification made to replicated document after the backup date) and it would be updated automatically. However, in the administration, there is the Recovery tool to help with this use case: you indicate a date range between which you might have missing messages (generally between the backup date and now) and it will ask other instances to send missing content. 

Blocked replication queue

In a sending instance, each linked instance is assigned an independent queue, so they cannot block each other. But for a given instance, the Replication is always sending messages in the produced order and will keep trying to send a failing message until it pass, before trying other messages. From replication point of view, a successful message transfer imply that the message was stored in the disk of the receiving instance (so it was not necessarily interpreted yet, it's just waiting to be handled). In general a failed transfer should imply one of the following reasons:

• a network problem prevents from reaching the target instance configured URL
• the target instance cannot store the message on disk (no space left, etc.)
• the target instance cannot store the message metadata in the "event" Solr core
• the receiving instance did not allow the sending instance to send it messages (the signature is wrong and interpreted as a possible attack attempt, the sender instance is not registered at all from receiving instance point of view, etc.)

In the administration of the sender, you will find details about the queue:

• how message are waiting
• what made the last message fail

If the problem is on receiving side, you will also most probably find more details on why a message failed to be stored by looking at the log (a detailed error will always be logged when a received message could not be stored in the receiving queue).

Delete a message

While it's really not recommended, there is sometimes no other choice than to remove a message to unlock the situation (it would generally suggest a bug in the Replication Application or something extending it). There is nothing in the user interface for this, but it's still possible to do it:

• get from the administration of the sending instance the identifier of the message to remove (it should be visible in the details of the error: "Failed to send message with id [88cab0e1-f1f7-498f-a6f2-d88f55851edf]...")
• stop XWiki
• go to the permanent directory and the folder of the message <permdir>/replication/sender/<uri of the target instance>/<md5 hash of the message id> (you can open the file metadata.properties to make sure it's the right identifier)

Fully reset Replication setup

Replication related data are stored in the following locations:

• Disk: the private key (to sign messages), the remaining messages to send and the not yet handled received messages are stored on disk in <data>/replication
• Instances linking: all metadata related to others known replication instances is stored on the page XWiki.Replication.Instances
• Document state: metadata related to the state of replicated documents (its owner instance, its readonly and conflict statuses) is located in the database in the table replication_document
• Standard replication configuration: the configuration that indicates where exactly documents are replicated (when the standard controller is used) is located in the database in the table replication_entity_instances

Debug log

It's not always easy to understand the outcome when in complex scenarios, and of course there is always the possibility of bug in the replication extension of a custom controller.

To help understand better what exactly happens, it's possible to enable debug log to get more detail (on what happen to the replication message, for example).

The root packages to track all Replication Application debug log is org.xwiki.contrib.replication. There are several ways to enable debug log.

With the Logging Admin UI

Use Logging Admin UI from the Administration section, add set TRACE or DEBUG level classes located in package org.xwiki.contrib.replication.

With the Logback configuration file

You need to add the following in WEB-INF/classes/logback.xml:

<!-- Replication debugging -->
<logger name="org.xwiki.contrib.replication" level="trace"/>

After this change you will need to restart XWiki.

Prerequisites & Installation Instructions

We recommend using the Extension Manager to install this extension (Make sure that the text "Installable with the Extension Manager" is displayed at the top right location on this page to know if this extension can be installed with the Extension Manager). Note that installing Extensions when being offline is currently not supported and you'd need to use some complex manual method.

You can also use the following manual method, which is useful if this extension cannot be installed with the Extension Manager or if you're using an old version of XWiki that doesn't have the Extension Manager:

1. Log in the wiki with a user having Administration rights
2. Go to the Administration page and select the Import category
3. Follow the on-screen instructions to upload the downloaded XAR
4. Click on the uploaded XAR and follow the instructions
5. You'll also need to install all dependent Extensions that are not already installed in your wiki

Versions

Dependencies

Dependencies for this extension (org.xwiki.contrib.replication:replication-ui 2.3.9):

• org.xwiki.contrib.replication:replication-default 2.3.9
• org.xwiki.contrib.replication:replication-entity-default 2.3.9
• org.xwiki.contrib.replication:replication-entity-controller 2.3.9
• org.xwiki.platform:xwiki-platform-rendering-macro-velocity 14.10.7
• org.xwiki.platform:xwiki-platform-display-macro 14.10.7
• org.xwiki.rendering:xwiki-rendering-macro-html 14.10.7
• org.xwiki.platform:xwiki-platform-component-script 14.10.7
• org.xwiki.platform:xwiki-platform-administration-ui 14.10.7