• Shawn Heisey
    Shawn Heisey, 2017/03/17 21:17

    This page includes RemoveDuplicatesTokenFilterFactory in the example analyzer and claims that the three "flip" terms will be reduced to one term by that filter.

    That is NOT what happens.  That filter will only remove duplicates if they have the same *position*, which normally only happens with things like the synonym filter.

    After running through the standard tokenizer and the snowball filter, the three "flip" terms will have different positions.  This screenshot shows what happens if the exact analyzer documented on this page is used with "flip flipped flipping" as the input:

    https://www.dropbox.com/s/k4hbtzkthse122i/xwiki-remove-duplicates-analysis-results.png?dl=0

    Documentation for the filter, which does say that terms have to have the same position to be considered duplicates:

    https://cwiki.apache.org/confluence/display/solr/Filter+Descriptions#FilterDescriptions-RemoveDuplicatesTokenFilter

    There is no filter included with Solr or Lucene that will remove duplicate terms at different positions.

  • Hassan Ali
    Hassan Ali, 2020/03/16 13:34

    It took me some time to find "Solr Administration" section. It's in the Global Administration and title is different (see my attachment:chrome_SKEI1bdXLy.png)

Get Connected