如何使用 Lucene 获取频繁出现的短语
我想用 Lucene 获得一些经常出现的短语.我从 TXT 文件中获取了一些信息，并且由于没有短语信息而丢失了很多上下文，例如信息检索"被索引为两个单独的词.
I would like to get some frequently occurring phrases with Lucene. I am getting some information from TXT files, and I am losing a lot of context for not having information for phrases e.g. "information retrieval" is indexed as two separate words.
有什么方法可以得到这样的短语?我在互联网上找不到任何有用的东西，所有的建议、链接、提示，尤其是示例，都非常感谢！
What is the way to get the phrases like this? I can not find anything useful on internet, all the advices, links, hints especially examples are appreciated!
我只按标题和内容存储我的文档:
I store my documents just by title and content:
Document doc = new Document();
doc.add(new Field("name", f.getName(), Field.Store.YES, Field.Index.NOT_ANALYZED));
doc.add(new Field("text", fReader, Field.TermVector.WITH_POSITIONS_OFFSETS));
因为对我来说最重要的是文件的内容.标题通常根本不具描述性(例如，我有许多 PDF 学术论文，其标题是代码或数字).
because for what I am doing the most important is the content of the file. Titles are too often not descriptive at all (e.g., I have many PDF academic papers whose titles are codes or numbers).
我迫切需要从文本内容中索引出现频率最高的短语，刚才我发现这种简单的词袋"方法效率不高.
I desperately need to index top occurring phrases from text contents, just now I see how much this simple "bag of words" approach is not efficient.
推荐答案
Julia，看来你要找的是 n-grams，特别是 Bigrams(也称为搭配).
Julia, It seems what you are looking for is n-grams, specifically Bigrams (also called collocations).
这里有一个 关于寻找搭配的章节 (PDF)，来自 Manning 和 Schutze 的 统计自然语言处理基础.
Here's a chapter about finding collocations (PDF) from Manning and Schutze's Foundations of Statistical Natural Language Processing.
为了使用 Lucene 做到这一点，我建议使用 Solr 和 SingleFilterFactory.有关详细信息，请参阅此讨论.
In order to do this with Lucene, I suggest using Solr with ShingleFilterFactory. Please see this discussion for details.
相关文章