[ lucene扩展 ] "Did you mean" feature with Apache Lucene Spell-Checker

转自：http://www.javacodegeeks.com/2010/05/did-you-mean-feature-lucene-spell.html

Google's "Did you mean" feature
After making an introduction to Lucene in a previous post, now it is time to take it up a notch and create a more sophisticated application. You are most surely familiar with Google's "Did you mean" feature (other search engines support this too). Here is an example of that:

Lucene SpellChecker Subproject
This feature can be implemented in Java using Lucene project and this tutorial will show you how. The implementation will be based on one of Lucene's sub-projects, named SpellChecker (see spell checker API). The inspiration for this post was a recent article that I read at InfoQ named “Implementing Google's "Did you mean" Feature In Java”. The author shortly describes the edit distance algorithms that are used and then provides a simple code example. I also found a similar article named “Did You Mean: Lucene?”, but this was written back in 2005, so I guess it is now quite outdated. It is a good idea to first take a look on those and then come back for some action.

Finding the base dictionary
For the purposes of this tutorial, an English dictionary text file will be used. You can perform a simple search for that or you can directly download this zipped file. The file is very simple and just contains the words separated by lines. The extracted file is named “fulldictionary00.txt”.

Using the Lucene SpellChecker API
Let's get started by creating a new Eclipse project, perhaps under the name “LuceneSuggestionsProject”, and import the necessary JAR files. Except for the main JAR lucene-core-3.0.1.jar, we will also need the lucene-spellchecker-3.0.1.jar. This can be found in the extracted tarball in the folder “lucene-3.0.1\contrib\spellchecker”. In case you don't know how to find the distribution and get started with the project, please check my previous post (An Introduction to Apache Lucene for Full-Text Search). Create a new class named “SimpleSuggestionService” and make sure a main method is included. The source code for this class is the following:

package com.javacodegeeks.lucene.spellcheck;

import java.io.File;

import org.apache.lucene.search.spell.PlainTextDictionary;
import org.apache.lucene.search.spell.SpellChecker;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;

public class SimpleSuggestionService {
    
    public static void main(String[] args) throws Exception {
        
        File dir = new File("c:/spellchecker/");
        
        Directory directory = FSDirectory.open(dir);
        
        SpellChecker spellChecker = new SpellChecker(directory);
        
        spellChecker.indexDictionary(
             new PlainTextDictionary(new File("c:/fulldictionary00.txt")));
        
        String wordForSuggestions = "hwllo";
        
        int suggestionsNumber = 5;

        String[] suggestions = spellChecker.
            suggestSimilar(wordForSuggestions, suggestionsNumber);

        if (suggestions!=null && suggestions.length>0) {
            for (String word : suggestions) {
                System.out.println("Did you mean:" + word);
            }
        }
        else {
            System.out.println("No suggestions found for word:"+wordForSuggestions);
        }
            
    }

}

 

Yes, it is as simple as that. If you run the application (using “hwllo” as the input word), the following output will occur:

Explaining the SpellChecker API
The code is very straightforward, but I will elaborate on some of the classes. To begin with, you need a Directory, which is a flat list of files. This class is abstract so we use FSDirectory as the actual implementation. FSDirectory stores the index files in the file system (for faster access RAMDirectory, which is the memory resident implementation, should be evaluated). The next step is to create a SpellCheckerinstance, which is the heart of our application. Note that the default StringDistance is the LevensteinDistance. If you desire to use an other algorithm (JaroWinklerDistance or NGramDistance), you can use the setStringDistance method. Next, we instantiate a PlainTextDictionary, which points to the text file we downloaded. The dictionary is indexed and then the SpellChecker object is used to retrieve the suggested words for a misspelled word.

I guess you are now ready to create the next big search engine. You can find the relevant Eclipse project here.

Enjoy!

Read more: http://www.javacodegeeks.com/2010/05/did-you-mean-feature-lucene-spell.html#ixzz1mKUgB5Mh