I am not precisely sure, but you should find something along the lines of a pipeline of processor classes that you can configure in Lucene that do a bunch of stuff during indexing, such as possibly stemming etc. You should be able to write your own class that conforms to the same interface as these processors and should be able to configure it as one of the stages in the pipeline.

Like I said, this is only a guess, but an educated one :). See how you go with that. You can find books on Lucene (there are a few around) to help you out, look around. Hope this helps.

I have a question:
I am working on a local search engine for master’s thesis. I configured “Nutch” (this is an open source search engine) and am able to pay with it. I found Nutch automatically indexing the documents it has crawled. But I want to do some preprocessing on the crawled documents before indexing. So can u please tell me how to go about (if u have any exposure of it)? Or can you tell me which group to join for discussion? Because that will greatly simplify my job.

Thanking u in advance and waiting for ur response soon!
-Tesfaye

I am glad you’ve been enjoying it. Yeah, it’s amazing just how much goes on behind the scenes when it comes to how search engines work. It is reasonably easy to set up a search system using something like Lucene, but this kind of deeper knowledge really helps when you want to tweak it to perform better, as well as debugging issues.

Thank you for this great article, it is written nicely.

I have done several search engine implementation using Lucene but these posts made me fully aware of all the decisions involved in constructing a search engine and I am looking forward to your next article.

Best regards,
Bert

I don’t know of a pure Ruby lemmatizer, but this might be helpful http://english.rubyforge.org/. It contains a stemmer as well as a lot of other language utilities.

I’m currently building a small search engine using Ruby on Rails and I found your article extremely helpful.

Now, do you know any good english lemmatizer written in Ruby?

Cheers,
Kartik