[
Lists Home |
Date Index |
Thread Index
]
On Aug 23, 2005, at 12:41 PM, Robert Koberg wrote:
> Wolfgang Hoschek wrote:
>
>> A starting point may be MemoryIndex (http://dsd.lbl.gov/nux/api/
>> index.html), which can be used for Lucene fulltext search over
>> comparatively small *transient main memory* XML documents, for
>> example as in Nux XQuery. Note, however, that it is not
>> straightforward to extend XML fulltext quering over transient
>> main memory to XML fulltext quering over huge persistent XML
>> document collections; the underlying technology is bound to be
>> vastly different wrt. data management, indexing and
>> transactional properties, though the high level search API may
>> indeed remain indentical or similar.
>>
>
> Cool. (I saw some of your posts on the lucene list).
>
> Does this require Saxon 8x?
The XQuery integration does, the
org.apache.lucene.index.memoryindex.* package does not.
Actually, I've become a lucene committer recently, so you'll fine
org.apache.lucene.index.memoryindex.* in the lucene SVN trunk as well.
>
> I have done some RAMDirectory searches, but for things like forum
> posts that need to deny postings that contain 'bad' words (for a
> young girls portal). But I need to search large collections of XML
> config and content.
>
> The worst thing about this is updating the index, but it is not a
> showstopper.
>
> Can you expand more on:
>
> "the underlying
> > technology is bound to be vastly different wrt. data management,
> > indexing and transactional properties"
>
> it would seem similar to me. That is, if the instance XML changes
> you have to open the reader, delete the Lucene document, reindex
> the new document, add it, optimize and close the reader.
>
If all you need is to index the document's flat text instead of XML
documents with their structure, that's straighforward, yes.
But presumably you'll want to combine structured search (e.g. XPath
navigation and predicates) with unstructured fulltext search, and now
you're in database terrain, wrt. choosing cost effective persistent
index data structures and execution plans for a mix of the main
expected queries/data types/access patterns/read&write frequency,
plus static and/or dynamic query optimization, including materialized
view maintenance, transactional updates, etc. All well known
problems, now in the context of XML and fulltext search, but without
easy solutions nonetheless.
Wolfgang.
|
