OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help



   Re: [xml-dev] indexing and querying XML (not XQuery)

[ Lists Home | Date Index | Thread Index ]

On Aug 23, 2005, at 12:41 PM, Robert Koberg wrote:

> Wolfgang Hoschek wrote:
>> A starting point may be MemoryIndex (http://dsd.lbl.gov/nux/api/  
>> index.html), which can be used for Lucene fulltext search over   
>> comparatively small *transient main memory* XML documents, for   
>> example as in Nux XQuery. Note, however, that it is not   
>> straightforward to extend XML fulltext quering over transient  
>> main  memory to XML fulltext quering over huge persistent XML  
>> document  collections; the underlying technology is bound to be  
>> vastly  different wrt. data management, indexing and  
>> transactional  properties, though the high level search API may  
>> indeed remain  indentical or similar.
> Cool. (I saw some of your posts on the lucene list).
> Does this require Saxon 8x?

The XQuery integration does, the  
org.apache.lucene.index.memoryindex.* package does not.
Actually, I've become a lucene committer recently, so you'll fine  
org.apache.lucene.index.memoryindex.* in the lucene SVN trunk as well.

> I have done some RAMDirectory searches, but for things like forum  
> posts that need to deny postings that contain 'bad' words (for a  
> young girls portal). But I need to search large collections of XML  
> config and content.
> The worst thing about this is updating the index, but it is not a  
> showstopper.
> Can you expand more on:
> "the underlying
> > technology is bound to be vastly  different wrt. data management,
> > indexing and transactional  properties"
> it would seem similar to me. That is, if the instance XML changes  
> you have to open the reader, delete the Lucene document, reindex  
> the new document, add it, optimize and close the reader.

If all you need is to index the document's flat text instead of XML  
documents with their structure, that's straighforward, yes.

But presumably you'll want to combine structured search (e.g. XPath  
navigation and predicates) with unstructured fulltext search, and now  
you're in database terrain, wrt. choosing cost effective persistent  
index data structures and execution plans for a mix of the main  
expected queries/data types/access patterns/read&write frequency,  
plus static and/or dynamic query optimization, including materialized  
view maintenance, transactional updates, etc. All well known  
problems, now in the context of XML and fulltext search, but without  
easy solutions nonetheless.



News | XML in Industry | Calendar | XML Registry
Marketplace | Resources | MyXML.org | Sponsors | Privacy Statement

Copyright 2001 XML.org. This site is hosted by OASIS