Seeking Information on Indexed Filesystems

Mar 18, 2008 / By Sheeri Cabral

Tags: , ,

Tonight I am catching up on older e-mails — here’s another question that came to me about 2 weeks ago from a user group member that I never had time to research and answer. I have directed the original author to this post so questions you pose in the comments can be answered.

Do you know anything about indexed file systems? I’m looking for a ‘nearline’ storage solution to help with data archiving.

We have a system which at it’s peak will be accepting 15 million short records / day. In order to keep the Web front end moving nicely, we want to drop data after about 3 weeks and shift it into a higher latency, higher capacity storage system. Indexed file systems seemed like a perfect solution for this. Ideally it would have a good front end to allow execution of arbitrary queries in some language (SQL would be nice).

The only thing I’ve been able to locate is an MS product called “Microsoft Index Server” and another product from a company called CopperEye. The MS product seems nice because it has an SQL front end, apparently. A reliable, easy to set up open source package would be ideal, though.

Do you know anything about these tools, or could you put me in touch with anyone who does?

5 Responses to “Seeking Information on Indexed Filesystems”

  • All file systems with indexes on them that maintain any form of consistency have all died. Everything else is asynchronous (e.g. Beagle, whatever Apple does) and is only really good for text search.

  • Francois Schiettecatte says:

    I would take a look at SOLR, it is an open source text indexing app based around Lucene, it is very easy to set up and administer, the only down side is that it does not accept SQL but full text searches.

  • Ilias says:

    DId you look at filesystems that support extended attributes (e.g. NTFS and XFS)? There is no support for SQL though and the front-ends are probably not so great…

  • Oscar Rylin says:

    I’d have a look at Sun’s Honeycomb project (available atm with StorageTek).
    From the whitepapers I’ve read, it sounds exactly what you’re looking for, unless I missed something?

  • NickP says:

    Folks,

    Thanks for your comments and advice. I’ll take a look at the solutions you mentioned.

    Thanks also to Sheeri for forwarding the query.

    Nick

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>