Seeking Information on Indexed Filesystems
Tonight I am catching up on older e-mails — here’s another question that came to me about 2 weeks ago from a user group member that I never had time to research and answer. I have directed the original author to this post so questions you pose in the comments can be answered.
Do you know anything about indexed file systems? I’m looking for a ‘nearline’ storage solution to help with data archiving.
We have a system which at it’s peak will be accepting 15 million short records / day. In order to keep the Web front end moving nicely, we want to drop data after about 3 weeks and shift it into a higher latency, higher capacity storage system. Indexed file systems seemed like a perfect solution for this. Ideally it would have a good front end to allow execution of arbitrary queries in some language (SQL would be nice).
The only thing I’ve been able to locate is an MS product called “Microsoft Index Server” and another product from a company called CopperEye. The MS product seems nice because it has an SQL front end, apparently. A reliable, easy to set up open source package would be ideal, though.
Do you know anything about these tools, or could you put me in touch with anyone who does?
March 19th, 2008 at 12:41 am
All file systems with indexes on them that maintain any form of consistency have all died. Everything else is asynchronous (e.g. Beagle, whatever Apple does) and is only really good for text search.
March 19th, 2008 at 7:20 am
I would take a look at SOLR, it is an open source text indexing app based around Lucene, it is very easy to set up and administer, the only down side is that it does not accept SQL but full text searches.
March 19th, 2008 at 12:13 pm
DId you look at filesystems that support extended attributes (e.g. NTFS and XFS)? There is no support for SQL though and the front-ends are probably not so great…
March 20th, 2008 at 7:28 am
I’d have a look at Sun’s Honeycomb project (available atm with StorageTek).
From the whitepapers I’ve read, it sounds exactly what you’re looking for, unless I missed something?
March 31st, 2008 at 6:33 pm
Folks,
Thanks for your comments and advice. I’ll take a look at the solutions you mentioned.
Thanks also to Sheeri for forwarding the query.
Nick