Are you aware of an existing DBA opening or consulting requirement in your organization? Enter your email for a chance to win one year's access to Safari Books.
Seeking Information on Indexed Filesystems
Tonight I am catching up on older e-mails — here’s another question that came to me about 2 weeks ago from a user group member that I never had time to research and answer. I have directed the original author to this post so questions you pose in the comments can be answered.
Do you know anything about indexed file systems? I’m looking for a ‘nearline’ storage solution to help with data archiving.
We have a system which at it’s peak will be accepting 15 million short records / day. In order to keep the Web front end moving nicely, we want to drop data after about 3 weeks and shift it into a higher latency, higher capacity storage system. Indexed file systems seemed like a perfect solution for this. Ideally it would have a good front end to allow execution of arbitrary queries in some language (SQL would be nice).
The only thing I’ve been able to locate is an MS product called “Microsoft Index Server” and another product from a company called CopperEye. The MS product seems nice because it has an SQL front end, apparently. A reliable, easy to set up open source package would be ideal, though.
Do you know anything about these tools, or could you put me in touch with anyone who does?
Mar 18, 2008
Category: MySQL, Non-Tech Articles, Not on Homepage
Tags: indexed file server, indexed fileserver, MySQL
All file systems with indexes on them that maintain any form of consistency have all died. Everything else is asynchronous (e.g. Beagle, whatever Apple does) and is only really good for text search.
I would take a look at SOLR, it is an open source text indexing app based around Lucene, it is very easy to set up and administer, the only down side is that it does not accept SQL but full text searches.
DId you look at filesystems that support extended attributes (e.g. NTFS and XFS)? There is no support for SQL though and the front-ends are probably not so great…
I’d have a look at Sun’s Honeycomb project (available atm with StorageTek).
From the whitepapers I’ve read, it sounds exactly what you’re looking for, unless I missed something?
Folks,
Thanks for your comments and advice. I’ll take a look at the solutions you mentioned.
Thanks also to Sheeri for forwarding the query.
Nick