Putting fulltext search outside the database is ideal for the reasons you stated.

However, it’s also not ideal for 3 reasons (people will have to weigh all the pros and cons for what’s good for their environment, as there’s no clear-cut “always” and “never” here…..):

0) The features of a [relational] database are taken away — most importantly, the ability to relate the text to other entities. You also take away sorting, but that’s easily fixable in OS’s, and perhaps most fulltext engines, such as Lucene*. A Google appliance is quite costly for just plain ol’ text searching. There are also companies out there that sell the service of searching your data quickly — Transparencee (http://www.transparensee.com/)* is one I’ve heard of.

This is very important, because it is an important point I make when talking about whether or not to put images in a database (see also http://sheeri.net/archives/39 ). One of the factors is easily being able to relate the data to each other — otherwise, you’re using a database to store information, not actually relate data to each other. If you want to just store data, you can put it all in flat files!

Being able to easily look up text is important, but so is being able to update and delete text that has some certain metadata associated with it — maybe that’s a user id, or a timestamp, etc. And honestly, it’s the same with images. Yes, there are costs to storing large BLOBS in the database, but that’s a factor, not an argument, because using a database simply for storage is a silly idea anyway.

1) In some cases you want to perform functions on the text in part or in whole. For example, encryption….

2) The data is in more than one place. This is similar to point 0 in that if you have all the data in the database, it’s all in one place. But it warrants having its own bullet point, because it’s more difficult to get a reliable point-in-time snapshots of the data for backup and recovery. You can’t replicate all the data, either.

I think it would be great if DBMS’ had API hooks, so you could choose whether or not to use the way they implement fulltext searching or not. That would add overhead, as the data and the search engine would have to communicate as opposed to the application going straight to the searching engine when it needs to. However, it would also make searching a black box, and not require application changes to use.

SQL Server already uses an external solution:

http://www.developer.com/db/article.php/3446891

It uses Microsoft’s search to look for data. However, this also means that it doesn’t automatically re-index when a change is made. (See the “Indexing Considerations” part at http://www.sitepoint.com/blogs/2006/11/12/sql-server-full-text-search-protips-part-1-setup/ )

And I know MySQL is working on a FULLTEXT solution that’s server-wide. Maybe they’ll include API hooks so we can drop in our own solution.

* note that Lucene and Transparencee aren’t products or companies Pythian/I endorse nor discourage — those are simply examples I’ve heard of, but never used myself.

Hey there,

If your comments are being dropped from our blog, rest assured I’m interested in that not happening. Please let me know if you’re willing to work with my team to get to the bottom of the issue.

Thanks

Paul
–
Paul Vallee, CEO, The Pythian Group, Inc.
http://www.linkedin.com/in/paulvallee

My email is vallee@pythian.com, reach out and we’ll figure out why your comments are being lost.

No we don’t - if you posted and it did not show up here, let me know and we can work together to figure it out. We only moderate for spam.

Paul

2. You cannot be RDBMS agnostic, pick whatever falls in your hands and consider fit for the job. Not after I have to run REPAIR TABLE on a 700+ mil rows MyISAM table, after some insert delay glitches.
It’s simply not professional to recommend something like this to your clients, knowing that might loose data and have downtimes.

2. Just look at http://www.postgresql.org/docs/8.3/static/release-8-3.html it’s compelling enough to make da move. Until it’s out … hang on, your Saviour is on the way. It’ll be a steep learning curve but I’m sure you’ll make it.

By the way, my day to day work is MySQL DBA. Hopefully, it’ll not be MySQL for a long time.
Heck, even SQLite is better than MySQL for the simple reason that it does what it states. Meeting expectations is everything.

Let’s assume for a minute that the full-text feature can be taken out of the existing database and the db migration objection can be cast aside.

You should know I’m no apologist for MySQL or even for Oracle for that matter. I’m all about the data, not the database platform. So…

I’m interested in arguments justifying building that full-text feature in the database at all, given the existence of the Google appliance. Assuming postgresql’s full-text features are so kickin’ compared to mysql’s, talk to us about that I’m eager to learn. Finally, if we’re gonna spend some money and it needs to be in a database, why wouldn’t we just spend $5000 on Oracle SE1 on a single quad-core CPU loaded with gads of RAM?

Paul