Pythian Group

Notice to readers:
This is an excerpt of my liveblogging of the Keynote where the Exadata Storage Server was announced.

It is a mix of my comments in real time, and my quotes from things Larry Ellison said that I felt were worthy of mention.

You may be interested in reading more about Oracle’s Exadata platform. I would suggest taking a gander at Oracle’s product page and also reading up on Christo Kutrovsky’s Analysis of the Exadata and Oracle Database Machine announcement from a different point of view.

You may also be interested in the complete liveblog transcript of the keynote which includes my liveblogging of the rather boring HP advertiseynote before the big show.

I also separated out the Oracle Database Machine liveblog if you just want to get to the rest of the juicy stuff.

So here goes with the liveblogging transcript:

Announcing Oracle’s first ever hardware product.

The exadata programmable storage server.

Building intelligence into the storage server.

Allows us to reduce the amount of data.

Confirming HP is the partner.

Storage server does not pass disk blocks back to the database server, it actually passes query results.

Note: A few startups are doing this sort of thing already. They should be totally freaking out right now.

Slide explaining how query processing works in traditional storage.

Stark contrast to a grid of exadata storage servers, with processing ability local to each and every disk drive.

“We actually pass the query from the database server directly into the storage servers.”

Explaining how this works.

This reminds me hugely of kickfire for mysql, but for Oracle.

Cool, they’re provisioning two infiniband pipes per storage server. Nice. 40gbps.

Marc Fielding: “The problem is still the disk drives.”

Larry: 1gb/s per exadata storage server, you can have dozens working in parallel.

Christo: 1gb/s??? that’s not that good.

Immediately available for Linux., will work with any Oracle database.
Available for x86. Christo: “not 64 bit???????”

By the way, I made a bet with Paul Cunningham that Kickfire would fail because of Moore’s law. I wonder if I should bet against this tech too.

More details to come…

I’ll be liveblogging the keynote here along with Marc Fielding, Christo Kutrovsky, Darrin Leboeuf and Luke Davies and Martin Wisniewski.

First observation: I can not believe I can not make this video full screen.

Even Pythian’s flash video player can go full screen.

Introduction time. Safra Katz.

Confirming HP is here as part of the announcement.

So presentation from HP is here first. Weird. Maybe this is a megabucks commercial keynote. Introducting Exec VP Anne Livermore.

“Very very very very very important event for HP. 100000 joint customers.”

Show of hands for HP adoption. A bit lame, she seems disappointed with how many hands are raised. :-)

The video stream is super saturated, the video is super skippy. Hopefully I can understand what’s going on well enough to do this liveblog.

So far it’s completely a vendor advertisement keynote.

“By 2010, more than 1/3 of CEOs and CIOs realize their datacenters will not meet their demand”.

This is not that big a deal according to Marc Fielding. Basically, Duh, 2/3 of datacenter CEOs do not need to invest further to make it through two years, 1/3 does. I agree.

Ugh another advertising slide. I hope there’s something meaty here.

One in six HP servers runs Linux. That’s interesting.

Bragging about HP winning vs. EMC.

HP would be the sixth largest software company in the world if you looked at only software revenue (interesting)

“Transforming the data center”

“True 24×7 lights out automated and energy efficient data center”

I am IMing with Alex Gorbachev who is at the keynote floor.

I asked him:

Paul Vallee
5:44
do you think HP is only going to advertise here?
5:41 PM

or are they part of this x

oracloid11g
5:44
I think they are

I think it’s crazy for Larry to allow HP to announce it then. He will come onstage with them?

Oh my god she’s actually playing an advertisement.

“next generation data center”

Christo: “I think we were too optimistic, thinking it was going to some amazingly cool piece of Oracle software. I’m starting to think we will be disappointed and it will be hardware.”

Alex Gorbachev is twittering live: http://twitter.com/alexgorbachev.

Darrin Leboeuf: “Maybe Oracle bought HP’s services arm!!! LOL”

HP spends 2% of revenue on IT. Interesting. Consolidated 60+ datacenters into 6.
BTW among Pythian’s customers that share such data, I am aware of one large company spending 1.4%.
Not as large as HP though. :-) Maybe there is a cost to scale.

Talking about virtualization now. Still completely unrelated to X as far as I can see.

So far this looks like one of those million dollar advertising keynotes.

29 minutes of this and I’m still awake. I’m pretty proud.

LOL silence…. then “can i have the next chart please”

Christo Kutrovsky: “Can we have the big news now?”
Darrin Leboeuf: “Can we have the next slide please??? please? Next?”

Now she’s talking about EDS. I missed the point as to why?

I guess they’re partnering with EDS on services. All-righty then. This is also not keynote-worthy material.

OK I get it thanks to Martin. HP bought EDS. How did I miss this news? :-)

17:56 Shilling for EDS’ outsourcing. Here’s a tip, choose Pythian instead.

“Performance Optimized Datacenters, our PODs”. Clever.

So HP is following Sun’s lead in container-based datacenters. Way to go.

17:57 Alex Gorbachev: “So boring.”

(Christo suggested I timestamp. All times EST.)

HP working on eliminating copper in computers, replacing with laser+optical, as a power-saving play. Interesting.

18:00 “to wrap up…” woo hoo

Everyone here in my office left except Christo and I. That should paint a picture.

Polite applause.

OK, Larry’s sailboat is on the screen.

“Extreme Performance” . It’s showtime.

Lots of sailboat visuals. Larry loves his sailboat. He seriously loves it. We get it.

18:03 “Ladies and gentlemen, Larry Ellison”

Wow now he’s talking about sailing. Incredible.

OK I’m giving up the sarcasm. The commercial is over and I’m getting ready to get excited.

“next slide please” wow this has not been rehearsed much. Probably because of the secrecy.

Looks like this will be about VLDB. Databases are tripling in size every two years.

“Disk systems today simply can not cope with the amount of data that has to be moved off those drives. We have a huge bandwidth problem”.

“You don’t have to have a 200tb db to start experiencing the slowdowns. 1tb is the elbow of the curve [pretty graph]”

Darrin and Marc are back. I guess it’s getting interesting again.

Two possible solutions:
1. reduce data going through to the storage systems.
2. wider, faster pipes, and more of them.

(reminds me of the bahd).

Announcing Oracle’s first ever hardware product.

The exadata programmable storage server.

Building intelligence into the storage server.

Allows us to reduce the amount of data.

Confirming HP is the partner.

Storage server does not pass disk blocks back to the database server, it actually passes query results.

Note: A few startups are doing this sort of thing already. They should be totally freaking out right now.

Slide explaining how query processing works in traditional storage.

Stark contrast to a grid of exadata storage servers, with processing ability local to each and every disk drive.

“We actually pass the query from the database server directly into the storage servers.”

Explaining how this works.

This reminds me hugely of kickfire for mysql, but for Oracle.

Cool, they’re provisioning two infiniband pipes per storage server. Nice. 40gbps.

Marc Fielding: “The problem is still the disk drives.”

Larry: 1gb/s per exadata storage server, you can have dozens working in parallel.

Christo: 1gb/s??? that’s not that good.

Immediately available for Linux., will work with any Oracle database.
Available for x86. Christo: “not 64 bit???????”

By the way, I made a bit with Paul Cunningham that Kickfire would fail because of Moore’s law. I wonder if I should bet against this tech too.

Second product announcement: The Oracle Database Machine (in partnership with HP).

Specs slides.

8 64-bit servers, 14 exadata storage servers, tons of ram.

Larry: “It will hold really a lot of songs”.

Three year development program.
Custormers:
Amazon, Yahoo, Countrywide, NPD, Quelle

M-Tel: a Bulgarian company, 10-72 times speedup. The worst speedup was 10x.

Alex G: “It’s mainframes!”

Larry: “Next slide”. Martin W: “why doesn’t he have the clicker himself!?”

Christo (a bulgarian) “I can’t believe M-Tel had this and managed to keep it a secret. That’s funny.”

28x P-Series competitive advantage at M-Tel. For half of one.

Darrin L: “Yeah but what’s the price difference. How do you license this!?”

Good questions IMHO. Oracle licenses based on server performance. There is a major problem to be solved there.

TPC-H query set. 30x average speed-up.

They have a paper describing it. Will be intersting to read it.

Christo: “This seems to be parallel-query only! It might be only for data warehouses!”

Larry: “With a conventional array, when you add storage, you don’t add data bandwidth. With this solution, every time you add storage server, you are adding not just disk capacity, but two infiniband pipes, two processors, and more cache”.

Makes Sense. Refer to BAHD again. Man I feel a bit smart right now.

Now bashing Teradata. I guess we’re into the advertising section.

Now Larry is saying how similar this is to Netezza. One processor per disk drive. Christo is saying it’s very similar.

Larry: two big differences:
1. our database machine runs oracle, theirs does not.
2. we lose a drive, ours keeps running, theirs does, their queries stop.
Christo “umm no they don’t they have three way mirroring”.

Great quip: “even I studied about B-Tree indexes in School!” LOL LOL

Slide – comparing vs. Netezza. Clearly this is the company they are targeting.
Oracle’s stuff is bigger and faster and has more cores and faster bandwidth.

Christo: “The question is, does that CPU sort.”

HP Oracle … 650,000 vs. 1.500,000 for Teradata system with less spec.

In the Oracle column, you need to pay the 1.7 mm software license.

Software license for Oracle though… 1.7mm. There is something interesting going on here.

“Even if you pay list for Teradata, it’s cheaper for capacity.”

Next slide……..

Will speed up OLTP as well as data warehousing.

Available today, 10-50x faster than current (in small type) oracle data warehouses”

I must have missed something. How is this not costing more than Teradata?

OK Christo explained, the Oracle system had triple the storage for modestly more cost.

Mark Hurd talking again. Can HP please hire somebody exciting? Maybe exciting is just not what they do.

Adds the fact that these storage servers are “completely open, proliant-based servers”.

I wonder how long this will be HP-only.

Next, I think you’ve listened to me for long enough, now take a moment and listen to this advertising interlude.

My comments: This is definitely a premium play. It will be interesting to follow the developments. It is not at all what I thought it would be, but it is fascinating.

I bet against Kickfire with Paul McCullagh because of Moore’s Law limiting the long-term value of the speed-up. Maybe I was wrong, Larry is betting a lot more money that I am.

Christo: Let me sum it up: This is parallel query being pushed down to the disk.

Christo: I have two words for you: Object Checkpoint.
Basically before you run a parallel query on any object, you have to checkpoint it. This is already there in 10g and is an enabling technology.

Darrin: When you start up an instance, does it start up an instance on the disk server?
Christo: Doesn’t need it, just needs a filter server.

Christo: The question is, does it sort???????? This is extremely important.

Christo: My first guess was a SAN. But I never imagined Oracle would start having a hardware play. (Or two).

Christo: They got tired of people not buying bandwidth, so they’re forcing people to buy it now. Smart.

I’m sorry if everyone else knew this already, but I just noticed the following from the Live Keynote page (click on Larry’s keynote to see it):

So Larry will be joined on stage by bigwigs from HP.

You will remember that HP bought Polyserve, Kevin Closson’s clustered file systems company, a little while back.

No coincidence, I think, but as we know from multiple sources, Kevin is now at Oracle as an Architect on this new technology that Larry is announcing.

At first I thought the HP keynote was one of those big-pay vendor closers. But now, with it being one session in the Live View, and what we’re starting to learn about the likely nature (storage) of the innovation, the following snippet can easily be re-interpreted (interesting bit in my bold):

Transforming Business and Technology Today and Tomorrow
Innovation is the lifeblood of information technology, but businesses are far more selective today in the kinds of technology they will buy and deploy to ensure they remain competitive. They’re looking for practical innovation that will optimize business results such as lowering IT costs, reducing risk and improving growth and profitability. Come learn about HP’s customer-focused innovation, including investments in research, product development and advanced services that have increased energy efficiency, provided new approaches to datacenter transformation, and also given us new advances in internet technology that are shaping the future of enterprise IT. Also highlighted: the joint innovation that HP and Oracle are delivering to their customers around the world.

Ann Livermore | Executive Vice President, Technology Solutions Group, HP | Biography [+/-]
Mark Hurd | Chairman of the Board and CEO, HP

The pieces are coming together. Stay tuned as I will be writing further about this as the keynote begins.

This is the 114^th edition of Log Buffer, the weekly review of database blogs.

I am sorry to say that this log buffer was supposed to be edited by Dave Edwards, but he’s suffering from severe and long-lasting tooth pain and until his root canal is done he’s KO’d by a killer combo of painkillers and the pain that the painkillers can’t kill. I’ve been there myself, twice, and here’s a tip Dave. It hurts until the dentist takes out the needle. Then the pain goes away while he digs. The pain comes back that night. The next morning it’s worse than ever, unbelievably, writhingly bad. But later that afternoon, blisssssssssssss. :-) Good luck man.

This Log Buffer has been generated in a completely automated way with the help of the incredibly awesome AideRSS.

To give you an idea of just how awesome it is, I was able to load up Dave’s complete OPML file of all the blogs he monitors for Log Buffer. And AideRSS applied it’s magical PostRank algorithm which scores blog posts based on how many comments, del.icio.us bookmarks, blog links from other blogs, etc. that it received, along with some more secret sauce they don’t publicly tell us about (kind of like Google with their Pagerank equivalent). The number to the left of each headline represents the linked item’s AideRSS PostRank.

It did a great job of automatically selecting the best posts from the last week.

To give you an idea of AideRSS’s helpfulness, here are a couple useful feeds I suggest you subscribe to:

1. PlanetMySQL, but only with posts that rank “Best”
2. OraNA.info, again only the posts that rank “Best”. Note that there is a bug in Eddie’s feed that makes it impossible to use all possible information on the ranking.
3. SQLBlogs.com processed by AideRSS to show only the best posts.

While I have no idea how AideRSS plans to make moolah, I think we can agree that is some kind of awesome if you’re like me and can’t afford to miss a big story, but can’t afford the time you would need to read it all. Many thanks to Andrew Baldwin and although that’s the AideRSS about page there there’s a good pic of Andrew on that page. I first met Andrew at MySQLConf 2008 this spring and he’s a great guy and a great advocate for this service.

10.0 – Random selection, with a bias ..

Read the rest of this entry . . .

All,

I’m writing to help get the word out that Microsoft announced a major security vulnerability in GDI+, a component that is included and vulnerable to remote code execution exploits in every supported release of SQL Server 2005.

You can find our more about the vulnerabilities and affected products (there’s a long list, not just SQL 2005) at the Microsoft announcement here.

There is an update already available, so you probably want to evaluate an accelerated deployment of that. If you are a current Pythian client, we’ll be reviewing this patch for you. If you are not, now would be a good time to sign up, and Michelle will take care of you. :-)

Snippet from the announcement:

Executive Summary

This security update resolves several privately reported vulnerabilities in Microsoft Windows GDI+. These vulnerabilities could allow remote code execution if a user viewed a specially crafted image file using affected software or browsed a Web site that contains specially crafted content. Users whose accounts are configured to have fewer user rights on the system could be less impacted than users who operate with administrative user rights.

This security update is rated Critical for all supported editions of Windows XP, Windows Server 2003, Windows Vista, and Windows Server 2008, Microsoft Internet Explorer 6 Service Pack 1 when installed on Microsoft Windows 2000 Service Pack 4, Microsoft Digital Image Suite 2006, SQL Server 2000 Reporting Services Service Pack 2, all supported editions of SQL Server 2005, Microsoft Report Viewer 2005 Service Pack 1 Redistributable Package, and Microsoft Report Viewer 2008 Redistributable Package.

This security update is rated Important for all supported editions of Microsoft Office XP, Microsoft Office 2003, 2007 Microsoft Office System, Microsoft Visio 2002, Microsoft Office PowerPoint Viewer 2003, Microsoft Works 8, and Microsoft Forefront Client Security 1.0. For more information, see the subsection, Affected and Non-Affected Software, in this section.

The security update addresses the vulnerabilities by modifying the way that GDI+ handles viewing malformed images. For more information about the vulnerabilities, see the Frequently Asked Questions (FAQ) subsection for the specific vulnerability entry under the next section, Vulnerability Information.

Recommendation. Microsoft recommends that customers apply the update immediately.

Known Issues. Microsoft Knowledge Base Article 954593 documents the currently known issues that customers may experience when installing this security update. The article also documents recommended solutions for these issues.

I tripped over an old oracle-l exchange (not that old, from March of this year) and I thought it would make good content for a blog post on the critical difference between outsourcing and offshoring.

It started when Ethan Post posted a link to this fascinating story at the Ludwig von Mises Institute about how the U.S. dollar’s collapse affects the outsourcing industry. As many of these posts do, the idea of outsourcing gets conflated with that of offshoring. What the author really means to say is that the “downward dollar delivers a blow to offshoring“, not outsourcing.

Let me explain further. I am now cribbing shamelessly from my oracle-l post and so if you read this already this spring, my apologies.

Ethan had posed the following question:

Interesting article on the effects of the dollar’s fall on outsourcing. Would be interesting to hear a few of you who are perhaps feeling these effects to comment.

To which I replied:

Our margins were definitely squeezed painfully from April 07 until late last year (follow that link to see a 20% or so decline in the USD/CAD exchange, and remember that a substantial chunk of Pythian’s costs are in CAD and about 70% of our income is in USD). So it hasn’t been really that much fun adjusting to our new currency realities. That being said, I think there is a meaningful difference between offshorers and outsourcers and that these different ideas get conflated a lot, including in this case.

If your company’s entire business model is simply shifting work from a country where wages are high to a country where wages are lower, you have two problems. First, you are very vulnerable to this type of currency shift because it is at the core of your profit model. This is called labour arbitrage. Second, as time marches on and we continue our trend to a global rate for any given IT service, your company will cease to have any reason to exist. This article from the Economist covers general India inflation, trust me focus on labour inflation in the information technology sector and the situation is much, much worse.

However, outsourcing properly conceived can be highly successful even when the resources are hired locally to the market being targeted; meaning without relying on exchange rate differences nor differences in global payscales. These companies are successful because by concentrating expertise, adopting best practices, innovating and reusing work they have found efficiencies that add up to more than their direct costs of services delivery + overhead (meaning, that profit model has nothing to do with currency or wage geography).

I would count Pythian among companies conceived along those lines, we are far from alone, certainly our direct competitors based only in the U.S. dbaDirect, Contemporary and DCC (hi guys! well done!) do not rely to any degree on wage differences or exchange rates as parts of their profit models. To put things into perspective, although Pythian has a presence in 10 countries now (including four U.S. cities), our profit model also has innovation, expertise, scale and re-use as it’s heart and soul, not labour arbitrage.

So anyway, we’ve successfully adjusted over here and I think offshoring will lose a lot from the collapse of the USD, but these companies that are offshorers only will be forced to morph into something better to survive, which is good for everyone.

I am traveling to Europe next week to brief major prospects in Germany (Daimler, MAN) as well as to attend to administrative matters at Pythian Europe in Prague and would love to meet any readers of this blog during this trip!

I’m especially interested in meeting:

• DBAs, Applications Administrators and Systems Administrators,
• Potential customers (IT Directors, DBA Managers, Supply Managers for IT), and
• Potential partners (IT product of service companies that could partner with Pythian to delight our mutual customers)

Here is my itinerary:

• Sunday, August: Frankfurt,
• Monday, August 4: Stuttgart,
• Tuesday, August 5: Munich, and
• Wednesday, August 6 through Saturday, August 9: Prague, Czech Republic.

Please reach out to me using vallee@pythian.com if you would like to meet!

I was happy to be invited by Brian Prince at eWeek to answer some questions he had posed to Pythian, NTirety and industry analysts Noel Yuhanna of Forrester and Peter O’Kelley of the Burton Group.

You can take a look at the end result here: How to Decide if Remote Database Admins are Right for you.

I found the process interesting. I had actually provided a lot more content, which I include below, and I strenuously disagree with some of the analyst statements, especially the statement that the processes must be totally licked before engaging an external vendor. Are all the other vendors staffed with rookies following established processes and that’s it? It’s a very strange statement to make given that many, most of our customers turn to us to help them define best processes in terms of capacity planning, availability optimization, and security. It’s specifically contradicted by Corey’s statement about how his shop optimizes and automates processes. We do the same thing, of course, and we routinely inherit shops with tons of low-hanging fruit where we can dramatically streamline the efforts.

Then again, maybe Noel is thinking of dbaDirect. If I may say, “eeks!”. Anyway, not all companies in this space are made alike, I guess.

I think it would be interesting for most to read the article linked above, and compare with the following answers to his questions that I provided to Brian, mostly because it generates some respect for the challenge of cutting down content, and also because it illustrates to what degree the media selects sources and answers in support of its pre-established story. Fascinating, no?

1)What are some of the benefits of taking a remote DBA approach to database administration?

There are several benefits that I could list, but ultimately it becomes a matter of resource availability and agility. With an outsourced provider, more technical skills are on tap, at all hours of the day, with more escalation support and for more weeks of the year than with a fully insourced strategy. Furthermore, agility is greatly increased as the service provider can scale with the project needs and the likelihood that the service provider has already performed a task or a project is much, much higher than for a single in-house hire. This can dramatically reduce technology adoption inertia. In larger teams, I might mention that Pythian’s blended insourced/outsourced model allows a best of both worlds strategy to be implemented.

2)Talk about Pythian’s business model. How do you price your services to make them competitive with paying a FT DBA?

Pythian’s service model is a no lock-in, scopeless, linear cost-to-effort model that is disruptive to the traditional outsourcing model of flat monthly rates over a lock-in period.

In the traditional outsourcing contract, services are limited to a service level agreement (SLA) covering a strict scope of work, and the vendor’s profit model is centered around minimizing their costs associated with delivering that scope of work. This means that in traditional outsourcing, the vendor is literally motivated by the contract to deliver the minimum value-add as possible while still satisfying the arbitrary SLA, which is set during the lifetime of the contract, sometimes as long as 5 or more years. This problem is at the heart of what is wrong with outsourcing as currently conceived: no matter how successful the vendor is at automating, streamlining, tuning, and improving processes, the customer does not see any of those savings as they are paying a previously-negotiated rate.

Pythian’s model allows customers to subscribe to a fraction of a small team of engineers, with no lock in whatsoever so that the quantity of effort the customer requires is changeable on 30 days’ notice or can be cancelled for convenience at any time. This means that Pythian is constantly earning our next month’s renewal, which keeps us on our toes and keeps us motivated to add as much value as we can within the allotted effort level. Our profit model is customer-friendly and well understood: it is a simple mark-up model on our costs of service delivery. This means that as we automate, streamline, tune, and improve processes the customer gets to tap into those efficiencies, either by re-sizing the contract downwards to claim cash savings with no morale penalty, or by increasing the responsibilities that flow to Pythian without needing to increase the allotment of hours.

In smaller shops that might only need one or fewer full-time DBAs, our model is cost competitive because only the effort required to run the shop needs to be provisioned, and our processes, delivery model and expertise all couple to outpace our markup quite handily. Often, it is not an insourced vs. outsourced decision, however. In many larger shops, Pythian is engaged alongside the in-house team in order to dramatically increase the team’s capabilities, technical experience, availability at all hours and project support. Both use cases work very well.

3)How does an organization get in touch with Pythian in the event of an emergency? (ie, the database suddenly fails). How fast can they expect a response?

In the event of an emergency, the likelihood is that our monitoring software will have already alerted us and we will be on the job immediately. Generally, Pythian acts as an extension of the customer’s existing team and as such any method that the customer has adopted to collaborate is supported by Pythian. For instance, we use every third party instant messaging platform alongside internal platforms, email of course and telephones, and also videoconferencing and telepresence technology. Whatever the customer is using to collaborate with a resource working on another floor of the same building, we are also using. It is completely seamless.

Our response guidelines set the expectation that, in an emergency, a Pythian resource should be available on the affected system in single-digit minutes or less, all year round. Customers have live access to a backup engineer at all times as well as to a service delivery manager, for a total of three resources on-call. Our escalation guidelines allow our customers to engage the backup engineer and the service delivery manager immediately with no mandatory waiting period. Some of the systems we manage have downtime costs in the six figures per hour range, so this is the standard of care that we have implemented as a result.

4)What about other functions DBAs perform like providing end user and developer support?

This is a key advantage of the scopeless model that Pythian champions. Any work that the client wishes to flow to Pythian, we can do. This includes end user support, developer support, back-end development of triggers and functions, data modeling, data warehousing dimensional modeling, building transforms, input on best practices for security, business continuity planning and change control, you name it.

5)How does the process work with Pythian – are customers assigned a specific DBA for all their needs?

Customers are assigned to a small team of three to five engineers with an appropriate skillset to support the customer’s implemented technology. That team is led by a team lead which takes primary responsibility for the excellence in service delivery for the team, however a primary DBA is specifically not assigned, because one of the goals of the service is to isolate the customer from the pain of turnover, vacation, sick days, etc. and that goal would be compromised if we failed to disseminate the client knowledge evenly over a large enough team to be able to absorb that kind of change. Each team maintains an on-call schedule of its own, so that it’s always the same small group of people doing the customer’s support, which creates a client intimacy that results in Pythian becoming a seamless extension of the client’s tech team. Furthermore, each team reports to a service delivery manager, which has on-call responsibilities as well, so that the client has management support from Pythian at any time.

6)Doesn’t doing this effectively require an understanding of a business’s apps and processes? How does Pythian deal with that?

Of course. In this regard, engaging the Pythian team is no different than making a new in-house hire that happens to work in another office, or another floor of the same office. On day one, we will be contributing primarily our technical expertise while being complete rookies on the internal company-specific applications and processes. As time goes on, however, our expertise on the in-house specifics will increase much in the same way a new hire will gain that expertise over time. Among our 110 customers, Pythian has two customers that have been customers continuously since 1999, and ten customers that have been customers continuously since 2002 or earlier. For those customers, we are in a real sense the “old-timers” in the shop and are a key source of organizational knowledge for application structure, data model, and processes!

Todd Hoff, who apparently learned a hell of a lot during a short stint at Yahoo followed by some startups has an extremely well-written and edutaining article about how scaling to a million or more users requires jettisoning more or less everything we know and love about relational modeling.

Even though he uses bigtable (Google’s distributed hash storage system) as his example, in reality this approach works well with relational datastores like MySQL and Oracle too, you just have to think about your data differently and use the databases differently. So I’m including this article in the MySQL and Oracle categories because I think it would be of interest.

Here’s a taste of how it reads:

How do you structure your database using a distributed hash table like BigTable? The answer isn’t what you might expect. If you were thinking of translating relational models directly to BigTable then think again. The best way to implement joins with BigTable is: don’t. You–pause for dramatic effect–duplicate data instead of normalize it. *shudder*

Flickr anticipated this design in their architecture when they chose to duplicate comments in both the commentor and the commentee user shards rather than create a separate comment relation. I don’t know how that decision was made, but it must have gone against every fiber in their relational bones…

But Flickr’s reasoning was genius. To scale you need to partition. User data must spread across the shards. So where do comments belong in a scalable architecture?

The answer is, in case you aren’t following yet, you store it everywhere you might need it and worry about keeping your multiple copies in sync later, if at all.

BigTable data ethics are more Mardi Gras than dinner with the in-laws. Data just wants to have fun. BigTable won’t stop you from hurting yourself. And to get the best results you may have to engage in some conventionally risky behaviors. But if those are the glass bead necklaces you have to give for a peak at scalability, why not take a walk on the wild side?

So anyway, this is awesome stuff and thanks Todd. For your reading and learning enjoyment: Todd Hoff’s “How I learned to stop worrying and use lots of disk space to scale”.

It is with great pride that I am able to announce that Pythian is making a large investment in Europe. As of this month, Pythian Europe s.r.o. is fully operational and we have headquartered the company in beautiful Prague. Additional offices are planned in Paris and Malta by the end of the summer.

Pythian Europe is launching with an elite, full-fledged team and I would like to introduce the founders:

On the left is Lukas Vysusil, who joins us from Oracle where he served for 6 years in a variety of roles, including Oracle Applications DBA, DBA Team Lead, Manager of the Configuration Queue for Oracle OnDemand outsourcing services, and also Senior Technology Consultant. He brings a wealth of experience in team leadership, troubleshooting, Oracle Apps, the pressure cooker of consulting in the enterprise database and applications technology space and formal configuration and change management processes to Pythian and will serve as Service Delivery Manager.

On the right is Jan Polnicky, who joins us from Oracle where he served for 6 years in a wide variety of roles. You’ll have to check his linkedin profile for the entire list, but suffice it to say he started out as a developer for Online Services, quickly took on a leadership role in that team, moved to OnDemand where he became a services team lead, then got promoted to EMEA queue manager for configurations, and then got promoted to OnDemand Services EMEA Manager – Release Management where he led a team of up to 15 engineers across geographies (UK, ES, CZ, EG + USA & APAC indirects) doing general Oracle Database & Apps management, tons of preventative maintenance and supervised a number of Oracle Applications upgrade projects. In his spare time, Jan is working on his Ph.D., I kid you not. Jan will serve as a peer to Lukas as Service Delivery Manager.

You may think that’s enough.

You may be thinking, OK, with these guys and the teams they will soon be leading now Pythian has added so much expertise and horsepower in Europe they’ll stand pat for a while.

But oh no. Not me. That was not enough!

To lead these guys, on the centre, we have also added Peter Simecka as Vice President, Pythian Europe. Peter joins us from, you might have guessed it, Oracle Corporation where he started out in 1994. Even before joining Oracle, he had substantial expertise on Oracle/UNIX, dating back to Oracle 4 (I first worked on Oracle 5, but 6 was already out by then). Over his career at Oracle, Peter has led teams as large as 60 engineers, served as Product Support Manager for five years, served as Customer Support Manager for four years, and then built and led the Oracle OnDemand Outsourcing centre in Prague for four years. To say that he brings a wealth of leadership experience, customer support and liaison experience, and outsourced services design, development and delivery experience is a woeful understatement. I am hoping and planning to learn a lot from him.

It’s funny because the way I presented these guys, it makes it seem like I selected each of them individually, but that’s not how it happened at all. I’ll leave that story for another day, or maybe Peter will want to tell it.

So, what are we planning to do with this ambitious operation in Europe? Stay tuned.