Pythian Group

Here are the slides from my presentation “From Requirements to Partitioning and Sharding and Everything in Between”. A big thanks to all the attendees for their interest and questions. I got a lot of questions, so the audience was definitely very engaged!

MySQLConf2012-RajThukral.pptx

A panel on “Future Perfect: The Road Ahead for MySQL”

Brian Aker (HP), Paul Mikesell (Clustrix), Sundar Raghavan (Amazon), Slavik Markovich (McAffee), Ori Hernstadt (Akiban)

If there’s one common theme to this panel, and indeed, this whole conference, it is “We’re hiring!” It is amazing how much talent there is at the conference this year and yet it isn’t enough. Pythian is hiring as well of course: http://bit.ly/pythianjobs

Interesting differenciation between the mindset of Oracle from MySQL from Brian Aker: database as a service, which is something MySQL seems to be getting to. It comes with its own problems especially around trust levels which will lead to more thinking around data security (rather than just database security)

MySQL vs. NoSQL. Different tools for different jobs

Security is definitely a big topic these days other than performance.

Mark Callaghan of Facebook: “What Comes Next for MySQL”

focus on Large, sharded deployments

Interesting numbers from their deployment (MySQL with Innodb):
60M QPS and 1.5B rows read/second in production

MySQL with InnoDB is “web scale”

scaled to 10x more data on the same servers by:
Start with MySQL 5.1, flashcache, find and fix stalls, use multi-threaded purge from Percona, ask the db-ops team to deploy a lot of changes, use OSC (Online Schema Change) to add many covering indexes, use Faker from Percona+Facebook to fix replication lag, Make InnoDB compression good for OLTP

“MySQL has made amazing progress”
InnoDB multi-core performance is impressive (yes, it’s finally overcome that early limitation!)
Replication is robust (global transaction IDs, multi-threaded, crash-proof, group-commits)

But a lot of work remains (of course)

And now on to the future of databases in general. Some very valid points around Flash vs. HDDs and how databases need to work differently for them (minimize iops vs. minimize size). Update in place vs. Write-optimized,

Backups account of 50% of their IOPS when run – this needs work (for sure)
Innodb Fragmentation causes lot of space usage vs. AltDB (write-optimized store)
Pooling is an issue – 50% of i/o capacity is used on average, but too many outliers are out of capacity. Solutions are hard (Reshard, migrate shards, deploy a SAN)

Consistency between shards within a region, between master and slave for the same shard. Problem isn’t unique to MySQL but solutions are hard and current solutions are too constrained

MySQL still can not max out modern hardware

And more specific issues as related to Facebook and the Facebook architecture.

An exciting and busy day yesterday – lots of good talks, good conversations and good beer! Back at the sessions this morning and the first keynote of the day by Sam Ghods of Box: “MySQL: Still the Best Choice for Mission-Critical Data”

The usual story of a (file sharing) application that started out on one MySQL instance to store metadata and ran into scaling bottlenecks. Interestingly, NoSQL did not work out for them and they ended up sharding MySQL. “If you use a NoSQL store, but need any advanced featuers in your data store, you end up building them yourself. If you’re willing to partition your data yourself, you can use MySQL’s fancy features”

Now on to specific MySQL features that Box uses, not directly provided by NoSQL:

Inter-row Consistency (aka Unique Key) to ensure unique filenames in folders
Transactions maintain integrity on file/folder renames where the whole tree needs to be updated
Transactions also help maintain consistency when modifying denormalized data
Indexes guaranteed to be consistent, fast, gets you only the required data, requires no maintenance
Tools (or – whats happening in your data store), Metrics, Benchmarking
Maturity and Reliability (of MySQL)

And just to prove that he’s not anti-NoSQL, Box is using NoSQL as well, just for different applications. In other words, choose the right tool for the right job and be sure to consider all the pros and cons of a solution

Don’t choose a database just because “it scales” (whats the trade-off)
Wade, don’t jump into new technologies
If you go with new technology, be aware that crazy things might happen
Make sure you’re not rebuilding MySQL

And lastly, from none other than The Brian Aker, a keynote on The New MySQL Cloud Ecosystem. He was formerly the Director of Architecture for MySQL and also the creator of Drizzle. He is currently a fellow at HP, leading their cloud architecture group.

A little history of MySQL of course. The drivers as seen my Brian over the years: Initially “Batteries Included” or embedded into a product, to “Enterprise” or feature-creep, market-parity, stored-procedures.. And of course the GPL license, which caused no end of confusion in the marketplace. Now on to DBAs (or the shortage of!), again something we can all relate to. Yes, Pythian is also always looking for good MySQL DBAS. Continuing on however, there are no more distribution/GPL concerns as MySQL is provided as a service in the cloud now, and software as a service in the cloud does not need to concern itself with distribution.

Now on to the current landscape
“Fear” of Oracle is largely irrelevant. Definitely true, we’re over that as Oracle continues to release new, better versions of MySQL
“SQL is complete” hmm. I may need to think about that
“SQL is still in Demand” True
Disk I/O is still a problem. Of course. Especially in the cloud where it is unpredicatable
Virtualization is costly (?), Multi-tenancy issues – yes, who hasn’t seen that in the cloud!

And of course, how all this leads to what Brian is doing at HP with their version of the cloud using OpenStack, complete with demo on how to spin up and use a new MySQL instance in their cloud!

and to conclude, from my point of view, this conference seems to be quite a nostalgia trip. It seems all the angst around the various acquisitions has settled down and we can now look back on it as history while looking forward towards new paradigms that are opening up new opportunities for MySQL as well as the MySQL ecosystem, which is far bigger than just MySQL alone now. Cloud is a big buzzword, as is SaaS, PaaS, DaaS. The next few years are definitely going to be exciting as these technologies mature.

The next keynote is from Marten Mickos, now with Eucalyptus systems, previously CEO of MySQL AB. He’s talking about making LAMP a Cloud. No surprises there, Eucalyptus is the leading open-source cloud computing platforms for on-premise use.

A brief history of MySQL, the first MySQL conference in 2003, Eucalyptus and how the two tie together. It is true that MySQL has been the most common database platform in the cloud. Certainly the other big databases are lagging when it comes to adoption and deployment in the cloud. One comment from Marten that resonates with us at Pythian and others in the MySQL services business – Oracle definitely needs to build out the partner ecosystem around MySQL.

Some good insights from Marten into how the database and software paradigm has evolved from scale-up to scale-out, from closed-source to open-source, from distributions to “Stacks” (eg LAMP) to APIs and interoperability. MySQL has evolved through all of these changes and continues to do so.

Here it is finally – the MySQL conference 2012 starts with the Keynote Sessions.

The first keynote speech is by Peter Zaitsev, founder of Percona and a very smart guy and also by Baron Schwartz (Percona), another very smart guy, the brains behind a number of toolkits for MySQL. They’re talking about the MySQL Evolution – what I alluded to in my first post regarding this conference – they ways in which MySQL has grown, evolved, scaled and continues to make new inroads into new applications and industries.

From Peter: “What is most important hasn’t changed – MySQL is still a great piece of technology and it is evolving very rapidly” (Love that quote!) Also “MySQL is also buzzword compatible: NoSQL, BigData”

From Baron: his own personal journey from closed-source, proprietory to open-source and the passion that went with it. I’m sure a lot of us working with/in the open-source community can relate. “Lets be here, Lets be now” He certainly had the audience engaged.

A very well attended Pedro’s dinner – I didn’t count, but we had 9 tables of 8-10 people or so – dare I say almost a 100 people? Lots of beer, margaritas and good conversations! Here are a few pictures from the event

Wow what a lot has changed since the last MySQL conference I blogged about in 2007

MySQL has been acquired twice, once as MySQL by Sun and the second time around bundled with Sun when Oracle bought Sun. The conference is no longer organized by O’Reilly but by Percona. And the MySQL database itself has changed – We were talking about new features in MySQL 5.1, which wasn’t released yet, along with Falcon (where did it go?). 5.1 has long since been released as has 5.5 and we’re now talking about 5.6 instead of 6.0. There was no “Cloud” on the horizon, nor was there MariaDB, XtraDB, Drizzle, Schooner or any of the other offshoots of MySQL, all of which are creating a new buzz around the product.

Yet one thing remains constant – the vibrant community around MySQL. In spite of all the changes in technology, in ownership, in versions, branches, forks and the competitive landscape, core MySQL technology continues to thrive and grow along with the MySQL community as MySQL and its variants are adopted in more and more areas either replacing or working alongside other databases.

The upcoming talks are certainly in the same vein as in 2007 with lots of discussions around large-scale architectures, performance, replication, scaling, sharding – the usual suspects – but more interestingly, there are plenty of talks that go into the whole MySQL ecosystem, which is certainly larger than it was in 2007. New platforms like the Cloud, new tools for troubleshooting, diagnosis, testing and metrics around MySQL, new ways to offload specialized tasks like full text searching and keyvalue stores from MySQL to more efficient tools, specialized versions of MySQL that take full advantage of non-traditional hardware like the cloud or SSDs, tools to enable more efficient replication.. the list is endless.

Definitely lots of excitement in the next couple of days. Stay tuned for more information as I live-blog from the keynotes and the sessions.

If you’re at the conference, please come join us for the MySQL Community Dinner at Pedro’s and meet some of the people behind Pythian. We’re always hiring so if you’re interested, please attend the talks and tutorials being presented by Pythian folk, talk to us or check out the job openings at http://bit.ly/pythianjobs

–Raj.

The Pythian Group added another feather to its cap today. Our Nicklas Westerlund passed his MySQL Cluster DBA Certification exam, making him part of an elite group of 57 MySQL Cluster Certified DBAs worldwide, and now two here at Pythian, the other being Augusto Bott. We have a wealth of hands-on experience setting up and maintaining the MySQL Cluster. Nick’s and Augusto’s certified creds make them our go-to guys for MySQL Clustering.

Congratulations, Nick!