Gleb has done excellent work. He is very knowledgeable with streams. More importantly he has been extremely diligent about monitoring the streams process. Great job!

Yvonne has done a great job on our new cluster installation.

I would like to give a big recognition to Marco Tusa for implementing Data Retention project for us.

Marco did a great job, exactly according to our requirements.

Well Done Marco!

You are a real MySQL expert

Best Regards,

Alex.

Again I'm so impressed with the tenacity of one of my Team 5 DBAs. Sudeep never gives up on an issue. He just keeps working on it until it's completely solved. Nagging little issues usually ignored and left as "little Oracle's mysteries" are hunted down, killed never to get seen again. Mystery solved, forgetaboutit!

Thanks Sudeep.

I would like to express my appreciation for the support that you provide us. We are 3 DBAs with a ton of work, and our support contract with you has made my life so much better. I feel that I am able to get a breather from support on nights and weekends, which is invaluable. Prior to having your support, I would be working nights and weekends on weeks (every 3rd week) where I was primary support. Having Pythian as a partner in SQL Support, has made a big impact in my work/life balance. I just wanted to take the time to express my personal Thank You to each and every DBA that supports the Covidien account.

I really enjoy working with the Pythian team. Danil and Singer are both great to work with.

Excellent job on this. Very knowledgeable.

Excellent work from everyone who worked on this and determined what the issue was. This fix was much needed for a personalization put into PROD for Order Management. Without it, a major project was dead in the water.

Kudos to everyone

In the short time that Pythian has taken over supporting our Oracle installation, working with the support team has been excellent.

It's always a pleasure working with John. He and everyone else who assisted in this did a great job in helping discover the root of the problem.

Many Thanks

Gary

Thank you so much, Sandi!

Great job.

Singer is GREAT! He's fast and attentive. And always does excellent work. We're lucky he's supporting the Virgin America team. -Tony

The DBAs at Pythian did a superb and thorough job for TravelClick. Everything was done within specification. Any issues were dealt with professionally and quickly. At the end of they day, everything worked... and worked well. I could not have ask for more.

Thanks - would never have figured this out!

Michelle Gutzait did a really great job helping us to find and fix problems with SSIS package and database itself. She applied her professional SQL DBA skills, developer skills and interpersonal skills for this project by giving right and timely recommendation for the server setting and helping IT to find the correct solution. She modified the package itself and result was reducing the execution time from almost 20 hours to 2 hours.

Thanks for the details analysis and follow-up. Doug

Yes, Subhajit, Sudeep, Vasu, Yuri and Maris did a great job to support McGrath. Appreciate your effort. We did not interface with David much this time but I know he is working hard in the background. Thank you all for your support.

Thank you Sandi, keep up the good work.

[OTR]I hope we're done for now with partition management...

Thank you. It is very helpful to have this patch list.

Francisco did a great job, he checked our servers and provided us with optimization recommendations. We've implemented everything accordingly.

Thank you so much!

Thanks for the follow-up. Appreciate the attention to detail.

Yes, Edwin did a superb job helping me with this!

Great work, thanks.

Marcelo helped me a lot, not just with pure DBA tasks, but helping debugging an existing application, finding dependencies and logical errors and fixing it. I am very impressed with his knowledge in both administration and development areas.

Marcelo always shows the highest level of knowledge. In this particular case he can found the root of the problem and did the necessary research as needed and provided the best way to solve it.

Alex did an excellent job helping get to the root cause of the issue.

Thank you for your great support with our MySQL servers, it's been a pleasure to work with a such dedicated and professional team.

Thank you Pythian - Marco Andrew, and James.

Best Regards,

Alex.

Once again Pythian folks helped us out during an urgent production issue. Fahd, Chuan, Yannick and Gleb were all very responsive and knew exactly what to do. It is very comforting to be able to count on the Pythian Team for their knowledge and expertise.

Thank you so much for the support you provide us.

I would like to emphasize that James and Raj are very professional DBAs, and they are always available when we need them.

It's been a real pleasure working with such dedicated people.

Good Job Pythian team.

Best Regards,

Alex.

The Medical community relies on Nordion to provide Medical Isotopes to benefit the lives of millions of people. Nordion IT relies on Pythian to monitor and maintain its critical business applications. Once again a Pythian DBA has saved us from unplanned downtime. Michelle Gutzait quickly diagnosed a SQLServer problem and put in place a permanent solution. Well done!

Our OMS/Grid control migration was a major migration for Ning. Gleb planned and executed it very well. Ning was really impressed with his accomplishment. I would thank him for his flawless work.

Mani.

Great thanks to Christo, Alisher and Kellyn for their work during production deployment night! Our first release of the year contained a large number of DDL changes in over 10 different environments. Their well executed and coordinated work allowed us to apply these changes in a short timeframe. Excellent team work and communication! Thank you!

Excellent work and understanding of the issue.

Thank you.

I wanted to thank you for your and your team's support this weekend, with upgrading 2 of our 10g databases to 11g.

Gwen's support was invaluable. She was patient, responsive and helped us resolve failures encountered in the course of the upgrade. We were able to complete these tasks in a timely manner.

Andrey G contributed to Ning database stability is enormous. He initiated many pro-active measures and put some good process and procedure in place. I would like thank and appreciate his effort on behalf of NING operations.

Thanks

Mani

Igor was persistent in completing the ticket (CR401579), systematic in his approach and clear in his communication of progress, next steps, and requests for input from us.

Andre did an amazing job. I don't know what else he has to work on because when he works with me he is incredibly efficient and more than solved the problem at hand, he went on to help correct a bunch of other concerning issues we uncovered. The solution was quick and effective. The moment he came online he understood the problem and worked hard to solve it in the best way possible. I know everyone here appreciates his work and we always hope to work with him.

I just wanted to thank all of the Pythian team for their professionalism and high standard of work. We resolved some big issues here at Casella with all their help.

I look forward to working with Pythian in the future, as we have identified some items that may require their help again.

Thanks again.

Gwen did an awesome job on restoring and recovering the database.

Andrey and Gleb did a great job investigating the issue and rebuilding the database in limited amount of time. I am totally happy with their service.

yes great job, thanks.

Yes Luke has always done oustanding work for us as most of the guys do.

Much appreciated, Matt

Thanks Catherine, David and Sudeep for excellent job in making this maintenance go smoothly.

Susheel

I just wanted to thank Pythian for what they did today. You succesfully restored a broken Oracle server within a couple of hours and handled this case in a very professional way saving the day for a bunch of developers.

Special thanks go to Virgil Glavan who immediately answered my call and to Jan Polnicky who cured our server.

Thank you guys.

Gabor Sebestyen

EU Edge, LLC

Budapest, Hungary

The restore of the elab database was able to be completed very quickly due to Pythian working on the Restore documents in conjunction with Working Instructions MDS was verifying. With the restore templates being made available, the restore happened within 2-3 hrs rather than a much longer downtime for the users, which was very beneficial for the BAS business unit.

I'd just like to recognize Andre Araujo's amazing work for us in query optimizations. I like to refer to Andre as the "SQL Magician" because he is able to turn queries that take several hours, even days, into ones that a fraction of the time. Most recently, we needed to run an ETL job to get at some really important customer data on a table with near 1 billion records. He was able to extract the data for us in a couple hours and come up with some amazingly fast, albeit complex, queries to perform analysis on the data. I am continually amazed by Andre's deep understanding of how Oracle works and how he is able to apply it to help us out. Thank you Pythian for having such talented DBA's on your staff!

John did a great job on CR310761 - he ws helpful and completed the task quickly.

We are very satisfied by the service given to us by Andre and Shakir in support of our recent data quality and reorganization initiative. Thanks!

Catherine did excellent on this CR. Her notes are very clear and shows her knowledge of the subject. She is prompt with communicating changes. I am very happy that she will be supporting the Team 5 and has excellent knowledge of Oracle Financials.

Andrew has done an excellent job on this project. He started working on this project late but able to manage it successfully. He has kept the team updated with any changes to the scope or time and made sure that we got all the action items taken care. I am glad that we have him manage the project by co-ordinating with resources from different continent.

Thanks Andrew for contribution.

Pythian did a very good job to fix our Oracle problem. I sent an e-mail to their support and got a call right away. Thanks to the sales team and the DBA, Karun, who fixed our very serious situation in a short timeframe.

Pythian were recently engaged to take a lead role in a high end infrastructure build project at Telstra. Our requirements were a combination of specific technical skills in clustered HA/DR environments and the ability to lead and communicate (verbally and documentation) across groups in a confident and clear manner. Alex's unique skill-set was an absolute must have on the project and he proved his worth repeatedly during the build process with fantastic troubleshooting skills and his ability to communicate complex requirements to demanding support teams. The communication component in today's marketplace is often overlooked and I believe Pythian's focus in this area coupled with their deep technical skill-base makes them a unique player in the professional services field. I would recommend Alex and Pythian highly and look forward to working with them again in the future.

I would like note the great work by Pythian on improving the performance and the stability of our production environment.

The Pythian team has helped us in identifying issues and worked with us to resolve them.

We had replaced our production servers and thought to be a long arduous task but Warner made that transition quick and simple. We were done several hours before the expected completion time which made our clients very happy. Another major task was when we added our new SAN array. Mohammed worked with us to make recommendations and outlined the steps needed to have the optimal configuration. He worked with us during several early hours and kept me up to date during the transition. Warner and Mohammed were very knowledgeable about the tasks and kept me up to date during the upgrade period which was very important since many parties were involved. They are both professional and great to work with.

I would highly recommend Warner, Mohammed and the Pythian team.

Thanks,

James Chong

IT Director

Pythian were recommended to me by a work colleague who had used their services previously. We started with a face to face meeting to discuss covering a month of leave, and signed up for 3 months. Within weeks it became obvious that they were not your average professional services organisation. As our relationship grew they understood what sort of work we were happy for them to do without our signoff, and what we wanted to have involvement with. This led to an immediate improvement in the state of health of our databases - Pythian were rapidly completing the sort of work that had previously been overlooked by other professional services organisations.

We have now engaged Pythian on an on-going basis and I would recommend them as highly as any other organisation that I've dealt with.

Dedicated and result driven, Lukas at Pythian consistently aims to deliver what our business requires. Overall Pythian have great flexibility and team spirit, which helps in our dynamic and changing environment.

Don Seiler has done an tremendous job offering LIM support over the last few months. He seems to always be readily available to answer our questions...even after business hours. We really appreciate his efforts.

I just wanted to let you know how much I appreciate the work that was done for us last week by David, Vasu, Balraj and Murali. We were short-staffed, and they came through every time. We asked for their help on many issues, gave them little notice on when the work needed to be done, and they very promptly reacted and performed the work accurately and rapidly. They are a great team to work with, and we so appreciate their efforts. David also does a fantastic job on keeping us informed and communicating with us and the rest of the Pythian staff.

Hi,

I just wanted to let you know that we went live successfully for both internal and external websites.

I want to thank Pythian for the great work and excellent support. Your level of expertise and response time is outstanding.

Keep up the Great work!

Regards

Nithin Shetty

McGrath RentCorp Inc

Pythian did a very good job for cloning this environment for the first time.

I just wanted to compliment Gerry on his thoroughness and clarity in the documentation on his work in CR 254247. Thanks!

Nicklas and Gerry did exceptional work for us, reacting swiftly to our production problems and restoring order to our databases.

One of the really nice things about how our hours work is the ability to use them for unexpected occurances. Our company was purchased awhile back and there was never any interest by the new owners in expanding the usage of Pythian to their databases. We just got off of a couple hour call with them. A really large table had been accidentially truncated 24 hours ago. They could get it back from tape, but ran into issues trying to get it restored. The import crashed and was in the process of rolling back and no one there had a clue how long it was going to take before they could try again or even if they were going to be able to successfully restore it.

As soon as they called us looking for any thoughts I contacted Shakir, our team lead, and after explaining the situation he brought in Christo. We had the person there establish a WebEx session and share out his desktop. It took Christo a few minutes to orient himself and then to figure out that it was going to take at least another 9 hours to complete the rollback. Long story short, Christo came up with two very workable options for them, tested them out and documented them. Because of auditing requirements they still need to do the physical work, but they were very happy and said they learned a lot. I'm suspecting that those couple hours could well generate some interest in having Pythian take over their DB's as well.

Thanks again Christo. Good job!

Kudos to Marc Fielding! Our tech group has been fighting a really weird data loss problem on one of our Linux database servers for several weeks now. It was related to Oracle writing a flat file out to a Samba mountpoint aimed at one of our Windows servers. It took Marc only a couple hours to diagnose the problem and have us running 100% again. These guys aren't just great DBA's!

Karun always does a good job for us. He is wonderful and everyone on our DBA team at LSS enjoys interacting with him. I trust him to follow up on any and all outstanding issues. Plus, I swear he is psychic and deals with erratic mood swings diplomatically.

Vasu and Balraj did a great job especially with all the bugs they encountered while applying the RUP6 patch.

We had been having intermittent indexing issues with one of our internal search applications. Our team lead Augusto Bott managed to pick out an obscure replication issue relatively quickly. His quick identification of the problem allowed us to bring a fast resolution to the application and bring things back onto the right track. This has been a reoccuring issue for us for about 6 months and Augusto figured it out in under 15 minutes. Thanks for being on the ball.

October is a stressful month at MDS as we gear up for our fiscal year end. Once again Pythian came through a with great game plan. Team Lead, David Ashlock, provided a proactive monitoring approach tailor made to suit our Finance team's needs. David's innovations continue to keep MDS's Global ERP running with the minimum downtime and maximum performance. His attention to detail and his ability to communicate make doing business with Pythian a pleasure.

Last night I made a critical error at 4:30pm. I basically ran a command that destroyed a large number of files on a server that was 2 days out from go live. The back up server was still in the test phase and no full back up had been done on the system.

If you're wondering what I did I ran a simple command to give high level access to a sub directory of a web directory to allow a user to complete their job. This was to be a temporary solution until we could lock down the directory. As root I ran "chmod 777 mydirectory / -R". So basically I recursively reassigned 777 to the root directory not the directory I intended.

Okay so this was all bad things I did and I knew it, but it did not change where I was at. Less then 2 days from go live on a production product, that was new to my entire team, and we needed the final two days to complete tasks on these servers such as backups and redundancy testing.

So from 4:30-7:00 myself and another senior team member troubleshot the back up server for this application. It was a LAMP install, rh4 linux, apache 2.2, mysql 5, php 5. It went up originally very simply, but the on the back up server php would not talk to the database.

So that everyone knows we do not work with Pythian currently for any support on our mysql, php, apache servers. They support our oracle servers only.

At a little after 7pm we contacted Greg Leger at Pythian for support on the server, he assured me he had senior team members to support me if I needed them. A few minutes later Singer contacted us. At 7:30pm we were back online and ready to go home!

The issue was simple but neither myself or the other team members here could see the issue. Turned out there were two sets of mysql running, when we researched this originally we searched for "rpm -qa | grep MySql" which is the way the new release is spelled. It turns out the orignal release on the server was "mysql" or "rpm -qa | grep mysql".

Thanks, for bailing me out on such short notice on systems you had not worked with previously.

Rob Hamel:

If you have an employee-of-the-month program, I'd like to nominate Marc for it. His work this weekend went above and beyond the call of duty. It is difficult finding competent help, let alone great help. It is tough to find a vendor that will care about your systems as if they were their own. Pythian appears to be one of those shining stars. I know we pay you well for your services, but I'd like to note my thanks to Marc for doing his job so well in general and especially this weekend.

Keep up the great work.

Just wanted to drop a quick note and say that I think that Isabel is likely the most cheerful/happy/fun to work with DBA i have come across! Its nice to work with someone who loves their jobs so much it shows at that level! I look forward to working with her in the future!

As always the tasks and results are exceptional and show extreme dedication.

I'm an experienced DBA, almost 10 years of senior experience. I had a major stumper today, and after banging my head for a few hours, Sheeri was able to help me in under 10 minutes. This just goes to prove the adage that everyone needs an extra pair of eyes. Pythian is proving to be a valuable partner to PalominoDB.

I want to extend my thank you to Augusto and Nick who helped us promptly to identify and fix the problem we had with one very weird behavior on version 4.1.11 where BOOLEAN type is not correctly mapped to TINYINT(1) and this broke the replication on our second most important server.

Once more - thank you guys.

Martin

July has been an extremely busy month for Register.com in terms of project work - both planned and unplanned. Pythian in its usual grace has supported us beautifully throughout, working with us under some difficult deadlines and coming out smelling like a rose. We also performed an extremely complex migration this past weekend from our old Solaris based Oracle Financials environment to a shiny new Linux one, and because of the detailed planning that Sylvain and his team of Balraj and Vasu produced, our project has literally gone off without a hitch - a truly incredible feat given the scope. We are all amazed (but we shouldn't be - this is just Pythian doing its thing).

Thanks!

With 30+ databases to patch and upgrade each quarter, it's nice to have a team who can reliably work on the 4th of July while we enjoy our holiday.

Thanks Virgil, Christo and Pythian team members who supported our upgrades this quarter.

Just wanted to leave a comment about the excellent work Nick Westerlund and Augusto Bott have done for us, particularly in the past two days when our application traffic has increased significantly. Augusto has always done good work and Nick, apparently a new addition to the team, has come on and hit the ground running.

Both of their excellent work, particularly in critical times on short notice, is very much appreciated.

Thanks, Marcelo, for taking care of two crucial services in our environment.

I wanted to pass along some feedback about David. I have heard nothing but good things about how he has been handing the work that Pandu was doing and he has by far exceeded any expectations we may have had about how this transition would go.

In fact, I would probably say that things are working even better than they were with him involved.

Please pass along our appreciation to David for the thoroughness of his knowledge transfer work along with his continuing work on our RT tickets.

thanks

The task of upgrading our oracle instances on our production box was done in a timely and professional manner. Thank you.

This agent deployment was done in the most *lazy dba* fashion (all scripted and automated ) as we love it here at CCI ... good job!

I'd like to commend Sheeri in the strongest possible language for her support last week with our rapid rollout of mysql proxy. Her dedication is greatly appreciated and reflects well on Pythian.

I just wanted to let you know we have been hammering your team with requests hard, especially since last Thursday. This is primarily due to our GSS project implementation. In addition to these requests, your team has also had to deal with a few production issues. They have been working very well and keeping a great attitude. Please thank them for me.

Re: CR#207213

I was pleased with the response time

and the service given.

Thanks a lot!

The MDS Infrastructure Consolidation Team has been working with Pythian for the past 3 months on site-specific consolidation efforts.

Karun Dutt, our dedicated Pythian DBA, has consistently performed over and above expectations to assist MDS with our hardware upgrade from Solaris to Linux, as well as an Oracle upgrade from 8i to 11g; trouble-shooting and finding the required solutions, often to fit a square peg in a round hole. He always makes himself available, even on very short notice, for team meetings and interfaces very well with the internal MDS technical team.

We have been very pleased with the work performed by Pythian, and Karun specifically, and look forward to continuing our engagement with them during our ongoing Global Consolidation efforts.

thanks to Alex, Vamsi and all members of Pythian team (including Singer/SA's) for the excellent handling of this critical event - thorough preparation, smooth execution and timely communications!

-- george

We just got back from Pythian Day, an annual (for us) event in which we go visit the Pythian offices, meet the people on our team, and hear about the latest events in the Pythian universe. Once again, we had a good time and found it very useful.

Sometimes just sitting with your team in front of a whiteboard for an hour and getting to know them in-person and getting to share knowledge in-person can be a really useful exercise, and I think it definitely was in this case. It's always nice to put a face to a name, or faces to names in this case.

As usual, the Pythian folks were gracious hosts and the trip was a success.

One of the most blood pressure testing exercises we go though now is our quarterly security patch upgrades. Our web applications process 25,000+ new sessions an hour and every minute we're down you can feel our clients fidgeting. We have 30+ databases and it takes 3 weekends to complete the updates. Tonight we hit the final critical nine.

The work was assigned to Aleksey Sintyurev who I believe is one of the newer DBAs on staff. I knew he was very senior but we had never really worked with him before. He had also just started working this morning with a monthly process that Pythian helps us out with on several of our databases. It was new to him, fairly complicated and he was running into problems. Our SLA mandated window to start and complete the patching was rapidly approaching and since 3 of the databases to be updated are beyond critical I have to admit I freaked out enough to call Paul. Literally within minutes I had 3 additional senior DBAs on the line and team members familiar with that monthly process took it over and freed Aleksey up to concentrate on our patch process.

First, thanks to Paul and the other team leads (Alex Gorbachev and Greg Leger) and Andrey Goryunov in Australia for great handling of a panicking client who was having a meltdown!

Second I can only say excellent job to Aleksey! You were not only under the normal pressure of delivering the level of DBA support we all expect from Pythian, but the added Pythian senior corporate level scrutiny that my panic attack brought. Nicely done!

Paul Moen performed like a champion when our master databases were melting down off-hours. It really proved the value of having DBAs on-call around the world!

Keeping in mind that he's not one of our regular team and that our server setup is quite unusual -- he came up to speed very, very quickly. He was up with access on the right machines discerning the nature of the problem when we first got in touch over IM.

He was able to move quickly and get multiple restore options going in parallel, choosing the best one when it became clear which that was. Paul got the app back up as soon as reasonably possible (by using the old slave) while the new master was being made ready for production. Later he added some suggestions about ways to have higher availability for the master in EC2.

Thanks Pythian!

MDS has been live with Pythian DBAs for 3 months now on our Global ERP. Team Lead Sylvain Fontaine conducted a smooth transition from the former support vendor and within weeks has initiated proactive maintance. Vasu has implemented changes that have improved performance and reduced overhead bringing direct savings to MDS. Team 5 has also begun clearing outstanding issues that have posed risk to our system.

MDS Functional and Development Managers agree that the team is providing fast, 24x7 and easily engaged support.

I would like to state how pleased I am with the professionalism, knowledge and work ethic the Pythian Group analysts have shown in my dealings.

Whether it has been grappling with STREAMS issues, setting up environments monitoring or backup strategies. They have been excellent.

These views are my own and may or may not reflect those of Vale Inco.

Sheeri provided excellent support on an effort that needed very quick turn around. Sheeri was able to assist in very tangible ways in helping us to determine the database architecture all while workinging a an environment that was almost entirely new.

This past weekend was very busy. Our Penn Community environment was successfully upgraded to Oracle 10g. While this may sound like a commonplace task, when it comes to Penn Community, it certainly is not. This is one of our most critical environments. A multitude of applications rely on this database. Needless to say, we all had to have our act together when we upgraded PCOM from 9.2.0.6 to 10.2.0.3.

Shervin did an excellent job with planning, testing, and carrying out the production upgrade. He left no stone unturned and actually delivered the environment to us ahead of schedule. This upgrade consisted of about 25 post-upgrade testers alone who had to get up at 1:30 AM this past Sunday to certify all of those production apps that use Penn Community.

The result is that we have had NO reported problems since going live with 10g on Sunday. Sr. management all breathed a sigh of relief when I sent out the all clear on Sunday. It is a pleasure working with Shervin.

Well done Shervin!

I'd like to comment on the overall work that was done by several members of the team first as the preparation for the downtime and second during the maintenance.

Virgil and Alex were very helpful all the time and follow on the guidelines that were directed by me to make sure that all parties involved into the maintenance are in synch.

I also appreciate the fact that guys did extra analyses to make sure that the systems are working fine and all scheduled task that were temporarily disabled are back to normal.

Thank you.