Josh Buckner

18 September 2020

Best practices to learn from Mizuho Financial’s database DevOps journey

Recently, I was lucky enough to co-host a webinar with James Phillips, VP of Corporate Technology at Mizuho, a leading global bank with a worldwide network of financial and business centers. He joined me to talk about how the bank had introduced database DevOps, why they did it, and the challenges they had along the way.

It’s a fascinating conversation and you can still watch the webinar online. In this post, I want to cover some of the issues that came up that are important to a business audience. Many of the companies and organizations I talk to in my role at Redgate are on the same journey to database DevOps or are thinking of embarking on it, and there are some really valuable insights from James that are worth sharing.

Start your journey with top-down support

James has been around databases for more than 20 years and was already familiar with database DevOps, having started it about 10 years ago when he was doing some consulting work. He was fortunate when he joined Mizuho because both the CTO and one of the managing directors at the bank already knew there was a problem with the database.

While there was a heavy investment in DevOps and CI/CD processes across most of the application platforms, including Java, .NET, Angular and Python, the database was the missing piece and there were constant issues that were slowing the development teams down.

The important point here is that James didn’t need to go around the business and find support from key stakeholders. There was a clear objective and board approval to introduce database DevOps to resolve an issue that had already been recognized. Many DevOps initiatives are slowed down at the very beginning of their journey because of the lack of such support, so look for resources that can help like the following blog posts:

• The top 7 benefits of DevOps for CEOs
• The top 7 benefits of DevOps for CIOs
• The top 7 benefits of DevOps for IT Managers

Be prepared to change direction

While there was management support and a desire to adopt database DevOps at Mizuho, there was still the giant question of how they were going to do it. Like many businesses, they turned to the native Microsoft SQL Server Data Tools (SSDT) and MSBuild for their pilot project. Having been through the process before at other companies, James was able to make it work and it was partly successful. There were, however, significant performance problems, as he explains:

“SSDT doesn’t scale very well when it comes to very large databases. I’m not talking terabytes, but about the number of objects that are actually in your database. We have one with over 8,000 objects and there are a lot of interdependencies. From check-in to deployment of a change to a single stored procedure took about 25 minutes.”

This might appear to be a setback but for James and Mizuho the experience was invaluable. Firstly, introducing database DevOps served to highlight the inconsistencies in internal processes. For example, it became obvious that there was a lack of controls and standards in the way changes were promoted through the different environments. The move to a DevOps ways of working made visible the real issues that needed to be resolved to become more efficient, so the pilot project was a success in many ways.

Secondly, it demonstrated that free and open source solutions were too lightweight and not robust enough for their needs. This is where Redgate came into the picture. James had worked with Redgate’s database development tools before and liked them, and it was a natural next step to introduce SQL Toolbelt for the whole team. 91% of Fortune 100 companies use Redgate software and the SQL Toolbelt contains the industry-standard tools for SQL Server development and deployments.

Invite Subject Matters Experts to join you

As well as bringing in the SQL Toolbelt, James also wanted to change the source control system they were using from SVN to Git. During the pilot project, they found that a lot of the principles of DevOps are not necessarily enforced with SVN and the database developers in particular were not used to it, so they were cutting corners. The SQL Toolbelt includes SQL Source Control which plugs into SQL Server Management Studio (SSMS) and works with version control systems like Git, enabling the database to be source controlled alongside the application.

However, getting up and running with Git and tools like SQL Source Control involved a big learning curve, and James found that standard training approaches were too generic and didn’t relate to the day-to-day experience of the developers. Instead, he encouraged internal Subject Matter Experts (SMEs) at Mizuho to create customized training that focused on the use cases and scenarios developers come across in their normal working day. Multiple lunch and learn sessions multiple times a day made a huge difference compared to the off-the-shelf training.

James also made sure that the SMEs were available when the new tools and processes went live to address any issues that came up. I like what he said during the webinar to explain why it’s good idea:

“I’m going to be honest here – you’re going to have issues. There’s no way you’re going to nail it all on that first rollout, no matter how much testing you did ahead of time, because you’re going to find a developer that does something you’ve never thought of.”

It’s not a race to the finish line

Many companies and organizations on their journey to DevOps take an all-in approach: they try to roll it out across every team at the same time in an attempt to complete their journey as fast as possible.

James advises against this – and speaks from experience too. He freely admits that he wished he had the discipline ten years ago, when he first started introducing database DevOps, to begin with one of the smaller databases first. That way, any problems can be ironed out and worked through, and the learnings can be used to create a win and then build on it. He calls it – and I do like this – “avoiding the big bang theory”. As he explains:

“We didn’t say ‘Okay, every SQL database we’ve got, you’re all going live on the same date, let’s go’. We started with smaller databases, we built up success stories there and then started moving into our larger databases that had bigger user bases and more developers.”

Don’t leave anyone behind

We’ve been talking about the tools and processes for introducing DevOps to the database, but in any DevOps conversation, another factor always comes into play: people. Donovan Brown from Microsoft sums it up neatly in his description of what DevOps is: “DevOps is the union of people, process, and products to enable continuous delivery of value to our end users.”

He puts people first deliberately because a lot of DevOps is about cooperating, collaborating and breaking down siloes. That needs a cultural shift in the way people think, as well as a change in processes and the tools they use. As James explains:

“DevOps alone is not going to solve all your problems. I can put all the automation processes in but if the culture doesn’t buy in, developers don’t buy in and there’s not a top-down enforcement of it, you are going to have problems and you are going to have stumbling blocks.”

The answer is communication – and more communication. Separate conversations with different stakeholders at different levels to identify where their pain points are. The key then is to collaborate with them and demonstrate how database DevOps will make deployments easier, or faster, or reduce the amount of rework they have to do when errors make it through to production.

Don’t forget your internal customers

One final and really important point that James brought up was to think about who your customers are. We traditionally tend to think of them as the people who buy or use our products and services but he stressed the importance of internal customers. The product managers, the marketing people, the analysts – the people who are asking for the new features to be developed. They too have targets to meet, objectives to deliver on, and part of the IT team’s task is to help them by delivering faster while maintaining the quality of what’s produced. As James commented:

“DevOps is much more than just ‘Okay, I’ve automated my deployment process’. It’s also about adding value and quality, with controls in place so that you have less rework. Less rework means less downtime, less downtime means more productivity for the customer and delivering enhancements rather than dealing with technical debt.”

I like that because it demonstrates that introducing DevOps doesn’t just improve the way IT teams work, it helps people throughout the company by delivering value and quality to internal as well as external customers.

Summary

James and the team at Mizuho learned a lot on their journey to database DevOps – and achieved a lot as well. They used to be limited to two deployments a day to the QA environment but they now deploy on-demand and there have been some instances where they’ve deployed 20 to 30 times in a day. Importantly, deployments are now virtually error-free. As James says:

“Once in a blue moon we get some weird error because there’s some table being replicated or something that somebody forgot about. Outside of that, we have error-free deployments which is a huge difference compared to where we were a year ago.”

It’s customary in posts like this to end with a good reason why you should explore database DevOps. This one is no different, but the words come from James who, at the end of the webinar, concluded with:

“So the real elevator pitch to this is that it’s been proven, not just within Mizuho but multiple organizations, that database DevOps does work and it does provide a very significant value. Now with that said, find your use case that is the best to start out and prove it within your organization. Show the results, get the buy-ins across the board from all the major stakeholders that you need involved and then start expanding it throughout your organization.

“That is the key to success and, to me, it’s beyond proven that DevOps works and there’s plenty of whitepapers you can go read out there, there’s plenty of videos you can go listen to that show it’s successful and it works. But go out and do it, don’t hesitate anymore. Don’t find the excuse of why not to do it, find the excuse of why to do it.”

For more information about how Database DevOps can help you deliver value faster while keeping data safe, visit Redgate’s Database DevOps resources page online.