The hardware guy: how Bank of America is building a new infrastructure

Reilly: constant reinvention is crucial to success

David Reilly, head of the Technology Infrastructure unit at Bank of America, began his career in financial services IT started in the UK working nightshifts in the kind of environment where magnetic tapes had to be loaded for batch processes to run and the concept of real-time processing was not a factor.

Since 2000 Reilly has been based in New York, the city that never sleeps, which he moved to with Goldman Sachs before crossing over to Bank of America in 2011, whose vast infrastructure is under his charge and real-time processing is very much the nature of the beast.

Yet he says some of his most important lessons come from those nightshifts all those years ago: “I started as a nightshift operator cleaning the printer bands with a toothbrush. It was a great grounding. When you work nights running batch jobs you see everything: the key processes that do or don’t work – change management is a good example; you see where that’s not being done right. In that environment of working at the back end of a 24 hour process you understand the value of testing, understand the value of how to recover when something goes wrong – lessons that have informed me throughout my career.”

It has a day-to-day resonance in his current role. “I think where you start does inform where you end up: if I’d grown-up as a database administrator or systems developer I’d bring a different set of experiences. I have always got an eye on the things that could go wrong in each scenario, just trying to see around the corner a little bit when it comes to the production environment,” he says. “When you get the kind of scale we work at now in any large enterprise the level of testing, and more specifically integrated testing, that you have to do to uncover those unintended consequences of the change, I go back to that. It is amazing how many times you go back to ‘what could we have done better in testing? What could we have done better in production planning?’“

Against this background, much of what Reilly’s team have been working on for the past few years comes down to the simplification of systems. “We have a phrase which is ‘driving out valueless complexity’. One of the things we do in infrastructure is ruthlessly benchmark our cost base and the size of our footprint and compare it to companies in the same business. Unless something is contributing significant risk reduction and or helping our clients do more, why do we have it?” he says.

While rationalisation is a continuous process, the bank is also rolling out new systems, including the Quartz capital markets platform and Next Gen in the consumer division.

“It’s a question of horses for courses. There are times when a legacy system is rock solid, reliable and completely appropriate and the right approach is to slowly demise it over time: it’s being thoughtful about where you apply those different tactics,” says Reilly. “In the data space, for instance, we would be better served by rationalisation of what we have. Like most businesses of our size we don’t lack information, we need additional insight, so we are coming up with a consistent data architecture that we can retrofit to the data stores.”

Whichever approach is taken, Reilly says that the human factor is a vital ingredient. “It doesn’t matter how good the system is – if the people you’ve got running it are not properly equipped then it doesn’t matter how shiny it is.”

At Bank of America there has been a significant concentration on aligning business and IT. “Our structure, I think, is somewhat unique in financial services and it is a differentiator for us. First of all, all technology and all operations are under a single leader: Cathy Bessant runs technology and operations for the whole firm and reports directly to the chief executive. We start at a place where the technology community has a seat at the table.”

Below Bessant, the corporate organogram has vertical and horizontal elements. “There are capabilities that sit across the organisation – infrastructure, my world, is one that affects all businesses in all geographies; information security is another and data is another,” he says. “Then there are individual lines of business organisations that deliver technology to capital markets, wealth management, consumer banking or payments. The third innovation is that the shared teams are aligned to create tension between the needs of the enterprise and the line of business areas. So I have executives who have dual responsibility, delivering to the line of business and also an enterprise responsibility, such as running storage for the whole firm. What that does, by placing those execs around the table, is create extremely short lines of communications: accountabilities are clear and the sense of shared ownership is much higher.”

A central component in the communication between these executives is the infrastructure cost benchmarks, which Reilly says allow the bank to directly attribute 95% of infrastructure costs to products and services that can be marked-to-market, so it know how network or data centre costs stack up against competitors.

“It gives us a level of precision and where we are not in front we set multi-year benchmarks to become best in class. I have to make sure I’m delivering value for money and absolute comparison helps me do that,” he says.

The focus on benchmarking is taking the bank further down the road of a service-oriented relationship with vendors, which Reilly describes as “smartsourcing”.

“Where there is more of a commodity service and we can’t deliver differentiation, we’ll use the service wholesale so we can focus on where we can deliver differentiation,” he says. “We are doing more of that in the networking space over the next 18 months, with a couple of projects in the data centre. We are big believers in the modular datacentre, which looks like a shipping container, with incredibly densely packed processor blades inside. Why would we want to engineer or build that? There are people far better at integrating that.”

Ultimately, however, the organisation wants to go even further along the dematerialisation of infrastructure and into the realm of software defined infrastructures. “We are dealing in a unit of currency that is services rather than platforms,” he says. “Virtualisation is interesting, but we think it’s a waypoint on the path to software defined infrastructure – that’s the megatrend that we are hoping to embrace. We see the application itself arriving with a manifest that describes all the infrastructure characteristics that it needs, which can then be provisioned in real-time on commodity hardware that doesn’t involve a human having to build and configure the environment. Probably 80% of our environments could be provisioned on a known set of very highly defined parameters: there will be no manual change management needed any more, like I used to do, because the system knows what good looks like. Configuration management becomes more accurate and simpler into the bargain.”

While Reilly says that Bank of America is “just at the beginning,” of this development, called Project Greenfield, he says it is a three to five-year timescale. “We expect to have 200 workloads up and running by the end of December in that software defined environment and should have 7,000 added next year. “It’s very, very early for us – you can’t buy this stuff today, we are developing it with our technology partners, some large some small. We certainly couldn’t create a software defined infrastructure on our own, and nor would we want to; we are trying to do this with the community.”

That includes involvement with Facebook’s Open Compute project, which Reilly describes as “a real enabler”. One of the stacks it will have running in December is an Open Compute stack.

“We are embarked on the journey, but it will be three to five years before we have it done,” he says. “If we are successful, it would mean that the environment that we are working so hard every day to make better will end up being consumed. But if you don’t constantly challenge yourself to reinvent like that you can’t be successful – the pace our industry moves at means that you can’t ever rest on your achievements.”

Even so, those lessons Reilly learnt on nightshifts all those years ago remind him to try to look round the corner and see what other outcomes might be possible, given the rapid nature of technological change. “We have to have a balanced portfolio: maybe some of the commercial models around cloud providers will get to a point where they make more sense. Maybe there will be a different way to consume compute resources; maybe we’ll have figured out how to really track the movement of data around different geographies – we have to be ready for that, so this idea of reinvention, which we have a separate part of the organisation driving, is not going away.”

David Reilly was the winner of the CIO of the Year in the Banking Technology Awards 2013.