Crime thriller: Trading Down, by Stephen Norman – part 3
Stephen Norman
Over the past few weeks, we’ve been sharing some extracts from a new crime thriller, Trading Down, by Stephen Norman, long serving CIO/CTO of RBS Global Markets. This is his debut novel.
Here, in the third and final extract, we return to the action at the Hamilton Datacentre crisis.
Catch up on the previous extracts here and here.
TRADING DOWN by Stephen Norman
Hamilton datacentre: 01:00 Eastern Standard Time
Mannie had no answers. No one had any answers. The vast, failsafe arrays of air conditioning had stopped working in both halls. Even though they were independent and unconnected. There was power. There was cold water in the chillers, but no cold air blasting out to cool the computers. It was as if the CRAC units had gone on strike.
The crisis in Hardy had started later, but seemed determined to catch up. Mannie went out there and felt for himself the hot, humid atmosphere. It was twenty-four degrees and rising. People on the bridge started talking about going to DR, about moving to Westchester. Paul Carpenter encouraged them. He made it sound like moving house.
Mannie knew it wasn’t that simple. Going to Disaster Recovery was like jumping off a cliff with a parachute in one hand. DR meant shutting down every application running in Hamilton and firing it up in Westchester – processing a mirror copy of the data. There were hundreds of apps, most of them in the middle of their end-of-day batch processing. Going to DR was like evacuating a town into new homes and factories, transferring the half-built cars and the half-cooked dinners and expecting everything to work. It would take all night to do it and the fallout would last for weeks. And in the case of Hamilton, it was brand new and they had never even practiced it, because the weekend change calendar was so full.
He could tell the AppDev guys felt the same way. He had heard Chris Peters arguing with Carpenter.
“Paul, we must fix the cooling, OK? Believe me, we will be 100% better off than going to DR.”
It was 1am. McQuade’s engineers claimed to be ten miles away on 287. Two of the day crew from IBM arrived. They rushed into the Hardy Hall to look at the mainframe.
The incident manager was trying to get Mike Tucker and Property Services on the call. No luck yet.
Frank Castagnetto from IBM joined the call, presumably from his home. Castagnetto sounded like a high school science teacher.
“Gentlemen, some facts for you. The temperature in the Hardy Hall is now 28 degrees and rising one degree every five or six minutes. When it reaches 32 degrees, we will shut down the mainframe to protect it. You have 20 minutes, give or take. The bank needs to make some decisions.”
“Even if we haven’t gone to DR by then?” That was Paul Carpenter. The head of TIS Americas was sitting, naked, in his loft apartment in Tribeca with a speakerphone in front of him. His white cat was stretched out on the sofa beside him. Neil Jenkins had forced him to outsource mainframe support to IBM last year and he had always known it was a mistake. He was filled with sober and righteous anger.
“Yes,” said Castagnetto.
“We need more time,” said Carpenter, “the electricians will just be arriving.”
“Twenty minutes,” said Castagnetto, “then we’re done. If you are going to DR, you need to decide now. Once we shut down the mainframe, you can’t swing it to Westchester.”
Carpenter did not reply.
“Okay,” said Castagnetto, “then where is Neil Jenkins?”
Mannie didn’t remember hearing Neil Jenkins for the last hour. He was drowning. And sweating. He was standing with Frank and the young IBM engineer by the mainframe in the Hardy Hall, watching the temperature rising. Now it was 29 degrees. Already some of the UNIX servers had shut themselves down. It was a total disaster.
He put his hands up to the grill of the nearest CRAC unit. Normally it blew cold air, gloriously chilled, like ice from a glacier. There was air wafting out all right. It was a warm gentle breeze, the same f**king temperature as everything else. A row of green lights twinkled from its status panel. As far as this CRAC unit was concerned, everything was A-OK.
Mannie kicked the CRAC unit several times, until the pain brought tears to his eyes. “Damn you,” he shouted. “Damn you! You sh*tty piece of garbage, why won’t you work? Why? What’s wrong with you? Give me cold air, damn you!”
He looked at the racks of servers on the nearest aisles, all cooking, all blowing out toxic heat. They were a mixed bag, mostly trading and risk systems for Global Markets. EOS was in the middle of its batch but the other systems were mostly done for the night. The markets opened at 7.30am tomorrow.
Out of the corner of his eye, he saw the temperature gauge move to 30 Celsius. Something snapped in Mannie’s head.
“Frank, shut down everything in aisles D and E.”
“OK,” said Frank, “I’ll warn the dbas to get ready …”
“Frank, I said shut down. Like this.”
He moved to the first cabinet. Inside the door frame was a circuit breaker. Flicking up the guard, he tripped the switch. Nothing happened. Dual power supply, he reminded himself grimly, no single points of failure. He found the second breaker and flipped that. The little lights stopped flickering and the fans wound down. It was like killing something.
“Do the rest, Frank. Now.”
“Mannie, oh my God, we can’t do that.”
“We don’t do it, it will all be gone in twenty minutes. We need time so some asshole can make a decision. Just do it, Frank. Now.”
He ran back to the NOC and found Luis, John and Karen.
“Luis, John, open all the doors. And the roller shutters, starting with the delivery bay. Everything. I want a breeze going through.”
“What about the guard dogs?” asked John nervously. “Don’t they let them loose at night?”
“F**k the dogs. Dogs don’t produce enough heat to worry about. Run, run, damn you, the mainframe is about to go!”
Hamilton datacentre: 01:23 EST
Marcie Schulman was back on the Tech Bridge, sounding stressed. “The call with the business to decide if we are going to DR is happening now. The business is on but no TIS representation. Neil, are you on? Has anyone heard from Neil Jenkins? Is he on another bridge?”
No-one had heard from Neil. In the silence, Mannie jumped in.
“Guys, this is Mannie Seibowitz in Hamilton. McQuade has arrived.”
“What about the mainframe?” asked the incident manager. “IBM’s twenty minutes are up.”
“We bought time. We shut down a sh*tload.”
“Could you be more precise?”
“Email, the Global Markets trading systems, all the risk servers, sales, you name it. About half the kit in the Hardy Hall.”
“Oh my God,” said the incident manager, forgetting his lines.
“IBM say the mainframe may last thirty minutes,” said Mannie. “Then it’s gone for sure.”
Hamilton datacentre: 01:33 EST
“Hamilton, do you have a McQuade update?” said the incident manager, for the fourth time. Mannie ran down to the Chiller Room and dragged the senior McQuade guy back to the NOC. He was a thin silver haired man in a dirty white T-shirt. His hands and fingernails were stained with silvery grease. He looked like a car mechanic.
“Well?” said Mannie.
“Someone’s changed the settings on the chiller controls,” the man said. “The pumps were all set to manual. We’ve changed them back.”
“And?” said Mannie, with a murderous glance at Frank.
The mechanic just sat there. Mannie wanted to slap him, wanted to scream.
The man shrugged.
“Made no difference. Power’s OK, water temperatures are good, pumps are good, fuses good, everything’s good.”
He looked up, faintly defiant. “Ain’t nothin’ wrong with those chillers.”
Hope evaporated. Even the incident manager was silent.
“Excuse me, guys, this is Chris Peters. Equity Technology.”
Chris was sitting on the front steps of his home on Thurleigh Rd, the dawn sunlight bright on his face. The incident reports said his US trading systems were all down. EOS Americas was down. So were all the US back office systems. In ten years, he had never experienced anything like it. He spoke softly, trying to hide his desperation.
“Guys, all the alarms so far… they are from servers. Individual computers. Right? But the air-conditioning units, the CRACs, they’re putting out warm air. They’re sucking in warm air. No alarms from them? Why not?”
You could almost hear Mike Tucker from Property trying to figure it out. Mannie could hear someone whispering to Mike in the background.
“Chris, the thermostats for the CRACs are all controlled from the Building Management System in India. Centrally, you know.”
“So why aren’t the CRACs complaining?” asked Chris Peters. “What’s their target air temperature?”
There was a static burst on the line and an Indian voice cut in, a thick South Indian accent.
“This is Murali in Chennai. I can answer that question. The CRAC units should be set to nineteen degrees. But we are checking now…”
There was an agonizing pause.
“Excuse me, please, this is Murali. We have investigated the BMS. Air conditioning in Hamilton is set to sixty-six degrees. I am very sorry indeed for this news…”
“Sixty-six degrees Celsius?” said the incident manager, disbelieving.
“Yes, I am confirming sixty-six Celsius. Hence previous speaker was asking the right question. There are no alarms because CRAC units think hot, hot air is righty-fine. When temperature reaches sixty-seven degrees, then they will turn on!”
“When they will be blowing cold air over a dead world,” said Uzgalis, stating the obvious poetically.
“Well, Jesus Christ, Murali, just reset the f**king temperatures, will you!” shouted Mike Tucker.
He had reason to be anxious. Quite apart from the damage to the US economy, if this went down as a Property screw-up, Neil Jenkins would destroy him in the corridors of power.
Murali was back two minutes later and he did not sound happy.
“There is a problem with permissions. Building Management System is not allowing. We change it but then it resets to sixty-six degrees. We have not seen this before.”
Back in Hamilton, someone was tapping on Mannie’s shoulder. He turned around. It was the young IBM engineer, his face white and sweaty.
“We’re up to thirty-three degrees. In E aisle. We’re going to bring the mainframe down.”
“No, no, stay with it. We’ve found the problem.”
“I can’t risk it. It’s probably damaged already.”
“The bank will take the risk.”
“I’d need it in writing.”
Mannie looked around. The Disaster Recovery playbook was open on the table. He ripped out a page and scrawled ‘IBM must run mainframe until further notice,’ on a blank space.
“That’s it, buddy. Now run that baby until it turns into a puddle of silicon, OK?”
Mannie was pumped. He was so far out in space, he had no fear of falling back to Earth. He was on his way to some distant planet for good or ill.
Hamilton datacentre: 01:40am EST
Marcie Schulman was back on the Tech bridge, sounding punchy.
“The executive committee has requested full Disaster Recovery to the backup datacentre in Westchester. IT asked for a twenty minute delay to resolve the situation in Hamilton, which takes us to 2am. The business has instructed IT to execute full DR at 2am. No further excuses or delays.”
“Mike, we’ve got twenty minutes to break into the BMS and turn down the temperature,” said John Uzgalis. “Can we help?”
“We’re trying,” said Mike Tucker. “The system was sold to us by GE, but it was made by Fujitsu. Their industrial controls division in Yokohama. We’ve got calls out everywhere. No joy so far.”
“Who configured it? You must have someone who understands it,” said Uzgalis.
“We did. He was a contractor. He’s on vacation in Thailand.”
“Can we shut it down? Can we unplug it?”
“It’s managing eight buildings including the two datacentres in London. And the trading floors. No, we bloody well can’t unplug it.”
Chris Peters was watching a milkfloat coming up Thurleigh road, Murali’s words still ringing in his head: When temperature reaches sixty-six degrees, they will turn on!
The idea appeared magically in his brain, fully-fledged, gift-wrapped. Frantically, he searched for Neil Jenkins’ private number. The guardian of the bank’s infrastructure sounded groggy.
“Neil? It’s Chris, listen… how about this? Just listen, will you?”
The noise of the argument carried down the street. The milkman stopped his cart to listen. The young guy sitting on the steps was pleading on the phone. He sounded like he was almost crying.
“Neil, why not? Just ask Mannie to try it. Help us, please. Please, just call him.”
Shrugging, the milkman continued on his rounds. He’d seen a lot of strange things in his time, but usually there was a woman involved.
Two minutes later, back in Hamilton, Mannie had a call from a UK number.
“Mannie? This is Chris Peters. Here’s an idea. Unusual, but it’s your last chance.”
Mannie listened with amazement.
Finally he said, “Are you f**kin’ crazy? I’m not doing that. Neil would kill me.”
“It’s Neil’s idea, for God’s sake!” came the reply, “I’m just the messenger. Neil says get on with it.”
What do I have to lose? thought Mannie, yelling for Karen, John and Luis.
“We need hairdryers. Here, as many as possible. In the next ten minutes. Go.”
“Just do it,” he shouted at their incredulous faces. “There’s a CVS drugstore, up by the intersection, it’s open twenty-four hours. Buy everything they’ve got, the bigger the better. Don’t think, run!”
Trading Down by Stephen Norman is on Amazon pre-order for Kindle delivery on 9 November 2017.
