Today at Berkeley Lab

Ratmaggedon: Or How a Large Rodent Took Down Several Lab Buildings and IT Systems Monday Morning, and How We Recovered

In the wee morning hours on Monday, Oct. 22, a night of foraging was coming to an end for the rat. So was its life.

But its tragic end was no match for a group of 25 electricians, plant maintenance technicians, HVAC technicians, electrical engineers, and a carpenter, who pitched in over the next three days to get the lights back on for the 726 people who work in the Building 50 complex.

Here’s how this week’s power outage affecting a number of Lab buildings and systems went down.

Monday, Oct. 22

The electrical substation located outside the Building 50 complex

5:30 a.m. A rat scurries into the switch that feeds the Building 50 complex, coming in contact with 12,000 volts of electricity. The rat dies. Breakers are tripped. The lights go out in the Building 50 complex.

5:30 a.m.  Equipment on the ALS beamline detects the power dip and protects itself by shutting down. Other sensitive equipment follows suit. Building 50 complex emergency generators kick on, providing electricity to sensitive equipment and emergency lighting.

5:32 a.m.  Maintenance receives a call from the Guest House reporting a power problem. Users of sensitive equipment that has shut down start calling in. Alarms go off notifying Lab Plant Maintenance Technicians, who monitor alarms 24/7, that heating and air conditioning systems in the Building 50 complex have gone offline. Work to trace the source of the outage begins.

8:00 a.m. The problem is isolated to a single feeder switch outside Building 50. An electrician dons personal protective equipment, including an arc-flash suit. De-energizing is complete on the 8-foot high, 5-foot wide substation that houses the 12,000 volt switch, a transformer, and secondary circuit breakers.

8:15 a.m. Electrician emerges with answers. “It was a rat.” A grisly scene.

8:30 a.m. Work begins on rebuilding fuses. A contractor is called to help with assessment and repair.

The damaged insulator (left) was replaced by a new one (right).

9:30 a.m. The 486 employees in Buildings 50A, 50C and 50E have power restored. The feed for these buildings was not damaged, just tripped, and the line is reset and power is restored. Technicians work to make sure heating and air conditioning systems safely come back online. Prognosis for the power feed to Buildings 50B and 50F is not good.

10:00 a.m.  Contractor arrives to assess damage and help with the clean-up. The team works to compile a list of what can be cleaned and what needs to be replaced. Nine new insulators are needed for the switch. The Lab has three on hand. The other six are available from a vendor — in Texas.

10:30 a.m. Building 50 complex houses multiple data centers. The cooling tower for computer systems in Building 50B is down. The heat in the enterprise systems server room builds up. HVAC team brings in portable cooling units. A carpenter joins the team to build a duct to get the heat out of the room.

11:00 a.m. Work on cleaning parts of the switch and replacing parts with available components continues into the evening.

1:30 p.m. Enterprise systems, such as FMS and HRIS, which had been shut down to protect them when the power first went out, begin to be restored once there is enough cooling.

Tuesday, Oct. 23

8:00 a.m. The team begins its last push to clean and repair with parts that are on hand.

11:00 a.m. Technician leaves Oakland Airport with a box of insulators for the switch.

12:00 p.m. Installation of new insulators begins.

3:00 p.m. Repairs are complete. Retesting and energizing of the switch begins.

5:00 p.m. The last connection is made. Power is restored to Buildings 50B and 50F. At last, 240 employees can return to their desks.

Wednesday, Oct. 24

8:00 a.m. It takes another day or two to fully restore computing services. Science systems, such as the High-Performance Computing Services and Science Virtual Machine Service in the Building 50 complex, use more power and generate more heat than enterprise systems, and aren’t backed up by generators. They begin to be restored once full cooling capacity is reached, which takes many hours.

“I’m proud of my electrical crew,” said Ryan Velasco, Facilities Manager for the Lab’s Electrical Group. “Everyone rallied together to get power back up. They are committed to the job every day, not just when we have a crisis like we had this week.”

“The electrical team was just one of many who sprung into action these past few days,” said Ellen Ford, Chief of Staff, Operations. “The IT team, facilities, and our tenants all worked together as critical contributors to One Lab, ultimately supporting our science mission.”