It was an existing operating facility that could not sustain any downtime, scheduled or otherwise, since every minute adds up to millions of dollars in billings for the company,” said Jim Mackey, president, Evergreen Power Systems, Seattle, in reference to Evergreen’s work on the expansion of Cingular Bothell 9 Enterprise Data Center, one of the major data centers in the Pacific Northwest. It was a $2.2 million project for Evergreen. General contractor Turner Construction Co., Seattle, furnished all the major electrical equipment—valued at approximately $10 million.
“The thing about these data centers,” said Dick Adams, superintendent, Turner Construction, “is that the money to be made by having them is so much greater than the cost of making them, that the companies want you to get online as fast as possible. What was successful about this project was that we did it fast, put it through a battery of difficult and painstaking tests to make sure it ran right and we came out with flying colors. It was a cool project.”
Changing power scenario
Before the expansion, the data center production area was designed for 40 watts per square foot. Over the last few years, the equipment being installed has had much higher power requirements, creating a much larger electrical demand.
“We’ve expanded service to the facility and distribution to accommodate a much denser electrical mode by increasing the size of the UPS system and adding emergency generators. It’s gone from 40 watts per square foot to 120 watts per square foot,” Mackey said.
As with any project, the expansion of the data center began with design. “There was a lot of programming and sequencing involved and consideration for how it was going to work and which were the critical loads,” Mackey said. “That’s when it got complicated and required synchronization because of the redundancy of every circuit with two redundant backup power systems on top of the normal utility power.
“This project was so complicated that we went to an outside consultant that specializes in these types of systems. Normally, on smaller, simpler systems, we’ll design it ourselves,” Mackey said. The company used the services of Gary Gerber at Gerber Engineers in Seattle.
“There was a large mechanical package associated with the upgrade, including installation of a cooling tower,” Mackey said. Holaday-Parks Inc., Seattle, designed it and did the end work. “With that much electrical demand, the electrical equipment produces heat, so the mechanical provides the cooling.”
In a 3-foot raised floor space on the second floor of the existing data center, Holaday-Parks installed a chilled water loop for equipment cooling through computer room air conditioning (CRAC) units, electrical and connectivity infrastructure.
A typical high-end data center topology might include either a single or redundant 12,470 volt utility feeder. The Cingular Bothell project includes a single 12,470V utility feeder and, in an upcoming phase, will include a redundant incoming feeder. As is typical, the feeder or feeders connect to the facility’s main electrical switchgear through a motor-operated breaker sharing a common bus with an N+1 12,470V generator plant. This 15kV switchgear also includes output breakers for redundant UPS system substations, a mechanical substation and, in the Cingular project, a possible backup substation. The substations on the Cingular project include a 12,470V-to-480V transformer and 480V output breakers, since without them, the 12,470 volts would be unusable by any equipment that would be installed in the Cingular or other high-end data centers.
Backup for backup
Further redundancy is provided with double-ended substations with two transformers in each lineup. The UPS substation output breakers feed the UPS modules. The mechanical substation distribution feeds mechanical and house loads.
“This separation of the house and mechanical loads in typical high-end data center installations and with the Cingular Bothell installation is used because, a lot of times, there are power issues related to mechanical systems that you don’t want reflected on your critical loads,” said Gil Hammond, project manager, Evergreen Power Systems. All of these electrical systems can be tied together with busway or feeders to provide bypass and alternate feed capabilities. On the Cingular Bothell project, this included connections to the standby substation, bypass of a UPS system, or connection to a load bank for testing. The goal of this entire system is no single point of failure.
While Evergreen’s expansion of Cingular’s operating data center shared many design similarities with a typical high-end data center topology, installation followed a much more difficult path due to the sensitive nature of the ongoing production requirements. The expansion involved an addition of 34,000 square feet of raised floor with a load projection of 60 watts per square foot on the second floor of the existing facility. Evergreen installed four new 750 kVA UPS modules and associated battery strings, and two new 2-megawatt generators.
“There was some uniqueness to that,” Mackey said. “We had to install and connect batteries and be aware of environmental concerns associated with batteries, such as spill containment.”
The result was the completed implementation of a 2N + UPS system, which equates to two independent UPS systems of four modules each that can be called system A and system B. To protect against failure of either system, each of these systems was independently loaded to no more than 40 percent its individual capacity.
“The result is that not only is there redundancy in each of the A and B systems, but also there is capacity to carry the entire critical load in the event of a total failure of either system,” Hammond said.
Evergreen also ran thousands of feet of 3-inch conduit feeders from the existing first-floor distribution equipment. This conduit ran under the second-floor’s raised floor and supplied power distribution units (PDUs), which contain a transformer and output breakers to feed remote distribution cabinets that contain individual load breakers that feed the equipment racks. Some single-corded equipment also required the installation of static switches. These are fed from both the A and B sources and provide a virtually instantaneous power transfer upon loss of either source.
Hood-Patterson & Dewar—a consulting firm that specializes in commissioning, testing, project management, engineering and maintenance—designed and performed tests to demonstrate to owner Cingular that the equipment works as it should.
“The challenges facing us and Evergreen included installing and connecting the new electrical equipment without disturbing the existing equipment,” said Dan Parker, president, Hood-Patterson & Dewar. “The second challenge was to integrate control of the new equipment with existing equipment and then prove all the sequences and all the controls without disrupting power to the existing equipment.
“On a brand-new facility, it’s a lot easier to do that,” Parker said. “You don’t have to worry if something fails to operate properly or fails to protect itself. We wanted to give the owner the assurance that the equipment worked, but also had to be mindful that we couldn’t just fail it and fix it. We had to anticipate failures and not allow them to occur. Of course, Evergreen was involved with us every step of the way. In some cases, we are viewed as a threat by the installation contractors, but Evergreen was totally interested in quality and we tried to approach it as a team. Evergreen was receptive to our ideas and the questions we had. They were good at explaining what they did and how they did it.
“It was a difficult, complex system and we had no interruption of power, no bad incidents and the system works well,” he said.
The project’s aim was to not affect uptime, a condition that can be impacted by many factors outside of the construction teams’ control including unreliable utility power, climate, the environment, machine or system failure, or human error. Evergreen’s 25-person crew had to be especially vigilant during the 13-month long project.
“These requirements imposed a lot of coordination and scheduling issues for us in terms of working in and around existing equipment and personnel,” Mackey said.
Since the end of the construction, Evergreen has been monitoring and maintaining the facility.
“I have electricians there 24/7 and we do startup and continuing maintenance on all of the components of the system. We are continually checking to make sure everything is good, when it is needed, and that it will start and work the way it is supposed to. We try to make it as foolproof as possible,” Mackey said.
Of course, there is always the chance of error. “A raised-floor environment is a special situation because of the constant changes in equipment being upgraded and installed,” Hammond said. “Anytime someone gets under the raised floor to deal with cabling, there’s a chance of something happening. Anytime the construction team or the regular facility personnel have to work on existing equipment, we have to develop a detailed methodical written procedure, which very clearly delineates the proposed schedule, personnel involved and all of the different functions that will be done to complete the work scope. This has to be approved all the way up the facility, and in some cases, the corporate chain of command,” he said.
Uptime of facilities is measured in terms of nines, which refers to the percentage of down time in a calendar year. Five nines, 99.999, amounts to five minutes of downtime in a calendar year and is considered the highest level that can be achieved.
So how does the Cingular Bothell 9 Enterprise Data Center rate on the “9s” scale? “They’re at 99.998 percent, very close to five nines,” Hammond said. “The gap goes back to when they had some early startup woes. They are proud of where they are since they’ve gone through five construction phases and still been able to maintain that kind of online uptime.” EC
CASEY, author of “Kids Inventing! A Handbook for Young Inventors” and “Women Invent! Two Centuries of Discoveries That Have Changed Our World,” can be reached at email@example.com or www.susancaseybooks.com.
Evergreen Power Systems—Lead electrical contractor, power and data center integrator
Holaday-Parks Inc.—Mechanical design/build contractor
Hood-Patterson & Dewar—Commissioning agent
Gerber Engineers—Electrical engineer
Turner Construction Co.—General contractor
Liebert, an Emerson Network Power company—UPS system
Russelectric Inc.—Medium voltage primary and paralleling switchgear