The flood of information being generated, stored and transmitted over the Internet has created a massive demand for new data centers. For electrical contractors that also do telecommunications work, data centers are immense opportunities, involving power, mechanical structures, cabling and security systems.

What is a data center?
Data centers are facilities where servers, storage, switches and routers fill racks, and cabling runs overhead and in under-floor cable trays. All this equipment is used to store and distribute the data on the Internet. With an estimated 100 billion plus webpages on more than 100 million websites, data centers contain a lot of data. With almost 2 billion users accessing all these websites, including a growing amount of high bandwidth video, it’s easy to understand, but hard to comprehend, how much data is being uploaded and downloaded every second on the Internet.

Data centers are filled with tall racks of electronics surrounded by cable racks. Data is typically stored on big, fast hard drives. Servers are computers that take requests and move the data using fast switches to access the correct hard drives. Routers connect the servers to the Internet.

Speed is of the essence. Servers are very fast computers optimized for finding and moving data. Likewise, the hard drives, switches and routers are chosen for speed. Interconnection uses the fastest methods possible. Faster speed means lower latency, which is the time it takes to find and send the data along to the requestor.

While speed is a primary concern for data centers, so is reliability. Data centers must be available 24/7 since those 2 billion Internet users are spread around the world. Reliability comes from designing devices with redundancy, providing backups for storage and offering uninterruptible power. Devices are most needed to fight the No. 1 enemy of reliability: heat.

All the electronics generate heat, and the faster they run, the more power they consume and the more heat they produce. Getting rid of heat requires powerful air conditioning systems, which can consume as much power as the data center electronics themselves. In addition, uninterruptible power requires generators, batteries or even fuel cells, which generate heat from inefficiency, too.

Data centers consume vast amounts of power. A few years ago, the magnitude of this consumption became apparent in surveys of giant data centers. Estimates are that data centers use up 1.5 percent of all the power consumed in the United States. Power consumption in a data center is more than 100 times as much per square foot as the average commercial property.

Within the data center, the focus is on moving data, reducing power and heat, and ensuring reliability. That is done by choosing components and systems, designing facilities and installing them properly.

Moving data over cabling
Every data center project begins with fiber optic connections to the Internet. Entrance facilities must be provided for multiple cables that connect to the outside communications networks. Incoming cables will terminate in racks with connections to routers that, in turn, connect to the servers hosted in the data center. These connections will carry vast quantities of data over single-mode optical fibers.

Within the data center, the goal is to move data as fast as possible with the lowest latency, and that means using the fastest possible data communications links. Gigabit Ethernet is way too slow. Ten gigabit Ethernet and Fibre Channel are commonly used today. Fibre Channel is moving to 16 gigabits per second (Gbps), and Ethernet is headed for 40 Gbps and 100 Gbps. The big data center owners want 40 or 100 Gbps now and are pushing standards committees and manufacturers to produce usable products as soon as possible.

At 10 Gbps, standard data links use multimode or single-mode fiber optics, coax cables or Category 6a unshielded-twisted pair (UTP) links. Of the three, fiber is the most reliable and has the lowest power consumption, especially if the links are long. Up to about 10 meters (33 feet), 10GBASE-CX4 on coax can be used on very short links effectively. Cat 6a consumes a lot of power to transmit 10 Gbps on longer links and generates more heat. Many data centers use coax for short links and fiber for longer links, but Cat 6a continues to be developed for lower power and is becoming more widely used.

Even inside the servers, increasing speed and reducing power has led to the development of optical fiber interconnects. For instance, Intel is actively developing board-level interconnects for its products, touting all the usual benefits of fiber.

Most data centers will contain a mix of fiber, coax and UTP cabling. The connections to the Internet coming in from the outside world are going to be on single-mode fibers. Multimode fiber, either OM3 or the new OM4 high-bandwidth fibers, are likely to be used for longer connections. Coax will be used on very short connections. And Cat 6a may be used for short and medium-length links.

Another option is becoming available: active optical cables. These are fiber optic cables with transceivers at either end to convert electrical ports to optical fiber, providing the advantages of fiber to conventional copper ports.

Whichever media is used for interconnects, one thing is certain, there will be many cables! Cables from routers to servers, servers to other servers or switches, and switches to storage devices. With the volume of cables involved and the number of routes they can follow, plan the pathways and spaces carefully to prevent chaos in the data center. Cables may be run overhead or under floors. All cable trays must be properly sized for the expected cable loads. Under-floor trays are generally wire mesh types, while heavy-duty overhead racks are usually needed, especially when expected to carry large quantities of weighty copper cabling. Additional room should be provided for Cat 6a, since cables cannot be neatly bundled to save space, due to the risk of alien crosstalk between adjacent cables.

Because a Cat 6a or coax cable makes only one connection link per cable, multifiber optical fiber cables are big space savers. Prefabricated fiber optic cable systems using multifiber-style connectors not only save space but also installation time, since they just require installation and plugging together, no termination or splicing required. Fiber optic cables that are “armored” are sometimes used in the under-floor trays to prevent cable damage when more cables are placed in the trays.

Cable testing is important for all cables. Cat 6a is highly stressed at 10 Gbps, so all terminations must be made carefully. Even fiber links, especially if they include multiple connections, must be tested since link loss at 10 Gbps and above is quite small and intolerant of bad connections.

Obviously, cable plant documentation and labeling are critically important. In a facility that may have thousands of cables, it’s necessary that each cable end and each port are logically marked to allow moving, tracing, testing and troubleshooting cables.

Powering and cooling the data center
As noted earlier, data centers consume vast amounts of power and require large uninterruptible power supplies for reliability. All power requires air conditioning for delicate electronic modules. Many data centers run dual power systems with dual power backup to ensure maximum reliability. This means a duality of UPS setups, generators and power distribution systems.

Because of the high power consumption, data centers have become a major focus of research on how to reduce power. Electronics manufacturers are developing lower power systems and using lower power data links. Even the location of the data centers can affect the cost of power and design of a data center. The owners of big data centers, such as Google, are well aware of the issues. In the Bay Area near its headquarters, Google uses solar photovoltaic systems to generate 30 percent of its power. Another major data center is located on the Columbia River, so it can use lower cost (and lower carbon) hydroelectric power.

When you consider that the air conditioning system will use as much power as the servers, careful design of the system is very important. Power cables are usually under the floor on separate aisles from data cables, but the actual layout of power and data cables has to be coordinated with the layout of equipment racks because cooling of the equipment has become quite sophisticated. Generally, data centers are kept much cooler than regular offices to ensure proper temperatures to protect all the equipment. Racks and electronics are designed for directional airflow to keep hot air away from other electronics, while feeding cooling air to the racks. Cable racks must not impede cooling airflow and must be placed accordingly.

Security
Data centers are considered critical facilities and, of course, involve massive investment. Owners invest millions in preventing intrusion from the Internet from hackers and also are concerned with physically securing the facility. A typical data center will include state-of-the-art personnel entry systems, intrusion alarms and numerous surveillance cameras. This will often cover not only the actual building, but the area in which it is located. Some data center owners even ban cell phones and Wi-Fi from the facilities because of worries about data interference.

Coordination is key
While data centers tap all the resources of the well-rounded electrical contractor, it is important that the contractor be involved with the customer from the initial concept of the project. A successful data center will involve coordination among the customer, vendors, contractors and even municipalities and utilities from the beginning of the project. Each party should contribute to the design to ensure that all issues, such as facility design, communications, power and security, are covered properly.

HAYES is a VDV writer and educator and the president of The Fiber Optic Association. Find him at www.JimHayes.com.