100-gigabit Ethernet technology coming to Fermilab

Jon Bakken

Jon Bakken, head of the Core Computing Division, wrote this column.

Upgrades are coming to Fermilab’s network capabilities. The upgrades will substantially increase the laboratory’s off-site bandwidth capacity. Perhaps just as significantly, the upgrades will keep the laboratory’s wide-area network infrastructure on the leading edge of network technology. To put things into perspective, after the upgrade our bandwidth will allow us to stream about 25,000 HD Netflix movies simultaneously—but we of course don’t do that. We have better things planned.

Today, off-site network connectivity is provided through a metropolitan-area network (MAN) called ChiMAN. ChiMAN is an optical network infrastructure based on 10-gigabit Ethernet (10GE) technology and has been in place at the laboratory since 2006. ChiMAN currently provides eight 10GE channels to the laboratory for an aggregate of 80 gigabit-per-second off-site capacity. Two of these 10GE channels are dedicated for general user traffic such as the use of Web, e-mail and collaborative tools. Five 10GE channels are allocated for high-impact scientific data movement such as LHC CMS data. The remaining 10GE channel is used for network-related R&D activities.

ESnet, the laboratory’s Internet service provider, is in the process of rolling out its latest network, ESnet5. ESnet5 is based on an advanced network infrastructure that includes 100-gigabit Ethernet (100GE) technology. Part of ESnet5 will be a new Chicago-area MAN, called ChiExpress, which will replace ChiMAN. The initial configuration of ChiExpress will provide the laboratory with one 100GE channel and three 10GE channels for an aggregate of 130 gigabit-per-second off-site capacity. The 100GE channel will be allocated primarily for moving high-impact scientific data, which makes up 95 percent of the laboratory’s off-site network traffic. The excess capacity on that channel will be made available for various network, middleware and application R&D activities. The ChiExpress 10GE channels will be dedicated to general user traffic, in a fashion similar to ChiMAN today.

The initial 100GE deployment at Fermilab will be a router to support the ChiExpress 100GE channel. Internal production network 100GE infrastructure should follow in the coming years as the technology becomes available on a wider and less expensive spectrum of network platforms.