Nortelís 10-Gbit/s transport platform
Delivering bandwidth to build on
By: Dino Diperna, James Frodsham, Terry Taraschuk, Louis-Rene Pare
Originally published in Telesis July, 1997
Nortel is the first supplier to introduce to the market 10-Gbit/sSONET/SDH transport over a single fiber. Fully compatible with the installed base, the 10-Gbit/s S/DMS TransportNode features dense WDM (wavelength division multiplexing) technology to offer 160-Gbit/s-ready transport today, and exploring product solutions for 400 Gb/s and beyond in the future. Dense WDM is supported by Nortelís Multi-Wavelength Optical Repeater System, the industryís first bidirectional optical amplifier capable of supporting sixteen wavelengths on a single fiber.
To create the 10-Gbit/s system, designers combined a powerful architectural approach with a host of leading-edge technology developments. Nortelís 10-Gbit/s platform is the industryís most cost-effective transport solution, enabling carriers to maximize the revenue-generating potential of existing fiber routes, while setting the stage for the delivery of emerging high-bandwidth video, interactive multimedia, and Internet services.
In the next four years, network bandwidth requirements are expected to grow by a factor of ten, propelled by increasing voice, private network, and wireless traffic, but more significantly by emerging bandwidth-hungry interactive multimedia, video, and Internet communications. Some carriers today already have requirements in excess of 80 gigabits per second (Gbit/s) on some of their routes, and predict that several routes will demand 100 to 200 Gbit/s by the end of the decade. Driven by this payload demand, the Synchronous Optical Network/Synchronous Digital Hierarchy (SONET/SDH) market is expected to grow at an annual rate of 17 to 27 percent, from $3.3 billion in 1996 to $6.5 billion by the year 2000, according to The Yankee Group, a leading U.S. research and market analysis firm.
Operating in the competitive grip of todayís global markets, carriers also require solutions that are both cost-effective and evolvable to meet changing demands. To avoid the high cost of installing new fiber, carriers must maximize the value of existing fiber by carrying more traffic and extending transmission reach. The ultimate goal for operators is to minimize the transport cost per bit per kilometer (km), while maintaining superior levels of reliability.
Nortelís 10-Gbit/s S/DMS TransportNode system (OC-192 in North America and other SONET markets, and TN-64X in SDH markets) is the industryís most cost-effective, flexible, and reliable means of increasing capacity on existing fiber routes. Delivering four times the traffic-carrying capacity of existing 2.5-Gbit/s (OC-48/STM-16) transport systems, the 10-Gbit/s system can be combined with sixteen-wavelength dense wavelength division multiplexing (WDM) technology to offer up to 160 Gbit/s of capacity.
Nortel also has developed a complete global family of optical amplifiers, including a new Multi-Wavelength Optical Repeater (MOR) System Ė the industryís only bidirectional multiwavelength amplifier capable of handling both 2.5-Gbit/s and 10-Gbit/s rates. The MOR system supports sixteen-wavelength WDM transmission, and allows carriers to extend the reach of their fiber-optic lines up to 400 km between terminals without
regeneration (see below).
Nortel is the first supplier to introduce a 10-Gbit/s SONET/SDH platform to the marketplace. The first company to deploy the 10-Gbit/s platform is Touch America, a wholly owned subsidiary of Montana Power Company, which is installing a seven-state, 5,000-km (3,000-mile) high-capacity transport network.
Most recently, Qwest Communications, a U.S. provider of communications services to businesses, consumers, and other carriers, agreed to purchase up to US$150 million of Nortel OC-192 and MOR equipment for its 21,000-km (13,000-mile) fiber-optic network. When fully operational in late 1998, it will be the first U.S. coast-to-coast, all OC-192 network, and the first such network in the world. Other customers signing volume agreements for the OC-192/MOR system include WorldCom, IXC Communications Inc., and Norlight Telecommunications Inc.
Nortelís 10-Gbit/s development builds extensively on the field experience and customer feedback gained with its industry-leading 2.5-Gbit/s S/DMS TransportNode system. Nortel is currently the worldís leading supplier of high-capacity 2.5-Gbit/s systems, with nearly 28,000 network elements shipped in North America alone.
Its global portfolio of compatible SONET/SDH products, together with WDM technology and optical amplifiers, enables Nortel to offer operators staged and flexible increases in capacity Ė ranging from 2.5 Gbit/s to 160 Gbit/s Ė in line with existing infrastructure, customer base, network size, and growth projections. The 10-Gbit/s system is completely interoperable with existing transport products, allowing carriers to integrate it seamlessly with their installed base for efficient network operation.
In addition, the 10-Gbit/s product and the MOR system can be managed by Integrated Network Management (INM) Broadband. Capabilities include rapid fault, etimely configuration, and efficient performance management. INM Broadband offers an open, distributed platform and an advanced graphical user interface for integrated management across multiple technologies/services (e.g. SONET, SDH, ATM, switch, digital video, radio, digital cross-connect, and digital loop carrier/access) from Nortel and other suppliers.
Development of the 10-Gbit/s system required a global, multisite Nortel team to develop several leading-edge technological breakthroughs in optoelectronics, integrated circuits, system packaging, and software design. Because of high performance requirements, many conventional technology solutions could not be used. At the same time, all new technologies were required to meet the same stringent dependability and reliability standards as well-established technologies. To ensure those standards were met and even surpassed, Nortel set up a design-for-dependability program to tackle from the very start of development such issues as reliability, availability, ease of installation and maintenance, and low lifecycle costs.
Early and continuous input from the manufacturing team during the development process ensured that Nortel could manufacture the product on time, in volume, and to the highest standards of quality. Working in partnership with developers, the manufacturing team tailored the design to support industry-leading production, test, and quality-control processes.
Ultimately, the strength of the 10-Gbit/s platform stems from advances made by the teams of engineers who coordinated development across a broad range of technologies. Together, they successfully combined a new architectural approach with innovations in hardware design, high-speed modules and optoelectronics, optical performance, physical design, and software design to deliver the most powerful transport platform available on the market today.
S Y S T E M A R C H I T E C T U R E
The 10-Gbit/s system is built around a modular, distributed hardware architecture that centralizes bandwidth management and supports the port cards Ė the high-speed 10-Gbit/s line-rate interfaces, and the low-speed 155-megabit-per-second (Mbit/s), 622-Mbit/s, and 2.5-Gbit/s tributaries (see figure 1.0).
With this architecture, the system can be easily upgraded, expanded, or evolved to incorporate new software features and more sophisticated bandwidth-management capabilities. Likewise, new port cards can be deployed to accommodate, for example, additional tributary types.
Most significantly, the architecture supports several capabilities that enable carriers to save equipment costs, simplify operations, and support a variety of customer requirements. These capabilities include powerful networking and protection switching schemes, flexible bandwidth management, and a distributed control/messaging system for real-time performance monitoring and recovery.
Powerful networking and protection: The 10-Gbit/s system can support a full range of networking configurations and protection switching schemes, including linear and ring configurations. (See Figure 2.0)
To provide customers with an easy evolution path, point-to-point OC-192/STM-64 terminals can be upgraded later to add-drop or self-healing ring configurations without interrupting service by simply replacing circuit packs and/or downloading software.
Flexible bandwidth management: For network operators, the goals of bandwidth management are twofold. First, they want to fill all available time slots in the bit stream carried on the fiber in order to maximize the revenue-generating potential of their routes. Second, they want to reduce network equipment and fiber costs by eliminating the need to backhaul traffic at each end of the network for bandwidth management at large, expensive cross-connects.
Nortelís 10-Gbit/s architecture meets both goals. All tributary and line traffic is handled at relatively low levels of STS-1 (50 Mbit/s) granularity, which allows operators to fill their fiber pipes more efficiently. Traffic coming in on an OC-48/STM-16 tributary, for example, is demultiplexed into 48 STS-1s (or 16 STM-1s), each of which can be routed to a different destination. At higher levels of granularity, say STS-12, traffic would have to be routed in blocks of 12 STS-1s, which may not match traffic demands or patterns on given routes at particular times. Much like an airline flying planes with empty seats, these partially filled STS-12 payloads cut into a carrierís profit margins.
Flexible bandwidth management also allows operators to eliminate expensive cross-connects and the need to backhaul traffic. Each of the STS-1s can be sent directly to its intended destination, unlike traditional bandwidth management systems where the traffic is sent in large bundles (for example, at STS-48 levels) over a backbone to large cross-connects located at a few specific offices in the network. The cross-connects groom the traffic, which then has to be backhauled over the same fiber to its ultimate destination. Using the airline analogy, the traffic can now "fly" direct in Nortelís 10-Gbit/s system, rather than being routed first to a hub airport and then to a connecting flight.
Further cost reductions are provided by the systemís capability to terminate a mix of tributary types and functions, and support mixed protection schemes. This capability eliminates subtended OC-48, OC-12, or OC-3 (STM-16, STM-4, or STM-1) shelves and uses direct optical interfaces to terminate at the tributary level. Centralized bandwidth management performs all the traffic grooming, routing, and hairpinning at the STS-1 level, avoiding the need for additional tributary nodes and floor space in the central office.
Centrally managing bandwidth gives carriers maximum flexibility to groom (process, manipulate, and integrate) mixed payloads of voice, data, ATM, and video services Ė and to reconfigure traffic assignment on demand. With this flexibility, Nortelís 10-Gbit/s system efficiently supports a full range of network features. (See Figure 3.0)
Distributed control/messaging: The higher number of ports and greater amounts of bandwidth in a 10-Gbit/s system place increased demand on the system for messaging, monitoring, and control. A fully loaded bay can have up to 40 OC-12/STM-4 or OC-3/STM-1 tributaries plus two OC-192/STM-64 line interfaces Ė more than four times as many ports as in previous systems. Each of these ports requires individual monitoring and control, as well as support for SONET/SDH data communications channels for such functions as remote provisioning, alarms, and software distribution.
To manage the complexity, Nortel designers developed a fully distributed control and messaging architecture. They incorporated on all the critical switch, line, and tributary cards a transport control subsystem (TCS) Ė a small microprocessor with associated RAM and flash memory. The TCSs are interconnected through an
internal, high-speed messaging LAN (local area network) that enables the cards to communicate with each other, and with the main network element processor (shelf controller). The TCSs enable software to be automatically distributed from one site to the entire network to add new features quickly and easily.
Together with associated local software (discussed later), the TCSs support processing-intensive functions, such as real-time performance monitoring, fault monitoring, fault correlation, protection switching, and hardware provisioning and control. Nortelís 10-Gbit/s system meets or exceeds 10-millisecond fault detection and 50-millisecond restoration standards. For its part, the shelf controller coordinates software downloads, provides a gateway to the Network Manager via the Operations Controller, and handles higher-level functions requiring less real-time processing, such as performance reports and statistics.
H A R D W A R E D E S I G N
Accommodating all the circuitry needed to deliver these unprecedented levels of performance, bandwidth management, and distributed control within a single two-meter-high (seven-foot-high) bay required a number of innovations to integrate more functions on each chip, circuit board, and circuit pack. Circuit boards were designed with multiple layers, and each pack may include multiple circuit boards. One complex unit, for example, has circuit boards with up to 18 layers.
To handle higher-bandwidth communications, designers incorporated a new high-density, high-speed backplane connector system in their design. In the 10-Gbit/s system, some of the units have more than 1,000 pins on the connector.
Nortelís design optimally balances high-speed performance demands with the pin count (the density or the closeness of the pins on the connector). Keeping the pin count at a manageable level is important to ensure product robustness and manufacturability, while maintaining signal integrity.
At the chip level, designers first partitioned functions onto different devices, and then chose the most appropriate technology to balance the need for high densities (gate counts) with high performance (bit rates). To meet the different requirements, they developed 10 application-specific integrated circuits (ASICs) across a spectrum of technologies, including:
ē .5-micron complementary metal oxide semicon-ductor (CMOS) technology, for such processing- and memory-intensive functions as synchronization and overhead processing that require relatively low bit rates (155 Mbit/s) and very high gate counts (300,000+);
ē .8-micron Nortel bipolar analog telecommunications MOS (BATMOS) technology, for backplane drivers/
receivers, custom memory structures, and clock distribution where higher bit rates and relatively high gate counts (100,000+) are required; and
ē gallium arsenide metal semiconductor field effect transistors (MESFET), for intermediate-stage multiplexing/demultiplexing at high bit rates but relatively modest gate counts (1,000+).
H I G H - S P E E D M O D U L E S
For functions that demanded even higher levels of performance, Nortel developed several new custom gallium arsenide heterojunction bipolar transistor (HBT) integrated circuits (ICs). Used in the high-speed line interface modules, these ICs perform such functions as: multiplexing and demultiplexing the 10-Gbit/s electrical signals from and to lower-speed interfaces; amplifying the high-speed data output to drive an optical modulator that performs electro-optic conversion; and, in conjunction with the photodetector, converting optical signals to an amplified voltage (optoelectronic conversion). This HBT technology
allows feature sizes of two microns or more, ensuring a more dependable and repeatable ASIC manufacturing process.
The greatest challenge, and the overriding goal, of the HBT design team was to achieve world-leading levels of reliability and performance, while concurrently addressing fundamental reliability, packaging, and manufacturing challenges associated with the HBT technology and electronic design.
As a new leading-edge IC technology, the HBT development was particularly challenging because of the relatively high transistor (gate) count. For example, one of the ICs, aptly named the SuperDecoder, has about four times the gate count demonstrated in comparable technologies at 10 Gbit/s. Used on the 10-Gbit/s high-speed receiver module, the SuperDecoder chip incorporates a unique, patented microprocessor-controlled scheme that monitors the error rate of the incoming data, and adaptively selects the best point in the waveform for sampling (to decide if the signal is a 0 or 1). This adaptive sampling technique guarantees low bit-error rates and extended reach in Nortel SONET/SDH transport products.
Engineers also incorporated several design advances in IC interconnect technologies and substrate materials in the cards. To control electrical losses in the substrate material, for example, they turned to ceramic substrates to replace traditional PCB materials (such as fiberglass), which, although appropriate for 2.5-Gbit/s systems, could not deliver acceptable loss characteristics at 10-Gbit/s speeds. And to ensure high-speed signal integrity for connections on the chips, they directly bonded a semiconductor die to the ceramic substrate. Traditionally, chips had wire leads that were soldered to the substrate or board, and the longer the length of the physical connection, the greater the signal degradation. In high-bit-rate 10-Gbit/s systems, the effect on signal quality is even more pronounced, but the problem cannot be solved by simply shortening the length of the connections. High speeds generate high temperatures, which over time can damage the soldered connections and affect the reliability of the product. By bonding the chip directly to the substrate, however, Nortel was able to minimize the physical length of
the connections, while maintaining the integrity of the connector bond.
H I G H - S P E E D O P T O E L E C T R O N I C S
The new 10-Gbit/s system also incorporates several advances in optoelectronics technology. Nortel, in fact, is the worldís second largest supplier of optoelectronic components (such as lasers, transmitters, receivers, and detectors) used in telecommunications products, including not only its own, but also those of a large and growing base of external customers.
The latest innovation incorporated in Nortelís 2.5- and 10-Gbit/s systems is an sixteen-wavelength dense WDM optical subassembly used in the transmitter module. A key component in the subassembly is a state-of-the-art, high-power laser that emits light within the narrow wavelength range required to support dense WDM. Compared with single-wavelength systems where the wavelength window (or tolerance) is about +/-2.0 nanometers (nm), the window in eight-wavelength dense WDM systems is only about +/-0.25 nm. To ensure global compatibility, Nortelís WDM implementation complies with the spectrum plan that the International Telecommunication Union (ITU) recently standardized for dense WDM systems.
One of the techniques Nortel employs to ensure the precise targeting of narrow wavelengths is a novel "rotating" metal organic chemical vapor deposition (MOCVD) process for growing wafers. (Wafers are diced up into individual laser chips, which are used in the module.) This process ensures a high degree of uniformity in the fabrication of the wafer, reducing the width of the wavelength distribution by a factor of five. This improvement results in higher or more predictable yields, which supports volume production and security of supply for the customer. In fact, Nortelís laser technology offers the best combination of power and reliability on the market today.
Another key innovation is a new patent-pending wavelength stabilization mechanism that keeps the laser locked precisely to the correct wavelength throughout the life of the system. With this feature, Nortel is delivering the worldís first single-package stabilized
laser source for dense WDM.
Unlike other systems using free-running lasers that are not wavelength referenced or locked, Nortelís new lasers can easily add new wavelengths or channels to upgrade capacity. The locking mechanism is designed to be upgradable to 16 wavelengths and beyond without replacing existing lasers, thereby future-proofing the system against major cost outlays and lowering the overall cost of ownership.
In addition to its industry-leading laser technology, Nortel has developed the highest power and most stable laser pump modules available commercially in the world today. These 980 nm pumps are used in Nortelís Multiwavelength Optical Repeater amplifier.
The foundation for this technology is Nortelís heritage in building high-reliability lasers for transoceanic fiber-optic cable systems where dependability is paramount. This ultra reliability is achieved through such methods as flux-free soldering, solvent-free packaging, and an innovative optical alignment technique for coupling the laser with a fiber-optic cable so that the alignment does not drift with age as do other systems.
O P T I C A L P E R F O R M A N C E
Another primary development focus was to ensure world-leading optical performance standards. Nortel implemented an end-to-end test program at all its manufacturing sites to continuously monitor such key parameters as broadband frequency response and amplifier noise. This pre-emptive analysis and action program tracks and corrects all performance-affecting parameter shifts at the manufacturing site before they reach the customer Ė ensuring a robust and optimally performing product.
Nortel scientists, for example, limited signal degradations through careful design and control of high-frequency responses and low-frequency cutoffs. They also took several measures to minimize the effects of such fiber transmission impairments as chromatic dispersion, polarization mode dispersion, and nonlinearities.
Dispersion is also dealt with through careful link performance budgets designed to meet particular network requirements. By limiting losses and dispersion effects between the transmitter and receiver and by maintaining the output power of the transmitter at agreed-upon levels, Nortel can guarantee the bit-error rate will never exceed a given value.
To correct for errors caused by dispersion or other optical nonlinearities, Nortel became the first vendor to incorporate forward error correction (FEC) into its SONET/SDH products. FEC employs a check code that is inserted at the transmitting end and recalculated at the receiving end. The two figures are then compared to identify and correct bits that have been corrupted by degradations in the transmission path. FEC ensures an extremely low bit-error rate, or optionally can be used to achieve a longer optical reach.
Although FEC is a standard communication technique, it has not traditionally been applied to fiber systems, because of the superior bit-error-rate performance of these systems and because performance standards have not required it. However, as capacities of SONET/SDH systems increase and payloads shift from voice to data and other special services, customers are demanding performance that is significantly better than specified by the standards. Furthermore, FEC becomes more important in networks where optical amplifiers are used. Because these devices do not perform bit-error-rate correction, they can amplify not only the optical signal but also any noise carried with it.
P H Y S I C A L D E S I G N
Beyond the component level, physical design played a major role in creating a global product that meets the specifications of both the American National Standards Institute (ANSI) and European Telecommunications Standards Institute (ETSI) markets, as well as Bellcore specifications and international standards for electromagnetic interference (EMI), electrostatic discharge (ESD), and other safety-related regulations. Achieving all the 10-Gbit/s functionality within the prescribed ANSI/ETSI physical envelope posed significant challenges in such areas as EMI compliance and thermal cooling.
Because EMI emissions are inherently higher in a 10-Gbit/s system, traditional methods of handling EMI shielding at the shelf level are inadequate. Instead, Nortelís mechanical design team mounted the circuit packs within their own sealed metal enclosures. Fully enclosing the packs, however, restricts cooling convection air flows, which in traditional designs dissipate the heat generated by components (such as ASICs) on the circuit packs.
Finding a new solution was even more critical because some of the high-density circuit packs in the OC-192/STM-64 can dissipate as much power as an entire shelf might in earlier designs. Moreover, Nortelís 10-Gbit/s design has extra built-in power margins that will allow the current platform to evolve as new functionality or new technologies are introduced.
To meet these more demanding thermal requirements, the mechanical design team developed a patented thermally conducting compound that is injected between the components on the circuit pack and the outer metal enclosure, which acts as a heat sink. Air flowing through vertically aligned channels, or "chimneys," on the outside of each enclosure then exhausts the heat through air vents in the bay using forced-air convection cooling methods.
At the customer site, installation and maintenance is eased through individual circuit pack covers that provide easier and quicker identification and access, protect connectors and fiber-optic cables, and provide EMI protection. Color-coded fiber-management hardware also gives craftspeople direct access to any of the up to 84 fibers on the system, eliminating the need to sort through a tangle of other fibers.
In addition, the shelf design was optimized to enable easy extraction and replacement of circuit packs in the field without interrupting service. This ability to easily swap circuit packs gives operators the flexibility to offer any mix of tributaries to meet the different bandwidth demands of customers.
S O F T W A R E D E S I G N
Like the hardware design, a primary goal of Nortelís software design was to ensure product robustness. In pursuit of this goal, the software design team made extensive use of the industryís latest software, utilities, and languages.
The team also capitalized on the extensive field experience and customer feedback gained from previous product implementations, and closely coordinated development with hardware designers to take maximum advantage of the systemís distributed architecture.
Software loads, for instance, are distributed to the transport control subsystem (TCS) microprocessors on each of the systemís circuit packs and on the shelf controller. These loads comprise three major elements:
ē generic framework software that handles functions common to all the TCS cards, such as intercard messaging and fault detection;
ē tables that personalize the generic software with the attributes for a particular card by encapsulating all its hardware definitions, such as the names, addresses,
and number of bits in an ASIC register. The tables are created from text files by a Nortel-developed compiler, and are used by the generic software to trigger specific actions (such as lighting an LED alarm lamp, performing diagnostics, or launching protection switching) in response to certain events; and
ē application-specific software that enables individual functions on different cards. Application-specific software on the external synchronization interface card, for instance, provides accurate timing to the rest of the system from the cardís internal clock.
The combination of base software (the generic software and personalized hardware definition tables) and application-specific software provides several benefits. The generic code promotes software reuse, as well as improved robustness and maintainability, because the software is already debugged before it is loaded onto the cards. The alternative would be a more error-prone and time-consuming method of writing individual code for each TCS card. Moreover, compiling the tables from text definition files Ė rather than hard coding them as is done in traditional designs Ė paves the way for fast and easy modifications to the systemís diagnostic capabilities.
A key component of the internal diagnostics of Nortelís 10-Gbit/s system is an innovative software system known as Fault Location and Alarm Reporting (FLARE). FLARE precisely identifies the cause of any type of system error, triggers alarms and protection switching, and informs network operators of any circuit pack or fiber failures.
In addition to FLARE, the software includes several other internal system self-checks. For example, traffic paths through the network element are monitored for silent failures that may not be easily or immediately detectable.
Another major capability is a fast and efficient "hot turn-up" software download system that enables new or upgraded system features to be introduced without interrupting service. New software can be downloaded to network elements during periods when spare bandwidth is available on the data communication channels. In this way, an entire subnetwork can be upgraded simultaneously, instead of one node at a time. Moreover, the system gives operators the choice of committing to the new load or returning to the existing load.
The close coordination between software, hardware, physical, and manufacturing design was the key to the development of Nortelís ultra-reliable, high-performance, global 10-Gbit/s transport platform.
Given its full interoperability with Nortelís existing transport products, its capability to incrementally increase capacity through multiwavelength WDM technology, and a supporting cast of optical amplifiers for extended reach, Nortelís new 10-Gbit/s system provides carriers with the means to maximize the revenue-generating potential of their existing routes, while preparing for the growing demands of higher-bandwidth multimedia, Internet, and video services.