A few weeks ago I witnessed a weird problem in one of our customers’ networks. The customer was complaining about poor data throughput between his central site and two remote sites, each connected with one Fast Ethernet (FE) and one Gigabit Ethernet (GE) connection over the Synchronous Digital Hierarchy (SDH), by means of a LAPS (Link Access Procedure) encapsulating technique. LAPS provides a means of transparently mapping Ethernet frames to SDH virtual containers.

On each customer site, NIL had installed, provisioned and maintained the data network, based on Cisco Systems equipment (switches, routers, firewalls etc.), while the optical transmission network SDH, based on Ciena equipment, was installed, provisioned and maintained by another company.

After receiving the customer’s trouble report, I immediately started performing troubleshooting procedures on the part of the data network equipment (mainly switches) that NIL had installed, and through which connectivity was established. First I performed some preliminary tests on data network equipment, pointing out problems along the optical transmission path that was based on the SDH technology. That report brought to the scene the company that had installed and maintained the SDH equipment. They proceeded with tests on the SDH equipment, which included measurement of the optical transmission link throughput on L1 and L2 (by sending the Ethernet frames) and on L3 and L4 (by sending the UDP datagrams). Tests that the other company performed by way of expensive optical equipment (i.e., IXIA 400T, SunSet MTT) showed maximum throughput of 100 Mbps for the Fast Ethernet link, and 1 Gbps for the Gigabit Ethernet link.

At that time, the problem seemed to be with the customer’s end equipment, such as PCs, or applications the customer was running on that equipment. Before I started bothering the customer with that solution, though, I decided to include the customer’s remote sites in the NIL Monitor system for at least a week, in order to get long-term traffic statistics for the affected optical transmission links.
One weeks’ statistic from the NIL Monitor system amazed me. Even though the NIL Monitor servers I used for measuring link bandwidth couldn’t generate traffic beyond data rates of 90 Mbps, it still revealed the bandwidth asymmetry of the affected optical transmission links and focused my attention back to the SDH system.

Measured bandwidth of the affected optical link

At that point, a question arose: Why didn’t the previous measurements (made with highly specialized and extremely expensive optical test and measurement equipment) show the bandwidth asymmetry, which was so clearly presented on the NIL Monitor system?

NIL Monitor measurement of the link bandwidth is based on TCP, a protocol that provides a reliable transport path over the data network. TCP does that by breaking up the data stream into packets and reassembling those packets in the proper order to reconstruct the original data stream at the receiving end. But sometimes packets can take different paths to their destination, arriving at different times and out of sequence. TCP solves this problem by temporarily storing packets until the missing packets arrive, so they can be restored to the correct order. What happens if a packet is damaged, or doesn’t arrive at all? TCP solves this issue by discarding the damaged packet and sending another one. Similarly, if a packet doesn’t arrive on time, another one is sent.

By contrast, UDP is an unreliable protocol because it doesn’t bother dealing with lost or damaged packets (handling of damaged or missing packets is left to a higher-layer protocol, such as in the applications).

To satisfy my curiosity, I captured the data traffic of the affected link. The output showed several packet retransmissions. At that point, I knew that the problem was in the SDH equipment, but I was still missing one piece of that puzzle. To solve the problem, I had to look into the configuration of the SDH equipment, where I finally found out that the source of the problem was the way in which the Fast Ethernet and Gigabit Ethernet frames were mapped into SDH virtual containers (VCs) and transmitted over the SDH network.
Fast Ethernet frames were mapped into two virtual containers (VC3, each with a capacity of 48.960 Mbps), while the Gigabit Ethernet frames were mapped into seven larger virtual containers (VC4, with a size of 150.336 Mbps), by the encapsulating technique LAPS. VCs (VC3s and VC4s) of the same Fast Ethernet and Gigabit Ethernet connection were traveling both ways around the SDH ring. Due to different optical path lengths, the VCs that were traveling one way around the SDH ring were subject to higher delay than VCs traveling the opposite way. On the receiving site, the corresponding VCs of FE and GE arrived at different times and therefore could not be reassembled on time. That problem reflected on the operation of TCP, which started the retransmission of missing packets, resulting in effective bandwidth being cut.

According to that finding, all VC3s and VC4s of the corresponding Fast Ethernet and Gigabit Ethernet were rerouted over one optical path around the SDH ring, immediately improving TCP performance. As the following picture shows, the bandwidth asymmetry disappeared after the VCs of the corresponding FE and GE connections were sent the same way around the SDH ring.

Measured bandwidth of the optical links after routing all corresponding VCs of the FE and GE the same way around the SDH ring

What have we learned?
As one of my teachers once said, “The new test and measurement equipment can give you results faster, but often we forget the theory behind those results. We just can’t blindly believe the measured results if we are not sure how the results were measured.”