A fiber upgrade at the Bank of Canada did not go smooth. Fortunately, the project ended better than it began.
December 1, 2002
At the BICSI Canadian regional conference in Quebec City earlier this fall, speaker Robert Horne outlined a complex and frequently frustrating optical fiber installation that was part of a $2.9 million network upgrade project at The Bank of Canada’s head office in Ottawa.
It is a project that the structured cabling expert, who specializes in network design, project management and training, will not soon forget. “My purpose for speaking at the conference was to share my experience with others, knowing that if I had problems with (it) others might have had too,” says Horne, an RCDD/LAN Specialist who currently provides client project support for the national Bank’s network services branch. “I wanted to let others benefit from my mistakes, and also learn from what we did right.”
The bane of his existence became the small-form factor (SFF) connector MT-RJ, developed by a coalition of industry heavyweights in the late 1990s that included Corning Cable Systems, Agilent Technologies, AMP NetConnect, US Conec and Fujikura. Following its release in mid 1998, the powerful MT-RJ Alliance was created. Its mandate: Reduce the cost of fiber-optic networking and promote both the connector and fiber installations.
According to a position paper from Anixter Inc., SFF connectors are a significant step toward breaking copper’s “stranglehold” on horizontal premises applications. Its size is important, the paper notes, because as more fiber is being used in private networks, more electronics are being squeezed into less space.
“No SFF connector has garnered as much attention — for better or worse — than the MT-RJ. The Alliance has created something of a juggernaut. It boasts that more than 30 companies that have agreed to support the connector. This support ranges from local area network equipment to integrated circuits. Its success in luring these companies onto the MT-RJ bandwagon can be attributed to the strength of the individual team members in their respective markets.”
The MT-RJ was just coming into play when Horne volunteered to design, tender and project manage a major upgrade at the Central Bank, which would see shared hubs replaced with a switched 10/100 architecture. The first challenge was deciding which type of fiber to install. In the end, he chose the 50/125 um laser optimized InfiniCor 600 cable from Corning, hoping it could support 10-Gigabit Ethernet down the road.
“Then I had to decide what type of connector I would use,” he says. “Would it be SC or MT-RJ? The safe decision would have been SC, but MT-RJ at the time was the talk of the industry.”
What followed was a type of connector tug of war. Horne’s first design was based on the MT-RJ, but after a preliminary investigation, he went back to SC. The latter was tried and true, while the “sexy newcomer” had a limited field install track record. Horne had also heard several “negative anecdotal” stories about the connector that made him nervous. There was, he says, also a fear of the unknown.
Any concerns were mitigated following a series of meetings with manufacturer reps from AMP NetConnect and Corning Cable Systems (known at the time as Siecor). What ultimately swung the design back the other way was Georgetown University in Washington, D.C. The university installed an all fiber network to establish a 622 Mbps (OC-12) ATM backbone with switched 10 Mbps Ethernet to dormitories and desktops, and switched 100 Mbps Fast Ethernet for servers and advanced applications.
“After that, I decided to go with MT-RJs,” says Horne. “There was great assurance from the manufacturers that this was a solid connector and they showed no reservations about supporting it. I liked the idea of not having to use a lot of hybrid patch cords. We would only have to use them in the Telecommunications Rooms since the access switches didn’t have MT-RJ interfaces. Still, this would only be a small portion of the patch cords used. Most would be MT-RJ to MT-RJ.
“I was cautiously optimistic, especially since I had support of the manufacturer, and the support of the Bank.” A Request For Proposal issued in November 2000 specified the MT-RJ connector and Corning InfiniCor 600 fiber as well as a single manufacturer. Vendors were limited to AMP NetConnect and Siecor. The following month, a contract worth an estimated $100,000 was awarded to an installation firm that bid on the AMP/Tyco product.
“I felt confident that we would have a smooth installation,” says Horne. “I was wrong.”
Phase One of the project began in March 2001 and within two months connector issues began to crop up when the installer began getting poor results on their tests. “AMP was called in and we were told that some of the first connectors used were of an older generation and would be replaced with the newest versions. We also had some issues with poor installation practices, such as fiber strands being left on the floor of the Telecommunications Rooms, which is something that had to be addressed several times to the contractor. That left me with a bad feeling as one of the basic rules of fiber installation safety is to properly dispose of all fiber pieces when striping and cleaving.”
Horne found himself in a Catch-22 position. With problems mounting, the contractor pointed to potential issues with the connector and 50/125 cable. The manufacturer countered that the problem was likely the result of using questionable installation techniques. The truth, says Horne, lay somewhere in the middle.
After AMP was brought in to investigate the situation, it was revealed that while a “bad batch” of connectors had been installed, the installation practices themselves needed to be improved. By this time, stress levels were rising. The contractor was losing money, the vendor was facing a PR nightmare and the consultant’s reputation within the Bank was at stake. The situation ended up getting worse before it got better.
“Sometimes stuff happens, but it’s not what happens, it’s how you deal with it,” says Horne. “Both the manufacturer and the installer worked together. AMP/Tyco changed the design and the installation practices of the connector.”
Still, there were other issues such as test procedures. The first phase ended in August with poor test results and a high failure rate (17 per cent), the result of what Horne calls poor installation test techniques.
“The test procedures were a major issue,” he says. “The contractor submitted test results that included failures, and many suspect results. I went through them with a fine toothcomb and did not like what I saw. I compiled my concerns and asked them to retest. The contractor had used two different test methods and subsequently had many false passes. In the end I had them retest all fibers using a new test method and one set of testers.”
Fortunately, all parties were committed to making it a success. Phase Two of the project began in December and by February of this year, the re-testing of all fiber installed began. Results of the new tests were submitted in June with much better results this time. The site is now certified and the infrastructure has a 25-year warranty.
“The installers tested every single strand that wasn’t in use. We had our support people here move all of our active connections from the working connection to the ones that were just tested to make sure they were OK. We had to retest everything.
“It’s isn’t the best business practice because you end up losing a pile of money. I feel very strongly that if manufacturers certify firms, they should make sure that they are properly trained.”
Horne’s advice is that manufacturers provide more training to certified installation firms and that contractors invest more commitment in the training of their employees. When it comes to installation of advances such as the MT-RJ connector, he adds, there must also be a thorough system of checks-and-balances.
“You share the pain,” says Horne. “And also, share the gain.”