Case Study

Fiber Outage in Pakistan June 27^th 2005 to July 8^th 2005

The Pakistan’s sole under sea optical fiber link, called Southeast Asia, Middle East and Western Europe-3 (SEAMEWE-3), stopped working due to a fault from the 27^th June to the 8^th of July 2005. This disruption halted the global connectivity of almost 10 million internet users in the country ( http://www.jang.com.pk/thenews/jun2005-daily/28-06-2005/main/main3.htm).

The PingER team carried out an analysis of the level of global connectivity and performance of the country’s network infrastructure before, during and after the outage. The statistics and graphs are based entirely on the PingER data. This includes monitoring various nodes from SLAC and the World. Some measurements have also been made from within Pakistan. This outage also gave us an opportunity to study the overall network infrastructure in the country, with a few very interesting results (Link to the country’s case study).

The data obtained is a result of monitoring the following hosts in Pakistan. Unfortunately, we do not have meaningful data for any host located in Karachi, which would have further broadened the scope of our research, by covering the three big cities in the country.

Nick Name	Node Name	Service Provider	Name of Institute/Node	Location of Host
PK.NIIT.EDU.N1	www.niit.edu.pk	NTC	NUST Institute of Information Technology	Rawalpindi
PK.QAU.EDU.N1	www.qau.edu.pk	NTC	Quaid-e-Azam University	Islamabad
PK.UET.EDU.N1	www.uet.edu.pk	NTC	University of Engineering and Technology, Lahore	Lahore
PK.PIEAS.EDU.N1	www.pieas.edu.pk	NTC	Pakistan Institute of Engineering and Applied Sciences	Islamabad
PK.DSL.NET.SVR.N2	mbl.dsl.net.pk	Micronet Broadband	Micronet Broadband	Islamabad
PK.DSL.NET.N1	www.dsl.net.pk	Micronet Broadband	Micronet Broadband	Islamabad
PK.DSL.NET.GTWY.N1	lo-0-gw.dsl.net.pk	Micronet Broadband	Micronet Broadband	Islamabad
PK.LCWU.EDU.N1	www.lcwu.edu.pk	Brain Net	Lahore College for Women University	Lahore
ORG.WB.SDNPK.N1	www.wb.sdnpk.org	Habib Rafiq	Area Development Programme, Balochistan	Rawalpindi

Table: 1

Note: This data has been deduced by traceroute results to these nodes.

Effect of Fiber Outage June-July 2005

The effect of the fiber outage is studied in a fair amount of detail. Although the backup satellite connectivity was provided, but the quality of the link, as expected, was very poor. Moreover, it was perceived that the full connectivity would be restored in 3-4 days, but in actual lasted for 12 days (http://www.jang.com.pk/thenews/jun2005-daily/30-06-2005/main/main13.htm).

Fig: 1

The sudden increase in loss seen towards the end of June indicates that during the fiber outage, the reliability of the connectivity downgraded to a huge extent. The packet loss from SLAC to Pakistan had increased from a few percent to over 10% and even beyond, before complete connectivity was lost. Quality levels for packet loss were set at 0-1% = good, 1-2.5% = acceptable, 2.5-5% = poor, 5%-12% = very poor, and greater than 12% = bad (http://www.slac.stanford.edu/comp/net/wan-mon/tutorial.html#loss). The median value in Fig 1 indicates that the outage period falls under the “bad” category, but data from Feb’04 and onwards shows that the overall connectivity of the Pakistani universities seen from SLAC in the US is poor. This, to some extent, also explains why real-time applications like streaming video and voice/video conferences over IP are of poor quality in the country.

Fig 2: Shows the minimum RTT measured by PingER from SLAC to the various universities in Pakistan

As the main fiber connectivity link halted, there was a sudden jump in the Minimum Round Trip Times as well. This is mainly due to the change from a land-line link to a geostationary satellite which imposes an RTT of ~ 600ms. This did not only happen with one or two nodes but every node located in Pakistan. Under normal circumstances, the traffic is routed via Singtel (Singapore).

The minimum RTT has remained fairly consistent for most universities since Dec’ 03, but the fiber outage proved disastrous. This explains why the call center industry in the country had come to a standstill as VoIP traffic via satellite had huge delays not suitable for real-time communication.

Fig 3

Usually, nodes in Pakistan are fairly reachable. However, there was a sudden increase in “Ping Unreachability” to Pakistan due to the fault. The big blue spike (PK.NIIT.EDU.N2) is not due to problems in the fiber outage. This node was down for some due to other reasons. Towards the end of June, the percentage of unreachability soared as high as 18-19% for most nodes. During the outage, quite a few nodes in Pakistan were not reachable from the outside world. Since the satellite link could not support the volume of traffic, a lot of packets were dropped.

RTT and Packet Loss from SLAC to NIIT

Fig 4

These graphs show that the links have had huge amount of packet losses. At least once a month, the losses go as high as 20% (small circles) and occasionally, reaching the cent per cent mark (red circle). This goes to show the extreme unreliability of the link. The big yellow circle towards the end shows fiber outage in Pakistan that paralyzed the country’s Internet users for almost twelve days. However, even other than the outage, the country’s link has not been exemplary either. With a lot of variation in the packet loss and slight in the RTT values, serious efforts are required in order to improve the overall infrastructure. The sudden jump in Average RTT and Packet Loss values suggest that the satellite link was fairly congested. A possible reason could be that the call-center traffic in the country was being provided highest priority (http://www.jang.com.pk/thenews/jul2005-daily/02-07-2005/main/main18.htm).

The unreachability of Pings from SLAC increased by magnitude of 5-6 before the connection was totally disrupted, as shown by the red circle. The green circle is not due to the outage. Actually, this machine was down for some time. However, as we see PK.NIIT.EDU.N1, which is a monitored node at NUST Institute of Information Technology, there have been many times when the unreachability factor has gone fairly up. As we went deeper into the data, we noticed that the last week of May’05 had a very high unreachability factor, especially the 28th of May. Also, the overall network of the country experienced problems between the 21^st and 24^th of May. This unreliability was evident in the case of NIIT, which is being provided DSL service by the NTC. Upon further investigation, it was revealed that even the Quaid-e-Azam University, (www.qau.edu.pk) and PIEAS (www.pieas.edu.pk) also faced a lot of problems. Both these universities are also provided broadband service by NTC.

Comparison of the Links to Various Universities during the Internet Outage

We also carried out a comparison of the overall reliability (based on average Packet Loss) from SLAC to various universities in the country. NIIT was found to be the most disrupted link during this period, with an average packet loss of around 20%. Packet Loss of over 12% makes link fairly unusable (http://www.slac.stanford.edu/comp/net/wan-mon/tutorial.html#loss).

Fig 5