International Committee for Future Accelerators (ICFA)

Standing Committee on Inter-Regional Connectivity (SCIC)

Chairperson: Professor Harvey Newman, Caltech

ICFA SCIC Network Monitoring Report

Prepared by the ICFA SCIC Monitoring Working Group

On behalf of the Working Group:
Les Cottrell cottrell@slac.stanford.edu

January 2006 Report of the ICFA-SCIC Monitoring Working Group

Edited by R. Les Cottrell and Aziz A. Rehmatullah on behalf of the ICFA-SCIC Monitoring WG

Created January 18, 2006. Last Update February 2, 2006

ICFA-SCIC Home Page | Monitoring WG Home Page

This report is available from http://www.slac.stanford.edu/xorg/icfa/icfa-net-paper-jan06/

Executive Overview

Internet performance is improving each year with packet losses typically improving by 40-50% per year and losses by 25%-45% per year, and for some regions such as S. E. Europe, even more. Geosynchronous satellite connections are still important to countries with poor telecommunications infrastructure. However, the number of countries with fiber connectivity has and continues to increase and in most cases, satellite links are used as backup or redundant links. In general for HEP countries satellite links are being replaced with land-line links with improved performance (in particular for RTT). On the other side of the coin number of Internet usage is increasing (see http://www.internetworldstats.com/stats.htm), the application demands (see for example [bcr]) are growing and the expected reliability is increasing, so we cannot be complacent.

In general, throughput measured from within a region is much higher than when measured from outside. Links between the more developed regions including Anglo America, Japan and Europe are much better than elsewhere (3 - 10 times more throughput achievable). Regions such as Russia, S.E. Asia, S.E. Europe and Latin America are 3-6 years behind. Russia and S.E. Asia are catching up slowly. However, Africa, S. Asia and C. Asia are 8-10 years behind and even worse appear to be falling further behind. Looking forward ten years to 2015, if the current rates of progress continue, then performance from N. America to Africa will be 1000 time worse than to Europe, to S. Asia and C. Asia will be 100 times worse than to Europe.

Africa and South Asia are two regions where the internet has seen phenomenal growth, especially in terms of usage. However, it appears that network capacity is not keeping up with demand in these regions. In fact many sites in Africa and India appear to have throughputs less that that of a well connected (cable, DSL or ISDN) home in Europe or Anglo America. Further the end-to-end networking is often very fragile both due to last mile effects and poor infrastructure (e.g. power) at the end sites, and also due to lack of adequate network backup routes.

For modern HEP collaborations and Grids there is an increasing need for high-performance monitoring to set expectations, provide planning and trouble-shooting information, and to provide steering for applications.

There is a positive correlation between the various economic and development indices. Besides being useful in their own right these indices are an excellent way to illustrate anomalies and for pointing out measurement/analysis problems. The large variations between sites within a given country illustrate the need for careful checking of the results and the need for multiple sites/country to identify anomalies.

To quantify and help bridge the Digital Divide, enable world-wide collaborations, and reach-out to scientists world-wide, it is imperative to continue and extend the PingER monitoring coverage to all countries with HEP programs and significant scientific enterprises. On the other hand, more support is required from these countries to enable adding more sites to PingER. Based on a few weeks of data, we have tried to highlight the problems in Pakistan’s case study. Similarly, support from people in India, Bangladesh and other developing countries is required not only to add remote nodes but also monitoring sites with remote nodes tailored to fit the needs of that region. This in turn will require help from ICFA to identify sites to monitor and contacts for those sites, plus identifying sources of on-going funding support to continue and extend the monitoring.

Introduction

The formation of this working group was requested at the ICFA/SCIC meeting at CERN in March 2002 [icfa-mar02]. The mission is to: Provide a quantitative/technical view of inter-regional network performance to enable understanding the current situation and making recommendations for improved inter-regional connectivity.

The lead person for the monitoring working group was identified as Les Cottrell. The lead person was requested to gather a team of people to assist in preparing the report and to prepare the current ICFA report for the end of 2002. The team membership consists of:

Table 1: Members of the ICFA/SCIC Network Monitoring team

Les Cottrell	SLAC	US	cottrell@slac.stanford.edu
Richard Hughes-Jones	University of Manchester	UK	rich@a3.ph.man.ac.uk
Sergei Berezhnev	RUHEP, Moscow State.Univ.	Russia	sfb@radio-msu.net
Sergio F. Novaes	FNAL	S. America	novaes@fnal.gov
Fukuko Yuasa	KEK	Japan and E. Asia	Fukuko.Yuasa@kek.jp
Sylvain Ravot	Caltech	CMS	Sylvain.Ravot@cern.ch
Shawn McKee	Michigan	I2 HEP Net Mon WG	smckee@umich.edu

Goals of the Working Group

Obtain as uniform picture as possible of the present performance of the connectivity used by the ICFA community
Prepare reports on the performance of HEP connectivity, including, where possible, the identification of any key bottlenecks or problem areas.

This report may be regarded as a follow on to the May 1998 Report of the ICFA-NTF Monitoring Working Group [icfa-98], the January 2003 Report of the ICFA-SCIC Monitoring Working Group [icfa-03] the January 2004 Report of the ICFA-SCIC Monitoring Working Group [icfa-04] and the January 2005 Report of the ICFA-SCIC Monitoring Working Group [icfa-05]. The current report updates the January 2005 report, but is complete in its own right in that it includes the tutorial information from the previous reports.

Methodology

There are two complementary types of Internet monitoring reported on in this report.

In the first we use PingER [pinger] which uses the ubiquitous "ping" utility available standard on most modern hosts. Details of the PingER methodology can be found in the May 1998 Report of the ICFA-NTF Monitoring Working Group [icfa-98] and [ejds-pinger]. PingER provides low intrusiveness (~ 100bits/s per host pair monitored¹) Round Trip Time (RTT), loss, reachability (if a host does not respond to a set of 21 pings it is presumed to be non-reachable). The low intrusiveness enables the method to be very effective for measuring regions and hosts with poor connectivity. Since the ping server is pre-installed on all remote hosts of interest, minimal support is needed for the remote host (no software to install, no account needed etc.)
The second method (IEPM-BW [iepm]) is for measuring high network and application throughput between hosts with excellent connections. Examples of such hosts are to be found at HEP accelerator sites and tier 1 and 2 sites, major Grid sites, and major academic and research sites in Anglo America², Japan and Europe. The method can be quite intrusive (for each remote host being monitored from a monitoring host, it can utilize hundreds of Mbits/s for ten seconds to a minute each hour). It also requires more support from the remote host. In particular either various services must be installed and run by the local administrator or an account is required, software (servers) must be installed, disk space, compute cycles etc. are consumed, and there are security issues. The method provides expectations of throughput achievable at the network and application levels, as well as information on how to achieve it, and trouble-shooting information.

PingER Results

The PingER data and results extend back to the start of 1995. They thus provide a valuable history of Internet performance. PingER has 34 monitoring nodes in 14 regions, that monitor 1037 remote nodes at over 750 sites in around 120 countries (see PingER Deployment [pinger-deploy]). These countries contain over 90% of the world's population (see Table 2) and over 99% of the online users of the Internet. Most of the hosts monitored are at educational or research sites. We try and get at least 2 hosts per country to help identify and avoid anomalies at a single host, although we are making efforts to increase the number of monitoring hosts to as many as we can. The requirements for the remote host can be found in [host-req]. Fig. 1 below shows the locations of the monitoring and remote (monitored sites).

Figure 1: Locations of PingER monitoring and remote sites as of Jan 2006.

There are around thirty seven hundred monitoring/monitored-remote-host pairs, so it is important to provide aggregation of data by hosts from a variety of "affinity groups". PingER provides aggregation by affinity groups such as HEP experiment collaborator sites, Top Level Domain (TLD), Internet Service Provider (ISP), or by world region etc. The world regions, as defined for PingER, and countries monitored are shown below in Fig. 2. The regions are chosen starting from the U.N. definitions [un]. We modify the region definitions to take into account which countries have HEP interests and to try and ensure the countries in a region have similar performance.

Figure 2: Major regions of the world for PingER aggregation by regions

More details on the regions are provided in Table 2 that highlights the number of countries monitored in each of these regions, and the distribution of population in these regions.

Table 2: Countries and populations by region

Regions	# of Countries	% of World Population	% of Monitored Population
Africa	31	11.5	12.8
Balkans	9	1.0	1.1
Central Asia	4	1.0	1.1
Europe	25	7.9	8.8
Latin America	18	8.2	9.1
North America	2	5.1	5.7
East Asia	4	23.2	25.9
South East Asia	6	6.2	6.9
South Asia	5	22.5	25.0
Middle East	5	2.8	3.1
Oceania	5	0.5	0.5
Russia	1	2.2	2.5

To assist in interpreting the results in terms of their impact on well-known applications, we categorize the losses into quality ranges. These are shown below in Table 3.

Table 3: Quality ranges used for loss
	Excellent	Good	Acceptable	Poor	Very Poor	Bad
Loss	<0.1%	>=0.1% & < 1%	> =1% & < 2.5%	>= 2.5% & < 5%	>= 5% & < 12%	>= 12%

More on the effects of packet loss and RTT can be found in the Tutorial on Internet Monitoring & PingER at SLAC [tutorial], briefly:

At losses of 4-6% or more video-conferencing becomes irritating and non-native language speakers become unable to communicate. The occurrence of long delays of 4 seconds (such as may be caused by timeouts in recovering from packet loss) or more at a frequency of 4-5% or more is also irritating for interactive activities such as telnet and X windows. Conventional wisdom among TCP researchers holds that a loss rate of 5% has a significant adverse effect on TCP performance, because it will greatly limit the size of the congestion window and hence the transfer rate, while 3% is often substantially less serious, Vern Paxson. A random loss of 2.5% will result in Voice Over Internet Protocols (VOIP) becoming slightly annoying every 30 seconds or so. A more realistic burst loss pattern will result in VOIP distortion going from not annoying to slightly annoying when the loss goes from 0 to 1%. Since TCP throughput for the standard (Reno based) TCP stack goes as 1/(sqrt(loss) [mathis]) (see M. Mathis, J. Semke, J. Mahdavi, T. Ott, "The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm",Computer Communication Review, volume 27, number 3, pp. 67-82, July 1997), it is important to keep losses low for achieving high throughput.
For RTTs, studies in the late 1970s and early 1980s showed that one needs < 400ms for high productivity interactive use. VOIP requires a RTT of < 250ms or it is hard for the listener to know when to speak.

It must be understood that these quality designations apply to normal Internet use. For high performance, and thus access to data samples and effective partnership in distributed data analysis, much lower packet loss rates may be required.

Loss

Of the two metrics loss & RTT, loss is more critical since a loss of a packet will typically cause timeouts that can last for several seconds, moreover, RTT increases with increase in distance between any two nodes and also, with the increase in the number of hops, whereas loss is less distance dependent. For instance RTT between a node at SLAC and somewhere in Europe is expected to be around 160ms.

Figure 3: December 2005 packet loss snapshot seen from USA sites to the world.

Fig. 3 shows a snapshot of the losses for December 05. We observe that very few countries have bad connectivity. Most of N. America, Europe, Oceania and Russia have excellent or good performance, meaning that the average packet loss is less than 1%.

Another way of looking at the losses is to see how many hosts fall in the various loss quality categories defined above as a function of time. An example of such a plot is seen in Fig 4.

Figure 4: Number of hosts measured from SLAC for each quality category from February 1998 through December 2005.

It can be seen that recently most sites fall in the good quality category. The numbers at the bottom indicate the percentage of total sites that see good or better packet loss at the start of the year. Also the number of sites with good quality has increased from about 55% to about 75% in the 9 years displayed. The plot also shows the increase in the total number of sites monitored from SLAC over the years. The improvements are particularly encouraging since most of the new sites are added in developing regions.

Towards the end of 2001 the number of sites monitored started dropping as sites blocked pings due to security concerns. The rate of blocking was such that out of 214 hosts that were pingable in July 2003, 33 (~15%) were no longer pingable in December 2003.

The increases in monitored sites towards the end of 2002 and early 2003 was due to help from the Abdus Salam Institute of Theoretical Physics (ICTP). The ICTP held a Round Table meeting on Developing Country Access to On-Line Scientific Publishing: Sustainable Alternatives [ictp] in Trieste in November 2002 that included a Proposal for Real time monitoring in Africa [africa-rtmon]. Following the meeting a formal declaration was made on RECOMMDENDATIONS OF the Round Table held in Trieste to help bridge the digital divide [icfa-rec]. The PingER project is collaborating with the ICTP to develop a monitoring project aimed at better understanding and quantifying the Digital Divide. On December 4th the ICTP electronic Journal Distribution Service (eJDS) sent an email entitled Internet Monitoring of Universities and Research Centers in Developing Countries [ejds-email] to their collaborators informing them of the launch of the monitoring project and requesting participation. By January 14th 2003, with the help of ICTP, we added about 23 hosts in about 17 countries including: Bangladesh, Brazil, China, Columbia, Ghana, Guatemala, India (Hyderabad and Kerala), Indonesia, Iran, Jordan, Korea, Mexico, Moldova, Nigeria, Pakistan, Slovakia and the Ukraine.

The increase towards the end of 2003 was spurred by preparations for the second Open Round Table on Developing Countries Access to Scientific Knowledge: Quantifying the Digital Divide 23-24 November Trieste, Italy and the WSIS conference and associated activities in Geneva December 2003.

The increases in 2004 were due to adding new sites especially in Africa, S. America, Russia and several outlying islands. See Fig. 1 and section “Accomplishments since last report”.

In 2005, the Pakistan Ministry Of Science and Technology (MOST) and the US State Department funded SLAC and the National University of Sciences and Technology’s (NUST) Institute of Information Technology (NIIT) to collaborate on a project to improve and extend PingER. As part of this project and the increased interest from Internet2 in “Hard to Reach Network Places” many new sites in the South Asia and Africa were added to increase the coverage in these regions and also to replace sites that were blocking pings. For instance we can find no sites in Angola that are pingable in Dec 2005. Also as part of this project we started to integrate PingER with the NLANR/AMP project and as a result a number of the AMP nodes were added as PingER remote hosts in the developing regions. With help of Duncan Martin and the South Africa Tertiary Education Network (TENET) (http://www.tenet.ac.za), we successfully set up a monitoring node in South Africa, which should be a great help in viewing the Digital Divide from within the Divide. With the help of NIIT (www.niit.edu.pk), a monitoring node was set up at NIIT and in Nov’ 05 another one at NTC (National Telecommunication Corporation www.ntc.net.pk), which is the service provider for the PERN (Pakistan Educational and Research Network www.pern.edu.pk). Although it is too early to provide any long terms predictions, more than almost two months of data gathered indicate certain interesting results which will be discussed later in more detail.

Fig. 5 below shows the long term trends for the various regions as seen from Anglo America.

Figure 5: Packets loss trends from Anglo America to various regions of the world.

The following general observations can be made for the losses:

For most regions the improvement in losses is typically between 25% and 45% per year.
The better rgions are achieving better than 1% packet for most of their sites seen from SLAC.

Fig. 6 shows the world's connected population fractions obtained by dividing countries up by loss quality seen from the US, and adding the connected populations for the countries (we obtained the population/country figures from "How many Online" [nua] for 2001 and from CIA World Factbook for 2005 [cia-pop-figures]).

Figure 6: Fraction of the world's connected population in countries with measured loss performance in 2001 and Dec 2005

It can be seen that in 2001, <20% of the population lived in countries with acceptable or better packet loss. By December 2005 this had risen to 79%. The coverage of PingER has also increased from about 70 countries at the start of 2003 to over 120 in December 2005. This in turn reduced the fraction of the connected population for which PingER has no measurements. The results are even more encouraging when one bears in mind that the newer countries being added typically are from regions that have traditionally poorer connectivity.

It is interesting to compare the packet losses seen by various regions with those seen by residential DSL customers in the San Francisco Bay Area. This is shown in Fig. 7 below.

Figure 7: Losses from SLAC to various world regions compared with that for residential customers in the San Francisco Bay Area.

RTTs

There are limits to the minimum RTT due to the speed of light in fibers or electricity in copper. Typically today, the minimum RTTs for terrestrial circuits are about 2 * distance / ( 0.6 * c), or roughly 100km/ms (RTT time,) where c is the velocity of light, the factor of 2 accounts for the round-trip, 0.6*c is roughly the speed of light in fibre. For geostationary satellites links, the minima are between 500 and 600ms. For U.S. cross country links (e.g. from SLAC to BNL) the typical minimum RTT (i.e. a packet sees no queuing delays) is about 70 msec.

Fig. 8 below shows the trends of the min-RTT measured from ESnet sites in Anglo America to the various regions of the world. The straight lines are exponential fits to the data (straight lines on a log-linear plot), and the wiggly lines are moving averages for the last 6 months.

Figure 8: Minimum RTT measured from ESnet sites in the US to sites in regions of the world

As is seen by comparing the exponential fits with the moving averages, the trends here are less clear. Europe and the Balkans and to a lesser extent Russia have been pretty stable since upgrading the links from say 45 to 155, 622 or 2400 or 10,000 Mbps implying that for high speed links, the actual link speeds have a small effect on the minimum RTT, the main effect being the distance. Central Asia on the other hand has been stuck with geo-stationary satellites and so little change is seen for it. The minimum RTT for Africa is partly increasing since we are extending the monitoring to reach more distant countries and more countries with satellite links. South Asia has been gradually upgrading the links within and outside the countries. Also, as is evident from the year 2000 minimum RTT map in Fig 9 below, India and Pakistan have moved from satellite to fiber optics, resulting in a decline in the minimum RTT values. Latin America took a huge step down in RTT at the end of 1999 going from mainly satellite (>500ms) to 200ms (i.e. mainly landlines). S.E. Asia looks like a gradual improvement.

Fig. 9 shows the RTT from the U.S. to the world in January 2000 and December 2005. It also indicates which countries of the world contain sites that were monitored (in the Jan 2000 map countries in green are not monitored, in the Dec 2005 apart from the US unmonitored countries are left white).

Figure 9: December 05 comparison of Minimum RTT with 2003 and 2000 results

It is seen that the number of countries with satellite links (> 600ms RTT or dark red) has decreased markedly in the 6 years shown. Today satellite links are used in places where it is hard or unprofitable to pull terrestrial-lines (typically fibers) to. Barring a few countries in Central and Eastern Africa, Bangladesh and Nepal most of the countries being monitored by PingER now have optical fiber connectivity. The Eastern Africa Submarine Cable System (EASSy [EASSy]) Project has been established to develop and implement a submarine cable system to provide fibre optic telecommunications facilities to the Eastern coast of Africa. The partners include Botswana, Burundi, Djibouti, Ethiopia, Kenya, Madagascar, Mozambique, Rwanda, Somalia, South Africa, Sudan, Zanzibar, Tanzania and Uganda so this could make a big improvement for this area. This cable will also link Northern and Southern African international gateways to the system.

Two interesting examples stand out in this data: Niger and Mali. Both countries have minimum RTTs greater than 600ms, indicating that both these links are satellite. However as seen in Fig. 3, the link loss quality from the US to the sites monitored in these countries is fairly good, with packet loss around 1%. This is much better than most of the South Asia, where the quality of the links are barely acceptable.

Throughput

We also combine the loss and RTT measurements using throughput = 1460Bytes[Max Transmission Unit]/(RTT * sqrt(loss)) from [mathis]. The results are shown in Fig. 10. The orange line shows a ~40% improvement/year or about a factor of 10 in < 7 years.

Figure 10: Derived throughput as a function of time seen from ESnet sites to various regions of the world. The numbers in parentheses are the number of monitoring/remote host pairs contributing to the data. The lines are exponential fits to the data.

The data for several of the developing countries only extends back for only about five years so some care must be taken in interpreting the long term trends. With this caveat, it can be seen that links between the more developed regions including Anglo America, Japan and Europe are much better than elsewhere (3 - 10 times more throughput achievable). Regions such as Russia, S.E. Asia, S.E. Europe and Latin America are 3-6 years behind. Russia and S.E. Asia are catching up slowly. However, Africa, S. Asia and C. Asia are 8-10 years behind and even worse appear to be falling further behind. Looking forward ten years to 2015, if the current rates of progress continue, then performance from N. America to Africa will be 1000 time worse than to Europe, to S. Asia and C. Asia will be 100 times worse than to Europe.

View from Europe

To assist is developing a less N. American view of the Digital Divide; we added many more hosts in developing countries to the list of hosts monitored from CERN in Geneva Switzerland. We now have data going back for almost four and a half years that enables us to make some statements about performance as seen from Europe. Fig. 11 shows the data from CERN as of September 2005. The lines are exponential fits to the data.

Figure 11: Derived throughputs to various regions as seen from CERN

The slow increase for N. America is partially an artifact of the difficulty of accurately measuring loss with a relatively small number of pings (14,400 pings/month at 10 pings/30 minute interval, i.e. a loss of one packet ~ 1/10,000 loss rate). The very slow increase in throughput for the Middle East, is an artifact caused by initially only monitoring hosts in 2 Middle East countries (Israel and Egypt) with one (Israel) having markedly better performance (factor of 20) than anywhere else in the Middle East. As we added hosts in more Middle East Countries (starting in July 2003), the median dropped dramatically as Israel had less effect. We have added several hosts to the Mid-East based on hosts being successfully monitored from SLAC. Apart from the special case of the Middle East mentioned above, the trends are similar to those seen from ESnet/US: the improvements are between 50% and 100% per year; Russia and S. E. Europe (Balkans) and to a lesser extent Latin America are catching up with Europe; the Middle East and S. Asia are falling behind. There is insufficient data at the moment to indicate how far the various regions are behind N. America or how long it will take to catch up

Variability of performance between and within regions

The throughput results, so far presented in this report, have been measured from Anglo America or to a lesser extent from Europe. This is partially since there is more data for a longer period available for the Anglo America monitoring hosts. Table 4 shows the throughputs seen between monitoring and remote/monitored hosts in the major regions of the world. Each column is for monitoring hosts in a given region, each row is for monitored hosts in a given region. The cells are colored according to the median quality for the monitoring region/monitored region pair. White is for derived throughputs > 5000 kbits/s (good), blue for <= 5000 kbits/s and >1000kbits/s (acceptable), yellow for <= 100kbits/s and > 500 kbits/s, and magenta for <= 500kbits/s (very poor to bad). The table is column ordered by decreasing median performance. The rows are sorted by region. The Monitoring countries are identified by the Internet two-character Top Level Domain (TLD). Just for the record CH=Switzerland, DE=Denmark, HU=Hungary, CA=Canada, RU=Russia, JP=Japan, BR=Brazil, IN=India, PK=Pakistan and ZA=South Africa. S. Asia is the Indian sub-continent; S.E. Asia is composed of measurements to Indonesia, Malaysia, Singapore, Thailand and Vietnam.

Table 4: Derived throughputs in kbits/s from monitoring hosts to monitored hosts by region of the world for December 2005

As expected it can be seen that within regions (the circled cells) performance is generally better than between regions. Also performance is better between closely located regions such as Europe and S. E. Europe, Russia and Europe, Russia and S.E. Europe.

To provide further insight into the variability in performance for various regions of the world seen from SLAC Fig. 12 shows various statistical measures of the losses and derived throughputs. The regions are sorted by the median of the measurement type displayed. Note the throughput graph uses a log y-scale to enable one to see the regions with poor throughput. The countries comprising a region can be seen in Fig. 2.

Figure 12: 25 percentile, median and 75 percentile derived throughputs and losses for various regions measured from SLAC for Oct-Dec '05

The difference in throughput for N. America and Europe is an artifact of the measurements being made from N. America (SLAC) which has a much shorter RTT (roughly between a factor of 2 and 20 times or for the average sites close to 3 to 4) to N. American than to European sites. Since the derived throughput goes as 1/RTT this favors N. America by about a factor of 3 to 4 times. The most uniform region (in terms of Inter-Quartile-Range/median for both derived throughput and loss) is Central Asia, probably since all the paths use a geo-stationary satellite. The most diverse are N. America and Europe. For Europe, Belorussia stands out with poor performance. Hopefully the Porta-Optica project (http://www.porta-optica.org) will improve this situation.

Case Study on NIIT, Pakistan

With NIIT being an important collaborator with SLAC, Caltech and CERN, we prepared a small case study with 3 PingER monitoring sites in Pakistan to provide a brief overview and a measure of the issues at NIIT in particular and Pakistan in general.

The Pakistan Education and Research Network (PERN) “is a nationwide educational intranet connecting premiere educational and research institutions of the country. PERN focuses on collaborative research, knowledge sharing, resource sharing, and distance learning by connecting people through the use of Intranet and Internet resources”.

PERN uses the services of NTC (National Telecommunication Corporation), which is the national telecommunication carrier for all official/government services in Pakistan, for the provision of infrastructure and bandwidth to the universities in Pakistan. The PingER project worked with NTC install a PingER monitoring site at NTC headquarters in Islamabad, to monitor the performance of various universities connected to PERN. This data was compared with that from two PingER monitoring hosts at NIIT. One of the hosts uses NTC/PERN to provide external connectivity at 1-1.5Mbits/s, the second uses Micronet, a commercial network with a 512kbits/s connection to NIIT. We analyzed the data to compare the results from the three monitoring hosts to a common set of sites in Pakistan, for seven weeks, from 7^th Dec 2005 thru 28^th Jan 06. All the sites were connected to PERN/NTC and information about them is provided in Table 5.

Table 5: Remote sites in Pakistan monitored from NIIT

	Remote Node	University Location	Service Provider	Traffic Enters the Country Via	End host location

1	PK.QAU.EDU.N1	Islamabad	NTC	Karachi