APerformance Comparison of TCP IP and MPI
on FDDI Fast Ethernet and Ethernet
Saurab Nog and David Kotz
Department of Computer Science
Dartmouth College
Hanover NH
saurab cs dartmouth edu dfk cs dartmouth edu
Dartmouth Technical Rep ort PCS TR
November
Revised January
Abstract
Communication is a very imp ortant factor a ecting distributed applications Getting a
close handle on network p erformance b oth bandwidth and latency is thus crucial to un
derstanding overall application p erformance Webenchmarked some of the metrics of net
work p erformance using twosetsofexperiments namely roundtrip and datahose The tests
were designed to measure a combination of network latency bandwidth and contention
We rep eated the tests for two proto cols TCP IP and MPI and three networks Mbit
FDDI Fib er Distributed Data Interface Mbit Fast Ethernet and Mbit Ethernet
The p erformance results provided interesting insights into the b ehaviour of these networks
under di erent load conditions and the software overheads asso ciated with an MPI imple
mentation MPICH This do cument presents details ab out the exp eriments their results
and our analysis of the p erformance
Keywords Network Performance FDDI FastEthernet Ethernet MPI
Revised January to emphasize our use of a particular MPI implementation MPICH
Copyright 1995 by the authors
INTRODUCTION
Intro duction
This do cument presents b enchmarking tests that measured and compared p erformance of
TCP IP and MPI Message Passing Interface implementations on di erentnetworks
and their results
Wechose these proto cols TCP IP and MPI b ecause
TCP IP
Greater exibility and user control
Higher p erformance
Wide usage and supp ort
MPI
Programming ease
A widely accepted standard for distributed computing applications
Since our MPI implementation MPICH runs on top of P which in turn uses TCP IP
a p erformance comparison of TCP IP and MPI givesagoodestimateoftheoverheads and
advantages asso ciated with using the higher level abstraction of MPI as opp osed to TCP IP
so ckets
We also compared three di erentnetworks to understand the inherent hardware and soft
ware limitations of our setup We rep eated the same set of exp eriments on
Mbps FDDI Fib er Distributed Data Interface token ring
Mbps Fast Ethernet
Mbps Ethernet
In most cases we cross checked our results with results from the publicly available ttcp
package The results seem to b e in very close agreement including the presence of spikes
and other anomalies
Setup Details
We ran two di erent tests roundtrip and datahose on all six combinations of proto col
TCP IP and MPI and network FDDI Fast Ethernet Ethernet Although our exp eri
ments were not sp eci c to any implementation of the hardware or software standards the
results were necessarily dep endent on the sp eci c implementations Other implementations
mayhave di erent p erformance
SETUP DETAILS
Hardware Software
The FDDI and Ethernet exp eriments used RS s running AIX at MHz The
FDDI was an isolated network and the Ethernet exp eriments were run when the Ethernet
was lightly loaded usually overnight
The Fast Ethernet tests used Pentium based PCs with MB RAM running FreeBSD
on an isolated network
Both the RS s and PCs had MPICH version released July running on
top of P
Environment
All the tests except those involving Ethernet were conducted in an isolated mo de For the
Ethernet exp eriments we did not disconnect ourselves from the rest of the world so wewere
careful to cho ose quiet times of the day for testing This minimized the risk of interference
from external tra c
We rep eated the tests several times at least running a few thousand iterations of each
test every time Wechose the b est values max for bandwidths and min for time as our
nal result Thus our numb ers approximate b est case p erformance results
NODELAY option except the Fast Ethernet datahose All TCP IP tests used the TCP
Normally the TCP IP proto col delays small packet transmission in the hop e of gaining
advantage by coalescing a large numb er of small packets This delay results in arti cially
high latency times for small packets We used TCP NODELAYtoovercome this b ehaviour
and eliminate the wait p erio d We also set b oth the sending and receiving side kernel
bu ers at K bytes maximum allowed on the RS s FreeBSD Pentiums allowupto
K The increased bu ers were prompted by the observation thatkernel bu ers are
the b ottlenecks for most network op erations
For MPI we used the default con guration in all cases
Performance Measures
In these tests wewere interested in the following measures of network p erformance
Total Bandwidth the sum of the individual bandwidths of concurrent pro cesses It
signi es the e ective bandwidth usage for the set of all participating hosts
Bandwidth Pair of pro cesses the average bandwidth each individual pair of pro cesses
was able to use
Time iteration the average time it takes to complete one iteration of roundtrip or
datahose
FreeBSD often crashed when o o ded with small messages
ROUNDTRIP
Roundtrip
The roundtrip test is designed to measure the following parameters
network latency
network contention overhead
synchronization overheads
All machines were connected to the same network but were paired for communication pur
p oses Each host talks to its designated partner only There is a master slave relationship
between the two pro cess in each pair The master initiates the communication and then
measures the time interval it takes for the slave to receive and send the message back This
completes one roundtrip iteration Our test do es many roundtrip iterations in a burst to
determine the average time iteration and the network bandwidth achieved Many iterations
are necessary to overcome clo ckgranularity cold cache e ects and accuracy problems
All pro cesses synchronize b efore switching to the next message size This serves twomajor
purp oses
Each message size is timed separately restricting timing and other errors to the phase
in which they o ccured
Synchronization prevents pro cesses from running ahead or lagging b ehind the group
Roundtrip is a store and forward test Eachslave pro cess receives the complete message
from its master and then sends it back Since at anygiven time only pro cess in a pair is
writing data to the network there is no contention b etween the master and slave pro cesses
of a pair
We ran roundtrip on and pairs up to pairs on FreeBSD of pro cessors simulta
neously to determine howcontention in uences network p erformance Results for various
combinations of network and proto col are presented in the following graphs
ROUNDTRIP
The pseudo co de for roundtrip is as follows
for all interesting message sizes
set message size
synchronize all master and slave processes
if master
I am the master
start timing Initialize timer
do a large number of iterations
for iteration count iteration count MAX ITERATIONS iteration count
Initiate communication with slave
write slave message message size
Wait for reply from slave
read slave buffer message size
stop timing Stop timer
else
I am the slave
do a large number of iterations
for iteration count iteration count MAX ITERATIONS iteration count
Wait for the master to initiate communication
read master buffer message size
Reply back to the master with the same message
write master buffer message size
End if else
End for
ROUNDTRIP
100
’roundtrip-tcp-fddi-4-totalbw’ ’roundtrip-tcp-fddi-3-totalbw’ ’roundtrip-tcp-fddi-2-totalbw’ 80 ’roundtrip-tcp-fddi-1-totalbw’
60
40 Bandwidth (Mbits/sec)
20
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip TCP IP FDDI Total Bandwidth
60
’roundtrip-tcp-fddi-4-bw’ ’roundtrip-tcp-fddi-3-bw’ ’roundtrip-tcp-fddi-2-bw’ 50 ’roundtrip-tcp-fddi-1-bw’
40
30 Bandwidth (Mbits/sec) 20
10
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip TCP IP FDDI Bandwidth p er Pair of Pro cesses Master Slave
1e+06
’roundtrip-tcp-fddi-4-time’ ’roundtrip-tcp-fddi-3-time’ ’roundtrip-tcp-fddi-2-time’ ’roundtrip-tcp-fddi-1-time’
100000
Averageg Time (micro-sec) 10000
1000 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip TCP IP FDDI Time p er Iteration
ROUNDTRIP
100
’roundtrip-mpi-fddi-4-totalbw’ ’roundtrip-mpi-fddi-3-totalbw’ ’roundtrip-mpi-fddi-2-totalbw’ 80 ’roundtrip-mpi-fddi-1-totalbw’
60
40 Bandwidth (Mbits/sec)
20
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip MPI FDDI Total Bandwidth
60
’roundtrip-mpi-fddi-4-bw’ ’roundtrip-mpi-fddi-3-bw’ ’roundtrip-mpi-fddi-2-bw’ 50 ’roundtrip-mpi-fddi-1-bw’
40
30 Bandwidth (Mbits/sec) 20
10
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip MPI FDDI Bandwidth p er Pair of Pro cesses Master Slave
1e+06
’roundtrip-mpi-fddi-4-time’ ’roundtrip-mpi-fddi-3-time’ ’roundtrip-mpi-fddi-2-time’ ’roundtrip-mpi-fddi-1-time’
100000
Averageg Time (micro-sec) 10000
1000 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip MPI FDDI Time p er Iteration
ROUNDTRIP
100
’roundtrip-tcp-fast-3-totalbw’ ’roundtrip-tcp-fast-2-totalbw’ ’roundtrip-tcp-fast-1-totalbw’ 80
60
40 Bandwidth (Mbits/sec)
20
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip TCP IP Fast Ethernet Total Bandwidth
60
’roundtrip-tcp-fast-3-bw’ ’roundtrip-tcp-fast-2-bw’ ’roundtrip-tcp-fast-1-bw’ 50
40
30 Bandwidth (Mbits/sec) 20
10
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip TCP IP Fast Ethernet Bandwidth p er Pair of Pro cesses Master Slave
1e+06
’roundtrip-tcp-fast-3-time’ ’roundtrip-tcp-fast-2-time’ ’roundtrip-tcp-fast-1-time’
100000
10000 Averageg Time (micro-sec)
1000
100 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip TCP IP Fast Ethernet Time p er Iteration
ROUNDTRIP
100
’roundtrip-mpi-fast-3-totalbw’ ’roundtrip-mpi-fast-2-totalbw’ ’roundtrip-mpi-fast-1-totalbw’ 80
60
40 Bandwidth (Mbits/sec)
20
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip MPI Fast Ethernet Total Bandwidth
60
’roundtrip-mpi-fast-3-bw’ ’roundtrip-mpi-fast-2-bw’ ’roundtrip-mpi-fast-1-bw’ 50
40
30 Bandwidth (Mbits/sec) 20
10
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip MPI Fast Ethernet Bandwidth p er Pair of Pro cesses Master Slave
1e+06
’roundtrip-mpi-fast-3-time’ ’roundtrip-mpi-fast-2-time’ ’roundtrip-mpi-fast-1-time’
100000
10000 Averageg Time (micro-sec)
1000
100 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip MPI Fast Ethernet Time p er Iteration
ROUNDTRIP
10
’roundtrip-tcp-ethernet-4-totalbw’ ’roundtrip-tcp-ethernet-3-totalbw’ ’roundtrip-tcp-ethernet-2-totalbw’ 8 ’roundtrip-tcp-ethernet-1-totalbw’
6
4 Bandwidth (Mbits/sec)
2
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip TCP IP Ethernet Total Bandwidth
10
’roundtrip-tcp-ethernet-4-bw’ ’roundtrip-tcp-ethernet-3-bw’ ’roundtrip-tcp-ethernet-2-bw’ 8 ’roundtrip-tcp-ethernet-1-bw’
6
4 Bandwidth (Mbits/sec)
2
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip TCP IP Ethernet Bandwidth p er Pair of Pro cesses Master Slave
1e+07
’roundtrip-tcp-ethernet-4-time’ ’roundtrip-tcp-ethernet-3-time’ ’roundtrip-tcp-ethernet-2-time’ ’roundtrip-tcp-ethernet-1-time’ 1e+06
100000 Averageg Time (micro-sec)
10000
1000 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip TCP IP Ethernet Time p er Iteration
ROUNDTRIP
10
’roundtrip-mpi-ethernet-4-totalbw’ ’roundtrip-mpi-ethernet-3-totalbw’ ’roundtrip-mpi-ethernet-2-totalbw’ 8 ’roundtrip-mpi-ethernet-1-totalbw’
6
4 Bandwidth (Mbits/sec)
2
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip MPI Ethernet Total Bandwidth
10
’roundtrip-mpi-ethernet-4-bw’ ’roundtrip-mpi-ethernet-3-bw’ ’roundtrip-mpi-ethernet-2-bw’ 8 ’roundtrip-mpi-ethernet-1-bw’
6
4 Bandwidth (Mbits/sec)
2
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip MPI Ethernet Bandwidth p er Pair of Pro cesses Master Slave
1e+07
’roundtrip-mpi-ethernet-4-time’ ’roundtrip-mpi-ethernet-3-time’ ’roundtrip-mpi-ethernet-2-time’ ’roundtrip-mpi-ethernet-1-time’ 1e+06
100000 Averageg Time (micro-sec)
10000
1000 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Roundtrip MPI Ethernet Time p er Iteration
DATAHOSE
DataHose
Datahose is designed to measure the raw bandwidth of the network The results are a
combination of the following factors
ability of a pro cess to pump data onto the network and the network s capabilityto
deliver it
network contention overhead
Unlike roundtrip datahose enforces no synchronization b etween the master and slave pro
cesses of a pair The master pro cess keeps writing data to the network and the slave reading
from it without caring ab out each other s state Packet owcontrol for datahose is thus
handled by TCP The time it takes for the slave to receive all data is the time for one
datahose iteration Many datahose iterations are done to get an accurate count
All datahose pro cess pairs synchronize b efore changing message sizes Synchronization is
done to prevent older messages from spilling over to the next stage This step is the only
time that a master and slave explicitly synchronize
Datahose o o ds the network with messages Since there is no owcontrol inherent in the test
except for that provided by TCP IP datahose with the TCP NODELAY option crashed
the FreeBSD Pentium machines rep eatedly This e ect forced us to un set TCP NODELAY
for the Fast Ethernet tests TCP IP is thus able to coalesce many small sized packets b efore
sending them over reducing the average time p er packet However for large messages
K TCP NODELAY has a minimal impact on p erformance
Datahose was run on and pairs up to pairs on FreeBSD of pro cessors si
multaneously to determine how the presence of other pro cesses on the network in uences
bandwidth pro cess Results for various combinations of network and proto col are presented
in the following graphs
DATAHOSE
The pseudo co de for datahose is as follows
for all interesting message sizes
set message size
synchronize all master and slave processes
if master
I am the master
do a large number of iterations
for iteration count iteration count MAX ITERATIONS iteration count
Just keep writing to the network
write slave message message size
else
I am the slave
start timing Initialize timer
do a large number of iterations
for iteration count iteration count MAX ITERATIONS iteration count
Keep reading from the network whatever the master process has written
read master buffer message size
stop timing Stop timer
End if else
End for
DATAHOSE
100
’datahose-tcp-fddi-4-totalbw’ ’datahose-tcp-fddi-3-totalbw’ ’datahose-tcp-fddi-2-totalbw’ 80 ’datahose-tcp-fddi-1-totalbw’
60
40 Bandwidth (Mbits/sec)
20
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose TCP IP FDDI Total Bandwidth
70
60 ’datahose-tcp-fddi-4-bw’ ’datahose-tcp-fddi-3-bw’ ’datahose-tcp-fddi-2-bw’ ’datahose-tcp-fddi-1-bw’ 50
40
30 Bandwidth (Mbits/sec)
20
10
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose TCP IP FDDI Bandwidth p er Pair of Pro cesses Master Slave
1e+06
’datahose-tcp-fddi-4-time’ ’datahose-tcp-fddi-3-time’ ’datahose-tcp-fddi-2-time’ ’datahose-tcp-fddi-1-time’
100000
10000 Averageg Time (micro-sec)
1000
100 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose TCP IP FDDI Time p er Iteration
DATAHOSE
100
’datahose-mpi-fddi-4-totalbw’ ’datahose-mpi-fddi-3-totalbw’ ’datahose-mpi-fddi-2-totalbw’ 80 ’datahose-mpi-fddi-1-totalbw’
60
40 Bandwidth (Mbits/sec)
20
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose MPI FDDI Total Bandwidth
70
60 ’datahose-mpi-fddi-4-bw’ ’datahose-mpi-fddi-3-bw’ ’datahose-mpi-fddi-2-bw’ ’datahose-mpi-fddi-1-bw’ 50
40
30 Bandwidth (Mbits/sec)
20
10
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose MPI FDDI Bandwidth p er Pair of Pro cesses Master Slave
1e+06
’datahose-mpi-fddi-4-time’ ’datahose-mpi-fddi-3-time’ ’datahose-mpi-fddi-2-time’ ’datahose-mpi-fddi-1-time’
100000
Averageg Time (micro-sec) 10000
1000 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose MPI FDDI Time p er Iteration
DATAHOSE
100
’datahose-tcp-fast-3-totalbw’ ’datahose-tcp-fast-2-totalbw’ ’datahose-tcp-fast-1-totalbw’ 80
60
40 Bandwidth (Mbits/sec)
20
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose TCP IP Fast Ethernet Total Bandwidth
70
60 ’datahose-tcp-fast-3-bw’ ’datahose-tcp-fast-2-bw’ ’datahose-tcp-fast-1-bw’
50
40
30 Bandwidth (Mbits/sec)
20
10
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose TCP IP Fast Ethernet Bandwidth p er Pair of Pro cesses Master Slave
1e+06 ’datahose-tcp-fast-3-time’ ’datahose-tcp-fast-2-time’ ’datahose-tcp-fast-1-time’
100000
10000
1000 Averageg Time (micro-sec)
100
10 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose TCP IP Fast Ethernet Time p er Iteration
DATAHOSE
100
’datahose-mpi-fast-3-totalbw’ ’datahose-mpi-fast-2-totalbw’ ’datahose-mpi-fast-1-totalbw’ 80
60
40 Bandwidth (Mbits/sec)
20
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose MPI Fast Ethernet Total Bandwidth
70
60 ’datahose-mpi-fast-3-bw’ ’datahose-mpi-fast-2-bw’ ’datahose-mpi-fast-1-bw’
50
40
30 Bandwidth (Mbits/sec)
20
10
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose MPI Fast Ethernet Bandwidth p er Pair of Pro cesses Master Slave
1e+06
’datahose-mpi-fast-3-time’ ’datahose-mpi-fast-2-time’ ’datahose-mpi-fast-1-time’
100000
10000 Averageg Time (micro-sec)
1000
100 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose MPI Fast Ethernet Time p er Iteration
DATAHOSE
10
’datahose-tcp-ethernet-4-totalbw’ ’datahose-tcp-ethernet-3-totalbw’ ’datahose-tcp-ethernet-2-totalbw’ 8 ’datahose-tcp-ethernet-1-totalbw’
6
4 Bandwidth (Mbits/sec)
2
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose TCP IP Ethernet Total Bandwidth
10
’datahose-tcp-ethernet-4-bw’ ’datahose-tcp-ethernet-3-bw’ ’datahose-tcp-ethernet-2-bw’ 8 ’datahose-tcp-ethernet-1-bw’
6
4 Bandwidth (Mbits/sec)
2
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose TCP IP Ethernet Bandwidth p er Pair of Pro cesses Master Slave
1e+07
’datahose-tcp-ethernet-4-time’ ’datahose-tcp-ethernet-3-time’ ’datahose-tcp-ethernet-2-time’ 1e+06 ’datahose-tcp-ethernet-1-time’
100000
10000 Averageg Time (micro-sec)
1000
100 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose TCP IP Ethernet Time p er Iteration
DATAHOSE
10
’datahose-mpi-ethernet-4-totalbw’ ’datahose-mpi-ethernet-3-totalbw’ ’datahose-mpi-ethernet-2-totalbw’ 8 ’datahose-mpi-ethernet-1-totalbw’
6
4 Bandwidth (Mbits/sec)
2
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose MPI Ethernet Total Bandwidth
10
’datahose-mpi-ethernet-4-bw’ ’datahose-mpi-ethernet-3-bw’ ’datahose-mpi-ethernet-2-bw’ 8 ’datahose-mpi-ethernet-1-bw’
6
4 Bandwidth (Mbits/sec)
2
0 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose MPI Ethernet Bandwidth p er Pair of Pro cesses Master Slave
1e+07
’datahose-mpi-ethernet-4-time’ ’datahose-mpi-ethernet-3-time’ ’datahose-mpi-ethernet-2-time’ ’datahose-mpi-ethernet-1-time’ 1e+06
100000 Averageg Time (micro-sec)
10000
1000 1 10 100 1000 10000 100000 1e+06 1e+07
Message Size (bytes)
Figure Datahose MPI Ethernet Time p er Iteration
ANALYSIS OF RESULTS
Analysis of Results
Characterization and analysis of end to end network p erformance is an involved task There
are many factors that contribute to the nal results machine architecture network proto col
software network card in the computer and its device driver the external hub concentrator
for rings like FDDI and so forth
Many features that we observed in our results were predictable and intuitively app ealing
They were
The time it to ok to pro cess messages b elow a certain threshold bytes for all the
six combinations was constant probably b ecause TCP NODELAYprevents message
coalescing So all small messages are sent in a packet of their own and the small size
makes the network latency a big factor as opp osed to bandwidth
FDDI and Fast Ethernet are an interesting comparison Because of the di erent
transmission mechanisms they p erformed di erently on the same b enchmarks We
analyse their TCP IP p erformance here
Low network load
Fast Ethernet p erformed b etter roundtrip Mbits sec Figure datahose
Mbits sec Figure than FDDI roundtrip and datahose Mbits sec
Figure Figure when the network was lightly loaded pair of pro cesses
One p ossible reason is that Ethernet Fast or otherwise based on CSMA CD
do es not wait for p ermission for each message Thus a transmitting pro cess
after sensing for absence of a carrier wrote to the network hoping that a collision
would not o ccur For light load conditions collisions do not o ccur or were rare
Thus the high throughput FDDI is based on token ring eachmachine in FDDI
has to wait until it can grab the token that is oating around on the ring Only
when a station has the token do es it start transmission This control mechanism
slows down the transmission pro cess even in a no load situation
High network load
FDDI p erformed much b etter roundtrip and datahose Mbits sec Figure
Figure in a high load situation pairs than Fast Ethernet roundtrip
Mbits sec Figure datahose Mbits sec Figure The reason again is
the network hardware mechanism As network load increased so did the number
of packet collisions in Ethernet Thus even though the individual pro cesses could
put out packets faster onto the network retries from collided packets degraded
p erformance drastically So the bandwidth p er pair of pro cesses dropp ed sharply
for Fast Ethernet as load went up FDDI is a collision free network and hence
this phenomenon did not a ect its p erformance We see almost aggregate
bandwidth utilization for FDDI
Although MPI was slower then TCP IP itwas more consistent TCP IP had very
abrupt spikes and dips in p erformance at various places The corresp onding MPI tests
were generally much smo other
REFERENCES
MPI was always except for a few data p oints in Fast Ethernet b oth roundtrip and
datahose slower than TCP IP The di erence gives us the software overhead involved
with using MPI MPI reduced p erformance in the two faster networks FDDI and
Fast Ethernet by ab out Mbits sec at the maximum load levels However MPI
and TCP IP were pretty close on Ethernet giving the impression that the software
p enaltywas masked by the muchslower network
Of the three networks welooked at FDDI seemed to b e the most predictable in terms
of p erformance Abrupt discontinuities whichwere present in the two Ethernets Fast
and Normal did not plague FDDI
While the results were generally consistent with exp ectations the presence of rep eatable
p erformance spikes and dips in TCP IP made the analysis interesting Weveri ed most
of the TCP IP discontinuities using the ttcp package They seem to b e consistenteven
across various machine typ es for Ethernet At presentwe do not have an insightful
explanation for these anomalies If you have some clue to the solution wewould appreciate
hearing from you
Finally wewould like to emphasize that our results are for sp eci c implementations of the
hardware and software standards Other implementations mayhave di erent p erformance
References
Christos Papadop oulos and Gurudatta M Parulkar Exp erimental Evaluation of
SUNOS IPC and TCP IP Proto col Implementation IEEE ACM Transactions on
Networking Vol No April Pages
Ralph Butler and Ewing Lusk User s Guide to the p Parallel Programming System
Version Argonne National Lab oratory ANL August
D W Walker The design of a standard message passing interface for distributed
memory concurrent computers ParComp Vol No Pages
Andrew S Tanenbaum Computer Networks nd Edition Prentice Hall