This is
pretty dry stuff...click here
to cut to the chase...
WITH TRACEROUTE |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||
|
1 205.139.105.254
(205.139.105.254) |
4.405 ms |
6.266 ms |
2.629 ms |
|
2
border1-serial3-5.Seattle.mci.net (204.70.52.65) |
21.628 ms |
21.056 ms |
31.816 ms |
|
3
core1-fddi-0.Seattle.mci.net (204.70.2.145) |
16.701 ms |
18.725 ms |
21.598 ms |
|
4 * *
sl-stk-1-H9/0-T3.sprintlink.net (206.157.77.66) |
36.615 ms |
|
|
|
|
|
|
|
|
5
sl-stk-1-H9/0-T3.sprintlink.net (206.157.77.66) |
34.934 ms |
40.946 ms |
74.192 ms |
|
6
sl-che-1-H2/0-T3.sprintlink.net (144.228.10.90) |
55.085 ms |
55.506 ms |
55.157 ms |
|
7
sl-che-3-F0/0.sprintlink.net (144.224.10.3) |
55.863 ms |
56.454 ms |
58.791 ms |
|
8
sl-cica-2-H0-T3.sprintlink.net (144.224.13.6) |
68.133 ms |
58.324 ms |
80.837 ms |
|
9 gw2.boulder.co.coop.net
(199.45.132.35) |
66.01 ms |
69.084 ms |
61.033 ms |
|
10 199.45.130.14
(199.45.130.14) |
67.436 ms |
70.048 ms |
69.716 ms |
|
11 199.45.143.15
(199.45.143.15) |
68.246 ms |
70.162 ms |
91.696 ms |
|
12 t1.boardwatch.com
(204.144.169.2) |
77.907 ms |
75.371 ms |
77.696 ms |
|
13 www.boardwatch.com
(204.144.169.6) |
81.938 ms |
84.054 ms |
74.27 ms |
This results in a list of routers between your site and the entered domain
with the first router encountered at the top of the list, and the destination
domain machine at the bottom. At each site, both the name and IP number are
listed, along with typically three timing values that indicate round-trip
packet propagation times.
In Windows95, traceroute is renamed TRACERT
for reasons only apparent to Microsoftians. It is also only available from a
DOS window. Drop to a DOS window and enter TRACERT
BOARDWATCH.COM to get essentially the same results.
In the example above, the first machine
encountered was 205.139.105.254 in Grants Pass, Oregon. The machine
failed a reverse domain name lookup and so only the IP number is listed. This
is actually a traceroute from a web server in Grants Pass TO www.boardwatch.com.
The second machine is a border router of MCI's
network in Seattle where we join MCI and machine three is one of the MCI core
backbone routers in Seattle. In machine four, we connect to Sprint in
Stockton California. It's worth noting that we did not go through one of the
NAP's apparently such as the PacBell SF NAP or MAE-WEST and so we've most
likely discovered a private peering exchange between Sprint and MCI in
Stockton California. The concept that all connections between private
backbones exist at one of the four NSF funded Network Access Points or NAPS
is simply erroneous at this point with hundreds of such private NAPS between
two carriers cropping up all over the country at any target of geographic
proximity/opportunity.
At lines 6 we pass through what looks like a
Stockton router where the interface is designated Cheyenne. At line 7 we
enter Sprint's hub facility in Cheyenne and line 8 is the exit point on the
way to Colorado Internet Cooperative Association, also known as the Boulder
COOP in line 9. Line 10 appears to be a Boulder Coop router. In line 11 we go
through an unidentified machines, actually at eSoft, Inc. in Aurora Colorado.
Line 12 is a T-1 connection to our gateway router here at Boardwatch and line
13 is the 204.144.169.6 machine we use as a web server on our LAN.
Note the time indications for each line - three
time values in milliseconds. These are roundtrip times for each leg, not
sums, differences, or otherwise related to each other. From Grants Pass to
Littleton, across two major backbones, and BACK looks like about 82 ms, 84
ms, and 74 ms for the three packets involved. Not bad actually. The point is,
times are complete roundtrip times for packets to THAT router on THAT line
and are unrelated to anything on previous or subsequent lines. Traceroute is
a relatively low priority service on most machines, and as such isn't really
a very good indicator of roundtrip times. The PING program actually works
slightly better for that purpose.
That said, you CAN derive some indication of
how busy the Internet is from examining these traceroute roundtrip times. If
the three times on any one particular line are fairly close together, this
indicates some consistency between the packets. If they are more variable, it
usually indicates some dynamic change in traffic levels between the three
packets. Similarly, if you get a relatively good time roundtrip to one router,
while the very next machine gives a very poor series of times, and then the
times are pretty good again on the NEXT router, you are seeing some traffic
surges on the network. Smaller pipes tend to be somewhat more sensitive to
such traffic surges, while larger ones tend to be a bit more forgiving.
Traceroute wasn't really designed as any
of the normal control message elements of any protocol. It is actually a
small program that relies on some predictable response activity of routers
and servers on the Internet.
Routers receive Internet Protocol packets, and
in fact packets of other protocols of various sizes and flavors, and in most
cases simply pass them on. A router will have several interfaces usually. It
receives a packet in one interface, opens the packet to find the DESTINATION
ADDRESS, and then does a lookup in a routing table to determine which of its
other interfaces (to other routers) is the best place to "route"
the packet. In basic operation, it's actually not a terribly smart device. To
a router with three interfaces, for example, the entire global Internet falls
into three categories - Interface 1, Interface 2, and Interface 3. The
Interfaces can be serial ports, ethernet cards, V.32 ports, or a specialized
smoke signal rapidly-flappable-wet-blanket (RFWB) interface device. It
doesn't matter. Open the packet, decide which port, and mail the puppy. In
almost all cases, a router is the ultimate finger pointer - "This packet
is somebody else's problem, and I'll pass it out port X for them to deal
with."
In the early days of TCP/IP networks, it was
discovered pretty quickly that if you sent out a packet and there was some
configuration anomaly in a single router in the network, you could create a
routing loop where packets could enter, but they would never get back out.
The packet would simply move from one router to another in a circle. These
packets would accumulate in the network quite quickly and seriously slow
things down for those packets that were actually going somewhere.
The solution was the addition of an 8-bit Time-To-Live
(TTL) field in the Internet Protocol packet header. The sending machine could
set this to any value between 0 and 255. Each router that handled the packet
decremented this value in the packet header by one when it passed on the
packet. If it received a packet that had a TTL of 0 or 1, instead of passing
on the packet, it killed it. In this way, after a set number of
"hops" through routers, in no case larger than 255, a packet died and
was removed from the network. In practice, TTL is the maximum number of hops
a packet can transit before death. The 255 hop total is also the maximum
"span" of the Internet today. If you are more than 255 hops from
anywhere, you can't get there from here using the existing Internet Protocol.
When a router kills a packet, it also sends out
an Internet Control Message Protocol (ICMP) error message to the
packet originator address indicating TIME EXCEEDED IN TRANSIT. This message
contains the IP address of the router sending the error message, as well as
the address of the machine that sent the original IP packet and any of the
original packet contents.
Traceroute takes advantage of these two
predictable reactions. When you enter TRACERT WWW.BOARDWATCH.COM,
traceroute first does a DNS lookup of WWW.BOARDWATCH.COM to get the 204.144.169.6
IP address. It then uses the somewhat more efficient User Datagram
Protocol (UDP) to create three small, typically 40 byte, packets
containing the originating address, the destination address, and a time stamp
of when the packets were created. It sends out three of these packets
with the TTL value set to 1. These packets arrive at the first router
in the path to Boardwatch and that router immediately
decrements the TTL, notes that it is now zero, and issues an ICMP
TIME EXCEEDED IN TRANSIT error message back to the sending machine -
including the original time stamp and the IP number of the router sending the
error message. The original machine that sent out the three packets receives
this ICM error message and notes the time of receipt, as well as the IP
number of the router that sent it. It then examines the time stamp
information returned inside the ICMP packet. It can then calculate the
difference between the time stamp information sent, and the time the ICMP
packet was received. This is how it calculates the round trip transit time in
milliseconds.
Traceroute then extracts the IP number of the
machine issuing the error message and does a reverse DNS lookup to retrieve
the name of the machine. It prints a sequence number, in this case 1,
followed by the name of the machine, the IP number of the machine, and the
round trip time for each of the three test packets. The "first"
machine in route is now diagrammed.
Traceroute then increments the TTL to 2
and sends out three more packets with new time stamps in them - again
addressed to 204.144.169.6 - still the ultimate destination. The first
router in the path receives the packet, decrements TTL from 2 to 1
this time, and since TTL has NOT expired, passes the packets on to the NEXT
router in the path. THAT router now decrements TTL from 1 to 0, and issues
the ICMP error messages back to the originating machine in the same way.
Tracert continues to increment TTL and send out
sorties of three packets each time. Each time TTL is incremented, the packets
make it to another router down the path before expiring and causing the
tattletale ICMP error message that allows traceroute to identify the router.
The traceroute machine will wait a fixed amount of time for a reply. If this
expires, it will print a series of ASTERISKS (*) indicating that there is a
machine there in the path, but it can't get it to respond with an ICMP error
message within the default time. It then increments TTL again and continues.
When TTL reaches a value where the UDP datagram
actually reaches the ultimate intended host, in this case www.boardwatch.com,
that would normally be the end of the packet. But the host will be surprised
to learn that the destination port number in the packet header is a
ridiculously implausible port number - usually 33,434 but in any case
something not ever recognized as a port. Web sites usually monitor port
number 80 for example while an e-mail server will monitor port 25. Port
33,434 is not only not one of the normal ports, but is not likely to ever be.
So the destination machine issues an ICMP error message as well - in this
case the PORT UNREACHABLE message. Traceroute reads this as a kind of
"mission accomplished" termination.
Traceroute can be very useful to list the
default path from your site to another site and that is its most common use.
However, we often use traceroute to examine backbone topologies and this at
times requires some bending of the rules. We often want to run packets around
in circles so to speak, and watch how they get there.
Conceptually, there are two options to UDP
datagrams that traceroute can take advantage of. These are the Loose Soure
Route Record (LSRR) and Strict Source Route Record (SSRR) usually referred to
as Loose Source Routing and Strict Source Routing. Rumor has it that Van
Jacobson had both in the original 1988 TRACEROUTE and disabled them at the
request of some administrators who didn't want it known how badly broken
their gateways really were. We can't verify that rumor, but find it
believable.
We haven't seen much in the way of traceroute
variants that do the Strict Source Routing, which in theory would allow you
to specify the precise path from one point to the next. But there are several
that support Loose Source Routing, which allows you to indicate intermediate
destinations.
Basically, this requires a bit more detailed
description of UDP packets. In the case of LSRR, you can enter up to nine
intermediate routers to define a path through the network. In most UNIX
traceroutes, this is the -g option switch where in the Windows95
TRACERT it is the -J option.
To see why this might be valuable, let's take a
look at a regular traceroute to Mustang Software Company (diagrammed above)
in Bakersfield California.
|
||||
|
1 |
2 ms |
2 ms |
2 ms |
t1.boardwatch.com
[204.144.169.2] |
|
2 |
9 ms |
7 ms |
7 ms |
199.45.143.15 |
|
3 |
11 ms |
10 ms |
12 ms |
t1.esoft.com [199.45.143.14] |
|
4 |
18 ms |
16 ms |
16 ms |
199.45.130.13 |
|
5 |
56 ms |
18 ms |
18 ms |
gw21-backbone-1.boulder.co.coop.net
[199.45.132.33] |
|
6 |
23 ms |
20 ms |
20 ms |
sl-che-3-H12/0-T3.sprintlink.net
[144.224.13.5] |
|
7 |
24 ms |
32 ms |
22 ms |
sl-che-1-F0/0.sprintlink.net
[144.224.10.1] |
|
8 |
44 ms |
49 ms |
45 ms |
sl-stk-1-H3/0-T3.sprintlink.net
[144.228.10.89] |
|
9 |
76 ms |
48 ms |
43 ms |
sl-stk-2-F/T.sprintlink.net
[198.67.6.2] |
|
10 |
223 ms |
224 ms |
307 ms |
sl-ana-2-H4/0-T3.sprintlink.net
[144.228.10.26] |
|
11 |
53 ms |
54 ms |
82 ms |
sl-ana-4-F0/0.sprintlink.net
[144.228.70.4] |
|
12 |
62 ms |
62 ms |
66 ms |
sl-mustang-1-s0-T1.sprintlink.net
[144.228.74.46] |
|
13 |
85 ms |
73 ms |
69 ms |
mustang.com [204.250.9.1] |
|
||||
This is a typical traceroute. Mustang is in California and like Boardwatch,
is basically connected to the Sprint backbone. As you can see, we start at Boardwatch
in line one, and then sequence through an in and out interface at eSoft, Inc.
in Aurora. We then go to the Boulder Coop, up through two Sprint backbone
routers, in Cheyenne Wyoming, and from their to Stockton California on the
Sprint backbone. From Stockton, we drop down to Anaheim California, and from
their to Mustang.
Note an interesting anomaly: We went through
DIFFERENT routers going OUT through Cheyenne and Stockton than we did coming
IN from Grants Pass. Backbone systems can be considerably more complex than
highway systems actually with multiple "lanes" for redundancy,
capacity, etc. But like a highway system, you often take a different exit
going north than you do going south . This also explains why roundtrip times
are not terribly valuable, the route the ICMP error message takes BACK to you
is unlikely to be the route the original sortie of three packets took OUT to
the machine in the path. The point is, traceroute diagrams paths in only one
direction. Reversing the direction often diagrams an entirely different path.
Well and good enough. But
we're pretty familiar with the Cheyenne/Stockton leg. What we would like to
do is find out what is going on going EAST to Kansas City from Cheyenne.
We're pretty sure that Kansas City is connected to Fort Worth, but Sprint
might have built a new connection directly from Kansas City to Anaheim. From
previous traceroute explorations, we know there is a Sprint router in Kansas
City - 144.228.10.81
|
||||
|
1 |
2 ms |
2 ms |
2 ms |
t1.boardwatch.com
[204.144.169.2] |
|
2 |
9 ms |
9 ms |
8 ms |
199.45.143.15 |
|
3 |
12 ms |
12 ms |
11 ms |
t1.esoft.com [199.45.143.14] |
|
4 |
18 ms |
18 ms |
19 ms |
199.45.130.13 |
|
5 |
22 ms |
40 ms |
20 ms |
gw21-backbone-1.boulder.co.coop.net
[199.45.132.33] |
|
6 |
60 ms |
214 ms |
233 ms |
sl-che-3-H12/0-T3.sprintlink.net
[144.224.13.5] |
|
7 |
253 ms |
219 ms |
208 ms |
sl-che-2-F0/0.sprintlink.net
[144.224.10.2] |
|
8 |
38 ms |
38 ms |
38 ms |
sl-kc-1-H2/0-T3.sprintlink.net
[144.228.10.81] |
|
9 |
42 ms |
40 ms |
39 ms |
sl-kc-2-F0/0.sprintlink.net
[144.224.20.2] |
|
10 |
116 ms |
252 ms |
232 ms |
sl-fw-5-H3/0-T3.sprintlink.net
[144.228.10.78] |
|
11 |
190 ms |
211 ms |
204 ms |
sl-fw-6-F/T.sprintlink.net
[208.12.128.6] |
|
12 |
249 ms |
257 ms |
117 ms |
sl-ana-1-H2/0-T3.sprintlink.net
[144.228.10.30] |
|
13 |
261 ms |
227 ms |
249 ms |
sl-mustang-1-s0-T1.sprintlink.net
[144.228.74.46] |
|
14 |
507 ms |
466 ms |
480 ms |
ns.mustang.com [204.250.9.1] |
|
||||
Examine this command line. We are listing two
IP numbers with the -J option. The rightmost number is actually our
destination, 204.250.9.1, the address of the MUSTANG.COM
machine. In this case, we have no interest whatsoever in this machine. We're
actually trying to verify the Sprint backbone through Cheyenne to Kansas City
and Fort Worth to Anaheim. A normal traceroute to MUSTANG.COM would
actually go through Cheyenne, to San Francisco, and down to Anaheim. But in
this case, we want to force a path east and out the south end of the
backbone. Our interest is in the backbone, and the Mustang site is a handy
endpoint of known connection that we can use.
The first IP number, 144.228.10.81 is a
Sprint router in Kansas City. The second is the MUSTANG.COM
destination - picked only because we know it is attached in Anaheim.
In the UDP datagram, the first address is
listed as a DESTINATION address, but a pointer in the datagram points to the MUSTANG.COM
address as the DESTINATION address. When the router in Kansas City is finally
reached, it notes the discrepancy between the pointer and the destination
address. It discards its own address as destination, shifts all addresses to
the left, and sends the packets on with the MUSTANG.COM address now
appearing to be the destination.
We could put up to nine intermediate IP
addresses in the command line. That is all the room available in the header.
As each address is reached, it is discarded and the remaining addresses
shifted left one position. In this way, we can "steer" a bit
through the Internet. But in practice, you have to have some knowledge of the
possible Intermediate addresses.
The advantage of loose source routing is that
you don't have to know ALL the routers in line as in Strict Source Routing.
But the routes you enter DO have to make some sort of sense, and the ultimate
destination has to make some sort of sense. But it does allow you to
"steer" a bit in exploring paths through the Internet.
As we can see from the results, we can go from
Boulder, to Cheyenne, and out to Kansas City. Kansas City is connected to
Sprint routers in Fort Worth, and Fort Worth is connected directly to
Anaheim. There is no link directly between Kansas City and Anaheim. We
already knew that Mustang was connected to Sprint at Anaheim. But we've just
successfully mapped the route from Cheyenne to Kansas City to Fort Worth to
Anaheim. We can now use the router numbers we get from those cities, to go
exploring further if we like.
Give me a lever and a place to stand and I can
move the world. Actually, whoever said this was a bit off. Over and over, and
in multiple fields of endeavor, we've found that you generally need THREE
places to stand in this world to do anything.
On the Internet, there are many backbones, and
your best source of reference is the one you are connected to. But it can be
of a powerful advantage to be able to see your site, and others, from other
vantage points on the Internet. As we noted, the path in one direction is
rarely the same path in the reverse direction. It can be of powerful
advantage to run traceroutes from OTHER locations back to US. In addition,
this can verify our "visibility" from other backbones and even
other continents.
Ray Davis of Carpe Net in Hofheim Germany (ray@carpe.net) has actually presented me
personally with a bit of a rose in this respect and the entire Internet it
would appear as well. Ray has done a small, fairly simple thing really quite
well. He connected traceroute and the World Wide Web with a very simple shell
script.. Anyone can run this shell script on their web and provide the world
with an online traceroute function accessible from a browser.
Davis is actually from Colorado, but moved to
Germany to serve as Technical Director at carpeNet Information Technologies
GmbH (carpe diem - seize the day, carpe net - seize the net).. CarpeNet is
connected to the central hub of one of the fastest and most extensive
European networks (Nacamar/www.nacamar.net). And soon they'll be one of the
highest bandwidth ISPs in Europe - with a 10 Mbps ethernet link into the new
MAE-Frankfurt building - destined to be a major European NAP. They hope to
develop some business as a European mirror site for popular U.S. web sites
and they seem to be making some interesting moves in getting there. http://www.carpe.net/index.html
In any event, Ray wrote a little bit of an HTML
script and a Common Gateway Interface (CGI) program script to allow anyone to
visit his web site and run traceroute. In fact, you can enter ANY
destination you want on the HTML form, and carpeNet's server will dutifully
traceroute from their location to the entered destination and print out the
results on your screen.
According to Davis: "Before we founded
carpeNet, I spent over a year researching networks here in Germany and as
many possibilities of connecting to the Internet as I could find. I often
found myself wanting to trace the route not only from where I am but from
other places back to me or from other places to other places. This way I
could figure out how various nets were utilized, who was peering with who,
how logical their routing was and something about their performance.
"Some months later, after carpeNet was
founded, I wrote a simple sh cgi script and put an ISINDEX web page in
front of it. Then I send a message to the inet-access mailing list asking if
anybody else thought it was useful to be able to traceroute from
elsewhere via a web page. I also included my traceroute URL and told everyone
to pick up the cgi source if they wanted to set up a traceroute page too.
"Quite a few people thought it was a great
idea, and some of them also put up a traceroute page and announced it. A
couple days later David Dennis (http://www.amazing.com)
put up some of these URLs on a page he called Club Traceroute."
Davis work has had several interesting effects.
First, you can check to see if your web site is really reachable from Hofheim
Germany. This will check to see that your Domain Name Service entry has
migrated through the system to Germany, and also that you are actually
reachable over the network. Finally, you get to see the connections between
European backbones to the U.S. and across the U.S. backbones. By selecting
different end points, you can again draw some very interesting routes.
So interesting that others wanted to setup
webserver traceroutes on THEIR web sites. And the more of these that go up,
the more interesting and useful it becomes. The accompanying table lists 88
of our favorites located in an afternoon. If you don't feel like hand
entering them, we've also put them up on a page on our web site at http://www.boardwatch.com/isp/trace.htm.
But there appear to be hundreds available in Japan, North America, Germany,
France, Belgium, Russia, Israel, Mexico, Czech Republic, Sweden, Australia,
and more. Just in North America, we can now trace different sites both from
and two on MCI, AGIS, UUNET, IBM/Advantis, Sprint and Sprint's overseas
cooperative, Global One, which is apparently becoming ever so popular.
This growing fleet of web traceroute servers
allows us to pick spots on different backbones and even different continents
to traceroute FROM.
Davis has actually moved beyond CGI scripts and
actually rewritten a C language traceroute to output HTML and BE the
CGI - somewhat more efficient on the server and perhaps more secure as well..
The C language source code is available at http://www.carpe.net/src/index.html.
ISP's particularly are putting up these simple
little pages all over the world. Soon, it would appear there will be enough
of them that we will have the illusion of being able to specify BOTH ends of
a traceroute, the source and destination. At that point, it will be
fairly easy to map the Internet to any degree of specificity desired, using
the common traceroute program.