Linux Mint - Free and powerful

Monday, 8 September 2014

bash - Script to extract highest latency from traceroute - Server Fault

bash - Script to extract highest latency from traceroute - Server Fault:



'via Blog this'



Stephan asks an interesting question. He is looking for latency in hops between networks to show why the connection is slow. Not all international connections are equal so how do you prove that a link in the connection is the cause of the delay.



I'm looking for a script that can extract the line with the highest latency hop from a traceroute. Ideally it would look at the max or avg of the 3 values by line. How can I so that?
This is what I tried so far:
     traceroute www.google.com | awk '{printf "%s\t%s\n", $2, $3+$4+$5; }' | sort -rgk2 | head -n1
     traceroute -w10 www.google.com | awk '{printf "%s\t%s\n", $2, ($3+$4+$5)/3; }' | sort -rgk2 | head -n1
It seemed a step in the right direction, except some of the values coming back from a traceroute are *, so both the sum and the average provide a wrong value.
Update Got one step further:
     traceroute www.cnn.com | awk '{count = 0;sum = 0;for (i=3; i<6; i++){ if ($i != "*") {sum += $i;count++;}}; printf "%s\t%s\t%s\t%s\n", $2, count, sum, sum/count }' | sort -rgk2
now need to intercept if I dont' have a column 4,5. Sometimes traceroute only provides 3 stars like this:
     17   207.88.13.153  235.649ms  234.864ms  239.316ms 
     18   *  *  *
share|edit
You will have to
  • Kick off a traceroute
  • Collect each line of output ( a pipe would likely work well here)
  • Use a tool like awk to
    • Analyze the line and extract the information you want
    • Compare the values you just got with previous values and store the current line if appropriate
    • At the end of the input print the stored value
share|edit
   
Yep - that's general approach. Got a code sample? –  stwissel 4 hours ago
4 
@stwissel got some $$$ to pass to me ? We generally don't write scripts and do people's jobs for them but we're happy to point you in the right direction or even debug stuff if you show us your workings. –  Iain 4 hours ago
   
Pecked the original question on glass, now added the attempts I made so far. For a "clean" traceroute I have a solution, but need a step when * comes back –  stwissel 3 hours ago
Stephan, you could try and use pchar a derivative of pathchar. It should be in the Ubuntu repository.
I takes a while to run though so you need some patience. It will show you throughput and that will be much better than latency for determining the bottleneck.
Here is an example:
rayd@raydHPEliteBook8440p ~ sudo pchar anddroiddevs.com
pchar to anddroiddevs.com (31.221.38.104) using UDP/IPv4
Using raw socket input
Packet size increments from 32 to 1500 by 32
46 test(s) per repetition
32 repetition(s) per hop
 0: 192.168.0.20 (raydHPEliteBook8440p.local)
    Partial loss:      0 / 1472 (0%)
    Partial char:      rtt = 6.553065 ms, (b = 0.000913 ms/B), r2 = 0.241811
                       stddev rtt = 0.196989, stddev b = 0.000244
    Partial queueing:  avg = 0.012648 ms (13848 bytes)
    Hop char:          rtt = 6.553065 ms, bw = 8759.575088 Kbps
    Hop queueing:      avg = 0.012648 ms (13848 bytes)
 1: 80.5.69.1 (cpc2-glfd6-2-0-gw.6-2.cable.virginm.net)
share|edit|delete|flag
Try:
$ traceroute 8.8.8.8 | awk ' BEGIN { FPAT="[0-9]+\\.[0-9]{3} ms" }
                           /[\\* ]{3}/ {next}
                           NR>1  {
                                   for (i=1;i<4;i++) {gsub("*","5000.00 ms",$i)}
                                   av = (gensub(" ms","",1,$1) + gensub(" ms","",1,$2) + gensub(" ms","",1,$3))/3
                                   if (av > worst) {
                                     ln = $0
                                     worst = av
                                   }
                                 }
                           ND { print "Highest:", ln, " Average:", worst, "ms"}'
which gives:
Highest:  6  72.14.242.166 (72.14.242.166)  7.383 ms 72.14.232.134 (72.14.232.134)  7.865 ms  7.768 ms  Average: 7.672  ms
If there are three asterix (asteri?) * * * the script assumes that the hop isn't responding with the IGMP response and ignores it completely. If there are one or two *in a line, it gives them the value of 5.0 seconds.
share|edit
   
Hmm - you divide through 3 in all cases so if I have 110.666ms * 159.334ms I would get 90ms which wouldn't be the accurate average of 135ms –  stwissel 2 hours ago
1 
I think the problem is the lack of a definition for *. In your comment, you chose to ignore it and divide by two. It could also be defined as 5.0 seconds as this is the time at which traceroute gives up and shows the asterix. It could also be defined as infinity as no reponse has been seen. In the latter case you could end up with more than one line as the worst case response. –  garethTheRed 2 hours ago
   
True. * isn't defined. It appears when the request times out. Typically that happens when a server is configured to ignore traceroute. I checked with 30 sec timeouts and still got *. So I concluded ignoring is as good as assuming a value (where the timeout - default 3sec - would be appropriate) Awk turns it into zero anyway. But having 2 values and dividing by 3 doesn't yield a usable result. –  stwissel 37 mins ago 
   
I've edited it (previously) so that if there are three * then it assumes the server is ignoring the request. However, if there are one or two * and a response, then the server isn't ignoring the request, but is taking a very long time to respond (> 5 sec by default). – garethTheRed 28 mins ago
Use mtr --raw -c 1 google.com. It's wayy faster and easier to parse.
share|edit
 My view on this issue is to use something a bit more sophisticated than traceroute. Traceroute was replaced by the original developer, Van Jacobson, with pathchar but the package is no longer maintained. It has been replaced by pchar



Here is a test output from Pchar.



pchar to anddroiddevs.com (31.221.38.104) using UDP/IPv4

Using raw socket input

Packet size increments from 32 to 1500 by 32

46 test(s) per repetition

32 repetition(s) per hop

 0: 192.168.0.20 (raydHPEliteBook8440p.local)

    Partial loss:      2 / 1472 (0%)

    Partial char:      rtt = 6.454726 ms, (b = 0.000683 ms/B), r2 = 0.193537

                       stddev rtt = 0.169716, stddev b = 0.000210

    Partial queueing:  avg = 0.005676 ms (8315 bytes)

    Hop char:          rtt = 6.454726 ms, bw = 11720.936536 Kbps

    Hop queueing:      avg = 0.005676 ms (8315 bytes)

 1: 80.5.69.1 (cpc2-glfd6-2-0-gw.6-2.cable.virginm.net)

    Partial loss:      0 / 1472 (0%)

    Partial char:      rtt = 5.095600 ms, (b = 0.000475 ms/B), r2 = 0.665108

                       stddev rtt = 0.041035, stddev b = 0.000051

    Partial queueing:  avg = 0.003673 ms (8315 bytes)

    Hop char:          rtt = --.--- ms, bw = --.--- Kbps

    Hop queueing:      avg = -0.002003 ms (0 bytes)

 2: 80.4.31.233 (glfd-core-2b-ae6-708.network.virginmedia.net)

    Partial loss:      161 / 1472 (10%)

    Partial char:      rtt = 7.467523 ms, (b = 0.000811 ms/B), r2 = 0.452354

                       stddev rtt = 0.131138, stddev b = 0.000143

    Partial queueing:  avg = 0.004913 ms (12006 bytes)

    Hop char:          rtt = 2.371923 ms, bw = 23809.456358 Kbps

    Hop queueing:      avg = 0.001240 ms (3691 bytes)

 3: 213.105.159.245 (popl-bb-1b-ae3-0.network.virginmedia.net)

    Partial loss:      0 / 1472 (0%)

    Partial char:      rtt = 7.251529 ms, (b = 0.000749 ms/B), r2 = 0.263173

                       stddev rtt = 0.173828, stddev b = 0.000189

    Partial queueing:  avg = 0.008567 ms (12006 bytes)

    Hop char:          rtt = --.--- ms, bw = --.--- Kbps

    Hop queueing:      avg = 0.003653 ms (0 bytes)

 4: 62.254.42.89 (popl-bb-2a-ae2-0.network.virginmedia.net)

    Partial loss:      0 / 1472 (0%)

    Partial char:      rtt = 7.371611 ms, (b = 0.000565 ms/B), r2 = 0.138153

                       stddev rtt = 0.171840, stddev b = 0.000213

    Partial queueing:  avg = 0.003385 ms (12006 bytes)

    Hop char:          rtt = 0.120082 ms, bw = --.--- Kbps

    Hop queueing:      avg = -0.005182 ms (0 bytes)

 5: 62.254.42.94 (popl-bb-1c-ae0-0.network.virginmedia.net)

    Partial loss:      0 / 1472 (0%)

    Partial char:      rtt = 7.102278 ms, (b = 0.000997 ms/B), r2 = 0.224520

                       stddev rtt = 0.225750, stddev b = 0.000279

    Partial queueing:  avg = 0.003131 ms (12006 bytes)

    Hop char:          rtt = --.--- ms, bw = 18501.528779 Kbps

    Hop queueing:      avg = -0.000254 ms (0 bytes)

 6: 213.105.159.117 (tele-ic-5-ae0-0.network.virginmedia.net)

    Partial loss:      5 / 1472 (0%)

    Partial char:      rtt = 8.002180 ms, (b = 0.000656 ms/B), r2 = 0.354165

                       stddev rtt = 0.120721, stddev b = 0.000134

    Partial queueing:  avg = 0.001926 ms (12006 bytes)

    Hop char:          rtt = 0.899902 ms, bw = --.--- Kbps

    Hop queueing:      avg = -0.001205 ms (0 bytes)

 7: 195.66.236.186 (1-7-2.pr01.ixhs.uk.exponential-e.net)

    Partial loss:      259 / 1472 (17%)

    Partial char:      rtt = 9.015488 ms, (b = 0.000428 ms/B), r2 = 0.528561

                       stddev rtt = 0.049230, stddev b = 0.000061

    Partial queueing:  avg = 0.002829 ms (12006 bytes)

    Hop char:          rtt = 1.013308 ms, bw = --.--- Kbps

    Hop queueing:      avg = 0.000903 ms (0 bytes)

 8: 31.221.38.97 (ser018524.thca.uk.exponential-e.net)

    Partial loss:      287 / 1472 (19%)

    Partial char:      rtt = 11.514627 ms, (b = 0.001036 ms/B), r2 = 0.475188

                       stddev rtt = 0.202132, stddev b = 0.000164

    Partial queueing:  avg = 0.002002 ms (12006 bytes)

    Hop char:          rtt = 2.499139 ms, bw = 13164.125611 Kbps

    Hop queueing:      avg = -0.000827 ms (0 bytes)

 9: 31.221.38.104 (alpha)

    Partial loss:      736 / 1472 (50%)

    Partial char:      rtt = 11.477751 ms, (b = 0.001552 ms/B), r2 = 0.837482

                       stddev rtt = 0.207185, stddev b = 0.000149

    Partial queueing:  avg = 0.002844 ms (13636 bytes)

    Hop char:          rtt = --.--- ms, bw = 15498.384615 Kbps

    Hop queueing:      avg = 0.000842 ms (1630 bytes)

10: 31.221.38.104 (alpha)

    Path length:       10 hops

    Path char:         rtt = 11.477751 ms r2 = 0.837482

    Path bottleneck:   11720.936536 Kbps

    Path pipe:         16816 bytes

    Path queueing:     average = 0.002844 ms (13636 bytes)

    Start time:        Mon Sep  8 09:59:12 2014

    End time:          Mon Sep  8 12:09:51 2014





---------------------



Provided by: pchar_1.5-1_i386 bug


NAME

pchar - Perform network measurements along an Internet path

SYNOPSIS

pchar  [ -cChnqSvV ] [ -a analysis ] [ -b burst ] [ -d debug ] [ -g gap
       ] [ -H hops ] [ -i interface ] [ -I increment ] [ -l origin ] [  -m mtu
       ]  [  -M mode  ]  [  -p protocol ] [ -P port ] [ -R reps ] [ -s hop ] [
       -t timeout ] [ -w file ]  -r file | host

DESCRIPTION

Pchar measures the characteristics of  the  network  path  between  two
       Internet   hosts,   on   either  IPv4  or  IPv6  networks.   It  is  an
       independently-written reimplementation of the pathchar  utility,  using
       similar  algorithms.   Both  programs  measure  network  throughput and
       round-trip time by sending varying-sized UDP packets into  the  network
       and  waiting  for  ICMP  messages  in  response.  Like traceroute, they
       modulate the IPv4 time-to-live (TTL) field or the IPv6 hop limit  field
       to get measurements at different distances along a path.

       In  its default mode, a run of pchar over a short path might produce an
       output that looks like this:
       pchar to dancer.ca.sandia.gov (146.246.246.1) using UDP/IPv4
       Packet size increments by 32 to 1500
       46 test(s) per repetition
       32 repetition(s) per hop
        0:
           Partial loss:      0 / 1472 (0%)
           Partial char:      rtt = 0.657235 ms, (b = 0.000358 ms/B), r2 = 0.989713
                              stddev rtt = 0.004140, stddev b = 0.000006
           Hop char:          rtt = 0.657235 ms, bw = 22333.268771 Kbps
           Partial queueing:  avg = 0.000150 ms (418 bytes)
        1: 146.246.243.254 (con243.ca.sandia.gov)
           Partial loss:      0 / 1472 (0%)
           Partial char:      rtt = 0.811278 ms, (b = 0.000454 ms/B), r2 = 0.995401
                              stddev rtt = 0.003499, stddev b = 0.000005
           Hop char:          rtt = 0.154043 ms, bw = 83454.764777 Kbps
           Partial queueing:  avg = 0.000153 ms (336 bytes)
        2: 146.246.250.251 (slcon1.ca.sandia.gov)
           Partial loss:      0 / 1472 (0%)
           Partial char:      rtt = 1.044412 ms, (b = 0.002161 ms/B), r2 = 0.999658
                              stddev rtt = 0.004533, stddev b = 0.000006
           Hop char:          rtt = 0.233133 ms, bw = 4686.320952 Kbps
           Partial queueing:  avg = 0.000100 ms (46 bytes)
        3: 146.246.246.1 (dancer.ca.sandia.gov)
           Path length:       3 hops
           Path char:         rtt = 1.044412 ms, r2 = 0.999658
           Path bottleneck:   4686.320952 Kbps
           Path pipe:         611 bytes
           Path queueing:     average = 0.000100 ms (46 bytes)

       The path here passes through three hops.  Each  hop  consists  of  four
       lines  of  output:  Partial loss documents the number and percentage of
       probe packets that were lost during  the  probes  for  that  hop.   The
       partial  char line shows the estimated round-trip time from the probing
       host through the current hop.  The hop char line shows estimates of the
       round-trip  time  and  bandwidth  for  the  current  hop.  Finally, the
       partial queueing shows an estimate of the average  queueing  along  the
       path, up to and including the current hop.

       Between  each  hop,  pchar prints the IP address and (if known) name of
       the host/router at the end of the link.

       After the last hop (usually the target host), pchar  prints  statistics
       on the entire path, including the path length and path pipe (the latter
       is an estimate of the delay-bandwidth product of the path).

       Pchar has another mode of operation,  called  trout  (short  for  “tiny
       traceroute”).   In  this  mode, packets of random sizes (one packet per
       hop diameter) are sent along the path to a destination.  No attempt  at
       estimating  link  properties  is  made; however, this mode is extremely
       fast.  It is intended for  use  as  a  part  of  a  larger  measurement
       infrastructure.  The output from this mode might look like:
       trout to bmah-freebsd-1.cisco.com (171.70.84.44) using ICMP/IPv4 (raw sockets)
       Packet size increments from 28 to 1500 by 32
        0: 171.70.84.42 (bmah-freebsd-0.cisco.com)
        1: 171.70.84.44 (bmah-freebsd-1.cisco.com) 352 -> 352 bytes: 0.318 ms

OPTIONS

-a analysis
              Set analysis type.  Current choices are lsq (the default), which
              uses a minimum filter followed by a least sum-of-squares fit  to
              estimate  link  bandwidths, kendall, which uses the same minimum
              filter  followed  by  a  linear  fit  based  on  Kendall's  test
              statistic,  lms, which does a minimum filter followed by a least
              median of squares fit, and lmsint, which is an implementation of
              the lms computations using only integer arithmetic.

       -b burst
              Set  the  size  of  packet  bursts.   A burst parameter > 1 will
              result in some number of ICMP_ECHOREPLY packets sent before  the
              probe  packet  to induce queueing.  These packets are useful for
              measuring   store-and-forward   switched   subnets,   but   make
              measurements of fast links behind bottlenecks inaccurate.

       -c     Ignore routing changes detected during running.  Normally, pchar
              will exit if it receives responses from more than one host for a
              given  hop,  assuming that this condition is caused by a routing
              transient.  However, certain  load-balancing  schemes  can  also
              cause  this  condition.  In such situations, using the -c option
              may be useful.

       -C     Use pcap(3) packet capture library (this must have been  enabled
              at  configure time).  Note that this option must be specified to
              enable TCP-based probes.

       -d debug
              Sets debugging output level.  Generally not useful except to the
              developer.

       -g gap Set  the  mean inter-probe gap in seconds.  The default is 0.25,
              which results in approximately four probes per second being run.
              Care  should  be  taken not to decrease this gap by too much, in
              order to avoid flooding the network.  The default value here  is
              deliberately  conservative;  users  with  the  need or desire to
              probe more quickly are presumed to have  at  least  perused  the
              documentation for the relevant command-line options.

       -G gaptype
              Set  distribution  used to select interprobe gap times.  Current
              alternatives are fixed (the default) and exp,  which  picks  gap
              times from an exponential distribution.  The latter option is an
              attempt to simulate a Poisson process of probe packets (a lot of
              aliteration), however due to the fact that each probe experiment
              takes a non-zero amount of time, this is only an approximation.

       -H hops
              Set the maximum number of hops that pchar will  probe  into  the
              network.   The  default  maximum  is  30  hops, the same as with
              pathchar and traceroute.

       -h     Print usage information.

       -i interface
              Set the interface to listen on for the -C option.

       -I increment
              Set the probe packet size increment.  Pchar will send IP packets
              with  sizes  that  are integer multiples of increment, up to the
              maximum specified by the -m option.  The default  is  a  32-byte
              increment.    Small  increments  should  produce  more  accurate
              results, but will result in more probes (thus taking  longer  to
              run).

       -l origin
              Set  the  local  source of probe packets.  This option is mostly
              useful on multi-homed hosts.  If not specified, it  defaults  to
              the value of hostname(3).  Note that this option must be used if
              the local hostname  cannot  be  resolved  to  an  IPv4  or  IPv6
              address.

       -m mtu Set  the  maximum  probe  packet  size.  This value should be no
              larger than the path MTU between the two hosts.  The default  is
              1500 bytes, the Ethernet MTU.

       -M mode
              Set  operational  mode.   The  normal operational mode is pchar,
              which uses active probes to characterize the bandwidth, latency,
              loss, and queueing of the links comprising a path.  Another mode
              is trout, a “tiny traceroute” that is intended to be used  as  a
              portion of a larger network management infrastructure.

       -n     Don't attempt to resolve host addresses to names.

       -p protocol
              Select  protocol to use.  Current options are: ipv4udp (UDP over
              IPv4), ipv4raw (UDP over IPv4, using raw IP  packets),  ipv4icmp
              (ICMP  over IPv4, using raw IP packets), ipv4tcp (TCP over IPv4,
              using raw IP packets), ipv6icmp (ICMPv6 over IPv6, using raw  IP
              packets),  and ipv6udp (UDP over IPv6).  The default protocol is
              either ipv4udp or ipv6udp, as appropriate to  the  network-layer
              address  associated  with  the hostname provided.  Compared with
              ipv4udp, the implementation of ipv4raw offers finer control over
              the contents of packet fields, but is otherwise identical.  Note
              that the ipv6icmp and ipv6udp options are only available if IPv6
              support  was  compiled  into  pchar,  which  can  be selected at
              configure time.   Finally,  the  ipv4tcp  option  requires  that
              pcap(3)  support be specified at configure time and enabled with
              the -C option.

       -P port
              Select starting UDP port number (the default is  32768).   Pchar
              uses consecutive port numbers starting from this value, counting
              up.  Care should be taken not  to  use  port  numbers  that  are
              actually in use by network services.

       -q     Quiet   mode,   suppressing   all  output.   Useful  if  writing
              statistics to standard out (see the -w option).

       -r file
              Read measurements in from a file named file, as written  by  the
              -w  option.   This  option  is  useful  for  experimenting  with
              different analysis algorithms over a fixed data set.

       -R reps
              Set the number of repetitions of each probe packet  size  to  be
              sent.   The  default is 32 packets of each size.  Smaller values
              speed up testing, at the expense of accuracy.

       -s hop Set the starting hop at which to begin probing.  The default  is
              1,  so  network  probing  will begin at the host adjacent to the
              host where pchar is being run.  Larger values allow  probing  to
              begin  farther  out  from  the testing host; this can be helpful
              when attempting to probe  outside  a  local  internetwork  whose
              characterisics are well-known.

       -S     Do  SNMP  queries at each hop to determine each router's idea of
              what it thinks its next-hop interface characteristics are.   Use
              of  this  features  requires  the  UCD  SNMP library, as well as
              enabling at configure-time using --with-snmp.

       -t timeout
              Set the amount of time (in seconds) that pchar will wait for  an
              ICMP  error message before declaring a packet loss.  The default
              is 3 seconds.

       -T tos Set the IP Type Of Service bits for outgoing UDP packets.   This
              option  isn't terribly useful for a lot of people, but it can be
              used, for example, to  force  a  particular  DiffServ  codepoint
              within  networks that support this functionality.  For values of
              -p that use IPv6 as a network-layer protocol, this  option  sets
              the  traffic  class  field  in  the IPv6 header according to RFC
              2460.

       -v     Verbose mode.  While each probe is in progress, print a synopsis
              of the hop number, repetition, and probe packet size on standard
              out.  Verbose mode mimicks the output of pathchar.

       -V     Print version and copyright information and exit.

       -w file
              Write statistics to a datafile named file.   This  file  can  be
              read  back in by specifying the -r option in a subsequent run of
              pchar for off-line analysis, or parsed  by  other  programs  for
              plotting, etc.

              If  file  is  given  as   - , then the statistics are written to
              standard out.  In this case, the quiet flag -q may be useful, to
              avoid cluttering the standard output stream.

SEE ALSO

pcap(3), ping(8), traceroute(8), pathchar(8)

NOTES

Because  pchar relies on measurements to drive its estimates of network
       characteristics,  it  may  occasionally  produce  some  seemingly   odd
       results.   Care  should be taken when interpreting the output of pchar.
       For example, the coeffecients of determination for  the  least  squares
       fit  can  be  useful  in  seeing  how “good” of a fit the bandwidth and
       round-trip time parameters describe the performance seen by  the  probe
       packets.   The  coefficient  of determination takes values from 0 to 1,
       where a value of 1 indicates that the  estimated  parameters  perfectly
       fit the data.

       Pchar  was  originally  named  pc, which was either an abbreviation for
       “path characteristics” or “pathchar clone”.

BUGS

Pathchar automatically determines an appropriate maximum packet size to
       use, based on a Path MTU discovery algorithm.  Pchar relies on the user
       specifying the maximum packet size manually.

       Some versions of  Solaris  rate-limit  the  generation  of  ICMP  error
       messages.   Any run of pchar through, or to, a Solaris machine may show
       abnormally high packet loss rates.  This  feature  of  Solaris  affects
       traceroute  and pathchar as well, but not ping.  Some versions of Linux
       appear to have similar rate-limiting.  In situations such as this,  the
       use  of  ICMP-based  probes  (selected by the -p option) may yield more
       satisfactory (or at least faster) results.

       Timestamps printed after each run are printed  relative  to  the  local
       time  zone.   Timestamps  saved in trace files are expressed as seconds
       past the epoch.

       There are way too many command-line options.

AUTHOR

Bruce A. Mah <bmah@acm.org>.   The  author  of  the  original  pathchar
       utility is Van Jacobson <van@ee.lbl.gov>.  The algorithms used by pchar
       were coded from Van Jacobson's viewgraphs describing the  operation  of
       pathchar.

                                15 January 2001                       pchar(8)






References:



http://www.brendangregg.com/Perf/network.html

http://tnlandforms.us/netlinks.html

http://docstore.mik.ua/orelly/networking_2ndEd/tshoot/ch04_02.htm

http://www.caida.org/research/performance/bandwidth/

http://www.caida.org/tools/utilities/others/pathchar/

http://www.caida.org/tools/utilities/others/pathchar/pathcharnotes.html

http://manpages.ubuntu.com/manpages/raring/man8/pchar.8.html



0 comments :

Post a Comment

Thank you for taking the time to comment. Your opinion is important and of value and we appreciate the positive feedback! If you are "Negative Nancy" then please do us, and humanity, a favor, and piss off.

Total Pageviews

Google+ Followers

Pages

Blog Archive

Popular Posts

Recent Comments

Rays Twitter feed

Ads

Web sites come and go and information is lost and therefore some pages are archived. @rayd123. Powered by Blogger.