Recently I have been experiencing problems with all of our customers who access our services through Shaw including my own home internet connection. These problems seem to be wide-spread throughout the city.
While throughput speeds and latency have remained good, jitter has become a major problem.
What this means is upload/download speeds would appear to be normal but applications that require real-time packet delivery are effected. At home I notice major problems with things like VoIP and gaming.
A packet analysis using ethereal indicates major spikes in the "diff" values and jitter with many packets arriving out of sequence. This only happens on at the destination end, not at the Shaw-customer end.
This is near impossible to report as a problem because the techs you get on the phone have never even heard of jitter and any upload/download test they get you to run will be fine because TCP is built to handle this.
However, things like First Person Shooter games are now a complete lost cause.
My question for the group is, is anyone else experiencing this problems? I'm trying to determine how wide spread this is before attempting to escalate the issue.
Regards,
John Lange
On Friday 25 August 2006 13:06, John Lange wrote this amazing epistle:
Recently I have been experiencing problems with all of our customers who access our services through Shaw including my own home internet connection. These problems seem to be wide-spread throughout the city.
<snip>
My question for the group is, is anyone else experiencing this problems? I'm trying to determine how wide spread this is before attempting to escalate the issue.
Regards,
John Lange
I've noticed things slowing down a little (then again I'm also running a couple of P2P programs). I don't know if this would help but I find I get booted from some P2P connections unless there are large numbers of peers. The same sort of thing happens when I connect with various nntp servers including Shaws. For example the download of binary articles frequently freezes and resets (BNR2 is good for that but pan is not). Even getting headers from large groups shows the same sort of behaviour. Before it was just Shaw. Now it's from stable servers outside Shaw.
You may also want to look at streaming video and VNC type connections. The streaming video may be the key. Look at some stuff from YouTube on both Shaw and MTS. This way you can isolate it. If you can use the same machine on both, so much the better because they can't claim it's your machine. Since it's video you can talk in language they understand. "The video freezes and drops frames. You say I'm getting all the packets. Maybe they aren't arriving in the correct order or they aren't arriving at a consistant speed. MTS doesn't have this problem and they are slower than you." Since video is Shaws core business, this is something they need to pay attention to.
Later Mike
On Fri, 2006-08-25 at 15:00 -0500, Mike Pfaiffer wrote:
You may also want to look at streaming video and VNC type connections. The streaming video may be the key. Look at some stuff from YouTube on both Shaw and MTS. This way you can isolate it. If you can use the same machine on both, so much the better because they can't claim it's your machine. Since it's video you can talk in language they understand. "The video freezes and drops frames. You say I'm getting all the packets. Maybe they aren't arriving in the correct order or they aren't arriving at a consistant speed. MTS doesn't have this problem and they are slower than you." Since video is Shaws core business, this is something they need to pay attention to.
Thanks for the suggestion Mike but I don't think streaming video will be the answer for a few of reasons;
1) The Video from those sites is buffered purposely to accommodate dropped or out of order packets. So in effect, even though the video is "streaming" its not real-time.
2) I'm pretty sure they are using TCP for this type of buffered video and the TCP stack will also accommodate for out of sequence packets.
3) Download is fine on Shaw, its upload that is the issue. The video you are referring to is download only.
What I'm trying to determine is if this is just a problem between us and Shaw or if its Shaw system-wide. Because of the issues with gaming I'm leaning toward a Shaw system-wide problem.
Just hoping someone else can verify.
John
John is correct that Video and VoIP are very different. Video requires a lot of bandwidth, since it is only one way can be buffered without any latency issues, generally TCP. On the other hand VoIP is low bandwidth, two way so buffering adds latency which is problematic and usually UDP.
I did some pinging this afternoon and there is some erratic ping times but I am impressed with the low times that I am seeing. pings to epic.ca average at 21 ms. The larger ping times are still under 300ms. As you say when you complain to Shaw it may be difficult for them to see a problem.
Of course pinging every sec does not simulate a RTP stream.
-- Bill
On 8/25/06, Bill Reid billreid@shaw.ca wrote:
Of course pinging every sec does not simulate a RTP stream.
This reminds me that I set up smokeping to watch for this type of stuff:
Here's a ping to my default route:
http://ertw.com/cgi-bin/smokeping.cgi?target=Gateway
And from home to work (probably going to go via Group Tel/Bell West)
http://ertw.com/cgi-bin/smokeping.cgi?target=Winnipeg
A fair amount of jitter (represented by the "smoke"), and there is a noticable difference in jitter in August (400 day graph of the Gateway graph), even after the data has been averaged.
Sean
Sean, you never cease to amaze me. Thats a fantastic graph and is exactly the kind of evidence I need to prove there is an issue.
I'm going to forward that link on to shaw's tech support.
Thanks!
John
On Fri, 2006-08-25 at 19:52 -0500, Sean Walberg wrote:
On 8/25/06, Bill Reid billreid@shaw.ca wrote: Of course pinging every sec does not simulate a RTP stream.
This reminds me that I set up smokeping to watch for this type of stuff:
Here's a ping to my default route:
http://ertw.com/cgi-bin/smokeping.cgi?target=Gateway
And from home to work (probably going to go via Group Tel/Bell West)
http://ertw.com/cgi-bin/smokeping.cgi?target=Winnipeg
A fair amount of jitter (represented by the "smoke"), and there is a noticable difference in jitter in August (400 day graph of the Gateway graph), even after the data has been averaged.
Sean
-- Sean Walberg sean@ertw.com http://ertw.com/
Sean Walberg wrote:
I am now using smokeping. A very nice monitoring package.
In the process I discovered cacti a replacement for MRTG. If you have struggled with setting up graphs with MRTG you will love this package. You configure everything via a web interface.
-- Bill
On 8/27/06, Bill Reid billreid@shaw.ca wrote:
In the process I discovered cacti a replacement for MRTG. If you have struggled with setting up graphs with MRTG you will love this package. You configure everything via a web interface.
It's an amazing package, we use it at Ceridian too. You can share graph templates via XML, there's a wealth of stuff that can be monitored that's up on the Cacti forums, or if you're willing to dig around, make it yourself.
On the topic of MRTG/RRDTool, if anyone's interested in monitoring web application performance with RRDTool I wrote a tutorial on the subject a few months ago: http://www.ibm.com/developerworks/edu/dw-esdd-webperfrrd-i.html The idea is to be able to graph various components of web site performance to determine what's to blame (DB, DNS, Internet, etc).
Sean
I also have now setup smokeping. Thanks again Sean.
The only problem I have with the RRDTool based programs is they are "lossy". Meaning, the don't keep the original data which the graphs are based on so you can't go back and query them for specific time periods or use them for accurate reporting.
This is why we use rtg instead of mrtg to keep track of bandwidth. rtg stores all its data in a mysql table which allows you to query any time period you have data for and see detailed results. This also means the bandwidth usage is accurate, where as with mrtg its just an average.
I wish smokeping had the same thing. Probably could be added without too much trouble. Might be a project I undertake one evening.
John
On Sun, 2006-08-27 at 10:54 -0500, Sean Walberg wrote:
On 8/27/06, Bill Reid billreid@shaw.ca wrote: In the process I discovered cacti a replacement for MRTG. If you have struggled with setting up graphs with MRTG you will love this package. You configure everything via a web interface.
It's an amazing package, we use it at Ceridian too. You can share graph templates via XML, there's a wealth of stuff that can be monitored that's up on the Cacti forums, or if you're willing to dig around, make it yourself.
On the topic of MRTG/RRDTool, if anyone's interested in monitoring web application performance with RRDTool I wrote a tutorial on the subject a few months ago: http://www.ibm.com/developerworks/edu/dw-esdd-webperfrrd-i.html The idea is to be able to graph various components of web site performance to determine what's to blame (DB, DNS, Internet, etc).
Sean
-- Sean Walberg sean@ertw.com http://ertw.com/ _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Somewhat co-incidentally, I just read an interesting essay on the whole net neutrality debate, which gives a good explanation of the causes of jitter and network delays in general. You can find it here:
http://itpolicy.princeton.edu/pub/neutrality.pdf
Gilles
On Aug 25, Sean Walberg wrote:
On 8/25/06, *Bill Reid* billreid@shaw.ca wrote:
Of course pinging every sec does not simulate a RTP stream.This reminds me that I set up smokeping to watch for this type of stuff:
Here's a ping to my default route:
http://ertw.com/cgi-bin/smokeping.cgi?target=Gateway
And from home to work (probably going to go via Group Tel/Bell West)
http://ertw.com/cgi-bin/smokeping.cgi?target=Winnipeg http://ertw.com/cgi-bin/smokeping.cgi?target=Winnipeg
A fair amount of jitter (represented by the "smoke"), and there is a noticable difference in jitter in August (400 day graph of the Gateway graph), even after the data has been averaged.
Sean
John Lange wrote:
My question for the group is, is anyone else experiencing this problems? I'm trying to determine how wide spread this is before attempting to escalate the issue.
I do, on occasion, play some FPS online games and have noticed that in the past, little while (not that long mind you) that my frame-rate was excruciatingly slow that it caused me to exit the game. I have shaw's regular high-speed, not the extreme.
Dan.
-
Bill Reid -
Dan & Michele -
Gilles Detillieux -
John Lange -
Mike Pfaiffer -
Sean Walberg