CDN update

Our idea of relaying incoming tracking data through our CDN seems to be unrealistic. It works near-perfectly, until there's even a minor network hiccup on either end (the CDN network or Clicky's home network). We're tracking so much data, when the network is unavailable even for just 10-20 seconds, data queues up insanely fast and this starts an inevitable downward spiral.

This is what happened today, a few different times. We tried many different tricks to make it work better, but nothing worked. There's a bit of missing data from your stats most likely. We apologize for that.

The CDN has been temporarily disabled so that all outgoing and incoming traffic points to our "real" servers again as soon as possible (DNS time-to-live values were only 15 minutes because of the auto-failover that Dyn DNS gives us, so this should happen very fast).

We're going to re-enable the CDN for our static files and tracking code within the next day or two, because that all worked perfectly. It was the incoming traffic data that was the problem, and this will now bypass the CDN entirely and go straight to headquarters. It's the outgoing data that's the most important anyways, so this isn't the biggest deal in the world. Logging incoming data would have been a nice addition to the arsenal, but life goes on.

Please don't assume we didn't test the CDN before deployment, because of course we did. But it wasn't until the full brunt of the entire world was sending incoming data to these servers that problems started surfacing. That scenario was untestable so the problems resulting it from it were not predictable until the system was live.

Update: The CDN is live again as of ~2AM PST on March 10. It is now only handling outgoing data. Things are working smoothly right now. If you notice anything strange, please let us know!

16 comments | Mar 09 2010 4:35pm

Reader Comments

I understand this situation since i work with CDN company. Continue to improve it ;-)

mrkay Mar 09 2010 6:03pm

So, why are you asking for me/us to reinstall the tracking code? If it's really necessary, I'll do it. But if not, I'd rather not, since it has to be done on every page. And, btw, Clicky is outstanding! Thank you.

Sister Mar 10 2010 7:14pm

Does this mean the clicky.log() call packets will still have to make the round-trip between our server in Amsterdam and the US?

Sander Mar 11 2010 12:04am

Unfortunately, yes. That shouldn't be a huge deal though, since that happens in the background.

Sean (Clicky) Mar 11 2010 12:04pm

Please start working on adding cookies to getclicky because i am thinking of moving to woopra ... And i dont eant i love getclicky .

Anonymous Mar 11 2010 8:17pm

That is on the roadmap for this month!

Sean (Clicky) Mar 11 2010 9:24pm

So do we have to update the tracking code or will it still work as is. I see you had problems with CDN and wondering if that affects the manual tracking code. If so, is the tracking code correct when I go to it from Preferences?

Gloria Mar 15 2010 7:25am

You don't have to update your tracking code, we always maintain backwards compatibility, but we definitely recommend doing it - especially if you're doing the self-hosted option. It will play better with our CDN.

Sean (Clicky) Mar 15 2010 8:06am

This round-trip thing is a big deal actually and it's the reason we'll probably have to stop using clicky, even though we love the product. We use clicky.log() on a HTTPS website to track certain actions on our website. Now every click results in the message "waiting for static.getclicky.com" in the status bar for anywhere between 400-1000 ms. Furthermore, the fact that clicky isn't asynchronous results in a delay of the onload event which causes our own javascripts to slow down as well. I wish there would be a solution for this, since we don't want to give up clicky, but we are forced to...

Sander Mar 18 2010 3:01am

You're in Amsterdam, the tracking code should load lightning fast. My tests from a server in that country showed 20-30ms total download times. That was non-SSL, but the SSL should still be very fast.

Furthermore, why does it matter if "waiting for..." shows up in the status bar, if the page is already fully loaded? It doesn't force your browser to actually wait for our servers to respond before it do the action you are tracking - it's in the background.

Sean (Clicky) Mar 18 2010 9:36am

Hi Sean,

Unfortunately I have to agree with Sander, as I'm very often seeing the same thing (non-SSL) (also based in Amsterdam). Not so much for the js file, but more so for the in.php script. Next results are with the Web Inspector Tool in Google Chrome and FireBug's Net Panel.

Google Chrome
------------
Homepage:
static.getclicky.com/js: 14ms latency, 53ms total download
in.getclicky.com/in.php: 418ms latency, 0ms download

Article:
static.getclicky.com/js: 113ms latency, 11ms total download
in.getclicky.com/in.php: 414ms latency, 1ms download

FireBug Net Panel
--------------
Homepage:
static.getclicky.com/js: 0ms DNS lookup, 36ms Connecting, 46ms Waiting - 82ms total
in.getclicky.com/in.php: 59ms DNS lookup, 202ms Connecting, 216ms Waiting - 477ms total

Article:
static.getclicky.com/js: 51ms DNS lookup, 87ms Connecting, 90ms Waiting - 228ms total
in.getclicky.com/in.php: 50ms DNS lookup, 199ms Connecting, 212ms Waiting - 461ms total

Jean-Paul Mar 22 2010 4:15pm

To add, the DOM isn't fully loaded, since my jQuery scripts are positioned after Clicky's, and they unfortunately are blocked from downloading while waiting for Clicky's in.php script to do it's thing. I would love to see an asynchronous solution like Google is offering.

Jean-Paul Mar 22 2010 4:18pm

Why not move the Clicky script so it's the very last thing?

Also, the tracking code can be configured to not log to in.getclicky.com until the page has fully loaded. It does that by default, until 2 seconds have passed, in which case it will log anyways, just to make sure it captures as much data as possible.

If you want to change it to log only after "onload" has fired, set clicky_custom.delay="onload";

http://getclicky.com/help/customization#delay

Sean (Clicky) Mar 22 2010 4:34pm

> Why not move the Clicky script so it's the very last thing?

Because W3 Total Cache (for WordPress) is injecting the concatenated and minified JavaScript files as the last DOM element before [/body] (and after wp_footer, which is where Clicky is inserted)

> If you want to change it to log only after "onload" has fired, set clicky_custom.delay="onload";

Thanks, I may have read about that option earlier, but will investigate again and see if it helps anything.

I really was hoping the CDN feature would speed up things (and it really does for the clicky.js file), but unfortunately the logging to base thru CDN didn't work out as you'd hoped. It's that part which is hurting us Europeans the most I guess.

Jean-Paul Mar 22 2010 6:56pm

Added: your proposed solution of adding the var looks very promising in early testing. I added it to Yoast' WordPress plugin for Clicky and from looking at Chrome's Developer Panel and Firebug's Net Panel in.php is now waiting after the onload event has fired. Thanks for the pointer.

Jean-Paul Mar 22 2010 7:28pm

Great!

Sean (Clicky) Mar 23 2010 1:41pm

Recent Comments

CDN update

Reader Comments