I just sent this same message to support (at) mixpanel.com.
EDIT: just to reiterate -- the major problem isn't the API being down. It's the extreme timeouts being experienced trying to load a static JS file.
This is the second time you guys have had a major outage in recent memory and your status page looks like a christmas tree. I really do love mixpanel, but unfortunately I'm going to relegate your service in my mind to 'untrustworthy third party service' every time I write code against your api.
Also the fact that I found a pretty major bug in your loader code which prevents it from working correctly with IE browsers (https://mixpanel.com/docs/integration-libraries/javascript) last week and today you are still distributing the broken code has me worried. Do you not take this stuff seriously?
The issue is that you can't insert b (the <script> tag) into the DOM until you've declared window.mixpanel because the code within mixpanel.2.js depends on window.mixpanel to be defined.
It is a subtle race condition that only seems to happen with IE, but when it would fail, it would cause one of those ugly IE dialogs to popup to the end user. Not good.
The correct code should look like this near the end... just move the insertBefore to be after the window.mixpanel:
a.__SV = 1.1; window.mixpanel = a; d.parentNode.insertBefore(b, d);
I wish that was true. But since the library doesn't handle service downtime at all gracefully, we're stuck with the callbacks not firing -- something we depended on.
The SnowPlow architecture diagram is here: https://github.com/snowplow/snowplow/wiki/Technical-architec...
Might I suggest something like Chaos Monkey? Real-time APIs are highly vulnerable to DDOS and other traffic conditions because they'll so readily accept new connections, might be something worth throwing on the crew regular-like. :)
We believe we have figured out the problem and have already gone ahead with a solution. The issue, at this time, appears to be that the traffic we normally get to our API has increased substantially. Whether it's legitimate or not is unknown but we're unfortunately at the mercy of waiting for DNS to update. It's also possible we may see bottlenecks further down our infrastructure path but we're actively thinking through what's next - fortunately, we have more control there.
We'll have a full transparent write up once things are back to normal.
If you're using our JavaScript libraries, your website should not be greatly affected by our downtime other than the data that will be lost. We're deeply sorry and disappointed ourselves.
Our support team is happy to talk to you at status.mixpanel.com to give you updates as they get them.
Suhail