Re:CCP Games brings Sentry to the PlayStation
by james33 on Friday January 10, 2014 @03:27PM
(#7039140)
Sentry really is a life saver. We use it for our HTML5 MMO for both the client-side Javascript and the back-end in Node.js, and it has helped us to dramatically improve our end user experience. Really cool seeing this used out of the browser!
Re:CCP Games brings Sentry to the PlayStation
by look_lookatme on Friday January 10, 2014 @03:17PM
(#7039089)
I've read the Wikipedia article on Stackless Python but I was wondering if someone could quickly explain why Stackless is more desirable to CCP than regular Python?
Re:CCP Games brings Sentry to the PlayStation
by sp332 on Friday January 10, 2014 @03:32PM
(#7039169)
Unlike most MMOs, which have multiple instances or "shards" of each location, EVE Online keeps all players in one shard. That means, for busy hub areas, they have to handle thousands of ships interacting with each other all on one server.
The lightweight concurrency lets them write little (and not-so-little) bits of code that execute in parallel without incurring overhead associated with OS threads.
Edit: sorry if I'm confusing "concurrent" and "parallel", I never really liked those terms :p
Re:CCP Games brings Sentry to the PlayStation
by cdash on Friday January 10, 2014 @03:57PM
(#7039314)
As far as I know nothing in EVE executes in parallel in a traditional sense. Each solar system is run on one thread and one python interpreter. While stackless is used for concurrent programming it is not used for parallel programing.
Re:CCP Games brings Sentry to the PlayStation
by krisvale on Friday January 10, 2014 @06:52PM
(#7040526)
Thousands of lightweight threads with cooperative scheduling. Allows you to have a myriad of little programs running, each perhaps servicing a network request, or running some long running game logic. Impossible with threads. Also, threads are pre-emptive, which means that you need to use locking almost everywhere.
Re:CCP Games brings Sentry to the PlayStation
by famousactress on Friday January 10, 2014 @04:51PM
(#7039749)
This is a handy reminder. We spun up Sentry a while back (like, maybe 1-1.5 years ago?) and spun it back down because logging to it via TCP was unacceptable for us.
For the folks using it, how do you get data to it? Curious how things have shaped up over time.
Re:CCP Games brings Sentry to the PlayStation
by famousactress on Friday January 10, 2014 @05:21PM
(#7039993)
We didn't want to write to something synchronously without having a fault-tolerant solution for latency... especially when we could afford the end-user latency of seeing the exceptions later. I don't want my logging aggregator to be a SPOF for the main app. Would much rather suck up the data asynchronously from syslog or files, etc.
Re:CCP Games brings Sentry to the PlayStation
by krisvale on Friday January 10, 2014 @06:50PM
(#7040510)
Hi, at CCP we have a custom transport that does the logging to sentry on a separate tasklet (think lightweight thread). Basically, the logging is done in the backgrond when there is time, freeing the thread whence the error came up to do other stuff immediately.
Re:CCP Games brings Sentry to the PlayStation
by zeeg on Friday January 10, 2014 @07:53PM
(#7040848)
There's two things to consider here:
1) Sentry is not designed to replace logging therefor things like syslog will never achieve the same result. Sentry is explicitly for catching outliers and TCP is the only method able to store it's data.
2) As with CCP most people no longer use synchronous transports. We even changed the default to be the threaded worker transport. That said, when your user hits a 500 adding 50-200ms is generally acceptable as you've already put the user in a bad situation.
Re:CCP Games brings Sentry to the PlayStation
by famousactress on Monday January 13, 2014 @12:16PM
(#7051891)
Late getting back to this but wanted to clear up some things:
1) Sorry, I think I was misunderstood. Was saying I'd rather populate Sentry from logs not that I think my logs are an alternative for Sentry.
2) Awesome. I'll look at the threaded worker transport stuff. As for the 50-200MS on 500's, I totally agree. Didn't get around to sorting out making connection timeout something like a couple hundred seconds though (at the time it wasn't clear to me if that was an exposed setting for the Sentry lib), and also I was hoping to use Sentry for situations where I might give our users a 200 but warn us internally about and use Sentry to notice a large uptick in the warnings.
The lightweight concurrency lets them write little (and not-so-little) bits of code that execute in parallel without incurring overhead associated with OS threads.
Edit: sorry if I'm confusing "concurrent" and "parallel", I never really liked those terms :p
For the folks using it, how do you get data to it? Curious how things have shaped up over time.
1) Sentry is not designed to replace logging therefor things like syslog will never achieve the same result. Sentry is explicitly for catching outliers and TCP is the only method able to store it's data.
2) As with CCP most people no longer use synchronous transports. We even changed the default to be the threaded worker transport. That said, when your user hits a 500 adding 50-200ms is generally acceptable as you've already put the user in a bad situation.
1) Sorry, I think I was misunderstood. Was saying I'd rather populate Sentry from logs not that I think my logs are an alternative for Sentry.
2) Awesome. I'll look at the threaded worker transport stuff. As for the 50-200MS on 500's, I totally agree. Didn't get around to sorting out making connection timeout something like a couple hundred seconds though (at the time it wasn't clear to me if that was an exposed setting for the Sentry lib), and also I was hoping to use Sentry for situations where I might give our users a 200 but warn us internally about and use Sentry to notice a large uptick in the warnings.