Re:Processing billions of events in real time
by EdwardDiego on Monday November 15, 2021 @06:24PM
(#29234330)
Sad to say, ad-tech is one of the more accessible spaces if you enjoy creating systems that have to dynamically scale, provide high throughput and/or near real-time streaming data solutions, while also ensuring data correctness (for a given value of correct).
Maybe fintech could offer similar challenges in some areas? I'm thinking HFT etc.
I greatly enjoyed the challenge of building data pipelines in my role at $LAST_COMPANY, an SSP, that delivered valid transactional + related entity data to a near realtime reporting system - and scaled up and down as needed to maintain data timeliness while minimising costs (as ad traffic (well internet traffic as a whole) has high seasonality throughout the day).
But I didn't enjoy working for a company in the ad-tech market though - far too many deals that, while legal, felt sleazy - and (from my limited experience) often seemed to begin with handshakes made by sales reps on booze / cocaine fueled nights in red light districts.
If I could solve similar business problems without being in a market that makes me want to have a long shower, I'd be keen.
Maybe fintech could offer similar challenges in some areas? I'm thinking HFT etc.
I greatly enjoyed the challenge of building data pipelines in my role at $LAST_COMPANY, an SSP, that delivered valid transactional + related entity data to a near realtime reporting system - and scaled up and down as needed to maintain data timeliness while minimising costs (as ad traffic (well internet traffic as a whole) has high seasonality throughout the day).
But I didn't enjoy working for a company in the ad-tech market though - far too many deals that, while legal, felt sleazy - and (from my limited experience) often seemed to begin with handshakes made by sales reps on booze / cocaine fueled nights in red light districts.
If I could solve similar business problems without being in a market that makes me want to have a long shower, I'd be keen.