Ironically, Urban Airship (now just Airship) itself is great example of something that people use even though they don't need it - due to hype. You have a mobile app and it needs to send notifications, but you don't have any in-house expertise on mobile notifications. Management says, "ok, let's outsource this to somebody who has some expertise!" They look around and find Urban Airship - the leader in mobile app notification handling!

So you "integrate" with Urban Airship, per management dictates (the same reason you used Hadoop, Cassandra and Kafka). They give you this big complicated presentation but you discover that, in order to use their "service" you have to:

1) instrument your application to send their API a notification every time something happens in your application.

2) set up a workflow in their system that tells them when to tell you that something happened that a notification needs to be sent for

3) instrument your application to read a dataset from them telling you when to send a mobile notification

4) actually send the mobile notification

You realize that they're charging your company a ton of money to do... absolutely nothing. You're still doing everything, responsible for everything, and they're nothing but completely empty overhead. They could have been replaced by a Postgres instance running on a single underpowered EC2 node (and it would be more reliable). But management sees "leader in mobile app notifications" the same as they saw "leader in distributed computing loads" and insisted that you use this thing that exists because it exists.

- Did you also build the audience segmentation system (and UI) for your marketing team?

- Did you build a scheduled / triggered push system with per-device (or user account) rate limiting?

- Did you build the streaming data interconnect to pipe audience data back to anywhere else in your organization?

- Did you build the capability to remove devices that uninstalled your app from your notification list (surprisingly tricky, at least a few years ago, and APNS and Google will start getting very upset with you if you don't do this)

- Did you build an embeddable notification inbox with state synced across devices?

- Did you build a CDN for in-app notifications with video / images?

Because Airship did, and they will sell you all of that in one package. They'll charge an arm and a leg for it, but they do a lot more than just sit as a middleman between you and APNS when sending notifications. The relatively simple backend tech is definitely not their product, and if that's all you used from them then I would understand your frustration.

I can assure you, Urban Airship was not providing "absolutely nothing". If they hadn't been so expensive, we'd have used them and it would have saved me a shitload of time. Open-source support for push at the time was rudimentary (I assume that's gotten better? I haven't looked at it in about a decade) and in any case, doing it at any scale and even somewhat-good reliability involved a lot of work and dragging in tons of extra services (at least, a database and some kind of message queue service, unless you wanted to write those yourself)

[EDIT] Now, somewhat after that 2010 date, maybe 2012 or 2013 IIRC, UA re-structured their pricing and offering a ton which made them even less appealing to us, since their focus seemed to be value-add features for marketers on top of push, which was something we had no use for whatsoever—it's possible that by then they were indeed more trouble than they were worth if you just needed push messaging, even putting aside the cost of the service itself. If your marketing folks were big on using that as a "channel", though, it was probably very much worth it. They had a lot of features for that kind of thing.

1) Your app needs to register with Apple to get a push token, using the app's push certificate (which you'd have to generate if your app used push messaging). It needs to send that to a server you control, because that's basically your "to" address to send messages. These were not to be treated as permanent, so to do it right you also had to be ready for these to change.

Then, similar story for Android, but all shoddier, less efficient, and worse-engineered, but also higher-level—so, situation normal for Android vs. iOS in (at least) the early '10s :-) So you're looking at two similar but necessarily entirely separate implementations of this sort of functionality, to cover both platforms.

2) You need to store all those tokens your apps are sending you.

3) To send the push, you need to generate a bunch of individual messages for each address. For Apple, you'll be using the app's push cert to authenticate over a raw (I think? Memory's fuzzy) UDP connection, then blindly firing a stream of data at it (no immediate return statuses, it's a one-directional communication channel). For Google, you'll need to manually chunk these into sets of a certain max size and post the chunks one at a time to some Web endpoint. For an app with one million users sending a broadcast notification to all of them, that's one million messages each time you send. Google's system required that you implement exponential backoff, too, if you didn't want to get banned.

4) Oh, if you're sending to multiple apps, you need a way to manage (add, update) the certs (or, for Google, IIRC, an API token of some kind?) for them and to select the correct one for a given push.

5) Google would tell you at send time if an address was bad (IIRC—it has been about a decade since I touched this stuff). For Apple, you had to wait a while after a send, then open a new connection and ask for a list of bad tokens for that app, which it will happily spit at you as another UDP data stream. These will be uninstalls, folks who've turned off push for your app, whatever. You have to remove those from your token list, facing vague threats of service bans if you keep sending to bad addresses for too long.

6) If your scale is non-tiny and you need any amount of delivery guarantees or reliability, it's plain by now that you need a queue of some kind. So if you're over that line but not too far over it, you hack something together in PostgreSQL or what have you, probably make some minor but OK-for-your-needs errors in the implementation, and live with it. If your needs are greater than that, this is where you start looking at shit like AMQP, which solves a ton of your problems while giving you an ongoing maintenance headache. Retries, fanout to multiple workers, aggregating failure data, et c. There's a lot going on there.

7) Do your senders want to schedule pushes for the future? Now you get to implement cron-for-push-messages, including a UI for it.

8) Do your senders want to send individualized messages? Now you need to hook into some kind of datastore that ties those push tokens to other user data so you can fill that stuff in while generating messages. And you need at least a rudimentary templating system.

9) Do your senders want to send messages in response to user actions in the app? Now you need a way to ingest potentially very bursty and high-volume traffic from a ton of clients, and connect those to configurable push-send triggers. Probably this'll be another thing you need to route through your queuing system, if you don't want to badly over-spend on servers while not having great reliability under load.

10) Push history and stats reporting is probably gonna be needed, at some point, by someone.

This list just keeps going, really.

3) and 5) in particular were significant hurdles -- allowing UA to handle the API and TOS differences between Apple and Google was a major benefit. It wasn't just a matter of not wanting to implement it ourselves, but the risk of implementing it incorrectly and getting banned from the service.

Small nitpick: "UDP is considered a connectionless protocol because it doesn't require a virtual circuit to be established before any data transfer occurs."

https://www.techtarget.com/searchnetworking/definition/UDP-U....

However, technically correct is the best kind of correct :-)

I may also be mis-remembering, and APNS might use TCP. Either way, you're spraying bits at a socket and not getting any kind of feedback (until later, on a separate connection), aside from getting dropped if your cert doesn't check out (again, IIRC).

Where I worked did eventually drop them and start using our own redis queue + long-running process and that was plenty sufficient for what we did. We lost the "+1 to iOS's counter" feature (this was prior to background processing of notifications existing at all), but we had solid evidence that it simply wasn't providing us much benefit for the complexity/cost compared to just a numberless dot.

Obviously we could do that too with some additional storage, but handling it across multiple OSes and weird iOS + Android lifecycle details was enough of a pain that we didn't end up doing it.

Maybe I'm missing something here, but it feels like you don't need some fancy realtime queue service for most push notifications, so why not use something that offloads the checks to the client?

RSS is a neat standard, but if you're writing your own app, you might as well use a custom JSON payload that's exactly what you need. If you use RSS but never actually use an RSS reader or expose it to the public, you may end up deviating from the standard, or doing extra work to be compliant with the standard that never gets used.

As far as push notifications, the reality is that "push" is a bit of a marketing term. Every time your phone wakes up and gets on the network, it contacts Apple or Google and sees if there are any pending pushes to be delivered. By using the "push notification" systems, you get to piggyback on that highly optimized connection already provided by Apple or Google. These pushes have to be much smaller than your typical RSS feed and are intended for high priority delivery.

An alternative, if you don't want your data going through the "push" networks, is to have your app "background fetch". You might want to do this if it's too much data (like an RSS feed would be) or if it's sensitive data (Apple and Google can obviously read all push messages). In this case, the iOS and Android have special "background fetch" use cases, which gives your app a chance to pre-load data. News apps are the canonical example.

> it feels like you don't need some fancy realtime queue service for most push notifications, so why not use something that offloads the checks to the client

Mostly because the realtime queue already exists, it's provided mostly for free, it comes with extra features like sounds and badges, it was available before background fetches, and business people know to ask for "push" by name.

- If I want push notifications for meetings, I want to get them exactly 10 minutes before my meeting, not 2 hours after, or 10 minutes before but +/- 30 minutes

- If a customer is submitting a critical bug to my bug tracker, I want to know immediately so I can start damage control.

- If a developer on my team responds to my comments on a Pull Request at 3PM, I don’t want to get the notification after I’ve clocked out at 5, causing us to essentially lose a day of work.

In practice, I don’t think there are many use cases where people don’t care about the punctuality of push notifications. And if you start by tackling those use cases and then an important customer says “hey, I actually need punctuality for this new class of notifications”, then you’ll suddenly need to rebuild your whole notifications stack.

Yeah, I didn’t look at them at all today. I’m still alive. As a user, I don’t care about how punctual they are since I don’t look at them until the end of my work day.

Not that there is anything wrong with that, and in many use cases it would be a fine alternative. But sometimes you really do need to push data to the user even if they aren’t polling for it.

Pill reminders would be something that you’d want to push. You can’t be sure the user is running your app (even in the background), and the actual state changes are rare. So some kind of mechanism where the client doesn’t initiate the check would be better.

Using a long poll for email or slack or something like that can make more sense in some scenarios.

From your app's point of view, you receive a "push" from the OS.

Depending on the use case, you may well depend on the real-time aspect of push notifications. If my door bell rang while using headphones, I'd want to know immediately. I'd also be pretty pissed if a messenger took more than 5 seconds between sending and receiving on the other end, event if "closed" or in background.

https://www.airship.com/app-experience-platform/pricing/

> AXP ENTERPRISE

> Starts at $25k/Annually

> 100k + MAUs

> Powerful App Experiences

> Mobile Data Hub

> Customer Journey Optimization

> Success Packages

> Deep Mobile Expertise

> * Overage rates apply

> * Success Package required – prices may vary

More charitably, I think Airship was useful before Android / iOS built their feature into the platform natively.

For small apps, that's another service to build and maintain. Airship serves well that purpose.

Now, if you are a large company, and have a large volume of pushes, then it doesn't makes sense to pay Airship, but build a service yourself, and have a team to maintain it. Usually, even the smallest service, if you are doealing with Apple's (and Google's) external APi, will have trouble time to time, and you will have to have a team to 'passively' monitor and maintain it.

For a company with less than 100 people, that doesn't really make sense. But with a later stage company, it totally makes sense to have your own service.

Anwyay, just my 2cents, as I have done both, build my own Notification service for a startup, and used Urban Airship for my own personal projects.

In Twillio's case after a half hour of you could end up with some sort of send_text(number, msg) function/method that you can just use all over the place and if anything changes in the text message world they handle it.

Id rather do that than have to deal with the carriers, etc. The way you describe sounds like a pain with vendor lock in on top.

These third-party push notification services are essentially overlays on top of the above, don't actually change the workflow at all (the above is the bare minimum you'd need for the functionality to work) but introduce extra complexity, moving parts, and more importantly, extra rent you now need to pay to them.

UNPHAT is a good approach. Any webshop essentially has Amazon's "a failed add to cart will lose us money"-problem and there are a lot of webshops out there, so it's even a common problem! Cassandra may be a good choice even for a smaller webshop (I'm not familiar with it myself, and may be a poor trade-off with respect of ROI, but that's another thing), but it's definitely not a good fit for a lot of other things like batch jobs.

At my current $dayjob we run a low-volume high-profit B2B business; we have a few hundred customers, and it'll never be more than a few thousand – several ten thousand at the most if we end up wildly successful. It's a whole forest of microservices and will scale to the moon, but it's also pretty complex and difficult to work with. All that for a fairly small customer base using a fairly basic web UI they rarely log in to. We do actually have high-availability requirements in the core business, but that's separate from the management UI. It's a good example of not having understood the actual problems and just using something "because I read about it on Hacker News".

Clearly, having HA and reliable transport is reasonable for your use-case but that isn't the same as microservices, hadoop or Kafka.

They talk about remediating issues with email alerts. That is exactly the kind of approach you can take when you are nowhere near Google's scale, instead of building a complex system to ensure all of these remediations can happen automatically without any human intervention. Sounds like they took exactly the right approach of not over-engineering a solution prematurely.

Ironically we kind of relied on e-mail alerts for too long and would kinda sorta break Gmail. While it was probably just the web/IMAP interface or browser/client itself struggling, trying to bulk delete tens of thousands of emails while hundreds more per second were coming in often had incoherent results.

I think I was mainly writing in agreement with the article’s closing:

> What we’re all imploring you to do is to think! And to actually understand the problem you are trying to solve.

I remember a project I worked on in the past where the requirement was to have the app display a notification to the user at a certain time, like an alarm clock. Pretty simple, and could be implemented with a line or two of code as a locally-generated notification.

But no, the lead declared that we needed a push notification from a server in order to do this properly. Even after the difference between a local notification and a push notification was explained to him (he didn't know that local notifications existed), he insisted that notifications = push notifications, and we must have a server. AND that server needed to run a complex web service. AND it needed a way for devices to log in with credentials, so it needed a database. AND it needed to be fault tolerant since this was a critical user journey in the app. AND the clients had to check in with the server periodically, since their location might have changed to a different time zone, therefore the push notification time had to change. AND this AND that AND so on. All for what was essentially an alarm clock--something you could implement in the client code running on the device in 5 minutes.

IIRC this didn't exist at the time but is indeed the ideal solution. (some more discussion of this downthread)

If pill reminders were sync'd between devices (or between mobile and web verisons of the product), then push notifications would still be necessary to inform one side or the other when to resync data if nothing else.

Still, 2010 was a wildly different time in mobile. We were all still figuring a lot out (clearly)!

Mobile has come a long way for sure!