News for hackers. Stuff that matters.
Re:Check If Email Exists by profmonocle on Thursday July 01, 2021 @11:45PM (#27707633)
A non-marketing example:

We were onboarding a large new client to our SAAS product. This process involved creating accounts for all of their employees (tens of thousands) and sending emails with an activation link. (Where they'd be able to set up their password.)

Our system sends these emails in batches, and as soon as the first batch went out we got an alert from our monitoring system that our bounce rate was surging - high enough to risk a sending pause from Amazon SES. We stopped sending and investigated the issue, and it turned out that the email list we were given was a mess - it included all current employees, but also a huge number of former ones. Just under 1/10th of the emails in our first batch were invalid.

We asked the client to give us a better list, but due to internal issues they couldn't get that to us any time soon. Meanwhile they were breathing down our necks to get these emails out ASAP, and they were a large enough client that our management wanted to keep them happy, so we tried out one of these email validation services. Unfortunately, it didn't work. It turns out that this technique doesn't work for all mail servers. It was reporting every email as valid, even ones we knew were invalid since they'd already hard bounced.

(Edit: thinking back - this was several years ago - I think it wasn't saying that they were valid emails, just that it couldn't tell whether they were valid or not - the service was able to detect that the server wasn't rejecting non-existent addresses.)

We ended up unpausing the emails and just hoping for the best. Ended up with something like an 8% bounce rate that eventually fell off our record as our normal sending patterns resumed. Amazon's guidelines say they might cut you off when you hit 10%, so we cut it pretty close.

Re:Check If Email Exists by ganafagol on Friday July 02, 2021 @02:16AM (#27708321)
This is clearly trying to solve a non-technical problem with technical means. The root problem is that AWS cancels you with too high bounce rate. The obvious solution is to talk to some AWS representative to at least temporarily not cancel you after explaining the situation. If AWS does not let you talk to them, then that's where the problem lies, not in some not cleaned up email list.

It's terrible to spend a lot of effort on this kind of tech just because some business partner has shitty customer support.

Re:Check If Email Exists by aembleton on Friday July 02, 2021 @04:33AM (#27709133)
But that would affect all customers of Amazon SES because the IP addresses would score low by other email servers.
Re:Check If Email Exists by kozziollek on Friday July 02, 2021 @09:50AM (#27711504)
Assuming that the customer's servers were hosted by some SaaS like Office, right?

If customer was hosting e-mail servers themselves they wouldn't notify any other mail servers?

Re:Check If Email Exists by golergka on Friday July 02, 2021 @06:28AM (#27709828)
> This is clearly trying to solve a non-technical problem with technical means.

Isn't that the whole point of what most of us are doing?

Re:Check If Email Exists by ganafagol on Friday July 02, 2021 @05:11PM (#27716931)
Is it? In my dayjob I'm solving technical problems with technical means.

World hunger is not a technical problem. You won't solve it with technical means. If you think you can, you have already lost the fight.

Climate change is not a technical problem. You won't solve it with technical means. If you think you can, you have already lost the fight.

And so on, and so forth. Technical means can help solving certain components needed for the overall solution. These are then technical (sub)problems though. For example, how to store more energy in a battery, or how to grow certain crops with less water. But the overall problems are social in nature. People need to understand that world hunger is a distribution problem. That one is easier to solve with certain (technical) tools available, but that won't be enough. People need to understand that we can't use more natural resources than get replenished. Not a technical problem. If only the tools get better, people will find new ways to be wasteful. Etc etc.

Re:Check If Email Exists by bobthebuilders on Friday July 02, 2021 @06:25PM (#27717680)
Climate change is technical problem. Carbon sequestering coal plants can remove all carbon from the atmosphere. World hunger is undoubtedly not a technical problem though.
Re:Check If Email Exists by golergka on Friday July 02, 2021 @06:25PM (#27717688)
World hunger is rapidly being solved by exactly technical means. So is climate change. How do you think these issues are solved?
Re:Check If Email Exists by ganafagol on Saturday July 03, 2021 @10:02AM (#27721676)
World hunger is caused by power imbalances. That's a social/political problem. Whatever technical solution you provide helps worsening the root cause and at best keeps the status quo.

We have observed dramatic technical advances in all fields related to food production in the last 3 decades. We've seen at best marginal improvements in world hunger in the same time, and even that only if you're beeing optimistic with your statistics.

Re:Check If Email Exists by femto113 on Friday July 02, 2021 @12:02AM (#27707696)
If they're checking using SMTP's VRFY command then it's actually considered a best practice for the server to always reply with a 252 "cannot verify" since otherwise it can be used to fish for valid addresses.
Re:Check If Email Exists by sildur on Friday July 02, 2021 @06:18AM (#27709780)
But... I could also fish for valid addresses by trying to send an email and waiting for a bounce...
Re:Check If Email Exists by e12e on Friday July 02, 2021 @07:00AM (#27709987)
Yes, and you could be eg: graylisted. Or server could accept all an silently drop. Or rate limit.

It's sad that VERIFY is basically dead due to spam. In the olden days, you might have been able to use finger - but it's also dead for (among other) similar reasons.

The spam cat and mouse game leads to quite silly situations for benign actors.

For some mx's you might be able to designate some ips as trusted, and do real verify for those.

Re:Check If Email Exists by bkuehl on Friday July 02, 2021 @12:23AM (#27707780)
I can't even imagine wanting to handle managing accounts and credentials for that many users at an enterprise! At that point SSO integration is well worth the money. How did you handle removing access when a user was no longer employed at the company?
Re:Check If Email Exists by moooo99 on Friday July 02, 2021 @03:04AM (#27708615)
Not OP, but also building a similar user system. I can totally understand the motivation to not use the internal SSO. With most companies I know, as soon as you actually connect to their private datasources, you have to do some extra steps to prove how you're securing your platform. This makes sense from the companies perspective, but also introduces a huge technical and organizational overhead for the startup which might be better spend elsewhere if your product does not absolutely rely on SSO
Re:Check If Email Exists by donmcronald on Thursday July 01, 2021 @11:54PM (#27707655)
It seems like the most practical solution to that should be calling AWS, explaining it for 5 min, and getting an exception. Is that kind of reasonable solution no longer possible with the cloud providers being so huge?
Re:Check If Email Exists by toomanybeersies on Friday July 02, 2021 @08:16AM (#27710459)
It's possible, you just need to pay the extra 10% for premium support.
Re:Check If Email Exists by friendzis on Friday July 02, 2021 @03:29AM (#27708748)
At that point would it not be easier to just spin up a VM in customer's infra and send emails directly to their exchange server?
Re:Check If Email Exists by geoduck14 on Friday July 02, 2021 @04:07PM (#27716215)
Ok, so first pass gives me a TERRIBLE idea, that would "get the job done". I'm sure you thought of this and dismissed it:

Keep your "overall bounce rate" low, by ALSO sending out extra emails to confirmed email addresses. Like, for every "confirmation" email, also send a "thanks for joining us" email to someone that already confirmed their email.

Re:Check If Email Exists by kapp_in_life on Thursday July 01, 2021 @11:56PM (#27707666)
I'd hope you'd at least be able to explain the situation to your account manager and get an exception(maybe for that single companies domain?), but I've never used AWS so I wouldn't know if thats possible.
Re:Check If Email Exists by polynomial on Friday July 02, 2021 @12:49AM (#27707898)
account manager? you fancy.
Re:Check If Email Exists by isbvhodnvemrwvn on Friday July 02, 2021 @06:25PM (#27717684)
TAMs you get with the higher support plans can be of some help in situations like these.
Re:Check If Email Exists by justinator on Friday July 02, 2021 @02:39AM (#27708443)
Why did all the employees have email addresses from different services, rather than a single @ourcompany.com address pool that they controlled?
Re:Check If Email Exists by kortilla on Friday July 02, 2021 @02:34PM (#27715076)
This is a reason you need an escape hatch from SES.

In the past when I worked on a system that needed to notify via email we always had a way to change delivery process for certain emails, domains, etc for exactly this reason. This is one of those cases where we would “deliver directly” (i.e. send directly to their mail provider).

Re:Check If Email Exists by killingtime74 on Friday July 02, 2021 @12:46AM (#27707892)
I’ve never worked with emails, could you not send these first emails yourself and not use SES
Re:Check If Email Exists by EnderWT on Friday July 02, 2021 @01:41AM (#27708141)
Have you looked into what SES or other email services provide? Sending emails is easy, while actually getting them delivered is harder. You have to make sure you're not getting flagged as spam, can handle bouncebacks, etc.

Here's one discussion: https://stackoverflow.com/questions/371/how-do-you-make-sure...

Re:Check If Email Exists by toomanybeersies on Friday July 02, 2021 @08:23AM (#27710522)
In GP's case, they probably could'e arranged with the client's sysadmins to whitelist the IP they were sending emails from.
Re:Check If Email Exists by indigo945 on Friday July 02, 2021 @03:12AM (#27708656)
Not getting flagged as spam isn't actually that hard, though. Besides, if you're using SES or some other hosted SMTP service, you still have to set up SPF for your domain, so you haven't even really gained much comfort. The only really useful thing is to gain an ip address with a high reputation, but you can generally get those at any reputable hosting provider as well. Just don't try sending emails from your residential internet connection.
Re:Check If Email Exists by killingtime74 on Friday July 02, 2021 @11:42PM (#27719185)
Ah I understand that part but if it’s going to a big client who knows the emails are coming is it besides the point? I understand day to day
Re:Check If Email Exists by toomanybeersies on Friday July 02, 2021 @08:11AM (#27710417)
Wouldn't the better solution have been to support SSO?