I receive a lot of fake user signups on the web tool I publish for free. These users (bots?) sign up using fake email addresses, but putting a captcha does not help much. I send a confirmation email to these users to make sure they exist, but if the email address does not exist, sending that confirmation email hits my account's deliverability (spam) score as it generates a bounce which my email sender provider does not appreciate.

For example, I've received multiple warnings from Intercom that I need to improve deliverability of my email, or they will ban my account. Ironically one of the suggestions is to use confirmation emails - but that's exactly where the problem is for me.

A tool like this helps me to weed out a ton of these undeliverable email addresses to avoid sending emails that will hit my spam score.

> checking for a bounce

So in my case, generating that bounce is exactly what I need to avoid in order to make sure my account remains in good standing.

I think what this is telling you is that users don’t value giving you their email address for your free tools. They know why you want their email address, and they don’t want that, but they do want your tools. Maybe it’s time to charge money for those tools or change models? Maybe not give access until the email address is confirmed? Worth considering I guess. Idk the details. I do this all the time. Ads and malpractice have made giving out your email address risky business. If a user can give you a fake email and get what they want without giving you anything (subject to your ads or whatever you’re using their email for) that’s the logical thing for them to do.

Not judging or anything though I know my tone might seem that way.

You seem to be assuming there's no reason these tools need a login system at all. Assuming they do and they're storing some kind of state or data for the user, how do you handle forgotten passwords if the user can't receive an email? A second free tier for people who can't be bothered to add a recovery email is too much work for a tool that's free anyway.

To the parent: Just send a validation email during the onboarding process. Don't create the account until they validate. And DO rate limit the number of these sent to any given address and the number of signup attempts by IP / fingerprint. Captchas are also good, but just for slowing fakes down.

> Just send a validation email during the onboarding process. Don't create the account until they validate.

This is literally the same thing the person you're replying to suggested.

Is it that risky? Talking, at least, as a somewhat knowledgeable person, which is the target of such web tools.

I usually have no problem using my email address to sign up for things. And yes, I do know that emails will come.

Sometimes they are actually useful and I keep them. Often, I don't care and I just unsubscribe. I some not-so-common cases, it is spam, in which case I mark it as such and I am never bothered by it again.

Sure, it's a bit of extra work. But I would not classify it as risky business.

I was thinking risky in terms of cost/benefit.

The cost for me to give a fake email is very close to 0, and the risk of getting spam emails or who knows what is much higher in comparison so why bother giving anyone my real email address? I mean I do give it out from time to time (you can easily find it if you want) I’m just speaking generally.

Issues I've had is after x years or some leak a ton of mail start pouring in.

You seem to be assuming “a lot of fake user signups” == “few real user signups.” OP only said the first. For all we know they also get plenty of real ones.

Weird. Same exact 5-word opening I was typing. I didn't see your post until I posted mine.

Great minds think alike? Heh

If you dont state clearly why you even need the emails, fake one is the way to go.

If a tool like this works for you, what stops you from sending validation emails yourself?

I'm not the OP, but the answer is typically

1) Time, and..

2) Deliverability

Home grown email systems require monitoring (e.g. deliverability rates, IP reputation monitoring) and maintenance of the system itself which can be a time sink.

The reality is without using SES, Sendgrid, or a similar 3rd party provider, avoiding people's spam folder is very tricky.

A non-marketing example:

We were onboarding a large new client to our SAAS product. This process involved creating accounts for all of their employees (tens of thousands) and sending emails with an activation link. (Where they'd be able to set up their password.)

Our system sends these emails in batches, and as soon as the first batch went out we got an alert from our monitoring system that our bounce rate was surging - high enough to risk a sending pause from Amazon SES. We stopped sending and investigated the issue, and it turned out that the email list we were given was a mess - it included all current employees, but also a huge number of former ones. Just under 1/10th of the emails in our first batch were invalid.

We asked the client to give us a better list, but due to internal issues they couldn't get that to us any time soon. Meanwhile they were breathing down our necks to get these emails out ASAP, and they were a large enough client that our management wanted to keep them happy, so we tried out one of these email validation services. Unfortunately, it didn't work. It turns out that this technique doesn't work for all mail servers. It was reporting every email as valid, even ones we knew were invalid since they'd already hard bounced.

(Edit: thinking back - this was several years ago - I think it wasn't saying that they were valid emails, just that it couldn't tell whether they were valid or not - the service was able to detect that the server wasn't rejecting non-existent addresses.)

We ended up unpausing the emails and just hoping for the best. Ended up with something like an 8% bounce rate that eventually fell off our record as our normal sending patterns resumed. Amazon's guidelines say they might cut you off when you hit 10%, so we cut it pretty close.

This is clearly trying to solve a non-technical problem with technical means. The root problem is that AWS cancels you with too high bounce rate. The obvious solution is to talk to some AWS representative to at least temporarily not cancel you after explaining the situation. If AWS does not let you talk to them, then that's where the problem lies, not in some not cleaned up email list.

It's terrible to spend a lot of effort on this kind of tech just because some business partner has shitty customer support.

But that would affect all customers of Amazon SES because the IP addresses would score low by other email servers.

Assuming that the customer's servers were hosted by some SaaS like Office, right?

If customer was hosting e-mail servers themselves they wouldn't notify any other mail servers?

> This is clearly trying to solve a non-technical problem with technical means.

Isn't that the whole point of what most of us are doing?

Is it? In my dayjob I'm solving technical problems with technical means.

World hunger is not a technical problem. You won't solve it with technical means. If you think you can, you have already lost the fight.

Climate change is not a technical problem. You won't solve it with technical means. If you think you can, you have already lost the fight.

And so on, and so forth. Technical means can help solving certain components needed for the overall solution. These are then technical (sub)problems though. For example, how to store more energy in a battery, or how to grow certain crops with less water. But the overall problems are social in nature. People need to understand that world hunger is a distribution problem. That one is easier to solve with certain (technical) tools available, but that won't be enough. People need to understand that we can't use more natural resources than get replenished. Not a technical problem. If only the tools get better, people will find new ways to be wasteful. Etc etc.

Climate change is technical problem. Carbon sequestering coal plants can remove all carbon from the atmosphere. World hunger is undoubtedly not a technical problem though.

World hunger is rapidly being solved by exactly technical means. So is climate change. How do you think these issues are solved?

World hunger is caused by power imbalances. That's a social/political problem. Whatever technical solution you provide helps worsening the root cause and at best keeps the status quo.

We have observed dramatic technical advances in all fields related to food production in the last 3 decades. We've seen at best marginal improvements in world hunger in the same time, and even that only if you're beeing optimistic with your statistics.

If they're checking using SMTP's VRFY command then it's actually considered a best practice for the server to always reply with a 252 "cannot verify" since otherwise it can be used to fish for valid addresses.

But... I could also fish for valid addresses by trying to send an email and waiting for a bounce...

Yes, and you could be eg: graylisted. Or server could accept all an silently drop. Or rate limit.

It's sad that VERIFY is basically dead due to spam. In the olden days, you might have been able to use finger - but it's also dead for (among other) similar reasons.

The spam cat and mouse game leads to quite silly situations for benign actors.

For some mx's you might be able to designate some ips as trusted, and do real verify for those.

I can't even imagine wanting to handle managing accounts and credentials for that many users at an enterprise! At that point SSO integration is well worth the money. How did you handle removing access when a user was no longer employed at the company?

Not OP, but also building a similar user system. I can totally understand the motivation to not use the internal SSO. With most companies I know, as soon as you actually connect to their private datasources, you have to do some extra steps to prove how you're securing your platform. This makes sense from the companies perspective, but also introduces a huge technical and organizational overhead for the startup which might be better spend elsewhere if your product does not absolutely rely on SSO

It seems like the most practical solution to that should be calling AWS, explaining it for 5 min, and getting an exception. Is that kind of reasonable solution no longer possible with the cloud providers being so huge?

It's possible, you just need to pay the extra 10% for premium support.

At that point would it not be easier to just spin up a VM in customer's infra and send emails directly to their exchange server?

Ok, so first pass gives me a TERRIBLE idea, that would "get the job done". I'm sure you thought of this and dismissed it:

Keep your "overall bounce rate" low, by ALSO sending out extra emails to confirmed email addresses. Like, for every "confirmation" email, also send a "thanks for joining us" email to someone that already confirmed their email.

I'd hope you'd at least be able to explain the situation to your account manager and get an exception(maybe for that single companies domain?), but I've never used AWS so I wouldn't know if thats possible.

account manager? you fancy.

TAMs you get with the higher support plans can be of some help in situations like these.

Why did all the employees have email addresses from different services, rather than a single @ourcompany.com address pool that they controlled?

This is a reason you need an escape hatch from SES.

In the past when I worked on a system that needed to notify via email we always had a way to change delivery process for certain emails, domains, etc for exactly this reason. This is one of those cases where we would “deliver directly” (i.e. send directly to their mail provider).

I’ve never worked with emails, could you not send these first emails yourself and not use SES

Have you looked into what SES or other email services provide? Sending emails is easy, while actually getting them delivered is harder. You have to make sure you're not getting flagged as spam, can handle bouncebacks, etc.

Here's one discussion: https://stackoverflow.com/questions/371/how-do-you-make-sure...

In GP's case, they probably could'e arranged with the client's sysadmins to whitelist the IP they were sending emails from.

Not getting flagged as spam isn't actually that hard, though. Besides, if you're using SES or some other hosted SMTP service, you still have to set up SPF for your domain, so you haven't even really gained much comfort. The only really useful thing is to gain an ip address with a high reputation, but you can generally get those at any reputable hosting provider as well. Just don't try sending emails from your residential internet connection.

Ah I understand that part but if it’s going to a big client who knows the emails are coming is it besides the point? I understand day to day

Wouldn't the better solution have been to support SSO?

The use case is marketing email. Sending to non-existent addresses can radically affect deliverability.

If the address is to a large host, then they will use reaching invalid email addresses as evidence that you are not keeping to best practices. They will throttle deliverability, and possibly reject email.

If your sending to an invalid host, then your mail sending provider (if your using one) may consider you a bad customer and send you through a lower grade of outbound IP addresses.

Frequently new registrations are processed at once as batch imports from another system or from a partner. There is a need to remove these invalid email addresses pre-sending and hurting sending reputation.

Any engineer or product team that uses this lacks ethics and is putting their company at risk for CANSPAM.

If you work at a company that would abuse hibp and a direct mention in CANSPAM you should refuse the work.

This is not for “marketing emails”. This is for spammers.

They’re the same thing, really.

Not really: but majority of spammers think that they are doing marketing.

And majority of marketing is perceived as spam.

we're on the cusp of defining a new internet law here

All marketing emails are spam, most spam is marketing emails.

What grinds my gears, is a company which has "important info" emails you cannot opt out of, but then it is still 95% spam.

My bank tried this during the start of the pandemic. Yeah, that kind of scummy.

"How to get CERB! Also, we're awesome and here are products to help you!"

(cerb == initial canadian relief payments)

Scum.

I've never once received a "marketing email" that wasn't spam.

Used to be in the same camp, before starting my own company. While the border is thin, it is clear, and there is a difference between marketing and spam.

I think you proved the point: when the spammer send spam they think they are doing marketing.

I don't know about you, but I'm subscibed to several newsletters out of my own volition.. Are they spamming me then?

> I'm subscibed to several newsletters out of my own volition.. Are they spamming me then? reply

No, since you signed up on your own volition, obviously they are not unwanted emails.

But I always uncheck the newsletter etc. box and still get tons of them. From legit companies. That's spam, and goes straight to junk mail.

If that was the case, all newsletter senders in the world are spammers too. And all spammers thinking they are marketers is not logically equivalent to all marketing is spamming

Fair enough, if I opt-in it isn't spam.

I have! Expensify. Not a customer, and never have been. Signed up because I was considering using the service. The emails are exceedingly infrequent and delightful to read.

That’s you (and the large part of HN who doesn’t know anything about normal users). When GDPR came into effect, we even had people write us that they were annoyed having to reconfirm, as obviously they still wanted our newsletter.

Some people want to receive marketing emails. The exact same emails are spam to the rest of us. If you default opt-in users and don't make blanket unsubscription from all marketing emails easy (and make the clear distinction [both internally and externally] between emails regarding the existing functionality of products/services that we've already paid for and emails to ask for more revenue from us, which includes copy supported by advertisers), then you're a spammer, simple as that.

Spammers aren't necessarily evil, but definitely annoying, and annoying customers is a good way to lose them. It's easy to drink the coolaid and eat the dogfood when you're paid to like it or have a passion for creating it but potential customers have roughly the opposite incentive and it pays to remember that.

I receive a lot of unread, unwanted email from companies I pay money to every month. It's ridiculous.

> If you default opt-in users and ... then you're a spammer, simple as that.

There is no "and". If you default opt-in users, you're a spammer.

> Spammers aren't necessarily evil

What?

There's some things right on the edge, like Netflix as an example. They're never trying to sell me anything; they've got my $N/month. The emails (that I don't read until now to sample them) are suggestions of shows I might like. User retention mail and promoting the shows they own the rights to are spam, but not all of it is.

My credit union sends infrequent warnings about recent phishing and scam techniques. I never opted in to those emails but they're not useless for everyone and possibly do net good.

I think a good heuristic is to look at the value provided in emails; if the recipient stands to benefit significantly more than the sender (on average) then it's not spam. Sure, my credit union has to deal with less hassle reversing charges or resetting passwords but it's quite a lot more hassle to be a victim of phishing or other scams.

Every few months, or when there's a big disaster, I get spam from Unicef, trying to get me to donate more money.

The emails are undoubtably spam, but (in my opinion) Unicef itself isn't an evil organisation.

I have a common name first year/generation gmail account. The volume of mail I receive that I didn't solicit is mind-boggling. Not just spammers, but it's obvious that people have saved the wrong address in their browser suggestions, and that people give it out when asked for an email addresses at retail. Disney employees, California private school parents, iPhone receipts and apple IDs. Taxes, warrants, bail bonds, social security information. People are f'ing stupid when it comes to email.

Everything should require a confirmation before you assume it's valid. Not because it doesn't exist but because it might not be who you think it is.

Yup, nobody wants those emails. Unfortunately sales people and marketers are excellent liars and they convince higher ups that it's worthwhile.

I agree with all you said. But there are many comments on HN that are "all marketing is spam in general".

If you’re sending to someone who actively subscribed and wants to hear from you it’s not “marketing”, it’s information. Basically a newsletter.

Otherwise it’s spam.

Only “marketers” make that distinction.

I don't know if it's always spam, but it's up the same creek as trying to brute-force your way into someone's heart.

I maintain a small fee tool for creating API endpoints to trigger email alerts for yourself (varmail.me). Since it's very basic and has no marketing, it doesn't get too many users. You have to click a link in an email to even log in, but I still get bots stuffing my login form with known bad email addresses (I've googled and some of them appear in honeypot lists). There is no chance they can get in, but this is bad because my login verification emails look like spam, since a large percentage of them are sent in response to these bots. So I definitely see some value in a way to pre-screen the emails there.

Interesting project, maybe you need a captcha to display and weed out bots before you actually send an email.

Yes, I've considered it. The thing is I really dislike captchas so I tried to avoid deploying one. I do have a hidden form field that gets populated with a secret value in JavaScript, so the bots must be running JS at least to function.

Personally, I do this kind of thing manually if I want to send some positive feedback to people who work at companies I like, but they don't have any obvious emails published.

A recent example was the CEO of Evernote for the work put in to their behind the scenes series although I don't expect anyone to read it of course. People are busy!

I wrote a bit about it here: https://utf9k.net/blog/email-lookup/

Now does this scale? Not at all and I haven't read the email spec or anything like that. It's also handy in a pinch if you wrote down an email but can't remember if it's spelled correctly or not.

The non-interactive nslookup for Windows invocation:

  nslookup -query=mx evernote.com

Edit: Trying out the macOS/Linux invocation on Windows also works:

  nslookup -q=mx evernote.com

Oddly, the first time I did this, I only got IPv4 results, subsequent queries for the same domain, included IPv6 as well.

For email confirmation.

The practice of email confirmation is still widely used, but the change in email deliverable rules has make it a pain to properly validate them.

Even if you are using a 3rd party provider like SES or mailgun, they have a email bounce limit. A considerable number of real world users give fake email address(which is even sometimes encouraged on HN) which triggers those bounce limits.

To fix it, there are paid services but they does not work very well. Fixing it yourself take a lot of engineering time, that is better spent elsewhere.

Providing an open solution to this problem (which is given in the github repo) is a double edge sword. As this gives a edge to spammers who created the problem in the first place.

I just watched a video where someone demo’d GitHub Copilot by writing a comment along the lines of

    // IsValidEmail takes a string and returns whether it’s a valid email address

The AI response was a nightmarish 100+ character regex that made my blood curdle.

I think of email validation like encryption: don’t roll your own, and don’t trust an AI to do it either.

Edit: Here's the regex: https://gist.github.com/cassidoo/6101ef0657665683b787aab5ae9...

100+ is nothing.

This is a regex that validates a string against the RFC822 "Standard for ARPA Internet Text Messages" and it contains 6424 characters.

http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html

This is a Tony the Pony situation almost.

This really seems like it is plagiarizing the regex from somewhere without citation.

It looks like it:

https://twitter.com/xooorx/status/1410776673985994754

https://stackoverflow.com/a/38137215

all AI is plagiarizing something, though. Computers cannot reason, they can just jam stuff together that satisfies whatever the program says.

ML is matrix math (generally) - stuff that you can do by hand, computers are faster at it. As fancy as GPT and co-pilot are, it's all copied from somewhere.

Does it work?

Yeah basically, but it doesn't validate an email address ([email protected]), it validates say, the From: header of a MIME message, which could contain an email address, a phrase (like a first and last name) comments, and all sorts of dumb dumb ideas like having phrases that have embedded comments and phrases that look like email addresses but aren't and on and on and on. This RFC is a trainwreck and whoever wrote it should feel bad about themselves.

>but couldn't that be determined by just checking for a bounce when a message is sent to it

The fewer times you bounce, the better your chances of not being marked as a spammer.

This could also be useful for an ecommerce site, where you want to be able to easily contact the buyer if there's a delay, address correction needed, etc. People typo their own email at a rate that's surprising.

I use email validation via MailGun for exactly this purpose with a productized service business. If we don't have a good email, then we can't deliver the service once complete. That leads to angry customers, even if the issue was a typo when they created the account. Easier to try and catch it during signup rather than at the time of delivery when emails start bouncing and you have no way to get ahold of the person.

Mailcheck is also helpful to catch things like "[email protected]" and other common typos when the user types it in:

https://github.com/mailcheck/mailcheck

Fork modified for React: https://github.com/eligolding/react-mailcheck#readme

I somewhat lol'd when the demo allowed "[email protected]" just fine. Guess kickstarter isn't using mailcheck anymore. Looks like it's an open issue:

https://github.com/mailcheck/mailcheck/issues/179

Oh, yeah, that's not good. Not sure why they didn't just repoint to the suggested https://codepen.io/ianneub/pen/BajKKBy demo.

I bought a new car a few years ago and I was unable to enter my .life email address into the SiriusXM registration form in the car. Had to use a .com address :(

I specifically bought a cheap .org since some companies have trouble with lesser known TLDs

I have a hyphen in the domain (.com) I use for personal email (it's shorter, just my initial and surname) and I still have had problems in some places with that - it's so frustrating.

I have a .red and a .me and I have never not been able to sign up.

.life has four letters. Lots of terrible attempts to validate email addresses reject TLDs that aren't 2 or 3 characters.

>Lots of terrible attempts to validate email addresses reject TLDs that aren't 2 or 3 characters.

which is sad because .info has been around for 20 years.

pipesandcigars.com let me sign up with a .miami but now I can't login because it says it's invalid. My ISP frantically called me up after starting service because they took all my info down and then the system didn't like the domain. My utility company kept sending me requests to go paperless and I had to call them to get an answer on why that happened since I had accepted the paperless prompt, the issue was the domain.

It's nice to have but I've had quite a few headaches with it

I had a similar experience with LG. You need a developer account in order to install applications on webOS TVs. I successfully registered on the LG Developer portal with something like [email protected], but I couldn't log in on the TV with that email address. Had to register again without the plus signs.

I had something worse(?) happen with one of my accounts for a hotel chain. I entered [email protected] and they just stripped out the + leaving me with an account that I couldn't use unless I made another account at the email provider.

Same, got a .email tld, and regularly get rejected.

Forget about AI and self driving car, we can't even get email validation right in 2021.

I am not sure there is any clear legitimate use for deep validation (beyond checking for syntax). If you are providing a valuable service to a customer, wether it's a catch-all or free mail shouldn't matter to you.

Now, if you are buying lists here and there to spam the hell out of it, the bounce rate would flag you very quick and you'd need to find another smtp provider every week. This service would be your life line.

Ideally you don't want to send an email only to find out it doesn't exist, in terms of marketing.

Do that too often and servers can start black listing your domain/IP because it looks like you're "scanning" for available email addresses.

Similar experiences here. I have [myfirstname]@[mylastname].party as my primary email. In meatspace it’s a curiosity, and people are sometimes incredulous. Online, I’m routinely given some variant of “enter a valid email address.”

We have been auditing legacy user accounts to begin a mapping to Zendesk, and Zendesk allows mapping to ‘organizations’ via email domain.

For some of these legacy accounts, the registered email addresses may have typos.

So, even a basic DNS check against the existence of the domain’s MX record is helpful.

Any of the ‘suspect’ email addresses can then be further evaluated by a human, and then removed or fixed.

Any kind of technical email validation is horribly unreliable at best and there is no chance that this GitHub repo is going to work any better.

As a result the only place such a service is useful is for someone who has a ton of low-value emails they don’t trust, and they don’t want a ton of bounces when they hit send (which risks losing your send privileges with pretty much high-volume email platform).

So they run all their emails through a service like this, and only send to the ones marked valid. This excludes a ton of emails that actually are valid, but failed the check (false negatives). But that’s ok because the emails were low-value to begin with.

If this sounds like a spammy operation… bingo. Technical email validation services are really only useful for people who are doing things like buying email lists from commercial providers, harvesting emails from sites like HN, or forcing people to enter an email address to do basic things with a free service.

Financial/fintech companies use services like these for fraud-detection on account opening. While validating an email is by no means and exhaustive and conclusive signal to classify a fraud/genuine user, verifying the validity of new customers's email addresses is a big help.

If you use Amazon SES and have a high bounce rate they will just block you. So, Eve could flood your registration with invalid emails that would cause bounces, that would stop all your emails from getting delivered. A DoS you can only recover from by switching to a different email provider

> Please hire me, SiriusXM.

Given their questionable business practices, their customer service dark patterns, their dated and awful UX, and their inevitable demise to much more popular streaming services, you’d be best to stay far away.

The service I work on emails users(B) after their email address has been input by another user(A) of our service. This would allow us to check that the email is reachable before we attempt to send the email. If it is not we can ask user(A) to check before they proceed.

I sometimes use it for sales. Looking up a company on LinkedIn, then finding the name of a person I want to each our to but am not sure about the email. Then I could try the classic first name.last name @ domain to see if it’s worth sending an email

I use to have a active product DearEle.com an email reminder service. There was a feature to cc everyone in email, which is used by spammers to send junk products to everyone. The source emails were all fake. Email check prevent fake emails.

> Please hire me, SiriusXM.

Honestly not where I thought that rant was heading ;)

I would love to clean up my addressbook removing no longer valid email addresses for contacts.

I’m tempted to add a spam detector rule that runs all From addresses through this.

Marketing?

It has honeypot detection. It's built for spamming people. There's literally zero reason to avoid honeypots otherwise

If you're asking for an email address in exchange for some piece of content (e.g. a soft paywall), I find that it's reasonably sensible to filter out the absolute spammiest-looking of email addresses in order to collect more real email addresses and prevent people just rolling their face across the keyboard and calling it a day. (It can also help with bots a little bit, although most bots are smart enough to use <random firstname>.<random lastname>@gmail.com or something). The best way to do this is obviously double opt-in, i.e. actually emailing the address in question. Most businesses believe (probably accurately; I don't have the stats) that this creates too much friction. I've used Sendgrid's email verification service in the past, and the actual scoring is basically garbage but if you set the threshold really low (e.g. reject all emails with a score between 0 and 0.1) then your precision is really good (almost every email address you reject is invalid, as measured by actually sending an email to them and the email hard-bouncing or being otherwise undeliverable). You let a decent number of bad emails through but it's an improvement over nothing.