To the extent I am capable of commenting publicly on this, I have not ever seen solid numbers which make gradual engagement (technical term for what you're talking about) look like a big win for the company, and to the extent I have seen numbers on gradual engagement, it has been the opposite of a win.

The only numbers I can comment on publicly are the gradual engagement test for BCC. That was my worst Product idea ever, hands down. It 10xed customer support costs because users expected their accounts to magically work (even when, e.g., accessing from home and school and why did your computer delete my cards you just ruined Halloween for my class you monster). The conversion rate for guest accounts to paying accounts was 2. Not two percent. Two accounts ever, in six months, actually saved their email addresses and then, some time later, paid for the software. (This was at least partially because I did not roadblock users at any point and e.g. ask for an email address with a greyed-out Skip option, which is a virtual necessity to get gradual engagement to work. #microUXtip) I have no email addresses for guest accounts nor permission to contact them, something which I have in abundance for trial accounts, and I can tell you that is worth a good deal of money over the long run.

Optimizely and Stripe are, indeed, the two YC companies that I can think of which do gradual engagement. The mafia are very generous with what they know, have an active mailing list, and are universally capable of knocking out a gradual-engagement enabled account system within a week or two. What does this suggest to you as to internal sentiments regarding gradual engagement?

If you want to do something like this, use a post-signup fully capable account for a product tour. Product tours, as compared to gradual engagement, have a) vastly easier UX problems to resolve and b) empirically are full of win. That video I have about the first run experience explains this in a lot of detail.

P.S. Global endorsement of everything btilly says on this thread.

This might not work for all web services, but I think it works well for stripe. I honestly doubt I would be using stripe today if they didn't have gradual engagement. When I was in the market I found stripe during my "research" mode. I stumbled into their site and as I "learned" about their product I was also "building" my account. In the end I felt little friction and I integrated stripe payments into LinkPeek.

tl;dr Gradual engagement success appears to depend on the type of person or market a service targets. Pulling from BCC we could deduce that gradual engagement may not work well for teachers, or that your implementation did not work well. I know from experience that this technique converted me from a spectator to a consumer. My personality might not be typical but I generally don't like pressure and often have a bad case of buyers remorse after making purchases. I didn't feel this when "researching" on stripe and I ended up a customer.

Yes, I know, you're hoping someone else will know the answer. But if you don't develop the habit of testing, you won't be in the habit of testing, and you won't be testing things that are unique to your business.

This is what patio11 keeps on complaining about. When there is a discussion of A/B testing, everyone says how great it is. But over 90% of you simply aren't doing it. And in a thread like this you can see that people really aren't doing it.

So, while it might be nice to a/b test everything - sometimes you just want to start out on the right foot and then test from there.

Hmmm.... I'd caveat that advice a bit. I'd have to be pretty sure that the company I'm copying is facing the same sort of problems I am. Especially since, after N instances of being wrong, I've learned that I'm often quite bad at figuring out whether company Foo is facing the same problem as company Bar!

Especially when you have limited resources and solving one problem means that you're not solving another.

I've seen far too many instances of people coping what Amazon or Facebook do and it not helping in any way because - well - they're not Amazon or Facebook.

Personally I'd try and spend a more time talking to / observing actual users if I don't have the numbers for more quantitative techniques. Something, sadly, most companies seem to do just about as much as they a/b test :-/ That'll give me some strong hints on what the most significant problems are.

Deploying a demo requires a certain amount of effort.

We're bootstrapping so I'm careful with our efforts.

I'm only asking around to figure out if for most people that did it, the effort was worth it.

If you have any traffic at all, the habit of A/B testing will quickly pay for itself.

I see your point though and do agree that once the demo is in place, A/B testing will make sense.

A/B testing is a statistical technique for teasing out the significance of changing one thing, in a world where lots of things are changing all of the time. If you just change stuff and throw it out there, you can make an educated guess about what mattered, but you don't really know and have no idea how good the data actually was.

If you want to learn more about what A/B testing is, and how to do it, http://elem.com/~btilly/effective-ab-testing/ is a good starting place.

It's like bait. Once they begin to use your product and put their data/actions into it, you simply remind them to save it with with their email (and password if necessary to save it). Their investment in trying it is a much stronger motivation to sign up than some features list with a promo video ever could be.

EDIT: Not to mention as we move more towards desktop style web apps, it will be the natural progression.

IMO it splits the difficulty into two steps (starting to use the tool, then sign-up up / paying).

For a product that is easy to get started, it seems to make sense.

Thanks for your feedback!

It wasn't so much a demo as it was an instant trial.

https://manage.stripe.com/register

Note: see the skip this step link

That's what I would have suspected as well. Weeding out the users that are not a good fit is very interesting indeed!

I suspect too that asking for CC info straight after the demo would be less of a repellent, once the visitor has actually tried the app (and it will probably lead to less chargeback, too).

This is the real benefit of free/no signup. If you can weed out your customers that aren't a complete fit your support costs drop dramatically.

Did you run it as an A/B test to be sure that the eventual rise in people paying wasn't because of some other change that you made which you're not thinking about?

There's a 30-day period before those people have to decide to pay, so I didn't run a full A/B test during that entire period because we were clearly getting more qualified signups, so I don't perfect data on exactly how big the increase was.

One issue with a/b testing something like this is that the cost to do a good online demo can be non-trivial - and judging whether it can be a success can be non-obvious without doing a good online demo first. Egg. Chicken. Chicken. Egg.

If I were me I might go about it something like this (assuming that building the demo feature isn't trivial - in which case just a/b test it :-)

0) I'd already be doing quick'n'dirty usability testing and talking with users regularly. This is even faster at showing up low-hanging fruit than a/b testing in many situations. I'd want some hint that the kinds of problems demos solve (e.g. not understanding what the product did) were showing up and there were no bigger fish to fry.

1) I'd guesstimate how much time/money it would take to build a good online demo.

2) I'd guesstimate how much of an increase in final conversion it would take for it to be worth while taking that time (considering that I could be building other features).

3) I'd add a "demo" button. I'd have it link to a "demo coming soon" page with an email signup so I could grab some "demo wanting" users for future quizzing. I'd split test that to see if anybody clicked on the demo button.

4) I now know what percentage of people are even vaguely interested in an online demo. So I have an upper bound for how much of an increase in conversions it could make. If it's worse than the number I came up with that would make it worthwhile - I can stop now.

5) I also now know whether "demo seeking" potential customers convert better/worse/same than non-demo-seeking potential customers.

6) I'd probably go talk to the demo-seeking users I gather on the e-mail list no matter what - to find out more about why they wanted the demo. Did they not understand the product? Were they looking for a particular feature? Did they want to play to figure out whether it was worth the money? Something else? Maybe you can fix their problem without building the demo. Maybe everybody wants to know whether it has FooFeature and you just need to add a bullet point to the home page.

7) From the previous step I now know more about why folk want an online demo. I'd try and build as little as possible to fix that particular problem. Stick that on the demo page - and see if we got any more conversions in the next round of testing.

Repeat until no longer improving :-)

We also notice that our users sometimes create an account, play around a bit, but then leave and come back again a few days later because they were in the middle of something. I'm curious what the effect is of having no way of contacting them, or leaving a reminder in their inbox of where to find back the website. Maybe just asking for an email address would help.

In any case it would be interesting to see some more data. So far we've just been using sign up forms, but always try to keep them short and 'friendly'. We're definitely considering giving this a try, so hopefully we have some meaningful data to show in a while.

If I present the visitor a thing or a service he can buy, is it bad that I need him to sign up separately and confirm his e-mail before he can actually press "place order"?

I guess it sounds bad from how I'm saying it, but my concerns are the security (confirmed e-mail = you can reach the person) and work investment (creating a form that would do two logically-separate things at the time is more work).

So what do you guys think? Should I let people do anything without a confirmed e-mail address and work on merging sign-up, sign-in and place-order forms together?

The company was Best Buy. Look at the stats which show that requiring registration essentially never helps. About the only time it would is for (very?) frequent visitors.

You don't need "security" for orders other than making order ids non-guessable. For customer service people can give their order id and other details if necessary (eg zip code, name).

I disabled the limits for unconfirmed users and will create a merged form as soon as possible.

The development flow looks like this to me right now: First I built a strictly REST/model-centered back-end, and now I have to smooth the corners between user-friendliness and those standards. Interesting observation.

(Also, if anyone is curious what I'm building, it's https://artistsnclients.com/)

On the one hand, you build a bigger mailing list + extra friction.

On the other, smaller mailing list, but more users get to try your service.

I recommend a split test.

it's taken us a while to get mixpanel installed so we can measure this properly.

our key metric is not how many people we get logging in but how many people move through our product and send out an email newsletter. we'll be a/b testing this with a "sign-up" button and an "open" app.

we should compare findings :-)

In case of LinkPeek the demo was basically just a free version of the SaaS product. It only required one webpage to be built and took on the order of about 2 hours to create.

If you are interested in seeing the demo go to linkpeek.com and click try now.

Thanks for you link as well!

However, at the same time if the demo looks crappy enough, chances of me signing up decrease exponentially.