I too always find the Intercom or similar bubbles incredibly annoying, intrusive and distracting, especially when a bot pops up saying something that I absolutely don't need to focus on.
I think the purpose of this bubble is to promote Intercom. It's for sales/marketing people who see it elsewhere first (but never use themselves) and think it's a great idea. Next thing you know, there's an order coming from the C-level to engage with Intercom and integrate it.
The effect of the integration is that some people will talk and it will seem like an improvement - compared to zero interactions previously, no doubt it is. As a consequence of this, nobody bothers to check if it's the best way to integrate the chat functionality.
(Although this is only one of the 1000 things that are wrong on the web today. All sorts of unnecessary things that you need to ignore, neutralize, refuse, dismiss etc. before you can get to the content.)
There was an article yesterday about the lack of increased web page performance to match the constantly growing infrastructure - one item that came up was the fact that many sites have unnecessary features that no one asked for. This quote:
> We first added Intercom when we launched Atlist— it just seemed like the startup thing to do.
Seems like a perfect example of a feature that wasn't born out of demand or a perceived demand - it was just one of the bells and whistles available as freebies in frameworks so why not throw it in there.
Isn't it weird that we have entire companies like Intercom or Rasa whose value add is pushing automated, AI-driven "assistants" onto websites, and then the companies that buy into that entire value add and codebases hacked on by ML experts find that none of it even works better than how it was in 1998?
I could understand the thought process at first glance. NLP is revolutionizing things. Startups want the lastest technology if it's backed by Arxiv papers underneath enough layers. There is the image of people working dead-end jobs in call centers answering questions that nobody wants to spend time on but need to get answered anyways. So why not just make all of that go away, by inserting this ready-made digital replacement into your package.json?
I feel like the point is being missed though. Because even a call center employee working at Comcast is still a human. I think people are getting way too ahead of themselves in thinking code and "neural" networks (what a misleading term) can suddenly automate all of these things that could never be automated before, like dynamic conversations. And even before that the article proves that a single HTML form element is 62% more effective than a VC-backed GPT2-powered "solution" to a problem nobody had. Why require a dynamic conversation at all if e-mail works better?
Maybe there's some property about human interaction with founders or call center workers that's fundamentally impossible to replicate with technology, at the philosophical level, and we'd be better off investing into human resources instead of spending all of this engineering effort trying to work around that fact. Or maybe there theoretically isn't such a limitation, but I have serious doubts that 2020 is the year we'd finally be ready to declare that we've gotten past them.
I see this issue a lot with social media, everyone wants to have share buttons and so on, but few step back and think about if that even makes sense and if their customers even want to engage with them in that way.
There's a perceived need for a Facebook page or Twitter, but how much value does it really add to either party. Especially in commerce where conversion correlates with speed, is it really worth paying that performance cost? As an example, last week I removed a bunch of integrations for a client that were adding no value and adding roughly 1.5s to First Contentful Paint.
Though as a strategy, it sometimes pays off to hurt your competitors by focusing them on features that are useless or cost them money.
I recently tried to buy an electric skateboard ($1000+) and couldn’t get to the add to cart button because of the chat interface. A competitor had a similarly rated/speced board so I bought theirs.
I have used them as a business owner of an e-commerce shop and they were surprisingly useful for our customers. I was reluctant to add one to the site but it was absolutely worth it. I did always wish that there was a way to integrate the chat window in to the page layout, though - I think that would be much less obnoxious than the chat bubble pop-up. I started trying to build one like that but never finished.
As retail stores started to die, some smart cookie said:
"Hey, remember those people who follow you around the store and intruded as you were just browsing? Those sucked! Let's bring that to the web". And lo, chatbots were born.
The worst ones are the embeds that open a small window within the page. Hope you didn't want that content in the bottom-right.
This is an American disease. In most European countries staff might great you, but they will try to discreetly shadow you and once you act like you need a help they will step in to assist.
I absolutely hated the US diners with multiple 'are you ok? do you need anything?' interruptions during single meal. I know why it happens, I just find it really annoying.
Those people followed you around because it led to higher sales overall. Or at least that's the idea. Pushy salesmen tend to move more product than laid back cashiers, even if the former tends to be more annoying.
I worked at a Best Buy back in the day and the name of the game was upselling. My most annoying coworkers were quite successful.
I think some believe this was the reason why Wal-Mart failed in Germany. They brought this "can I help you" with them and German customers rather went to Aldi & co where they could shop in peace.
It's particularly frustrating since the pop up "Chat Bot" is almost always just a crappy search "Bot" which does a worse job than the normal search engine.
You can block the offending domains with uBlock Origin or similar tool. It's a relatively small effort to block the most used ones after which you won't see most of the popups anymore.
I think whether or not this is true for a site depends heavily on the target demographic. Highly technical people with above-average focus would probably see it as an interruption and view it negatively.
In my experience chat buttons are close to useless. First of all that's usually not a chat at all at first, there is some bot with a series of dumb checks and links. And if you manage to get a human on the other side they usually have a timer and if you dare to type your reply longer than 30 seconds they will just drop your session at any arbitrary moment with a comment "user didn't respond".
My internal ranking of contact options is like this:
Email - completely useless, won't even try. Nobody ever replied me on my email sent to the contact listed on a company website. (my address is my first plus last name at gmail, so nothing "weird" there)
Chat bubble - almost useless, will try only when desperate, like when other channels don't respond or gated (e.g. phone number gated by ten layer deep voice menu)
Telegram/Whatsapp chat associated with work number - almost always work and quick, but rarely made available for customers. Or available only after you've already contacted them some other way.
Phone - always works but needs a lot of attention and interruption from both parties
"Always" seems excessively generous here. There are many times I've contacted a company and reach a human who says "sorry, I can't help you, but you can file a ticket on the website" let alone the times I've received "sorry, we're experiencing more than normal callers, please try again another time" without even the option to wait on hold.
Almost every phone support center I’ve called has been hot garbage even after getting past hold. From the automated options menu, which doesn’t have what you’re looking for, to the rep needing to forward me somewhere else.
Huh? That's how I always reach out about something, and probably get a response 90% of the time?
I don't think you're giving e-mail enough credit here. Not sure why you think it's useless.
I detest all the synchronous waiting associated with chat/IM/phone hold, I much prefer to send an e-mail and get a reply the next day whenever possible.
I would much prefer using emails too. My mails are usually to some reseller about specific unmentioned details of the product - like what is the type of power socket, is there a certain feature present in that specific model etc. Anything that is not easily searchable. I assume that regular call operators who probably also read emails don't have an easy answer and just ignore mails altogether.
If it's a large enough company and/or you're asking for something very specific and not something they are willing to give you, they usually won't even bother replying.
"And if you manage to get a human on the other side they usually have a timer and if you dare to type your reply longer than 30 seconds they will just drop your session at any arbitrary moment with a comment "user didn't respond"."
I've personally never had that issue, but my formative years on the internet were on IRC, and as a result when I see a chat text box, I usually send chat messages tersely and fast.
Hey
I could use some help
For some reason I can't log in.
Not sure why?
Looking at the network
the login request didn't seem to send my password.
ect.
I've only recently learned that this is apparently not universal behavior. Two of my co-workers at my company actually fill out large, properly formatted, paragraphs of text in the chat box.
There was a point where one co-worker chastised me for continually interrupting them before they could finish writing. Funnily enough, when I talked to them about using a rapid style of fast terse messages, they immediately adapted to using it. So I think this is just a culture thing. It comes down to what your primary modes of communication are.
So with that in mind, I suspect that when you fill out large chunks of text like this, it looks to the support staffer that you started filling out the text, then moved off the webpage and forgot about it. They were likely expecting a terse sentence.
I have a coworker who does this, and personally I can't stand it. Everyone else on our work slack can take 30 seconds and think through what they say in advance. There is a certain rhythm and quantity to the slack notifications he generates; I've never said to myself "oh that's <x>" after the third notification in a row and been wrong.
If even a single other person sent messages this way, maybe I'd change my notification settings and see everything much later and less often. But since it's just him, he annoys me on a regular basis with this behavior. Just... don't press enter. Why can't you write the same things, without pressing the button that makes noises on 12 devices miles away every time you stop to think? Why???
I also tend to write in separate phrases but usually make an effort to remember and write full message at once and then send. Especially when describing a problem, since I work in QA it's a habit to write every possible relevant info before sending help request.
As an example last time in a "contact bubble" with my cell operator - I had an issue with their app for smart tv, human operator requested me to reboot it, by the time I did it she already hang up. Almost the same happened with my bank, they asked some address, and while I was asking it (like 2-3 minutes tops) they are gone.
PS: I one had a chat with a USA Amazon and that was super helpful, quick and they never hang up even after we were done with discussing a problem, they just waited with open session (I wasn't sure who need to hang first), and asked "do you want anything else?"
So sometimes this stuff works fine.
I would actually put chat bubbles higher than phone. It’s probably the primary way users are supported at my current job (with real humans in the other end) and it works super well. And when I see the Intercom icon, I can almost guarantee that I’ll be talking with a real person (sometimes even the founder) based on past experience.
In the full blog post, they show that the numbers went from:
* Chat bubble: 34 conversations from 8,004 visitors (0.42%)
* Nav link: 45 conversations from 6,622 visitors (0.68%)
It probably performs at least on par with the chat bubble, but it doesn't seem like enough data to say confidently that the navbar outperforms the chat bubble.
I agree that it's a net win to remove the intrusive chat bubble if they're not sacrificing conversations, but the title is overstating the evidence.
How much data would you need for confidence? According to my calculations this is 98% confidence. That feels like 'enough' to make a decision for my small startup.
I'm pretty explicit in the blog post that this isn't meant to be universally applicable— it's just what happened to us.
First of all, kudos for quantifying your results instead of hand waving them. Yes, your results look like a ~60% improvement in conversion rate from the A to the B test, with a p value of 0.02 and a statistical power of around 80% for a two-tailed test. So that's good.
However context is important - at this level of significance you'd expect to see a similarly strong, but ultimately spurious, effect going from the A to the B test about 1 in 50 times.
Since you're not working on something safety critical, that's probably an acceptable false positive rate for you. But generally speaking, and in particular here since the absolute numbers and changes are quite small, I would be wary of trusting such a result. It seems promising but inconclusive. Maybe run a few more tests with disjoint (or nearly so) samples of visitors?
There are a few other things that could possibly confound the result - off the top of my head, your screenshots look like different pages between the A and B test. I'm not sure if that's how you ran the experiment or if you just happened to use two different page screenshots, but that would typically disqualify the result and require another test.
1) Why did the two groups have such different N sizes? If it was intended to be run as a 50-50, a large delta would make me wonder if there was an exposure bias
2) For the baseline rate (0.4%), this test is underpowered for even a 50% change, meaning you will have a high false discovery rate
I'm somewhat of a layman, but I'd wager A/B pages printed 50:50 (by IP for instance) could lead to a rather solid conclusion if ran long enough. On the other hand, eh, chat bubbles suck and you can quite confidently say they don't help, so might as well keep it this way. On a personal note I do feel like I would be much more prone to click a chat request as another menu option than a bubble.
Just as a note: if I ran an A/B test and one bucket had 8,004 visitors and the other had 6,622 visitors, I'd be concerned about the way the test was setup.
There are some valid possibilities to explain the discrepancy, but there are also a lot of possibilities in which the test wasn't setup properly and it isn't measuring the right thing.
When you get a test result that confirms your suspicion, the first thing you should do is challenge those results.
Personally i dont mind the in web chat features (intercom being one) but what I really HATE is when you click on it and it says 'Were not available pls send us an email'. So program the damn thing with your working hours and dont display a chat bubble if there is no one to chat with at that moment in time. If I want to email I will find your 'Contact Us' page. Sorry for the rant perhaps I am alone in this...
I really hate those live chat bubbles, and many people feel the same way. When I see one of those I instantly go looking for alternatives to your site. Sure, you might win more clicks or whatever, but it's also worth considering the goodwill lost.
It's kind of like hiring a used car salesman to do your sales. Sure, perhaps he'll sell more in the short run, but if that's what you're after there are plenty of unsavory tactics out there you can use. Wouldn't you rather customers just had a good, clean experience they're happy to share with others?
> When I see one of those I instantly go looking for alternatives to your site. Sure, you might win more clicks or whatever, but it's also worth considering the goodwill lost.
I think we should keep in mind that HN readers are far from the typical internet user.
I think somewhere along the line the expectations for what people get when they click on a floating chat box changed.
When we started Olark (http://www.olark.com) in 2009 it was novel to show status before you clicked on the chat button, and to float on the screen.
I both were super important, because like you I hated clicking on a box, button, icon, for instant help and having it say "leave us a message" - and I didn't like having to dig around to find some way to contact a business.
Now-a-days many chat/messaging products occlude presence in favor of collecting as much contact information as possible without letting you know if someone is actually there to answer the question --- or if you will be funneled to a bot.
There are a couple of reasons for this:
- the relatively high cost of having a person talk to you.
- the allure of getting leads for low cost (i.e. chatbot < $$ than person)
- it's far easier technically to ignore presence as boot-time or never even implement it.
- the growing lack of user expectation for an immediate response.
I think good human-to-human conversations are essential whether you are starting out or scaling a business. The trick is consumer behavior is changing due to dark patterns.
Readit News
I think the purpose of this bubble is to promote Intercom. It's for sales/marketing people who see it elsewhere first (but never use themselves) and think it's a great idea. Next thing you know, there's an order coming from the C-level to engage with Intercom and integrate it.
The effect of the integration is that some people will talk and it will seem like an improvement - compared to zero interactions previously, no doubt it is. As a consequence of this, nobody bothers to check if it's the best way to integrate the chat functionality.
(Although this is only one of the 1000 things that are wrong on the web today. All sorts of unnecessary things that you need to ignore, neutralize, refuse, dismiss etc. before you can get to the content.)
> We first added Intercom when we launched Atlist— it just seemed like the startup thing to do.
Seems like a perfect example of a feature that wasn't born out of demand or a perceived demand - it was just one of the bells and whistles available as freebies in frameworks so why not throw it in there.
I could understand the thought process at first glance. NLP is revolutionizing things. Startups want the lastest technology if it's backed by Arxiv papers underneath enough layers. There is the image of people working dead-end jobs in call centers answering questions that nobody wants to spend time on but need to get answered anyways. So why not just make all of that go away, by inserting this ready-made digital replacement into your package.json?
I feel like the point is being missed though. Because even a call center employee working at Comcast is still a human. I think people are getting way too ahead of themselves in thinking code and "neural" networks (what a misleading term) can suddenly automate all of these things that could never be automated before, like dynamic conversations. And even before that the article proves that a single HTML form element is 62% more effective than a VC-backed GPT2-powered "solution" to a problem nobody had. Why require a dynamic conversation at all if e-mail works better?
Maybe there's some property about human interaction with founders or call center workers that's fundamentally impossible to replicate with technology, at the philosophical level, and we'd be better off investing into human resources instead of spending all of this engineering effort trying to work around that fact. Or maybe there theoretically isn't such a limitation, but I have serious doubts that 2020 is the year we'd finally be ready to declare that we've gotten past them.
There's a perceived need for a Facebook page or Twitter, but how much value does it really add to either party. Especially in commerce where conversion correlates with speed, is it really worth paying that performance cost? As an example, last week I removed a bunch of integrations for a client that were adding no value and adding roughly 1.5s to First Contentful Paint.
Though as a strategy, it sometimes pays off to hurt your competitors by focusing them on features that are useless or cost them money.
Give me the option, if I need it I will use it. Force it on me I will leave
"Hey, remember those people who follow you around the store and intruded as you were just browsing? Those sucked! Let's bring that to the web". And lo, chatbots were born.
The worst ones are the embeds that open a small window within the page. Hope you didn't want that content in the bottom-right.
I absolutely hated the US diners with multiple 'are you ok? do you need anything?' interruptions during single meal. I know why it happens, I just find it really annoying.
I worked at a Best Buy back in the day and the name of the game was upselling. My most annoying coworkers were quite successful.
No answer, of course.
Brilliant.
A simple "live chat" link in the navigation or footer like they've done works so much better than the intrusive popups.
Automatic popups are like the annoying salesmen in stores asking you unsolicited questions.
(The floating bubble doesn't help as well even if it's more discrete, and I believe not a lot of people associate it with "live chat")
Me: "Go away."
Deleted Comment
Got a phone call from a Sales rep 10 minutes later asking if I need help filling it out or if I was having second thoughts!
I had to sign up anyway (it was for work), but still... that was a shock.
My internal ranking of contact options is like this:
Email - completely useless, won't even try. Nobody ever replied me on my email sent to the contact listed on a company website. (my address is my first plus last name at gmail, so nothing "weird" there)
Chat bubble - almost useless, will try only when desperate, like when other channels don't respond or gated (e.g. phone number gated by ten layer deep voice menu)
Telegram/Whatsapp chat associated with work number - almost always work and quick, but rarely made available for customers. Or available only after you've already contacted them some other way.
Phone - always works but needs a lot of attention and interruption from both parties
"Always" seems excessively generous here. There are many times I've contacted a company and reach a human who says "sorry, I can't help you, but you can file a ticket on the website" let alone the times I've received "sorry, we're experiencing more than normal callers, please try again another time" without even the option to wait on hold.
Huh? That's how I always reach out about something, and probably get a response 90% of the time?
I don't think you're giving e-mail enough credit here. Not sure why you think it's useless.
I detest all the synchronous waiting associated with chat/IM/phone hold, I much prefer to send an e-mail and get a reply the next day whenever possible.
I've personally never had that issue, but my formative years on the internet were on IRC, and as a result when I see a chat text box, I usually send chat messages tersely and fast.
ect.I've only recently learned that this is apparently not universal behavior. Two of my co-workers at my company actually fill out large, properly formatted, paragraphs of text in the chat box.
There was a point where one co-worker chastised me for continually interrupting them before they could finish writing. Funnily enough, when I talked to them about using a rapid style of fast terse messages, they immediately adapted to using it. So I think this is just a culture thing. It comes down to what your primary modes of communication are.
So with that in mind, I suspect that when you fill out large chunks of text like this, it looks to the support staffer that you started filling out the text, then moved off the webpage and forgot about it. They were likely expecting a terse sentence.
If even a single other person sent messages this way, maybe I'd change my notification settings and see everything much later and less often. But since it's just him, he annoys me on a regular basis with this behavior. Just... don't press enter. Why can't you write the same things, without pressing the button that makes noises on 12 devices miles away every time you stop to think? Why???
As an example last time in a "contact bubble" with my cell operator - I had an issue with their app for smart tv, human operator requested me to reboot it, by the time I did it she already hang up. Almost the same happened with my bank, they asked some address, and while I was asking it (like 2-3 minutes tops) they are gone.
PS: I one had a chat with a USA Amazon and that was super helpful, quick and they never hang up even after we were done with discussing a problem, they just waited with open session (I wasn't sure who need to hang first), and asked "do you want anything else?" So sometimes this stuff works fine.
* Chat bubble: 34 conversations from 8,004 visitors (0.42%)
* Nav link: 45 conversations from 6,622 visitors (0.68%)
It probably performs at least on par with the chat bubble, but it doesn't seem like enough data to say confidently that the navbar outperforms the chat bubble.
I agree that it's a net win to remove the intrusive chat bubble if they're not sacrificing conversations, but the title is overstating the evidence.
How much data would you need for confidence? According to my calculations this is 98% confidence. That feels like 'enough' to make a decision for my small startup.
I'm pretty explicit in the blog post that this isn't meant to be universally applicable— it's just what happened to us.
However context is important - at this level of significance you'd expect to see a similarly strong, but ultimately spurious, effect going from the A to the B test about 1 in 50 times.
Since you're not working on something safety critical, that's probably an acceptable false positive rate for you. But generally speaking, and in particular here since the absolute numbers and changes are quite small, I would be wary of trusting such a result. It seems promising but inconclusive. Maybe run a few more tests with disjoint (or nearly so) samples of visitors?
There are a few other things that could possibly confound the result - off the top of my head, your screenshots look like different pages between the A and B test. I'm not sure if that's how you ran the experiment or if you just happened to use two different page screenshots, but that would typically disqualify the result and require another test.
1) Why did the two groups have such different N sizes? If it was intended to be run as a 50-50, a large delta would make me wonder if there was an exposure bias
2) For the baseline rate (0.4%), this test is underpowered for even a 50% change, meaning you will have a high false discovery rate
Deleted Comment
Deleted Comment
Deleted Comment
p0 = 34
p1 = 45
n0 = 8004
n1 = 6622
z = -2.0924
p = 0.03662
So, the improvement is statistically significant at 95% confidence (p < 0.05)
As far as practical significance, that's debatable . . .
There are some valid possibilities to explain the discrepancy, but there are also a lot of possibilities in which the test wasn't setup properly and it isn't measuring the right thing.
When you get a test result that confirms your suspicion, the first thing you should do is challenge those results.
It's kind of like hiring a used car salesman to do your sales. Sure, perhaps he'll sell more in the short run, but if that's what you're after there are plenty of unsavory tactics out there you can use. Wouldn't you rather customers just had a good, clean experience they're happy to share with others?
I think we should keep in mind that HN readers are far from the typical internet user.
When we started Olark (http://www.olark.com) in 2009 it was novel to show status before you clicked on the chat button, and to float on the screen.
I both were super important, because like you I hated clicking on a box, button, icon, for instant help and having it say "leave us a message" - and I didn't like having to dig around to find some way to contact a business.
Now-a-days many chat/messaging products occlude presence in favor of collecting as much contact information as possible without letting you know if someone is actually there to answer the question --- or if you will be funneled to a bot.
There are a couple of reasons for this: - the relatively high cost of having a person talk to you. - the allure of getting leads for low cost (i.e. chatbot < $$ than person) - it's far easier technically to ignore presence as boot-time or never even implement it. - the growing lack of user expectation for an immediate response.
I think good human-to-human conversations are essential whether you are starting out or scaling a business. The trick is consumer behavior is changing due to dark patterns.