As their largest competitor, I still hold that his is the right attitude. Everyone in this space had to do massive scaleups, and some bumps inevitably happen.
We have identified the issue causing users to be unable to authenticate to the Zoom website (zoom.us) and unable to start and join Zoom Meetings and Webinars
Even with a distributed architecture, auth/login flows are often some of the harder problems to horizontally scale without introducing inconsistencies of behavior.
I know this isn't the right forum, but where does one report bugs to Webex? It never lets me enter a full 7 letter tld in an email address, if you care.
Well now this would get all the infrastructure guys enraged when they hear that they (Zoom) proudly rely on Oracle's infrastructure, to scale to millions of users [0]
Let's wait until the first person asks if they use 'Kubernetes' yet.
Unfortunately, the same incentives that keep us from both developing or government requiring/legislating that the underlying capabilities, i.e., video/audio conferencing, be interoperable (i.e., you can call people on GoToMeeting/Teams from Zoom, or vice versa); will also likely prevent them from putting out any good information on these events. Video conferencing is yet another example of a monopolist trying to take the whole market.
I realize that the old internet of distributed services and actual competition is likely something those who grew up in the facebook world will not even have a conceptual understanding of, but all of the monopolistic type centralization of services (everyone on Zoom, everyone buying from Amazon, everyone using Uber, everyone getting information through Google, etc.) is immensely dangerous, to a degree that simply cannot be overstated. Its digital, literal monoculture, where one event can have devastating cascading impacts.
In many ways the current virus situation is precisely that, a devastating impact due to a monoculture created and pursued by globalism, where diversity is destroyed and everyone is an equal and controlled human entity. The very crowd that is always demanding "diversity", is also the crowd destroying actual diversity by destroying differences and trying to turn us all into the same, equal thing.
I was impressed at how well they were able to scale up when the 'rona hit. I'd still like to know how Zoom specifically rose up so fast, given that it's one of a billion video conferencing tools. Was it because they offered many-to-many video chats of over whatever Skype offers for free? I don't even know what Zoom's business model is.
In the last startup I joined, back in ‘16, they had two things going for them:
- It was very easy to get people to get on the same zoom call as everyone else
- It had the best quality for video conferencing.
At the time, Hangouts was unreliable. Slack had not had a video chat feature (and it was crappy when it first came out). Goto Meeting had poor quality video and was more difficult to get everyone together.
We also thought that Zoom had encryption. (Turns out ...). There were very easy screen shares.
The startup I joined is remote-first, though it did not start that way. After I joined from out of state, we just all started doing that. We used Zoom a lot.
In short, I think Zoom got as big as they did because they addressed enough of the pain points in the end user experience.
All other video conferencing solutions suck! Somehow Zoom figured out how to solve feedback/echo elimination, audio latency, video latency, and video quality.
It can't be overstated! They won over with sheer quality. I suffered through evaluating several video conferencing products over a few month period, and Zoom was the only one that worked properly.
The created a product that works, setting themselves apart in a sea of garbage.
Every conference call or presentation or whatever is held up by the stupidest user with the shittiest laptop from their incompetent IT department.
Zoom had the lowest friction, most idiot proof solution while still being decent quality and having enough useful features for classes and meetings, etc.
(as it turned out, some of those low-friction design choices also had some negative security and integrity consequences)
Using zoom was frictionless, specially pre-Corona Times.
- Keyboard shortcuts for most common tasks (new meeting, copy meeting id)
- Just send a link to invite someone to a meeting (no need to wait for the "e-mail" to arrive, or to download an ics attachment with the meeting details).
- Just click on the link to join a meeting
- Just click share to share screen (no more "can you make me the presenter" conversations)
- Super easy way of asking remote control (no "click to take remote control" message)
- Easy to use audio testing feature
- Meeting ID on top of every meeting so it was easy to see and share (this was been deprecated though, after someone from the Uk gov't published their meeting id on social media)
- Added everyone on your domain name as a contact (this was disabled as a default setting after this happened with an European ISP e-mail addresses)
While competing solutions still required meeting id, meeting password, typing your name and e-mail, "taking the presenter role" before screen sharing, downloading a temporary executable file, showed weird gray boxes during screen sharing.
Other nice features where; Multiplatform Win/Mac/Linux/Browser based, Android, iOS, lots of dial-in phone numbers, even in my third world country (!), also audio/video quality was more stable even with our third world internet.
Did I also mention their executable clients where super responsive.
You need a paid account for meetings over 40 minutes with more than two participants. If you're a business or school, you're likely going to want a paid account.
I'm told it's much better than most others. I haven't really noticed a dramatic difference, but that seems to be the consensus.
I don't know if this is the case with WebEx, but in my experience working in enterprise software, a handful of huge corporate clients that generate an outsize percentage of revenue pretty much dictate what gets attention. They're generally more worried about interoperability with their environments and minimizing their own internal training/support expenses than having an optimal product.
For them, something mediocre but gaplessly sufficient is preferable to something being mostly excellent with a few big caveats.
Not a fan of WebEx.. but Enterprises often have very different requirements that are deal breakers..
For example, it has to run as a on-premise hosted service, or has to tie into an arcane security environment which hasn't been updated in a long time where bandwidth is already an issue.
Imagine running that many copies of custom software in one environment, each with their ideal solution
When the rest of the whole technology industry, giant corporation after giant corporation, treated video chat as an opportunity for vendor lock-in, service up-sell and competetive wedge...
Zoom just made a tool that could be used by everyone, everywhere, for free. And they saved us.
Pretty much the only time I've used my cellphone for work for months was when my power went out for a couple of days because of a storm. (We don't use Zoom--Meet and Bluejeans--but same idea.)
Not because they have the best quality, most secure, or most reliable product, though. So rather than cut them some slack and wait, this is a good opportunity to explore alternatives.
That's debatable. Perhaps not the most secure, but possibly the best quality and most reliable (to this point, at least). Depends how you define quality, but the ease of use and high quality video are perhaps the leading factor in their huge success.
I’d argue most reliable for the most people, however. Even if that’s only because many people already have the client installed and are accustomed to it. It’s not completely random that zoom is so popular.
Our school district's entire eLearning plan relies on Zoom for full day live streaming. Basically every teacher of every class live streaming all day. Not sure how many other districts and colleges had similar plans.
Fwiw, most school districts also already have either MSFT Teams or Google Classroom licenses so switching away from Zoom shouldn't be too problematic for ad hoc lectures.
My district started with Zoom in the spring, then switched to Meet when the Zoom security issues got hot, and ultimately settled on Webex for the fall because it was the most like Zoom, carried enterprise support, and because Meet is still missing far too many necessary features for teachers (breakout rooms, raise hand, etc).
Yes, some schools are planning on this. Breaks for lunch / recess.
Too many schools are taking the approach that our learning wasn't structured enough in the spring, so we are going to the opposite extreme.
It's going to have a negative effect on the children and caregivers.
We have non-technical people on tight deadlines trying to reproduce brick and mortar on a computer screen. It's not going to end well. I feel for them, they are trying their best.
I have a child starting kindergarten. 1/2 the class is attending in person M and Tu and the other 1/2 is via a chrome book and Google Meet. Wed all kids are attending via Google Meet (the classroom will be deep cleaned Wed). And then Th/Fr is like M/T, but the group that was in person is now at home and vice versa. Then they deep clean again on the weekend. (Their day is 6 hrs (less recess and lunch)).
Not only is it going to be challenging for the kindergarten kids to stay focused on their class via chrome book, the SAME teacher is managing the kids in the classroom and the kids online. The kindergarten teachers each have an assistant too (as they did before Covid-19) so that helps a bit.
I am fortunate my wife stays at home with the kids. I can't imagine what it will be like for parents of kindergartners who both work, or parents with special needs kids or parents who don't speak English. I am supportive of social distancing in these times. But its going to be tough on the kids.
There are many districts that expect kids, including early elementary, to sit and stare at screens a majority of the day (with breaks for things like lunch).
Yes, my daughter is doing remote this year, but the majority of students are in class. So she has to watch a live video of the lecture. The district doesn't want to record the videos since they think students won't pay attention. They also periodically take attendance.
It seems to vary. Our district has our kindergartner in front of a screen for about 2 hours (not continuous). Then they give us activities to do with our kids, at our convenience, that we submit via an app. The teacher reviews and comments on it. There's around 7 per day, but they aren't too difficult for our 5 year old: write 10 T's, read them a book, collect then count some stuff, etc.
Our school is 100% distance learning. When they were going to allow distance learning and in person, they were planning to have completely separate classes for this so they could better optimize for your particular medium.
Sadly yes. Palo Alto has been doing this for the last couple of weeks (grade 7-12; dunno about younger). 10 minutes between classes; 30 for lunch. I don’t think it’s sustainable.
One improvement: the high schooler has some sessions with 90 kids (!!). So at least they can use their phones to goof off and text each other which they couldn’t do IRL.
There’s no easy answer to this, but I really wonder about the effects of this on kids, especially the younger ones. Kids, even liners, are pretty gregarious by nature, and there’s a lot of important physical contact among the little kids (and, for different reasons, teens :-)). Seems like there will likely be some developments issues down the road.
I'm surprised there hasn't been noise about schools wanting Zoom to reinstate their eye tracking code so the teacher knows when Little Johnny isn't paying attention.
I made a web app to help my wife with our pre-K kids 7 zoom classes, it has made a huge difference and he is able to connect by himself most of the time. I have no idea how non-technical people can cope with this, its still insane but at least manageable.
Say more (school admin considering deploying a custom webapp of the sort for our parents, where links/etc. are more easily laid out and managed for our audience).
That's interesting. My side project, StatusGator, monitors and aggregates status pages and I've been wondering recently if we should target the education... and how to.
We've had a handful of school district customers sign up recently and use us to build aggregated status pages or send notifications to teachers and administrators when Zoom (or Blackboard or whatever) is down. So there seems to be a use case there beyond the "DevOps" market we currently cater to.
This issue only affects joining Zoom meetings through the web client. If you join using the desktop or mobile client to the meeting ID directly, you should still be able to join a meeting.
Doesn't do me a lot of good then since I can't run the Zoom application on my only laptop :/ (corporate policy). They've really screwed up here by de-emphasizing the web client in favor of the application.
This flow (clicking on a link, opening on their website, and opening the app) is what's currently broken. Opening in the app directly bypasses the problem for this reason.
The problem is that a ton of the people hit with this are students on Chromebooks. Our district standardized on Chromebooks and issued one to every student. Assuming the teacher isn't on a Chromebook this would likely mean about half the class still couldn't get into the meeting.
When everyone started working from home, Rogers in Toronto (1 of the 2 cellular networks) basically crashed on the voice side.
It was painful to get them to admit anything was wrong, until I asked if this would impact 9-1-1 calls too, then they admitted an issue with full-digit dialing (aka: 10 digit dialing).
Even units that were resistant to switching from the conference line migrated over.
I wonder how this afternoon will go if this continues.
Originally the Zoom site listed meetings and webinars as operational during the time the outage was happening. Now they are listed as 'Partial Outage.' I would have thought something like an outage is detected automatically and the status page updated automatically.
And the error message is strange:
>There is no accout for zuora account id:2c92a00d6ff0e970016ffbde74ae767c (3,201)
It seems that everyone having issues is seeing the same error.
Building a status page is usually an endeavour taken upon when it is a business requirement, e.g. you have clients with whom you have signed an SLA with.
Unfortunately this provides a perverse incentive to NOT report an outage, as even a small outage would be a contract breach for a "many nines" SLA, whereas many small breaches added up over a month could also breach a contract for (relatively) smaller companies. If the client doesn't notice, then it doesn't count.
Anecdotally, the only status pages I've seen that update automatically have no monetary incentive to maintain the page.
"even a small outage would be a contract breach for a "many nines" SLA"
Ain't that the truth !
I remember working with organisations that had Verizon leased lines.
100% SLA the Verizon sales reps would say, normally accompanied with a sheet of paper that said "100% SLA" in big bold letters.
Of course the line would go down. Some manager would the come to my desk, thumping his fist and talking about the 100% SLA. I'd say "sure, but did you read the small print".
In all my years, I don't think I ever saw anyone get a SLA payment from Verizon.
It's subscription/billing management, and it doesn't really do payment processing itself, but rather you might configure Zuora with something like Stripe/PayPal/etc.
Gitlab had an outage today related to Zuora going down. My guess is the backends directly call to these APIs and they don't have any form of caching or storing the info from these APIs elsewhere.
Before the whole virus situation started, I had never even heard of Zoom, and now it seems it's everywhere. What happened to Skype? I remember it being the thing I was forced to use during pretty much every client meeting back in the day.
Microsoft bought it and gutted it, it has terrible usability now. I think when the pandemic hit the most usable system was bound to get tons of traction quickly, and I guess that happened to be Zoom.
Trends become Buzzwords. The less syllables the better... Mix that with some perfectly timed marketing and heaps of free licenses, and Zoom becomes ubiquitous within a few months.
For many schools, zoom is the backup plan. Covid caused the in-person outage.
This isn't their area of expertise, and asking teachers to migrate to the backup and inform and assist their students to get online during an outage of unknown length isn't reasonable.
Snow days in 2020 will be replaced with Zoom Outage days.
If Zoom had an outage in early april, that would be legitimate. However it's been 5 months since schools first closed, and we've known for most of that time that this would be a long term issue. Do you really want the person who can't figure out two meeting programs over the course of several months to be responsible for your kid's education?
I made a joke about this earlier in the morning on my team's slack: "I wonder how many systems admins are going to wake up later this week and find Jitsi installations dotted throughout their infra" when the more tech-savvy users try to 'self-service' their video conferencing
Most institutions didn't even plan to go into Zoom, assuming they have some well-thought-out redundancy when they don't even have a well-thought-out primary is misunderstanding how disorganized most institutions are handling this whole situation: Most did nothing over summer to prep.
Sure. But everyone is already flying by the seat of their pants here. That there isn't a backup video conferencing system set up isn't overwhelmingly surprising to me.
(plus, training out of practise professors on two video conferencing systems? No way that would end well)
Readit News
We have identified the issue causing users to be unable to authenticate to the Zoom website (zoom.us) and unable to start and join Zoom Meetings and Webinars
Even with a distributed architecture, auth/login flows are often some of the harder problems to horizontally scale without introducing inconsistencies of behavior.
I wish 'em a speedy recovery!
Let's wait until the first person asks if they use 'Kubernetes' yet.
[0] https://www.oracle.com/customers/zoom.html
I realize that the old internet of distributed services and actual competition is likely something those who grew up in the facebook world will not even have a conceptual understanding of, but all of the monopolistic type centralization of services (everyone on Zoom, everyone buying from Amazon, everyone using Uber, everyone getting information through Google, etc.) is immensely dangerous, to a degree that simply cannot be overstated. Its digital, literal monoculture, where one event can have devastating cascading impacts.
In many ways the current virus situation is precisely that, a devastating impact due to a monoculture created and pursued by globalism, where diversity is destroyed and everyone is an equal and controlled human entity. The very crowd that is always demanding "diversity", is also the crowd destroying actual diversity by destroying differences and trying to turn us all into the same, equal thing.
We also thought that Zoom had encryption. (Turns out ...). There were very easy screen shares.
The startup I joined is remote-first, though it did not start that way. After I joined from out of state, we just all started doing that. We used Zoom a lot.
In short, I think Zoom got as big as they did because they addressed enough of the pain points in the end user experience.
It can't be overstated! They won over with sheer quality. I suffered through evaluating several video conferencing products over a few month period, and Zoom was the only one that worked properly.
The created a product that works, setting themselves apart in a sea of garbage.
Zoom had the lowest friction, most idiot proof solution while still being decent quality and having enough useful features for classes and meetings, etc.
(as it turned out, some of those low-friction design choices also had some negative security and integrity consequences)
Using zoom was frictionless, specially pre-Corona Times.
- Keyboard shortcuts for most common tasks (new meeting, copy meeting id)
- Just send a link to invite someone to a meeting (no need to wait for the "e-mail" to arrive, or to download an ics attachment with the meeting details).
- Just click on the link to join a meeting
- Just click share to share screen (no more "can you make me the presenter" conversations)
- Super easy way of asking remote control (no "click to take remote control" message)
- Easy to use audio testing feature
- Meeting ID on top of every meeting so it was easy to see and share (this was been deprecated though, after someone from the Uk gov't published their meeting id on social media)
- Added everyone on your domain name as a contact (this was disabled as a default setting after this happened with an European ISP e-mail addresses)
While competing solutions still required meeting id, meeting password, typing your name and e-mail, "taking the presenter role" before screen sharing, downloading a temporary executable file, showed weird gray boxes during screen sharing.
Other nice features where; Multiplatform Win/Mac/Linux/Browser based, Android, iOS, lots of dial-in phone numbers, even in my third world country (!), also audio/video quality was more stable even with our third world internet.
Did I also mention their executable clients where super responsive.
I'm told it's much better than most others. I haven't really noticed a dramatic difference, but that seems to be the consensus.
For them, something mediocre but gaplessly sufficient is preferable to something being mostly excellent with a few big caveats.
For example, it has to run as a on-premise hosted service, or has to tie into an arcane security environment which hasn't been updated in a long time where bandwidth is already an issue.
Imagine running that many copies of custom software in one environment, each with their ideal solution
Enterprise
Zoom just made a tool that could be used by everyone, everywhere, for free. And they saved us.
Yeah, I think that's something to celebrate.
Lots of people are using their paid zoom accounts more than their cell phone during the day and even for work.
Downtime means work is down too - not just for classrooms.
Dead Comment
My district started with Zoom in the spring, then switched to Meet when the Zoom security issues got hot, and ultimately settled on Webex for the fall because it was the most like Zoom, carried enterprise support, and because Meet is still missing far too many necessary features for teachers (breakout rooms, raise hand, etc).
This showed “coming in fall”: https://educationblog.microsoft.com/en-us/2020/06/20-updates...
Are they expecting students to “sit in class” by watching a live zoom lecture?
Too many schools are taking the approach that our learning wasn't structured enough in the spring, so we are going to the opposite extreme.
It's going to have a negative effect on the children and caregivers.
We have non-technical people on tight deadlines trying to reproduce brick and mortar on a computer screen. It's not going to end well. I feel for them, they are trying their best.
Not only is it going to be challenging for the kindergarten kids to stay focused on their class via chrome book, the SAME teacher is managing the kids in the classroom and the kids online. The kindergarten teachers each have an assistant too (as they did before Covid-19) so that helps a bit.
I am fortunate my wife stays at home with the kids. I can't imagine what it will be like for parents of kindergartners who both work, or parents with special needs kids or parents who don't speak English. I am supportive of social distancing in these times. But its going to be tough on the kids.
Our school is 100% distance learning. When they were going to allow distance learning and in person, they were planning to have completely separate classes for this so they could better optimize for your particular medium.
One improvement: the high schooler has some sessions with 90 kids (!!). So at least they can use their phones to goof off and text each other which they couldn’t do IRL.
There’s no easy answer to this, but I really wonder about the effects of this on kids, especially the younger ones. Kids, even liners, are pretty gregarious by nature, and there’s a lot of important physical contact among the little kids (and, for different reasons, teens :-)). Seems like there will likely be some developments issues down the road.
Deleted Comment
Not sure how four hours in front of a virtual lecture works for kindergarten, or why it was seen as a viable approach.
Seriously lamenting the lack of innovative thinking around this. What a wasted opportunity to do something truly interesting with education.
We've had a handful of school district customers sign up recently and use us to build aggregated status pages or send notifications to teachers and administrators when Zoom (or Blackboard or whatever) is down. So there seems to be a use case there beyond the "DevOps" market we currently cater to.
This issue only affects joining Zoom meetings through the web client. If you join using the desktop or mobile client to the meeting ID directly, you should still be able to join a meeting.
It was painful to get them to admit anything was wrong, until I asked if this would impact 9-1-1 calls too, then they admitted an issue with full-digit dialing (aka: 10 digit dialing).
Even units that were resistant to switching from the conference line migrated over.
I wonder how this afternoon will go if this continues.
But hyperlinks are not.
And the error message is strange:
>There is no accout for zuora account id:2c92a00d6ff0e970016ffbde74ae767c (3,201)
It seems that everyone having issues is seeing the same error.
Unfortunately this provides a perverse incentive to NOT report an outage, as even a small outage would be a contract breach for a "many nines" SLA, whereas many small breaches added up over a month could also breach a contract for (relatively) smaller companies. If the client doesn't notice, then it doesn't count.
Anecdotally, the only status pages I've seen that update automatically have no monetary incentive to maintain the page.
Ain't that the truth !
I remember working with organisations that had Verizon leased lines.
100% SLA the Verizon sales reps would say, normally accompanied with a sheet of paper that said "100% SLA" in big bold letters.
Of course the line would go down. Some manager would the come to my desk, thumping his fist and talking about the 100% SLA. I'd say "sure, but did you read the small print".
In all my years, I don't think I ever saw anyone get a SLA payment from Verizon.
Zuora seems to be a bit like Stripe? Maybe it's their outage, or maybe someone at Zoom forgot to pay the Zuora bill..
Gitlab had an outage today related to Zuora going down. My guess is the backends directly call to these APIs and they don't have any form of caching or storing the info from these APIs elsewhere.
https://trust.zuora.com/
Dead Comment
This isn't their area of expertise, and asking teachers to migrate to the backup and inform and assist their students to get online during an outage of unknown length isn't reasonable.
Snow days in 2020 will be replaced with Zoom Outage days.
(plus, training out of practise professors on two video conferencing systems? No way that would end well)