What Google Cloud needs is a better sales story. AWS consistently beats out Google in this respect. This story is welcomed and there should be more success stories published like this.
My business is a heavy GC user (AppEngine, Datastore, CloudStorage, BQ, ComputeEngine, ManagedVM) and I couldn't imagine achieving what we have achieved in such a short amount of time on any other platform.
We (myself and another guy) started at zero and shipped our product in 3 months on what is effectively an infinitely auto-scalable platform where we don't have to do any devops or carry a pager. I'd much rather work on features than chores. =) 4 months later and we've had profitable growth and zero downtime (knock on wood) and we're hiring.
I've been relying on multiple Cloud offerings (including PaaS, IaaS etc) outside Google for many years, but for some unclear reason, I cannot manage to consider Google a reliable partner for a type of hosting or another, at this point.
I would love to be proven wrong really, because I often look at their GC offering, and I /really/ want to dive in, yet I feel that I would risk having to migrate quickly in 2 weeks at some point because they are changing things significantly (either shutting down or other major change).
So to my question: for how long have these services been around in a fashion that is stable (feature-wise, quality-wise etc)? Is there anyone here with a longer history of using Google Cloud offering, able to comment on this?
EDIT: thanks to everyone who replied below. I think I'll give it a try :-)
For Google Cloud products that have made it to General Availability (GA), we have a one-year deprecation policy (https://cloud.google.com/terms/, Section 7.2). That is, if we're going to make some big change, we give you a 1 year heads up.
Disclaimer: I work on Compute Engine, but I'm not a lawyer ;).
The problem (I suffer from it as well), is that people naturally feel that the corporate offerings from Google will mirror the experience that they have with the consumer offerings from Google. And the big issue with Google's consumer offerings is that they seem to have limited to no support and that there is a very real chance that Google will just shut them down all of a sudden. This is the exact opposite of what you want in the enterprise space, where Service & Stability are paramount. Even though I know that Google's corporate offerings are treated differently, that little niggling doubt remains. Plus, Google really needs to hire some damn marketing experts to sell the features & benefits of their offerings and make them cohesive - they just have their stuff plastered everywhere.
Anecdotal: AppEngine deprecated their Master/Slave datastore[0] in April 2012, and it was actually shutdown on July 6, 2015. So that's 3+ years to move to newer tech.
I wonder how big you'd have to be to get a real SLA from Google, Azure, or AWS. I imagine 25% off your hosting bill doesn't really cut it for those companies running millions of dollars in revenue through it.
The main reason I tend to shy away from GC is that it's blocked in China. AWS isn't (which also means Heroku isn't blocked since Heroku runs on AWS, so I use AWS for IaaS and Heroku for PaaS).
If you plan to offer multi-lingual content, anticipate having viewers in that part of the world, and your content isn't something they would be interested in blocking, it's just a practicality worth factoring in.
Another reason is that I find the GC console a mess. I had to simultaneously use two versions of the console to get different tasks done; they didn't have one console that had all features of the system. Heroku was a lot cleaner and easier to navigate in this respect.
On the contrary, GC is NOT entire blocked in China. Some of the Google services (including GC console / dashboard) are indeed blocked but GC's IP address pool and the applications running on GC does seem to be accessible in China.
My company is running mostly on Google Cloud and I had just returned from a trip in China where I've verified it.
In my experience (3 months in Shanghai at the end of last year) AWS does occasionally get blocked or severely slowed down in China. I'd say it was inaccessible about 5% of the time (for significant periods of time).
You can of course run in the AWS Beijing region which requires you to have a Chinese business presence with a ICP license.
It'll be interesting to see what happens now that Google is expanding its presence in China [1]. I ran into some Googlers on the Maps team while in Shanghai - not likely a coincidence.
The old App Engine / API / Admin console era is over. All tasks have been migrated to https://console.cloud.google.com/ including some of the last App Engine specific administration things.
One of the reasons I avoid Google is because they have taken lot of pain to make their customer care totally suck. I mean even IRS and Comcast support does better compared to Google.
I published a very critical release to my Android app few weeks back and realized that it was not getting published at all. It took me a month to get the problem resolved.
I don't want to buy a cloud service where the support might be nothing more than bunch of "Help yourself" pages.
Spotifier here. One of the big risks that we identified with the Google partnership was exactly this: Google isn't exactly known for awesome customer service.
We've been very pleasantly surprised. The cloud team has been pretty awesome to work with, including lots of engineer<->engineer contact, walk-throughs of systems and code for critical dependencies, and solid support and collaboration. Exceeded our expectations.
We offer paid support for Cloud (http://cloud.google.com/support) that let you trade off responsive time (e.g., 4 hours vs 1) for price.
The support personnel (and engineering!) also do best effort scanning of Stack Overflow, Server Fault, and per-product mailing lists. You can get links to those here: https://cloud.google.com/support/#community.
Disclaimer: I work on Compute Engine, but not in Support (though I do jump in and support customers!)
I've had problems with their customer support. It got to the point, I just gave up, and left my videos on YouTube. (My problem was that slick move they made when they bought YouTube, and messed with the login names.)
I just try to forget they are up there. I got to one person at Google, and when they realized I didn't want to advertise, it was "Go to the help boards. I don't deal with that stuff."
Do I still use their products--yes, but don't trust them like I used to. I usually avoid customer service like the flu virus, but thought Google would be different?
One of the big advantages of Google Cloud platform is "Scalability".
Load balancers: Scale to millions of users seamlessly. No warming up, no tickets, ...
PubSub: You can send the whole Internet 10 times in a day through it. Google does that every day
Big Query: Its equivalent to spinning up a 100+ node cluster in a matter of seconds and it can process data at speeds of GB's of data per sec, I heard couple of use cases where the user was able to process at ~ 50 GB/s
Kubernetes: 1000's of nodes running 100's of thousands of containers.
Funny you should mention that. :) I work at Google on PerfKit Benchmarker (https://github.com/GoogleCloudPlatform/PerfKitBenchmarker). Not only can you measure and test GCP (and other clouds), but we're trying to make it very easy for you to do so!
(PKB doesn't have a benchmark for App Engine yet, though. Sorry.)
Are you implying that we don't measure and test? We do and I'm very happy with the results. =) StackDriver, Google Cloud console, Intercom, Mixpanel and Pingdom are all being used as well.
Our company motto is that we will strive to never hire a devops role and will work to design and engineer our platform in a way that backs up that claim.
I have tried it last summer and was quite disappointed. CPU-bound programs that completed in a few seconds on my own computer took about ten times as long on AppEngine despite similar nominal specs (GHz and RAM).
If you like the App Engine model, but want a "full computer worth of horsepower", you might want to consider our "Managed VM" environment built on top of Compute Engine: https://cloud.google.com/appengine/docs/managed-vms/
Traditional App Engine is not particularly cost-effective for compute-intensive loads, but it works great for basic web workloads. If you have special needs, Managed VMs have the performance characteristics of the underlying Compute Engine node. We've found that App Engine is not "all or nothing"; it's the backbone of our app but we run services where they make the most sense (usually GCE).
In addition to all the excellent comments here. Let's not forget that the massive massive Snapchat runs almost entirely on Google AppEngine. So if AppEngine can handle their scale, it can handle most others :)
App Engine expects your web apps to return almost immediately (in a matter of few ms). For anything that takes longer, you need to use task queues in App Engine. Or maybe try AppEngine Managed VM's?
GearLaunch provides merchants with software that allows them to build and run online storefronts, and also manages production, logistics and customer service for all products sold. GearLaunch is the only e-commerce software provider to cover the entire value chain, enabling marketers to focus on marketing.
My experience with upper management is that Google isn't "serious" about their cloud offerings based on how much of Amazon's profit margin is tied up in AWS. It's an enragingly pointyhaired sentiment to deal with.
Who's upper management? As I mentioned in another thread, we acquired Diane Greene's company so that she would run Cloud and Apps at Google, back in November:
and Sundar (CEO) and Ruth (CFO) mentioned Cloud a lot on Alphabet's most recent earnings call. I'm not disagreeing that AWS is massively important to Amazon, but to say that Google (and Alphabet) aren't serious about Cloud is incorrect.
The only thing that is stopping us from having Google Cloud is lack of an Australian data centre. I'd love to be on Google Cloud, but requirements prevent us from doing so unfortunately :-/
I assume your business is GearLaunch? If so, why do I need to enable javascript just to see the landing page, which is just a logo...? I'm curious if this was intended, or a consequence of the framework you're using (e.g. Meteor).
Edit: It seems to be on purpose, with the use of noscript tags.
I'm on mobile so this initial response will be short:
AWS' core product is AWS. They do a damn fine job of providing this.
GC's core is "who knows what" - they do a damn poor job of even making an offering if switch thousands and thousands of servers to - let alone thousands and thousands of man hours.
Ironically, Google has massive domain knowledge of how to run infra... Yet clueless how to adopt and support and foster companies other than google.
It sucks.
But I've been in the business of fork lifting many people to aws, for all the reasons you'd expect:
Find me a single ops eng on the market who has done scale in GC? Nope.
> Find me a single ops eng on the market who has done scale in GC?
Every user of appengine, for a start. You need to code to their datastore, but apart from that scaling is practically infinite - and completely invisible. We don't need to configure, provision or even * understand * load balancers or geographically redundant servers.
Concrete examples? The royal wedding website was served from appengine a few years back; I imagine you'd be impressed with how that scaled from nothing to phenomenal traffic loads and back again. And Khan Academy, Snapchat, and now Spotify ought to meet your definition of "done scale".
I want to commend some of the Google Compute Platform employees that commented in this thread.
I was at AWS for 6 years (left 2 years ago... today!), and I've always been a proponent of being more open and communicative with developers, but it rarely happened - I guess that AWS' PR policy and such are a big showstopper for these kind of discussions. Although, some individuals did their best (e.g. Jeff Barr) to try to share as much as possible.
AWS employees may not speak out very often, but they hardly need to. AWS is a marketing machine, from Jeff's regular blog posts, to the volume of free information they provide and to the conferences that they are running everywhere. I think Microsoft is starting to catch on with Azure promotions & conferences, but it's like Google didn't even know that the there was a race .... and they left their running gear at home.
For me personally[0], the difference isn't being able to speak in terms of marketing, but in terms of engaging with users who have problems. It may not translate to much in terms of run rate, but I like to think (or at least hope) the presence of GCP staff on HN, Stack Overflow, and in any other community we happen to be part of will in the long run pay off in terms of goodwill.
I am free to comment on posts about GCP here on HN. If there are workarounds, I can post them. If it's a bug I can comment as such and file it internally. I can express opinions as long as they're clearly marked as my own. I didn't have to take any training or end up on a whitelist; Google expressed their faith in my good judgement when they hired me. If nothing else, it was a morale boost to feel trusted to act responsibly.
At the other end of the spectrum, GCP has started building out in terms of large-scale engagement. Cloud has been a big part of Google I/O; now there's GCP NEXT as a Cloud-specific user conference: https://cloudplatformonline.com/NEXT2016.html. Building more in this space seems easier to me than a cultural change which encourages (rather than prohibits) customer and community engagement on the individual level, but perhaps I'm being naïve.
[0]: I currently work at Google on Compute Engine; I formerly worked at Amazon, although not on AWS.
I tried Google Container Engine (GKE) and really liked it - it's the best cloud solution for deploying Docker to production in my opinion, mainly due to its use of Kubernetes. Unfortunately in my Web apps I make heavy use of Postgres-specific features, and since Cloud SQL only supports MySQL, Google Cloud is a total non-starter for me.
So for now I'm on AWS, using Postgres on RDS and deploying containers with ECS. ECS is a lot simpler than Kubernetes, but since my apps are pretty simple (a half dozen task definitions), it's not a big deal. I really hope Google adds Postgres to Cloud SQL at some point.
There are vendors running managed Postgres services on Google Cloud Platform, like ElephantSQL [1] and Aiven [2]. And you can of course run your own on GCE, even to the extent of 24/7 commercial support - you can run EnterpriseDB from Cloud Launcher [3].
And, you can also run Kubernetes on AWS - we have a group focused on making sure it's an excellent experience.
I work for Google Cloud Platform; ping me if you'd like more help with either option.
> There are vendors running managed Postgres services on Google Cloud Platform, like ElephantSQL
I did check out ElephantSQL but my pricing needs are somewhere between their $100 and $20 plans and there seems to be a lack of configurability compared to RDS's parameter groups (e.g. enabling extensions).
> you can also run Kubernetes on AWS
I've had success turning up Kubernetes clusters on AWS for demo purposes, but I really don't want to manage a k8s cluster myself (anecdotes I've read about etcd failures / partitions especially scare me). Also I use Terraform for provisioning, and kube-up.sh is not something that fits into that paradigm. I've also made the mistake running kube-up.sh with the wrong arguments after a previous invocation that had created a cluster, which caused it to try and create a new cluster, which wiped the local cache of the previous cluster I had made, making kube-down.sh unable to automatically clean up the old cluster (so I had to do it manually in the AWS console).
The other thing I tried was the kube-aws CoreOS tool, which is nice, but it comes baked with a 90-day expiration due to TLS certificate expiration, so I'd have to set up some sort of PKI process to make that production-ready. All in all just too much work for a single person trying to deploy a small number of containers for small to medium sized projects; if I was a medium-sized company with hundreds of containers and some dedicated DevOps resources maybe it would be worth it, but for myself I'd prefer a turn-key solution like ECS or GKE.
IMO one of the benefits of a platform service is that you get the whole platform from one vendor, so if you're having some problem, you can work with one vendor to sort it out. Trying to get Google support to work with a 3rd-party vendor and my hypothetical company on an issue sounds like a nightmare. There are already many other options for platform services which provide e.g. Everything you'd need to run a Rails app in dev and prod, so that's where the bar is.
I sincerely hope you will add PostgreSQL support to CloudSQL soon.
The current Postgres offerings are not great. ElephantSQL is extremely expensive compared to their offering: 4 cores, 15GB RAM, 1TB data for $1,000/mo. CloudSQL (2nd gen) with the exact same specs would be $370/mo. Aiven doesn't advertise exact prices until you sign up, so I can't compare, but I see that the number of instances (max 3 nodes) is very small, so not really an option.
Google Cloud SQL is no replacement for Amazon RDS in my opinion. Cloud SQL instances run outside of your project's private network, so connections from Google Compute Engine have to be over a Public IP. This means accepting connections from any host (insecure) or whitelisting each Google Compute Engine VM's IP address (pain in the ass).
I've resorted to running my own MySQL instance inside of Google Compute Engine and setting up replication and off-site backups myself. It's definitely not as convenient as Amazon RDS, but the rest of Google Cloud has some great features like Google Container Engine.
Would love to see an RDS-like solution from Google that runs in a project's private network and supports more than just MySQL.
So, basically something like AWS's VPC? Given that it took them a few years to get that right even with tons of people asking for it, I'm not super optimistic that Google could match it any time soon. I'd love to be wrong!
I am sure Google Cloud has VPC stuff for all the components, including Cloud SQL. It lacks the variety (No PostgreSQL, Oracle, MSSQL, or MariaDB). All traffic is encrypted by default on all posts (Google Cloud does this for free) too.
I'm tentatively happy with our move to GCE as well, although we need to live with it more. What really delights me is the quality of the web dashboard, command line tool, and the ability to open web based SSH windows with a single click.
> Google has long been a thought-leader in this space, and this shows in the sophistication and quality of its data offerings. From traditional batch processing with Dataproc, to rock-solid event delivery with Pub/Sub to the nearly magical abilities of BigQuery, building on Google’s data infrastructure provides us with a significant advantage where it matters the most.
Big Data is the core strength of Google Cloud. Good to see this move by Spotify!
> What really tipped the scales towards Google for us, however, has been our experience with Google’s data platform and tools. Good infrastructure isn’t just about keeping things up and running, it’s about making all of our teams more efficient and more effective, and Google’s data stack does that for us in spades.
What I really really liked about Google Cloud is the ease of use. Spin up a VM, start Cloud shell, SSH into your instance, install a bunch of software and you will know what I mean. It's "Quality" Cloud.
At work we moved to GCE at the beginning of this year, from Linode after they were having stability issues over the christmas break. No complaints from us. So far have been very happy with it. We were considering moving to AWS, but to realize the same pricing as GCE we'd have to purchase reserved instances - the sustained usage discounts have been huge for us.
Additional things we liked:
- gcs is way more responsive than S3. also, was fairly painless to migrate our S3 buckets to GCS via the web console.
- peering with cloudflare lets us save on bandwidth costs!
- network load balancer has shown itself to be very reliable and solid for holding open A LOT sustained websocket connections.
- http load balancer has shown itself to be very capable of ssl termination & routing (love that we can route /api to our API servers, and the rest to our static servers). additionally, no pre-warming was required. when we did the datacenter switch, it was just 5 mins of downtime. didn't have to worry about pre-warming for our production load like we would have on ELB.
Things we didn't like:
- salt-cloud's driver for GCE is still lacking. Can't specify disk size for provisioned storage. Can't specify local SSD storage either. Also parallel provisioning didn't work. Something about PyCrypto not liking the way it forked. Not GCE's fault
- billing support non-existent unless you purchase a support plan.
- documentation still needs some work.
I tried to sign up for trial. Entered my credit card info. All cleared. Try to use bigquery and got some confusing error about billing not being set. Contacted support a couple of days ago. It said response time is 2-3 hours. Haven't heard since... Anecdotal? Yes. Makes me re-consider even trying google cloud offering? You bet.
Thanks for the reply! I don't remember the specific issue, but we had a billing inquiry and were told we had to upgrade to a paid support plan to get an answer.
The big point of this move is the hit it will have on on-premises Hadoop offerings: Cloudera, Hortonworks, MapR. It's a massive vote of no-confidence in their offerings.
Spotify are effectively ditching a 2000 server, 90 PB Hadoop cluster to go GCE. As Spotify say themselves, that's big news....
Spotifier here. This is an important point. I have nothing bad to say about Cloudera or HWX (disclaimer: we're an HWX customer - we've had a pretty good experience), but I don't really see a compelling reason at this stage to manage your own cluster(s) (HIPAA/regulatory constraints, maybe?)
Getting shared-storage and indepedently operated/scaled compute clusters on top of that storage isn't easily achievable with the standard Hadoop stack, and building that on top of HDFS is non-trivial.
In fact, I don't think large orgs like you (Spotify) really want independently operated clusters. That prevents easy sharing of data, causing data silos to appear. You really want to have true multi-tenancy, which isn't in Hadoop yet. Hadoop has worked more on Kerberos support at the cost of features like easy-to-use access control - Apache Ranger or Sentry anybody!?!?
We host a lot of our applications on Google Compute Engine but have had many issues with even simple features available from other cloud providers. For example, Google's HTTP(S) Load Balancer offering does not support SNI [1], HTTP to HTTPS forwarding [2], or sticky sessions [3], making it unusable for us. The support we pay for has been pretty helpful, but all we hear is "we have a feature request for this but I cannot provide an ETA on when and if this will be implemented". Many of the issues in the Compute Engine public issue tracker [4] haven't been updated in many months or over a year in some cases. Additionally, it can be very confusing finding out which beta services can only be accessed via the gcloud tool and not the web console. Without the ability to even schedule a window for maintenance on our VMs, we often feel like we do not have the control we need over our hosting environment. Please increase your transparency around addressing Google Cloud issues, all the Docker updates have been great, but many of these issues are over a year old and still show-stoppers for some users. Thanks again for churning out new features/services constantly, but please help close out some of these older feature requests!
Readit News
My business is a heavy GC user (AppEngine, Datastore, CloudStorage, BQ, ComputeEngine, ManagedVM) and I couldn't imagine achieving what we have achieved in such a short amount of time on any other platform.
We (myself and another guy) started at zero and shipped our product in 3 months on what is effectively an infinitely auto-scalable platform where we don't have to do any devops or carry a pager. I'd much rather work on features than chores. =) 4 months later and we've had profitable growth and zero downtime (knock on wood) and we're hiring.
Thanks Google!
I would love to be proven wrong really, because I often look at their GC offering, and I /really/ want to dive in, yet I feel that I would risk having to migrate quickly in 2 weeks at some point because they are changing things significantly (either shutting down or other major change).
So to my question: for how long have these services been around in a fashion that is stable (feature-wise, quality-wise etc)? Is there anyone here with a longer history of using Google Cloud offering, able to comment on this?
EDIT: thanks to everyone who replied below. I think I'll give it a try :-)
Disclaimer: I work on Compute Engine, but I'm not a lawyer ;).
[0] http://googleappengine.blogspot.com/2012/04/masterslave-data...
If you plan to offer multi-lingual content, anticipate having viewers in that part of the world, and your content isn't something they would be interested in blocking, it's just a practicality worth factoring in.
Another reason is that I find the GC console a mess. I had to simultaneously use two versions of the console to get different tasks done; they didn't have one console that had all features of the system. Heroku was a lot cleaner and easier to navigate in this respect.
My company is running mostly on Google Cloud and I had just returned from a trip in China where I've verified it.
You can of course run in the AWS Beijing region which requires you to have a Chinese business presence with a ICP license.
It'll be interesting to see what happens now that Google is expanding its presence in China [1]. I ran into some Googlers on the Maps team while in Shanghai - not likely a coincidence.
[1]: http://www.bbc.com/news/technology-34698642
I published a very critical release to my Android app few weeks back and realized that it was not getting published at all. It took me a month to get the problem resolved.
I don't want to buy a cloud service where the support might be nothing more than bunch of "Help yourself" pages.
We've been very pleasantly surprised. The cloud team has been pretty awesome to work with, including lots of engineer<->engineer contact, walk-throughs of systems and code for critical dependencies, and solid support and collaboration. Exceeded our expectations.
The support personnel (and engineering!) also do best effort scanning of Stack Overflow, Server Fault, and per-product mailing lists. You can get links to those here: https://cloud.google.com/support/#community.
Disclaimer: I work on Compute Engine, but not in Support (though I do jump in and support customers!)
It sounds like you're avoiding the generic Google in the same way you'd not buy something from Amazon vs. using AWS.
I just try to forget they are up there. I got to one person at Google, and when they realized I didn't want to advertise, it was "Go to the help boards. I don't deal with that stuff."
Do I still use their products--yes, but don't trust them like I used to. I usually avoid customer service like the flu virus, but thought Google would be different?
I guess they are big enough to not care?
You never know what you don't measure and test
Load balancers: Scale to millions of users seamlessly. No warming up, no tickets, ...
PubSub: You can send the whole Internet 10 times in a day through it. Google does that every day
Big Query: Its equivalent to spinning up a 100+ node cluster in a matter of seconds and it can process data at speeds of GB's of data per sec, I heard couple of use cases where the user was able to process at ~ 50 GB/s
Kubernetes: 1000's of nodes running 100's of thousands of containers.
Can't beat that!
Funny you should mention that. :) I work at Google on PerfKit Benchmarker (https://github.com/GoogleCloudPlatform/PerfKitBenchmarker). Not only can you measure and test GCP (and other clouds), but we're trying to make it very easy for you to do so!
(PKB doesn't have a benchmark for App Engine yet, though. Sorry.)
I have tried it last summer and was quite disappointed. CPU-bound programs that completed in a few seconds on my own computer took about ten times as long on AppEngine despite similar nominal specs (GHz and RAM).
Disclaimer: I work on Compute Engine.
https://cloud.google.com/appengine/docs/python/modules/#Pyth...
Google does give an additional level of comfort - seamless scaling, No-Ops, powerful Big Data tools, flexibility.
I'm going to shamelessly promote one of my opinions on this topic when it comes to BigQuery:
https://cloud.google.com/blog/big-data/2016/02/visualizing-t...
Dead Comment
https://angel.co/gearlaunch
http://www.nytimes.com/2015/11/20/technology/google-picks-di...
and Sundar (CEO) and Ruth (CFO) mentioned Cloud a lot on Alphabet's most recent earnings call. I'm not disagreeing that AWS is massively important to Amazon, but to say that Google (and Alphabet) aren't serious about Cloud is incorrect.
Edit: It seems to be on purpose, with the use of noscript tags.
Look at the names themselves CloudStorage Vs Redshift/s3, ComputeEngine Vs EC2. Cookies vs. Oreo.
AWS service names are a brand in themselves while Google service names are too generic.
:puzzled:
I just like to see what other people can achieve in 3 months of time, it's kind of a motivation for me :)
AWS' core product is AWS. They do a damn fine job of providing this.
GC's core is "who knows what" - they do a damn poor job of even making an offering if switch thousands and thousands of servers to - let alone thousands and thousands of man hours.
Ironically, Google has massive domain knowledge of how to run infra... Yet clueless how to adopt and support and foster companies other than google.
It sucks.
But I've been in the business of fork lifting many people to aws, for all the reasons you'd expect:
Find me a single ops eng on the market who has done scale in GC? Nope.
EDIT: may I please have a rebuttal?
Every user of appengine, for a start. You need to code to their datastore, but apart from that scaling is practically infinite - and completely invisible. We don't need to configure, provision or even * understand * load balancers or geographically redundant servers.
Concrete examples? The royal wedding website was served from appengine a few years back; I imagine you'd be impressed with how that scaled from nothing to phenomenal traffic loads and back again. And Khan Academy, Snapchat, and now Spotify ought to meet your definition of "done scale".
I was at AWS for 6 years (left 2 years ago... today!), and I've always been a proponent of being more open and communicative with developers, but it rarely happened - I guess that AWS' PR policy and such are a big showstopper for these kind of discussions. Although, some individuals did their best (e.g. Jeff Barr) to try to share as much as possible.
It seems that the Google guys know how to do it.
Keep doing it. It will help your business a lot.
Is there a particular ticket (or tickets) you ran into trouble with? I can try to follow up.
I am free to comment on posts about GCP here on HN. If there are workarounds, I can post them. If it's a bug I can comment as such and file it internally. I can express opinions as long as they're clearly marked as my own. I didn't have to take any training or end up on a whitelist; Google expressed their faith in my good judgement when they hired me. If nothing else, it was a morale boost to feel trusted to act responsibly.
At the other end of the spectrum, GCP has started building out in terms of large-scale engagement. Cloud has been a big part of Google I/O; now there's GCP NEXT as a Cloud-specific user conference: https://cloudplatformonline.com/NEXT2016.html. Building more in this space seems easier to me than a cultural change which encourages (rather than prohibits) customer and community engagement on the individual level, but perhaps I'm being naïve.
[0]: I currently work at Google on Compute Engine; I formerly worked at Amazon, although not on AWS.
I used to work at AWS and the things we could say were very controlled. It could just be my team/managers though.
So for now I'm on AWS, using Postgres on RDS and deploying containers with ECS. ECS is a lot simpler than Kubernetes, but since my apps are pretty simple (a half dozen task definitions), it's not a big deal. I really hope Google adds Postgres to Cloud SQL at some point.
And, you can also run Kubernetes on AWS - we have a group focused on making sure it's an excellent experience.
I work for Google Cloud Platform; ping me if you'd like more help with either option.
[1] https://www.elephantsql.com/blog/2014-11-17-google-compute-e... [2] https://aiven.io/ [3] https://cloud.google.com/launcher/solution/public-edb-ppas/e...
I did check out ElephantSQL but my pricing needs are somewhere between their $100 and $20 plans and there seems to be a lack of configurability compared to RDS's parameter groups (e.g. enabling extensions).
> you can also run Kubernetes on AWS
I've had success turning up Kubernetes clusters on AWS for demo purposes, but I really don't want to manage a k8s cluster myself (anecdotes I've read about etcd failures / partitions especially scare me). Also I use Terraform for provisioning, and kube-up.sh is not something that fits into that paradigm. I've also made the mistake running kube-up.sh with the wrong arguments after a previous invocation that had created a cluster, which caused it to try and create a new cluster, which wiped the local cache of the previous cluster I had made, making kube-down.sh unable to automatically clean up the old cluster (so I had to do it manually in the AWS console).
The other thing I tried was the kube-aws CoreOS tool, which is nice, but it comes baked with a 90-day expiration due to TLS certificate expiration, so I'd have to set up some sort of PKI process to make that production-ready. All in all just too much work for a single person trying to deploy a small number of containers for small to medium sized projects; if I was a medium-sized company with hundreds of containers and some dedicated DevOps resources maybe it would be worth it, but for myself I'd prefer a turn-key solution like ECS or GKE.
The current Postgres offerings are not great. ElephantSQL is extremely expensive compared to their offering: 4 cores, 15GB RAM, 1TB data for $1,000/mo. CloudSQL (2nd gen) with the exact same specs would be $370/mo. Aiven doesn't advertise exact prices until you sign up, so I can't compare, but I see that the number of instances (max 3 nodes) is very small, so not really an option.
I've resorted to running my own MySQL instance inside of Google Compute Engine and setting up replication and off-site backups myself. It's definitely not as convenient as Amazon RDS, but the rest of Google Cloud has some great features like Google Container Engine.
Would love to see an RDS-like solution from Google that runs in a project's private network and supports more than just MySQL.
http://googlecloudplatform.blogspot.co.nz/2015/12/the-next-g...
I too hope so.
Big Data is the core strength of Google Cloud. Good to see this move by Spotify!
> What really tipped the scales towards Google for us, however, has been our experience with Google’s data platform and tools. Good infrastructure isn’t just about keeping things up and running, it’s about making all of our teams more efficient and more effective, and Google’s data stack does that for us in spades.
What I really really liked about Google Cloud is the ease of use. Spin up a VM, start Cloud shell, SSH into your instance, install a bunch of software and you will know what I mean. It's "Quality" Cloud.
Additional things we liked: - gcs is way more responsive than S3. also, was fairly painless to migrate our S3 buckets to GCS via the web console. - peering with cloudflare lets us save on bandwidth costs! - network load balancer has shown itself to be very reliable and solid for holding open A LOT sustained websocket connections. - http load balancer has shown itself to be very capable of ssl termination & routing (love that we can route /api to our API servers, and the rest to our static servers). additionally, no pre-warming was required. when we did the datacenter switch, it was just 5 mins of downtime. didn't have to worry about pre-warming for our production load like we would have on ELB.
Things we didn't like: - salt-cloud's driver for GCE is still lacking. Can't specify disk size for provisioned storage. Can't specify local SSD storage either. Also parallel provisioning didn't work. Something about PyCrypto not liking the way it forked. Not GCE's fault - billing support non-existent unless you purchase a support plan. - documentation still needs some work.
Disclaimer: I work in Cloud Support.
Deleted Comment
Deleted Comment
Getting shared-storage and indepedently operated/scaled compute clusters on top of that storage isn't easily achievable with the standard Hadoop stack, and building that on top of HDFS is non-trivial.
[0]http://www.csc.kth.se/~gkreitz/spotify-p2p10/spotify-p2p10.p...
[1] https://code.google.com/p/google-compute-engine/issues/detai... [2] https://code.google.com/p/google-compute-engine/issues/detai... [3] https://code.google.com/p/google-compute-engine/issues/detai... [4] https://code.google.com/p/google-compute-engine/issues/list