I was initially a huge fan but now I hate it. It re-invents so many unnecessary wheels that it is downright obnoxious. There are no loops, if statements, variables. Things that have been in programming languages since day 1. It re-invents modules, badly. Has weird hacks like null resources and external data providers to get around shortcomings in their "declarative" model. And a few other things I'm forgetting. Oh, now they're adding "environments". That's just dandy because they got everything else so right.
Every place I've seen Terraform used invariably runs into these shortcomings and the workarounds range from using Erb and Jinja templates to generate Terraform templates to just ditching it entirely and using home-grown solutions. Terraform should have been a library in an actual programming language instead of a gimped external DSL that re-invents the wheel.
I would highly recommend Nixops. It allows you to describe your infrastructure in the Nix expression language. Which is a proper dict-based programming language. It supports many popular clouds. I'm not sure if it is Nixos only, but Nixos itself is awesome to use with it. You just put your provisioning configuration inside your "network" configuration and boom, all your servers and hard drives are created and fully provisioned.
I am actually okay with not having loops or advanced constructs. Those things made using Ansible for more complex provisioning scenarios (a) unreadable and (b) horribly kludgey.
m
The things that Terraform solves for, namely keeping environment state and infrastructure relationships, are really hard to do with other CM tools without diving deep i.e. writing python or ruby
It's not the lack of "advanced" constructs as you say. It is also the total disregard for all the human centuries of effort that have gone into making programming languages and environments convenient vehicles for expressing computation. Things like stack traces, syntax highlighting editors, auto-complete, interactive debuggers, and a bunch of other stuff you get with an actual programming language that you don't get with an external DSL. There is a reason those things are not part of the package. It is because making those things is extremely hard. Whereas slapping together a parser is not but feels like actual work.
You might consider those things advanced. I consider it the bare minimum to get proper work done.
HCL is atrocious. Terraform's main issue is that it tries very hard to be declarative, then it has to invent a template language as a superset of the declarative JSON it uses to make up for it. And that superset is invariably shit.
It's like you said; no loops, no proper variables etc.
Terraform itself though is fantastic and powerful. It's lacking some pretty important stuff still (for example, not having to refresh the entire infrastructure state when you change one single variable for one single tiny little service), but that just makes it an alpha. It's still better than the alternatives by a mile and then some.
But, yeah. Fuck HCL. The good news though is that I think this is "easily" fixable by someone who would really work on it: Because the underlying environment is 100% json, it's easy enough to generate; so you could implement a saner environment for specifying the state and feed that to terraform. That's enough for me to bet my infrastructure on it: knowing that today's issues are fixable, rather than the effort being doomed from the start.
We've really enjoyed templating terraform. It solves a lot of the problems you're mentioning, and we can write the complex logic we need without interacting _directly_ with cloud APIs...
It also seems pretty much the same as ansible (which suffers the same problems as you mentioned).
I pretty much agree with you, they should of provided 2 entrypoints IMHO, they can make there DSL and all that and make it (eventually) turing complete and at the same time they should just provide composeable objects in <some> language as well.
I've never quite figured out who the target audience is for these kind of DSL(s), is it people that can't program, or people that can't program well/at all (without adult supervision)?
I don't know either. Puppet, ansible, salt, and friends are in the same boat. The best part is that the special syntax doesn't buy them anything. They just end up being gimped languages for expressing some kind of dependency graph.
Apparently these people think it makes it "simpler". Well I'm sorry but me looking up how to format things with yaml and no autoformat and no intellisense just isnt good and doesnt make things simple.
Even xml with a backing xsd would be better than that, but that's apparently not cool enough anymore.
Some basic conditional logic was added in 0.8 [0]. The reliance on 'interesting' declarative tricks, e.g. counts for conditionals and loops, is a little annoying I agree. However, it looks like they're listening to the community and will build on these features in the future. There's a lot to work on :)
Take a look at jsonnet: http://jsonnet.org. It extends JSON with loops, conditionals, variables and more in an elegant way. We don't write JSON anymore and treat it like an "assembly language." We use jsonnet to generate CloudFormation templates, terraform templates and Kubernetes resource definitions.
Hear, hear! Devops right now is in the Middle Ages, kind of like what web frontend programming was a few years ago. Remember all those weird cobbled down logicless DSLs that were supposed to save us from ourselves.
I am expecting the Renaissance you mentioned in devops shortly, just like how React transformed the scene.
In short, DSLs should be inside programming languages and we shouldn't be afraid to write code.
But this isn't a programming language. it's a DSL that lets you get the job done. I get that you want to use flow control statements to make your life easier, but I feel that part of the reason why Terraform was made and has gained a lot of popularity. For me, it doesn't make things overly complicated. Except for how they handle variables with modules.That stuff sucks.
I don't really agree that Terraform re-invents wheels. In fact it feels like a lot of the things you cite as issues are actually due to Terraform being pretty innovative. There's not much like it for "version-controlled" infrastructure, so the team and community are learning as they go, and can't be expected to come up with perfect solutions right away. I agree the looping hacks are very ugly, really dislike the whole split and join everywhere thing.
As for DSL vs. programming languages, the halting problem provides a pretty compelling argument in favor of limiting your system if you want to have "verifiable convergence". Personally I'm a huge fan of HCL too. It fixes a lot of what's bad about YAML and TOML and has a built in pretty printer.
Ahh just like Ansible with it's YAML'ish templating language. If you want to add variables and scoping, loops, a module resolution system, etc., then I'd rather use a real programming language.
Huge Terraform fan. The first time it creates VM's, disks, IPs, firewall rules it is like magic. Infrastructure definition should be code, it just makes sense.
If your looking to dive in, I wrote a short introduction blog post on getting started with Google Compute Engine.
I love terraform! It is one of the few tools I've found that was easy to understand and easy to integrate with legacy stuff, and it very quickly saved me a huge amount of time.
It only took me about 30 minutes to setup a terraform file that could bring up and teardown an entire web stack with a single command. I was so shocked by how straightforward it was that I brought the stack up and down a few times just to make sure it was actually repeatable.
We had a tough time with Terraform on AWS. The syntax is undeniably better than Cloudformation and having the concept of modularity built in is a big win.
However, when things went wrong we ended up with cryptic error messages and often when `terraform up` failed, then `terraform destroy` would fail too, leaving us using AWS console to jump in and start clearing up the resources. Particularly painful because AWS has no awareness that your stack came from a Terraform config, so you have to navigate through the subresource menus destroying things one-by-one.
We ended up switching over to Cloudformation. The extra verbosity does suck, but the tooling for running/updating stacks feels far safer. We can review the changeset when we update the stack, view a realtime event log as resources are created, deleted or updated, and best of all we can always tear the stack down from a single point.
Amazing platform. We've recently taken about 6 months to define our entire infrastructure in Terraform. We've been using it since version 0.6 and with version 0.9, the biggest feature to come for us is definitely the remote back-end and the remote locking mechanism (which we use Terragrunt for atm). The remote back-ends mean that the state will not longer be stored on the local machine which is obviously a big plus for security as the state actually contains all of your infrastructure information and even secrets. Anyway, huge project and huge potential, so much so, I've made it the headline in my CV! Thank you Hashicorp, keep it coming!
It's allowed us to do the following for our developers:
Full integration with the CI/CD process.
Open a branch, commit it to GitHub, add a special label and you will get a completely new setup, just to test your branch. New application name in the service discovery tool (Consul/Eureka), new RDS instances, new network, etc, etc. Once happy, the developers can merge into dev and the branched version of the infrastructure is destroyed.
I cannot recommend Terraform enough for teams that are looking to gut their old infrastructure deployment methods. We used Terraform ~0.7 to build out our EU VPC on AWS. After a bit of pain understanding how to build modules we were finally able to say we have our infrastructure truly in code. We were even able to reproduce our exact infrastructure deployment, with a bit of tweaking, for partner that had strict data privacy requirements. Terraform allowed us to provision a smaller scale deployment of our production deployment on their AWS account in about a days worth of work. We plan on redoing our US deployment Terraform and we we are going to POC their Pro/Enterprise version very soon
Can anyone compare Fugue[0] with Terraform/Terraform Enterprise.
I'm launching a venture that requires deploying and maintaining cloud infrastructure across the three major cloud providers (AWS, GCP, Azure) and am considering options. Most of our infrastructure is kubernetes but Fugue caught my eye for their AWS deployments. Curious if anyone has used both.
Readit News
Every place I've seen Terraform used invariably runs into these shortcomings and the workarounds range from using Erb and Jinja templates to generate Terraform templates to just ditching it entirely and using home-grown solutions. Terraform should have been a library in an actual programming language instead of a gimped external DSL that re-invents the wheel.
https://Nixos.org/nixops
The things that Terraform solves for, namely keeping environment state and infrastructure relationships, are really hard to do with other CM tools without diving deep i.e. writing python or ruby
You might consider those things advanced. I consider it the bare minimum to get proper work done.
It's like you said; no loops, no proper variables etc.
Terraform itself though is fantastic and powerful. It's lacking some pretty important stuff still (for example, not having to refresh the entire infrastructure state when you change one single variable for one single tiny little service), but that just makes it an alpha. It's still better than the alternatives by a mile and then some.
But, yeah. Fuck HCL. The good news though is that I think this is "easily" fixable by someone who would really work on it: Because the underlying environment is 100% json, it's easy enough to generate; so you could implement a saner environment for specifying the state and feed that to terraform. That's enough for me to bet my infrastructure on it: knowing that today's issues are fixable, rather than the effort being doomed from the start.
Compilers are like, useful.
I pretty much agree with you, they should of provided 2 entrypoints IMHO, they can make there DSL and all that and make it (eventually) turing complete and at the same time they should just provide composeable objects in <some> language as well.
I've never quite figured out who the target audience is for these kind of DSL(s), is it people that can't program, or people that can't program well/at all (without adult supervision)?
0: https://www.hashicorp.com/blog/terraform-0-8/#conditional
I am expecting the Renaissance you mentioned in devops shortly, just like how React transformed the scene.
In short, DSLs should be inside programming languages and we shouldn't be afraid to write code.
As for DSL vs. programming languages, the halting problem provides a pretty compelling argument in favor of limiting your system if you want to have "verifiable convergence". Personally I'm a huge fan of HCL too. It fixes a lot of what's bad about YAML and TOML and has a built in pretty printer.
If your looking to dive in, I wrote a short introduction blog post on getting started with Google Compute Engine.
https://blog.elasticbyte.net/getting-started-with-terraform-...
- The new 'State environment' feature should resolve the issues discussed here: https://charity.wtf/2016/03/30/terraform-vpc-and-why-you-wan...
- The new locking feature means we don't need to use https://github.com/gruntwork-io/terragrunt
It only took me about 30 minutes to setup a terraform file that could bring up and teardown an entire web stack with a single command. I was so shocked by how straightforward it was that I brought the stack up and down a few times just to make sure it was actually repeatable.
However, when things went wrong we ended up with cryptic error messages and often when `terraform up` failed, then `terraform destroy` would fail too, leaving us using AWS console to jump in and start clearing up the resources. Particularly painful because AWS has no awareness that your stack came from a Terraform config, so you have to navigate through the subresource menus destroying things one-by-one.
We ended up switching over to Cloudformation. The extra verbosity does suck, but the tooling for running/updating stacks feels far safer. We can review the changeset when we update the stack, view a realtime event log as resources are created, deleted or updated, and best of all we can always tear the stack down from a single point.
It's allowed us to do the following for our developers:
Full integration with the CI/CD process.
Open a branch, commit it to GitHub, add a special label and you will get a completely new setup, just to test your branch. New application name in the service discovery tool (Consul/Eureka), new RDS instances, new network, etc, etc. Once happy, the developers can merge into dev and the branched version of the infrastructure is destroyed.
If you want a great starting point for learning how to do things the right way https://github.com/hashicorp/best-practices/tree/master/terr... helped us out a lot.
I'm launching a venture that requires deploying and maintaining cloud infrastructure across the three major cloud providers (AWS, GCP, Azure) and am considering options. Most of our infrastructure is kubernetes but Fugue caught my eye for their AWS deployments. Curious if anyone has used both.
[0] https://fugue.co/