One of the biggest benefits of go vs python is that you only manage your dependencies during build. You end up with a single executable that is easy to deploy and mostly immune to the deployment of other things. In python, you have to also worry about deployment. To be safe, you need to carry the vm and other dependencies with you.
This means you can't just give any python program to someone and have it just work, unless it's pretty trivial.
Go also carries all its dependencies with it; this is why it produces large single static binaries.
Python lacks a tool to painlessly produce the same, that is, a self-contained binary that includes the VM and all the dependencies. Determining Python dependencies may be slightly tricky, because `import` is an executable statement, and dependencies may be native-code libraries.
There are several tools for that, but all have their downsides, and none is in widespread use. Polishing such a tool until it's as simple and fool-proof as Go's build tool could be quite beneficial for Python community.
Everyone seem to agree the generation of a static binary is easy with go. That's not my experience as soon as CGO and the glib is involved. I would gladly pay 100$ to make it work on my open source project using libvips to anyone who can create the static build: https://github.com/mickael-kerjean/nuage
Why does it have to be a self-contained binary? When you pack everything into an virtualenv you get more or less the same result, don't you? If you want to have a single file you can zip it :)
Right, its not magic to build such a tool. But until it exists, this is a benefit for Go. This is also why I keep using Tcl, with tclkit you can build single-file executables for you tcl programs.
Speaking from the linux distro perspective, this is not true.
You can develop an application in python that utilizes distro-supplied python libraries. If you package your library in .deb or .rpm, it's trivial to utilize the dependencies from the distro.
Otherwise, you can install from pypi via pip, and especially into a virtualenv. You can declare each dependency's specific version if you so choose. Modern distros support python2.7 and python 3.x.
If you're on some other operating system, yeah, using python sucks for you.
While it's fair to say python is not statically compiled, it's no different than managing dependencies in other dynamically linked software.
Unlike python, go applications have to be cross compiled. For the most part, native python code will run anywhere the interpreter runs.
In the world of containerized applications, vendoring dependencies is a moot point anyway. We're all shipping around big tarballs with everything baked in anyway, so go doesn't offer any advantages in that space other than smaller binary size.
Not to say that go doesn't have some advantages over python, but I think the dependency management problem is vastly overstated, at least when it comes to python.
> You can develop an application in python that utilizes distro-supplied python libraries.
And unless you're aggressively tracking your distro's package releases you'd better hope that the new libdep doesn't introduce any breaking bugs. Also your app is no longer for Linux it's for Ubuntu 18.04 updated roughly on 2018-08-03.
The same goes for the actual python. Distributions apply lots of downstream patches and backport fixes so your python 2.7 is really redhat-python2.7-13 so you might want to test against that -- or you can bundle your own and be done with it.
Unless you have the wall clock time to actually define and test supported distributions you probably want to pretend the system python doesn't exist.
> pypi via pip, and especially into a virtualenv
I would never subject my users to this workflow. What you're describing are source distribution channels which are (ab)used to distribute applications. I have no idea if some dependency's native extension will even compile on their system. Do they even have build tools? Why would they?
Depends on your usecase, really. Static linking has been there for a long time and it has its disadvantages. I have more than a dozen programs on my system that share the same Python runtime and libraries. Compare that to every Go program shipping its own copy of the same things.
>every Go program shipping its own copy of the same things.
Is that really an issue? It takes up a bit more space, but nothing that's really any concern on a modern system. People often resort to using Docker to manage dependencies in other programming languages and end up shipping an entire copy of an operating system.
I honestly don't see many downside to shipping a static binary. At least in that case you can chroot the entire thing, something I really wish Python (and most other languages) would make a whole lot easier.
Security fixes require a recompilation, that perhaps the largest downside, but if you use Docker, then you need to rebuild that image as well.
First let me state that a language is just a brush with which to paint your logic. I really don't subscribe to the X is better than Y. I might concede that x is better than y for certain tasks, but not a sweeping generalisation.
The first point about inconsistent spelling mistakes, I 100% sympathise. I suffer from this terribly. Pylint and syntastic are a silver bullet for me. I have been tempted by a more fancy IDE, but I'm so used to vim, I'm reluctant/scared to change.
Parallelism: I do look over the wall at Go and wish that python 3.x had spent more time on removing the GIL, but that is old news now.
However I enjoy threading with python(not its limitations), because its so trivially easy. The Queues primitive saves 90% of the headache. as for ctrl-c not doing anything set all your threads to daemon (<threadObj>.setDaemon(True)) When the main thread exits, so does everything else. Go shines by having built in true parallelism
If I need horsepower, then I have to run multiple processes (not multiprocessing, that has drawbacks.) which has the annoyance of serializing and message passing. (but that does mean scaling horizontally less painful further on). Again, Go really shines here.
Statically compiling thing: YES. This means that I can 90% of the faff with docker (xyz version not working any more, the build fails) and use a real scheduling system that runs on real steel(or AWS) and not the omnifaff that is kubernetes.
> I have been tempted by a more fancy IDE, but I'm so used to vim, I'm reluctant/scared to change.
I'm happy to report that JetBrains' VIM plugin is almost flawless (Block-based editing, window splitting, save/close, it's all where it's supposed to be).
I used to be in the same camp. But if you add a linter and parser and full blown backend to make lookups for code completion to your VIM configuration you might as well use an IDE. I've switched to IntelliJ for Java.
And they provide a free all pack subscription for open source contributors. Its nice to see all the features at one place though their pipenv integration is still a bit shaky[0]. I haven't got a response yet.
I think when you really need performance it's a great example of a time to start looking at alternative Python implementations, CPython just isn't the best choice for a lot of tasks that actually have processing speed requirements.
and run it as an executable if your machine has stack installed. If not -- compile it with a `stack ghc --resolver=lts-12.2 ./file.hs` and you'll have a `./file` executable. No project creation, no need to list the dependencies.
I've written in Python for ~13 years and I'm really enjoying Go. It's fast and its libraries (the few there are) seem to make a lot of sense. I don't think I would use it for daily scripting like the author, but for a large new project I would definitely pick Go over Python now.
I like the expressiveness of Rust in theory, but for the programs I write I don't feel like I get sufficient payoff for fighting the borrow checker, as compared to writing in Go. Would I be correct in saying OCaml is pretty much Rust without the borrow checker? I have no prior experience with the language.
Does someone know a ressource where I can easily learn how to switch from Bash to Go?
To give some context: When it comes to serious programs I use Go already, but often I just want to code a little script to watch a website for changes for example:
#!/usr/bin/env bash
url="https://example.com"
mem="memory.txt"
oldsum="$(cat "$mem")"
newsum="$(curl -sL "$url" | sha512sum)"
if [ "$oldsum" != "$newsum" ]; then
xmpp_notify "$url changed"
echo "$newsum" > "$mem"
fi
I haven't tried it yet, but in my mind it feels like writing such a program would be more time consuming/complex in Go than writing it in bash. Neverthelss, I feel like Go is overall a much better language than Bash (with all those quotes), so I would like to transition to writing more short programs in Go and for that I would like to learn how to do things like reading/writing/grepping files as easily in Go as I do it in Bash. Any idea if someone wrote some hints together already?
The benefit of bash is that it has system binaries as first class citizens; you can leverage the hundreds of thousands of lines of GNU projects with a single keyword. It's hard to beat that expressiveness.
Saying "Go is overall a much better language" is like saying a wrench is a better tool than a screwdriver. You should keep both in your toolbelt and switch out when the need arises.
For your question, start with the regexp and os package, learn how to open a file, read in the contents, and then while iterating over the contents, apply a regex to the text. That should get you to the point that you can search plaintext for character strings. Good luck.
Don't listen to the naysayers. Go is fantastic for scripts like this - it's really good at network stuff, has a ton of built in functions, etc. And you get actual robustness and error checking that isn't a complete joke. Look how many ways your Bash script could fail! That's fine if you're literally watching the output with your eyes. If not it's going to cause issues when `curl` fails for example.
The only downside really is that you have to compile the Go code, which is fast but not as fast as just interpreting a Bash script (the first time).
Another possible issue is executable size - if you have a lot of Go scripts they may take up a fair bit of space.
I wish Go had an interpreter / JIT mode for scripts.
Edit: Before someone says "but that's 3 times longer than the bash code!" please write the equivalent Bash code that has proper error checking.
I think for this small size of an example program the error checking in Go is pure overhead, but as soon as the program starts to have a few more functions it will be very useful to have more options than just to abort on fail (Bash trap style / set -e). So I don't have any problem with 'polluting' my source with `if err != nil` checks ;-)
What I am more concerned about are the different ways I have to call functions from different libraries like
That is a lot more 'special' than the other function calls and the bash version and probably requires more documentation reading than '$ curl --help'. But hey, thanks to your example I have a starting point and probably are going to experiment with it in the new future.
For all the others who wrote that Bash is the right tool for the job: I read your warnings, opinions and suggestions and will keep them in mind while exploring the possibilities of using Go for script like jobs.
That is so repetitive to handle errors. Doesn't go have a cleaner way of handling errors than making the code look far less readable without unified error handling like exceptions?
Just a few months ago I replaced an almost 100-line Go "script" with two lines in the Makefile. The fact that Go is great for big projects doesn't automatically make it good for everything. Go is for programs, not scripts.
My problems come from the other end: My Bash scripts tend to become larger. A few weeks ago I wrote a 'cache' utility function (very useful to cache curl requests for example). A few days ago I wanted to use it in a second script was just about to write a package manager for bash modules... I think that is the kind of situation you are getting into when the line count of a bash script increases and you are trying to do proper software development ;-)
So I am not going to build every script in Go from today on, but I would like to learn a bit more on when Go is a viable alternative to Bash.
Being such a concise language, Haskell could be a good option here. You can do quick scripting while still having a powerful type system (especially compared to Go's) backing you up. I somewhat quickly typed this code up and haven't tested it, but it should replicate that bash script pretty closely:
#!/usr/bin/env stack
-- stack script --resolver=lts-12.2
import System.Process (system)
import Control.Lens
import Crypto.Hash.SHA256
import qualified Data.Bytestring.Lazy as B
import Network.Wreq
url = "https://example.com"
mem = "memory.txt"
main = do
oldsum <- B.readFile mem
newsum <- (hashLazy . view responseBody) <$> get url
when (oldsum /== newsum) $ do
exitCode <- system $ "xmpp_notify " ++ url ++ " changed"
B.writeFile mem newsum
I didn't make use of it, but there's also the Turtle library which aims to provide a fairly solid shell scripting experience, providing a lot of coreutils as simple Haskell functions. You also get a REPL in the form of GHCI, which comes default in Haskell installations.
I love Go and use it for allllmost everything, but I think bash is still better for something like this. The ease with which it can run programs and capture output is the killer feature.
I'd love to see some future bash like language that's strongly typed, has a consistent and simple syntax, but retains these advantages.
Go is the wrong tool for that. I'm no fan of bash, but it's often the the quick, clear (enough) solution for small tasks. I'm not a python dev - maybe that would be a suitable compromise?
for me, I use bash for anything less than 10 lines long, python for anything between 10 and 500 lines, and then I switch to a heavier programming language for anything longer.
> I’m not a fan of full blown IDEs that require their own configuration.
Hmm, but those IDEs actually do solve most of the problems author has had with Python. If one regularly code in Python, spending a few hours configuring the IDE is justifiable. I for example never write scripts outside the IDE, because why?
Note: I like Python, so this is me playing devil's advocate here.
If you need an IDE to solve common pain points of a language, isn't it fair to say that another language which lacks those pain points is a better fit for you? Using an IDE to aid you, to smooth things out and allow for more rapid development and better workflow is one thing. Being forced to use one because the language is lacking is a bad look on the language!
You could also say that this attitude holds us back. What if Smalltalk had only been accessible from vi back in the day, would Smalltalk have been as appreciated? I think not.
No they don't. No IDE can detect all typos in Python and do accurate code completion like they can for statically typed languages because the problem is unsolvable in general.
They can use heuristics to get fairly okish. Pycharm is not bad and VSCode's new Python extension seems to work fairly well. But it's never going to be as rock solid as Java code completion for example.
Prospector does a better job and it prevents you from wasting time with what essentially are stylistic choices and BS like a hard 80 chars limit per line (remember, A Foolish Consistency is the Hobgoblin of Little Minds)
And yes, yes, automated stilling tools are better
Intelisense is needed but even the standard vim autocompletion plugin does an ok job
> You're going async by using Golang, why not in python?
In Golang, you're still using a thread-based abstraction.
In Python, "going async" involves switching to a slightly different style of programming and using different I/O libraries. While that has advantages, using threads lets you stick with the standard style.
This means you can't just give any python program to someone and have it just work, unless it's pretty trivial.
Nuitka (nuitka.net) has been able to compile python programs to a single stand alone executable reliably, robustly and easily for a while now.
It's a real gem and deserves to be more popular. We are very far from the quirks of pyinstaller or cxfreeze : it just works, even with numpy or qt.
Yes, go is still superior for deployment :
- it can cross compile
- executables are smaller
- it's built in so it's just an easier experience
- it doesn't depend on libc
But it's important to spread the word: the time were your python program was hard to ship is over.
Just use nuitka.
> To be safe, you need to carry the vm and other dependencies with you.
Python lacks a tool to painlessly produce the same, that is, a self-contained binary that includes the VM and all the dependencies. Determining Python dependencies may be slightly tricky, because `import` is an executable statement, and dependencies may be native-code libraries.
There are several tools for that, but all have their downsides, and none is in widespread use. Polishing such a tool until it's as simple and fool-proof as Go's build tool could be quite beneficial for Python community.
You can develop an application in python that utilizes distro-supplied python libraries. If you package your library in .deb or .rpm, it's trivial to utilize the dependencies from the distro.
Otherwise, you can install from pypi via pip, and especially into a virtualenv. You can declare each dependency's specific version if you so choose. Modern distros support python2.7 and python 3.x.
If you're on some other operating system, yeah, using python sucks for you.
While it's fair to say python is not statically compiled, it's no different than managing dependencies in other dynamically linked software.
Unlike python, go applications have to be cross compiled. For the most part, native python code will run anywhere the interpreter runs.
In the world of containerized applications, vendoring dependencies is a moot point anyway. We're all shipping around big tarballs with everything baked in anyway, so go doesn't offer any advantages in that space other than smaller binary size.
Not to say that go doesn't have some advantages over python, but I think the dependency management problem is vastly overstated, at least when it comes to python.
And unless you're aggressively tracking your distro's package releases you'd better hope that the new libdep doesn't introduce any breaking bugs. Also your app is no longer for Linux it's for Ubuntu 18.04 updated roughly on 2018-08-03.
The same goes for the actual python. Distributions apply lots of downstream patches and backport fixes so your python 2.7 is really redhat-python2.7-13 so you might want to test against that -- or you can bundle your own and be done with it.
Unless you have the wall clock time to actually define and test supported distributions you probably want to pretend the system python doesn't exist.
> pypi via pip, and especially into a virtualenv
I would never subject my users to this workflow. What you're describing are source distribution channels which are (ab)used to distribute applications. I have no idea if some dependency's native extension will even compile on their system. Do they even have build tools? Why would they?
Is that really an issue? It takes up a bit more space, but nothing that's really any concern on a modern system. People often resort to using Docker to manage dependencies in other programming languages and end up shipping an entire copy of an operating system.
I honestly don't see many downside to shipping a static binary. At least in that case you can chroot the entire thing, something I really wish Python (and most other languages) would make a whole lot easier.
Security fixes require a recompilation, that perhaps the largest downside, but if you use Docker, then you need to rebuild that image as well.
You should make it easy as possible for users of your software .
The first point about inconsistent spelling mistakes, I 100% sympathise. I suffer from this terribly. Pylint and syntastic are a silver bullet for me. I have been tempted by a more fancy IDE, but I'm so used to vim, I'm reluctant/scared to change.
Parallelism: I do look over the wall at Go and wish that python 3.x had spent more time on removing the GIL, but that is old news now.
However I enjoy threading with python(not its limitations), because its so trivially easy. The Queues primitive saves 90% of the headache. as for ctrl-c not doing anything set all your threads to daemon (<threadObj>.setDaemon(True)) When the main thread exits, so does everything else. Go shines by having built in true parallelism
If I need horsepower, then I have to run multiple processes (not multiprocessing, that has drawbacks.) which has the annoyance of serializing and message passing. (but that does mean scaling horizontally less painful further on). Again, Go really shines here.
Statically compiling thing: YES. This means that I can 90% of the faff with docker (xyz version not working any more, the build fails) and use a real scheduling system that runs on real steel(or AWS) and not the omnifaff that is kubernetes.
I'm happy to report that JetBrains' VIM plugin is almost flawless (Block-based editing, window splitting, save/close, it's all where it's supposed to be).
I used to be in the same camp. But if you add a linter and parser and full blown backend to make lookups for code completion to your VIM configuration you might as well use an IDE. I've switched to IntelliJ for Java.
[0] : https://youtrack.jetbrains.com/issue/PY-30855
It dumps numpy arrays in /dev/shm which can loaded by other processes by the filepath, muuuuuch nicer than passing things around
Not even if X is better than Y for every specific task?
To give some context: When it comes to serious programs I use Go already, but often I just want to code a little script to watch a website for changes for example:
I haven't tried it yet, but in my mind it feels like writing such a program would be more time consuming/complex in Go than writing it in bash. Neverthelss, I feel like Go is overall a much better language than Bash (with all those quotes), so I would like to transition to writing more short programs in Go and for that I would like to learn how to do things like reading/writing/grepping files as easily in Go as I do it in Bash. Any idea if someone wrote some hints together already?Saying "Go is overall a much better language" is like saying a wrench is a better tool than a screwdriver. You should keep both in your toolbelt and switch out when the need arises.
For your question, start with the regexp and os package, learn how to open a file, read in the contents, and then while iterating over the contents, apply a regex to the text. That should get you to the point that you can search plaintext for character strings. Good luck.
The only downside really is that you have to compile the Go code, which is fast but not as fast as just interpreting a Bash script (the first time).
Another possible issue is executable size - if you have a lot of Go scripts they may take up a fair bit of space.
I wish Go had an interpreter / JIT mode for scripts.
Edit: Before someone says "but that's 3 times longer than the bash code!" please write the equivalent Bash code that has proper error checking.
I think for this small size of an example program the error checking in Go is pure overhead, but as soon as the program starts to have a few more functions it will be very useful to have more options than just to abort on fail (Bash trap style / set -e). So I don't have any problem with 'polluting' my source with `if err != nil` checks ;-)
What I am more concerned about are the different ways I have to call functions from different libraries like
That is a lot more 'special' than the other function calls and the bash version and probably requires more documentation reading than '$ curl --help'. But hey, thanks to your example I have a starting point and probably are going to experiment with it in the new future.For all the others who wrote that Bash is the right tool for the job: I read your warnings, opinions and suggestions and will keep them in mind while exploring the possibilities of using Go for script like jobs.
Just a few months ago I replaced an almost 100-line Go "script" with two lines in the Makefile. The fact that Go is great for big projects doesn't automatically make it good for everything. Go is for programs, not scripts.
My problems come from the other end: My Bash scripts tend to become larger. A few weeks ago I wrote a 'cache' utility function (very useful to cache curl requests for example). A few days ago I wanted to use it in a second script was just about to write a package manager for bash modules... I think that is the kind of situation you are getting into when the line count of a bash script increases and you are trying to do proper software development ;-)
So I am not going to build every script in Go from today on, but I would like to learn a bit more on when Go is a viable alternative to Bash.
I'd love to see some future bash like language that's strongly typed, has a consistent and simple syntax, but retains these advantages.
Hmm, but those IDEs actually do solve most of the problems author has had with Python. If one regularly code in Python, spending a few hours configuring the IDE is justifiable. I for example never write scripts outside the IDE, because why?
If you need an IDE to solve common pain points of a language, isn't it fair to say that another language which lacks those pain points is a better fit for you? Using an IDE to aid you, to smooth things out and allow for more rapid development and better workflow is one thing. Being forced to use one because the language is lacking is a bad look on the language!
They can use heuristics to get fairly okish. Pycharm is not bad and VSCode's new Python extension seems to work fairly well. But it's never going to be as rock solid as Java code completion for example.
- I make stupid mistakes in Python constantly. I misname a variable or a function or I pass in the wrong arguments: Try mypy
- If the task is IO bound: Threads are the wrong the answer to that. You're going async by using Golang, why not in python?
- With Python, I have to ensure all the packages I need are installed on the remote machine: pipenv Like slezyr mentions
- Consistent Styling: Try autopep8 or black
- Intellisense: I would say "use Pycharm" but vscode is getting better I heard
Prospector does a better job and it prevents you from wasting time with what essentially are stylistic choices and BS like a hard 80 chars limit per line (remember, A Foolish Consistency is the Hobgoblin of Little Minds)
And yes, yes, automated stilling tools are better
Intelisense is needed but even the standard vim autocompletion plugin does an ok job
In Golang, you're still using a thread-based abstraction.
In Python, "going async" involves switching to a slightly different style of programming and using different I/O libraries. While that has advantages, using threads lets you stick with the standard style.