This is really significant. At this moment in history, the growth of computer power has made a bunch of important signal-processing and statistical tasks just feasible, so we are seeing things like self-driving cars, superhuman image recognition, and so on. But it's been very difficult to take advantage of the available computational power, because it's in the form of GPUs and clusters. TensorFlow is a library designed to make it easy to do exactly these things, and to scale them with your available computing power, along with libraries of the latest tricks in neural networks, machine learning (which is pretty close to "statistics").
As a bonus, it has built-in automatic differentiation so that you can run gradient descent on any algorithm — which means that you can just write a program to evaluate the goodness of a solution and efficiently iterate it to a local maximum. If you do this enough times, hopefully you'll find a global maximum. There are a variety of other numerical optimization algorithms, but gradient descent is very simple and broadly applicable.
And it runs in IPython (now Jupyter), which is a really amazingly powerful way to do exploratory software development. If you haven't tried Jupyter/IPython, google up a screencast and take a look.
I'm just repeating the stuff that's on the home page in a different form, but this is a really big deal. Most of the most significant software of the next five or ten years is going to be built in TensorFlow or something like it. (Maybe Dan Amelang's Nile; see Bret Victor's amazing video on the dataflow visualization he and Dan built for Nile recently at https://youtu.be/oUaOucZRlmE?t=24m53s, or try the live demo at http://tinlizzie.org/dbjr/high_contrast.html, or download and build the Nile system from https://github.com/damelang/nile.)
Apparently the Googlers think that too, because among the 39 authors of the white paper are several shining stars of systems research, including some who may be up for a Turing Award in the next decade or two.
What about Torch or Theano which allow you to use multiple GPUs and clusters? they also have a wide array of libraries which allows you to extend capabilities (itorch etc.). torch is also very fast, most parts written in C so I don't know if Tensorflow would be really that fast compared to existing librairies.
One thing find I found interesting is the ability to use the software you designed in research directly in production (without having to rewrite the whole thing in java)
That said, I'm excited to see Google release this. Hopefully it would encourage Facebook or Baidu to also release
As far as I can tell, this may not be faster than theano/torch, but it doesn't sound like it'll be slower. It takes a similar approach of sending all the big computation stuff out to efficient implementations. The "slow" language code is simply building up a graph of work to be done by the efficient implementations.
As a side note, it feels like the front page picture of interactive computational graph is a bit misleading - looks like the interface is still using Python (or C++), not fancy GUI... But yes, it looks very similar to a great tool from Yoshua Bengio's group - Theano. So, great! Another amazing tool to take away the grind of doing ML projects!
Agreed, this is huge. I've been thinking along these lines for a while, but more focused on traditional programming systems. The approach of programming via data-flows and visually representing those is going to be enormous in the coming decade, IMHO. Especially in asynchronous and heterogenous programming environments.
Actually, it feels like I've been "scooped" somewhat. However, at the same time it's awesome to find that many of the core assumptions I've been working in formulating for traditional programming systems are actually already being used by other top engineers/researchers at Google. Thanks for posting about the whos-who of systems research. I'll have to check up on their research more!
I think there is a lot of room left to apply this approach to a lot of existing systems outside of machine learning. We're working on releasing a prototype in the coming months at our new startup (vexpr.com) based on applying these same data-flow principles to normal programming projects. Actually this might give a huge boost in presenting the underlying approach to systems engineering to audiences.
I wouldn't worry too much about being scooped by this--the idea of data flow programming has been around since the 60s. Tensorflow looks like a very nice implementation of those ideas for neural network training, and is potentially useful for more general numerical computing, but to what extent remains to be seen.
From the AMS statement on The Culture of Research and Scholarship in Mathematics:
"For this reason, mathematicians traditionally list authors on joint papers in alphabetical order. An analysis of journal articles with at least one U.S. based author shows that nearly half were jointly authored. Of these, more than 75% listed the authors in alphabetical order. In pure mathematics, nearly all joint papers (over 90%) list authors alphabetically."
Why is alphabetical order great? It just only means that people with last names starting with z will get penalized every single time because of choosing wrong parents! I would think randomized order would be chosen - especially if you were mathematician.
If only it were so simple. It really depends on the field, but in the biological sciences it really matters where you are in the authorship list. If you are first it is assumed you did most of the work and if you are last you were the head honcho of the group. When you apply for scholarships and grants the committees will only consider the first and last authors as important and everyone else is just filler. It really important to be last when you are trying to get grants as so much emphasis is placed on track record.
Depending on how long it takes to "evaluate the goodness of a solution" techniques like multi-start gradient decent can rapidly become intractable though, especially in higher dimensions. There are a handful of open source libraries out there that try to tackle this time consuming and expensive black box optimization problem from a more Bayesian approach [1] [2]. There's also a YC company to do this as a service (full disclosure, I'm a co-founder) [3]. Combining these methods with TensorFlow could be very powerful and is one step closer to more automatic ML flows though.
Nile is a language built by Dan Amelang, a researcher @ viewpoints research institute. It is a language made for graphics rendering, it has the goal of having no incidental complexity.
2D vector graphics rendering as an example, takes ~10.000 lines of code using traditional libraries like Cairo (used by Firefox) or Skia (used by Android and Chrome). In Nile, it's 200 lines of code.
This is awesome, thanks so much for sharing this - and I love how whimsical the docs are:
Our pond is a perfect 500 x 500 square, as is the case for most ponds found in nature.
A feedback on the PDE aspect: is there any nice way to write your model and grid using TensorFlow, and still obtain the free derivatives/optimized computation graph if you use a different solver to actually integrate the equations? The example shows off a very simple toy first order Euler solver, but in practice this is grossly inadequate. I suspect that it's very difficult if not impossible - especially since most solver libraries are very old Fortran libs, and carrying forward the reverse mode differentiation there is a nightmare. Still, this makes for very nice building blocks.
Another question - would you consider adding an example of using this together with Google Ceres - to make the two autodiffs play nice together?
Maybe of interest: this is (amongst others) by Jeff Dean, famous for Bigtable, Map Reduce, Spanner, Google Web Search, Protocol Buffers, LevelDB, and many, many other things. He's a Google Senior Fellow, but also doubles as the patron saint of Google Engineering.
The whole scientific Python stack is available for Python 3. This seems like a somewhat backwards thing to do -- or perhaps the requirement is intended to mean at least 2.7?
This "backwards" meme doesn't make any sense. Python 2 and Python 3 are different languages that happen to share the same name, a lot of syntax, and near-source compatibility, but they've been developed concurrently for ten years now. It's kind of like C and C++. If the current Python 2 team (which is part of the Python Software Foundation) abandons it, probably somebody else will take over maintenance, because it seems unlikely that it will fall out of use.
If the Python Software Foundation drops the ball, someone else will pick it up.
Let me start by saying I understand what you're saying. I understand why what you're saying is true. However, many people see (and consequently think) this:
"Short version: Python 2.x is legacy, Python 3.x is the present and future of the language"
... which was lifted from https://wiki.python.org/moin/Python2orPython3. It follows that some people would think something along the lines of "...so this library was built for legacy Python?"
I am really excited about this library. Tensors are the future; matrices are going to look so old in a few years. In fairness, Theano already knew that. I cannot wait to dig into this and am very relieved that Google's best are still using Python for this endeavour...but...using Python 2.7.
I have just recently been persuaded by the community that 3.5 is cost free, and here I have this enormous counterexample. For the science guys, clearly, the message is not getting through, and I'm not surprised: 3.x offers them nothing. Hence they're blissfully continuing with 2.7.
So I guess I'll have to run two Pythons on my system. Not the ideal situation.
Now if Python 3.(x > 5) would give us native GPU, which might require putting (a GPU-enabled) Numpy into the standard library...as opposed to, say, spending 2 years re-inventing Tornado....
> Tensors are the future; matrices are going to look so old in a few years.
I am honestly curious about this point of view. Is there any example where actual multidimensional tensor have any relevance? What I mostly see around is just standard linear algebra operations on matrices and vectors lifted to higher-dimensional tensors point-wise (for instance applying a certain operation to all 2-dimensional subtensors of a given tensor).
I never saw the need for modeling matrices in terms of tensors - the latter seem just more complex to use and usually tensor operations (like symmetrization, antisymmetrization, wedge products...) are not even available in libraries
(And by the way, both matrices and tensors are now more than one century old...)
Methinks embrace the magical library written by engineers and researchers at Google instead of by a random goulash of machine learning grad students with no investment in the outcome other than graduating.
Well, wait to see how support goes etc... In the meantime take on Keras or Lasagne, and do your stuff, i.e. acquire hands-on experience now. When time comes that you need to go into production, or you need a model that is not available in your current lib or framework, then consider a switch. Point being that experience acquired with one framework will empower you whatsover.
In my view, these frameworks are like programming languages, you do need to pick a new one up from time to time, but you can do most of the things with all of them.
Right now Keras is your best bet for most deep learning tasks, and Caffe is best for image processing (i.e. convolutional neural networks). Keep an eye on tensorflow though because it may come to be the gold standard over the next several months to year.
Edit: Btw, François Chollet, the author of keras, made the following tweet this morning: "For those wondering about Keras and TensorFlow: I hope to start working on porting Keras to TensorFlow soon (Theano support will continue)."
I tried going through the site and also the comments but couldn't wrap my head around what this library actually is. It sounds awesome based on the response/comments. Can anyone explain this to a layman?
It is a library for the optimization of machine learning algorithms, similar to Theano. You write a model you want to optimize as a graph of symbolic expressions, and the library will compile this graph to be executed on a target platform (could be CPU, GPU). This is really neat because you essentially wrote your program in Python and now it's going to be running as optimized C++ code (or as optimized kernels on your GPU).
The fact that you defined your model as a symbolic graph means you can derive gradient equations automatically, hence ridding you of a rather tedious step in optimization. With such tools, the workflow for a practitioner is much simpler: 1-code models as graphs of symbolic expressions, and 2-define objective functions (ex. for a regression problem, your objective would be the root mean squared error, plus some regularization). Disclaimer: I did not look at the code, but from what I understand it is pretty much the same as Theano (which I've been using a lot lately).
If you want to train neural nets, you can either rewrite everything from scratch and get a bug-ridden sub-optimal implementation, or you can use a kind of off-the-shelf library.
The problem is that there are about 3-5 alternatives out there, and none of them are mature enough or convincing enough to dominate. The field changes so fast that it's easy for them to become obsolete.
What you're seeing here is the enthusiasm of people who really want to get a good tool with proper support, and be able to stick with it. I'm still not sure if TensorFlow is that tool, but it depends on what will happen to it during the coming years.
It looks like to me the big win here is that this is a ML library just like all the others, except that the code is not written for a specific type of processor (GPU versus CPU), nor is it written for a specific compute level (desktop, server, phone). It's target is to be a general purpose ML library. This should have the affect of getting ML solutions into useable applications more quickly, since you don't have to develop on one platform, then port to another. It just works everywhere. Best guess on my part based on what I'm reading on the site.
EDIT: To be a bit more specific, autograd just requires that one writes the forward model, and it can involve any program logic like conditionals, etc, so long as the end result is differentiable. It then takes care of all the gradient calculations.
i really wish people would stop calling multidimensional arrays tensors.
there is already well defined language for this. multidimensional array is fine, psuedotensor is fine. tensor is confusing if you have any previous background with tensor calculus before the word became machine learning flavour of the month.
still. this does look pretty cool overall. processing large volumes of data is becoming increasingly less specialised and more generic, which can only be good. :)
I'd say tensor is exactly as confusing as vector -- vectors are just elements of some vector spaces, and you get a list of numbers by choosing a basis of that vector space. Similarly, tensors are just elements of a tensor product of some vector spaces (or modules), and you get a multidimensional array by choosing bases in the factors.
The problem is complicated a bit more because physicists use the word "tensor" to mean a tensor field, i.e. a collection of tensors on which calculus makes sense.
Tensor is a much better term than multidimensional array:
- It's not as verbose
- The concept of tensor has linear algebra operations associated with it while multidimensional array is just a programming term without any attached Math operations.
> The concept of tensor has linear algebra operations associated with it while multidimensional array is just a programming term without any attached Math operations.
But that's the precise problem. The multidimensional arrays that programmers call "tensors" do not generally have the operations defined on them that you would expect from a tensor, such as index contraction, tensor products, or differentiation. At the same time, a tensor is different from an array, just like a number is different from its decimal representation. These distinctions are important when you want to change the basis, a very useful and frequent operation for both tensors and numbers.
Then again, this battle was already fought and lost for vectors, which programmers specialised to mean "arrays" instead of meaning "elements of a linear space".
Readit News
As a bonus, it has built-in automatic differentiation so that you can run gradient descent on any algorithm — which means that you can just write a program to evaluate the goodness of a solution and efficiently iterate it to a local maximum. If you do this enough times, hopefully you'll find a global maximum. There are a variety of other numerical optimization algorithms, but gradient descent is very simple and broadly applicable.
And it runs in IPython (now Jupyter), which is a really amazingly powerful way to do exploratory software development. If you haven't tried Jupyter/IPython, google up a screencast and take a look.
I'm just repeating the stuff that's on the home page in a different form, but this is a really big deal. Most of the most significant software of the next five or ten years is going to be built in TensorFlow or something like it. (Maybe Dan Amelang's Nile; see Bret Victor's amazing video on the dataflow visualization he and Dan built for Nile recently at https://youtu.be/oUaOucZRlmE?t=24m53s, or try the live demo at http://tinlizzie.org/dbjr/high_contrast.html, or download and build the Nile system from https://github.com/damelang/nile.)
Apparently the Googlers think that too, because among the 39 authors of the white paper are several shining stars of systems research, including some who may be up for a Turing Award in the next decade or two.
One thing find I found interesting is the ability to use the software you designed in research directly in production (without having to rewrite the whole thing in java)
That said, I'm excited to see Google release this. Hopefully it would encourage Facebook or Baidu to also release
Actually, it feels like I've been "scooped" somewhat. However, at the same time it's awesome to find that many of the core assumptions I've been working in formulating for traditional programming systems are actually already being used by other top engineers/researchers at Google. Thanks for posting about the whos-who of systems research. I'll have to check up on their research more!
I think there is a lot of room left to apply this approach to a lot of existing systems outside of machine learning. We're working on releasing a prototype in the coming months at our new startup (vexpr.com) based on applying these same data-flow principles to normal programming projects. Actually this might give a huge boost in presenting the underlying approach to systems engineering to audiences.
I think that's fantastic and wish academia went that way too (yeah right).
From the AMS statement on The Culture of Research and Scholarship in Mathematics: "For this reason, mathematicians traditionally list authors on joint papers in alphabetical order. An analysis of journal articles with at least one U.S. based author shows that nearly half were jointly authored. Of these, more than 75% listed the authors in alphabetical order. In pure mathematics, nearly all joint papers (over 90%) list authors alphabetically."
http://www.ams.org/profession/leaders/culture/CultureStateme...
Deleted Comment
Also consider that a single "order" might not make any sense. How do you determine if somebody "contributed" more than anybody else?
[1]: https://github.com/Yelp/MOE (full disclosure, I co-wrote this)
[2]: https://github.com/hyperopt/hyperopt
[3]: https://sigopt.com/cases/machine_learning
http://download.tensorflow.org/paper/whitepaper2015.pdf
starts with the Author list.
I can't find it on Google.
2D vector graphics rendering as an example, takes ~10.000 lines of code using traditional libraries like Cairo (used by Firefox) or Skia (used by Android and Chrome). In Nile, it's 200 lines of code.
Check out this repo, and the pdf inside the readme https://github.com/damelang/nile
http://www.tinlizzie.org/ometa/
https://en.wikipedia.org/wiki/OMeta
That tool can be used to solve a lot more problems. REBOL/Red and Racket are also taking DSL approach.
Our pond is a perfect 500 x 500 square, as is the case for most ponds found in nature.
A feedback on the PDE aspect: is there any nice way to write your model and grid using TensorFlow, and still obtain the free derivatives/optimized computation graph if you use a different solver to actually integrate the equations? The example shows off a very simple toy first order Euler solver, but in practice this is grossly inadequate. I suspect that it's very difficult if not impossible - especially since most solver libraries are very old Fortran libs, and carrying forward the reverse mode differentiation there is a nightmare. Still, this makes for very nice building blocks.
Another question - would you consider adding an example of using this together with Google Ceres - to make the two autodiffs play nice together?
The whole scientific Python stack is available for Python 3. This seems like a somewhat backwards thing to do -- or perhaps the requirement is intended to mean at least 2.7?
Edit: added context
Edited again: More careful wording
Tracking here: https://github.com/tensorflow/tensorflow/issues/1
Congrats on the library, which looks awesome.
Deleted Comment
If the Python Software Foundation drops the ball, someone else will pick it up.
"Short version: Python 2.x is legacy, Python 3.x is the present and future of the language"
... which was lifted from https://wiki.python.org/moin/Python2orPython3. It follows that some people would think something along the lines of "...so this library was built for legacy Python?"
I have just recently been persuaded by the community that 3.5 is cost free, and here I have this enormous counterexample. For the science guys, clearly, the message is not getting through, and I'm not surprised: 3.x offers them nothing. Hence they're blissfully continuing with 2.7.
So I guess I'll have to run two Pythons on my system. Not the ideal situation.
Now if Python 3.(x > 5) would give us native GPU, which might require putting (a GPU-enabled) Numpy into the standard library...as opposed to, say, spending 2 years re-inventing Tornado....
I am honestly curious about this point of view. Is there any example where actual multidimensional tensor have any relevance? What I mostly see around is just standard linear algebra operations on matrices and vectors lifted to higher-dimensional tensors point-wise (for instance applying a certain operation to all 2-dimensional subtensors of a given tensor).
I never saw the need for modeling matrices in terms of tensors - the latter seem just more complex to use and usually tensor operations (like symmetrization, antisymmetrization, wedge products...) are not even available in libraries
(And by the way, both matrices and tensors are now more than one century old...)
"Tensor said the tensor. Tension, apprehension, And dissension have begun."
https://en.wikipedia.org/wiki/The_Demolished_Man
Tensors were the future even before they were cool.
Deleted Comment
I want to start deep learning now, to implement RNNs, autoencoders, Q-learning on a very specific application (not images).
I've read a lot of papers, but not done any DL implementations yet (although many classical ML so far), my question is very simple :
Where do I start ????
Should I use Torch ? Theano ? Theano + pylearn2 ? Theano + pylearn + Lasagne ? Caffe ?
Or should I just switch to this magical new library directly ?
I feel confused.. any advice ?
Come on. Grad students are motivated by advancing the field.
... are they not? Maybe you can also be saying that the ones who write libraries are motivated only by graduating.
In my view, these frameworks are like programming languages, you do need to pick a new one up from time to time, but you can do most of the things with all of them.
Edit: Btw, François Chollet, the author of keras, made the following tweet this morning: "For those wondering about Keras and TensorFlow: I hope to start working on porting Keras to TensorFlow soon (Theano support will continue)."
The fact that you defined your model as a symbolic graph means you can derive gradient equations automatically, hence ridding you of a rather tedious step in optimization. With such tools, the workflow for a practitioner is much simpler: 1-code models as graphs of symbolic expressions, and 2-define objective functions (ex. for a regression problem, your objective would be the root mean squared error, plus some regularization). Disclaimer: I did not look at the code, but from what I understand it is pretty much the same as Theano (which I've been using a lot lately).
The problem is that there are about 3-5 alternatives out there, and none of them are mature enough or convincing enough to dominate. The field changes so fast that it's easy for them to become obsolete.
What you're seeing here is the enthusiasm of people who really want to get a good tool with proper support, and be able to stick with it. I'm still not sure if TensorFlow is that tool, but it depends on what will happen to it during the coming years.
https://blog.twitter.com/2015/autograd-for-torch
EDIT: To be a bit more specific, autograd just requires that one writes the forward model, and it can involve any program logic like conditionals, etc, so long as the end result is differentiable. It then takes care of all the gradient calculations.
there is already well defined language for this. multidimensional array is fine, psuedotensor is fine. tensor is confusing if you have any previous background with tensor calculus before the word became machine learning flavour of the month.
still. this does look pretty cool overall. processing large volumes of data is becoming increasingly less specialised and more generic, which can only be good. :)
- It's not as verbose
- The concept of tensor has linear algebra operations associated with it while multidimensional array is just a programming term without any attached Math operations.
But that's the precise problem. The multidimensional arrays that programmers call "tensors" do not generally have the operations defined on them that you would expect from a tensor, such as index contraction, tensor products, or differentiation. At the same time, a tensor is different from an array, just like a number is different from its decimal representation. These distinctions are important when you want to change the basis, a very useful and frequent operation for both tensors and numbers.
Then again, this battle was already fought and lost for vectors, which programmers specialised to mean "arrays" instead of meaning "elements of a linear space".