This article started slow for me, I almost stopped reading, glad I didn’t. Great read when the discussion got into the CLEVR data sets and hybrid system.

Hybrid AI has been a keen interest after I retired from almost 5 years of work with deep learning. I just took a job on a knowledge graph team and it feels great to be working on a different approach to knowledge representation and reasoning. My personal opinion is that Hybrid AI is the future and I wouldn’t be surprised to see an exponential growth of practical results, similar to what happened with deep learning around 2013.

We have been working on this recently on the Google Brain team.

We are working both on synthesizing programs from scratch (see https://arxiv.org/abs/2002.09030 for example) and on understanding computer programs using machine learning (see e.g. https://arxiv.org/abs/1911.01205).

I'm always happy to correspond with people about these topics.

I think you are underselling the potential of a model which deeply understand programming. Imagine combining such a model with something like AutoML-Zero: https://arxiv.org/abs/2003.03384 It may not be 'creative', but used as tab-completion, it's not being rewarded or incentivized or used in any way which would expose its abilities towards creating a new sort algorithm.

I've been reflecting on exponentials and humans in light of coronavirus.

AlphaGo showed the same dynamic: I don't know if everyone remembers, but before the Lee Sedol tournament, most 'sensible' 'respectable' people assumed that there was no way AlphaGo would win, any claims it would stomp Lee Sedol were hyperbolic AI alarmism, and the only question was whether DeepMind would be saved from acute embarrassment by eking out one victory over Lee Sedol or if he would simply clean the board with AG. And in the absolute worst-case scenario, it might be a fairly even match. After all, anyone could look at the Fan Hui vs AG games which were released a few months before and see that AG was really crappy, hardly even human pro level; Lee Sedol wouldn't break a sweat. Yes, it would improve if they trained it more and tweaked it more, but so what? There were so many ELO points between Fan Hui and Lee Sedol, and AIs lack human intuition. Just drawing a line is mindless curve-fitting, which sensible respectable people know better than to do.

Meanwhile, people like Eliezer Yudkowsky considered the training time and thought through the exponential graph, and concluded that, given all those months between Fan Hui and the tournament and the fact that DM was willing to do the tournament at all with so much money on the lines, AG would either lose terribly or could well go 5-0. As it happens, his prediction was wrong. It actually went 4-1, because Lee Sedol got lucky and accidentally triggered a 'delusion' (fixed in AlphaZero).

We went from 'the best MCTS might be somewhat competitive with a very low-ranked human pro head-on' to 'superhuman' in a matter of months, because AG performance scaled with compute. We've seen similar capability leaps with DoTA2 or SC2: the best non-cheating AI was so low level it basically didn't exist (DoTA2) or was nowhere near competitive with any pros (SC2) and then within a few months of finding a working architecture (compare Oriol Vinyal's demo at Blizzcon to the matches just a few months later with SC2 pros), they reach better-than-most-pro levels even if they are not strictly superhuman yet.

That's all fun and games when it's just board games or computer games, of course...