The guy who birthed modern deep learning and dipped
the "alex" in "alexnet"
[day 6/7 - sloppost had to crank something out before bed]
When learning a new discipline the first thing you have to contend with is that one is always standing on the shoulders of giants.
One of the cool things about ML is the giants tend to still be alive.
Hinton & Sutskever are household names in machine learning. They revolutionized the world by training, for the first time, an actual well-performing neural network for image classification.
I kinda can’t overstate the impact this paper had. Deep learning was not taken seriously as a a SOTA method for image recognition at the time. SOTA at the time was some truly fucked up shit.
This paper introduced so many concepts vital to machine learning today it’s baffling they fit all of it into a humble 8 pages: CNNs at scale, max pooling, dropout, data augmentation, ReLU1 and training on multiple GPUs. All massive parts of his future and our present.
This architecture was called AlexNet. It absolutely demolished the then state of the art. The architecture is named after the papers first author.
It’s first author??? Yes. Along Hinton and Sutskever is a guy named Alex Krizhevsky. He is very much not a household name. His google scholar looks like this.
But if you sort by most recent, you’ll see this:
He’s not doing any AI research at all anymore. His legacy are a handful of papers with a number of citations higher than I’ll ever earn in a year.
If Geoffrey Hinton is the “modern grandfather” of AI, then this guy is the father.
Contrary to Sutskever and Hinton, he actively avoids media attention. Apart from the initial AlexNet talk, he has done exactly one “interview”, and it is humblingly humble.
He does not like being filmed, but he did answer some questions. FOr example: when asked how much everyone contributed to the paper, he said the following.
I think if you measure time spent it’s definitely me, but all three of us made contributions so that if any one of us wasn’t there it wouldn’t have happened.
You changed the world, right?
With a reluctant smile on his face, he answers, almost sarcastic sounding: “yeah, totally”
His name is very literally already written in the history books. And what does he do with it? He does not show his face. He worked at google until 2018 and “lost interest” and just left.2 What is he doing now? I could not even find out. The man is linkedinless. Completely free.
This story makes me reflect on the amount of heroes we don’t hear about. The people with lasting impact on the world who despite potential for media attention, resist the pull and just go on living as they have. Silently massively changing the world. There is no clearer evidence that the work Alex did was pure utter love of the game.
It feels like a viscerally wise thing to do. It feels free-spirited. I am not built like this currently. But maybe one day.
In any case: Alex, I hope you’re doing alright.
sigmoids were still used in practice then
indeed he said he got bored. He gets bored of things after 10 years of studying them, it’s a pattern for him.
linkedinles~ my new favorite word!
Also shoutout to Dzmitry Bahdanau, the creator of Attention from "Attention Is All You Need"