Recently, OpenAI’s ChatGPT software has been lauded for its ability to generate natural-language responses to human prompts. Many think ChatGPT is unique since it’s so new and engaging.
ChatGPT’s Underlying Techniques
AI scholars disagree. Yann LeCun, Meta’s top AI scientist, told a Zoom meeting of reporters and executives last week that ChatGPT’s fundamental algorithms are not unique. “It’s nothing innovative,” LeCun stated. “It’s just neatly put together, nicely done.”
Similar Technology Developed by Other Companies and Research Labs
LeCun said many firms and institutes have constructed data-driven AI systems. He denied that OpenAI works alone. LeCun claimed OpenAI is not an advance compared to other laboratories. “It’s not only just Google and Meta, but there are half a dozen startups that basically have very similar technology to it,” added LeCun. “It’s not rocket science, but it’s shared, if you will.”
ChatGPT’s Composition
LeCun remarked that ChatGPT and OpenAI’s GPT-3 are made up of numerous technologies developed over many years by different parties. LeCun noted that ChatGPT leverages Transformer designs pre-trained in this self-supervised manner.
He advocated self-supervised learning before OpenAI existed. LeCun observed that Google’s 2017 language neural net, Transformers, underpins many language systems, including GPT-3.
History of Language Programs
LeCun noted decades of language programme development. “Large language models, the first neural net language model — at the time, it was large, by today’s standards, it’s little — was by Yoshua Bengio, about 20 years ago,” stated LeCun. Google used Bengio’s attention research for the Transformer, and all language models use it.
Reinforcement Learning and Human Feedback
Reinforcement learning through human feedback, like Google’s Page Rank for the web, is used extensively in OpenAI’s initiative to improve machine output. He stated DeepMind, not OpenAI, pioneered that method.
Decent Engineering
LeCun said the ChatGPT software is more about good engineering than science. He compared the programme to IBM’s Watson computer, which played Jeopardy! in 2011, and Sebastian Thrun’s self-driving vehicle, which won DARPA’s 2005 Grand Challenge. “It was just really beautifully engineered,” LeCun remarked of Thrun’s award-winning tech.
Read more: OpenAI has released a paid version of ChatGPT Professional