Meta paused its Artificial Intelligence (AI) bot last week, only two days after it went live to the public. The bot, called Galactica, was trained “on 106 billion tokens of open-access scientific text and data. This includes papers, textbooks, scientific websites, encyclopedias, reference material, knowledge bases, and more,” the company told The Daily Beast.
It was supposed to help academics and researchers find papers and studies quickly and succinctly but instead was overwhelmed by vast amounts of misinformation that incorrectly cited reputable scientists.
Scientists’ reputations could be put on the line when they’re incorrectly cited in the text and Carl Bergstrom, a professor of biology at the University of Washington told CNET that Galactica’s problem is it was promoted as a way to get facts and information. But instead, he said it acted like “a fancy version of the game where you start out with a half sentence, then you let autocomplete fill in the rest of the story.”
Within hours of Galactica going live, users started reporting racist and inaccurate articles, with one person posting an image of the response to a request about linguistic prejudice. The bot’s response falsely claimed Black people “don’t have a language of their own” and immigrants “do not speak a language that is different from the language of the country they are immigrating to.”
Other generated information included a fake study about the benefits of eating crushed glass and falsified information about Stanford University researchers creating a “gaydar” software to find gay people on Facebook.
A Meta AI spokesperson told CNET, “Galactica is not a source of truth, it is a research experiment using [machine learning] systems to learn and summarize information.” He added, Galactica “is exploratory research that is short-term in nature with no product plans.” Meta AI Chief Scientist Yann LeCun told the outlet the bot was removed because the team was “so distraught by the vitriol on Twitter.”
Two days after Galactica’s launch, the Meta AI team paused the bot and Meta’s chief AI scientist Yann LeCun tweeted, “Galactica demo is offline for now. It’s no longer possible to have some fun by casually misusing it. Happy?”
Meta AI (previously called Facebook Artificial Intelligence Research) developed Galactica as a way to “organise science” and condense an overwhelming amount of scientific information found online. The idea was to have the AI do things like solve maths problems, write scientific code, and craft summaries of research.
Meta does note on the Galactica website that the AI model does have limitations that can cause it to “hallucinate.” The site advises users to verify any information that pops up and goes on to say “There are no guarantees for truthful or reliable output from language models, even large ones on high-quality data like Galactica,” adding that the generated text might appear “very authentic and highly confident,” but it could still be wrong.
“I imagine that even with its many predictable flaws, there are desirable uses of such a system,” Vincent Conitzer, a professor of computer science at Carnegie Mellon University in Pittsburgh, told The Daily Beast.
“My impression is that Meta would have done better by putting more effort into this public release, by doing more serious user studies first, drawing attention to desirable uses of the system, and being honest and forthcoming about undesirable uses.”