What does ChatGPT really know about physics?

The art of conversation may be dead, but you wouldn’t know it from the attention being paid to ChatGPT. It’s not a person with sparkling repartee, but a chatbot that accesses a terabyte-sized database of words drawn from the Internet. Type in anything and ChatGPT (using an AI language model called Generative Pre-trained Transformer 3) selects the words that are most likely to follow. It then forms thematically linked words into responsive, coherent and grammatical sentences and paragraphs.

ChatGPT was put online in November 2022 for people to try. Having had a go myself, I’ve found the real-time experience is like texting with a person, enhanced by our human tendency to project personality and consciousness onto things that have neither. As ChatGPT itself will tell you, it is not sentient, although its facility with language can make you think you’re engaging with a conscious being.

But is the algorithm that powers ChatGPT able to present scientific ideas and facts with accuracy?

Informal testing by the millions of people who have tried out ChatGPT suggests that it could be used for educational purposes even if unscrupulous users might present its work as their own. But is the algorithm that powers ChatGPT able to present scientific ideas and facts with accuracy? How, in other words, does ChatGPT cope with a subject as specialized and complex as physics?

When I asked it, for example, what goes on inside a black hole, I was told the situation is “extremely extreme” and was then given accurate information about its density, the event horizon and other physical properties. In response to the question “What did LIGO measure?” I was given a long detailed answer about gravitational waves and LIGO’s facilities.

I also tried a few higher-level questions, first asking ChatGPT to tell me the main problem in combining general relativity with quantum physics. “They are based on fundamentally different principles,” I was told. “General relativity…describes gravity as the curvature of space–time…Quantum physics…describes the behaviour of matter and energy at the subatomic level…they make different predictions…in certain regimes.”

To the query “Is condensed-matter physics the same as solid-state physics?” ChatGPT replied that, unlike solid-state physics, condensed-matter physics includes research on liquids and complex fluids, and described the properties studied. I even got a balanced and meaningful response when I asked it if “understanding dark matter is more important than having artistic ability”, being told, in part, that it “depends on one’s personal values and goals”.

If real physics undergraduates had handed in these mini essays, I would have rated them as more than acceptable

I found no misinterpretations of physical concepts in any of these answers. Except for the catchy phrase “extremely extreme”, the writing was not particularly lively although it was perfectly clear and logically laid out. Overall, if real physics undergraduates had handed in these mini essays, I would have rated them as more than acceptable (had I not known where they came from).

ChatGPT also responds to equations. When I asked if “F=mv” and “R=VI” are correct, I was informed that these are the wrong forms of Newton’s and Ohm’s laws and was given the right equations and an explanation for them. Entering “E=mc²” generated a response that correctly identified the equation, but mistakenly stated that a small energy can be converted into a large mass.

Similarly, there are reports that ChatGPT isn’t so good at turning problems given in written form into mathematics, even if it can numerically solve problems in algebra and calculus. What’s more, ChatGPT doesn’t always get its facts right. When I entered my name, for example, it accurately described my writing and career, but got a book title wrong and granted me two awards I have never received.

ChatGPT did well, however, in expressing the lighter side of physics. When I asked it to tell me a physics joke, it replied: “Why don’t scientists trust atoms? Because they make up everything!” It can even do poetry. When I asked it to put the three laws of thermodynamics into verse, it came up with four stanzas. They weren’t great, but they rhymed and got the idea across. Here’s the first one:

The first law’s a balance that we must hold,
Energy’s constant, it can’t be bought or sold.
It flows like a river, in and out,
A never-ending cycle, without a doubt.

Getting more confident, I decided to ask what I thought was a trick question. “Could one build LIGO with LEGO?” The response: “It is unlikely…LIGO is a highly complex scientific instrument [that would be] difficult, if not impossible, to achieve using LEGO…LIGO requires advanced technology, such as lasers…that would not be available in a LEGO set.” I could almost believe that ChatGPT was slyly telling me how dumb I am. But no, the algorithm was just following its rules and being literal.

My adventures with ChatGPT show that it can deliver valid physics knowledge and lore in readable form but needs human oversight before its output can be fully trusted. Real human feedback to its responses also helps train the ChatGPT language model, while its Internet database arises from human knowledge and wisdom, so the chatbot inevitably reflects our own biases and errors. Perhaps the objectively verifiable facts and theories of science mean that chatbots can be improved more quickly here than in fields where answers are less definite or more controversial.

In 1950 when Alan Turing first wrote about thinking machines, he reckoned that it would be as difficult to train a machine as it is to educate a child. That may be so, but I think, with ChatGPT, we have taken the first steps in machine education.