What Inspires Us?

In order to make beautiful things, every craft starts with raw ingredients. Furniture from wood. Orchards from soil. Pottery from clay. CERN from metals and plastics. We are inspired by the crafts that transform raw, unstructured elements into beautifully useful things.

Our raw ingredients are not soil, sunshine, and silica per se, but chattering robot text from Large Language Models (LLMs) like ChatGPT.

At Rohe Nordberg Review, we craft prompts and processes to evaluate scientific manuscripts with these innocent chattering robots. We developed this craft starting in 2020 by creating an undergraduate statistics course on “statistical reading comprehension”; you can read more about this in our origin story.

We use many different robots/LLMs (e.g., ChatGPT and others). Our prompts and processes are developed to foster discussions among them (far easier than promoting classroom discussion!). We forge and polish their raw text into rigorous evaluations of scientific manuscripts.

Hand-blown glass, ceramics, glazes, and many other crafts start with raw ingredients that have unavoidable inconsistencies and thus randomness baked into the process. Good crafting controls these inconsistencies and releases the nature that is both useful and beautiful. In the same way, robot text is often innocent nonsense, inconsistent and random. Our work is to craft it.

Traditionally, statisticians’ craft is to create valid inferences from noisy experiments. We build on this tradition in multiple ways. First, we rely on replicates, asking many different robots the same questions multiple times. Then, we create conversations among them to identify the outliers, the residuals, the robot text that does not seem to agree with the others. Sometimes the outlier-robot-text makes a surprising insight and sometimes it is innocently chattering. We’ve had success coaxing the robots into telling the difference between insights and chattering. We craft these conversations to control and release the robot text into something that we hope you will find both useful and beautiful.

Just as statistical inferences are good guesses based upon noisy data, so are ours. We rigorously calibrate and validate our outputs. We are continually crafting the prompts and conversational processes to smooth over imperfections before they reach your eye. If and when you inevitably find a rough edge—either something wrongly stated or maybe something not-quite-right—we apologize and hope you will share that with us so that we can continue to refine our processes.

We make this for the experts (the scientists, statisticians, doctors, and editors) who have little patience for low-quality science. We trust that they can tell the difference in our craft. To them, we offer our reports as tools, so they may chisel and hammer a better science and a brighter future.