Accuracy & Method

Physicstan does not publish an accuracy percentage. This page explains why, what the system actually checks, what it cannot check, and how to use it without getting burned.

If you read nothing else: read the reasoning, not just the final number. The reasoning is where a wrong answer becomes visible.

Why there is no accuracy figure

Competing physics AI sites advertise 98.5% and 99.7% accuracy. Neither states a test set, a scoring method, a sample size, or who did the testing. A percentage without those is not a measurement.

Accuracy also is not one number. It varies enormously by problem type. The published research on general-purpose models makes this concrete. Researchers at Stanford and Auburn ran GPT-4 against forty problems from a college engineering physics course. On well-specified problems, where every needed value appeared in the question, it solved 62.5% correctly. On under-specified problems, where a quantity had to be estimated, it managed 8.3%. Same model, same subject, an eightfold difference depending on how the question was framed (Wang, Burkholder, Wieman, Salehi & Haber, Frontiers in Education, 2024).

A single headline number would hide exactly the variation that matters to you.

The three ways physics AI fails

The same study identified three distinct failure modes, and they are worth knowing by name because they fail differently and you catch them differently.

1. Building the wrong model of the situation. The AI misreads the physical setup: wrong reference frame, a force that should not be there, the wrong conservation law, an incline treated as flat. The algebra downstream is then flawless and useless. This is the most dangerous failure because the working looks impeccable.

2. Unreasonable assumptions about missing data. When a problem does not supply everything needed, something has to be assumed. A bad assumption, made silently, produces a confident wrong answer.

3. Calculation errors. Language models predict text. They do not natively perform arithmetic. Experienced tutors report the same pattern constantly: correct formula, correct rearrangement, wrong number.

Physicstan is built around these three specifically.

What the system does about each

Against wrong models

Every answer is required to state its interpretation of the problem before doing anything else, along with the physical principle it is applying and why that principle applies here. The setup is separated from the algebra so you can check it independently.

The model is instructed to choose the simplest valid model and to explain the choice, to preserve units when extracting known quantities, and to check sign conventions and perform a physical sanity check where one is meaningful.

There is a related rule that matters: dimensional consistency is not treated as proof that a physical model is correct. An answer can have the right units and still describe the wrong physics.

Against bad assumptions

Assumptions appear in a dedicated block in every answer, written out plainly, where you can see and reject them. A hidden assumption is how a wrong answer passes inspection.

The model is instructed never to invent a missing numerical value. If a problem cannot be solved as written, it should say which quantity is missing rather than supplying a plausible one. Similarly, if a photo or handwriting is ambiguous, it should state what it thinks the problem says and ask you to confirm rather than guessing and solving a different problem.

If you ever see it invent a value, please report that answer. That is a rule violation, not expected behaviour.

Against calculation errors

This is the part that runs outside the AI entirely.

When the model produces arithmetic that can be independently verified, it submits that calculation to our server as a check request. Our backend then evaluates the expression using a purpose-built parser that has nothing to do with the language model: a tokeniser, a shunting-yard conversion, and a stack evaluator supporting numbers, the four operations, exponentiation and parentheses. No AI is involved in that evaluation, and the code deliberately does not use PHP’s `eval`.

The result is compared against what the model claimed, within a tolerance. Three outcomes are possible and all three are displayed honestly:

• Passed. The independent calculation agrees.
• Failed. It does not agree. You see a mismatch warning, not a hidden correction.
• Not completed. The expression was outside what the checker supports, so no claim is made either way.

The same applies to unit dimensions. The backend maps a stated unit to its dimensional class from a fixed internal library, covering metres, seconds, kilograms, newtons, joules, watts, pascals, hertz, coulombs, volts, ohms, teslas and the common velocity and acceleration forms, and compares it against the dimension the answer expects. Units outside that library return “not in the current verified unit library” rather than a false pass.

The AI is explicitly forbidden from declaring its own answer verified. The instruction is that it must never claim independent verification unless the backend supplies that evidence, and that the backend decides which checks actually pass. Anything you see marked as an independent check is a check that genuinely ran on our server.

What is not verified

Being clear about the boundary is the point of this page.

• The physics. No automated check confirms that the chosen principle, model or approach is correct. A wrong method that is arithmetically clean will pass every check we run.
• Anything outside the checker’s scope. Arithmetic with functions, symbolic algebra, calculus and unit-bearing expressions beyond the fixed library are not independently evaluated.
• Diagram interpretation. If the AI misreads your photograph, nothing downstream catches it. This is why it is instructed to confirm ambiguous images with you.
• Whether your specific course expects a particular method. Instructors differ on conventions and required working.
• Advanced and specialised material. Reliability drops as you move past introductory university level. For upper-division, highly specialised or research-level work, treat output as a study aid and verify against your course materials.

Where the physics methodology comes from

Physicstan is not a new foundation model. It specialises a strong provider model through protected physics system instructions, a fixed problem-solving workflow, a structured answer contract, deterministic backend checks, and a benchmark test-case library.

The methodology and product behaviour are designed around the work of a physics PhD researcher, and the attribution shown in the application names the individual responsible.

On the bundled dataset, an honest statement of current status: the plugin ships an original seed set of benchmark-style physics questions, written for this purpose rather than copied from textbooks, with topic coverage aligned to openly licensed references including OpenStax. Those cases are marked pending review by default and are not enabled as few-shot examples until a physics specialist has reviewed each one. We are not claiming expert-verified answer quality on the basis of a dataset that has not completed review. When that changes, this page will say so.

Reporting an error

Every AI answer carries a thumbs up, a thumbs down and a Report button. Report opens a short comment box, and reports reach us with the message and request identifiers attached.

This is the most useful thing you can do with a wrong answer. A reported error with a description is what turns into a benchmark case and a fix. Feedback with no detail is much harder to act on, so a sentence about what was wrong is worth writing.

You can also push back in the conversation itself. Ask it to re-derive, use a different method, or justify a step you doubt. If an answer looks wrong to you, that instinct is usually worth following.

How to use this well

1. Read the assumptions block first. More wrong answers are caught here than anywhere else.
2. Check the setup before the algebra. If the interpretation is wrong, nothing after it matters.
3. Treat a failed check as a stop sign. If the independent arithmetic check reports a mismatch, the number is not trustworthy.
4. Treat a passed check narrowly. It means the arithmetic is right, not that the physics is.
5. Verify anything that affects a grade against your textbook, your notes or your instructor.
6. Use Tutor mode when you have time. Being guided to an answer you produce yourself removes the reliance question entirely.
7. Cover the solution and redo the problem. If you cannot, you have not learned it, whatever the mark says.

Related

• Physics AI Solver for the main tool
• Physics AI Tutor for guided help
• Physics homework help for assignments
• Pricing for credits and plans

Common questions

So is it accurate or not? It is reliable on standard high school and introductory university problems and progressively less so as problems get more specialised or more open-ended. That is the honest answer, and any site giving you a single percentage instead is not measuring anything.

Which AI model is behind it? Physicstan runs on a commercial provider model, configurable by the site administrator, wrapped in physics-specific instructions and our own verification layer. The wrapper and the checks are what differentiate the output, not exclusive access to a model.

Does it ever say it does not know? Yes. If essential information is missing it returns a clarification rather than a solution, and it should tell you exactly which quantity it needs.

Why did it ask me to confirm what my photo said? Because the image was ambiguous. Confirming is much better than silently solving a different problem, particularly under deadline.

What does “unit dimension not in the current verified unit library” mean? The unit fell outside the fixed set the backend can check. It is a statement that no check was performed, not a statement that anything is wrong.

Can I trust it for exam preparation? For practice and understanding, yes, with the verification habits above. Do not use it during a graded assessment unless your instructor has explicitly permitted outside tools.

What happens to a wrong answer I report? It reaches us with the conversation context and feeds the benchmark library used to test changes.