Testing Legal AI Chatbots with Quiz Questions

How far can we trust a legal AI assistant when it has to give a precise answer to a concrete legal question? Recently I set out to test the most promising legal AI platforms. I worked with a comprehensive dataset of 181 questions to get a realistic picture of what these models can do in a real-world situation.

The test subjects were AZTA AI, Libra AI and Bryter AI (BEAMON). The goal was not only to find out how smart these systems are, but also how stable they stay under automated testing.

The results at a glance

Bryter AI performed best, with Libra AI a close second. Both answered the legal quiz questions clearly more accurately than AZTA AI, which fell well behind. Consistency showed a similarly sharp difference: Bryter and Libra almost always gave the same answer when a question was asked again. AZTA AI’s answers were far less reliable, and frequent freezes made testing it harder.

• Accuracy: the share of correct answers among all questions asked.
• Consistency: how often a model gave the same answer to the same question across repeated runs.
• Error rate: the share of cases where the system froze or returned no answer because of another technical problem.

How we tested

The test used a set of 181 quiz questions compiled by ELTE. The questions were asked in 12 runs in total: 2 for AZTA AI, 5 for Libra AI and 5 for Bryter AI. I automated the process with Python’s playwright package, which submitted the questions and collected the answers through each platform’s web interface, the way a real user would.

During the tests I discovered several important technical limits. First, it is not worth sending more than 20 questions in a single session, because memory handling or the context window makes the systems start behaving oddly. On top of that, for the sake of stability it paid to leave a 10–15 second pause between questions so the AIs would not be overloaded.

Since I was working with legal quiz questions, fast and unambiguous answers mattered. This is where prompt engineering came in: with AZTA AI, for example, it was very useful that a short prompt could customise the quality of the answer. To maximise speed, I specifically asked the model to return only the letter of the correct answer, supplemented by a short justification of at most one or two sentences.

This promised a fast and efficient process in theory, but in practice AZTA AI turned out to be the worst-performing model. It responded extremely slowly to the questions, and it froze on a given question remarkably often, with the interface simulating an endless loading screen. Meanwhile, the other two AIs (Libra and Bryter) stayed very stable, and I very rarely ran into this kind of interruption with them.

Detailed results

Accuracy

The first chart immediately shows the gap between the models in the share of correct answers. Bryter AI stands out with a result of over 90%, and Libra AI is also in a strong band above 80%, while AZTA AI lags significantly behind at under 60%. The chart shows clearly which models are worth considering if accuracy is the main criterion.

Consistency

Answer consistency shows a similar trend. Asked the same question repeatedly, Bryter AI gave the same answer almost every time, and Libra AI was not far behind. AZTA AI’s answers changed from run to run far more often.

Error rate

Bryter AI and Libra AI both have a negligible error rate below 1%. AZTA AI, by contrast, froze or failed to answer because of other technical problems in more than 32% of cases. In practice, that means manual intervention for about one question in three.

Price and value

Taking prices into account, the more expensive platforms (Libra, Bryter) clearly provide a more stable and more accurate service than the cheaper AZTA AI.

Accuracy figures use the corrected answer key (see below).

Prices are per user, converted to forints at the current exchange rate and rounded so they can be compared. The plans differ mainly in the number of questions that can be asked and in extras such as central user management or teamwork support. The AI chat itself is the same across all of a platform’s plans.

Where every model failed – or did it?

Beyond the statistics, it is worth looking at the cases that tripped up even the best-performing models. According to the original quiz key, there were 4 questions that none of the AIs answered “correctly” in any of the 12 runs. We checked them against the original quiz cards and the law in force, and it turned out that the models were at fault in only half of the cases:

• On 2 questions (#70, #153) the models really were wrong, all three in the same way: once by citing a rule that does not exist, once by reading a legal term in its everyday sense.
• On 2 questions (#78, #165) the quiz’s answer key is wrong: the models gave the legally correct answer.

The accuracy figures in this article already use the corrected answer key. The correction raised each model’s result by 0.8 to 1.6 percentage points but did not change the ranking. There are two lessons. The models can indeed be wrong, sometimes all in the same way, but an AI test is only as reliable as its answer key, so “shared failures” are always worth double-checking.

Conclusion

The experience and the data show that stability and accuracy come at a price in the legal AI market. A cheaper solution such as AZTA AI can look tempting on paper. During testing, however, its high error rate and annoying freezes caused far more trouble than its price advantage was worth. The automated tests made one thing clear: if you need a reliable tool, the premium tools (above all Bryter AI and Libra AI) currently hold up well.

Even the best model, though, got roughly one question in ten wrong, and it often repeated its wrong answers as consistently as its right ones. Legal AI tools can be a real help, but they cannot replace expert review of their answers.