What was hallucinated

Fabricated: Case Law | Applicant’s BBM SFIC contained citations to non-existent cases in paras 35–37, later withdrawn before hearing. || Misrepresented: Legal Norm | IISA framed the test as requiring “new scientific or technical knowledge” and that the new knowledge be “based on principles of established science,” which the Tribunal corrected as not the statutory test. || Misrepresented: Legal Norm | IISA asserted hypotheses must identify technical knowledge gaps with “causal relationships between technical variables…,” which the Tribunal said is not the statutory requirement. || Misrepresented: Legal Norm | IISA stated that meeting observation and evaluation requires “analysis of numerical data using established statistical techniques,” which the Tribunal found is not required by the statute. || Misrepresented: Legal Norm | IISA argued there is a legal requirement to keep documentation and contemporaneous records to satisfy s 355-25; Tribunal held no such statutory requirement exists (documentation may be expected but is not mandated). || False Quotes: Doctrinal Work | Applicant attributed a sentence to the ManUp study that does not appear in the article; Tribunal verified the quote was not in the paper.

Details

"Nevertheless, due to that withdrawal being requested prior to the hearing, I have not considered those paragraphs, these reasons for decision do not take account of those paragraphs and I merely make some general comments below applicable to all parties that appear before the Tribunal.The use of Chat GPT is problematic for the Tribunal. It perhaps goes without saying that it is not acceptable for a party to attempt to mislead the Tribunal by citing case law that is non-existent or citing legal conclusions that do not follow, whether that attempt is deliberate or otherwise. All parties should be aware that the Tribunal checks and considers all cases and conclusions referred to in both parties’ submissions in any event. This matter would have inevitably been discovered, and adverse inferences may have been drawn. To ensure no such adverse inferences are drawn, parties are encouraged to use publicly available databases to search for case law and not to seek to rely on artificial intelligence."

Sanction teardown · Administrative Review Tribunal, Australia · 2025-01-24

Body by Michael Pty Ltd and Industry Innovation and Science Australia

What happened

In Administrative Review Tribunal, Australia, a filing relied on ChatGPT to help draft legal argument. The court identified the following problems with the citations in that filing:

Fabricated (Case Law)

Applicant’s BBM SFIC contained citations to non-existent cases in paras 35–37, later withdrawn before hearing.
Misrepresented (Legal Norm)

IISA framed the test as requiring “new scientific or technical knowledge” and that the new knowledge be “based on principles of established science,” which the Tribunal corrected as not the statutory test.
Misrepresented (Legal Norm)

IISA asserted hypotheses must identify technical knowledge gaps with “causal relationships between technical variables…,” which the Tribunal said is not the statutory requirement.
Misrepresented (Legal Norm)

IISA stated that meeting observation and evaluation requires “analysis of numerical data using established statistical techniques,” which the Tribunal found is not required by the statute.
Misrepresented (Legal Norm)

IISA argued there is a legal requirement to keep documentation and contemporaneous records to satisfy s 355-25; Tribunal held no such statutory requirement exists (documentation may be expected but is not mandated).
False Quotes (Doctrinal Work)

Applicant attributed a sentence to the ManUp study that does not appear in the article; Tribunal verified the quote was not in the paper.

Which AI tool

ChatGPT. Note: Charlotin's public database records tool attribution only where a court order, brief, or reporting on the matter states it explicitly; "unidentified" or "implied" means the record indicates AI use but does not name a specific product — we do not guess.

Outcome

Fake references withdrawn before the hearing

Additional detail

"Nevertheless, due to that withdrawal being requested prior to the hearing, I have not considered those paragraphs, these reasons for decision do not take account of those paragraphs and I merely make some general comments below applicable to all parties that appear before the Tribunal.The use of Chat GPT is problematic for the Tribunal. It perhaps goes without saying that it is not acceptable for a party to attempt to mislead the Tribunal by citing case law that is non-existent or citing legal conclusions that do not follow, whether that attempt is deliberate or otherwise. All parties should be aware that the Tribunal checks and considers all cases and conclusions referred to in both parties’ submissions in any event. This matter would have inevitably been discovered, and adverse inferences may have been drawn. To ensure no such adverse inferences are drawn, parties are encouraged to use publicly available databases to search for case law and not to seek to rely on artificial intelligence."

How Citation Safe would have caught this

Citation Safe runs three deterministic layers before a brief is filed: (1) does the citation exist against CourtListener's database of published opinions, (2) if quoted, does that exact language appear in the source, (3) does the cited case actually support the proposition it is cited for. Fabricated case citations fail Layer 1. Fabricated or misattributed quotations fail Layer 2 even when the underlying case is real. Misrepresented holdings — a real case cited for a proposition it does not support — are the target of Layer 3. None of these checks involve asking another language model whether the citation looks right; they are lookups and text-matches against the actual source, which is why a hallucinated citation has to survive a direct lookup against the authoritative source — not another model's opinion — to earn a VERIFIED stamp; our measured false-verify rate is published live at /quality.

Check a brief before you file it → · See our live false-verify rate

Source: https://www.damiencharlotin.com/documents/543/Industry_Innovation_and_Science_Australia_Australia_24_January_2025.pdf, via Damien Charlotin's public AI Hallucination Cases Database (CC0).