What was hallucinated

Fabricated: Case Law || Misrepresented: Case Law || False Quotes: Case Law || Outdated Advice: Repealed Law || Fabricated: Legal Norm

Details

""101 The use of AI technology in the Courts has been the subject of judicial observations, particularly regarding legal practitioners who are subject to professional and ethical obligations and responsibilities. However, as Bell CJ observed in May v Costaras [2025] NSWCA 178 at [15], with whom Payne JA and McHugh JA agreed, in the context of considering the use of AI in the preparation of submissions, that “(a)ll litigants are under a duty not to misled the court or their opponent.” The reliance on unverified materials produced by generative AI does have the potential to misled the Court.102 Although the termed used in relation to erroneously generated references by AI is “hallucinations”, this is a term which seeks to legitimise the use of AI. More properly, such erroneously generated references are simply fabricated, fictional, false, fake and as such could be misleading.103 All persons appearing before the Court have a duty to verify that the case law and legislation referred to and relied on, is accurate and that such materials actually exist. The references in Ayinde at [85] and [86] and in Costaras at [14]-[15], to matters involving litigants who are acting in person who rely on AI generated material clearly supports the position that all are required to verify the submissions made to the Court. There are many publicly available legal research websites, some which are accessible without a fee. Further, and without attempting to be exhaustive, the Queensland Supreme Court library is open and available to the public.104 The use of generative AI to prepare submissions that may include fake authorities will nearly always introduce added costs, complexity and add to the burden of other parties and to the Court: Costaras at [16] and [49]. I gratefully adopt the observation from Ayinde at [9] that “(t)here are serious implications for the administration of justice and public confidence in the justice system if artificial intelligence is misused.” The Court in Ayinde observed from [10] to [31] the existing guidance, regulatory duties of the profession, referrals and the Court’s powers. Matters which are within the Court’s own domain include in the most serious of cases contempt of Court. The observations in Ayinde at [26]-[28] regarding contempt of court are not limited to legal practitioners.105 As the reasons above demonstrate, the circumstances of this case involved many fake authorities, fabricated quotes and false propositions. It is unhelpful for the Court to be referred to, and for parties to rely on, such matters."

Sanction teardown · Federal Court of Australia, Australia · 2025-08-19

JML Rose Pty Ltd v Jorgensen (No 3)

What happened

In Federal Court of Australia, Australia, a filing relied on an unnamed/unconfirmed AI tool to help draft legal argument. The court identified the following problems with the citations in that filing:

Issue

Fabricated: Case Law
Issue

Misrepresented: Case Law
Issue

False Quotes: Case Law
Issue

Outdated Advice: Repealed Law
Issue

Fabricated: Legal Norm

Which AI tool

an unnamed/unconfirmed AI tool. Note: Charlotin's public database records tool attribution only where a court order, brief, or reporting on the matter states it explicitly; "unidentified" or "implied" means the record indicates AI use but does not name a specific product — we do not guess.

Outcome

N/A

Additional detail

""101 The use of AI technology in the Courts has been the subject of judicial observations, particularly regarding legal practitioners who are subject to professional and ethical obligations and responsibilities. However, as Bell CJ observed in May v Costaras [2025] NSWCA 178 at [15], with whom Payne JA and McHugh JA agreed, in the context of considering the use of AI in the preparation of submissions, that “(a)ll litigants are under a duty not to misled the court or their opponent.” The reliance on unverified materials produced by generative AI does have the potential to misled the Court.102 Although the termed used in relation to erroneously generated references by AI is “hallucinations”, this is a term which seeks to legitimise the use of AI. More properly, such erroneously generated references are simply fabricated, fictional, false, fake and as such could be misleading.103 All persons appearing before the Court have a duty to verify that the case law and legislation referred to and relied on, is accurate and that such materials actually exist. The references in Ayinde at [85] and [86] and in Costaras at [14]-[15], to matters involving litigants who are acting in person who rely on AI generated material clearly supports the position that all are required to verify the submissions made to the Court. There are many publicly available legal research websites, some which are accessible without a fee. Further, and without attempting to be exhaustive, the Queensland Supreme Court library is open and available to the public.104 The use of generative AI to prepare submissions that may include fake authorities will nearly always introduce added costs, complexity and add to the burden of other parties and to the Court: Costaras at [16] and [49]. I gratefully adopt the observation from Ayinde at [9] that “(t)here are serious implications for the administration of justice and public confidence in the justice system if artificial intelligence is misused.” The Court in Ayinde observed from [10] to [31] the existing guidance, regulatory duties of the profession, referrals and the Court’s powers. Matters which are within the Court’s own domain include in the most serious of cases contempt of Court. The observations in Ayinde at [26]-[28] regarding contempt of court are not limited to legal practitioners.105 As the reasons above demonstrate, the circumstances of this case involved many fake authorities, fabricated quotes and false propositions. It is unhelpful for the Court to be referred to, and for parties to rely on, such matters."

How Citation Safe would have caught this

Citation Safe runs three deterministic layers before a brief is filed: (1) does the citation exist against CourtListener's database of published opinions, (2) if quoted, does that exact language appear in the source, (3) does the cited case actually support the proposition it is cited for. Fabricated case citations fail Layer 1. Fabricated or misattributed quotations fail Layer 2 even when the underlying case is real. Misrepresented holdings — a real case cited for a proposition it does not support — are the target of Layer 3. None of these checks involve asking another language model whether the citation looks right; they are lookups and text-matches against the actual source, which is why a hallucinated citation has to survive a direct lookup against the authoritative source — not another model's opinion — to earn a VERIFIED stamp; our measured false-verify rate is published live at /quality.

Check a brief before you file it → · See our live false-verify rate

Source: https://www.damiencharlotin.com/documents/717/JML_Rose_Pty_Ltd_v_Jorgensen_Australia.pdf, via Damien Charlotin's public AI Hallucination Cases Database (CC0).