In recent years, the use of generative Artificial Intelligence (AI) tools has increased greatly and is expected to increase further over the upcoming years, particularly as the technology learns and gets better.
While AI platforms, such as ChatGPT can offer an array of advantages for users, including assisting with legal research, drafting documents, sourcing information and increasing efficiency, AI may not always be the best option as it often makes up its information such as cases and legislation.
It is crucial that people remain aware of the risks and limitations associated with AI usage when creating legal documents and obtaining legal advice or information.
How the AI Tools Function
It is important to understand how generative AI platforms operate. Generally, a user will ask the AI tool to provide an answer or response to a particular question and the AI tool will provide that answer in a one of a number of different forms (e.g. words, document, picture etc.).
However, a fundamental and well known issue with current AI tools is their ability to generate false, inaccurate and made-up responses, largely known as ‘hallucinations’, whereby the tool will formulate information or cite non-existent cases or commentary that gives you the answer it thinks you are seeking and that appears scarily on point with what you are looking for.
Inaccuracies and Misinterpretation
In a legal context, AI hallucinations can involve fabricating case law or providing incorrect legal information and advice. Alternatively, AI tools may cite legitimate cases or legislation but misinterpret the information within them and reach incorrect conclusions that the user then relies on as accurate.
The risks of relying on AI without careful human oversight have already been demonstrated a number of times in legal practice. Examples of which are set out below.
In the U.S. case of Roberto Mata v Avianca, a lawyer used ChatGPT to assist with legal research. Relying on the AI-generated content, the lawyer filed a court document citing several non-existent cases that AI had made up, later admitting that he had not previously used ChatGPT for legal research and was unaware that the tool could fabricate information. The court found the lawyer responsible for submitting misleading and erroneous material and fined the lawyer as a result.
A mayor from a Victorian council initiated legal action against ChatGPT after the platform falsely claimed he had been imprisoned for bribery. In reality, he had been a whistleblower in the bribery investigation and had never been charged with any crime. Although the inaccurate information was later removed, the case highlights the serious legal consequences that may arise from AI-generated misinformation.
An Australian lawyer has been referred to a state legal complaints commission, after he had used ChatGPT to write court filings in an immigration case and the AI made up case citations that did not exist and were relied upon by the lawyer.
Further, a Melbourne lawyer was referred to the Victorian legal complaints body after using AI software in relation to a family court matter. The AI platform had generated false case citations forcing the hearing to be adjourned and the lawyer referred to the complaints body.
Further, depending on what version of an AI platform you are using, AI knowledge is generally based on a largely limited timeframe of information (from and to certain dates), restricting its ability to consider recent legislative changes or new case law, posing further risks for individuals who rely on it without independently verifying the information.
Recognising these risks, ChatGPT acknowledges that its platform has many limitations they are working to address, including hallucinations, and states on the input page that “ChatGPT can make mistakes, check important info”.
Privacy and Confidentiality
Just as lawyers and businesses must take great care not to disclose confidential client information when using AI platforms, they must also be vigilant about protecting personal data.
In Australia, the collection, use and disclosure of personal data is regulated majorly by the Privacy Act 1988 (Cth). Businesses covered under this Act hold great responsibility for sensitive personal and client information and must ensure that their use of AI tools complies with these legal requirements.
Many AI products actively collect information about their users. As of August 2023, ChatGPT’s privacy policy indicates that it gathers data such as users’ IP addresses, browser details, and records of interactions with the platform. More concerningly, the policy states that OpenAI may share users’ personal information with third parties for business purposes, and such sharing may occur without direct notification to the user.
This raises significant risks for businesses. Sharing personal or client-related information, even unintentionally, could lead to breaches of privacy obligations, damage to client trust, and potential legal liability. The possibility that personal information could be accessed, stored, or disclosed by third parties outside the business’ control highlights the need for extreme caution in using these platforms.
Accordingly, the input of any personal, sensitive, or identifiable client information into AI platforms should be strictly avoided, unless there is certainty that the platform adheres to privacy and data security standards, and only with the informed consent of the client. Where possible, data should be anonymised to reduce possible risks.
What you can do?
While generative AI presents great benefit for increasing efficiency, its use for legal document creation and advice must be approached with caution to ensure that AI-generated content is accurate, ethically sound, and compliant with legal and privacy obligations.
In such a way, any AI-generated content must not be relied on blindly. We recommend that any information you get be closely reviewed, verified and compared to independent research to determine the accuracy of the information.
Alternatively, you can ask NECA Legal your legal query and we can provide advice accordingly.
Advice and Support
Should you have any questions or require further clarification, we encourage you to connect with CTI Lawyers Workplace Relations Specialists to get specific guidance and expert assistance on this matter. They are highly knowledgeable in these areas and will be able to provide you with the necessary support.
BIBLIOGRAPHY
- IBM (2023) What are ai hallucinations?, IBM. Available at: https://www.ibm.com/think/topics/ai-hallucinations (Accessed: 29 April 2025).
- Limitations and risks of using AI in legal practice (2023) Legal Practitioners’ Liability Committee. Available at: https://lplc.com.au/resources/lplc-article/limitations-risks-ai-in-legal-practice#:~:text=This%20article%20examines%20the%20limitations%20and%20risk%20strategies,capacity%20to%20generate%20false%20responses,%20known%20as%20%E2%80%98hallucinations%E2%80%99. (Accessed: 29 April 2025).
- Mayers, L., Martin, S. and Rybicki, D. (2023) Victorian mayor May Sue OpenAI after chatgpt ‘accuses’ him in bribery case, ABC News. Available at: https://www.abc.net.au/news/2023-04-06/hepburn-mayor-flags-legal-action-over-false-chatgpt-claims/102195610 (Accessed: 29 April 2025).
- Roberto Mata v Avianca, Inc. No. 1:2022cv01461.
Was this article helpful?