January 22, 2024
You may be surprised to find out that yes, ChatGPT passed the Bar Exam – and within the 90th percentile. Weirdly enough, you read that correctly. ChatGPT “passed the multiple-choice portion of the exam and both components of the written potion” of the Uniform Bar Exam (UBE) with a 298 out of 400.1 In fact, ChatGPT actually “exceed[ed]…the average score of real-life bar exam takers.”2 So, if you are just finding out that ChatGPT beat your bar passage score, you are not alone.
For those of you who do not know, ChatGPT is a chatbox, powered by Artificial Intelligence (“AI”), that responds to open-ended text inquiries. ChatGPT was trained through reinforcement learning, which allows an AI system to learn through trial and error. The AI system then receives feedback, which may either be positive or negative.
Reinforcement learning is similar to standard supervised learning in that it utilizes punishment vs. reward, with the ultimate goal of maximizing the reward. This, however, is the only attribute the two have in common. Reinforcement learning, unlike standard supervised learning, does not receive feedback in the form of the correct set of actions/answers. Instead, in reinforcement learning “there is no answer key,” rather, the AI-powered system “decides what to do itself to perform the task correctly.”3 In other words, the AI-powered system mimics how we, as humans, learn. To that effect, the AI-powered system “seeks to acquire the best strategy for taking repeated sequential decisions…in a dynamic system…to learn the winning policy.”4
Now you may be wondering, can ChatGPT practice law? In the U.S., one cannot practice law without admission to the bar, which is in general similar throughout the fifty states. To that point, typically to be admitted to the bar, an applicant must be a resident (or employed), have good moral character, have graduated from an accredited law school, and have passed the bar examination. ChatGPT, however, does not claim to be a resident, to have good moral character, or to have graduated from an accredited law school. ChatGPT also does not claim to be licensed to provide legal advice. To that end, when given a prompt to provide legal advice, ChatGPT answers with a generic message, stating:
“I’m sorry, but as an AI language model, I am not authorized to provide legal advice. Providing legal advice requires a thorough understanding of the specific laws and regulations in the relevant jurisdictions, as well as an assessment of the specific facts and circumstances of the situation. Only a licensed attorney who is authorized to practice law in your jurisdiction can provide you with legal advice. It’s important to consult with a qualified attorney who can provide you with personalized legal advice based on your specific needs and circumstances.”
If you’re feeling a sense of relief, you should be. ChatGPT expressly states that “[o]nly a licensed attorney who is authorized to practice law in your jurisdiction can provide you with legal advice.” ChatGPT further advises that the user should “consult with a qualified attorney.” Unfortunately, this sensation will not last long. This is because ChatGPT responds differently depending on the prompt. In other words, ChatGPT can, and will, perform lawyerly tasks.
ChatGPT, unlike previously artificially developed intelligence, can perform legal research and review contracts. ChatGPT can also prepare discovery, draft legal documents, develop a legal strategy, and perform brief analyses. Sadly, what was once revered as attorneys’ professional responsibilities, is now ChatGPT’s capabilities. That’s right, with a swipe of your mouse and finessing of the fingers, you can acquire an artificial esquire. But, at what cost?
As you may have heard, two New York attorneys recently utilized ChatGPT to prepare a filing and now may be facing possible punishment in the form of sanctions. This potential punishment, in case you were not aware, is no joke. It is more than just a slap on the wrist. Sanctions signify misconduct on behalf of the attorney, the client, or both, and may result in disqualification of counsel, dismissal of the action, entry of default, a fine, etc., depending on the misconduct that occurred. It is important to understand that the attorneys’ misconduct and successive sanctions are not the result of their mere usage of ChatGPT. Rather, it is the result of their usage of ChatGPT’s fabricated information. Yes, ChatGPT actually invented and supplied fake case law that was, subsequently, cited in a filing. Fortunately, the presiding judge caught wind of the falsities in the filing and acted accordingly. Unfortunately, this may not always be the case…
Now, you may be weary of ChatGPT and its ability to provide accurate information. And maybe you should be. Technological tools, once previously used to enhance the legal profession, may now be an inherent danger. Whether the ultimate danger will be the replacement of human attorneys, we do not yet know. As with anything else, only time will tell.
1 https://www.businessinsider.com/list-here-are-the-exams-chatgpt-has-passed-so-far-2023-1#gpt-4-scored-in-the-90th-percentile-of-the-bar-exam-with-a-score-of-298-out-of-400-2
2 https://law.stanford.edu/2023/04/19/gpt-4-passes-the-bar-exam-what-that-means-for-artificial-intelligence-tools-in-the-legal-industry/
3 https://online.york.ac.uk/what-is-reinforcement-learning/#:~:text=Reinforcement%20learning%20(RL)%20is%20a,using%20feedback%20from%20its%20actions.
4 https://developer.ibm.com/learningpaths/get-started-automated-ai-for-decision-making-api/what-is-automated-ai-for-decision-making