State Bar did not mention AI in contracts for February exam

Revelations that the State Bar's contract with ACS Ventures did not restrict the use of artificial intelligence roiled the California legal community on Tuesday. Experts say that clauses governing the use of AI should be a standard part of contracts--particularly when the deliverables include written content or other intellectual property.

"It is increasingly common to have clauses in contracts specifically governing the use of AI, although it is context specific," Ryan Abbott, a partner, Brown, Neri, Smith & Khan, LLP in Los Angeles, responded in an email. "For instance, if a contract is for a vendor to create new intellectual property, it may be important to have rules around the use of AI in the processes because US law currently largely fails to protect intellectual property rights in AI-generated content. Similarly, it is increasingly common to have rules around the use of provided content for AI training."

On Tuesday morning, Bloomberg reported the bar's contracts with ACS did not mention AI at all. The company was hired to serve as a psychometrician for the exam, charged with evaluating questions on the new California-only exam for accuracy and difficulty. But when the bar found it did not have enough multiple-choice questions available from Kaplan Test Prep and the First-Year Law Students' Examination, ACS provided an additional 29 questions.

Last month, in a petition to the California Supreme Court requesting permission to adjust the passing scores for the scandal-plagued exam, the bar admitted ACS used ChatGPT in crafting the questions. In response to the court's request for clarification on the use of AI in writing the exam, the bar also disclosed the ACS questions were disproportionately likely to have been flagged for typos or incorrect facts.

"While AI-specific clauses are not commonplace at this time, the need is growing," said Jeremy M. Evans, CEO of California Sports Lawyer in Los Angeles, in an email. Evans chaired the California Lawyers Association Task Force on Artificial Intelligence, but emphasized he was speaking only in his personal capacity and not for the association.

He added, "In general, the proper use of AI mainly includes the following when using generative artificial intelligence platforms: (1) not disclosing confidential, private, intellectual property, or privileged information, (2) have mandatory human review of inputs and outputs, and (3) disclosure where required or appropriate. The use of AI is not a concern, it is how the AI is used." 

San Francisco School of Law professor Katie Moran has been a consistent critic of the bar since the February exam was plagued with both technical issues and issues around its content. She said the bigger problem was the ACS was crafting questions at all.

"I've read all of the contracts between ACS and the bar," Moran said. "The contracts either say ACS would do exam evaluation or review; the more recent contracts include exam development."

She added, "If the State Bar contemplated that ACS Ventures would write multiple-choice questions for the bar exam, it would be a huge oversight. But if the bar never planned for their own psychometric company to draft questions, it is careless for another reason: lack of expertise."

The contracts show that ACS Founding Partner Chad Buckendahl signed a series of agreements and work orders with the bar between late 2023 and early 2025, calling for a total of $502,000 in payments. Chief Administrative Officer Steve Mazer or Chief Financial Officer Aracely Montoya-Chico signed off on each for the bar.

The initial agreement called for ACS to provide "psychometric services and analysis related to the California Bar Examination and the California First‐Year Law Students' Examination," and to perform "other research, advice services, and studies as requested by the State Bar, as well as training sessions."

None of the contracts contain any mention of artificial intelligence or explicitly state that ACS would write questions. They do contain provisions allowing the parties to change the scope or work. A deal signed in March, after the exam fiasco, called for payments of $204,000 for extensive evaluation of the exam and scoring.

"The decision to have ACS develop a small subset of questions was done to ensure there were a sufficient number of questions in all subtopics of the subject areas," the bar said in a statement shared by spokesman Rick Coca. "Regarding AI use, it was inappropriate for this to have occurred without levels of transparency and policy-level decision making. Structural changes have been made within Admissions to ensure this will not happen again."

The statement continued, "It is also important to note that all questions, including the small number (29 out of 200) developed by ACS, were reviewed by legal subject matter experts."