In August, a public school district in Iowa reportedly used ChatGPT to help it comply with the stateโs controversial book ban law. That lawโlike counterparts passed in Florida, Texas, Missouri, Utah, and South Carolinaโseeks to limit discussion of gender identity and sexuality in schools by barring school libraries from carrying books that touch on topics related to sexual content. Driven by a growing number of organized groups and political pressure from state lawmakers, such laws threaten studentsโ rights to access information and run the risk of chilling speech on topics such as sexuality, teen pregnancy, and sexual health. And using generative AI tools to implement these laws only compounds the problem.
Widely accessible generative AI chatbots like OpenAIโs ChatGPT and Googleโs Bard provide individuals free access to tools built on large language models. Earlier language models and other analogous automated tools have been used by businesses to detect spam, recommend products, and moderate online content. Years of experience in content moderation shows that these tools are simply not suited to making decisions based on the types of vague and subjective standards that characterize many book bans. Even for the narrower category of sexual content, AI tools are often overinclusive.
The Iowa school district sought to use ChatGPT only to scan books for whether they included โa description or visual depiction of a sex act,โ rather than the broader โage appropriatenessโ standard laid out in the law. According to Mason Cityโs Assistant Superintendent of Curriculum and Instruction Bridgette Exman, the school district chose to ask a narrower, โmore objectiveโ question: โDoes [insert book] contain a description or visual depiction of a sex act?โ
But even for this more limited type of query, these technologies have significant limitations. By now, it is well known that generative AI tools can produce unreliable results and fabricate plausible, but false, answers. ChatGPT and the like include disclaimers to this effect. In Mason City, ChatGPT wrongly identified three books as depicting sexual acts. These mistakes were caught when Exman reviewed its results, demonstratingโonce againโthe importance of human evaluation of the output of AI tools.
These tools are also inconsistent, producing different results from similar inquiries. When journalists tried to replicate Mason Cityโs prompts asking whether specific books contained โdescriptions or visual depictions of a sex act,โ ChatGPT generated different results than those received by the school district. And repeat inquiries gave contradictory answers.
To be sure, AI tools have mostly been successful at identifying and removing child sexual abuse material (CSAM), by using digital hash technology to assign a unique fingerprint, or โhash,โ to known CSAM images and to compare uploaded content against a large database of those โhashedโ images. Using these, social media platforms have generally been able to identify and remove CSAM images before they are posted online.
But when used to detect and remove content containing adult nudity and sexual activity, AI tools have been less reliable. They have struggled to understand the context of posts and are unlikely to be able to distinguish problematic content from, say, posts discussing or depicting subjects like breastfeeding or mastectomies. Metaโs Oversight Board acknowledged these limitations in a case where the companyโs automated systems flagged and removed an image of uncovered female nipples showing breast cancer symptoms with corresponding descriptions. Even though Metaโs Adult Nudity and Sexual Activity policy allows such images for โeducational or medical purposes,โ the companyโs machine learning classifier failed to understand the context of the photographs and wrongly removed the image. According to the Board, the case highlighted the concern that relying on inaccurate automated tools to enforce such policies โwill likely have a disproportionate impact on women.โ
Indeed, machine learning models have been shown to amplify biases. A recent study of AI tools used to flag sexual conduct found they were more likely to label photos of women as โsexually suggestive,โ especially if the photo depicted pregnant bellies, exercise, or nipplesโpossibly due to the bias of the primarily heterosexual male staff who trained the tools. Even images released by the U.S. National Cancer Institute that demonstrate how to do a clinical breast exam were flagged in researchersโ tests as โexplicitly sexual in nature.โ
While research is still ongoing, early analyses suggest that generative AI tools similarly reflect biases. These tools are trained on vast databases that include data from websites like Reddit and Wikipedia. Studies have shown that misogyny and other types of bias are overrepresented in these modelsโ training data. And ChatGPT is highly sensitive to user prompts, even bypassing its own rules around producing hateful content in response to usersโ engineered prompts. Therefore, prompts canโdeliberately or unconsciouslyโaffect which books are flagged.
Itโs not just schools that may be tempted to use ChatGPT to flag sexual content. In Texas, a new law requires all book vendors who supply school districts to rate their books based on whether they contain depictions or descriptions of sexual conduct. Nearly 300 booksellers will have to rate every book on their shelves (as well as titles previously sold) as either โsexually explicit,โ โsexually relevantโ โ meaning containing sexual conduct and part of required school curriculum โ or โno sexual conduct.โ This labor-intensive task may seem ripe for the use of AI, but if the tool replicates the inaccuracies and biases seen in content moderation, it could tag books depicting or describing educational subjects like breastfeeding or pregnancy as โsexually explicit.โ Even if ChatGPT misclassifies only 1% of the book titles it examines, hundreds of books could be inappropriately flagged and taken out of circulation.
Ultimately, one of the broader concerns about the use of AI is that it allows decision-makers to escape accountability and hide behind a veneer of objectivity and neutrality. As more states seek to limit discussion of race, gender identity, and sexuality in schools, decision-makers may rely on generative AI tools like ChatGPTโwith all its limitations, biases, and sensitivity to prompt designโto make overinclusive lists of inappropriate books under the guise of objectivity.
Broad book bans are a threat to free speech. Using generative AI tools to comply with those bans only makes them more dangerous.
