Honors Thesis 2024 - Jacob Choi

When Large Language Models Meet Religious Text

Jacob Choi

Highest Honor in Computer Science

Abstract

The field of AI has been quickly expanding outside of Computer Science, including areas such as healthcare, transportation, and the humanities. The intersection between AI and religion is also a growing field, but there exists a lack of computational work done from an application-based perspective. The current intersection in research between AI and religion often involves observing information that the models have learned, such as religious bias. For works that more directly impact communities, commercial AI-powered tools are available to help users learn more about religious texts, but lack transparency, which may be alarming for some.

To contribute to the field of AI application in religion from a computational perspective outside of AI model bias observation, we perform a case study on the Bible by creating a verse extraction tool using deep learning techniques to showcase the process of creating such a tool for religious communities to use. To do this, we first explore a challenge common to those who study the bible by finding references. We utilized a semantic similarity search and the Hungarian algorithm to identify references, which we found infeasible yet impactful. We then introduce six datasets that we use to train a llama-2-7b-chat model to respond to user queries with Bible verses. Additionally, we create two test sets to evaluate models, the first asking fact-based questions and the second asking theological questions. We find that state-of-the-art commercial models still come out on top with the highest accuracy of 62.5 and 58.5, and we describe the next steps to encourage research toward this direction of application-based tools in the computer science domain for religion.

Department / School

Computer Science / Emory University

Degree / Year

BS / Spring 2024

Committee

Jinho D. Choi, Computer Science and QTM, Emory University (Chair)
Davide Fossati, Computer Science, Emory University
Hiram Maxim, German Studies, Emory University
Helen J. Kim, American Religious History, Emory University

Links

Anthology | Paper | Presentation

Hiram Maxim, Helen Kim, Jacob Choi, Davide Fossati, Jinho Choi