Enhancing Library Search System with AI Technology at Columbia University

Columbia University's AI Community of Practice hosted an enlightening session led by Columbia Libraries/Information Services and Columbia graduate Ashish Dubey. The session highlighted innovative work on augmenting library discovery systems with artificial intelligence (AI) and large language models (LLMs). The demo and discussion highlighted the integration of AI and machine learning (ML) techniques to advance academic research and discovery.

Transforming University Library Search System with AI

Columbia Libraries' AI project aims to enhance the search capabilities of CLIO, the university's library search system, by integrating AI technology. The goal is to improve the accuracy and relevance of search results, moving beyond traditional keyword-based searches. By leveraging AI, specifically retrieval augmented generation (RAG), Columbia Libraries can link its library database to an AI system, enabling the system to deliver highly relevant answers by comprehending the semantic meaning of documents.

Highlighted Advantages of the AI-enhanced System

• Specificity: AI-enhanced system excels at processing complex research queries and providing highly specific and relevant documents. For example, a detailed query about climate models and biodiversity in the Amazon rainforest yielded precise and contextually appropriate results.
• Natural Language Processing: It can handle natural language queries, making it more accessible to users unfamiliar with advanced search techniques. For instance, a query about conservation efforts for the Great Barrier Reef in an environmental chemistry context produced relevant research papers seamlessly.
• Multilingual Capabilities: One of the standout features is the AI's ability to handle and translate documents in different languages. This functionality enables researchers to access valuable resources irrespective of language barriers.
• User Feedback Loop: It incorporates a feedback mechanism where users can approve or disapprove search results, allowing the AI to refine its recommendations iteratively. This continuous learning loop ensures increasingly accurate and tailored results.

Technical Architecture and Process

The AI-enhanced system's technical framework consists of three main processes:

1. Document Vectorization: first process converts plain text documents into numerical representations known as vector embeddings. These representations capture the meaning of the content, enabling retrieval based on the semantics of the documents rather than exact keyword matches.
2. Vector-Based Search: next process identifies and ranks documents based on their semantic relevance using a cosine similarity search algorithm to the user's query, ensuring that search results are contextually relevant.
3. AI-Assisted Retrieval: final process combines the results of the vector-based search with a large language model (LLM), such as OpenAI's GPT-4 Turbo, to produce detailed summaries and usage recommendations for each document. This involves a dynamic feedback loop where the AI system continuously learns from user interactions to refine and enhance search outcomes. 

Looking Ahead

As the amount of data and the number of users grows, the AI-enhanced system needs to maintain high performance and accuracy, which requires robust computing. Running LLMs at scale can be costly, mainly when relying on external APIs. To address these issues, the team is considering hosting the LLMs locally to control costs. Although this approach requires an initial investment in hardware and infrastructure, it could lead to more predictable and manageable long-term costs.

This initiative marks a major advancement in using AI for academic research and discovery. It demonstrates Columbia University's innovative spirit and showcases how AI can improve academic research. As the project progresses, its goal is to establish new standards for library discovery systems, making research more efficient and accessible for faculty, researchers, and students in higher education.

Emerging Technologies' mission is to empower faculty, researchers, and administrative members to utilize next-generation technologies to explore, adopt and integrate new applications and strategies. Please visit our AI: Community of Practice for discussion highlights on AI topics and an interest intake form to join the monthly gatherings.