Multi-Modal Retrieval-Augmented Generation (RAG) with Text and Image Processing

Project Overview

Research documents become directly and efficiently analyzed and summarized through a research assistant that runs on AI infrastructure which utilizes natural language processing (NLP) optical character recognition (OCR) and vector search methods. Research papers go through the system which extracts text images and tables then performs semantic searches through Hugging Face embeddings and ChromaDB followed by generating AI summaries of text and image explanations with GPT-4o. Users can submit research-related inquiries through the system which delivers specific relevant content sections thus cutting down the time needed for manual reading. This project delivers increased academic research speed because it incorporates PyMuPDF, pdfplumber, Tesseract OCR, OpenCV, LangChain, and OpenAI to optimize document processing along with AI retrieval capabilities. This allows students along with researchers and analysts to obtain automated content retrieval and available access to literature reviews, academic insights and research paper analysis.

Prerequisites

Python 3.8+ with Google Colab or Jupyter Notebook for execution.
The text and image analysis with GPT-4o requires an OpenAI API Key as a prerequisite for operation.
The program relies on the Tesseract OCR & Poppler-utils combination to extract text content from PDF files alongside image documents.
LangChain, ChromaDB & Hugging Face Embeddings for semantic search and AI-powered retrieval.
The program requires PyMuPDF together with pdfplumber and pdf2image to extract text and images along with tables from PDF documents.
Pandas, NumPy, Matplotlib & IPython Display for data processing and visualization.

Approach

This research project uses an AI-based method that combines Multimodal RAG with OCR, NLP, vector-based retrieval to efficiently extract and analyze academic content. The processing steps for PDFs begin with the application of PyMuPDF and pdfplumber and pdf2image for text table and image extraction and Tesseract OCR for processing scanned documents. Text and tables extracted from the documents are embedded with Hugging Face sentence transformers to create entries in ChromaDB for semantic search functions.

The system passes base64-formatted images to GPT-4o for structured figure and table and visual data explanations. RAG into the system allows users to achieve text embeddings together with AI-generated insights which provide context-dependent document comprehension. The AI querying capability of GPT-4o allows it to retrieve pertinent research paper sections automatically for users as well as generate summarized content from their search requests which enhances the speed and precision of research results.

Pandas DataFrame serves as structured storage that supports advanced analysis and visualization for the organized extracted content. This RAG system enables efficient academic research delivery because it removed the requirement for manual document assessment and delivered improved summary solutions that process images along with text content.

Workflow and Methodology

Workflow

The system handles PDF upload and processing duties which result in reading and analysis of these documents.
It extracts structured text data alongside tables from PDFs through PyMuPDF and pdfplumber tools.
Employs Tesseract OCR as part of its OCR for Scanned PDFs feature which extracts text from scanned or image-based PDFs.
Utilizes GPT-4o to convert academic figures into base64 while creating explanations through AI technologies.
Saves extracted content under ChromaDB embedding format using Hugging Face for semantic search operations.
Users can use this system to perform queries that enable them to find sections containing research answers.
Structure Summaries for all file types come from GPT-4o during automated summarization processes.
The system transforms raw data contents into Pandas DataFrames for both analysis and visualization purposes.

Methodology

Loads the PDF, determines its structure and extracts text, tables and pictures.
Uses Tesseract OCR and OpenCV to convert scanned PDFs and figures to text.
Using Hugging Face sentence transformers, convert extracted information into numerical embeddings.
Implements semantic search by storing and retrieving contextually relevant document portions using ChromaDB.
Multimodal Analysis with GPT-4o - AI is used to analyse both the textual and visual components of the study paper.
Uses user input to dynamically retrieve and summarise relevant portions.
Context-Aware Summarisation creates organised, AI-driven summaries of study findings.
Provides extracted and summarised material in a structured manner for review and analysis.

Data Collection and Preparation

Data Collection

Research Paper PDFs are collected using Google Scholar for publicly available academic papers and Sci-Hub for accessing paywalled research.
Academic Journals, Theses, and Conference Papers are sourced to ensure a diverse dataset.

Data Preparation Workflow:

PDFs are processed to extract text, tables, and images using PyMuPDF, pdfplumber, and Pandas.
Tesseract OCR retrieves text from scanned documents for better accessibility.
OpenCV enhances figures and converts them to base64 for AI-based analysis.
Extracted content is labeled with sections, page numbers, and metadata for structured retrieval.
Hugging Face embeddings are generated and stored in ChromaDB for semantic search.
Processed data is structured into a Pandas DataFrame for AI-driven querying and analysis.

Code Explanation

STEP 1:

Mounting of Google Drive

This code mounts your Google Drive into the Colab environment so that you can access files stored in your drive. Your Google Drive is made accessible under /content/drive path.

from google.colab import drive
drive.mount('/content/drive')

This code installs various tools for working with PDFs, enabling text extraction through OCR, and utilizing AI models. It configures Poppler and Tesseract OCR for recognizing text, along with LangChain, OpenAI, and ChromaDB for AI-driven document processing. Essentially, it readies your system to manage PDFs, images, and AI-enhanced text extraction.

!apt-get update && apt-get install -y poppler-utils tesseract-ocr libtesseract-dev
!pip install --no-cache-dir pdf2image pdfplumber opencv-python-headless unstructured[all-docs] chromadb langchain openai python-dotenv pillow tiktoken pymupdf pytesseract
!pip install langchain_openai
!pip install -U langchain-community --quiet

This code imports various libraries to manage PDFs, images, AI models, and data processing tasks. It utilizes utilities such as os and dotenv for managing the environment, along with numpy and pandas for handling data. For PDF extraction, it incorporates pdfplumber and PyMuPDF. Additionally, it configures pytesseract for optical character recognition (OCR) and employs OpenAI and LangChain for AI-driven text analysis, while also using matplotlib for data visualization.

# Basic Libraries
import os
import uuid
import base64
import io
from pathlib import Path
from dotenv import load_dotenv
from google.colab import userdata
#  Data Processing & Utilities
import numpy as np
import pandas as pd
# PDF Processing
import fitz  # PyMuPDF
import pdfplumber
from pdf2image import convert_from_path
from unstructured.partition.pdf import partition_pdf
# Image Processing
import cv2
import pytesseract
from PIL import Image
# OpenAI & LangChain (LLM & Embeddings)
import openai
from openai import OpenAI
from langchain_openai import ChatOpenAI  # Updated OpenAI API wrapper
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.chat_models import ChatOpenAI
from langchain.schema import SystemMessage, HumanMessage, AIMessage
# LangChain Document Handling
from langchain.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter
from langchain.docstore.document import Document
# Visualization
import matplotlib.pyplot as plt
from IPython.display import display, HTML, Markdown
from IPython.display import display

Setting Up OpenAI API Key

This code fetches the OpenAI API key from the userdata in Google Colab and saves it in the environment variable OPENAI_API_KEY. After that, it sets up the OpenAI client with this key, enabling the program to communicate with OpenAI’s API for various AI-related tasks.

os.environ["OPENAI_API_KEY"] = userdata.get('OPENAI_API_KEY')
OPENAI_API_KEY=os.environ["OPENAI_API_KEY"]
client = OpenAI(api_key=OPENAI_API_KEY)

Initializing Hugging Face Embeddings

This code imports Hugging Face embeddings with the "sentence-transformers/all-MiniLM-L6-v2" model. It transforms text into numerical vectors, which are beneficial for tasks such as semantic search, text similarity and AI-driven retrieval.

from langchain_community.embeddings import HuggingFaceEmbeddings  
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")

Setting Up Output Directory

This code establishes a path for the output directory using Path from the pathlib module. It designates /content/drive/MyDrive/Badhon/output_imagess as the location for saving processed images or other files.

from pathlib import Path
output_dir = Path("/content/drive/MyDrive/Badhon/output_imagess")

Extracting and Processing PDF Content

The extract_text_from_image function utilizes OpenCV to read an image, convert it to grayscale, apply adaptive thresholding and extract text through Tesseract OCR, all while managing errors effectively. The process_research_paper function is designed to extract text, tables and images from an academic PDF. It begins by reading the PDF with PyMuPDF (fitz) and organizes sections such as Abstract, Introduction, Methodology and Results based on keyword detection. Next, it employs pdfplumber to extract tables and uses PyMuPDF to gather images, saving them in the specified output directory. Any extracted images are processed with the extract_text_from_image function to capture any embedded text. Ultimately, the function returns a structured list of the extracted elements, which includes text, table and images along with their metadata.

def extract_text_from_image(image_path):
"""Extracts text from an image using OpenCV + Tesseract OCR."""
try:
img = cv2.imread(str(image_path))
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
processed = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 31, 2)
return pytesseract.image_to_string(processed)
except Exception as e:
print(f"Error extracting text from image {image_path}: {e}")
return "OCR Failed"
### --- FIXED PDF EXTRACTION FUNCTION --- ###
def process_research_paper(pdf_path: str):
"""Extracts text, tables, and images from an academic PDF."""
processed_data = []
doc = fitz.open(pdf_path)
# --- Extract Text Content ---
for page_num in range(len(doc)):
page = doc[page_num]
text = page.get_text("text")
# Detect sections
section = "General Content"
if "abstract" in text.lower():
section = "Abstract"
elif "introduction" in text.lower():
section = "Introduction"
elif "methodology" in text.lower():
section = "Methodology"
elif "results" in text.lower() or "findings" in text.lower():
section = "Results"
processed_data.append({
"element_id": f"page_{page_num}",
"type": "Text",
"content": text,
"metadata": {"section": section, "page_number": page_num + 1}
})
# --- Extract Tables using pdfplumber ---
with pdfplumber.open(pdf_path) as pdf:
for page_num, page in enumerate(pdf.pages):
tables = page.extract_tables()
for idx, table in enumerate(tables):
if table:
table_text = "\n".join([" | ".join(str(cell) if cell else "" for cell in row) for row in table if any(row)])
processed_data.append({
"element_id": f"table_{page_num}_{idx}",
"type": "Table",
"content": table_text,
"metadata": {"table_number": f"Table {idx+1}", "page_number": page_num + 1}
})
# --- Extract Images using PyMuPDF ---
for page_num in range(len(doc)):
page = doc[page_num]
img_list = page.get_images(full=True)  # Get all images in the page
for img_index, img in enumerate(img_list):
xref = img[0]  # Reference for extracting the image
base_image = doc.extract_image(xref)  # Extract image data
img_bytes = base_image["image"]  # Get raw image data
img_ext = base_image["ext"]  # Image format (PNG/JPG)
#  Save extracted image
img_path = output_dir / f"figure_{page_num+1}_{img_index+1}.{img_ext}"
with open(img_path, "wb") as f:
f.write(img_bytes)
# Run OCR on the image (for image-based tables)
extracted_text = extract_text_from_image(img_path)
processed_data.append({
"element_id": f"image_{page_num+1}_{img_index+1}",
"type": "Image",
"content": str(img_path),
"metadata": {"figure_number": f"Figure {img_index+1}", "page_number": page_num + 1},
"ocr_text": extracted_text
})
return processed_data

Running the PDF Processing Pipeline

This code executes the process_research_paper function on a PDF file named 2021clustur.pdf to extract text, tables, and images. The extracted components are saved in the elements list. To check the results, it prints a sample of the first three extracted elements, displaying their type (Text, Table, or Image) along with a preview of their content.

#Run Pipeline
pdf_path = "/content/drive/MyDrive/2021clustur.pdf"
# Process the research paper
elements = process_research_paper(pdf_path)
# Debug: Check extracted content
print("\n=== Extracted Elements Sample ===")
for elem in elements[:3]:
print(f"{elem['type']}: {elem.get('content', '')[:600]}...")

Displaying Extracted Tables and Images

This code goes through the first 20 elements extracted from the PDF, counting and showing tables and images. When it encounters a table, it transforms the text-based table into a Pandas DataFrame for improved readability and displays it. For images, it loads and shows the image using PIL (Pillow). Additionally, it prints out metadata such as the page number and figure/table number to help identify the source of the extracted content.

import pandas as pd
import IPython.display as display
from PIL import Image
table_count = 0
image_count = 0
for elem in elements[:20]:  # Check more elements for debugging
if elem["type"] == "Table":
table_count += 1
print(f"\n🔹 Table Extracted (Page {elem['metadata']['page_number']} - {elem['metadata']['table_number']}):\n")
# Convert text-based tables into DataFrame
table_rows = [row.split(" | ") for row in elem["content"].split("\n") if row]
df = pd.DataFrame(table_rows)
# Display table as a proper table
display.display(df)
elif elem["type"] == "Image":
image_count += 1
img_path = elem["content"]
print(f"\nImage Extracted (Page {elem['metadata']['page_number']} - {elem['metadata']['figure_number']}): {img_path}\n")
# Display the extracted image
img = Image.open(img_path)
display.display(img)

Generating Academic Summaries for Research Paper Elements

This function analyzes text, tables and images from a research paper to create summaries using AI technology. For images, it transforms them into a base64 string, offers a structured analysis prompt and requests the model to highlight key patterns. For tables, it pulls out important variables, trends and implications. For text sections, it pinpoints hypotheses, methodology and findings. The summaries generated by AI are saved within each element and any errors are managed effectively by marking summaries as unavailable if problems occur.

def generate_academic_summaries(elements):
"""Generate context-aware summaries for research paper elements."""
for elem in elements:
try:
if elem["type"] == "Image":
# Convert the image file to a base64-encoded string
with open(elem["content"], "rb") as img_file:
base64_image = base64.b64encode(img_file.read()).decode("utf-8")
# Define the text portion of the prompt for the image
prompt_text = (
"Analyze this academic figure:\n"
"- Describe key elements and labels\n"
"- Identify statistical representations\n"
"- Explain significance in the paper context\n"
"- Note any patterns or anomalies"
)
# Construct a multimodal message: text + image_url (using a data URI)
messages = [
{"role": "user", "content": [
{"type": "text", "text": prompt_text},
{"type": "image_url", "image_url": {"url": "data:image/png;base64," + base64_image}}
]}
]
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
temperature=0.2,
max_tokens=400
)
elem["summary"] = response.choices[0].message.content
elif elem["type"] == "Table":
# Create a prompt that includes the table content
prompt = (
f"Analyze this research table:\n{elem['content']}\n\n"
"1. Identify key variables and metrics\n"
"2. Summarize main relationships\n"
"3. Note significant values/trends\n"
"4. Explain potential implications"
)
messages = [{"role": "user", "content": prompt}]
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
temperature=0.3,
max_tokens=500
)
elem["summary"] = response.choices[0].message.content
else:  # For text content
section = elem.get("metadata", {}).get("section", "General Content")
prompt = (
f"Summarize this {section} section from a research paper:\n{elem['content']}\n\n"
"Focus on:\n"
"- Key hypotheses/research questions\n"
"- Methodology components\n"
"- Significant findings\n"
"- Theoretical contributions"
)
messages = [{"role": "user", "content": prompt}]
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
temperature=0.4,
max_tokens=400
)
elem["summary"] = response.choices[0].message.content
except Exception as e:
print(f"Error processing element {elem.get('element_id', 'unknown')}: {str(e)}")
elem["summary"] = "Summary unavailable"
return elements
# Generate academic summaries
summarized_elements = generate_academic_summaries(elements)

Debugging and Checking Summaries

This code outputs a sample of the first three summaries created for various elements of research papers. It takes the first 1000 characters from each summary and presents them, making sure that the AI-generated summaries are well-structured and informative. This process aids in checking the accuracy and thoroughness of the extracted content.

# Debug: Check summaries
print("\n=== Summary Sample ===")
for elem in summarized_elements[:3]:
print(f"{elem['type']} Summary: {elem['summary'][:1000]}...")

Displaying Summaries for Extracted Images

This code iterates through all summarized elements and outputs summaries specifically for images. It shows the figure number and page number alongside the AI-generated summary. A separator ("-" * 20) is included for better readability, facilitating the review of multiple image summaries simultaneously.

for elem in summarized_elements:
if elem["type"] == "Image":
print(f"Image Summary (Figure {elem['metadata']['figure_number']}, Page {elem['metadata']['page_number']}):")
print(elem["summary"])
print("-" * 20)

Creating a DataFrame from Summarized Elements

This function transforms the components of the research paper into a Pandas DataFrame for organized analysis. It pulls out essential information such as element ID, type, content, summary and OCR text, as well as metadata like page number and section. Each component is recorded as a row, simplifying the process of filtering, analyzing and visualizing the extracted research material. Ultimately, the DataFrame is generated and shown using df.head().

def create_dataframe_from_elements(elements):
"""Creates a Pandas DataFrame from the summarized elements."""
data = []
for elem in elements:
# Build a row from required keys and any metadata available.
row = {
"element_id": elem.get("element_id", ""),
"type": elem.get("type", ""),
"content": elem.get("content", ""),       # Fallback to empty string if missing
"summary": elem.get("summary", ""),
"ocr_text": elem.get("ocr_text", "")
}
# Add metadata fields as individual columns (if any)
row.update(elem.get("metadata", {}))
data.append(row)
df = pd.DataFrame(data)
return df
# Create DataFrame from your processed elements (summarized_elements)
df = create_dataframe_from_elements(summarized_elements)
print("DataFrame head:")
print(df.head())

Creating a Vector Database for Research Elements

This code sets up text embeddings and saves them in a Chroma vector database to enable efficient retrieval. It starts by merging the summary, content and OCR text into one string (text_to_embed) for embedding purposes. Next, it generates LangChain Document objects, ensuring that metadata is preserved for context. Finally, it initializes a Chroma vector database, transforming text into embeddings with the Hugging Face model, which allows for semantic retrieval and AI-driven document analysis.

df['text_to_embed'] = df['summary'] + " " + df['content'] + " " + df['ocr_text']
# Create Document objects from each row.
documents = [
Document(page_content=text, metadata=metadata)
for text, metadata in zip(
df['text_to_embed'].tolist(),
df.drop(columns=["text_to_embed"]).to_dict(orient="records")
)
]
# Create the vector database using Chroma.
vector_db = Chroma.from_documents(
documents=documents,
embedding=embeddings,
)

Explaining and Displaying Academic Figures with GPT-4o

This function processes an image, encodes it in base64 and submits a multimodal request to GPT-4o for an in-depth analysis. The explanation generated by the AI covers essential components, statistical representations, significance and noteworthy patterns. The image and its corresponding explanation are presented together in a well-structured HTML format using IPython.display. This approach improves research analysis by offering organized insights into academic figures.

import base64
import openai
import imghdr
from IPython.display import display, HTML
def explain_and_show_image(image_path):
"""
Given the path to an image, this function:
- Reads and encodes the image as a base64 data URI.
- Sends a multimodal request to GPT-4o for image analysis.
- Displays the image alongside its explanation.
- Returns the explanation text.
"""
# Open and encode the image file in base64.
with open(image_path, "rb") as img_file:
image_bytes = img_file.read()
# Detect MIME type dynamically
mime_type = imghdr.what(image_path) or "png"  # Default to PNG if detection fails
base64_image = base64.b64encode(image_bytes).decode("utf-8")
data_uri = f"data:image/{mime_type};base64," + base64_image
# Define the detailed text prompt for GPT-4o
prompt_text = (
"Analyze this academic figure and provide a structured explanation:\n"
"1️⃣ **Description of Key Elements and Labels**\n"
"2️⃣ **Statistical Representations** (e.g., bar chart, scatter plot, trend analysis)\n"
"3️⃣ **Significance in the Paper Context** (Why is this figure important?)\n"
"4️⃣ **Patterns & Anomalies** (Notable trends, spikes, outliers)\n"
"Please format the explanation neatly using numbered sections."
)
# Construct a multimodal request with an image
messages = [
{"role": "user", "content": [
{"type": "text", "text": prompt_text},
{"type": "image_url", "image_url": {"url": data_uri}}
]}
]
# Call GPT-4o to generate the explanation
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
temperature=0.3,
max_tokens=800  # Increased to allow for more detailed responses
)
# Extract the generated explanation
explanation = response.choices[0].message.content
# Generate HTML for a neat display
html_content = f"""
\
\
\
\
\
\📊 Image Explanation\
\{explanation}\
\
\
"""
# Display the formatted HTML content
display(HTML(html_content))
return explanation
# Example usage: Provide your actual image path
explanation_text = explain_and_show_image("/content/drive/MyDrive/Badhon/output_image/figure_3_1.png")

Querying an Academic Paper with AI-Powered Search

This feature enables users to pose research-related inquiries and obtain pertinent information from a Chroma vector database through semantic search. It effectively identifies and merges the most relevant sections of documents, subsequently prompting GPT-4o to create a well-organized, easy-to-understand response. The AI-generated reply is presented in Markdown format, enhancing visual clarity with bullet points and key highlights for improved understanding.

from langchain_openai import ChatOpenAI
from langchain.schema import SystemMessage, HumanMessage
from IPython.display import display, Markdown
def query_system(query: str, vector_db, k: int = 10) -> None:
"""
Retrieves an answer for ANY query related to an academic paper.
- Dynamically retrieves the most relevant content based on query intent.
- Uses AI to analyze the query and find the best response.
- Formats the response for easy readability.
Parameters:
- query (str): The user question (e.g., "Which method performed best?", "What is the dataset used?")
- vector_db: The Chroma (or any compatible) vector store containing document embeddings.
- k (int): Number of top matching documents to retrieve.
Returns:
- None (Displays a well-structured response)
"""
# Retrieve documents using similarity search
retrieved_docs = vector_db.similarity_search(query, k=k)
# Combine retrieved content
context = "\n\n".join([doc.page_content for doc in retrieved_docs])
# Define the AI prompt (NO STATIC SECTIONS - Fully Open-Ended)
prompt = (
"You are an expert research assistant analyzing academic papers.\n"
"Answer the user's question as clearly as possible using the extracted context from the research paper.\n"
"Provide well-structured, easy-to-read answers using bullet points and key highlights.\n\n"
" **User Query:** {query}\n\n"
" **Extracted Context:**\n{context}\n\n"
" **Your Answer:**"
)
# Initialize the AI model
llm = ChatOpenAI(model_name="gpt-4o", temperature=0.3)
# Generate AI response
messages = [
SystemMessage(content="You are an AI assistant specializing in research paper analysis."),
HumanMessage(content=prompt.format(query=query, context=context))
]
response = llm.invoke(messages)
# Display the structured response in Markdown format
display(Markdown(response.content))
return response.content

Example Research Paper Queries

These queries are designed to extract key insights from the academic paper using AI-powered search.

# Example Queries (Ask Anything!)
query_1 = "Which method performed best in this study?"
query_2 = "What dataset was used in this research?"
query_3 = "Summarize the main conclusions."
query_4 = "How does this paper compare different clustering techniques?"
query_5 = "What are the limitations of this study?"

Executing AI-Powered Query for Best-Performing Method

This command executes the query_system function with the question: "Which method performed best in this study?". The Chroma vector database retrieves the top 10 most relevant sections from the research paper and GPT-4o analyzes the extracted content to create a well-organized answer. The response is presented in a clear bullet-point format, allowing researchers to quickly pinpoint the best-performing method in the study.

query_system(query_1, vector_db, k=10)  # Finds best-performing method

query_system(query_5, vector_db, k=10)  # Lists study limitations

Conclusion

The AI-supported research assistant performs academic document analysis of academic papers PDFs and research documents through the combination of NLP, vector search, LLMs and Multimodal RAG capabilities. OCR along with LangChain integrates with OpenAI’s GPT-4o and ChromaDB and Hugging Face embeddings to create a system that performs automatic text extraction and semantic searches and AI-driven summaries and intelligent image understanding.

AI uses Multimodal RAG technology to perform text and image-structured retrieval which helps it interpret data between textual content and visual elements. Context-sensitive document understanding becomes better through this integration which results in enhanced accuracy of both queries and summaries. Both textual and visual information allows researchers to achieve automatic retrieval of essential findings and analysis of figures and research papers through automated methods.

The solution modifies academic research together with document processing along with AI-based content retrieval to serve as an effective tool for students and scholars and data analysts who pursue AI automation methods.

Challenges New Coders Might Face

Problem: OpenAI API Key Not Found
Solution: Ensure you have a valid OpenAI API Key, store it in the environment (os.environ["OPENAI_API_KEY"]), and verify it before running the project.
Problem: Tesseract OCR Not Installed
Solution: Install Tesseract manually using !apt-get install -y tesseract-ocr libtesseract-dev (for Linux/Colab) and ensure the path is set in your environment (pytesseract.pytesseract.tesseract_cmd).
Problem: Poppler-Utils Missing
Solution: Install Poppler with !apt-get install -y poppler-utils (Linux) or download it from the official site for Windows and set the path correctly.
Problem: ChromaDB Not Storing Data
Solution: Ensure the embeddings are correctly generated and stored using Chroma.from_documents(). If the database is empty, re-run the data embedding process.
Problem: Large PDFs Causing Memory Issues
Solution: Use chunking techniques with CharacterTextSplitter to process documents in smaller parts, reducing memory usage.
Problem: Image Analysis Not Working with GPT-4o
Solution: Ensure images are correctly converted to Base64 format before sending them in API requests and check if GPT-4o supports multimodal inputs in your region.

Frequently Asked Questions (FAQ)

Question 1. How can I extract text from academic PDFs using AI?

Answer: You can use PyMuPDF, pdfplumber and Tesseract OCR to extract text from PDFs, including scanned documents. This project automates the process using AI-powered text analysis and summarization with GPT-4o.

Question 2. How do I install and use Tesseract OCR for scanned PDFs?

Answer For Colab, install it with !apt-get install -y tesseract-ocr libtesseract-dev. For Windows, download Tesseract from its official website and set the path in pytesseract.pytesseract.tesseract_cmd.

Question 3. How does this AI research assistant use GPT-4o for research papers?

Answer: This system integrates GPT-4o with LangChain and ChromaDB, enabling semantic search, AI-driven summarization, and image analysis of research documents. You can ask questions like “What is the main conclusion of this paper?” and get AI-generated responses.

Question 4. Can I perform semantic search on research papers?

Answer: Yes! Extracted text is converted into Hugging Face embeddings and stored in ChromaDB, allowing AI-powered vector search to retrieve the most relevant sections of a paper.

Question 5. How do I analyze tables and figures from a research paper?

Answer: Tables are extracted using pdfplumber and structured into Pandas DataFrames, while figures are processed using OpenCV and GPT-4o to generate AI-powered explanations.