Scroll Top

Advanced RAG Multi-Modal Techniques for Accurate Data Extraction

Advanced RAG Techniques blog cover page-01-01-01 (1)

In the vast digital landscape, Portable Document Format (PDF) files have emerged as a ubiquitous medium for sharing and preserving information. From academic papers to business reports, PDFs encapsulate diverse content in a standardized and universally accessible format. While PDFs are renowned for their visual consistency across devices, extracting specific data, especially from tables within these documents, can often be a daunting task.

Another important aspect in this ever-evolving landscape of information technology is the ability to harness the power of Large Language Models (LLMs) by augmenting with accurate context to derive invaluable insights. However, the effectiveness of these models relies heavily on the quality and accuracy of the contextual data they receive. This blog delves into the pivotal role of extracting accurate information from PDF tables for Retrieval Augmented Generation, emphasizing its significance in ensuring LLMs provide correct and reliable answers.

In this blog post, we delve into the crucial use case of retrieving data from PDFs that contain tabular content. From discussing the various scenarios where this capability proves indispensable to exploring the importance of efficient data extraction, we aim to shed light on the significance of harnessing the potential hidden within PDF tables.

|Data retrieval Use Cases and scenarios

Data extraction from PDF files, including images and tabular data, becomes indispensable in several important use cases:

Financial Statements Analysis: Extracting data from financial reports, invoices, and statements enables financial analysts and accountants to perform comprehensive financial analysis, track expenses, and monitor financial performance accurately.

Medical Records Digitization: Converting medical records, lab reports, and patient charts from PDF format into structured data facilitates electronic health record (EHR) management, patient care coordination, and medical research.

Legal Document Processing: Extracting data from legal contracts, agreements, and court documents streamlines document review processes, enables keyword searching, and supports compliance with legal requirements.

Business Intelligence and Analytics: Extracting data from PDF reports, market research studies, and industry publications provides valuable insights for business decision-making, market trend analysis, and competitive intelligence.

Academic Research: Extracting data from scholarly articles, research papers, and academic journals supports literature reviews, citation analysis, and data aggregation for academic research and publication.

Insurance Claims Processing: Extracting data from insurance claim forms, policy documents, and medical records automates claims processing workflows, improves accuracy, and accelerates claim adjudication and settlement processes.

Real Estate Transactions: Extracting data from property listings, mortgage documents, and title deeds facilitates property valuation, market analysis, and real estate transaction management.

Customer Relationship Management (CRM): Extracting data from customer surveys, feedback forms, and contact lists enables businesses to analyze customer behavior, personalize marketing campaigns, and improve customer engagement and retention.

Supply Chain Management: Extracting data from shipping manifests, inventory reports, and purchase orders enhances supply chain visibility, inventory management, and demand forecasting for efficient supply chain operations.

Government and Regulatory Compliance: Extracting data from regulatory documents, compliance reports, and government publications helps organizations stay informed about regulatory changes, ensure compliance with industry standards, and mitigate legal risks.

Now that we have an overall understanding of different use cases, let’s shed some light on the importance of multimodal data retrieval.

|Importance of Multimodal data retrieval  

Data extraction from PDF files offers several benefits that contribute to improved efficiency, accuracy, and decision-making across various industries and domains:

Automation of Manual Processes: Extracting data from PDF files automates manual data entry tasks, reducing the need for human intervention and minimizing the risk of errors associated with manual data entry.

Time Savings: Automated data extraction from PDF files saves time compared to manual data entry methods, allowing organizations to reallocate resources to more strategic tasks and initiatives.

Improved Data Quality and Reliability: Automating multimodal data extraction minimizes errors and ensures consistent, reliable data for enhanced analysis and insights.

Enhanced Data Analysis: Extracted data from PDF files can be transformed into structured formats suitable for analysis using data analytics tools and techniques. This enables organizations to derive actionable insights, identify trends, and make informed decisions based on data-driven analysis.

Streamlined Business Workflows: Data extracted from PDF files can be integrated seamlessly into existing business systems, applications, and databases, streamlining business workflows and enhancing overall operational efficiency.

Facilitates Compliance and Reporting: Automated data extraction ensures consistency and accuracy in data reporting and compliance with regulatory requirements. Extracted data can be used to generate compliance reports, audit trails, and regulatory filings more efficiently.

Enables Search and Retrieval: Structured data extracted from PDF files enables easy search and retrieval of information, improving accessibility and usability of data for users across the organization.

Data-Driven Decision Making: Rapid data extraction from PDFs provides timely insights for informed decisions and supports data-backed strategies in dynamic environments.

Enhances Customer Experience: Streamlined data extraction processes enable organizations to respond to customer inquiries, process orders, and resolve issues more quickly and accurately, leading to improved customer satisfaction and loyalty.

Facilitates Digital Transformation: Data extraction from PDF files is a key component of digital transformation initiatives, enabling organizations to digitize and unlock valuable insights from unstructured data sources, such as scanned documents and image-based PDFs.

Enhanced Efficiency and Productivity: Automating table data extraction reduces manual labor, boosts productivity, and streamlines workflows for efficient data analysis and decision-making.

Versatility and Scalability: Modern multimodal extraction tools offer wide format compatibility and scalability, facilitating efficient data handling across industries and large datasets.

In this blog series, we will explore some of the most effective ways of using PDF data. In the current blog, we will make use of Multi-Modal extraction techniques in Retrieval Augmented Generation to achieve accurate results from a scanned PDF or PDF containing images with text.

Now we have a fair understanding of the importance of multimodal data retrieval. But how to get it going in a specific scenario? Let’s explain this in a step-by-step manner.

|Getting Started 

The main prerequisites which will be used in this solution:

  1.  Microsoft’s Table Transformer model which offers a promising solution for detecting tables within images.
  2. GPT4-V – The Open AI model with multi-modal capabilities.
  3. Llama-Index for orchestration and data-connectors.
  4. Other supporting libraries


Import the required libraries to get started with reading the PDF documents.

Define your model

Load the pdf, extract the pages, and convert the pdf into images.

The pdf used for this blog is taken from

This creates a subfolder with the name of the file and saves all the images here.

View the extracted images created under a folder of the name of the pdf document.

Using Microsoft’s Table Transformer to crop tables from the images and use the cropped data to get the required information.

Crop and extract tabular data from each page.

Now load the cropped data using SimpleDirectoryReader.

Now your data is ready. Start asking your questions.

The response:

The images provided appear to be pages from a Partnership Agreement document, which outlines the terms, conditions, and various provisions related to the operation of a partnership. The document includes amendments and restatements effective as of April 1, 2004. Here is a summary of the key sections and amendments based on the images provided:

  • **Capital Contributions (Section 2.01)**: This section likely details the initial contributions made by the partners to the partnership.
  • **Allocation of Profits and Losses (Section 3.03)**: This section describes how profits and losses are to be allocated among the partners.
  • **Economic Allocations (Section 3.08)**: This section specifies how net gains or losses are allocated among the partners for tax purposes.
  • **Adjustment of Basis of Partnership Property (Section 3.10)**: This section addresses the adjustment of the tax basis of partnership property upon the transfer of a partnership interest or other specified events.
  • **Taxes Withheld (Section 3.11)**: This section outlines the responsibilities for withholding taxes from distributions to partners.
  • **Independent Auditors (Section 8.01)**: This section mandates that the partnership’s books of account and records be audited annually by an independent certified public accountant.
  • **General Provisions (Section 9.01 – 9.10)**: These sections cover various general provisions, including amendments to the partnership agreement, powers of attorney, tax matters, and the entire agreement clause.
  • **Index of Defined Terms**: An index is provided to define specific terms used throughout the agreement
  • **Amendments to Partnership Agreement**: While specific amendments are not detailed in the images, the document mentions that it has been amended and restated, indicating that changes have been made to the original agreement.

The section numbers are provided next to the titles of each section in the images, and they correspond to the detailed provisions within the agreement. The document is structured to cover all aspects of the partnership’s operation, including financial matters, partner responsibilities, and legal compliance.

Please note that the images do not provide the complete text of each section, so this summary is based on the section titles and partial content visible in the images. For a comprehensive understanding of the amendments and the agreement as a whole, one would need to review the full document.

Reference collab 

This approach is useful when the PDF comprises images and tables from which we need to extract details needed for analysis.


Advanced RAG techniques, coupled with multi-modal capabilities, empower professionals across various domains to unlock the potential hidden within these documents. By automating data extraction, this approach streamlines workflows, enhances data quality, and facilitates data-driven decision-making.

Our in-house document parsing solution Intellexi can extract structured and unstructured data from different types of documents.


Leave a comment

Privacy Preferences
When you visit our website, it may store information through your browser from specific services, usually in form of cookies. Here you can change your privacy preferences. Please note that blocking some types of cookies may impact your experience on our website and the services we offer.