Multimodal RAG Langchain Model with OpenAI API

How to launch app

• Create your virtual environment (python -m venv 'your_env_name') and initialize it (source 'your_env_name'/Script/activate, **on Git Bash Windows)

• Install all the necessary libraries (pip install -r requirements.txt)

• Go to your .streamlit/secrets.toml in the app folder and complete the necessary information with your own (specially the OpenAI API KEY)

• From your proyect path, launch the app (streamlit run 'home.py') and log in with the username and password you created in the secrets.toml file

• Install all the necessary libraries pip insta

Project Overview

This repository contains the code and documentation for a Project that incorporates different data science and ML strategies, addidng additional Generative AI features:

• MySQL: The app includes a page where the user can apply certain filters and build a dataframe using a sql query
• Computer Vision ML model: Images Classificator using a CNN where user can upload an X-Ray image and the model will predict if there is a fracture or not.
• Retrieval Augmented Generation (RAG) model: This model enables users to upload a PDF document, ask questions, and receive coherent, complete, and relevant responses generated by an integrated large language model (LLM).
• Streamlit app: Models deployes in Streamlit interface to enhance user experience The RAG model dynamically generates a prompt from the user's query, incorporating instructions, context, and restrictions to create specific, contextually aware responses.

Data Content

• Cardiology Vector Store: 452-page document containing information on Pediatric Cardiology. Source: Pediatric Cardiology: Essential Pocket Guide, book.
• Nephrology Vector Store: 785-page document containing information on Pediatric Nephrology. Source: Hand Book of PEDIATRIC NEPHROLOGY, book.
• Dataset with 200 datapoints and 20 features containing information about the nursing crew (personal information, workplace information and education certifications)

RAG Model Architecture

Model Selection

The model architecture is centered around Qdrant for vector store and OpenAIEmbeddings API as the text transformer. Key libraries used include:

• OpenAI: for embeddings and LLM
• Langchain: For text extraction and model chaining.
• Qdrant: To create and manage the vector stores (both locally and on Qdrant server).
• Docker: Necessary to use Qdrant server

Components

• Document Loader: pdfplumber handles document uploads and text parsing.
• Embeddings: OpenAIEmbeddings transforms text into vector representations.
• Text Extraction: RecursiveTextCharacterSplitter and Qdrant handle text processing and vectorization.

Chain Architecture

* Retrieval of Information:

User queries retrieve a set of k documents (where k=3 in the code) from the Qdrant vector store using similarity_search().

* Prompt Engineering:

• A context is built using the selected documents.
• This context is passed to the dynamic prompt-generating function.
• A specific, context-aware prompt is created based on the user’s query.

* LLM Implementation:

The prompt is sent to the LLM via the OpenAI API to generate the desired response.

Model Evaluation

A second LLM model is used as a "judge" to evaluate the generated responses based on the following criteria:

• Relevance(0-5)
• Accuracy(0-5)
• Completeness(0-5)
• Clarity(0-5)

Through prompt engineering, a dedicated evaluation prompt is used to assess the quality of each response.

ML Model Architecture

• CNN: for image classification

Streamlit App Architecture

The model is deployed on a multipage Streamlit app, providing a user-friendly interface. Users have access to different features based on their 'role':

• Pages:

  • Home page: log in required to access the app features (secure system) and determined user's role (patient, provider, admin)
  • Peds AI page: User can ask questions (text-input or voice-input) about Pediatric Cardiology and Nephrology,and receive responses formatted in Markdown, followed by the LLM-based evaluation. This design also allows user to upload images of their patients to have a better idea of what type of wounds they are delaling with.
  • AI Personal Assistant: Users can upload their own document and ask questions about it. The length of the document will determine if the document is fed directly into the prompt, or if there is a need to embed it and store it in a temporary vector store using Qdrant and OpenAIEmbeddings.
  • X-Ray AI page: users can upload an image of their patient's X-ray and the model will return a prediction between 'fractured' or 'not fractured'.
  • Education Center: Integrated SQL queries allow users to access information about the whole team, and also see what kind of questions are being asked to the Peds AI page to further evaluate the need to review certain topics.

• Roles and permissions:

  • Patient | Family Member: Peds AI page
  • Providers: Peds AI page, AI Personal Assistant, X-ray AI
  • Admin | Manager: All previous pages, plus the Education Center page

Conclusions

Conclusion 1: The RAG model demonstrated high efficiency in terms of response time and relevance to user queries. However, a local version would make the app suitable for internal applications. Conclusion 2: The Image classificator offers a limited feature, since the model only provides the prediction on fractures. It does not point the fracture nor provides further details on the injury and treatment. Conclusion 3: The Education Center page offer limitless possibilities to retrieve data from the nurses dataset, allowing the user to manage and target the needs for further educational programs and review courses for the nurses. However, the information about user's queries are store in the session cache, not allowing the manager to retrieve them in their next session.

Future Optimizations

• Local embeddings, local Vector Stores and local LLM response generation to provide secure interface for sensitive information handling.
• Further developing of diagnostic X-Ray prediction model to retrieve information about the fractures and recommended treatment.
• Continuous local storing of user's queries for future analysis, beyond the current session.

Repository

• Data folder: pdf and csv documents
• functions.py where all the functions and code used accross the app is organized
• Pptx presentation
• ReadMe.md
• Requirements.txt with all the neccessary libraries for this project
• Streamlit folder ('app') to deploy the code in Streamlit platform and test it in an application
• Tableau Dashboard link: https://public.tableau.com/app/profile/dani.siaj/viz/Book1_17310642525580/Dashboard1?publish=yes