Geo-Foundation Model Research Assistant

A system that integrates document retrieval and advanced language generation to provide contextually accurate answers to user queries about academic research on foundation models. The tool uses LangChain for orchestration and Hugging Face transformers for LLM-based text generation.

Key Components
Document Processing:
Extracts text from journal papers (PDFs).
Splits text into manageable chunks using LangChain's CharacterTextSplitter.

Embedding Generation:
Generates embeddings for text chunks using HuggingFaceEmbeddings (e.g., all-MiniLM-L6-v2).
Saves embeddings in metadata to avoid redundant computations.

Vector Indexing:
Stores embeddings in a FAISS vector database for efficient similarity-based retrieval.

Query Handling:
Retrieves relevant chunks based on user queries using a custom retriever (SimpleVectorRetriever).

LLM Integration:
Uses an instruction-tuned LLM (e.g., Flan-T5-small) to generate answers based on retrieved context.
Incorporates LangChain's RefineDocumentsChain for iterative refinement of responses.

Optimizations:
Implements caching and metadata storage for faster execution.
Adjusts parameters like temperature and top-p to control randomness in responses.