DeepCrawl-Chat: An Intelligent Web Crawler and RAG System
Published:
Engineered ‘DeepCrawl-Chat,’ a comprehensive system for advanced web crawling and conversational search. The system allows users to crawl specified websites, extracts and processes the content, and builds a queryable knowledge base. Users can then interactively query the crawled content using natural language through a RAG pipeline. This project involved designing a scalable crawling architecture and integrating it with modern LLMs for accurate, context-aware responses.
Tech Stack: FastAPI, LangChain, FAISS, Docker, Uvicorn, NVIDIA AI, Hugging Face, Groq, SQLAlchemy, Redis.