Enhancing Educational Support for JetBrains MPS with a Retrieval-Augmented LLM Chatbot: A Structured Knowledge Integration Approach

Journal: International Journal on Software and Systems Modeling (SoSyM)

Abstract:

In this paper, we present a Large Language Model (LLM)-based chatbot designed to support learners of JetBrains MPS, a language workbench widely used in domain-specific language (DSL) development and model-based software engineering (MBSE). MPS poses a steep learning curve due to its complexity and high prerequisite knowledge in language engineering. To mitigate these challenges, our chatbot leverages retrieval-augmented generation (RAG) using the LlamaIndex library, integrating official documentation and expert-authored resources into a context-aware support system. We propose a method that improves chatbot performance through structured semantic summaries and composable graph-based indexing, enabling targeted retrieval and scaling to advanced modeling concepts such as maintainable MPS generator development. We evaluate the approach using the RAGAs framework, showing improvements in answer faithfulness, contextual relevance, and safety. The results demonstrate that this LLM-based assistant can effectively lower barriers to mastering modeling tools like MPS. Our contribution offers a reproducible method for enhancing modeling education through structured integration of language-specific knowledge in LLM-powered assistants.

Source: Manual