A Domain-Agnostic Framework for Visual Element Detection from High-Level Descriptions

Modern web applications feature dynamic and visually complex interfaces that challenge the reliability and maintainability of traditional automated testing frameworks. These conventional approaches often rely on fixed document object model locators, such as XPath or CSS selectors, which are highly sensitive to minor user interface changes. As a result, tests frequently break, leading to increased maintenance costs and reduced accessibility for non-technical users. To address these challenges, we propose a unified and lightweight framework that combines a domain-specific restricted natural language for intuitive, high-level test specification, with a vision-language module capable of real-time user interface element detection directly from live screenshots. This eliminates the brittleness of fixed locators and enables robust test execution even under frequent user interface changes. To validate our approach, we implemented and deployed a fully functional prototype, demonstrating that the system can operate efficiently on consumer-grade hardware without reliance on cloud services offering a highly cost-effective solution suitable for enterprise use. A key enabler of our framework is the streamlined dataset construction process: we introduce a hybrid data acquisition strategy that combines automated annotation from large-scale web crawling with a scalable human-in-the-loop refinement pipeline, significantly reducing manual effort while maintaining high data quality. Notably, the framework is domain-agnostic and can be readily adapted to other disciplines requiring efficient vision-language fine-tuning, broadening its potential impact beyond web testing. Experimental evaluations show substantial improvements in test robustness, maintenance overhead, and execution efficiency, establishing our framework as a practical and scalable solution for next-generation automated UI testing.