Smarter Crowdsourcing with NLP and Attention Mechanisms for Task Complexity Prediction

Competitive Crowdsourcing Software Development (CCSD) has emerged as a powerful tool for developing software solutions, attracting researchers and the development market. Using crowdsourced collective intelligence, CCSD ensures the delivery of innovative, cost-effective, and high-quality solutions within specified time frames, making it an attractive approach for addressing complex challenges in software development. However, as the CCSD environment gains popularity, it also introduces challenges, particularly in predicting job complexity, which must be tackled to optimize the crowdsourcing process. In software platforms, client organizations register and upload jobs related to development projects. These jobs are manually reviewed by the copilots responsible for assessing the complexity of the job, a process that can lead to delays and an overburden of experts. To streamline this process, we propose an approach that automatically predicts job complexity and classifies it accordingly. We collect data from the TopCoder CCSD platform, focusing on projects related to software development. The collected data are pre-processed and tokenized using NLP techniques. We convert the text data into a word embedding vector matrix using a pre-trained GloVe model, which captures the semantic and contextual meaning of the text. The Sequence Attention (SA) mechanism is introduced in LSTM to identify the key parts of the input sequence that are most relevant for predicting the output, thereby improving job complexity classification. The proposed approach is then trained on these word embeddings using SA-LSTM and benchmarked against LSTM and other state-of-the-art techniques. The proposed approach, GloVe-based (GB) SA-LSTM, outperforms other approaches by achieving an accuracy improvement of 31.4%, 152.04%, 20.07%, 10.99%, 9.16%, and 17.64% over ZeroR, RP, LR, BERT, SVM, and DT. To verify the impact of SA, the proposed approach is compared with GB-LSTM. The GB-SA-LSTM outperforms GB-LSTM by 4.88%. This enhanced accuracy in task complexity prediction within software crowdsourcing platforms can significantly contribute to the efficient and effective development of crowdsourced projects.