Skip to main navigation Skip to search Skip to main content

Integration of LLMs for Multitasking Workload Prediction in Mixed Reality Environments

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Multitasking in mixed reality (MR) environments introduces unique cognitive demands, particularly in workload management. Accurate workload prediction is critical for optimizing user experience, safety, and performance in such settings. This study proposes a novel framework that integrates large language models (LLMs) with traditional workload assessment tools to enhance prediction accuracy in MR multitasking scenarios. A multitasking experiment involving 36 participants was conducted, combining realworld and virtual tasks, with workload evaluated using NASA-TLX. To address limited sample sizes, synthetic data was generated using generative adversarial networks (GANs), enabling robust model training. We employed a hybrid deep learning model that integrates LLM-generated text embeddings with numerical features in a feedforward neural network (FNN). Results show that integrating LLMs, specifically BERT and GPT-2, significantly improves workload prediction accuracy, with a root mean square error (RMSE) reduction from 6.82 (FNN-only) to 0.95 (BERTintegrated model). The findings underscore the potential of LLMs to augment cognitive workload assessment, supporting more adaptive and scalable human-machine collaboration in MR environments.

Original languageEnglish (US)
Title of host publicationProceedings of the Annual Conference of the Prognostics and Health Management Society, PHM
EditorsChetan S. Kulkarni, Marcos E. Orchard
PublisherPrognostics and Health Management Society
Edition1
ISBN (Print)9781936263295
DOIs
StatePublished - 2025
Externally publishedYes
Event17th Annual Conference of the Prognostics and Health Management Society, PHM 2025 - Bellevue, United States
Duration: Oct 25 2025 → Oct 30 2025

Publication series

NameProceedings of the Annual Conference of the Prognostics and Health Management Society, PHM
Number1
Volume17

Conference

Conference17th Annual Conference of the Prognostics and Health Management Society, PHM 2025
Country/TerritoryUnited States
CityBellevue
Period10/25/25 → 10/30/25

Keywords

  • LLMs
  • Mixed environment
  • NASA-TLX
  • Workload prediction

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Electrical and Electronic Engineering
  • Health Information Management

Fingerprint

Dive into the research topics of 'Integration of LLMs for Multitasking Workload Prediction in Mixed Reality Environments'. Together they form a unique fingerprint.

Cite this