TY - JOUR
T1 - Developing common qualitative tools for cross ERC education program evaluation
AU - Zhao, Zhen
AU - O'Donnell, Megan
AU - Lyra, Marcus
AU - Jordan, Michelle
AU - Savenye, Wilhelmina C.
AU - Roehrig, Gillian
N1 - Funding Information: This work is supported by the National Science Foundation Grant EEC-2023275. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We would also like to thank the leadership team, education team, and the evaluation team of all partner ERCs, for their support and participation in knowledge sharing, data collection, and offering constructive feedback. Funding Information: 75 Engineering Research Centers (ERCs) across the nation have been funded by the National Science Foundation (NSF) since 1985; 14 are currently operating [1]. These ERCs have played a big part in not only advancing engineering and technology but also integrating research, education, and workforce development [1]. NSF requires each ERC to provide educational and professional development opportunities for professionals, post-doctoral researchers, graduate students, undergraduate students, K-14 teachers, and K-12 students. At the same time, data-driven approaches are recommended to evaluate and track the performance and impacts of these opportunities [2] findings are required as part of the center¶s annual report and site visit presentations. The responsibility falls on each ERC to develop and implement an evaluation plan. The effort, in general, is led by center education directors/leadership collaborating with external evaluators. Publisher Copyright: © American Society for Engineering Education, 2022
PY - 2022/8/23
Y1 - 2022/8/23
N2 - National Science Foundation (NSF) funded Engineering Research Centers (ERC) are required to develop and implement education and outreach opportunities related to their core technical research topics to broaden participation in engineering and create partnerships between industry and academia. Additionally, ERCs must include an independent evaluation of their education and outreach programming to assess their performance and impacts. To date, each ERC's evaluation team designs its instruments/tools and protocols for evaluation, resulting in idiosyncratic and redundant efforts. Nonetheless, there is much overlap among the evaluation topics, concepts, and practices, suggesting that the ERC evaluation and assessment community might benefit from having a common set of instruments and protocols. ERCs' efforts could then be better spent developing more specific, sophisticated, and time-intensive evaluation tools to deepen and enrich the overall ERC evaluation efforts. The implementation of such a suite of instruments would further allow each ERC to compare its efforts to those across other ERCs as one data point for assessing its effectiveness and informing its improvement efforts. Members of a multi-ERC collaborative team, funded by the NSF, have been leading a project developing a suite of common instruments and protocols which contains both quantitative and qualitative tools. This paper reports on the development of a set of qualitative instruments that, to date, includes the following: (a) a set of interview/focus group protocols intended for various groups of ERC personnel, centered around five common topics/areas, and (b) rubrics for summer program participants' verbal poster/presentations and their written poster/slide deck presentation artifacts. The development process is described sequentially, beginning with a review of relevant literature and existing instruments, followed by the creation of an initial set of interview questions and rubric criteria. The initial versions of the tools were then pilot-tested with multiple ERCs. Feedback sessions with education/evaluation leaders of those piloting ERCs were then conducted, through which further revision efforts were made.
AB - National Science Foundation (NSF) funded Engineering Research Centers (ERC) are required to develop and implement education and outreach opportunities related to their core technical research topics to broaden participation in engineering and create partnerships between industry and academia. Additionally, ERCs must include an independent evaluation of their education and outreach programming to assess their performance and impacts. To date, each ERC's evaluation team designs its instruments/tools and protocols for evaluation, resulting in idiosyncratic and redundant efforts. Nonetheless, there is much overlap among the evaluation topics, concepts, and practices, suggesting that the ERC evaluation and assessment community might benefit from having a common set of instruments and protocols. ERCs' efforts could then be better spent developing more specific, sophisticated, and time-intensive evaluation tools to deepen and enrich the overall ERC evaluation efforts. The implementation of such a suite of instruments would further allow each ERC to compare its efforts to those across other ERCs as one data point for assessing its effectiveness and informing its improvement efforts. Members of a multi-ERC collaborative team, funded by the NSF, have been leading a project developing a suite of common instruments and protocols which contains both quantitative and qualitative tools. This paper reports on the development of a set of qualitative instruments that, to date, includes the following: (a) a set of interview/focus group protocols intended for various groups of ERC personnel, centered around five common topics/areas, and (b) rubrics for summer program participants' verbal poster/presentations and their written poster/slide deck presentation artifacts. The development process is described sequentially, beginning with a review of relevant literature and existing instruments, followed by the creation of an initial set of interview questions and rubric criteria. The initial versions of the tools were then pilot-tested with multiple ERCs. Feedback sessions with education/evaluation leaders of those piloting ERCs were then conducted, through which further revision efforts were made.
KW - ERC Evaluation
KW - Evidence-based practice paper
KW - Qualitative Tools
UR - http://www.scopus.com/inward/record.url?scp=85138293526&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85138293526&partnerID=8YFLogxK
M3 - Conference article
SN - 2153-5965
JO - ASEE Annual Conference and Exposition, Conference Proceedings
JF - ASEE Annual Conference and Exposition, Conference Proceedings
T2 - 129th ASEE Annual Conference and Exposition: Excellence Through Diversity, ASEE 2022
Y2 - 26 June 2022 through 29 June 2022
ER -