Reducing Measurement Error to Increase the Statistical Reliability of Academic Performance Measures

As previously noted, the extent of measurable learning loss related to the COVID 19  pandemic is undeniable and requires new strategies, approaches, and thinking. Fortunately, to accelerate progress for the students who have fallen furthest behind—students with special needs and ELLs—Director Schneider of the Institute of Education Sciences (IES) is advocating for a widescale overhaul of education research, data collection, and analysis. Accountability data that reliably measure school performance is essential for identifying the schools that most need support. However, measurement error impacts the random differences between students’ true abilities and their test scores and thereby can impact a school’s true performance. 

 The Every Student Succeeds Act (ESSA, 2015) requires states to designate schools with low-performing student subgroups for Targeted Support and Improvement (TSI) or Additional Targeted Support and Improvement (ATSI). Measurement error is more likely and impactful to small districts, schools, or student subgroups, whereby random factors affecting fewer students create an outsized impact on the reported average score. Ensuring the reliability of school performance measures is incumbent upon state and local education agencies that must identify and provide support to the districts, schools, and students that need it.

Bayesian Interpretation of Estimates (BASIE) is an innovative framework for using an evidence-based Bayesian approach to interpret traditional impact estimates (Deke, Finucane, & Thal, 2022). Using the BASIE framework, a study of how the Pennsylvania Department of Education (PDE) identifies its schools for TSI and ATSI used Bayesian stabilization to improve the reliability of subgroup proficiency measures. The Regional Education Laboratory Mid-Atlantic study team (Farrow, Starling, & Gill, 2023) applied two statistical models to subgroup-specific proficiency rates. One model aligned with PDE’s accountability rules for ATSI.  The other model aligned with rules for TSI. To assess whether stabilization increased the statistical reliability, the results of the stabilization models were then compared with the un-stabilized proficiency rates that are currently used in accountability calculations.

 A goal of statistical analysis should be is to reduce measurement error and thereby increase the statistical reliability of academic performance measures.  Therefore, the BASIE framework shows promise.  In the 2023 study, Bayesian stabilization improved statistical reliability, showing similar variation across subgroup sizes, indicating more reliable results by reflecting less measurement error. Therefore, as the Regional Education Laboratory Mid-Atlantic study team concluded, utilization of Bayesian stabilization could allow for inclusion of subgroup sizes under 20 because stabilization showed proficiency rates for subgroups with 10–19 students varied less than un-stabilized proficiency rates for subgroups with 20–29 students.

 The 2023 Regional Education Laboratory Mid-Atlantic study suggests that stabilized data using an evidence-based Bayesian approach to interpret traditional impact estimates meets the current reliability requirements. More studies should further test this hypothesis. Improving the reliability of proficiency rates will absolutely benefit interpretation of subgroup proficiency measures and thereby optimize the identification of schools that most need support.

 

To cite:

Anderson, C.J. (February 28, 2023). Reducing measurement error to increase the statistical

            reliability of academic performance measures. [Web log post] Retrieved from

            http://www.ucan-cja.blogspot.com/

 

 

References

Anderson, C.J. (July 31, 2021). Generalizing virtual strategies that worked and planning for

accelerated learning. [Web log post] Retrieved from http://www.ucan-cja.blogspot.com/

 

Anderson, C.J. (January 31, 2023). Understanding the Needs of Struggling Learners 

            in Relation to Post COVID Accelerated Learning Goals. [Web log post] Retrieved from

            http://www.ucan-cja.blogspot.com/

Deke, J., Finucane, M., Thal, D. (2022) The BASIE (BAyeSian Interpretation of Estimates)

            framework for interpreting findings from impact evaluations: A practical guide for

            education researchers. U.S. Department of Education, Institute of Education

            Sciences, National Center for Education Evaluation and Regional Assistance

 

Farrow, L. Starling, J. & Gill B. (2023). Stabilizing subgroup proficiency results to improve the

             identification of low-performing schools. Regional Education Laboratory (REL)

            Mid-Atlantic study team. Retrieved from

            https://ies.ed.gov/ncee/rel/Products/Publication/106926