Showing posts with label data analysis. Show all posts
Showing posts with label data analysis. Show all posts

Tuesday, February 28, 2023

Reducing Measurement Error to Increase the Statistical Reliability of Academic Performance Measures

As previously noted, the extent of measurable learning loss related to the COVID 19  pandemic is undeniable and requires new strategies, approaches, and thinking. Fortunately, to accelerate progress for the students who have fallen furthest behind—students with special needs and ELLs—Director Schneider of the Institute of Education Sciences (IES) is advocating for a widescale overhaul of education research, data collection, and analysis. Accountability data that reliably measure school performance is essential for identifying the schools that most need support. However, measurement error impacts the random differences between students’ true abilities and their test scores and thereby can impact a school’s true performance. 

 The Every Student Succeeds Act (ESSA, 2015) requires states to designate schools with low-performing student subgroups for Targeted Support and Improvement (TSI) or Additional Targeted Support and Improvement (ATSI). Measurement error is more likely and impactful to small districts, schools, or student subgroups, whereby random factors affecting fewer students create an outsized impact on the reported average score. Ensuring the reliability of school performance measures is incumbent upon state and local education agencies that must identify and provide support to the districts, schools, and students that need it.

Bayesian Interpretation of Estimates (BASIE) is an innovative framework for using an evidence-based Bayesian approach to interpret traditional impact estimates (Deke, Finucane, & Thal, 2022). Using the BASIE framework, a study of how the Pennsylvania Department of Education (PDE) identifies its schools for TSI and ATSI used Bayesian stabilization to improve the reliability of subgroup proficiency measures. The Regional Education Laboratory Mid-Atlantic study team (Farrow, Starling, & Gill, 2023) applied two statistical models to subgroup-specific proficiency rates. One model aligned with PDE’s accountability rules for ATSI.  The other model aligned with rules for TSI. To assess whether stabilization increased the statistical reliability, the results of the stabilization models were then compared with the un-stabilized proficiency rates that are currently used in accountability calculations.

 A goal of statistical analysis should be is to reduce measurement error and thereby increase the statistical reliability of academic performance measures.  Therefore, the BASIE framework shows promise.  In the 2023 study, Bayesian stabilization improved statistical reliability, showing similar variation across subgroup sizes, indicating more reliable results by reflecting less measurement error. Therefore, as the Regional Education Laboratory Mid-Atlantic study team concluded, utilization of Bayesian stabilization could allow for inclusion of subgroup sizes under 20 because stabilization showed proficiency rates for subgroups with 10–19 students varied less than un-stabilized proficiency rates for subgroups with 20–29 students.

 The 2023 Regional Education Laboratory Mid-Atlantic study suggests that stabilized data using an evidence-based Bayesian approach to interpret traditional impact estimates meets the current reliability requirements. More studies should further test this hypothesis. Improving the reliability of proficiency rates will absolutely benefit interpretation of subgroup proficiency measures and thereby optimize the identification of schools that most need support.

 

To cite:

Anderson, C.J. (February 28, 2023). Reducing measurement error to increase the statistical

            reliability of academic performance measures. [Web log post] Retrieved from

            http://www.ucan-cja.blogspot.com/

 

 

References

Anderson, C.J. (July 31, 2021). Generalizing virtual strategies that worked and planning for

accelerated learning. [Web log post] Retrieved from http://www.ucan-cja.blogspot.com/

 

Anderson, C.J. (January 31, 2023). Understanding the Needs of Struggling Learners 

            in Relation to Post COVID Accelerated Learning Goals. [Web log post] Retrieved from

            http://www.ucan-cja.blogspot.com/


Deke, J., Finucane, M., Thal, D. (2022) The BASIE (BAyeSian Interpretation of Estimates)

            framework for interpreting findings from impact evaluations: A practical guide for

            education researchers. U.S. Department of Education, Institute of Education

            Sciences, National Center for Education Evaluation and Regional Assistance

 

Farrow, L. Starling, J. & Gill B. (2023). Stabilizing subgroup proficiency results to improve the

             identification of low-performing schools. Regional Education Laboratory (REL)

            Mid-Atlantic study team. Retrieved from

            https://ies.ed.gov/ncee/rel/Products/Publication/106926

 

Monday, May 6, 2013

How the Common Core Standards can be a Part of the Solution to Educational Reform


The publication of “A Nation at Risk: The Imperative for Educational Reform” (1983) renewed interest in education reform efforts.  The result was most states adopting some form of content-based learning standards by the end of the 20th century.  During this period, business leaders, such as Bill Gates of Microsoft, Craig Barrett of Intel and Louis V. Gerstner, Jr. of IBM, began suggesting the key educational issue should be identifying the sort of student needed to emerge from a high school education.  As a result, in 1996 state governors and business leaders created Achieve, Inc.  As a bipartisan, non-profit organization its mission was to help states raise academic standards, improve assessments and strengthen accountability to prepare all young people for postsecondary education, work and citizenship.  Initially, five states developed the American Diploma Project (ADP).  By developing a series of rigorous standards identifying the skills and knowledge expected of any student receiving a high school diploma, these five states and Achieve, Inc., formed a basis for educational reform that eventually became the Common Core Standards Initiative.  By 2009, thirty states had aligned their learning standards with those previously explicated by the American Diploma Project (ADP).  The rationale for the ADP, and subsequently the Common Core State Standards, is communities will be better positioned to compete successfully in the global economy when American students are fully prepared for the future.
As with every federally backed initiative since the ESEA (1965) and PL 94-142 (1975), it is difficult to argue with a well-scripted rationale.  Pragmatically, not too many politicians wanted to raise their hand in objection to NCLB, (2001).  After all, who wanted to go on record as being the person willing to leave a child behind?
If the rationale of most federal initiatives is logical and sensible, then what causes the educational inertia perpetuating failure?  Some may argue politics are to blame.  Others feel justified in their complaints with the bureaucracy interfering with change.  However, what if the political and bureaucratic shortcut requesting “alignment” was actually the source of the problem?  Politically, it is too often diplomatically prudent to help decision-makers feel better by suggesting, “you already created a great structure, all you really need to do is some rearrangement.”  From a budgetary perspective, it is cost efficient to tell an assessment publisher to adapt its pool of test items created for the previous learning standards and align items with the Common Core State Standards.  Promoting “alignment” with what is proven to be ineffective rather than honestly seeking effective, sustained change, may be the epitome of Einstein’s quote, “We can't solve problems by using the same kind of thinking we used when we created them.”
Tying adoption of the Common Core State Standards to the Obama administration’s Race to the Top competition was political genius.  By the middle of 2010, the American Recovery and Reinvestment Act of 2009 funds were being exhausted.  Therefore, thirty-nine states facing massive educational budget shortfalls by 2011, did in less than 100 days what was unattainable for the previous 100 years.  Yes, throughout the United States, states adopted common standards for English and mathematics guiding learning each year from Kindergarten through High School.  In most cases, enlightenment was not the impetus.  Indeed, the need for a piece of the 3.4 billion dollars being offered to state education departments winning the September 2010 Race to the Top competition accomplished what logic, reason, or the best interests of all children could not do previously.  Currently, forty-five states and the District of Columbia have adopted the Common Core State Standards.
Bureaucratically, the educational assessment machine that is Pearson will make breaking the current inertia of failing to monitor and adjust based on student data very difficult.  Alignment is synonymous with profits.  Any assessment company will charge much less for alignment rather than a complete overhaul.  Yet, it still charges for a “new assessment” although most items are from the previous test item bank. 
High expectations explicated as learning standards are a good thing.  Ensuring every state promotes high expectations within its learning standards is a good thing too as the United States must compete in a global economy.  So, could the problem once again be poor implementation? 
In a recent Los Angeles Times editorial (April 22, 2013) the dilemma facing implementation of the Common Core State Standards was well explicated:
Experts are divided over the value of the new curriculum standards, which might or might not lead students to the deeper reading, reasoning and writing skills that were intended.  But on this much they agree: The curriculum will fail if it isn’t carefully implemented with meaningful tests that are aligned with what the students are supposed to learn.  Legislators and education leaders should be putting more emphasis on helping teachers get ready for common core and giving them a significant voice in how it is implemented.  And if the state can’t get the right elements in place to do that by 2014, it would be better off delaying the new curriculum a couple of years and doing it right, rather than allowing common core to become yet another educational flash in the pan that never lives up to its promise.
High expectations explicated as learning standards are part of the variables called leading indicators for success.  Subsequently, test results produce a trailing indicator-exhibiting either student success or failure.  Effective implementation is what comes between the high expectations and the test results. 
A mission of promoting learning for all requires a district to either create a vertically aligned, curriculum-based system or modify a purchased system to make it appropriate for the district’s needs.  An effective system would be able to monitor and adjust established “nonnegotiable goals for achievement and instruction” (Marzano & Waters, 2009, p. 23).
The increased expectations for monitoring and adjusting in real time would be nearly impossible without a technology-based student information and instructional management system.  The Kennewick School District case study provides an actual example of the value of a technology-based student information and instructional management system.  Research by Fielding, Kerr, and Rosier (2007) documented how the achievement gap between economically poor and disadvantaged students and their non-disadvantaged counterparts could be closed by a four-step Targeted Accelerated Growth (TAG) loop process.  The TAG loop process includes the following steps:
(1)   discovering through the administration of diagnostic assessment the 
      sub-skill deficiency,
(2) providing increased direct instructional time,
(3) focused teaching to the deficient sub-skill, and
(4) retesting to assure that learning has actually occurred (Fielding et al, 2007, 
      p. 19). 
The TAG loop process is a process not a linear model.  With any process, change resulting from the interpretation of reliable data is crucial for success based on application of correct micro-adjustments.  It would seem diagnostic testing and professional development for the teaching staff optimizes effectiveness in efficiently using data to implement proportional micro-adjustments in instructional time.  Therefore, with an efficient and effective student information and instructional management system, how to utilize data to effectively and efficiently diagnose and implement needed change becomes possible.    
Before implementing the principle of monitor and adjust effectively, leaders must identify the problem and then lead continuous improvement systems and processes.  To lead continuous improvement systems and processes, the effective educational leader evaluates the Five Ts of Continuous Improvement: Theories, Teams, Tools, Time, and Technology.  The effective educational leader then ensures appropriate performance criteria are established.
A school or district’s continuous improvement goals for a given academic year must succinctly indicate implementation steps for each improvement goal.  The current framework utilized for continuous school improvement encourages no more than three goal statements per year.  Given the complexity of the school system, too many goals adversely impact the level of human energy devoted to the initiative.  Displaying a limited number of continuous improvement goals allows followers and stakeholders to monitor growth through well-explicated action plan.  Therefore, anything more than three continuous improvement goals will be visually overwhelming and thereby perceived as unmanageable.  
The Re-Inventing Schools Coalition (RISC) system frequently monitors student performance and ensures alignment between the “intended curriculum”…,“implemented curriculum”…, and the “attained curriculum” (DeLorenzo et al, 2008, p. 64).  Based on research and best practices, implementation of the correlates for Effective Schools is therefore prudent when seeking school improvement.  Disaggregated data helps “the district, its schools, and its teachers to evaluate their effectiveness” (Davenport & Anderson, 2002, p. 62). 
The result of frequent monitoring and analysis means data guides instructional decisions.  However, effective use of data depends on how well educational leaders are able to guide the process.  The continuous improvement process in education should develop a building-wide culture whereby all systems, processes, strategies, and actions define “how we do things around here” Lezotte & Snyder, 2011, p. 141).  Without interdependent implementation of all the correlates of Effective Schools Research, any progress is adversely impacted.

To cite:
Anderson, C.J. (May 6, 2013) How the common core standards can be a part of the solution to
educational reform.  [Web log post] Retrieved from http://www.ucan-cja.blogspot.com/

References:
Davenport, P., & Anderson, G. (2002). Closing the achievement gap: No excuses. Houston, TX:
                APQC
Delorenzo, R. A., Battino, W., Schreiber, R. M., Carrio, B. G. (2008). Delivering on the promise:
                The education revolution. Bloomington, IN: Solution Tree. ISBN-13: 9781934009420
Fielding, L., Kerr, N., & Rosier, P. (2007). Annual growth for all students, catch-up growth
             for those who are behind. New York: Foundation Press.
Lezotte, L. W., & Snyder, K. M. (2011). What effective schools do: Re-envisioning the
                correlates. Bloomington, IN: Solution Tree Press.
Marzano, R. & Waters, T.(2009).  District Leadership That Works. Bloomington, In: Solution
                Tree Press

Tuesday, December 4, 2012

Aligning a Response to Intervention System with Required Teacher Competencies


In collaboration with a multidimensional team of stakeholders that includes parents or guardians, a crosswalk between the correlates of Effective Schools Research and tenets of an effective Response to Intervention (RTI) program allows RTI to become an efficient and effective system.  This promotes equity in quality within schools committed to the pursuit of learning for all.  While there is a danger to prescribing interventions in a limiting fashion, RTI anthologies have the potential for providing a clearinghouse of research-based and success-proven strategies and interventions.  
Subsequent alignment of the district's non-negotiable goals with individual school needs and experiences (Marzano & Waters, 2009) should expand rather than limit the district's clearinghouse of research-based and success-proven strategies and interventions.  Promoting such a clearinghouse of identified successes ensures professional development is available.  This optimizes access, review, and implementation of a range of interventions, thereby enriching teachers through defined autonomy.
The core competency required for effective implementation of RTI interventions is the ability to correctly collect, analyze, and utilize data.  For many prospective and in-service teachers, data equates to math.  Unfortunately, too many teachers are uncomfortable with mathematics.  To address this, we will eventually need to recognize this country's math scores are related to too many teachers being over reliant on the text book rather than proficiency in "thinking mathematically” (Edelmuth, 2006). 
Therefore, Schools of Education and Alternate Route Teacher Preparation Programs need to ensure graduates are able to collect data, evaluate results, and be an honest consumer of the resulting data.  This becomes possible through better statistics and data analysis courses for prospective teachers.  Effective districts will need to complement this evolution through in-service professional development on collecting data, evaluating results, and being an honest consumer of the resulting data.
Frequent monitoring of student progress, and adjusting as indicated by results, is a correlate of continuous school improvement within Effective Schools.  This correlate requires teacher competency in collecting data, evaluating results, and being an honest consumer of the resulting data.  When teachers identify and prescribe an intervention, they often have difficulty accepting the need to change (adjust) if the prescribed intervention proves ineffective.  Too often the failure of the intervention is perceived a personal failure of the initial prescription, which can then delay the necessary adjustment.  For this reason, a district and school is well-advised to consider the following six ideas for successful development of an effective RTI system:
  1. Encourage participation by key stakeholders during planning and implementation.
  2. Elicit strong administrative support in staff development, instructional integrity, and data collection.
  3. Provide in-depth staff development with mentoring, modeling, and coaching.
  4. Begin follow-up trainings at the beginning of each school year.
  5. Distribute a manual outlining procedures and materials.
  6. Build Problem Solving Models including RTI into school schedules and the student improvement process (Lau, Sieler, Muyskens, et al, 2006).
Implementation of the effective RTI system can then begin.  However, potential problems will be omnipresent without administrative support and ongoing professional development.  The following core principles identify potential challenges for implementing RTI.   
  • effectively teaches each and every student,
  •  provides early intervention,
  • uses a multi-tier model of service delivery,
  • uses a problem-solving method to make decisions within the multi-tier model,
  • uses research-based validated interventions/instruction,
  • monitors student progress to inform instruction,
  • uses data to make decisions, and
  • uses assessment for three purposes: screening, diagnostics, and progress monitoring.
      Both pre-service and in-service teachers openly admit to the difficulty of monitoring student progress to inform instructional decisions.  The inter-relationship between the identified core principles makes the pursuit of a hierarchy subjective at best and futile at worst.  The correlate of frequent monitoring and subsequent adjustment drives the core principles for implementing RTI.  The ability to collect data, evaluate results, and be an honest consumer of the resulting data promotes the correlate of frequent monitoring and subsequent adjustment.  Teacher proficiency with data must therefore become a professional competency. 

To cite:
Anderson, C.J. (December 4, 2012) Aligning a response to intervention system with teacher
            competencies [Web log post] Retrieved from http://www.ucan-cja.blogspot.com/

References:
Edmonds, R. (1979). Effective Schools for the urban poor. Educational Leadership, 37,
15-24.
Lau, Sieler, Muyskens, Canter, VanKeuren, & Marston (2006).  Perspectives on the use of 
            the Problem-Solving Model from the viewpoint of school psychologist, administrator, and
            teacher. Psychology in the Schools, 43 (1), 117-127.
Marzano, R. & Waters, T.(2009).  District leadership that works. Bloomington, In: Solution
            Tree Press