How a Focus on Data Quality, Capacity, and Culture Supports Data-Driven Action to Improve Student Outcomes
A PCG Education White Paper (June 2010)
By David Ronka, Robb Geier, and Malgorzata Marciniak
The current age of greater accountability in schools has challenged educators to seek effective ways to incorporate data into their decision making processes from the central office to the classroom. However, this is not just a matter of collecting more data. For data to inform decisions about policy, programs, practice, and student placement, three critical factors need to be taken into consideration: data quality, data capacity, and data culture. This White Paper describes a research-grounded model for data use and discusses these three factors, why they are important, and how they support effective data use in schools and districts.
Schools rely on “random acts of improvement” (Bernhardt 2006, p. 30) when educators do not set clear targets for improvement and then use data to track progress against measurable indicators to reach those targets. Data can be used to formulate appropriate and effective education policy and to measure the effectiveness of programs and instructional interventions. Data can also be used to measure individual student progress, guide the development of curriculum, determine appropriate allocation of resources, and report progress to the community. But despite the leverage that can be gained by using data effectively, many schools still struggle with data-driven decision making (Mason, 2002; Ingram, Louis, & Schroeder, 2004; Boudett & Steele, 2007; Stid, O’Neill, & Colby, 2009). This paper discusses a theory of action that links the conditions necessary for data use to the types of decisions that can be informed by data to improve student outcomes. The paper will present the overall theory of action followed by a discussion of the two primary components (conditions for data use and examples of datadriven decision making in schools) and will end with a discussion of the implications for school and district leaders.
Theory of Action
Fifteen case studies published between 2002 and 2009 were analyzed to identify conditions in schools and districts that support data-driven decision making at the district, school, and classroom levels. Specific data-driven actions were documented within and across the cases in order to formulate a description of what effective data-driven decision making looks like in a district and school. The theory of action that emerged is represented in the graphic below. It contains three foundational conditions for data use (conditions), that enable different types of data-driven actions related to policies, programs, practices, and student placement (actions), and that together are linked to improved student outcomes (results).
According to the theory of action, if the necessary conditions for data use (data quality, data capacity, and data culture) are in place, and data are being used to formulate policy, evaluate and design programs, guide practice, and place students in appropriate instructional settings, then increased student achievement will result. However, it does appear that for data use to have a profound impact on student achievement, data use must be sustained over time, take place systemically throughout all levels of the organization, and be student centered. This theory of action, which emerged through a coding of the case studies (see appendix), has been reinforced by our work with schools and districts over the past decade.
Conditions for Data Use
There has been much progress in the area of data use by educators at the district, school, and classroom level. However, many schools and districts still only use data superficially. Superficial data use happens when data are used inconsistently and/or inappropriately in pockets of the organization without systematic procedures, expectations, and accountability in place. In these environments, there may be some who engage in effective data use practices. However, in the same school or district, data may also be used to punish educators, to justify the status quo, or to make critical placement decisions based on single data points (e.g., one assessment’s results) that restrict options and opportunities for students. Systemic data use, on the other hand, is where data are routinely and collaboratively used at all levels to inform organizational, program, and instructional improvement decisions directed at improving student outcomes. But this doesn’t just happen. It takes a concerted and deliberate effort for school and district administrators to put the necessary conditions in place that support and empower data-driven actions.In this paper we are primarily focused on student outcome data—that is, information about student learning (e.g., assessment or test data) and student engagement (e.g., attendance, conduct, graduation rates). There are many types of data that can inform schools of their progress toward goals, (e.g., incidents of vandalism, number of certified teachers, number of students enrolled in advanced classes). Our focus in this paper is on how schools and districts can most productively use data directly related to student outcomes to identify and understand issues related to curriculum, instruction, and assessment and make changes in how they operate in order to improve those outcomes.
Successful conditions that were present in many of the case study schools and districts can be distilled into three categories: data quality, data capacity, and data culture. It appears that these conditions are fundamental to effective data-driven decision making. These three areas synergistically interact to create an environment where data use is powered by high quality data, enabled by various data capacities, and supported by a culture of accountability and collaboration. In the next sections of this paper, each of these is discussed.
Access to high quality data can lead to greater levels of systemic data use and ultimately to improved student outcomes.
Data quality includes:
- Using multiple measures to ensure relevance and the ability to triangulate
from more than one data set;
- Making sure data are well organized and presented in data displays that
are easy to interpret;
- Using accurate data that have been standardized and cleansed;
- Making data available to stakeholder groups before the data “shelf life”
has expired; and
- Disaggregating data for analyzing across multiple factors.
Without high quality data, stakeholder groups can lose faith in the value of data and become discouraged. At worst, educators can use poor quality data — data that are old, that are not disaggregated, or that are presented in confusing or inaccurate ways — and draw false conclusions about district or school needs. This can result in “data-driven” actions that can actually cause harm. It is important for districts and schools to put safeguards in place to address data quality.
Data capacity is the next condition for data use. Without the capacity to access, understand, and use the data that are available, no amount of data (high quality or not) will lead to meaningful data use. In fact, without data capacity, the more data an organization has, the less it will be able to do with it. If data quality is the fuel, data capacity is the engine that converts the fuel to energy. Data capacity includes:
- Organizational factors such as team structures, collaborative norms, and clearly defined roles and responsibilities that support data use;
- Technology that can integrate data from multiple sources;
- Data accessibility that allows multiple users to have access to data in formats that are easy to interpret; and
- Data literacy and assessment literacy skills so data consumers know how to analyze multiple types of data and properly interpret results.
Schools and districts can improve data capacity by ensuring there has been adequate staff training on how to analyze and interpret test results, setting aside time for instructional and administrative teams to meet and discuss data, and establishing processes and procedures for accessing relevant data.
A culture of data use can only develop if data quality and capacity are in place. A strong data culture results when an organization believes in continuous improvement and regularly puts that belief into practice. Schools and districts that have a strong data culture emphasize collaboration as a keystone for success and they empower teachers and administrators to make decisions for which they are held accountable. Elements of a strong data culture include:
- Commitment from all stakeholder groups to make better use of data;
- A clearly articulated vision for data use;
- Beliefs about the efficacy of teaching and the value of data in improving teaching and learning;
- Accountability for results coupled with empowering teachers to make instructional changes;
- A culture of collaboration at all levels;
- Modeling of data use by school and district leaders; and
- Commitment to making ongoing instructional and programmatic improvements.
Questions to consider when assessing the extent to which a culture of data use is present within a district or school include:
- Is there commitment by all key stakeholders to use data for continuous improvement?
- Are people held accountable for the use of data at the school and classroom level?
- Is collaboration among staff highly valued?
- Do school leaders model data-driven decision making as a key aspect of their roles and responsibilities?
- Do teachers believe that data can and should be used to inform instruction?
- Are teachers open to changing their instruction based on data about student learning?
Data quality, capacity, and culture are the conditions necessary for systemic data use to exist within a school or district. But they are not the same as data-driven action. Rather, they are the foundation for data-driven action. Our analysis of the 15 case studies was framed by two key questions: What does a data-driven school or district look like? What kinds of data-driven actions do schools and districts take that successfully use data to improve student achievement? Four categories of data-driven actions emerged from our analysis. These categories also have been evident in our work with schools and districts across the United States and in Canada. Successful data-driven districts and schools use data in four key areas: to formulate sound policy, design and evaluate educational programs, guide classroom practice, and inform student placement.
Policy decisions lay the groundwork for educational practice. Data driven policies can have a powerful impact on needs assessment and planning processes, professional development, resource allocation, and teacher evaluation. Schools that model effective data use determine overall school needs through data drawn from multiple sources. Student performance data are used to drive the school- and district-improvement planning process. Professional development is informed by gaps identified in student performance data as well as by instructional data collected during walkthroughs and classroom observation. Resources such as time and staff are allocated based on the identified needs of students, and student assessment data are used as supplementary information in the performance evaluation of teachers.
Educational programming is the vehicle for ensuring that instruction is appropriate, targeted to identified learning needs, and aligned to established curriculum frameworks and benchmarks. In schools and districts that strive to continuously improve student outcomes, data are used to identify best practices across classrooms, to identify gaps in the curriculum, and to determine which programs are effective and which programs should be discontinued.
What happens in school hallways and classrooms in terms of practice directly influences student learning. These are the habits and actions that, taken collectively, form a learning environment that either supports or hinders growth. Data-driven practices include sharing and discussing performance data with students and parents, using data to develop lesson objectives, and adjusting teaching strategies based on evidence of student learning. Examples of what this looks like include teachers observing one another’s classrooms, leaders sharing data about progress toward school improvement goals, and instructional teams developing action plans to address specific areas of need identified through data analysis.
Finally, data should be used to ensure student placement into educational settings that are appropriate and optimally designed for student success. Teachers and administrators can use data to identify students who are at risk of academic failure or of dropping out, to guide flexible groupings of students for more focused and differentiated instruction, to identify appropriate supports and interventions, and to monitor the progress of students.
What Does Effective Data Use Look Like in Practice?
In order to show how these conditions and data-driven actions look in actual schools and districts, this section of the paper presents five short descriptions of data use drawn from the 15 case studies that were analyzed. These “snapshots” reflect data use practices found in schools and districts throughout the United States during the past 10 years. These summaries demonstrate the interplay between data quality, capacity, and culture, and demonstrate how data use practices emerge when leaders are deliberate about putting in place these conditions for effective data use.
Supovitz and Klein (2003) conducted a study highlighting how different schools and districts use multiple measures to gauge student performance. They reported that the schools in their study drew achievement data from three primary sources: external standardized tests, schoolwide periodic formative assessments, and classroom-based customized assessments. The most prevalent of these sources was external data from the state and district. A few of the schools began to experiment with systematic schoolwide assessments intended to provide interim feedback on progress toward school and grade-level goals. In classrooms, individual teachers fashioned creative and highly customized assessments. School leaders systematically analyzed a variety of student performance data at both the classroom and school levels. Rather than just relying on one individual test to provide guidance, innovative school leaders built more comprehensive systems of assessment that provided better interim information from multiple perspectives. By introducing this type of comprehensive system of assessments, teachers and school leaders could support an inquiry-oriented approach that involved ongoing and sustained investigations into the kinds of teaching that produced greater student learning.
Assuring access to quality data turned out to be critical to reducing the dropout rate in one urban district (Stid, O'Neill, & Colby, 2009). The case study illustrated how a district with only 54 percent of its high school students graduating was able to significantly address the dropout problem over the course of one calendar year. The district collected data that allowed them to conduct an initial diagnostic analysis that focused on the characteristics of students who were dropping out of high school. On the basis of this analysis, middle schools prepared reports which listed struggling students and data about their academic performance, attendance, behavior, and information about whether they had faced certain life challenges (e.g., pregnancy and parenting, homelessness, placement in foster care). These reports were provided to high school leaders early enough in the school year for them to identify and implement focused and tailored interventions for these at-risk students at the beginning of their first year in high school. In the case of ninth-grade students from one high school, such actions based on the right data at the right time resulted in a 25-percentage-point reduction in the number of students experiencing three or more core class failures in the ninth grade, which was identified as a critical threshold to prevent students from dropping out.
A study of six schools in another urban district (Mason, 2002) demonstrated the process of building capacity as a necessary intermediate step between collecting data and taking strategic action based on the data. The schools in the study faced several critical challenges: sustaining a commitment to transform data into knowledge, making data use a high priority, putting an effective ata management and integration system in place, developing analytic skills in school leaders, and building capacity to link data to school improvement planning. The district engaged the schools in a two-year project that provided training and support. Some schools experienced moderate successes, but not without some hard lessons. Participants of the project realized how challenging it was to develop collaborative norms, build the necessary internal support for the data use initiative, build the capacity among staff to use and analyze data, and then apply that knowledge strategically. At the end of the project, participants agreed that the process of using data needed a continuous and systematic focus, intensive professional development, and commitment to incorporate data use into everyday operations.
Brunner, et al. (2005) looked specifically at data use actions taken by effective teachers. The study reported that these teachers regularly used data to meet the needs of diverse learners, identify struggling students, create differentiated and individualized assignments, and provide learning materials appropriate to students’ levels. Teachers used data reports in conversations with other teachers, parents, administrators, and students. Many of the teachers used data to reflect upon the effectiveness of their own instruction and to shape their own professional development. Teachers also encouraged selfdirected learning by giving the data to students to help them take ownership over their academic performance and learning.
Ronka (2007) conducted interviews of school leaders at an elementary school during their first year of implementing a schoolwide data use initiative. The case demonstrates the importance of attending to the organizational and cultural aspects of introducing data use into the school environment. Specifically, the principal established a data team comprised of members who were representative of the school staff and who were critical to bringing about the kinds of programmatic and instructional change that might result from effective data use. The team met monthly throughout the year to monitor progress and to lay the groundwork for continuous data use by planning professional development on various uses of data, identifying data quality issues, taking action to address those issues, and coordinating data use across content areas and instructional teams. Stakeholder commitment at multiple levels was evidenced by the amount of time committed to planning and monitoring activities, and the principal’s strong leadership created an environment that was based on collaboration and focused on continual improvement.
Implications for Schools and Districts
In the case studies reviewed for this paper, each school or district applied a data-driven decision making approach for inquiry and action. The specific approach chosen, however, does not appear to be the major determinant for successful change over the long term. Making the approach “stick” requires a long-term vision for changing the way educators in the system make decisions and work to improve student results. It is this vision for changing the way decisions are made, when broadly communicated and shared throughout the organization, which guides sustainable growth through a particular data use approach. It is the task of school and district leaders to establish the vision and work toward it with strategic attention given to the three conditions for data use previously described.
Using the theory of action presented in this paper as a guide, leaders can create strategic plans to improve data quality, capacity, and culture. This can lead to a productive inquiry and action process focused on improving the conditions that support effective data-driven action. The table below presents questions schools and districts can ask to identify areas for improvement in the three foundational conditions for data use.
Careful and thoughtful attention to the conditions in which data are being used is an essential component of leadership in today’s educational environment. The proliferation of data and data systems has afforded educators the opportunity to fundamentally change the way they meet the needs of diverse students. When fostering and monitoring these conditions is a priority, then data-driven actions in areas related to policy, programming, practice, and student placement can be strategically focused on improving student achievement.
The theory of action presented in this paper advocates effective data use when making decisions about initiatives and instructional changes intended to improve student learning and achievement. When planning additions to the types and extent of data collection, enhancements to data systems, or data use professional development, we encourage education leaders at all levels to also consider the components of the theory presented in this paper. Assessing the extent to which specific strategic actions are supported by multiple types of data and a skilled culture of data use exists will enhance the likelihood that district and school improvement efforts will gain traction and ultimately lead to improved student results.
Data use initiatives too frequently fail to thrive and grow because of inattention to one or more aspects of data quality, capacity, or culture. Initiatives to expand data collection, increase data access, or foster data use that are not connected to authentic and important data-driven actions (policy, programs, practice, and placement) are not sustainable over time if the extra work they require doesn’t lead to transformative change and positive student results.
Balfanz, R., & Byrnes, V. (2006). Closing the mathematics achievement gap in highpoverty middle schools: Enablers and constraints. Journal of Education for Students Placed at Risk (JESPAR), 11(2), 143–159. Lawrence Erlbaum Associates, Inc.
Bernhardt, V. L. (2006). Using data to improve student learning in school districts. Larchmont, NY: Eye on Education, Inc.
Boudett K. P., & Steele, J. L. (Eds.) (2007). Data wise in action: Stories of schools using data to improve teaching and learning. Cambridge, MA: Harvard Education Press.
Brunner, C., Fasca, C., Heinze, J., Honey, M., Light, D., Mandinach, E., & Wexler, D. (2005). Linking data and learning: The grow network study. Journal of Education for Students Placed at Risk. 10(3), 241–267.
Fiarman. S. E. (2007). Planning to assess progress: Mason Elementary School refines an instructional strategy. In K. P. Boudett & J. L. Steele (Eds.), Data wise in action: Stories of schools using data to improve teaching and learning (chapter 7, pp. 125–147). Cambridge, MA: Harvard Education Press.
Forman, M. L. (2007). Developing an action plan: Two Rivers Public Charter School focuses on instruction. In K. P. Boudett & J. L. Steele (Eds.), Data wise in action: Stories of schools using data to improve teaching and learning (chapter 6, pp. 106–124). Cambridge, MA: Harvard Education Press.
Ingram, D., Louis, K. S., & Schroeder, R. G. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, (106)6, 1258–1287.
Kaufman, Trent E. (2007). Examining instruction: Murphy K–8 School unlocks the classroom. In K. P. Boudett & J. L. Steele (Eds.), Data wise in action: Stories of schools using data to improve teaching and learning (chapter 5, pp. 87–104). Cambridge, MA: Harvard Education Press.
Love, N., Stiles, K. E., Mundry, S., & DiRanna, K. (2008). The data coach's guide to improving learning for all students: Unleashing the power of collaborative inquiry. Thousand Oaks, CA: Corwin Press.
Mason, S. (2002). Turning data into knowledge: Lessons from six Milwaukee Public Schools. Madison, WI: Wisconsin Center for Education Research.
Ronka, D. (2007). Organizing for collaborative work: Pond Cove Elementary School lays the groundwork. In K. P. Boudett & J. L. Steele (Eds.), Data wise in action: Stories of schools using data to improve teaching and learning (chapter 1, pp. 11–28). Cambridge, MA: Harvard Education Press.
Ronka, D., Lachat, M. A., Slaughter, R., & Meltzer, J. (2008, December/January 2009). Answering the questions that count. Educational Leadership, 66(4), 18–24.
Snipes, J., Doolittle, F., & Herlihy, C. (2002). Foundations for success case studies of how urban school systems improve student achievement. How urban school systems improve student achievement. Washington, DC: Council of the Great City Schools.
Steele J. L., (2007). Acting and assessing: Community academy gets serious about homework. In K. P. Boudett & J. L. Steele (Eds.), Data wise in action: Stories of schools using data to improve teaching and learning (chapter 8, pp. 149–165). Cambridge, MA: Harvard Education Press.
Stid, D., O'Neill, K., & Colby, S., (2009). Portland Public Schools: From data and decisions to implementation and results on dropout prevention. Boston Dropout Prevention. San Francisco, CA: The Bridgespan Group, Inc.
Supovitz, J., & Klein, V. (2003). Mapping a course for improved student learning: How innovative schools systematically use student performance data to guide improvement. Philadelphia: Consortium for Policy Research in Education.
Teoh, M. B. (2007). Creating a data overview: McKay K-8 School learns to lead with data. In K. P. Boudett & J. L. Steele (Eds.), Data wise in action: Stories of schools using data to improve teaching and learning (chapter 3, pp. (53–69). Cambridge, MA: Harvard Education Press.
Thessin, R. A. (2007). Building assessment literacy: Newton North High School gets smart about data. In K. P. Boudett & J. L. Steele (Eds.), Data wise in action: Stories of schools using data to improve teaching and learning (chapter 2, pp. 29–50). Cambridge, MA: Harvard Education Press.
Tomberlin, T. (2007). Digging into data: West Hillsborough Elementary School dives deep. In K. P. Boudett & J. L. Steele (Eds.), Data wise in action: Stories of schools using data to improve teaching and learning (chapter 4, pp. 71–86). Cambridge, MA: Harvard Education Press.
About the Authors
David Ronka is a Manager at Public Consulting Group, Inc., and is located in Portsmouth, NH. David designs and delivers professional development for school leaders helping them use data to improve student outcomes. He has implemented educational data management systems for school districts and helps schools design informative and useful data displays and reports. David earned his Master of Education from Harvard and is a Teaching Fellow for Harvard’s Data Wise summer institute, working with educators around the world to help them make better use of their data. David currently teaches a graduate course in data use to aspiring principals from Boston Public Schools. Relevant publications include: Contributing author of Data Wise in Action: Stories of Schools Using Data to Improve Teaching and Learning (Boudett & Steele, 2007); Co-author of Answering the Questions that Count, Education Leadership (Ronka, et al., 2008).
Robb Geier is Director of Data Services at Public Consulting Group, Inc., and is located in Portsmouth, NH. Robb develops tools, protocols, and curricula for establishing district and school data teams focused on improving collaborative data use throughout the district. Robb also works with district and school data teams to conduct data audits to assess data quality, capacity, and culture and build strategic plans to improve processes, access, and use of data throughout the system. Robb’s work with schools includes facilitating school data teams and teacher teams, and coaching and training data coaches to lead instructional change driven by data use and inquiry. He has served as a Teaching Fellow for Harvard’s Data Wise summer institute and also worked for 10 years as a ninth-grade English teacher where he was part of an award-winning cross-curricular team, serving both general education and special education students.
Malgorzata Marciniak is a Senior Associate at Public Consulting Group, Inc., and is located in PCG’s office in Łódź, Poland. She manages projects, provides consultancy, and delivers professional development to educators. She has 10 years of managerial experience in international educational projects gained in Europe and in the U.S. Malgorzata is pursuing her PhD on using data for improving school and student performance. Her research includes school leadership and academic intervention strategies. Malgorzata holds a Master degree in English Philology from University of Łódź. She also completed the Intercultural Communication Program at the University of Tampere in Finland and the Project Management program at Harvard University.
About PCG Education™
PCG Education helps schools, school districts, and state departments of education to maximize resources, achieve their performance goals, and improve student outcomes. With more than two decades of K–12 consulting experience and over 700 professionals in 29 offices across the U.S. and in Canada, as well as its first European office in Łódź, Poland, PCG’s expertise, capacity, and scale help educators improve their decision making processes and achieve measurable results. To learn more about PCG Education, contact us at firstname.lastname@example.org, (800) 210 6113, or visit our web site at PublicConsultingGroup.com
The authors wish to thank Dr. Mary Ann Lachat, Founder and former President of the Center for Resource Management and Dr. Julie Meltzer, Senior Advisor at PCG Education, for their thoughtful feedback on this paper.
Download the document in it's entirety here:
Data-Driven District_Practical Ideas.pdf (279.35 kb)