Accountability in Education
Content
Accountability in education
Jo Anne Anderson
1
p
catio Edu n
o
l
International Academy of Education International Institute for Educational Planning
ic
y
ries se
The International Academy of Education
The International Academy of Education (IAE) is a not-for-profit scientific association that promotes educational research, its dissemination, and the implementation of its implications. Founded in 1986, the Academy is dedicated to strengthening the contributions of research, solving critical educational problems throughout the world, and providing better communication among policy makers, researchers, and practitioners. The seat of the Academy is at the Royal Academy of Science, Literature and Arts in Brussels, Belgium, and its co-ordinating centre is at Curtin University of Technology in Perth, Australia. The general aim of the Academy is to foster scholarly excellence in all fields of education. Towards this end, the Academy provides timely syntheses of research-based evidence of international importance. The Academy also provides critiques of research, its evidentiary basis, and its application to policy. The members of the Board of Directors of the Academy are: • Erik De Corte, University of Leuven, Belgium (President); • Barry Fraser, Curtin University of Technology, Australia (Executive Director); • Monique Boekaerts, Leiden University, The Netherlands; • Jere Brophy, Michigan State University, USA; • Eric Hanushek, Hoover Institute, Stanford, USA; • Denis Phillips, Stanford University, USA; • Sylvia Schmelkes, Departamento de Investigaciones Educativas, Mexico. The following individuals, all Fellows of the Academy, are the members of the Editorial Committee for the Education Policy Booklet Series.
• • • •
•
Lorin Anderson, University of South Carolina, USA Eric Hanushek, Hoover Institute, Stanford, USA T. Neville Postlethwaite, University of Hamburg, Germany Kenneth N. Ross, International Institute for Educational Planning, Paris, France The Director of the International Institute for Educational Planning, Paris, France (Ex-officio). http://www.smec.curtin.edu.au/iae/
The International Institute for Educational Planning
The International Institute for Educational Planning (IIEP) was established in Paris in 1963 by UNESCO, with initial financial help from the World Bank and the Ford Foundation. The French Government provided resources for the IIEP’s building and equipment. In recent years the IIEP has been supported by UNESCO and a wide range of governments and agencies. The IIEP is an integral part of UNESCO and undertakes research and training activities that address the main priorities within UNESCO’s overall education programme. It enjoys intellectual and administrative autonomy, and operates according to its own special statutes. The IIEP has its own Governing Board, which decides the general orientation of the Institute’s activities and approves its annual budget. The IIEP’s mission is capacity building in educational planning and management. To this end, the IIEP uses several strategies: training of educational planners and administrators; providing support to national training and research institutions; encouraging a favourable and supportive environment for educational change; and co-operating with countries in the design of their own educational policies and plans. The Paris headquarters of the IIEP is headed by a Director, who is assisted by around 100 professional and supporting staff. However, this is only the nucleus of the Institute. Over the years, the IIEP has developed successful partnerships with regional and international networks of individuals and institutions – both in developed and developing countries. These networks support the Institute in its different training activities, and also provide opportunities for extending the reach of its research programmes. http://www.unesco.org /iiep/
E d u c a ti o n p ol
1
I S B N : 9 2 - 8 0 3 -1 2 7 6 - 6
ic
y
se
ri e s
Preface Education policy booklet series
The International Academy of Education and the International Institute for Educational Planning are jointly publishing the Education Policy Booklet Series. The purpose of the series is to summarize what is known, based on research, about selected policy issues in the field of education. The series was designed for rapid consultation “on the run” by busy senior decision-makers in Ministries of Education. These people rarely have time to read lengthy research reports, to attend conferences and seminars, or to become engaged in extended scholarly debates with educational policy research specialists. The booklets have been (a) focused on policy topics that the Academy considers to be of high priority across many Ministries of Education – in both developed and developing countries, (b) structured for clarity – containing an introductory overview, a research-based discussion of around ten key issues considered to be critical to the topic of the booklet, and references that provide supporting evidence and further reading related to the discussion of issues, (c) restricted in length – requiring around 30-45 minutes of reading time; and (d) sized to fit easily into a jacket pocket – providing opportunities for readily accessible consultation inside or outside the office. The authors of the series were selected by the International Academy of Education because of their expertise concerning the booklet topics, and also because of their recognized ability to communicate complex research findings in a manner that can be readily understood and used for policy purposes. The booklets will appear first in English, and shortly afterwards in French and Spanish. Plans are being made for translations into other languages. Four booklets will be published each year and made freely available for download from the web site of the International Institute for Educational Planning. A limited printed edition will also be prepared shortly after electronic publication.
Accountability in education
I
This booklet
As the economies of nations compete for strong positions within a competitive global market place, many governments have become increasingly interested in the performance of all aspects of their education systems. This trend, coupled with the enormous expenditures that are devoted to education, has also precipitated widespread public requests for higher levels of scrutiny concerning the quality of education. These demands for information about school system performance can only be addressed through the implementation of systematic accountability systems. Historically, the education profession has conformed to the requirements of regulatory or compliance accountability systems (usually based on government statutes), and has also subscribed to professional norms established by associations of educators. However, at the beginning of the 21st Century, accountability systems have also been required to respond to demands that professional performance be judged by the results that have been achieved. This booklet offers a set of principles and strategies to be considered in the development and implementation of results-based accountability systems. Technical and political issues are addressed as well as the ways in which educators, policymakers, and community members can use the information from accountability systems to improve results. The statements presented here about accountability systems are likely to be generally applicable throughout the world. Even so, they should be assessed with reference to local conditions, and adapted accordingly. In any educational setting or cultural context, suggestions or guidelines for practice require sensitive and sensible application, and continuing evaluation.
II
Education policy series
1
Jo Anne Anderson
has been active in the development and evaluation of educational policy for over twenty years, serving directly or in an advisory capacity to state, regional, and national organizations. She currently holds the position of Executive Director of the Education Oversight Committee for the state of South Carolina in the United States. This agency is responsible for the creation and implementation of the State Accountability System.
Accountability in education
III
This publication has been produced by the International Academy of Education (IAE) and the International Institute for Educational Planning (IIEP). It may be freely reproduced and translated into other languages. Please send a copy of any publication that reproduces this text in whole or part to the IAE and the IIEP. This publication is available on Internet in its printed form, see: http://www.unesco.org/iiep The author is responsible for the choice and presentation of the facts contained in this publication and for the opinions expressed therein which are not necessary those of IIEP (UNESCO) and do not commit the Organization. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of IIEP (UNESCO) concerning the legal status of any country, territory, city or area, or of its authorities, or concerning the delimitation of its frontiers or boundaries.
Jointly published by: The International Institute for Educational Planning (IIEP) 7-9 rue Eugène Delacroix 75116 Paris France and: The International Academy of Education (IAE) Palais des Académies 1, rue Ducale 1000 Brussels Belgium Design and layout by: Sabine Lebeau Printed by: Stedi Média © UNESCO 2005
ISBN: 92-803-1276-6
IV
Education policy series
1
Table of contents
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.
Types of accountability system Values and aspirations The goals of schooling The main components of accountability systems
1 3 5 7
Aligning the components of an accountability system 10 The use of student assessment data Information about the context of accountability systems The need for high quality information Performance standards The generation of useful information Parent and community involvement 12 14 16 18 20 22 24
References
Accountability in education
V
1
Types of accountability system
There are three main types of accountability systems that are sometimes applied simultaneously in education systems.
In the field of education there are three main types of accountability system: (a) compliance with regulations, (b) adherence to professional norms, and (c) results driven. School accountability systems operate according to a set of principles and use a variety of implementation strategies. In this booklet, these principles and strategies are described, with particular attention given to the political and technical aspects of accountability. Accountability systems are not new. The differences between current systems and those employed previously are matters of “for what” and “to whom.” Educators have worked mostly within three accountability systems, often simultaneously. The first system demands compliance with statutes and regulations such as those embodied in the British Office for Standards in Education. Anchored in an industrial model of education, compliance systems view the school as the embodiment of constant processes and allow for variation in results, generally attributed to the varying characteristics of students. Simply stated, educators were accountable for adherence to rules and accountable to the bureaucracy. The second system is based upon adherence to professional norms. Although neither mandated nor required, the impact of widespread agreement on certain principles and practices has done much to elevate education as a profession. In the United States, the curriculum and evaluation
Accountability in education
1
standards for school mathematics (National Council of Teachers of Mathematics, 1989), the standards for educational and psychological testing (American Educational Research Association, 2000), and the program evaluation standards (Joint Committee on Standards for Educational Evaluation, 1994) exemplify the professional norm approach to accountability. Within this system, educators are accountable for adherence to standards and accountable to their peers. The third accountability system is based upon results, with results defined in terms of student learning. This system has emerged from increasing political involvement in education. The “No Child Left Behind” requirements in the United States and the Australian National Education Performance Monitoring Task Force are examples of results-based systems. In these systems educators are accountable for student learning and accountable to the general public. Educators often find themselves responding to all three systems, attempting to balance the requirements of each. Professional norms complement both compliance and results systems. On the other hand, compliance and results systems often conflict. Part of this conflict stems from the fact that the emergence of results systems has been fostered by dissatisfaction with historic results; that is, those achieved under compliance systems. At present, accountability systems focus less on compliance and more on results. What are the components of a workable, defensible accountability system that is based primarily on results, while at the same time being attentive to professional norms and regulatory compliance requirements. First, the system defines educators’ responsibility for all students, regardless of the advantages or disadvantages they bring to school. Second, the system must be built upon aligned components—objectives, assessments, instruction, resources, and rewards or sanctions. Third, the technical aspects of the system must meet high standards. Fourth, the system must provide the vehicle for positive change.
2
Education policy series
1
2
Values and aspirations
Accountability systems embody prevailing societal values and aspirations.
The relationship between the educational attainment of citizens and the quality of their life has grown from a point of research interest to a call for action. In the second half of the 20th century governments in a number of Western nations experienced (a) low relative performance of their students on academic assessments when compared with students from certain Asian nations; and (b) a loss of historic industries (and jobs) to these nations. Within the United States, the insistence on comprehensive accountability systems was intensified by two events: widespread publication in the popular press of results from the 1995 Third International Math and Science Study (TIMSS) and the 1996 National Governors Association Education Summit. The TIMSS results suggested that United States students in Grade 3 were slightly behind their peers in other developed countries and, importantly from a policy perspective, this difference increased the longer they remained in school. At the Summit the governors from almost every state committed to introduce strong accountability measures to ensure that public schools performed at the level necessary for economic supremacy. Within two years, United States educators were grappling with the change imposed by the shift in accountability systems from those based on compliance and professional norms to one based on results. United States educators are not alone. Reviews of accountability programs throughout the world provide evidence
Accountability in education
3
that accountability is an international issue. England has a national curriculum accompanied by assessments and measures for rating schools. France, Hong Kong, China, Japan, and others use national assessments to measure student and school progress and to make decisions about each. Many European systems use examinations to determine student access to the next level of education. All these systems are based on explicit definitions of what students are expected to learn and to what level they are expected to perform. Furthermore, examinations are used to monitor student learning, with the data providing the basis for changes within the system. Educational opportunity, an extension of civil rights and economic inclusion, has been redefined: concerns for equal access and treatment have been replaced with an emphasis on equal attainment. To have equal attainment, however, variations in access and, particularly, treatment must be available to meet the needs of increasingly diverse populations of students. This focus on equal attainment has led us back to the age-old question, “What’s worth learning?” That is, what should we expect students to attain as a result of the formal education they receive? The answer to this question depends primarily on societal values. The population of students to whom this question applies depends to a great extent on the aspirations societies have for their citizens.
4
Education policy series
1
3
The goals of schooling
Accountability systems are based on the expectation that students can and will achieve the goals of schooling.
Traditionally, schools have been expected to teach students. However, there has been general acceptance that only those students who bring advantages to the school are likely to benefit from the exposure to this teaching. Minority students, economically disadvantaged students, disabled students, and other groups simply have not been expected to learn at the level of their advantaged peers. Current research findings counter the premise that some students cannot benefit from schooling. Almost a quarter century ago, Ron Edmonds’ (1979) work on effective schools identified principles that should underlie school practices. Subsequently, teaching practices have been identified and instructional models developed that promote high levels of learning for large numbers of students, regardless of the disadvantages they bring into the classroom. Intense study of Asian school systems suggests that the combination of national aspiration, cultural support, and individual effort overcomes both real and perceived barriers. Assuming responsibility for the learning of all students transforms the school and the classroom environment and, to some degree, the way that teachers view their profession and themselves. The popular literature is replete with heroic educators who, despite overwhelming odds, are able to change and improve a school through their zeal. A challenge of accountability systems is to make the heroic, customary. In results-based systems, students’
Accountability in education
5
learning failures are attributed to weaknesses in educational programs and practices rather than to students’ characteristics and backgrounds. Schools that are accomplishing the goal of all students achieving success are most likely to have strong and stable teachers and administrators. Strength comes from factors such as greater content knowledge and visionary instructional leadership. Stability, in terms of commitment to the school over time, is needed to shape the school culture and climate. Stability enables the development of relationships with parents and the community that are anchored in mutual trust and focused upon students’ present and future needs. Why then, is there scepticism about goals based upon all students learning? Educators may find themselves overwhelmed by the disadvantages that students bring to the learning environment over which they have no control. Educators also have little control over the resources available to them to achieve the goals. Administrators must build consensus around the goals and cultivate a professional dialogue that encourages the definition of solvable problems. This dialogue must be extended to the broader community so that the disadvantages students bring to the school can be ameliorated over time. Workable, defensible accountability systems are built upon aligned components—objectives, assessments, instruction, resources, and rewards or sanctions.
6
Education policy series
1
4
The main components of accountability systems
Accountability systems should include five components: objectives, assessments, instructions, resources, and rewards or sanctions.
Analyses of current results-based accountability systems reveal agreement on five guiding questions: What do we expect students to know and be able to do? How satisfied are we that students have mastered the established content standards? How are teachers prepared to be effective in their classrooms with all students? How and to what degree is the public informed about school results and the contributors to those results? How does society respond to the information they receive about the performance of schools? Content standards have shifted from the trivium of ancient Greece to today’s workforce preparation. Throughout the world, education systems emphasize literacy, mathematical reasoning, scientific inquiry, and historical and social understanding to support civic participation. Within developing nations, literacy is the most often defined learning expectation. Within developed nations, the emphasis is on increasing mathematical and scientific competence. In general, curricula mirror the economic focus of nations. The establishment of content standards impacts on the nature and structure of the curriculum. Teachers must exhibit an understanding of the structure of the curriculum both horizontally (within levels) and vertically (across levels). Access to a variety of learning resources (including supplementary materials) and extended or enriched in-
Accountability in education
7
formal learning opportunities are important. In practical terms, the introduction of content standards has proceeded at a much faster pace than have the learning resources and supportive opportunities that must be aligned with the standards if the intended learning is to occur. The use of assessments to inform decisions about students, schools, and personnel has been accelerated by the rise of results-based accountability systems. Aligned with the content standards, assessments are used to make decisions about student eligibility for and progress to the next level of school; for administrator and teacher employment and rewards; and for resource allocation. When these assessments are used in this way, they are referred to as “high stake” assessments. These “high stakes” decisions generate demands that information from assessments can be used to improve the teaching-learning process. Because they are designed for administration to large numbers of students, however, accountability assessments generally do not offer sufficient diagnostic information for teacher planning and in-class work with individual students. Some assessment programs release items and/or parallel assessments so that teachers are comfortable with both the content to be tested and the manner in which each standard is assessed. Changes in expectations about students should lead to changes in instruction. The rapid change of the curriculum, particularly in mathematics and science, has left many teachers responsible for teaching content they may not have learned in a formal setting. Teachers also are expected to adapt their teaching for students from diverse backgrounds, exhibiting a range of motivations and prior experiences. Instead of a consistent methodology yielding differentiated results, teachers are expected to differentiate their methodologies to yield consistent results for diverse student populations. Results-based accountability systems utilize public reporting to a greater degree than do the compliance or professional norms systems. In the latter two systems information about student performance is held within
8
Education policy series
1
the profession. Results-based systems rely upon widespread communication of results to parents and the general public. Many results-based systems generate school report cards or school profiles for distribution to general audiences. These reports include summaries of the performance of students or subgroups of students as well as information about resources (for example, per student expenditures), programs (for example, participation in accelerated courses), and behaviour (for example, student attendance.) Providing this information to the public has required that teachers and administrators become comfortable discussing strengths and weaknesses, explaining a variety of statistical data, and facilitating positive change. This new communications role for educators can be intimidating as educators struggle both to understand underperformance and to inspire confidence that they can lead the change process needed to improve performance. Finally, in most results-based accountability systems performance is publicly acknowledged and rewards, sometimes financial, are provided to those schools or individuals exhibiting high and/or improving performance. Schools not succeeding are provided encouragement and often technical assistance. Technical assistance is most effective when the local school assumes ownership of the results-based change process. Schools needing to improve dramatically benefit from increased attention and resources. Yet these schools also may be overwhelmed by the infusion of new practices and greater expectations for simultaneous rapid and long-lasting change. In extreme circumstances another layer of educational governance may assume management of the school. The continuum from providing technical assistance to taking control often is ill defined. Technical assistance should provide immediate and temporary support whereas assumption of responsibility extends to governance and data management. All schools are most vulnerable when the public demands quick change, rather than exercising the patience to implement sustainable changes. Long-lasting change requires integration of remedies across community agencies and responsibilities.
Accountability in education
9
5
Aligning the components of an accountability system
Attention, scrutiny, and discipline should be exercised to ensure that the five components are aligned, with concerns for alignment evident from planning through implementation.
The foundation of results-based accountability systems is clear expectations for student learning, both what students are to learn and how that learning is to be demonstrated. Thus, content standards and the accompanying assessments are the components with which the other components, most importantly, instructional materials and teaching-learning strategies, must be aligned. When content standards, assessments, materials, and strategies are aligned, students have the maximum opportunity to learn. Also, when the public understands data derived from an “aligned” accountability system, they are more likely to respond to the performance of schools in a thoughtful and supportive way. Concerns for alignment are relatively new. Throughout much of the 20th century, textbooks formed the basis for instructional planning. Although the structure and content of textbooks changed in response to discipline-based organizations, the presumption was that textbooks incorporated all that was needed to facilitate the desired student learning. Thus, alignment was part and parcel of buying into the textbook “package”. As access to multi-media and a wider range of materials increased, reliance on a primary textbook for the design of an instructional plan began to fade. Currently, the specification of content standards presumes independence from a primary text and the use of diverse materials and teaching-learning strategies.
10
Education policy series
1
How are decisions of alignment reached? Policy and disciplinary organizations, government agencies, and local school districts typically employ a professional judgment methodology. In the United States, the Council of Chief State School Officers (State Education Improvement Partnership, 1996) and the American Association for the Advancement of Science (AAAS, 2003) are among the organizations that have developed structured processes for the review of content standards and assessments. These processes require agreement on the depth and breadth of the knowledge expected within a content standard or assessment, the degree of cognitive demand and evidence of discrete or integrated knowledge, the emphasis placed on the standard in instruction or assessment, and the ways in which student learning is reported. These methodologies are relatively new and there are not similarly consistent strategies for use by local administrators and teachers. As studies of alignment expand to address instructional validity, practitioner tools and skills should be developed to inform local decisions about instructional materials and the teaching-learning process within each school community.
Accountability in education
11
6
The use of student assessment data
Data from student assessments should be the primary source for identifying the problems to be solved.
Similar concerns for alignment are evident in the testing industry. Accountability systems emphasize student mastery of specified content and rely more on criterionreferenced assessments than on norm-referenced ones to determine how well students are learning. If these measures are misaligned with content standards, the information they yield is irrelevant to determining school effectiveness. Assessments in results-based accountability systems must be of sufficient technical quality to support the decisions that are based on the results. In the United States, recommended voluntary standards for the construction and use of accountability systems have been developed in a collaborative project between the National Center for Research on Evaluation, Standards and Student Testing (CRESST) and the Consortium for Policy Research in Education (CPRE) (Baker et al., 2002). When these standards are examined in the context of the Standards for Educational and Psychological Testing, some general principles for using assessments in accountability systems emerge: • Make explicit the purposes that the assessment system and individual assessments are intended to serve; • decide on a strategy to meet the testing requirements at various grade levels;
12
Education policy series
1
• determine the degree to which validity evidence is available or could be accumulated for multiple purposes and “the widest possible range of students;” • determine a standard of adequacy for technical quality; and • make plans to acquire needed technical quality information during piloting, field trials and implementation. Assessment systems can “lower the stakes” when educators and others have sufficient documentation that the assessments have met technical standards and there is clear understanding of how the assessment data are to be used. The stakes also are lowered when assessment data are used for positive purposes such as providing technical assistance to schools, initiating supplemental services to students, and amending policies and practices that interfere with goal attainment. When the stakes are consistently negative, the assessment data are viewed sceptically; when the stakes lead to improvements, assessment data can become accepted as an integral and necessary part of the decision-making process that leads to educational improvement.
Accountability in education
13
7
Information about the context of accountability systems
Supporting information about teacher quality, curriculum rigor, and resource allocation should provide the basis for selecting or designing strategies that are most likely to solve problems.
School quality is not only evident in assessment results, but also in the diversity of programs offered, the preparation and performance of educational professionals, student behaviour and attitudes, and the relationship between the school and the community. School reports should publish contextual and programmatic information along with assessment results. This additional information provides a more complete description of the school and enhances the public’s understanding of its overall performance. The information also offers a point of comparison among schools as patterns of inputs, processes, and outputs are related to levels of school performance. Inputs include fiscal and other resources, teacher quality, students’ backgrounds, and parent/community norms. Processes include the organization of schools, the curriculum and pedagogy, and opportunities for student participation in non-academic activities. Outputs include student achievement, participation, attitudes, and aspirations (Porter, 1991). Other potentially useful information includes attendance (both teacher and student), student behaviour (or misbehaviour), teacher professional development, and parents’ and students’ perceptions of the school. As school reports gain public attention, program advocates view publication of data as a way of ensuring much needed attention to their programs.
14
Education policy series
1
Collecting and reporting these data are mammoth tasks. Few countries have educational data systems with the flexibility to extract contextual information. Most reporting systems, therefore, rely upon supplementary self-reported data. As reliance on self-reporting increases, data on program characteristics are vulnerable to hurried collections, natural inclinations to present the factor positively, and inadvertent errors. At school sites, data collection is relegated to one of many tasks in a busy environment and often becomes secondary to more immediate concerns. Results-based accountability systems require both educators and the public to understand the meaning of data, the implications of the ways in which data are aggregated, and, of greater import, ways in which the data can be used to make improvements. For example, disaggregated student mathematics scores are interesting and may point to a gap in achievement, but only when those data are interpreted within our knowledge of the curriculum and instruction are we able to determine how best to improve student performance.
Accountability in education
15
8
The need for high quality information
All data collection instruments and procedures used in the construction of information systems must meet or exceed specified standards of quality.
Accountability systems demand that schools establish and maintain data bases that can be manipulated in response to a variety of inquiries. The most extensive system includes different security levels and permits inquiries on a school, classroom, or student basis. Data systems, however simple or complex, require administrative time and attention to accuracy. When the data are meaningful to those reporting them, use of the data is more likely to impact the quality of reporting. As data are used in decision-making at the school level, attention to accuracy should increase. Users of the data should not forget that while standardized collections offer uniformity and consistency, the unique aspects of a school or program may be sacrificed to standardization. There are several ways of enhancing the validity, credibility, and positive impact of assessments used for accountability purposes while minimizing their negative effects. Linn (2000) recommends the following five actions: • Provide safeguards against selective exclusion of students. One way of doing this is to include all students in accountability calculations. • Make the case that high-stakes accountability requires new high-quality assessments each year that are equated to those of previous years. Failure to do this can result in distorted results (for example, inflated,
16
Education policy series
1
non-generalizable gains) and distortions in education (for example, narrowly teaching to the test). • Place more emphasis on comparisons of performance from year to year than from school to school. This allows for differences in starting points while maintaining expectations of improvement for all. • Consider both value added and status measures in the system. A value added measure provides schools that start out far from the goal a reasonable change to show improvement. In contrast, a status measure guards against “institutionalizing” low expectations for these same students and schools. • Recognize, evaluate, and report the degree of uncertainty in the reported results. Assessments do not yield perfect data. Rather, all data are flawed in some way. The amount of error in the data as well as in the decisions made based on the data should be recognized, reported, and evaluated. In addition, the use of multiple assessments (rather than a single assessment) enables educators to better understand and take into consideration the nature and magnitude of the error.
Accountability in education
17
9
Performance standards
There is a need to establish clear and explicit performance standards by which success will be determined.
Results-based accountability systems are based on student performance. There are three general ways in which student performance can be interpreted and reported: status of a cohort of students against a criterion; change in status of a cohort of students over time; and longitudinal change in the performance of individual students. Status against a criterion is the simplest to collect, report, and explain. Cohorts of students are used as the unit of analysis. The report might state that “68 percent of our students in grades three through five met the standard.” Extensions of this type of reporting include the percentage of students scoring at various performance levels or the achievement patterns of various subgroups. Reporting change in status of a cohort over time is based on the assumption that school performance should improve from one year to the next, regardless of the students who make up the cohort. This report might state that the “percentage of elementary students meeting the standard this year is twelve percent higher than last year.” Subgroup performance also can be reported. In the longitudinal change model, the student, not the cohort, is the unit of analysis. Individual students are followed from one year to the next and the stability or change in performance is reported. The report might say that “This year 34 percent of students scored at a higher level than they (the same students) scored last year.”
18
Education policy series
1
This approach provides greater measurement precision by tracking assessment data for individual students over time but requires more frequent administration of assessments. This booklet began with the premise that current resultsbased accountability systems are broadening the responsibility of educational systems for all students. At the same time, however, reliable and valid measures of the impact of schooling necessitate that students be enrolled in the school for an amount of time sufficient for the school to have an impact. Therefore, in practice, accountability systems have had to address several questions. • For what portion of the school year must students be enrolled for the school to be held accountable for their performance? • Are there groups of students that should not be included in the system (e.g., students with severe disabilities, non-native language speakers)? • Because the results are the basis for substantive organizational decisions and the results are available to the general public, should a minimum number of students in a group be required before the data are reported? Performance standards simultaneously must protect the individual student, support needed changes, and promote the aspirations of the society for its educational system.
Accountability in education
19
10
The generation of useful information
Accountability systems should provide data that enable educators to do their job better.
Results-based accountability systems should provide information that is understood and can be used by a variety of audiences. Systems fail when they yield only a single level of analysis and fall prey to the assumption that one report satisfies the needs of all audiences. Each audience should have sufficient information to carry out its particular responsibilities. For educators, the information must enable them to identify needed services and resources (in terms of both substance and quality) and evaluate the impact. Stewardship of resources such as time, teacher quality, and positive working relationships with parents and the community stimulates higher levels of student performance. School personnel generally focus their energies on those elements over which they can exercise control. For example, thorough analysis of student and teacher performance data can help educators identify the conditions they can alter to increase attendance. Parents, on the other hand, view schools differently and, in surveys, have suggested that they are interested in issues of school and student safety, teacher qualifications, and student performance indicators such as dropouts or graduation rates. Parents and the community may be less interested in reviewing student demographics than educators are in presenting them. Educators argue that the demographics enable parents and the community to understand the context in which the school performances should be in-
20
Education policy series
1
terpreted. Parents and the community often lament that schools use the demographics as excuses for low performance. Educators tend to benefit when the results-based accountability reports are accompanied by substantiating technical information. As schools seek to improve, reports should provide a sufficiently high level of detail so that their accuracy and validity can be maintained. At various organizational levels, expanded assessment reports (for example, information about curricular strands and objectives, performance of subgroups of students on specific objectives) are essential to plan for program changes. Although using indicator data has the potential to increase understanding, a balance must be achieved. Placing too great an emphasis on one factor can distort perceptions and lead to questionable decisions. For example, high levels of teacher attendance are desirable, but not at the cost of denying teachers opportunities to participate in meaningful professional development. Missing two days of face-to-face teaching to learn an effective instructional strategy could lead to higher results than perfect teacher attendance. Some systems employ a data warehouse with varying access to levels of analysis. Parents may have access to information about their individual child but are precluded from data on other children or teacher performance. Decisions about warehousing data should consider retrieval strategies and security.
Accountability in education
21
11
Parent and community involvement
Accountability systems should provide data that increase parent involvement and community support as well as inform public policy and the allocation of resources.
Each audience should have sufficient information to carry out its responsibility within the overall accountability framework. Parents have a responsibility to make decisions for their children, to advocate for their children, and to support positive changes in the educational system (both locally and elsewhere). Useful reports for parents are those that help them understand what is in the best interest of their child(ren). Consequently, these reports should be private, but allow for interpretation of the child’s progress against explicit standards, against grade level expectations, and in comparison to peers. School reports should be provided in formats that are easy-to-read and at reading levels appropriate to the general population. Graphic representations should be used and ancillary materials provided to parents who wish to go beyond the published summary. Parents are expected to use the information to encourage and motivate their children and as a basis for interacting with school personnel. Ultimately parents also are expected to portray the school factually to the community and to advocate citizen responsibility for creating a culture of high expectations and performance. Policymakers range from local officials through members of state and national governing bodies. What do those
22
Education policy series
1
who make the rules want to know? They require information to help them understand what progress is being made, to inform their resource allocation decisions, and to enable them to ensure that the system meets not only the present requirements and needs but those of the future. Public polices provide the framework for the actions of those who work in and benefit from results-based accountability systems. Those policies serve as both the foundation and the subject of the system. This booklet began with the premise that accountability systems embody the values and aspirations of a society. Societies communicate their values through their policies and practices. Policymakers need to know that the measures used to assess student performance and evaluate school performance provide valid descriptions of the quality of education. They need to understand the meaning of the assessment results. They also need to know the populations of students to whom the results do (and do not) apply. Because no society has unlimited resources, those charged with policy development must examine the available information to determine how resource allocations promote or hinder achievement of the primary goals. Patterns of allocation and usage that slow progress must be redirected. Finally, as advocates for the future, policymakers need information to ensure that the system continues to improve. As substantive and technical challenges arise, accountability systems can be modified to focus more intently on desired learning, to assess that learning more accurately and precisely, and to communicate assessment results in proper forms to a variety of audiences. Sound and defensible policies provide for these changes, enabling growth over time.
Accountability in education
23
References and sources of information
American Association for the Advancement of Science (2003). Resources for Science Literacy: Curriculum Materials Evaluation. Cary, NC: Oxford University Press. American Educational Research Association. (2000). Standards for educational and psychological testing 1999. Washington, DC: Author. Baker, E., Linn, R., Herman, J., & Koretz, D. (2002). Standards for educational accountability systems. Los Angeles: Center for Research on Evaluation, Standards and Student Testing. Blank, R. K. (1993). Developing a system of education indicators: Selecting, implementing, and reporting indicators. Educational Evaluation and Policy Analysis, 15, 65-80. Brown, R. S. (1999). Creating school accountability reports. The School Administrator, 56 (10), 12-17. Cebulka, J. G. & Derlin, R. L. (1995). State educational performance reporting policies in the US: Accountability’s many faces. International Journal of Educational Research, 23, 479-492. Committee for Economic Development (2001). Measuring what matters: Using assessment and accountability to improve student learning: A statement by the Research and Policy Committee of the Committee for Economic Development. New York: Author. Curran, B. (1999). State accountability issues, challenges, and strategies. Washington, D. C.: National Governors Association.
24
Education policy series
1
Elmore, R., Abelman, C. H. & Fuhrman, S. H. (1996). The new accountability in state education reform: From process to performance in holding schools accountable. Washington, D.C.: The Brookings Institution. Finn, C. E. (2002). Real accountability in K-12 education: The marriage of Ted and Alice. In M. E. Williamson and H. J. Walberg (Eds.), School accountability. Stanford, California: Hoover Institution Press. Fuhrman, S. H. (2002). The new accountability. University Park, PA: Center for Policy Research in Education. Gong, B., Blank, R. K., & Manse, J. G. (2002). Designing school accountability systems: Towards a framework and process. Washington, D. C.: Council of Chief State School Officers. Hanushek, E. A. & Raymond, M. E. (2002). Lessons and limits of state accountability systems. Paper presented at the Taking Account of Accountability: Assessing Policy and Politics conference. Cambridge, Massachusetts: Harvard University. Holloway, J. H. (2003). A global perspective on student accountability. Educational Leadership, 60 (5), 74-76. Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards: How to assess evaluations of educational programs. Thousand Oaks, CA: Sage. Linn, R. L. (2000). Assessments and accountability. Educational Researcher, 29 (2), 4-16. MacPherson, R. J. S. (1995). Struggling for the soul of education: Towards accountability policy research. International Journal of Educational Research, 23. 561566. Meyer, D. & Ralph, J. ( 2001). Key indicators of school quality. Paper presented at Key Indicators of Children’s Well-being conference. Bethesda, Maryland.
Accountability in education
25
Merrow, J. (2001). Undermining standards. Phi Delta Kappan, 82, 652-659. National Council of Teachers in Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author. Porter, A. C. (1991). Creating a system of school process indicators. Educational Evaluation and Policy Analysis, 13, 13-29. Southern Regional Education Board. (1997). Accountability in the 1990s: Holding schools responsible for student achievement. Atlanta, Georgia: Author. State Education Improvement Partnership. (1996). Measuring results: Overview of performance indicators. Washington, D. C.: Council of Chief State School Officers. Tucker, M. S. & Clark, C. S. (1999). The new accountability. The American School Boards Journal, 186, 26-29.
26
Education policy series
1
Jo Anne Anderson
1
p
catio Edu n
o
l
International Academy of Education International Institute for Educational Planning
ic
y
ries se
The International Academy of Education
The International Academy of Education (IAE) is a not-for-profit scientific association that promotes educational research, its dissemination, and the implementation of its implications. Founded in 1986, the Academy is dedicated to strengthening the contributions of research, solving critical educational problems throughout the world, and providing better communication among policy makers, researchers, and practitioners. The seat of the Academy is at the Royal Academy of Science, Literature and Arts in Brussels, Belgium, and its co-ordinating centre is at Curtin University of Technology in Perth, Australia. The general aim of the Academy is to foster scholarly excellence in all fields of education. Towards this end, the Academy provides timely syntheses of research-based evidence of international importance. The Academy also provides critiques of research, its evidentiary basis, and its application to policy. The members of the Board of Directors of the Academy are: • Erik De Corte, University of Leuven, Belgium (President); • Barry Fraser, Curtin University of Technology, Australia (Executive Director); • Monique Boekaerts, Leiden University, The Netherlands; • Jere Brophy, Michigan State University, USA; • Eric Hanushek, Hoover Institute, Stanford, USA; • Denis Phillips, Stanford University, USA; • Sylvia Schmelkes, Departamento de Investigaciones Educativas, Mexico. The following individuals, all Fellows of the Academy, are the members of the Editorial Committee for the Education Policy Booklet Series.
• • • •
•
Lorin Anderson, University of South Carolina, USA Eric Hanushek, Hoover Institute, Stanford, USA T. Neville Postlethwaite, University of Hamburg, Germany Kenneth N. Ross, International Institute for Educational Planning, Paris, France The Director of the International Institute for Educational Planning, Paris, France (Ex-officio). http://www.smec.curtin.edu.au/iae/
The International Institute for Educational Planning
The International Institute for Educational Planning (IIEP) was established in Paris in 1963 by UNESCO, with initial financial help from the World Bank and the Ford Foundation. The French Government provided resources for the IIEP’s building and equipment. In recent years the IIEP has been supported by UNESCO and a wide range of governments and agencies. The IIEP is an integral part of UNESCO and undertakes research and training activities that address the main priorities within UNESCO’s overall education programme. It enjoys intellectual and administrative autonomy, and operates according to its own special statutes. The IIEP has its own Governing Board, which decides the general orientation of the Institute’s activities and approves its annual budget. The IIEP’s mission is capacity building in educational planning and management. To this end, the IIEP uses several strategies: training of educational planners and administrators; providing support to national training and research institutions; encouraging a favourable and supportive environment for educational change; and co-operating with countries in the design of their own educational policies and plans. The Paris headquarters of the IIEP is headed by a Director, who is assisted by around 100 professional and supporting staff. However, this is only the nucleus of the Institute. Over the years, the IIEP has developed successful partnerships with regional and international networks of individuals and institutions – both in developed and developing countries. These networks support the Institute in its different training activities, and also provide opportunities for extending the reach of its research programmes. http://www.unesco.org /iiep/
E d u c a ti o n p ol
1
I S B N : 9 2 - 8 0 3 -1 2 7 6 - 6
ic
y
se
ri e s
Preface Education policy booklet series
The International Academy of Education and the International Institute for Educational Planning are jointly publishing the Education Policy Booklet Series. The purpose of the series is to summarize what is known, based on research, about selected policy issues in the field of education. The series was designed for rapid consultation “on the run” by busy senior decision-makers in Ministries of Education. These people rarely have time to read lengthy research reports, to attend conferences and seminars, or to become engaged in extended scholarly debates with educational policy research specialists. The booklets have been (a) focused on policy topics that the Academy considers to be of high priority across many Ministries of Education – in both developed and developing countries, (b) structured for clarity – containing an introductory overview, a research-based discussion of around ten key issues considered to be critical to the topic of the booklet, and references that provide supporting evidence and further reading related to the discussion of issues, (c) restricted in length – requiring around 30-45 minutes of reading time; and (d) sized to fit easily into a jacket pocket – providing opportunities for readily accessible consultation inside or outside the office. The authors of the series were selected by the International Academy of Education because of their expertise concerning the booklet topics, and also because of their recognized ability to communicate complex research findings in a manner that can be readily understood and used for policy purposes. The booklets will appear first in English, and shortly afterwards in French and Spanish. Plans are being made for translations into other languages. Four booklets will be published each year and made freely available for download from the web site of the International Institute for Educational Planning. A limited printed edition will also be prepared shortly after electronic publication.
Accountability in education
I
This booklet
As the economies of nations compete for strong positions within a competitive global market place, many governments have become increasingly interested in the performance of all aspects of their education systems. This trend, coupled with the enormous expenditures that are devoted to education, has also precipitated widespread public requests for higher levels of scrutiny concerning the quality of education. These demands for information about school system performance can only be addressed through the implementation of systematic accountability systems. Historically, the education profession has conformed to the requirements of regulatory or compliance accountability systems (usually based on government statutes), and has also subscribed to professional norms established by associations of educators. However, at the beginning of the 21st Century, accountability systems have also been required to respond to demands that professional performance be judged by the results that have been achieved. This booklet offers a set of principles and strategies to be considered in the development and implementation of results-based accountability systems. Technical and political issues are addressed as well as the ways in which educators, policymakers, and community members can use the information from accountability systems to improve results. The statements presented here about accountability systems are likely to be generally applicable throughout the world. Even so, they should be assessed with reference to local conditions, and adapted accordingly. In any educational setting or cultural context, suggestions or guidelines for practice require sensitive and sensible application, and continuing evaluation.
II
Education policy series
1
Jo Anne Anderson
has been active in the development and evaluation of educational policy for over twenty years, serving directly or in an advisory capacity to state, regional, and national organizations. She currently holds the position of Executive Director of the Education Oversight Committee for the state of South Carolina in the United States. This agency is responsible for the creation and implementation of the State Accountability System.
Accountability in education
III
This publication has been produced by the International Academy of Education (IAE) and the International Institute for Educational Planning (IIEP). It may be freely reproduced and translated into other languages. Please send a copy of any publication that reproduces this text in whole or part to the IAE and the IIEP. This publication is available on Internet in its printed form, see: http://www.unesco.org/iiep The author is responsible for the choice and presentation of the facts contained in this publication and for the opinions expressed therein which are not necessary those of IIEP (UNESCO) and do not commit the Organization. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of IIEP (UNESCO) concerning the legal status of any country, territory, city or area, or of its authorities, or concerning the delimitation of its frontiers or boundaries.
Jointly published by: The International Institute for Educational Planning (IIEP) 7-9 rue Eugène Delacroix 75116 Paris France and: The International Academy of Education (IAE) Palais des Académies 1, rue Ducale 1000 Brussels Belgium Design and layout by: Sabine Lebeau Printed by: Stedi Média © UNESCO 2005
ISBN: 92-803-1276-6
IV
Education policy series
1
Table of contents
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.
Types of accountability system Values and aspirations The goals of schooling The main components of accountability systems
1 3 5 7
Aligning the components of an accountability system 10 The use of student assessment data Information about the context of accountability systems The need for high quality information Performance standards The generation of useful information Parent and community involvement 12 14 16 18 20 22 24
References
Accountability in education
V
1
Types of accountability system
There are three main types of accountability systems that are sometimes applied simultaneously in education systems.
In the field of education there are three main types of accountability system: (a) compliance with regulations, (b) adherence to professional norms, and (c) results driven. School accountability systems operate according to a set of principles and use a variety of implementation strategies. In this booklet, these principles and strategies are described, with particular attention given to the political and technical aspects of accountability. Accountability systems are not new. The differences between current systems and those employed previously are matters of “for what” and “to whom.” Educators have worked mostly within three accountability systems, often simultaneously. The first system demands compliance with statutes and regulations such as those embodied in the British Office for Standards in Education. Anchored in an industrial model of education, compliance systems view the school as the embodiment of constant processes and allow for variation in results, generally attributed to the varying characteristics of students. Simply stated, educators were accountable for adherence to rules and accountable to the bureaucracy. The second system is based upon adherence to professional norms. Although neither mandated nor required, the impact of widespread agreement on certain principles and practices has done much to elevate education as a profession. In the United States, the curriculum and evaluation
Accountability in education
1
standards for school mathematics (National Council of Teachers of Mathematics, 1989), the standards for educational and psychological testing (American Educational Research Association, 2000), and the program evaluation standards (Joint Committee on Standards for Educational Evaluation, 1994) exemplify the professional norm approach to accountability. Within this system, educators are accountable for adherence to standards and accountable to their peers. The third accountability system is based upon results, with results defined in terms of student learning. This system has emerged from increasing political involvement in education. The “No Child Left Behind” requirements in the United States and the Australian National Education Performance Monitoring Task Force are examples of results-based systems. In these systems educators are accountable for student learning and accountable to the general public. Educators often find themselves responding to all three systems, attempting to balance the requirements of each. Professional norms complement both compliance and results systems. On the other hand, compliance and results systems often conflict. Part of this conflict stems from the fact that the emergence of results systems has been fostered by dissatisfaction with historic results; that is, those achieved under compliance systems. At present, accountability systems focus less on compliance and more on results. What are the components of a workable, defensible accountability system that is based primarily on results, while at the same time being attentive to professional norms and regulatory compliance requirements. First, the system defines educators’ responsibility for all students, regardless of the advantages or disadvantages they bring to school. Second, the system must be built upon aligned components—objectives, assessments, instruction, resources, and rewards or sanctions. Third, the technical aspects of the system must meet high standards. Fourth, the system must provide the vehicle for positive change.
2
Education policy series
1
2
Values and aspirations
Accountability systems embody prevailing societal values and aspirations.
The relationship between the educational attainment of citizens and the quality of their life has grown from a point of research interest to a call for action. In the second half of the 20th century governments in a number of Western nations experienced (a) low relative performance of their students on academic assessments when compared with students from certain Asian nations; and (b) a loss of historic industries (and jobs) to these nations. Within the United States, the insistence on comprehensive accountability systems was intensified by two events: widespread publication in the popular press of results from the 1995 Third International Math and Science Study (TIMSS) and the 1996 National Governors Association Education Summit. The TIMSS results suggested that United States students in Grade 3 were slightly behind their peers in other developed countries and, importantly from a policy perspective, this difference increased the longer they remained in school. At the Summit the governors from almost every state committed to introduce strong accountability measures to ensure that public schools performed at the level necessary for economic supremacy. Within two years, United States educators were grappling with the change imposed by the shift in accountability systems from those based on compliance and professional norms to one based on results. United States educators are not alone. Reviews of accountability programs throughout the world provide evidence
Accountability in education
3
that accountability is an international issue. England has a national curriculum accompanied by assessments and measures for rating schools. France, Hong Kong, China, Japan, and others use national assessments to measure student and school progress and to make decisions about each. Many European systems use examinations to determine student access to the next level of education. All these systems are based on explicit definitions of what students are expected to learn and to what level they are expected to perform. Furthermore, examinations are used to monitor student learning, with the data providing the basis for changes within the system. Educational opportunity, an extension of civil rights and economic inclusion, has been redefined: concerns for equal access and treatment have been replaced with an emphasis on equal attainment. To have equal attainment, however, variations in access and, particularly, treatment must be available to meet the needs of increasingly diverse populations of students. This focus on equal attainment has led us back to the age-old question, “What’s worth learning?” That is, what should we expect students to attain as a result of the formal education they receive? The answer to this question depends primarily on societal values. The population of students to whom this question applies depends to a great extent on the aspirations societies have for their citizens.
4
Education policy series
1
3
The goals of schooling
Accountability systems are based on the expectation that students can and will achieve the goals of schooling.
Traditionally, schools have been expected to teach students. However, there has been general acceptance that only those students who bring advantages to the school are likely to benefit from the exposure to this teaching. Minority students, economically disadvantaged students, disabled students, and other groups simply have not been expected to learn at the level of their advantaged peers. Current research findings counter the premise that some students cannot benefit from schooling. Almost a quarter century ago, Ron Edmonds’ (1979) work on effective schools identified principles that should underlie school practices. Subsequently, teaching practices have been identified and instructional models developed that promote high levels of learning for large numbers of students, regardless of the disadvantages they bring into the classroom. Intense study of Asian school systems suggests that the combination of national aspiration, cultural support, and individual effort overcomes both real and perceived barriers. Assuming responsibility for the learning of all students transforms the school and the classroom environment and, to some degree, the way that teachers view their profession and themselves. The popular literature is replete with heroic educators who, despite overwhelming odds, are able to change and improve a school through their zeal. A challenge of accountability systems is to make the heroic, customary. In results-based systems, students’
Accountability in education
5
learning failures are attributed to weaknesses in educational programs and practices rather than to students’ characteristics and backgrounds. Schools that are accomplishing the goal of all students achieving success are most likely to have strong and stable teachers and administrators. Strength comes from factors such as greater content knowledge and visionary instructional leadership. Stability, in terms of commitment to the school over time, is needed to shape the school culture and climate. Stability enables the development of relationships with parents and the community that are anchored in mutual trust and focused upon students’ present and future needs. Why then, is there scepticism about goals based upon all students learning? Educators may find themselves overwhelmed by the disadvantages that students bring to the learning environment over which they have no control. Educators also have little control over the resources available to them to achieve the goals. Administrators must build consensus around the goals and cultivate a professional dialogue that encourages the definition of solvable problems. This dialogue must be extended to the broader community so that the disadvantages students bring to the school can be ameliorated over time. Workable, defensible accountability systems are built upon aligned components—objectives, assessments, instruction, resources, and rewards or sanctions.
6
Education policy series
1
4
The main components of accountability systems
Accountability systems should include five components: objectives, assessments, instructions, resources, and rewards or sanctions.
Analyses of current results-based accountability systems reveal agreement on five guiding questions: What do we expect students to know and be able to do? How satisfied are we that students have mastered the established content standards? How are teachers prepared to be effective in their classrooms with all students? How and to what degree is the public informed about school results and the contributors to those results? How does society respond to the information they receive about the performance of schools? Content standards have shifted from the trivium of ancient Greece to today’s workforce preparation. Throughout the world, education systems emphasize literacy, mathematical reasoning, scientific inquiry, and historical and social understanding to support civic participation. Within developing nations, literacy is the most often defined learning expectation. Within developed nations, the emphasis is on increasing mathematical and scientific competence. In general, curricula mirror the economic focus of nations. The establishment of content standards impacts on the nature and structure of the curriculum. Teachers must exhibit an understanding of the structure of the curriculum both horizontally (within levels) and vertically (across levels). Access to a variety of learning resources (including supplementary materials) and extended or enriched in-
Accountability in education
7
formal learning opportunities are important. In practical terms, the introduction of content standards has proceeded at a much faster pace than have the learning resources and supportive opportunities that must be aligned with the standards if the intended learning is to occur. The use of assessments to inform decisions about students, schools, and personnel has been accelerated by the rise of results-based accountability systems. Aligned with the content standards, assessments are used to make decisions about student eligibility for and progress to the next level of school; for administrator and teacher employment and rewards; and for resource allocation. When these assessments are used in this way, they are referred to as “high stake” assessments. These “high stakes” decisions generate demands that information from assessments can be used to improve the teaching-learning process. Because they are designed for administration to large numbers of students, however, accountability assessments generally do not offer sufficient diagnostic information for teacher planning and in-class work with individual students. Some assessment programs release items and/or parallel assessments so that teachers are comfortable with both the content to be tested and the manner in which each standard is assessed. Changes in expectations about students should lead to changes in instruction. The rapid change of the curriculum, particularly in mathematics and science, has left many teachers responsible for teaching content they may not have learned in a formal setting. Teachers also are expected to adapt their teaching for students from diverse backgrounds, exhibiting a range of motivations and prior experiences. Instead of a consistent methodology yielding differentiated results, teachers are expected to differentiate their methodologies to yield consistent results for diverse student populations. Results-based accountability systems utilize public reporting to a greater degree than do the compliance or professional norms systems. In the latter two systems information about student performance is held within
8
Education policy series
1
the profession. Results-based systems rely upon widespread communication of results to parents and the general public. Many results-based systems generate school report cards or school profiles for distribution to general audiences. These reports include summaries of the performance of students or subgroups of students as well as information about resources (for example, per student expenditures), programs (for example, participation in accelerated courses), and behaviour (for example, student attendance.) Providing this information to the public has required that teachers and administrators become comfortable discussing strengths and weaknesses, explaining a variety of statistical data, and facilitating positive change. This new communications role for educators can be intimidating as educators struggle both to understand underperformance and to inspire confidence that they can lead the change process needed to improve performance. Finally, in most results-based accountability systems performance is publicly acknowledged and rewards, sometimes financial, are provided to those schools or individuals exhibiting high and/or improving performance. Schools not succeeding are provided encouragement and often technical assistance. Technical assistance is most effective when the local school assumes ownership of the results-based change process. Schools needing to improve dramatically benefit from increased attention and resources. Yet these schools also may be overwhelmed by the infusion of new practices and greater expectations for simultaneous rapid and long-lasting change. In extreme circumstances another layer of educational governance may assume management of the school. The continuum from providing technical assistance to taking control often is ill defined. Technical assistance should provide immediate and temporary support whereas assumption of responsibility extends to governance and data management. All schools are most vulnerable when the public demands quick change, rather than exercising the patience to implement sustainable changes. Long-lasting change requires integration of remedies across community agencies and responsibilities.
Accountability in education
9
5
Aligning the components of an accountability system
Attention, scrutiny, and discipline should be exercised to ensure that the five components are aligned, with concerns for alignment evident from planning through implementation.
The foundation of results-based accountability systems is clear expectations for student learning, both what students are to learn and how that learning is to be demonstrated. Thus, content standards and the accompanying assessments are the components with which the other components, most importantly, instructional materials and teaching-learning strategies, must be aligned. When content standards, assessments, materials, and strategies are aligned, students have the maximum opportunity to learn. Also, when the public understands data derived from an “aligned” accountability system, they are more likely to respond to the performance of schools in a thoughtful and supportive way. Concerns for alignment are relatively new. Throughout much of the 20th century, textbooks formed the basis for instructional planning. Although the structure and content of textbooks changed in response to discipline-based organizations, the presumption was that textbooks incorporated all that was needed to facilitate the desired student learning. Thus, alignment was part and parcel of buying into the textbook “package”. As access to multi-media and a wider range of materials increased, reliance on a primary textbook for the design of an instructional plan began to fade. Currently, the specification of content standards presumes independence from a primary text and the use of diverse materials and teaching-learning strategies.
10
Education policy series
1
How are decisions of alignment reached? Policy and disciplinary organizations, government agencies, and local school districts typically employ a professional judgment methodology. In the United States, the Council of Chief State School Officers (State Education Improvement Partnership, 1996) and the American Association for the Advancement of Science (AAAS, 2003) are among the organizations that have developed structured processes for the review of content standards and assessments. These processes require agreement on the depth and breadth of the knowledge expected within a content standard or assessment, the degree of cognitive demand and evidence of discrete or integrated knowledge, the emphasis placed on the standard in instruction or assessment, and the ways in which student learning is reported. These methodologies are relatively new and there are not similarly consistent strategies for use by local administrators and teachers. As studies of alignment expand to address instructional validity, practitioner tools and skills should be developed to inform local decisions about instructional materials and the teaching-learning process within each school community.
Accountability in education
11
6
The use of student assessment data
Data from student assessments should be the primary source for identifying the problems to be solved.
Similar concerns for alignment are evident in the testing industry. Accountability systems emphasize student mastery of specified content and rely more on criterionreferenced assessments than on norm-referenced ones to determine how well students are learning. If these measures are misaligned with content standards, the information they yield is irrelevant to determining school effectiveness. Assessments in results-based accountability systems must be of sufficient technical quality to support the decisions that are based on the results. In the United States, recommended voluntary standards for the construction and use of accountability systems have been developed in a collaborative project between the National Center for Research on Evaluation, Standards and Student Testing (CRESST) and the Consortium for Policy Research in Education (CPRE) (Baker et al., 2002). When these standards are examined in the context of the Standards for Educational and Psychological Testing, some general principles for using assessments in accountability systems emerge: • Make explicit the purposes that the assessment system and individual assessments are intended to serve; • decide on a strategy to meet the testing requirements at various grade levels;
12
Education policy series
1
• determine the degree to which validity evidence is available or could be accumulated for multiple purposes and “the widest possible range of students;” • determine a standard of adequacy for technical quality; and • make plans to acquire needed technical quality information during piloting, field trials and implementation. Assessment systems can “lower the stakes” when educators and others have sufficient documentation that the assessments have met technical standards and there is clear understanding of how the assessment data are to be used. The stakes also are lowered when assessment data are used for positive purposes such as providing technical assistance to schools, initiating supplemental services to students, and amending policies and practices that interfere with goal attainment. When the stakes are consistently negative, the assessment data are viewed sceptically; when the stakes lead to improvements, assessment data can become accepted as an integral and necessary part of the decision-making process that leads to educational improvement.
Accountability in education
13
7
Information about the context of accountability systems
Supporting information about teacher quality, curriculum rigor, and resource allocation should provide the basis for selecting or designing strategies that are most likely to solve problems.
School quality is not only evident in assessment results, but also in the diversity of programs offered, the preparation and performance of educational professionals, student behaviour and attitudes, and the relationship between the school and the community. School reports should publish contextual and programmatic information along with assessment results. This additional information provides a more complete description of the school and enhances the public’s understanding of its overall performance. The information also offers a point of comparison among schools as patterns of inputs, processes, and outputs are related to levels of school performance. Inputs include fiscal and other resources, teacher quality, students’ backgrounds, and parent/community norms. Processes include the organization of schools, the curriculum and pedagogy, and opportunities for student participation in non-academic activities. Outputs include student achievement, participation, attitudes, and aspirations (Porter, 1991). Other potentially useful information includes attendance (both teacher and student), student behaviour (or misbehaviour), teacher professional development, and parents’ and students’ perceptions of the school. As school reports gain public attention, program advocates view publication of data as a way of ensuring much needed attention to their programs.
14
Education policy series
1
Collecting and reporting these data are mammoth tasks. Few countries have educational data systems with the flexibility to extract contextual information. Most reporting systems, therefore, rely upon supplementary self-reported data. As reliance on self-reporting increases, data on program characteristics are vulnerable to hurried collections, natural inclinations to present the factor positively, and inadvertent errors. At school sites, data collection is relegated to one of many tasks in a busy environment and often becomes secondary to more immediate concerns. Results-based accountability systems require both educators and the public to understand the meaning of data, the implications of the ways in which data are aggregated, and, of greater import, ways in which the data can be used to make improvements. For example, disaggregated student mathematics scores are interesting and may point to a gap in achievement, but only when those data are interpreted within our knowledge of the curriculum and instruction are we able to determine how best to improve student performance.
Accountability in education
15
8
The need for high quality information
All data collection instruments and procedures used in the construction of information systems must meet or exceed specified standards of quality.
Accountability systems demand that schools establish and maintain data bases that can be manipulated in response to a variety of inquiries. The most extensive system includes different security levels and permits inquiries on a school, classroom, or student basis. Data systems, however simple or complex, require administrative time and attention to accuracy. When the data are meaningful to those reporting them, use of the data is more likely to impact the quality of reporting. As data are used in decision-making at the school level, attention to accuracy should increase. Users of the data should not forget that while standardized collections offer uniformity and consistency, the unique aspects of a school or program may be sacrificed to standardization. There are several ways of enhancing the validity, credibility, and positive impact of assessments used for accountability purposes while minimizing their negative effects. Linn (2000) recommends the following five actions: • Provide safeguards against selective exclusion of students. One way of doing this is to include all students in accountability calculations. • Make the case that high-stakes accountability requires new high-quality assessments each year that are equated to those of previous years. Failure to do this can result in distorted results (for example, inflated,
16
Education policy series
1
non-generalizable gains) and distortions in education (for example, narrowly teaching to the test). • Place more emphasis on comparisons of performance from year to year than from school to school. This allows for differences in starting points while maintaining expectations of improvement for all. • Consider both value added and status measures in the system. A value added measure provides schools that start out far from the goal a reasonable change to show improvement. In contrast, a status measure guards against “institutionalizing” low expectations for these same students and schools. • Recognize, evaluate, and report the degree of uncertainty in the reported results. Assessments do not yield perfect data. Rather, all data are flawed in some way. The amount of error in the data as well as in the decisions made based on the data should be recognized, reported, and evaluated. In addition, the use of multiple assessments (rather than a single assessment) enables educators to better understand and take into consideration the nature and magnitude of the error.
Accountability in education
17
9
Performance standards
There is a need to establish clear and explicit performance standards by which success will be determined.
Results-based accountability systems are based on student performance. There are three general ways in which student performance can be interpreted and reported: status of a cohort of students against a criterion; change in status of a cohort of students over time; and longitudinal change in the performance of individual students. Status against a criterion is the simplest to collect, report, and explain. Cohorts of students are used as the unit of analysis. The report might state that “68 percent of our students in grades three through five met the standard.” Extensions of this type of reporting include the percentage of students scoring at various performance levels or the achievement patterns of various subgroups. Reporting change in status of a cohort over time is based on the assumption that school performance should improve from one year to the next, regardless of the students who make up the cohort. This report might state that the “percentage of elementary students meeting the standard this year is twelve percent higher than last year.” Subgroup performance also can be reported. In the longitudinal change model, the student, not the cohort, is the unit of analysis. Individual students are followed from one year to the next and the stability or change in performance is reported. The report might say that “This year 34 percent of students scored at a higher level than they (the same students) scored last year.”
18
Education policy series
1
This approach provides greater measurement precision by tracking assessment data for individual students over time but requires more frequent administration of assessments. This booklet began with the premise that current resultsbased accountability systems are broadening the responsibility of educational systems for all students. At the same time, however, reliable and valid measures of the impact of schooling necessitate that students be enrolled in the school for an amount of time sufficient for the school to have an impact. Therefore, in practice, accountability systems have had to address several questions. • For what portion of the school year must students be enrolled for the school to be held accountable for their performance? • Are there groups of students that should not be included in the system (e.g., students with severe disabilities, non-native language speakers)? • Because the results are the basis for substantive organizational decisions and the results are available to the general public, should a minimum number of students in a group be required before the data are reported? Performance standards simultaneously must protect the individual student, support needed changes, and promote the aspirations of the society for its educational system.
Accountability in education
19
10
The generation of useful information
Accountability systems should provide data that enable educators to do their job better.
Results-based accountability systems should provide information that is understood and can be used by a variety of audiences. Systems fail when they yield only a single level of analysis and fall prey to the assumption that one report satisfies the needs of all audiences. Each audience should have sufficient information to carry out its particular responsibilities. For educators, the information must enable them to identify needed services and resources (in terms of both substance and quality) and evaluate the impact. Stewardship of resources such as time, teacher quality, and positive working relationships with parents and the community stimulates higher levels of student performance. School personnel generally focus their energies on those elements over which they can exercise control. For example, thorough analysis of student and teacher performance data can help educators identify the conditions they can alter to increase attendance. Parents, on the other hand, view schools differently and, in surveys, have suggested that they are interested in issues of school and student safety, teacher qualifications, and student performance indicators such as dropouts or graduation rates. Parents and the community may be less interested in reviewing student demographics than educators are in presenting them. Educators argue that the demographics enable parents and the community to understand the context in which the school performances should be in-
20
Education policy series
1
terpreted. Parents and the community often lament that schools use the demographics as excuses for low performance. Educators tend to benefit when the results-based accountability reports are accompanied by substantiating technical information. As schools seek to improve, reports should provide a sufficiently high level of detail so that their accuracy and validity can be maintained. At various organizational levels, expanded assessment reports (for example, information about curricular strands and objectives, performance of subgroups of students on specific objectives) are essential to plan for program changes. Although using indicator data has the potential to increase understanding, a balance must be achieved. Placing too great an emphasis on one factor can distort perceptions and lead to questionable decisions. For example, high levels of teacher attendance are desirable, but not at the cost of denying teachers opportunities to participate in meaningful professional development. Missing two days of face-to-face teaching to learn an effective instructional strategy could lead to higher results than perfect teacher attendance. Some systems employ a data warehouse with varying access to levels of analysis. Parents may have access to information about their individual child but are precluded from data on other children or teacher performance. Decisions about warehousing data should consider retrieval strategies and security.
Accountability in education
21
11
Parent and community involvement
Accountability systems should provide data that increase parent involvement and community support as well as inform public policy and the allocation of resources.
Each audience should have sufficient information to carry out its responsibility within the overall accountability framework. Parents have a responsibility to make decisions for their children, to advocate for their children, and to support positive changes in the educational system (both locally and elsewhere). Useful reports for parents are those that help them understand what is in the best interest of their child(ren). Consequently, these reports should be private, but allow for interpretation of the child’s progress against explicit standards, against grade level expectations, and in comparison to peers. School reports should be provided in formats that are easy-to-read and at reading levels appropriate to the general population. Graphic representations should be used and ancillary materials provided to parents who wish to go beyond the published summary. Parents are expected to use the information to encourage and motivate their children and as a basis for interacting with school personnel. Ultimately parents also are expected to portray the school factually to the community and to advocate citizen responsibility for creating a culture of high expectations and performance. Policymakers range from local officials through members of state and national governing bodies. What do those
22
Education policy series
1
who make the rules want to know? They require information to help them understand what progress is being made, to inform their resource allocation decisions, and to enable them to ensure that the system meets not only the present requirements and needs but those of the future. Public polices provide the framework for the actions of those who work in and benefit from results-based accountability systems. Those policies serve as both the foundation and the subject of the system. This booklet began with the premise that accountability systems embody the values and aspirations of a society. Societies communicate their values through their policies and practices. Policymakers need to know that the measures used to assess student performance and evaluate school performance provide valid descriptions of the quality of education. They need to understand the meaning of the assessment results. They also need to know the populations of students to whom the results do (and do not) apply. Because no society has unlimited resources, those charged with policy development must examine the available information to determine how resource allocations promote or hinder achievement of the primary goals. Patterns of allocation and usage that slow progress must be redirected. Finally, as advocates for the future, policymakers need information to ensure that the system continues to improve. As substantive and technical challenges arise, accountability systems can be modified to focus more intently on desired learning, to assess that learning more accurately and precisely, and to communicate assessment results in proper forms to a variety of audiences. Sound and defensible policies provide for these changes, enabling growth over time.
Accountability in education
23
References and sources of information
American Association for the Advancement of Science (2003). Resources for Science Literacy: Curriculum Materials Evaluation. Cary, NC: Oxford University Press. American Educational Research Association. (2000). Standards for educational and psychological testing 1999. Washington, DC: Author. Baker, E., Linn, R., Herman, J., & Koretz, D. (2002). Standards for educational accountability systems. Los Angeles: Center for Research on Evaluation, Standards and Student Testing. Blank, R. K. (1993). Developing a system of education indicators: Selecting, implementing, and reporting indicators. Educational Evaluation and Policy Analysis, 15, 65-80. Brown, R. S. (1999). Creating school accountability reports. The School Administrator, 56 (10), 12-17. Cebulka, J. G. & Derlin, R. L. (1995). State educational performance reporting policies in the US: Accountability’s many faces. International Journal of Educational Research, 23, 479-492. Committee for Economic Development (2001). Measuring what matters: Using assessment and accountability to improve student learning: A statement by the Research and Policy Committee of the Committee for Economic Development. New York: Author. Curran, B. (1999). State accountability issues, challenges, and strategies. Washington, D. C.: National Governors Association.
24
Education policy series
1
Elmore, R., Abelman, C. H. & Fuhrman, S. H. (1996). The new accountability in state education reform: From process to performance in holding schools accountable. Washington, D.C.: The Brookings Institution. Finn, C. E. (2002). Real accountability in K-12 education: The marriage of Ted and Alice. In M. E. Williamson and H. J. Walberg (Eds.), School accountability. Stanford, California: Hoover Institution Press. Fuhrman, S. H. (2002). The new accountability. University Park, PA: Center for Policy Research in Education. Gong, B., Blank, R. K., & Manse, J. G. (2002). Designing school accountability systems: Towards a framework and process. Washington, D. C.: Council of Chief State School Officers. Hanushek, E. A. & Raymond, M. E. (2002). Lessons and limits of state accountability systems. Paper presented at the Taking Account of Accountability: Assessing Policy and Politics conference. Cambridge, Massachusetts: Harvard University. Holloway, J. H. (2003). A global perspective on student accountability. Educational Leadership, 60 (5), 74-76. Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards: How to assess evaluations of educational programs. Thousand Oaks, CA: Sage. Linn, R. L. (2000). Assessments and accountability. Educational Researcher, 29 (2), 4-16. MacPherson, R. J. S. (1995). Struggling for the soul of education: Towards accountability policy research. International Journal of Educational Research, 23. 561566. Meyer, D. & Ralph, J. ( 2001). Key indicators of school quality. Paper presented at Key Indicators of Children’s Well-being conference. Bethesda, Maryland.
Accountability in education
25
Merrow, J. (2001). Undermining standards. Phi Delta Kappan, 82, 652-659. National Council of Teachers in Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author. Porter, A. C. (1991). Creating a system of school process indicators. Educational Evaluation and Policy Analysis, 13, 13-29. Southern Regional Education Board. (1997). Accountability in the 1990s: Holding schools responsible for student achievement. Atlanta, Georgia: Author. State Education Improvement Partnership. (1996). Measuring results: Overview of performance indicators. Washington, D. C.: Council of Chief State School Officers. Tucker, M. S. & Clark, C. S. (1999). The new accountability. The American School Boards Journal, 186, 26-29.
26
Education policy series
1
