- Peer Review: Online Evaluation Resource Library: Teacher Education
Online Evaluation Resource Library: Teacher Education
- Jan 15, 2004 by Teacher Education
The site includes specific examples of project evaluation components for teacher education and other projects that should substantially increase the quantity and quality of teachers in science and mathematics. Created primarily as a resource for potential project developers and evaluators in the planning and implementing of program evaluations, the site contains a library of plans and reports of projects funded by NSF along with data collection instruments used in those projects. Additionally OERL contains a set of professional development modules on designing an evaluation and designing written questionnaires. The professional development modules have broad appeal to anyone interested in learning about program evaluation so this review will focus on that component of OERL.
- Type of Material:
The modules present step-by-step strategies with scenarios and case studies of how the strategies can be applied to specific evaluation projects. Some modules present exercises in which the user can apply discrete technical skills that are part of the strategy.
- Recommended Uses:
Faculty who write and submit grants to improve teacher education through curriculum development, faculty development, laboratory improvement, targeting under-represented populations, or integration of technology into instruction should use this resource to improve the project evaluation components. As project evaluation becomes an increasingly important component of graduate teacher education programs, the professional development modules on Evaluation Design (Methodological Approach and Sampling) and on Development of Written Questionnaires would also be appropriate for use in graduate teacher education courses. The material is appropriate for either independent study or classroom exercise. Students could work through the modules prior to developing an evaluation plan for a program of their choosing. The use of OERL as a support would be most motivational if the program were at their school. The case studies in the modules could be done as group projects in class and discussed, or done outside of class and then discussed.
- Technical Requirements:
Network connection and browser.
- Identify Major Learning Goals:
The goal of OERL is to improve evaluation practice using Web site resources. The professional development modules can be used in graduate teacher education courses. The student should learn how to design an evaluation, including the methodology to employ and the appropriate sampling techniques to use. Additionally the student should recognize for what kinds of evaluation questions the use of written questionnaires would be appropriate as a data collection instrument. Furthermore, the student should learn how to construct and administer such questionnaires.
- Target Student Population:
This library was developed for professionals seeking to design, conduct, document, or review project evaluations. But graduate students in education or school counseling should also be taught to use the resource.
- Prerequisite Knowledge or Skills:
As the modules state, some knowledge of statistics would be helpful but not
The content of program evaluation is an important topic that has traditionally been given
insufficient emphasis in educational research courses. If the typical masters-level student in education gets involved in educational research it is most likely to be program evaluation so it seems reasonable to emphasize it in basic educational research courses. The OERL site contains high quality materials with excellent reference material that can be used to introduce important issues in program evaluation to such a population. The sample evaluation materials were selected to represent sound evaluation practice as delineated by Program Evaluation Standards: 2nd Edition (1994) and other evaluation sources. Evaluation resources in the online library can be used as is, as models for the structure and content of new plans, instruments, or reports, or as adaptations for the evaluation of a similar project. The diversity of high quality examples makes it likely that an existing analysis plan or instrument may be adaptable for a new project.
Quality criteria are
Comparison tables that identify the content of instruments in specific categories (for example, Course Evaluation Instruments that have been used to evaluate Teacher Education Projects) make it possible to compare how different projects among the examples measure similar variables such as Student Demographics, Attitudes & Beliefs, or Student/Teacher interactions (to name a few of the many variables measured). User scenarios describe specific approaches to using OERL's evaluation resources. They take the user step-by-step through several scenarios that describe how evaluators might use OERL to develop plans, instruments, or reports.
The references are outstanding, with
many linked to more detailed material online.
Potential Effectiveness as a Teaching Tool
Graduate students in teacher education courses would benefit greatly from knowing the quality criteria for sound project evaluation, organized here into three sections reported to be the three principal characteristics of instruments: (1) Design, (2) Technical Quality, and (3) Utility. A criteria overview details specific components of these quality criteria, with key terms like validity, reliability, and stakeholders clearly defined. The alignment table overview for sound project evaluation instruments with Glossary, Quality Criteria, and Professional Standards & References to Best Practice provides quick access with logical organization to high quality examples. There are also alignment tables for plans and reports.
The professional development modules are outstanding. The modules are navigable to from the home page, which is always one click away if you click the OERL lighthouse logo, which appears in the upper left corner of every page on the site. Each module has the following structure: Key Topics, Strategy, Scenario, Case Study, and References. In the Key Topics section the main ideas of the module are introduced and learning objectives are stated. In the Strategy section the professional activity, such as planning an evaluation or designing a questionnaire, is presented as a sequence of steps. For each step, one or more examples from an NSF program evaluation is given to illustrate the process. Following this section the Scenario section takes the user through the entire professional activity step-by-step for one complete evaluation problem. Next, the Case Study section asks the user to apply the entire procedure to a new problem. These can be used in whole class format with discussions or as tutorials for independent study. As the instructional research on worked examples has demonstrated, such an instructional procedure (a worked example followed by an example to be worked) is much more effective than simply asking the user to work a problemafter an exposition of the procedure.
Ease of Use for Both Students and Faculty
OERL is a professionally designed website with an integrated navigational system. The navigation bar is explained on the index page and appears consistently across the website. For a very complex and information-dense website, it is fairly easy to find material and to navigate about it. Items are cross-referenced between resource type and project type, making it easy to find all sorts of useful examples of evaluation material. The three ways to search the evaluation resources are a strength of the project. A keyword search finds keywords of interest across the collection. An evaluation resources search allows the user to search for resources within Plans, Instruments, Reports, or Contributing Projects, and a component resources search allows the user to search for resources across plans and reports. The glossary and quality criteria can also be viewed separately. Evaluation instruments for projects are logically organized by respondent (Teacher/Faculty, Student, Observation, Case Study Protocol).
As described above, the professional development modules all have the same structure, which aids in working through them.
It may be hard to find the professional development modules. The modules are navigable to from the home page - the OERL lighthouse logo in the upper left corner of every page.
- Other Issues and Comments:
ALL faculty who write grant proposals or who teach students who may some day write grant proposals should review and use these project evaluation resources!
While all the materials on this site are of high quality, the professional development modules are an especially useful resource for teacher educators. They should be accessible from every page on the site. The authors plan to make that change this year (2 004).
- Creative Commons: