STANDARD 2 - ASSESSMENT SYSTEM AND UNIT EVALUATION

         Element 1: Assessment System

At the time Purdue University North Central (PNC) became an autonomous program in 2003 the unit assessment system was aligned with the Purdue West Lafayette conceptual framework.  As the PNC Education Department’s conceptual framework was developed, the prior assessment system was evaluated and scrutinized to ensure that it reflected our conceptual framework.  The portfolio, field experience and student teaching evaluation forms were changed immediately to reflect the PNC conceptual framework. Other changes such as curricular changes and the contents of the portfolio have been made after a careful analysis of the data has been collected as result of this system.  The unit assessment system is in place and has been fully implemented.

         The PNC education faculty developed the unit assessment system with involvement from its school-based and university partners.  Initially, the partners including teachers, principals and PNC faculty were involved in ongoing workshops to discuss the conceptual framework and the assessment system.  In 2004, the University Council on Teacher Education (UCTE) was formed.  Table 2.1 describes the membership of this group.  Its function is to serve as an advisory group that will oversee the education program’s effectiveness.  In November, 2005 the UCTE reviewed the data that had been collected from the assessment system including data related to the gates, field experiences, student teaching and surveys of graduates and employers. The UCTE was asked for input concerning changes they felt should be made to the Education program based on the data.  The overall response was positive. It was suggested that the gate system could be revised to ensure that the best predictors of candidate success were being assessed and that an evaluation tool should be created to help the members of UCTE assess the data.  (Exhibit 2.1.1: Minutes from UCTE meeting 11/05 TEAM Meetings and Candidate Council)

Table 2.1 shows how the UCTE as well as other stakeholders are involved in multiple formats with various responsibilities:

Table 2.1: Stakeholder Involvement

The PNC conceptual framework states that PNC candidates will demonstrate that they are effective educators who are committed to continuous growth and school reform by meeting the ten PNC outcomes and the associated knowledge, performance and disposition indicators.  The PNC outcomes were created by the faculty and have been aligned with the INTASC Principles and the Indiana Content and Developmental Standards for Early and Middle Childhood. (Exhibit 2.1.2: Table of Alignment of Outcomes and Standards)  The PNC outcomes were described in the Conceptual Framework section of this report and a complete description may be found in the exhibit room.  (Exhibit 2.1.3: PNC Outcomes)  To ensure PNC candidates demonstrate their effectiveness at determined points during their candidacy, evaluation forms, rubrics, and key assignments are aligned with the PNC outcomes and reflect the PNC conceptual framework.  (Exhibit 2.1.4: Field Experience and Student Teaching Evaluation forms)

Key Assignments

Developmental key assignments (Table 2.2) based on these outcomes have been designed to ensure that candidates are provided opportunities to demonstrate their competency in meeting the outcomes throughout the elementary education program.  The key assignments are course assignments that are the required artifacts for the PNC Professional Portfolio.  They were developed by the faculty and aligned with the PNC outcomes to ensure that all candidates have the opportunity to meet all of the outcomes.  The faculty member who is teaching the course uses a rubric that is aligned with the outcomes to assess the key assignment.  Although some key assignments have the same title, they are designed to build upon one another developmentally and provide appropriate background knowledge for the next assignment.  Each assignment has a different purpose associated with it and a different rubric based on the outcomes is used to evaluate it.  Since the key assignments are the required portfolio artifacts, candidates must complete the assignments at an appropriate level of performance in order for the portfolio to be acceptable.  Dispositions related to the PNC outcomes are evaluated during courses, and are included on the field experience and student teaching evaluation forms.

Table 2.2: Key Assignments

At Purdue University North Central, candidates are assessed at key points during their program using multiple assessment tools.  Data is gathered from both internal and external sources to assure that candidates are maintaining a level of knowledge, performance and disposition during their candidacy.  These assessments are also used to manage and improve the education program and the operations of the unit.  Table 2.3 provides the sources of internal and external data that are used.  

Table 2.3: Internal and External Sources of Data

A Gate System is in place that allows for the assessment of candidates at various transition points which include before admission to the program, during the program, as they graduate, as they are recommended for a teaching license, and after they are employed (Table 2.4).  Multiple measures are used to ensure a system of continuous improvement and candidate development.  The system is also designed to provide feedback for program review and revision.  Disposition and technological competence are assessed at the gates with the portfolio and during field experiences and student teaching.  

Table 2.4: Gate System

In the fall 2005, data that was collected from the gate assessment was reviewed by the Education faculty.  It was determined that the best predictors of candidate success were GPA and Praxis.  These predictors differentiated among candidates and the data showed that those candidates who progressed easily through the gates had usually passed the Praxis I and II exams on the first try and also had acceptable or above GPA’s.  The collected data also suggested that the present portfolio assessment procedure did not provide substantial evidence of candidate knowledge, performance and disposition.  It also found the signature form was in fact not a gatekeeper, but a program requirement.  It was further determined that the data collected from the gates did not provide enough information concerning how well candidates were meeting specific outcomes including performance and disposition.  There did not seem to be adequate evidence for making judgments related to program improvement.  Faculty consulted with the stakeholders, shared the results of the collected data, and the decision was made to revise the gates for candidates admitted to the education program beginning fall 2006.  An additional result of the data analysis was the removal of the gate before Block I and the removal of the signature form.  This was done because the data showed us that some candidates who were qualified according to GPA and Praxis I, which are the best predictors of success, were not able to proceed.  Field experience forms and key assignments were added to the gates.  These assessments were chosen because they directly assess the PNC outcomes and their associated performance and disposition indicators. Table 2.5 shows the new gate system. (Exhibit 2.1.5: UCTE minutes, Department Meeting minutes)   

Table 2.5: Gate System Effective fall 2006

Candidates must meet program and gate requirements in order to successfully complete the elementary education program.  These requirements are monitored by the academic advisor at the completion of each semester to ensure that they are met.  If a candidate does not meet all requirements, he/she may be removed from the program.  This is done only after consultation with the academic advisor and the candidate to determine a suitable plan of action.  This plan could entail changing to another program within the university, retaking courses as needed, and taking courses to help remediate in a certain area. Under certain circumstances a candidate may be allowed to continue in the program for a probationary period even though some gate or program requirements have not been met. (Exhibit 2.1.6: File on Special Cases and Outcomes) 

PNC faculty members are committed to a standard of equity and fairness in assessing candidates’ performance in coursework and progress through the gates.  To ensure that standard among all faculty members, training in the use and scoring of rubrics was completed.  In addition, for each major assessment point, inter-rater reliability data was collected.  For example, in spring 2005, one portfolio out of 14 was reviewed independently by two different instructors.  There was 100% agreement on the overall and summary analysis.  There was 80% agreement on the five point rubric for assessing the 10 outcomes.  No score included more than a one point differential.  Again in fall 2005, two reviewers assessed one candidate independently in Block V during the portfolio interview which resulted in 100% agreement in scores. (Exhibit 2.1.7: Inter-rater reliability data)

         To foster candidate success, candidates are made aware of the assessments and gates upon their entry into the program.  They are also provided support by faculty as they complete coursework and field experiences.  Care is taken to ensure that all candidates are aware of expectations and program requirements throughout the program.  Information related to the gates can be found at the education website (http://www.pnc.edu/ed) and candidates are assigned an advisor who is a full-time faculty member when they are admitted to Block III. 

         The PNC Professional Portfolio

         The portfolio and its associated key assignments and rubrics used for scoring are aligned with the PNC outcomes.   The PNC faculty views the portfolio as a tool for candidates to demonstrate and reflect on their progress, and plan for their own future growth in meeting the PNC outcomes.    Candidates are expected to place at least two artifacts in the portfolio at the end of every block of Education courses.  They must include the required key assignment artifacts, and at least one other artifact of their choice.  The goal is for the candidate to demonstrate their growth in the knowledge, performance and disposition indicators aligned with the PNC outcomes.  They describe each artifact and reflect on why they feel it meets the outcome in a Professional Growth Summary.  They also reflect on which outcomes they feel they need to develop further in a Future Growth Summary.  At Gates 2 and 3 candidates have a scheduled conference with a faculty member for a portfolio interview.  The candidate is asked to discuss their Professional Growth and Future Growth Summaries, and the artifacts contained in the portfolio. The faculty member uses an evaluation form to rate the portfolio as acceptable or unacceptable. (Exhibit 2.1.8: Portfolio Handbook, Interview Evaluation form)

         The PNC Professional Portfolio's required artifacts are the key assignments.  These assignments are evaluated by the course instructors before they are placed in the portfolio.  If they are not acceptable, the candidate is not allowed to pass through the gate as these are also gate requirements.  Candidates are given the opportunity to revise unacceptable key assignments. 

         If faculty members determine that a portfolio is not acceptable, candidates are notified and a remediation plan is developed.  This plan consists of allowing the candidate to resubmit the portfolio at a later time after suggested revisions are made.  If the revisions are not made, the candidate will not be allowed to enroll in the next block of education classes.  Candidates may appeal these decisions by following the Appeals Procedure. (Exhibit 2.1.9: Portfolio Appeals Procedure) Other candidate complaints and appeals are kept on file by the Department Chair. 

         Education faculty have also worked together to develop the rubrics used to evaluate the key assignments.  (Exhibit 2.1.10: Key Assignments and Rubrics)  These rubrics are aligned with the PNC outcomes and conceptual framework and their use and scoring have been discussed to ensure that the scorers are fair and there is consistency. (Exhibit 2.1.11: TEAM Meeting Minutes)

         The student teaching evaluation tool is aligned with the PNC outcomes.  Both the university supervisors and cooperating teachers use this tool to evaluate candidates.  Cooperating teachers and university supervisors are in-serviced on the use of these forms during the orientation meeting that is held in August for fall student teaching and in December for spring student teaching. (Exhibit 2.1.12: Cooperating Teacher and University Supervisor Orientation) Student teaching evaluations were examined for accuracy by comparing the cooperating teacher evaluations with those of the university supervisors in spring, 2005.  Both groups differed slightly in their evaluation of the candidates, and both ended up with the same aggregate score of 3.5/4.0.  The largest discrepancy of a student teacher evaluation was .85 (3.05 vs. 3.9).  Four out of 14 of the student teachers had .5 or more difference in their scores.  The correlation coefficient was .37 (Exhibit 2.1.13: Data on student teaching evaluations)  Based on the data results, a change was made in the orientation meeting to have both university supervisors and cooperating teachers attend at the same time so that they could review and discuss the form together.  An additional meeting has also been added at mid-semester for the university supervisors to review their midterm evaluations with one another and the Director of Student Teaching. 

         Element 2:  Data Collection, Analysis, and Evaluation

         The information in Table 2.6 shows that data is regularly collected and analyzed to ensure that the assessment system is maintained.  At the end of every semester, candidate performance is monitored through the analysis of disaggregated data so that only qualified candidates proceed in the program and remediation plans are developed for candidates who are encountering difficulty.  The data that is collected from key assignments, evaluation forms, surveys and Praxis is entered into a data base and the aggregated data is used to assess program effectiveness.  Reports are written to summarize the data, and are annually presented to faculty and stakeholders who then make recommendations concerning program improvement based on that data. (Exhibit 2.2.1: Example Data Reports)    

Table 2.6: Timeline for Data Collection, Analysis and Evaluation

The Purdue North Central Information Services Department is in the process of developing an Education Data Base for utilization by the unit.  When it is complete, it will be the primary method used to organize the data for analysis.  This should simplify the system and make it more manageable.  In the meantime, the unit uses four methods to collect and analyze data for unit evaluation.  The method that has been used the longest and for most data collection is Microsoft Excel.  Data from field experiences, student teaching, follow-up surveys and PRAXIS have been entered into a spreadsheet by the Director of Field Experiences, the Academic/Licensing Advisor or the Education secretary.  Disaggregated data is used to assess individual candidates.  Aggregated data is shared with faculty and stakeholders in the form of tables, charts and reports. 

The Director of Field Experiences also uses Microsoft Access to enter data related to the diversity of field experiences.  This data is used to ensure that all candidates are provided opportunities to work in diverse field sites and at different grade levels throughout the program.

A third method used for data collection and reporting is through EdVanta.  EdVanta is a campus-wide system used to maintain student information from admission to program completion or termination.  Data related to GPA, plans of study and personal information for each candidate are stored in EdVanta.

The fourth method is a web-based tool called TaskStream.  TaskStream was introduced to all teacher education candidates enrolled in block courses in fall 2005.  All candidates were required to purchase it at that time.  Beginning in spring 2006, Block I and any new candidates were required to purchase it and continuing candidates are required to keep their subscription up to date.  Candidates are required to submit their key assignments from each block course on TaskStream.  These assignments are the required artifacts for their Purdue Professional Portfolio.  Rubrics developed by the education faculty which are aligned with the PNC outcomes were used to assess how well candidates are meeting the outcomes as well as whether the elementary education program is providing sufficient opportunities for candidates to meet them. 

         Candidate Complaints

         The Candidate Council was formed in fall 2004 to help provide an opportunity for candidates to voice their concerns and complaints about the program.  It also allows for the unit to strengthen the communication between the candidates and the Education Department.  Each Block selects a student representative to the council which meets at least once a semester with the Department Chair.  Before the meeting, the representative seeks input from other members of their block about any concerns or complaints that should be discussed.  During the meeting, there is opportunity for each representative to voice concerns and discuss them with the other members of the council and the Department Chair.  (Exhibit 2.2.2: Candidate Council Minutes) These issues are then reviewed with the faculty during department meetings.             

Element 3:  Use of Data for Program Improvement

         The Education Department at PNC is committed to regularly and systematically using data for program improvement.  It is only through the careful examination of the data that we can be assured that our candidates and graduates are becoming effective educators committed to continuous growth and school reform.  The examination of the data also allows us to be assured that we are providing appropriate opportunities through our program for candidates to develop the knowledge, performances and dispositions expected of professional educators which are evidenced by meeting the PNC outcomes in coursework, field experiences and student teaching.  

         Candidates and faculty members both use data to make informed decisions related to individual and program performance.  After candidates reflect in their portfolio about how they are progressing in meeting the PNC outcomes, they must write a Future Goals Statement in which they discuss how they will use this information to plan for their own future growth in meeting the outcomes.  This allows them to use the data related to how they are progressing to develop a personal plan for improvement.  This is discussed with faculty during the portfolio interviews that are held after the completion of Blocks II and V.  During these interviews, candidates are also asked for their thoughts on the effectiveness of the program and how well they feel the program is supporting them in the meeting of the outcomes.  Faculty make use of the data received through their assessment of key assignments as well as other coursework, student evaluations, surveys and evaluations filled out by students, cooperating teachers and university supervisors to make improvements in their teaching and in the education program.   

         Academic autonomy was received from the Purdue West Lafayette campus in 2003, and therefore, many of the assessments that are used are relatively new.  To date, there is not sufficient data collected to make major program changes.  However, the collected data has supported the changes that are presented in Table 2.7.

Table 2.7: Data Driven Program Changes

The unit will continue to systematically collect data as the presently designed assessment system becomes the accepted tool for collecting data to make decisions about candidates and the unit.