Evaluation of 
Teaching Effectiveness

SJSU Academic Senate Retreat - October 12, 2001

[Web Site Prepared by Professor Annette Nellen, Retreat Planner]

Why is this web page here?

Table of Contents

Overview - to the Web Page and Retreat
Retreat and Web Page Topics:

• The Meaning of Teaching Effectiveness and Considerations in Evaluating It

• Types of Evaluation Tools and Techniques:

  • Student Input (SOTEs)

  • Peer Review

  • Teaching and Course Portfolios

  • Other Evaluation Techniques and Tools

• Summative versus Formative Review

• Notes from the Retreat

Additional Resources

Contact Information

Overview

Purpose of this Web Page

Please read the next section on the purpose of the Senate Retreat before October 12, 2001. Beyond that, this web page is NOT required reading for the Senate Retreat!  Instead, it serves to provide additional information should you want to read more in preparation for the Retreat, if you are curious as to what types of information about evaluation of teaching effectiveness are available on the Internet, or you are interested in helpful resources to supplement the discussions I hope you will have with your colleagues following the Retreat.

Purpose of the Senate Retreat

There has been much discussion on campus recently about issues surrounding the SOTE  instrument. Part of this discussion is likely due to campus forums held by the Senate's Student Evaluation Review Board (SERB) and the testing of new questions in Spring 2001. During the past year, in discussions with members of the Senate's Professional Standards Committee and others, I suggested that some of the concerns many express about the SOTE result from it becoming too significant a part of the evaluation of teaching effectiveness primarily because as a campus, we don't effectively use other measures of evaluating teaching effectiveness. In addition, I suggested that we don't always use the SOTE results effectively or correctly.  I did not find anyone who disagreed with me. Over the past two years, the Professional Standards Committee has reviewed a sampling of peer review forms used across campus - there is very wide discrepancy among them and it seems that too often, the entire review consists solely of a single classroom visit and perhaps no actual dialogue between the reviewer and reviewee.  In addition, while individual faculty may decide to include a course and/or teaching portfolio in their dossier, there is no requirement to do so and there is little discussion on campus of what potential benefits exist with portfolios. Finally, reviews of teaching effectiveness at SJSU are used almost exclusively for summative review, rather than formative review. Thus, we miss out on opportunities both for dialogue on teaching practices and to help ourselves and colleagues improve and enhance teaching and learning. Broader and more inviting opportunities to discuss teaching and learning with our colleagues would enhance our campus culture of teaching.

The Retreat will focus on evaluation of teaching beyond student ratings. Since the topic of the Retreat is one for which we all have experience and ideas, there will be several opportunities to discuss techniques for evaluating teaching with the outcomes of these discussions made available to the appropriate Senate committees and for attendees to take back to their departments and colleges.  It is hoped that the discussions and sharing of ideas at the Retreat will result in attendees continuing to discuss these ideas with colleagues such that the work at the Retreat can serve as a strong start to helping to change the campus culture to consider multiple measures of teaching effectiveness as well as using them effectively for both summative and formative purposes.

Elements of Evaluation 

I view an evaluation system as requiring four key elements:

1. Purpose - summative or formative?
2. Criteria - what is being assessed? what is the evaluation scale? how is the scale to be used?
3. Documentation - what is needed to document performance?
4. Process - when is the evaluation needed, who is to do it, how often is it to be done?

At the Senate Retreat, we will spend most of our time discussing item 3.  Item 4 - Process is specified in the contract and university policy (see below).  We will also spend some time on item 1 by considering how formative (developmental) review can be incorporated into existing review structures and into new dialogues on campus about teaching and learning. Item 2 is covered in the university policies calling for evaluation - RTP (S98-8), post-tenure review (S97-5), as well as in guidelines established by departments and colleges. We will have a discussion of the meaning of teaching effectiveness at the start of the Retreat.

Contract and University Policy Relevant to Evaluation of Teaching

• CSU/CFA Contract Provisions on Evaluation - paragraphs 15.14 - 15.31 are most pertinent to forms of evaluation.
• Senate Policy on Evaluation of Teaching Effectiveness - S91-9 (this policy speaks only to the frequency of SOTEs and peer review, and does not address how to evaluate teaching effectiveness).

Return to Table of Contents.

The Meaning of Teaching Effectiveness and Considerations in Evaluating It

We will begin the Senate Retreat with a discussion of what "evaluation of teaching effectiveness" means to us - you may want to give this some thought as you prepare for the Retreat and consider what you have done in the past to enable reviewers to evaluate your teaching effectiveness. In order to know if we are collecting and utilizing appropriate data to evaluate teaching effectiveness, we need to know what it is we are trying to evaluate - that is, what are the dimensions of teaching effectiveness.

• Course Anatomy: The Dissection and Analyses of Knowledge Through Teaching, Lee S. Shulman, President, The Carnegie Foundation for the Advancement of Teaching (this article refers to five elements of teaching and then leads into a discussion of teaching portfolios)
• Chickering and Gamson's Seven Principles of Good Practice - from Indiana  University-Bloomington
  • Background on the Seven Principles by Zelda F. Gamson
• "Effective Teaching Principles and Practices" from the University of British Columbia (a clear set of 7 principles for effective teaching that also notes "exemplary practices" for each principle)
• Characteristics of Effective Teachers
  • University of Texas
  • Stanford
• Information from Stanford's Center for Teaching and Learning

Return to Table of Contents.

Student Evaluations of Teaching Effectiveness (SOTE)

Contract and University Policy Requirements

• CSU/CFA Contract Provisions on Evaluation - paragraphs 15.14 - 15.17 are most pertinent to forms of evaluation.
• SJSU's SOTE and SOLATE Interpretation Guide (S98-4) - guidance in interpreting SOTE results + the statement: "Information from SOTE and SOLATE surveys are but one source of information for assessing teaching effectiveness."

Update on the Work of SJSU's Student Evaluation Review Board (SERB)

• Presentation slides about the SOTE rewrite process from the October 12, 3001 Senate Retreat

Truths and Myths and Debates about Student Evaluations of Teaching Effectiveness 

Most of the literature found indicates that there are many myths about the value of student evaluations. For example, studies have reported that there is no correlation between evaluation scores and easy teaching or grading. The first three sites below discuss some of these myths.  In contrast, the last three sites below raise issues about the validity of student ratings.

• "What Do They Know Anyway?" by Richard Felder, Chem. Engr. Education, 26(3), 134-135 (Summer 1992). 
• "Myths & Misperceptions" from University of Illinois at Urbana-Champaign
• "Cornell study finds student ratings soar on all measures  when professor uses more enthusiasm - Study raises concerns about the validity of student evaluations" (9/97)
• "Student evaluations get failing grades" from the APA Monitor (2/98)
• "Student Evaluation of Teaching: A Methodological Critique of Conventional Practices, " by Robert Sproule, Bishop's University (Canada) (11/00)

Information about Effective Use of Student Evaluations

• "Using Student Evaluations to Improve Teaching," from Stanford University's Speaking of Teaching (Fall 1997)
• "Making Evaluations Effective," by Richard Felder, Chem. Engr. Education, 27(1), 28-29 (Winter 1993). 

Return to Table of Contents.

Peer Review

• "Peer Review of Teaching - "From Idea to Prototype" Lessons from a Current AAHE Teaching Initiative Project." by Pat Hutchings

Overview: Provides reasons why peer reviews are needed, addresses the question: "What kinds of peer interaction -- between whom and under what conditions -- will contribute to the things we care most about, including student learning?", and provides suggestions for making peer review more than just classroom observations.

• Hints for constructive peer review from The University of Liverpool

• "Identifying the Focus of a Peer Review of Teaching System" 

• "Classroom Observations" 

  Both summaries above are from the Peer Review of Teaching: A Sourcebook, by Nancy Van Note Chism of  The Ohio State University. Summaries provided by Terri Thames, Chair of the SJSU Professional Standards Committee.

• Books on Peer Review:
  • The Peer Review of Teaching -- Excerpt from Making Teaching Community Property, by Pat Hutchings
  • Innovative Higher Education, Spring 1996, Daniel J. Bernstein and Kathleen M. Quinlan, Guest Editors
    Copies of the Spring 1996 journal are available for free. I found the articles to be interesting. Click here for info on ordering.

   

• Information on the AAHE The Peer Review of Teaching Project

Return to Table of Contents.

Teaching and Course Portfolios

A course portfolio focuses on what a professor has done with a single course from developing it to assessing it. A teaching portfolio is broader and would include more than one course and focus more on the instructor's general teaching philosophy than a course portfolio might. A portfolio can be viewed as analogous to what an artist would assemble to demonstrate his or her talents and career progression.

Course and teaching portfolios typically includes the instructor's syllabus, sample assignments, tests, student evaluations (SOTEs), and peer reviews. Thus, it may seem that the dossier used in SJSU's RTP process is a portfolio. However, there is a significant difference between the dossier and a portfolio. The SJSU dossier just calls for inclusion of the course documents. There is no requirement for any type of explanation from the faculty member. (Per S98-8, "Faculty members under review may include an analysis of any of the materials in the dossier and a statement of their teaching methods and goals" (emphasis added).)  A portfolio would include reflective statements about teaching and the course - what are the professor's goals and how are they accomplished in the course?  The course portfolio would also describe what changes the professor has made to the course and how they have worked.  The course portfolio can help the professor track the changes they have made to a course over time and note changes planned for the future.  The course portfolio can also be useful to explain what the course is all about (particularly as taught by the professor who created the portfolio).

In The Teaching Portfolio - Capturing the Scholarship in Teaching, an AAHE publication, the following four reasons are given to support the use of portfolios:

1. "Portfolios capture the complexities of teaching."
2. "Portfolios place responsibility for evaluating teaching in the hands of faculty."
3. "Portfolios can prompt more reflective practice and improvement."
4. "Portfolios can foster a culture of teaching and a new discourse about it."

Course Portfolios

• The Course Portfolio - This article at the American Psychological Society web site starts with a quote about the meaning of teaching from Shulman.  The author likens the portfolio to a "manuscript in progress" as prepared in scholarship activities.
• Handbook on Course Portfolios - from the University of Wisconsin-Madison
• Portfolio Projects of the AAHE Teaching Initiative
• Sample Course Portfolios:
  • Randy Bass at Georgetown (English)
  • University of Nebraska-Lincoln 

Teaching Portfolios

• Preparing a Teaching Portfolio - A Guidebook -   This site from UT Austin provides a brief overview to the purpose of a teaching portfolio, how to create it, contents, 
• Developing a Teaching Portfolio - This site from The Ohio State University also provides a brief overview to creating a teaching portfolio.
• Teaching Portfolios - This site from Penn State provides information on how to get started on your portfolio.
• "Promoting a Culture of Teaching: The Teaching Portfolio" - From Stanford University's Speaking of Teaching (Spring 1996)
• Sample Statements of Teaching Philosophy (you can likely find many more with a Google search):
  • Anil B. Jambekar at Michigan Tech (Operations Management)
  • Jeffrey A. Reimer at UC Berkeley (Chemical Engineering)

Other Evaluation Techniques and Tools

• Interviewing Students about Their Learning Experience to Show Evidence of Teaching Performance - from University of Wisconsin at Madison
• A variety of assessment documents from the University of Nebraska-Lincoln that look at student time, faculty time, and course materials.

Return to Table of Contents.

Notes from the Retreat

Click here.

Additional Resources

There are a lot of reference materials available in the Library on the subject of student evaluations of teaching effectiveness.

There are a variety of books on evaluating teaching effectiveness from:

• AAHE
• Ankar Publishing
• Jossey Bass

Return to Table of Contents.

Contact Information

Return to Table of Contents.

San José State University
Academic Senate

Last updated February 4, 2005