Participating teachers & classes, fall 2001 (27 sample papers)
Charles Darling	Introduction to Literature
Sam Goldberger	Western Civilization
Evelyn Farbman	Literature elective
Peter Wursthorn	Calculus
Carmen Yiamouyiannis	General Biology

Participating teachers & classes, spring 2002 (73 sample papers)
Sam Goldberger	Western Civilization
Anne Kan	Introduction to Music
Barbara diOrio	Art History
Nancy LaGuardia	Marketing Management
Mara Maislin	Early Childhood Education
Lilliam Martinez	General Psychology
Jose Ricardo-Rivera	Social Welfare
Carmen Yiamouyiannis	General Biology

Topics of articles to which students responded

• Views of the Atlantic slave trade
• Analysis of a poem
• Research about ghosts
• Rap music and lyrics
• Stem cell research
• Chemistry experiment applying math principles
• A community's response to a disaster
• Older employees and computer skills
• Marketing to kids
• Evolution—fact and/or theory
• Teenage single motherhood
• Towers of light memorial
• Inclusion of children with special needs

Assignment

Students in all classes received the same assignment sheet, which asked them to read the article their teacher selected and compose a response to it. They were asked to devote their first paragraph to introducing and summarizing the information in the article and to spend the rest of the paper presenting their thoughts about the topic. They were urged to support their ideas with references to the article, to other sources of information, or to their own experiences. Regardless of the range of topics and varying difficulty of the readings, the common conceptual task was to find a balance between objective and subjective discourse—a balance which had been identified as troublesome for students in a preliminary writing scan.

Students were given a week and a half to prepare the paper, and no formal revision guidance was offered. Students prepared one copy of the paper to be read by their own teacher as part of required class work. They made a second, anonymous, copy to submit to the pool from which samples would be randomly selected for the assessment scoring.

Evaluation of the process

Students were asked one question: "What did you like or dislike about writing this paper?" and their responses were copied and then summarized as positive, neutral (including no comment), or negative.

Participating teachers were given an evaluation form with 14 areas to rate (see Instructors' Evaluation below for detail). Seven out of the twelve teachers completed the evaluations. Their responses were summarized to guide the presentation and focus of the next assessment implementation.

Logistical lessons from the project

• For comparable samples, instructions must be very clear and supported by many conversations with teachers.
• About _ of teacher volunteers will not be able to follow through or will turn in samples that have to be disqualified.
• Preparing the samples for scoring is time-consuming (checking for names to black-out, stapling report forms and screens with uniformity, recording Banner numbers and answers to questions on the student report form, etc.).
• Selecting the norming samples is guess-work, since evaluation isn't objective. Need anchor paper(s) from previous session along with fresh ones that will elicit discussion focused on different scoring levels.
• Scorers need clear materials (schema & rubric, sample scoring forms, list of topics, wording of assignment, uniform lay-out of samples, etc.)
• Scoring session needs two leaders to manage balance between logistics and discussion and to provide authority.
• Norming takes up to an hour and a half. Participants feel that it shouldn't be rushed. Scorers can then work at a pace of about 15 papers per hour. (It takes about an hour for 10 people to score 75 papers via 150 readings plus a few extras to resolve splits.)
• After norming and _ hour of scoring, readers need a break. After the break, a quick confirmation norming with one paper can refocus the group. Best to go no more than two hours without a break.
• Scoring phenomena:
  • Fatigue shows up first in the holistic reading, as readers find themselves scanning for analytic categories rather than reading for the whole.
  • When reading a paper on the same topic as an earlier one, readers are tempted to compare the two papers, sometimes unduly applying higher or lower standards to the second one
  • Readers who are knowledgeable about the content of a paper may be more critical than readers who know less about the subject. Opposite may also hold, with the reader compensating for lack of knowledge.
• Questions and issues that arise in scoring session should be noted and pursued (suggestions for methodological revisions, concerns about areas of student need, ideas about implementations, etc.)
• Timely score compilation depends on advance work by data personnel (Banner profiles ready before scoring, anonymous Key Codes generated, spreadsheet completed with all but the CWA scores).
• Interpretation of compiled scores depends on clear purposes and questions. Assessment Team can generate some questions for reports. Departments should generate others for data personnel to pursue using the data compiled.
• Findings should be distributed carefully and with annotation that they are for the purpose of program improvement. They are still subject to statistical challenges as well as oversimplified uses. Interpretive communities need experience.
• Results should include indications about how the process itself improves student learning by demystifying writing throughout the campus:
  • Offering students a specific writing challenge in courses where they might not have written in the designated way, expanding student writing practice
  • Showing students and teachers a scoring rubric, clarifying standards
  • Eliciting interest in writing among participating teachers
  • Eliciting investment in the establishment of standards among readers at the scoring sessions
  • Establishing a cross-departmental notion of common goals.

Instructors' Evaluation Detail

Participating instructors were asked to indicate their agreement with each of the following statements. 4 = agree strongly 3 = agree 2 = disagree 1 = disagree strongly Seven out of the twelve participating teachers responded. Results are summarized in the first column below.
+	1) The writing task was compatible with my syllabus.
split	2) The writing task was clearly described for students.
++	3) The directions for teachers were clear.
-	4) Students were able to follow the directions about format and the extra copy.
split	5) Students expressed interest in reading the article, either before or after.
-	6) Students expressed interest in writing the essay, either before or after.
+	7) The assignment helped students to learn something of value to our coursework..
++	8) Reading / grading the students' essays was worth my time.
+	9) The assignment was useful in my class planning.
+	10) The project has led me to reconsider the way grade student writing.
split	11) My field requires better writing than our students generally demonstrate.
++	12) Courses in my field should incorporate more writing.
+	13) I would like to participate in further work on improving student writing at Capital.