Capital Community College - Common Writing Assignment Report

About Capital • Contact • Giving to Capital • Website Directory

Home | Faculty & Staff | Institutional Assessment Portfolio

Assessment Resources

Related Information

NEASC Accreditation Self Study

Report on Assessment of Writing
through a Common Writing Assignment
2001-2002

INTRODUCTION

In Capital Community College's design for Student Learning Assessment, individual programs are responsible for refining and assessing their goals according to cycles that fit with program review, certification, or internal examination schedules. At the same time, the broader curricula of General Education and Developmental Education require ongoing assessment, and the plan is to implement one large cross-curricular assessment each year for these areas. Since both General Education and Developmental Education share the goal of effective writing, the first year of implementation was devoted to sampling student essays to determine how well our students are meeting that goal.

METHOD

Instrument: course-embedded essay

Twelve teachers in diverse fields ranging from literature to marketing to math gave a common writing assignment to their students in fifteen different classes. Students in all classes received the same assignment sheet (see Assignment), which asked them to read an article selected by their teacher and compose a response to it. They were asked to devote their first paragraph to introducing and summarizing the information in the article and then to spend the rest of the paper presenting their thoughts about the topic. They were urged to support their ideas with references to the article, to other sources of information, or to their own experiences. Regardless of the range of topics and varying difficulty in the readings, the common conceptual task was to find a balance between objective and subjective discourse-a balance which had been identified as troublesome for students in a preliminary writing scan.

Students were given a week and a half to prepare the paper, and no formal revision guidance was offered. Students prepared one copy of the paper to be read by their own teachers as part of required class work. They made a second, anonymous (temporarily identified by Banner student ID number), copy to submit to the pool from which samples would be randomly selected for the assessment scoring.

Scoring: holistic and analytic

One hundred samples representing all fifteen classes were scored on a scale of 1-4 by members of the Assessment Implementation Team in two scoring sessions, one in November and one in May. Readers read each paper twice: once for holistic scores and then again for analytic scores according to a rubric (see Rubric for detail). Each scoring session began with a little over an hour of norming. For the May scoring session, norming samples included two anchor papers from the November scoring. After norming, each paper was read by two readers. If any paper received holistic scores differing by more than one point on the four-point scale, it was read by a third reader. No reader was able to see the previous readers' scores.

Data management

For each sample, readers' scores were averaged in each category and entered on a spreadsheet showing students' academic histories for analysis. By this point, student Banner ID numbers had been replaced by assessment key code numbers in an irreversible translation, assuring full anonymity for participants. Further information on data management is available on the Reporting Design page.

Validity and reliability

As a writing task that was part of the work of ongoing courses, the essay had high content validity. As performance indicators of effective writing, the sampled essays were authentic texts produced by students rather than responses to abstract measures intended to predict performance. The scoring scale and analytic rubric were based on previous readings of student papers that had led to consensus among Capital Community College staff about the key elements of writing. Further, the assignment had high consequential validity in its stimulation of writing and discussion of standards across the curriculum.

In holistic scoring, inter-reader reliability (agreement of two readers within one point on a four point scale) was 84%. Sixteen papers required a third reader. Analytic scoring on the rubric's four criteria showed greater variation on some criteria, less on others. An additional possible indication of reliability was offered by the observation that of the six students who wrote essays in two different classes, four received identical scores in all categories, and one received scores differing by only half a point.

RESULTS

Process lessons

As a pilot project in the assessment of General and Developmental Education goals, the Common Writing Assignment presented many opportunities for learning about:

logistics of course-embedded measures
collegial establishment of standards and criteria
scoring challenges, reliability and validity
data collection and management
reporting, interpretation, and application of results

Participating students were asked how they felt about the assignment, and participating teachers were given an evaluation questionnaire about the project.

Half of the students reported positive responses to the assignment (either reading, writing, or both). Twenty-nine made no comment or neutral comments, and twenty-one reported negative responses.
Seven of the twelve teachers who embedded the assignment in their courses responded to an evaluation of the project. Teachers' greatest agreement was that reading and grading the essays was worth their time and that courses in their fields should incorporate more writing. Their reports of students' mixed levels of enthusiasm for the project roughly match the student's self-reports.

Many of this year's lessons can be applied to the next cross-curricular assessment of goals. Further details of the process are collected on the Common Writing Assignment Process page.

Findings

Writing problems clustered around use of standard written English and inadequate development of points. Strengths were evident in the areas of purpose and structure-students found appropriate avenues into the task and were usually able to frame their compositions, using introductions and conclusions with some confidence.

Data supported the following conclusions:

A trend line showing the relationship between holistic CWA scores and Grade Point Averages indicates that students who write effectively (according to our readers) are doing better in school than students who struggle with writing.
CWA scores show a rough bell-shaped distribution with the median peak at 2.5 on the 4 point scale.
Analytic scores based on the rubric range from highest to lowest as follows: purpose & audience, organization, development, language.
In holistic scoring, students who enter at the developmental level (based on writing Accuplacer scores) are less likely than other students to reach the level of proficiency (3 or above).
In the analytic categories, students who enter at the developmental level score consistently lower in every category, with the greatest divergence showing in the category of language.
Students who receive a B- or better in either the upper level-developmental writing course or English composition are likely to score close to or at the level of holistic proficiency on the CWA.
Self-reports from students about the amount of writing they'd done in other courses (besides writing classes) indicate that those who had written essays in other classes across the curriculum score higher on the CWA and on GPA than those who had not practiced writing in other classes. However, this data may be unreliable, since it is based on self-reporting, so it opens an area for further study.
Students who reported clear feelings about participating in the CWA, either positive or negative, scored better than those who were neutral or silent.

CONCLUSIONS AND RECOMMENDATIONS

Overview and Suggested Goals

The most global findings for 2001-2002 indicate that proficient writing is related to success in school and that 30% of our students are writing at the level of proficiency (holistic score 3.0 or above). The mean holistic score for the entire cohort is 2.3.

The Student Learning Assessment Team suggests that units throughout the college use the findings to design interventions leading toward achievement of the following goals by the fall of 2005:

To raise the holistic mean by .5, from 2.3 to 2.8
To increase the percentage of students writing proficiently by one third, from 30% to 40%.

The Team further suggests that the Common Writing Assignment be repeated in 2005-2006 to assess progress toward these goals.

Recommendations

A. That the Humanities Department's Committee on Writing Standards (COWS) initiate and coordinate dialogue and professional development activities for the purpose of improving student writing across the college. Further, that COWS annually assess student writing in developmental courses, College Success, English Composition, and courses across the curriculum, reporting findings to the Student Learning Assessment Team, the Academic Dean, and faculty and staff across the college (see Specific Findings & Interventions #1, 2, & 5 below).

B. That departments, curriculum planners, academic advisors, counselors, and enrollment services staff collaborate in designing and enforcing policies that support early and continuous student practice in writing (see Specific Findings and Interventions #3,4, & 5 below).

Specific Findings Suggesting Interventions in Curriculum, Pedagogy, and Professional Development

1. The main obstacles to proficiency lie in the areas of development (support of ideas with evidence, examples, elaboration of topics, etc.) and language (effective use of sentence structures, word choices, and mechanics of standard written English).

Therefore:

In order to reach the suggested goals, curricular planning and professional development activities should explore methods of increasing students' skills in the categories of development and language, as defined in the scoring rubric.
The rubric explaining the four analytic categories of writing proficiency should be distributed to students and staff throughout the college in order to open discussion of Capital Community College writing standards.

2. Students who enter the college at the developmental level show a pattern of writing scores parallel to but lower than those of non-developmental students. Since developmental students comprise at least one third of the total cohort, the college cannot reach the goals listed above without raising the scores of the developmental group, a group for which the skills in the category of language are a particular challenge.

Therefore:

Interventions designed for developmental students should explore methods of building students' skills in all categories, and with particular attention to methods of increasing mastery of standard written English.

3. Grading in required CCC writing classes roughly matches the broader pattern of CWA scores, providing assurance that various college approaches to writing pedagogy are sharing roughly similar goals and assessment criteria. In addition, students who successfully complete English 101 are likely to be writing at least at the essential level on the CWA. Higher GPA's are associated with CWA scores at and above this level.

Therefore:

The college has a policy that students should complete English 101 within their first 15 credits. Academic advisors, counselors, and support staff should advise students of the relationship of this policy to their overall success in school, and the policy should be strictly followed in the registration process.

4. Students who report having written essays in classes other than English 006 and 101 demonstrate greater levels of writing skill and are more likely to have reached the proficient level.

Therefore:

The college should continue to develop learning communities that pair writing courses (e.g., English 006, 107, and 101) with introductory courses in the disciplines. The purpose is to support writing across the disciplines and engage more faculty in assigning, assessing, and improving student writing.
The college should pilot W-designated writing-intensive classes in business, science, social science, and other fields. It should support these pilot courses with professional development activities, limited enrollments, and assigned tutors to provide supplemental instruction in writing.

5. Showing growth over time, scores indicate steady increases in students writing at the proficient level in three sequential categories of writing-intensive classes: English 006, English 101, and writing beyond English 101. However across these three stages, a significant number of sampled students continued to write below the essential level.

Therefore:

The college should develop goals and practices for increasing the percentage of students writing at or above the essential level upon successful completion of English 101.
The college should build on this year's baselines showing increases in percentages of students writing at or above the proficient level, setting benchmarks at each stage for additional growth by 2005.

IMPLICATIONS FOR FUTURE ASSESSMENTS

The design and implementation of the Common Writing Assignment was guided by the following values:

a) Distillation of criteria to reveal the big, pervasive questions
The goal to be assessed was one that mattered to many educators and students and one that subsumed a variety of more specific objectives. The criteria for the scoring rubric were arrived at collegially, addressing the large goal and, in the process, clarifying it.

b) Content or face validity
The measurement tool directly measured performance on an authentic writing task.

c) Balance of local thinking and information from national models
The Student Learning Assessment Team adapted national models to local needs and took guidance from reports of writing assessment experiences elsewhere, particularly concerning the challenges of sampling and scoring authentic texts.

d) Open-ended content and structure
The assignment gave teachers and students room for choices. The uniformity of the writing task provided comparability but also encouraged diversity of student approaches and strategies. Wide participation from staff was invited for the embedding in courses and the scoring of samples.

e) Consequential validity
The consequences of the implementation were supportive of learning. For participating students and teachers, the project encouraged writing across the curriculum. For the assessment practitioners at the College, the implementation provoked discovery and further inquiry about the purposes and mechanisms of assessment. For other staff, it elicited constructive discussion of writing and standards.

f) Genuinely useful results
The data collection and management design carefully avoided the accumulation of data that was not directly useful to the improvement of student learning. It limited its focus to patterns of writing in anonymous aggregates of students, focusing on fields that had potential to guide interventions for program improvement.

These values, along with the choices that they prompted, provide groundwork for the design of future assessments. They are already guiding the preparation of the next General and Developmental Education assessment, which will focus on quantitative reasoning. The data management design, which extracts information on academic history and then assures student anonymity in reporting, is replicable and adaptable to many other types of student learning assessment. In addition to providing useful findings about student learning, the Common Writing Assignment has yielded confidence in the assessment process and momentum towards the development of an ongoing culture of inquiry at Capital Community College.

SUMMARY

For a three-page condensation of key points about the CWA, see the Summary Report.

Strategic Planning

Assessment of Student Learning

Institutional Effectiveness Data