Common assignment: assessing the rhetorical analysis

Hi there. It’s your friendly Assessment Research Coordinator, Daniel Ernst, checking in with an update on things we’ve learned so far from some of the 2018 ICaP common assignment pilots. A group of ICaP instructors recently read and rated a very popular first-year writing assignment, the rhetorical analysis. Raters not only came to consensus about their ratings, but found strong evidence of student improvement.

Assessment methods and results

For this assessment, we randomly selected 23 pairs of essays from a pool of 60 submitted by instructors. Each pair included two rhetorical analysis essays, one written as a diagnostic pre-test and one written as a post-test to conclude a four week class unit on rhetoric. In total, 46 essays (23 pre-tests and 23 post-tests) were read and rated at least twice by eight graduate student instructors and one professor. Essays were de-identified to prevent raters from knowing who wrote them, which class or instructor they were written for, and whether they were a pre or post test. Raters used a simple rubric built from ICaP outcomes one and three, which focus on rhetorical and analytical knowledge and critical thinking skills.

Comparing the ratings from the nine raters showed substantial agreement and statistically reliable results. (Pearson’s product-moment correlation coefficient of .73.) The highest essay scored an 11/12; the lowest scored a 3/12. Most pre-tests scored at or below 6/12; most post-tests scored at or above 7/12. Here’s what we found:

Pre (5.78) and post test means (7.63) respectively.
Mean pre-test score, 5.78; mean post-test score, 7.63 [t(22)=4.25, p < .001; Cohen’s d=1.04]
There is a highly significant (p < .001) improvement in mean scores between the pre and post test essays. We can confidently say the improvement in mean scores is not likely due to chance but instead likely due to the effect of the treatment: the class and concepts taught. Additionally, the improvement is not just significant but meaningful: the Cohen’s d value of 1.04 indicates the distribution average improved by one standard deviation from pre to post test. This means that a pre test essay scoring at the 84th percentile of all pre tests would score at just the 50th percentile of all post tests. Finally, the post test mean score (7.63 +/- .41) sits right at the midpoint (7.5) of our rating scale (3-12), indicating a distribution of student performance around the true mean of our scale.

Graph showing overlaid distributions
Pre and post test score distributions

So far, so good… both methods and results

Although the sample is limited in certain ways (size, variety, degree of randomness), we are seeing evidence of significant and meaningful growth in writing quality on rhetorical analysis assignments over the course of a single unit in ICaP courses. The evidence is bolstered by the strength of the rating instrument, which almost perfectly sorted pre and post test writing and scored the mean of post test essays at its exact midpoint, as well as the high correlation coefficient obtained by the nine raters using the ICaP Outcomes-derived scaled rubric (.73). As we begin to develop methods for assessment which will be applied program-wide, these results suggest the ICaP Outcomes and Detailed Learning Objectives can serve as source material for designing assignment rubrics, at least for the outcomes on rhetorical and analytical knowledge.

To be sure, we should expect such strong results due to the design of this specific assessment. Any writing done before learning concepts and skills and then measured again after should demonstrate significant improvement. But by building our rubric and scale directly from the ICaP Outcomes, we also show that instructors are meeting the outcomes related to rhetorical and analytical knowledge as our program currently articulates them. Content knowledge about rhetorical concepts and ability to critically analyze texts are fundamental to any writing class. We’re pleased not only with the scores, but the conversations our readers had as we rated sample essays and discussed the rubric ICaP staff and I developed.

We are currently preparing a full report on the common assignment pilots that will use results and data from the rating sessions to make evidence-based decisions on the future direction of program-wide common assignments and the culture of assessment within ICaP. The next steps going forward include revising the rhetorical analysis and other assignment instructor guides for our second generation of common assignments, as well as refining rubrics and assignment sheets. We welcome feedback and questions from any participating instructors or those interested.

Thanks again to all instructors and raters involved in this assessment, including but not limited to Parva Panahi, Mac Boyle, Deena Varner, Libby Chernouski, Julia Smith, Joe Forte, Carrie Grant, April Pickens, and Bradley Dilger.

Update on common assignment pilots

We are continuing our initiative to use assessment data to refine our curriculum and learn more about the success of our approaches to teaching English 106 and 108. Last Thursday, we held our second reading and rating session for common assignment pilots. This was the second reading and rating session led by ICaP assessment research coordinator Daniel Ernst, following a similar event working with a professional emails assignment.

We began by discussing the rubrics used to rate the rhetorical analysis assignments under consideration, and isolating potential issues for discussion. We then read and rated four assignments, discussing the results each time, helping everyone think about the rubrics similarly and also giving us ways to refine the rubrics on the long term. Readers then spent about an hour rating assignments, with each receiving at least two readings.

After considering the results of these reading and rating sessions and instructor feedback on the pilots, we’ve decided on the following process for AY2018–19:

  1. We will have a second generation of multiple pilots in Fall 2018. All ICaP instructors will participate by selecting common assignments, implementing them, submitting data for assessment, and participating in both reading/rating sessions and other measurements.
  2. We will use the same six assignments as in Spring 2018, but assignment materials themselves will be updated to reflect what we’ve learned from our pilots. Requirements for each assignment will be released by July.
  3. We will ask that instructors follow the templates as closely as possible. Assignment materials will describe permissible modifications (e.g. selecting texts or changing timing of deliverables).
  4. Assignments must be graded and assigned points. This helps ensure students do their best work. Individual instructors will determine how the common assignments are integrated into their course assignment sequences and grading structures.
  5. Rubrics will be provided for each assignment. Instructors can customize the format of the rubric to fit with the rest of their course (ie. point values, rating scales, etc.), and may add additional assessment criteria, but all common assignment rubric criteria must be represented in the customization.

The pilots have already been very successful in helping us understand how to balance the needs of syllabus approaches with the overall purpose of the common assignment: ensuring we have a data set from each semester which can be evaluated against ICaP outcomes and in comparison to other semesters. We have some more ideas about Fall processes which we’ll be sharing soon — not to mention the results from the assessments themselves.

We are grateful for the ICaP instructors who have been involved with this work since the beginning and we hope the strong participation will continue. Thank you to our readers: Parva Panahi, Allegra Smith, Amanda Leary, Ingrid Pierce, Mac Boyle, Deena Varner, Libby Chernouski, Julia Smith, and Joe Forte. Thanks as well to the many instructors who are participating in the pilots and focus groups, and to Daniel, Carrie Grant, and April Pickens for preparation, data processing, and continuing analysis.

ICaP Showcase Winners, 2017–2018

Bailey Galema’s presentation for her creative project, “The First One”

Thank you to all of the students and instructors who presented at the 2018 Writing Showcase! We’re pleased to announce our award winners:

Student awards

Best Remediation to Presentation Design
Claire Bailey, “The Impact of the Work Environment on Employees”
Irwin Weiser, Instructor

Best Multimedia Project
Sam Conkle, “Purdue Half Marathon Review”
Lauren Mallett, Instructor

Best Primary Research
Taryn Coyle, “The Benefits and Feasibility of Implementing an Open-Interaction Model at Natalie’s Second Chance Dog Shelter”
Joe Forte, Instructor

Best Visualization of Research
Felix Fernandez, “Artificial Intelligence: Humanity’s Most Powerful Tool”
Ashley Matney, Instructor

Best Creative Project
Bailey Galema, “The First One”
Daye Phillipo, Instructor

People’s Choice
Amy Santos, “Ally Politics in Eco-Feminism: Activism without Cooption”
Victoria Ruiz, Instructor

Instructor awards

Most Innovative Syllabus Approach Application
Joshua Galat, “Engaging the Community: Composing Through Service Learning”

Best Application of Pedagogical Theory
Bianca Batti & Alisha Karabinus, “Composing Through Play”

Best Assignment Scaffolding Design
Derek Sherman, “Intersectional Research: Digging Deeper”

Thank you

Thank you to the Purdue students and ICaP instructors who shared their writing and digital media with us today. We really appreciate your hard work.

Thank you to the ICaP staff, especially Carrie Grant, Joy Kane, and April Pickens, for the logistical work which made Showcase a success.

We appreciate the support from the College of Liberal Arts, and our sponsoring publishers: Pearson, VanGriner, Bedford/St. Martin’s, Fountainhead, and W. W. Norton.

And thank you to our friends in Professional Writing, who’ve announced the awards for their Showcase too.

Second CILMAR grant for Transculturation team

Congratulations to Rebekah Sims, Hadi Banat, Parva Panahi Lazarjani, and Phuong Tran (below, from left), the instructors leading the Transculturation research project in Introductory Composition. They were recently presented a second CILMAR Mini-Grant award of $2,000 to support their research in developing linked courses and curricula which support the growth of intercultural competence. Presenting the award (far left) is Dr. Dan Hirleman.

Well done!

Instructor Spotlight: Victoria Ruiz

Victoria Ruiz: Navigating identities, culture, and performing rhetoric via English 106

ICaP instructor and rhetoric & composition doctoral student Victoria Ruiz

Victoria Ruiz started her graduate school two and half years ago at Purdue and since then she has been teaching First Year Composition. She recently completed teaching her fifth batch of her English 106 class. Her sections are themed around identity as being a negotiation of time and space. In her own words, she designs her classes in such a way that students learn about identities, culture, explore, critique, and perform rhetoric in various cultural scopes across different genres.

In addition to teaching identity themed English 106, Ruiz is also the leader of the Syllabus Approach UR@. UR@ is one of the six different syllabus approaches that instructors of ICaP choose to teach their English 106 class. During their first year, instructors follow their mentor’s syllabus approach for a year. After a year of taking the Teaching Practicum Course with their mentor, instructors can choose different syllabus approaches to teach their classroom, and based on their preferences, they also introduce various assignments that meet ICaP’s six major outcomes.

According to Ruiz, she has made significant changes to her English 106 course as she now structures her course around her own research that is on identity, culture.

“I feel teaching and learning is a both-way process. As much as my students learn from me, I do learn from them as well. Additionally, the UR@ syllabus approach is flexible enough to support changes in the syllabus as per the student needs, and the students really seem to enjoy it,” Ruiz said.

Ruiz’s class is based on identity and through her assignments the students are asked to look at the way identity is constructed and performed through communication and writing for varied audiences, in various genres, and with specific purposes.

She said, “Over the past two years, I’ve curated a selection of additional reading assignments that allows students to know more about writing process, instruction & application, and identity. My students discuss about issues like race, class, ability, privilege, and so on throughout the semester. I have seen their identities change throughout the semester and it is really cool how students get well informed about the real world.”

Every English 106 course is divided into three types of class. Students meet twice a week in a traditional classroom, once in an one-on-one conference, and once in a computer classroom. These varied spaces also help instructors to implement their pedagogy by taking various pedagogical approaches. Most importantly, students and instructors work very closely to meet the goals of the class as set by ICaP.

“Over the past two years, I’ve found that it’s all the little things that we do as instructors that add up and impact our students’ lives. Each of my projects helps frame identity in different ways, and by the end of the semester the students are able to reflect on just how far they’ve come, all the work that they’ve completed, and everything they’ve learned… all of this helps them become better writers,” Ruiz shared.

Victoria’s Crucial Assignment: Personal Portfolio/Auto-Ethnography

The assignment that she is currently teaching  “is probably her students’ favorite” – it is a hybridized personal portfolio/auto-ethnography. Ruiz tries to place a lot of emphasis on the personal portfolio aspect of it so that students feel as though something is at stake and they take a vested interest in finishing out the semester with enthusiasm. This project asks students to compose materials that highlight the transferable skills that they might put to use outside the 106 classroom. In effect, when her students finish the semester – they then have something that they can continue adding to over the course of their collegiate career. This project is also one that they might offer to show potential employers during a job interview, etc. It is their chance to showcase their best work and what makes them most marketable.

Ruiz says she also really loves this assignment because it is structured in such a way that she is able to gauge what the students have learned and how they’ve grown throughout the semester. The reflection portion of the project puts the students and their work in the spotlight, so to speak.

Reaching Beyond the Classroom with English 108-S

Every semester, specialized sections of introductory composition take on the theme of “Engaging Public Discourse.” These accelerated courses ask students to think about public issues and community contexts beyond Purdue’s campus through community engagement.

English 108-S courses can take a number of formats: A class may partner with one community organization, developing projects to meet organizational needs. Students may individually volunteer in the community and write reflective projects related to their service. Assignments may ask students conduct research and take action on community-based issues. In each of these forms, community engagement encourages students’ civic participation and real world applications for learning.

If you are interested in community engagement or teaching 108-S:

ICaP is currently accepting applications to teach ENGL 108-S for Fall 2018. Instructors teaching the course will be required to enroll in English 680: Experiential Learning and Engagement in the fall as a seminar and practicum on community engagement pedagogy. If you think you may be interested in teaching English 108-S sometime in the future, we also encourage you to take this excellent course with Professor Bay.

To apply to teach English 108-S, send an application email to by Wednesday, March 21 addressing the following:

  1. Why you’d like to teach an service-learning/accelerated course
  2. Your background, experience, or interest in community engagement or community service
  3. Ideas you have about community partnerships or assignments you would be interested in developing for your 108-S course
  4. Whether you would be able to take English 680: Experiential Learning and Engagement in the fall, or have already taken it.

If you have any questions about teaching 108-S or the application process, please reach out to Carrie Grant, Assistant Director of Introductory Composition ( or Bradley Dilger, Director of Introductory Composition (

Administrative position selections

We’re pleased to announce our Introductory Composition administrative staff for AY2018–19. Welcome to:

Assessment research coordinator
Derek Sherman
Assistant director
Alisha Karabinus
Assistant mentors
Rebekah Sims and Kylie Regan
Online course developer
Bianca Batti
Technology mentor
Dee McCormick (joining Patrick Love)

Our applicant pool was large and very experienced, so it was difficult to make these decisions. We are grateful to everyone who applied and interviewed with us.

Please join us in congratulating our new staff.

Quintilian awards, Fall 2017

We’re pleased to announce the winners of the Quintilian Awards for teaching in Fall 2017. Congratulations to these excellent scholar-teachers.

  • Hadi Banat
  • Bianca Batti
  • Pamela Carralero
  • Libby Chernouski
  • Diana Clarke
  • Steven Dawson
  • Stacey Dearing
  • Javan DeHaven
  • Hannah Dellabella
  • Megan Denton Ray
  • Daniel Ernst
  • Joseph Forte
  • Daniel Froid
  • Joshua Galat
  • Lee Hibbard
  • Patrick Hoburg
  • Mitch Hobza
  • Aidan Holtan
  • Jen Hughes
  • Mitchell Jacobs
  • Alisha Karabinus
  • Ge Lan
  • Amanda Leary
  • Wei Lu
  • Kyle Lucas
  • Arielle McKee
  • John Milas
  • Michelle Parsons-Powell
  • Daye Phillippo
  • Ingrid Pierce
  • Kylie Regan
  • Margaret Sheble
  • Derek Sherman
  • Allegra Smith
  • Kenneth Tanemura
  • Beth Towle
  • Jessica Varkonyi
  • Zhaozhe Wang
  • Sebastian Williams

ICaP administrative positions 2018–19

We are now accepting applications for 2018-19 administrative positions in Introductory Composition. We have six half-time appointments available:

  • Assistant Director
  • Assistant Mentor (two positions)
  • Assessment Research Coordinator
  • Online Course Developer
  • Technology Mentor (two year appointment)

Applications are due by noon February 15, 2018. Please see the full position announcement for job descriptions, required qualifications, and the application process.

For questions about specific positions, contact Bradley Dilger or one of our current staff. For questions about the application process, contact Dilger.

Update 3/07: We’ve filled these positions. Thank you to everyone who applied, and congratulations to the new staff.

Spring Convocation 2018

Every semester, ICaP instructors and staff gather for news, announcements, and professional development. In Spring 2018, the focus of our Convocation will be assessment, including development of the common assignments we’ll be piloting in 2018.

This post will be updated.