ICAP Convocation, Spring 2020

Here are the materials for our January 10 Convocation in Grissom Hall.

All of the assessment materials shared have been added to our November blog post, and will be integrated into the new ICAP web site as well.

Assessment Updates

Three updates as we near the end of Fall 2019:

Norming, reading, and rating, Spring 2020

As in Spring 2019, we will be reading and rating Fall 2019 work as a large group. This process will begin with norming at our Spring 2020 Convocation (January 10, 2020, 11:00am) and will continue into the Spring 2020 semester. 

We will have four sessions before Spring Break. Please plan to attend at least two.

  • February 4: 9:00 to 11:45a (KRAN G002) 
  • February 5: 12:30 to 3:20p (REC 309)
  • March 2: 12:30 to 3:20p (REC 309)
  • March 5: 9:00 to 11:45a (KRAN G002)

If you teach English 106 and need to cancel conferences to make attending these sessions easier, please do. (Make sure to let Joy Kane know, of course.)

Student guide for creating the portfolio is ready

The tech mentors have graciously created a guide to creating the portfolio for students (PDF). In this guide, students will find information on how to create, combine, and format PDFs. In addition, students will find information on how to make the portfolio readable for accessibility purposes. Please inform your students of this guide to help them compose the final portfolio. 

Standardizing the Portfolio Format: Spring 2020 and beyond

Based on our assessment of the Summer 2019 portfolios, ICaP and the Assessment Committee have agreed that working with portfolios would be easier if the format and organization were  standardized to some degree. We ask instructors to use this format and organization in Spring 2020 if possible. We will gather feedback and finalize guidelines to be used by everyone in Summer 2020 and beyond. 

We request the following format guidelines: 

  • Title page with student name, instructor name, and course number 
  • Table of Contents with all writing projects  and their respective parts listed 
  • Clear labels within the portfolio for each part of each writing project  (e.g., proposal, rough draft, final draft, etc.)
  • Each component should start on a separate page 

We also ask that this organization be used: 

  1. Title Page 
  2. Table of Contents 
  3. Final Reflection 
  4. Course Writing Projects in chronological order
  5. Each writing project should also include its components and/or milestones in chronological order, i.e.
    1. Brainstorming 
    2. Rough Draft(s)
    3. Feedback (Peer and/or Instructor)
    4. Final Draft 
    5. Mini-reflection (memo in English 106Y)  

Again, this change is voluntary for the Spring 2020 semester. 

Update: Additional Voluntary Changes

Based on feedback from the portfolio submissions from Fall 2019, instructors are welcome to make the following additional changes to the portfolio assignment. This handout summarizes them.

  • Final Reflection: You may create your own prompt that focuses on some outcomes rather than all
    • Throughout the semester, however, students should have reflected on each outcome once
    • Continue to use the same rhetorical moves suggested in the Instructor’s Guide to the Portfolio
  • Final Reflection Audience: You may change the audience from an employer to an audience that more closely fits your syllabus theme or preference
    • You will need to submit this assignment sheet to Qualtrics
  • Reflective memos/logs: You are free to create your own prompt that fits the major assignment
    • Continue to use the same rhetorical moves suggested in the Instructor’s Guide to the Portfolio
    • Make sure these remain relatively small
  • Beyond the PDF Portfolio: Some instructors in Fall 2019 experimented with digital/web-based portfolios, so you may also use this format. Please see any of those instructors for advice
    • All components of the portfolio would remain the same
    • Submission would require a link

Thank you!

We would like to thank all Summer 2019 instructors for their commentary on the portfolios, especially those who attended the norming session on October 22, 2019. The commentary and suggestions we received have helped us continue to refine our assessment process.

The Assessment Committee is also hard at work at creating additional resources for teaching the portfolio and reflection. We will share those materials with you as soon as we can. 

If you would like to further contribute to the assessment conversation, please join the Assessment Committee. Please contact Derek Sherman (sherma11@purdue.edu) for more information. 

Thank you all for your hard work!

Save the Date! Norm, Read, and Rate Session

ICaP has scheduled a norm, read, and rate session for Tuesday, October 22nd, 2019 from 9:00 a.m. to 11:00 a.m. in PHYS 110 (see map below). This session is voluntary for instructors, but we have scheduled it during mentor group so that most instructors will be free. We will norm the portfolios from Summer 2019 to understand what may need to be revised for Spring 2020. 

Our goals for this norming session, include the following:

  • Create a consensus on what a portfolio should contain and how to read it 
  • Create a consensus on what the reflective log and final reflection should contain and how to read them
  • Discuss variations in how each 106 course meets ICaP goals and the portfolio’s need
  • Read and rate portfolios to illustrate changes for Spring 2020 common assignment

We will provide coffee, tea, and bagels at this session, so please come and help us out. The goal is to have incentives, so stay tuned for more updates. This post will be updated once more information is available. If you have questions, comments, or concerns, please email Derek Sherman (sherma11@purdue.edu).

Common Assignment Update: Plans for 2019-2020 Academic Year

As a full year of piloting our common assignments comes to an end, we want to highlight our assessment efforts and what’s to come for the 2019-2020 academic year. We, the ICaP team, would sincerely like to thank all of you who have helped us in this assessment effort because we couldn’t have done it alone. There were certainly bumps along the way with our assessment goals and what it means to assess, but we feel that our efforts have allowed us to showcase the powerful work that our instructors and students create. From our successful pilots, feedback sessions, Spring 2019 Convocation, and our norm, read, and rate sessions, we believe that it was a successful year that has allowed us to collaboratively think about next year. Again, thank you all for your help and insight into this project.

The future of our ICaP assessment efforts will be shifting from four common assignments down to one: the portfolio. Portfolios were chosen based on several reflective sessions that we have had with instructors, norm, read, and rate participants, and especially the many voices at Convocation:

  1. Portfolios can measure all six outcomes instead of individual outcomes–which would require multiple common assignments
  2. They allow us to assess our new syllabus themes with a common assessment protocol already in place; essentially, we are able to tell which themes best fit students’ and ICaP’s needs.
  3. Portfolios push students towards metacognition with the reflective component, while also preparing them to work towards greater proficiency in writing.
  4. Portfolios allow instructors to continue a diversity of teaching assignments with only minor standardization in the reflective component.
  5. Students are provided with a product that can be showcased to future employers to demonstrate that they are able to write effectively in multiple rhetorical situations and genres, and are able to address multiple audiences that goes beyond the teacher-as-audience model
  6. The portfolio allows students to guide instructors and raters through their writing and learning processes instead of the instructors making those choices without student input

In all, the portfolio, in terms of assessment, allows ICaP to assess students’ ability to meet all six of our outcomes while at the same time placing the onus of learning on the students. For assessment purposes, we have standardized the reflective essay, assignment sheet, and components (e.g., rough drafts, peer review feedback forms, etc.) that are required.  By creating a commonality amongst all of our 106, 106Y, and 108 sections, ICaP is able to assess how well each of our courses are doing in terms of meeting the outcomes set forth by the Undergraduate Core Curriculum committee for information literacy and written communication. This rhetorical move on our part allows students to develop the metacognition that is required as they move throughout their general education and major curriculum.

The assessment committee is finalizing the portfolio assignment now, and we’ll share that with supporting materials here shortly.

Foundational outcomes assessment, part one

For the 2018-2019 academic year, ICaP has two extensive assessment projects underway. First, we are continuing our own programmatic assessment based on the external review completed by the Council of Writing Program Administrators’ Consultant-Evaluator Service in 2017. ICaP instructors will recognize this as our common assignment initiative. We’re currently scheduling two program-wide norm, read, and rate sessions this semester which will allow us to finalize (a) the selection of the common assignment to be implemented in all ICaP courses in AY19–20 and beyond, and (b) a comprehensive assessment plan which will include both short- and long-term measurements of  ICaP’s effectiveness.

Second, we are completing an assessment of English 106 and 108 focusing on their role in Purdue’s Undergraduate Core Curriculum (UCC) Foundational Outcomes, specifically written communication and information literacy. This assessment draws on the direct measurement of student writing ICaP instructors are supporting through our norm, read, and rate sessions, and adds background to showcase how ICaP policies like syllabus review, as well as our assessment efforts, ensure we meet and exceed Purdue’s UCC Foundational Outcomes.

These two assessment projects overlap in many ways, so we are sharing the preliminary report we’ve written for the Foundational Outcomes assessment for the Office of Institutional Research, Assessment and Effectiveness. We’ve published some of the appendices already, but those new to ICaP may be interested in them.

In a follow-up report, ICaP will provide the additional data requested for the foundational outcomes assessment.

My staff and I would like to take this opportunity to thank the student workers, instructors, and administrators who have helped us with these assessment efforts. Without our “grassroots curricular assessment model” (Conti, LaMance, and Miller-Cochran, 2017), our assessment efforts would not have been as rich or as rewarding as they are now. And I would like to highlight Derek Sherman’s work as Assessment Research Coordinator: he’s built on previous efforts extremely well, and I’m looking forward to our next steps.

Attachments: Foundational outcomes assessment preliminary report (as PDF, 284kb) and appendices (as a single PDF, 5.9mb, and also as separate PDFs).

Spring 2019 syllabus prep materials ready

Everything instructors need for spring syllabus preparation is now ready. We’ve made no changes to common assignment instructor guides, and we’ll be using the same syllabus review checklist from the fall. Remember, all ICaP courses must include one of these five common assignments in Spring 2019. (For those teaching online 106, English 106-Y, we’ve already shared materials by email.)

Please use the checklist in the construction of your syllabus to ensure you’ve included all the necessary details. If you get stuck, check Linda Haynes’ syllabus template, developed last fall. (Again—no changes.) This template, based on materials given to new instructors, offers everyone a second way to check their syllabi and provides robust examples of policies and possibilities. Not sure how to frame something? Check the template for examples.

There are two changes instructors should keep in mind, however:

First, instead of emailing syllabi to Dilger, we’ll ask you to fill out this Qualtrics form, which allows you to submit multiple syllabi at once (if necessary). We think this will make processing syllabi easier for us.

The second change is a required addition for your calendars. In Spring 2019, we will be asking all ICAP instructors to participate in two program-wide assessment norming, reading, and rating sessions. We will have sessions both before and after Spring Break, with multiple time slots to minimize schedule conflicts.

  • Mon Feb 25, 12:30p to 3:00p
  • Thu Feb 28, 9:00a to 11:30a
  • Fri Mar 01, 10:30a to 2:00p
  • Tue Apr 02, 9:00a to 11:30a
  • Wed Apr 03, 2:30p to 5:00p

Please plan to attend at least two of these five sessions. If necessary, you may cancel classes to facilitate the time needed to participate. We will share the locations and specifics at Convocation (including some incentives for participation). For now, please make the appropriate plans in your course calendars.

Need something else? Let us know. We welcome instructor feedback and continued suggestions for resources which can support teaching.

Common Assignments: A Year in Review

We have completed the pilots of the ICaP common assignments. In the Fall of 2017, ICaP administrators in conjunction with the grad student-run Pedagogical Initiatives Committee (PIC) began developing six different common assignments to trial run in English 106 and 106-I courses during the Spring 2018 semester. This was a decision made largely in response to the Council of Writing Program Administrator’s (CWPA) external review of our program a year prior, which suggested a common assignment could be a good way to introduce more consistency to our diverse syllabus approach system while at the same time preserving instructor freedom.

Since then, a team led by and consisting entirely of graduate students across the English department has collectively developed and taught, as well as rated and analyzed samples of, six different common assignments piloted by 39 instructors and completed by more than 780 Purdue composition students. From the outset, this project strived to be a grassroots, bottom-up effort so that those with the most skin in the game had a seat at the table. At every stage of this project, we tried to engage with grad students and lecturers in order to cultivate a localized assessment initiative attuned to the actual experiences of ICaP instructors.

We learned a great deal from these pilots, and we have taken that knowledge into account to make evidence-based decisions about changes to the common assignment options going forward. Given that participation was voluntary for these pilots, our findings should be interpreted with a certain amount of caution. More specifically, there are issues with sample size and randomization, which impact generalizability. Nonetheless, the data we did gather and analyzed helped to start assembling the larger picture, and we used appropriate caution when making final decisions. As the common assignment component of ICaP becomes mandatory this semester, the assignments have been updated to address the most pressing issues we encountered during these pilots.

The assessment report, linked below, contains a full write up of the conditions leading to the advent of the common assignments, the process undertaken developing them, the findings of initial assessment efforts, and the changes we made and recommend making moving forward. We share this report to maintain the grassroots spirit and transparency of a truly bottom-up assessment project we set out with. Any questions, comments, or concerns should be directed to Daniel Ernst at ernst9@purdue.edu. Thanks.

Assessment Report 2017-18

Common Assignment Assessment Questions

The information below answers most of the questions from the assessment Group-Think-Share session. We transcribed these questions from the whiteboards and added questions received via email. On Tuesday, September 4, we will share a handout that outlines the protocol for turning in your completed common assignments.

We have included some general guidelines for you and students as you begin the common assignments in the General Questions section. If you have further questions, please let Derek Sherman know via sherma11@purdue.edu.

General Questions that apply to all four common assignments:

  1. How do we structure assignment sheets and rubrics? What actually needs to be on the assignment sheets vs. rubrics?
    • To be transparent with students about the common assignment, we ask that you align your assignment sheet AND rubric to the outcomes listed in your common assignment’s instructor’s guide found here. Please use the common assignment outcomes-based rubric as a starting point in the creation of your assignment sheet and rubric. As long as your assignment sheet and rubric show students the outcomes that determine their grade, your assignment and rubric are aligned with ICaP assessment.
  2. How should students submit assignments? In what format?
    • It’s easiest for everyone if students submit assignments via Blackboard in a .doc or .docx format. Even if you don’t use Blackboard extensively, please ask students to submit their assignments there so that you can easily download and share completed work with us.  Electronic submissions on Blackboard and in a .doc or .docx format make anonymizing for assessment purposes easier.
  3. Do we need to share our rubric and assignment sheet as part of assessment?
    • We would like you to share your rubric and assignment sheet with us so that we can understand how various instructors approached these common assignments. However, we will use the outcomes-basic rubric for assessment purposes.
  4. Would students recognize a logical progression of assignments?
    • This depends on how each instructor frames their course. Since we do not want to force a standardized curriculum on English 106/108 instructors, instructors must be transparent with their students on how this assignment is scaffolded into the curriculum. If students are made aware of this progression in the syllabus and classroom instruction, students would most certainly recognize a logical progression of skills and assignments.

Research-based Essay:

  1. Can the research essay be argumentative?
    • Yes, the research-based essay can be argumentative. However, it is not required to be argumentative.
  2. What is the relevance of the research-based essay beyond 106?
    • ICaP outcomes address research writing in several ways, so this essay is meant to accompany our diverse teaching approaches. ICaP has a strong foundation of teaching research-based skills that programs on campus search for in their students, whereas other “writing” programs lack in teaching research-based skills. This assignment, therefore, was built to appeal to faculty and programs who want students with research-based skills. This assignment introduces students to academic research to better prepare them for future research-based assignments. Since Purdue is also striving for more undergraduate research, focusing on research helps students join that movement, not to mention ICaP as a whole. this also helps ICaP promote itself as an establisher of research-based skills.
  3. Should instructors use solely secondary or primary sources — or will a mix of both source types be acceptable?
    • Instructors may choose to include all primary or secondary sources, or they may choose to have a mix of both. No matter what, students should be expected to summarize, analyze, and synthesize at least five sources in total, according to the research-based essay instructor guide, in whatever format you, the instructor, expects.
  4. How students will find, use, and integrate CREDIBLE sources?
    • Credibility varies depending on the student/class’s topic and what type of research the instructor chooses/values. For secondary research, students become acquainted with Purdue Libraries’ databases to search for peer-reviewed articles, thus engrossing students in academic-based research. Primary research, however, varies depending on the research method and participant(s) chosen; therefore, guidance from the instructor should be given to determine what equates to credible research. Often, only interviews or surveys are feasible given the limited time for 106/108.
  5. How are analysis, synthesis, and summary evaluated differently?
    • These are the three skills that often accompany academic-based work; therefore, for ICaP’s assessment purposes we are looking at these skill holistically (see ICaP outcome three: “Critically think about writing and rhetoric through reading, analysis, and reflection”). By holistically, we want to see if students are able to summarize, analyze, and synthesize academic sources cohesively into one academic paper. We are not seeking to isolate these skills or assess them individually.

Rhetorical Analysis:

  1. Is there still a pretest and posttest for the rhetorical analysis essay?
    • Yes, please have students complete a pretest without any instruction. After teaching the rhetorical analysis unit, students should complete a second rhetorical analysis. Each essay should be approximately three pages in length.
  2. How do we frame “rhetorical?” Is it important to focus on terms and strategies?
    • The big issue here is that we want to make sure students go beyond analyzing solely the text and the author’s use of rhetorical/literary devices (e.g., ethos, pathos, logos, simile, metaphor, etc.). In this sense, students should focus on  analyzing how the text and the author(s) use of rhetorical/literary devices affect the context and audience. The text, consequently, should be not viewed in isolation but as working against and amongst the author, audience, and larger context.

Portfolio

  1. What are the preferable methods for draft collection?
    • Since our assessment includes evaluating students’ rough drafts and whether or not they have addressed instructors’ comments and have shown growth per ICaP Outcome Four (i.e., “Provide constructive feedback to others and incorporate feedback into their writing”), it is important that students save a copy of their rough drafts. Please ask students to save a draft in a word document (e.g., ResearchEssay_Draft) and then create a new document for their final essay (e.g., ResearchEssay_Final).
  2. What should be the portfolio’s medium?
    • We are looking into Purdue’s new portfolio tool and other options and will have an update as soon as we can. We’re hoping to offer several methods. We welcome your suggestions.

Professional Email:

  1. What is ICaP’s most important outcome for the professional email?
    • Rhetorical awareness of the situation is the most important outcome with other outcomes such as purpose and persuasion built in.
  2. How flexible in terms of when during the semester can it be assigned? The time?
    • Per our discussion at Convocation, the timing is up to the instructor. If you can fit the assignment into an already existing assignment, please go ahead and include it whenever you teach that assignment. As Alisha Karabinus noted, this assignment was originally a kickstarter for rhetorical awareness, but you may include it wherever you see fit.
  3. What are the two audiences?
    • According to the instructor guide, it states the following considering audiences:  “In this assignment, you [students] write two distinct emails for two distinct situations:
      1. In the first email, you will write to an instructor about a missed assignment or exam you would like to attempt to make up. You will create your own reasons for the make-up.
      2. In the second email, you will write to a fellow student with whom you’re working on a group project that isn’t going so well. This email is meant to help establish better deadlines, clarity, communication, teamwork, or any of the above—you can invent a situation for this email or consider a problematic group project you’ve worked on in the past to create your reasons for this email.”
  4. 200 words is a lot for an email?
    • You are welcome to suggest a lower maximum as long as the focus remains on achieving the outcomes on the rubric (i.e., addressing the audiences effectively, etc).
  5. We would like to see student samples?
    • Currently, we don’t have any samples that fit our two new audiences. Because this assignment was updated this summer, our previous samples no longer coincide with this update. The professional emails that students produce in Fall 2018 may serve as samples for the future.
  6. What are the results from the previous semester’s assessment?
  7. How is the APE exercise relevant to the professional email assignment?
    • Linda Haynes’s APE exercise is designed to be a fun, low-stakes activity which teaches  editing and efficiency. While it certainly isn’t an essential activity, many instructors have had success working it in during this unit as a way to make the editing process visible and accessible.
  8. How is ICaP quantifying positive improvement for the professional email?
    • In terms of student improvement, growth cannot be quantified because the two audiences are different scenarios and we cannot have a measurement of growth without a pretest. Because raters from the professional email’s prior assessment varied in their definition of an effective email, we are hoping that this more focused audience approach will help us establish a more consistent definition of an “effective email.” If a more consistent definition is established, we may be able to move forward in quantifying growth with a pretest and posttest in the future. However, our goal right now is to measure achievement of the outcomes and not individual student growth.
  9. How do we make a short assignment feel more substantial?
    • The email is a shorter assignment, but it is also effective in teaching rhetorical situations and catering one’s message to one’s audience. Instructors, therefore, may choose to teach this as an introduction to rhetorical awareness at the beginning of the semester or they may embed it within a current unit. Ultimately, scaffolding this assignment in with others is the best way to make this assignment feel more substantial.
  10. Is it pretest and posttest?
    • No, this is two separate emails to two separate audiences.
  11. What are the number of drafts required?
    • This is up to the instructor. Instructors may guide students through sample emails first and then accept only one draft. Or, instructors may conduct peer review sessions on multiple drafts. The choice is up to the instructor.
  12. How many class session should be dedicated to the professional email?
    • This, too, is up to the instructor. This assignment is not meant to take up a considerable amount of time; therefore, scaffolding it into a current unit may help help in establishing how many class sessions you want to dedicate to this common assignment.

Again, if you have any other questions, please let Derek Sherman know at sherma11@purdue.edu.

Fall 2018 common assignments ready

Our second generation of common assignment pilots are ready to go! Remember, all ICaP courses must include one of these five common assignments in Fall 2018 (excepting online 106, English 106-Y, which draws from a different set particular to those courses).

We’ve made a few updates based on instructor feedback and the reading and rating sessions completed in spring and summer:

  1. We are dropping the reading annotations assignment from this semester’s pilot.
  2. The literature review has been replaced by a research-based essay.
  3. The email assignment has been adjusted to require two different emails to two entirely different recipients (and has been further adjusted for content).
  4. The information literacy pre/post test assignment has not changed.

We’ve updated the common assignments page accordingly. All the new instructor guides have been adjusted to streamline the assignments for assessing program outcomes. The instructor guides also now include the rubrics used for assessment so instructors may better see how these common assignments are being used to assess the program. Each pilot guide still includes resources and ideas for how to incorporate the assignment into your course.

We hope the changes make both assignments and instructional guides more clear for instructors. At Convocation, we’ll say a bit about the assessment data which supported these changes, and dedicate time to breakout groups so instructors working on common assignments can get assistance, share materials, and make plans for the semester. As always, if you need assistance developing your syllabus or integrating a pilot assignment into your assignment sequence, reach out—the ICaP support staff is happy to help.

I want to thank Carrie Grant, Alisha Karabinus, Daniel Ernst, and Derek Sherman for their help finalizing this second generation of pilots. I’m also grateful to everyone who helped us develop the assignments or participated in the first pilot in Spring 2018: Bianca Batti, Mac Boyle, Elizabeth Geib, Patrick Hoburg, Mitchell Jacobs, Amanda Leary, Alex Long, Eugie Ruiz, Margaret Sheble, Phuong Tran, and Sharry Vahed.