Faculty teaching FWIL courses in Fall 2016 were asked to choose any assignment they felt was rich with Information Literacy, submit student work to us through Blackboard Assignments, electronic transfer, or hard copy. As the work arrived, it was logged, redacted, and filed. We later pulled a stratified random sample, ensuring strong representation from all 3 FWIL paths: TAP, ENGS1, and HCOL85. The Office of Institutional Research collaborated with us on the randomization, and later on the statistical analysis. Throughout the fall semester, the FWIL committee tested, discussed, and refined the rubric and rating sheets to be used on rating day. Eventually, we chose to rate in three ways: 1) a holistic (overall) score based on a rubric, 2) a score focused only on the depth of engagement with source material, and c) a rough quantity count for numbers of sources used.

On January 11, 2017, twenty-two raters met in the Alumni Center for an all-day rating session. We began with 2 hours of norming using a shared rubric and 3 anchor papers. We then rated 241 student papers (97 from TAP, 97 from ENGS1, and 47 from HCOL85), with each paper read by two raters. Inter-rater reliability was high, and validity indicators are also strong. During the event, raters were encouraged to note larger patterns, trends or concerns on a paper to leave behind; they were also asked to jot down insights for their own teaching on a separate sheet of paper to take home. Before the day ended, the entire group had a chance to debrief the experience and share their impressions.

See the results in the PowerPoint (PDF).

Assessment Faculty Fellow J. Dickinson provided student-led focus groups on Information Literacy across all three FWIL paths (ENGS1, HCOL85, and TAP seminars). Students positively responded to courses with one-off library sessions, and many students indicated using more sophisticated databases and search strategies as a result of their FWIL course. At the same time, most students seem to be arriving at UVM with decent textual research skills from high school, so some expressed that they expected more rigor than they experienced in their first semester here. In particular, they expected longer papers; at the same time, they also noted that their short papers are expected to be of high quality with more sophisticated thinking than in high school. There were multiple requests for more clearly written assignment prompts. Click here for the summary report (PDF).

Surveys created by the College of Arts and Sciences ad hoc committee on the First-Year Experience, as well as listening tours conducted by Deb Noel, Caroline Beer, and Libby Miles, all voiced concerns with FWIL's implementation. In some cases, layering FWIL onto an existing class was a final workload-breaking straw, while in other cases demanding conformity to a common syllabus was seen as demoralizing and demeaning. Thus, the assessment process itself created data of another sort: it elicited a hue and cry about the implementation of the FWIL requirement, as well as deep suspicion towards assessment as a concept. Throughout the assessment process, the FWIL team listened as faculty communicated strongly about their fears and resentments. Such emotional reactions are both valid and valued, and they revealed tensions that had not been discussed openly - and needed to be heard. These responses led to a highly productive re-examination of the ways in which FWIL has been implemented, with recommendations for structural changes allowing for opting in and out.

The Deans Office and Departments of the College of Arts and Sciences, as well as the 2017-2018 FWIL Committee (click here for membership) will work toward implementing the following changes.

Structural recommendations: Align FWIL with other General Education areas, in which faculty, programs, and departments opt in to teaching courses coded as a Gen Ed. The FWIL Committee will work on a system that a) ensures enough FWIL courses are being offered, b) establishes an appropriate bar for what constitutes a FWIL course, and c) provides a mechanism that does not annoy those who are clearly teaching the FWIL outcomes.
Curricular recommendations: Faculty, programs, and faculty will have the opportunity to intentionally decide if their courses should fulfill FWIL or not. We have recommended that ENGS1 become more regularized across sections to match established expectations, but that thematized (non-TAP) seminars also be considered for non-CAS students, or those who choose not to enroll in TAP seminars. More choices should mean more faculty teaching courses they want to teach, and more students taking courses they want to take.
Pedagogical recommendations: What makes the difference between a 2.5 and a 3.0? In other words, a text that is close to unanimous "proficiency" but not quite? English Graduate Student Kimberly Dean analyzed every FWIL sample rated 2.5 and below, and found the following patterns, all of which are very teachable:
Organization and flow: 2.5s across the board scored 2s on organization. They tended to have problematic transitions not only between paragraphs but within them as well. Many were overly formulaic key-hold essays.
Idea development: 2s and below generally contained good ideas that were underdeveloped and unsupported.
In-text citation: 2s and below typically did not know how to indicate or include sources within the body of their paragraphs.