Results and Analysis

From collaborativeconvergences.wiki.hss.rpi.edu
Jump to: navigation, search

Both of the instructors who integrated the wiki assignment into their undergraduate courses sought to advance their course goals by achieving specific learning objectives, namely more reflective, more recursive, more deliberative writing and, as a result, higher-quality submissions. Additionally, both of the instructors saw opportunities to improve their assessment process using the wiki. Beyond the scope of the two courses and their associated objectives, the wiki research team sought to assess the wiki's ability to enhance group cohesion by increasing trust among student team members, or, in other words, to determine the extent to which the wiki facilitated "deep collaboration." This section reviews and analyzes the results achieved in meeting three sets of goals--learning, assessment, and social networking--associated with the wiki project.

Learning Goals

Congruent with the underlying motivation for the larger research project, both classes in which the wiki research was carried out sought to foster deep collaboration on writing assignments among student team members. Specifically, the instructors sought to encourage students to write early and then to read, reflect upon, and comment upon other team members' writing. This deliberation on other student's writing was to occur “in process,” rather than waiting for a formal, final “hand off” for review and revisions after the primary writing was “finished.” Effectively, the wiki assignment was intended to force students to start their writing early enough to receive and accommodate peer and instructor feedback, encouraging both early content generation and structured on-line discussion among team members regarding that content.

The extent to which the student teams engaged the wiki assignment varied widely, both in frequency and in depth of use. In the PDI Entrepreneurship course, some teams satisfied the bare minimum requirements of the assignment, following the instructions to the letter. Other teams used the wiki regularly from the initial assignment until near-completion of their business plans. Out of seven team projects, one team reviewed the wiki’s capabilities and immediately requested permission to switch to Google Documents<ref>This team already had been using Google Documents to coordinate their collaborative writing and had invested considerable time in that system. The instructor allowed them to continue using Google Documents provided they do their best to satisfy the spirit of the wiki assignment while using Google Documents. Despite differences in the interface and its capabilities, the instructor evaluated this team's on-ling collaboration in parallel with other teams' use of the wiki.</ref>; two teams satisfied only the minimum requirements of the week-long assignment; one team continued using the wiki for one week after completion of the wiki component of the larger assignment<ref>This team discussed their collaborative process and added some initial content and notes, but then gave up using the wiki.</ref>; one team used the wiki actively for two additional weeks before switching to Google Documents; and two teams used the wiki consistently over the entire course of the project. Similarly, PDI students' depth of engagement with the wiki spanned the spectrum, from simply distributing task assignments on one end of the spectrum to relatively high-level deliberation regarding business plan content on the other end. Task-scheduling and team management postings occupied a middle spot along this spectrum.

The majority of PDI students perceived the wiki as inhibiting their progress in writing, not because they had to slow down to reflect upon others’ contributions but because the wiki assignment added an administrative “make work” step to what was already--in their assessment--a highly collaborative process. Both instructors in the course encountered teams that were struggling with the assignment, where students thought they had to project (through the wiki) a writing process that did not accurately represent how they worked. These concerns were partly allayed when the instructors clarified that the point of the assignment was not to convey a certain mode of working, but rather to document how teams were already working. The clarification helped, but, since students did not "deeply collaborate" when writing even jointly-authored documents, the requirement that students "reflect on other students' content" was, in fact, additional work. Worse, since the students had ample time in class to discuss the assignment and the work of their peers, the on-line exercise seemed unnecessary to them if the real goal was to foster collaboration.

Likewise, students in the Introduction to Engineering Design (IED) course engaged their wiki assignment to varying degrees. A few students did not participate in the exercise at all, making no contributions to their team's wiki document. Some teams failed to follow the assignment instructions and assigned one or two students to do all the uploading of content onto the wiki, regardless of who originally wrote that content. Obviously, this violated the spirit of the wiki assignment.<ref>It also made it difficult to assess individual contributions to the assignment, as discussed in the following section.</ref> Despite these problems, however, a majority of the students participated more-or-less actively in the exercise.

Although student engagement with the assignment was variable, students’ experience with the wiki, and particularly the chosen interface, was strikingly consistent: they didn’t like it. As one student from one of the most engaged PDI teams stated in a comment within her team’s wiki space, “I feel like this interface will seriously inhibit our ability to colaborate [sic] and work together since only one person can be working on a section at the same time. The interface actually encourages us to divide up the work rather than work in a more collective manner.” When asked to reflect on their experience with the wiki assignment, IED students also shared a general view of dissatisfaction with the wiki interface. Their main concern had to do with lack of flexibility in document formatting--including difficulty in handling non-text content--which dissuaded them from using the wiki to generate complete reports. It seemed, the more students knew about collaborative authoring software, the more frustrated they were with the assignment requiring them to use a tool they otherwise would not have chosen.

In both classes, the learning goals of using the wiki were moderately achieved, at best. While the majority of students "participated actively" by contributing content to their respective writing assignments, there was considerably less discussion of the content, especially among team members. Even in the PDI course where "discussion" contributions were required, the majority of students tended only to "go through the motions," either discussing project management or offering superficial comments on other students' content contributions. Only rarely did we see elaborate reflections, suggestions, or critiques of one student's contribution by another student. Even students who used the wiki consistently over the assignment period used it primarily as a document manager--a common space to hold the latest version of each section--and not as a tool for writing that was “deeply collaborative.” On the other hand, having student contributions available to the course instructors did allow earlier feedback and direct communication with students having specific problems (e.g., non-participation, improper focus, etc.). Still, at least in our course environments, where students already have considerable time to work together collaboratively on common assignments, the wiki’s contribution to deep collaboration among students was not readily apparent.

Assessment Goals

Although both course wiki assignments proved unsatisfactory in fostering deep collaboration among students, they were surprisingly effective in enabling instructor assessment of individual students. As is typical in assessing group projects, the instructors needed methods for determining individual contributions to jointly authored submissions, and the wiki proved to be a useful tool for this. Instructors of both courses found that the wiki allowed a more nuanced understanding of who contributed what than could be established only by knowing the primary student responsible for each section. The wiki provided a clear record of usage, so the instructors could easily track who were the most assiduous users as well as the nature of their contributions, allowing insight on the effectiveness and dynamics of the student teams as well as helping in assigning credit more fairly to individual team members. In other words, the wiki provided a tool for relatively easily determining who contributed what and when, which allowed assessment of both amount and quality of individual contributions to team projects. It was also nice to always have access to the latest version of the document being generated and to have access to some of the background and behind-the-scenes discussions regarding the exercise.

Beyond the practicalities of assessment--determining who contributed what and when--the wiki provided new opportunities for instructor engagement in the students' writing process. Consistent with the logic of requiring multiple draft document submissions so that the instructor can provide input to student writing in-process, the wiki allowed the instructors to provide targeted input to individual students or teams based on the most-current version of the document available at the time of assessment. This real-time assessment opportunity removes the otherwise inevitable time lag accompanying the typical assessment cycle, where hard-copy documents are submitted then the instructor(s) review the documents (usually of the entire class) and then marked-up drafts are returned to students with comments. Although the same round of activities occurs with the wikis, the feedback occurs on the latest version of the document, at the point in time when the instructor is available to assess it, and the students have access to that feedback immediately.

Social Networking Goals

Table 1

In addition to the specific learning and assessment goals within the PDI and IED courses, the larger wiki research team also sought to explore the role of wikis in mediating team dynamics and, specifically, facilitating cohesiveness among collaborators. To this end, the PDI and IED students were surveyed on their experiences with team dynamics both prior to and upon completion of the wiki exercise.

Survey 1 was sent out to students at the beginning of the semester. Our initial intention was to collect relational data about students in terms of whether or not they knew each other in person prior to class and how well they knew each other before using the wiki online. We had hoped to take this information and perform social network analysis to investigate and gain insights about how social relations among students and the structure of these social relations affected their collaboration using wiki. However, we were ultimately unable to complete this task because the wiki assignment was group-based and the student groups were too small to allow for meaningful social network analysis. Also, because it was difficult to establish a clear "base-line" for their initial networks, it was not possible to directly link the wiki with any change that might have occurred as a result of collaboration on the wiki.

What follows below are the data culled from responses to the second survey, distributed and collected after students had completed the required collaboration on the wiki. Table 1 presents the descriptive statistics of the survey results, and Table 2 presents the correlation results. Consistent with our contention, our survey results showed the following correlations:

  1. "Group cohesion" (which reflects the closeness of relationships among group members) is significantly correlated with "trust within the group" (r=.607, p=.000);
  2. "Trust within the group" is highly correlated with "valued insight brought by peers to the project" (r=.538, p=.000);
  3. "A high valuation of contributions from group members" is significantly correlated with "perceived equal contributions from group members" (r=.434, p=.003);
  4. "Perceived equal contribution from group members" is correlated with "students’ satisfaction with using wiki in the group project" (r=.409, p=.005); and
  5. "Group members kept promises by fulfilling their expected tasks" is correlated with "group members reaching consensus easily" (r=.547, p=.000).
Table 2

Together, these results suggest that facets of effective social networking, including relations of trust, reciprocity, and cooperative norms, are tightly bound with practices of deep collaboration, where interactions are satisfying and effective in achieving consensus in jointly-authored work. However, the survey results also corroborated the instructors' sense that students rarely engaged in debate or discussion on the wiki, and, instead, they tended to regularly engage in debates and discussions outside the wiki. Apparently, then, despite clear correlations between the quality of social networking and the degree to which deep collaboration was achieved, the wiki was not perceived by the students to be important in facilitating collaboration within these courses. Worse, students' comments indicate that many perceived the wiki to hinder their collaboration. In students' words:

  • "Deep collaboration has worked for previous projects (even ran our own wiki for it) but this software wasn’t useful. We collaborated in class and divided the rest."
  • "Essentially the requirements of the project and lack of decent interface and features removed any actual teamwork from the process."
  • "[The wiki was] Not too effective [considering our team had] very few group members that met frequently."
  • "The wiki delayed our work process."
  • "Enjoyed working with Google docs before the wiki was required."

The correlations listed above are consistent with teams that achieve deep collaboration, yet it seems it was not the wiki that facilitated positive collaborative experiences in our courses. Instead, we assume that the collaborative relationships were either already well established or build around face-to-face interactions, including, perhaps ironically, those face-to-face interactions required to negotiate the wiki assignment! While wikis have been shown to enhance collaboration when participants are dispersed in time and space, in our studio courses--where students were working closely together and many had already established effective strategies for collaboration--the wiki was determined to be ineffective in promoting deeper collaboration, perhaps even inhibiting deep collaboration by reducing time available for direct, face-to-face negotiation of students' writing content and process among team members.


Notes <references />


Continue reading the Challenges and Recommendations section.

Personal tools