OBSTACLES AND OPPORTUNITIES INIMPLEMENTING E-PORTFOLIOS
SCHWARTZMAN, ROY
UNIVERSITY OF NORTH CAROLINA AT GREENSBORO
COMMUNICATION STUDIES DEPARTMENT.
MOORE, CHANTEL L.
UNIVERSITY OF NORTH CAROLINA AT GREENSBORO
COMMUNICATION STUDIES DEPARTMENT.
Dr. Roy Schwartzman
Communication Studies Department
University of North Carolina at Greensboro.
Ms. Chantel L. Moore
Communication Studies Department
University of North Carolina at Greensboro.
Obstacles and Opportunities in Implementing e-Portfolios
Synopsis:
A pilot implementation of e-portfolios across several disciplines at two universities yields insights regarding how to address barriers in adopting, using, and assessing these collections of student work. Lessons from this project suggest the primary technical, instructor, student, institutional, and attitudinal factors that can impede or impel the successful use of e-portfolios in higher education.
Obstacles and Opportunities in Implementing e-Portfolios
Roy Schwartzman, Ph.D. and Chantel L. Moore
University of North Carolina at Greensboro
Electronic portfolios have been embraced widely throughout higher education as a useful ancillary or alternative to standardized tests. Offering collections of authentic student work products alongside reflections on how and why they were produced, e-portfolios can provide concrete developmental feedback to improve performance. The electronic nature of e-portfolios enables sharing for peer review, revision, or submittal to prospective employers. The Pilot Rollout of e-Portfolios (PREP) Project was conducted over the course of a full academic year to fulfill the mandate of a state grant to test the institution-wide use of e-portfolios for qualitatively assessing undergraduate student competencies in written communication and critical thinking. This pilot implementation of e-portfolios across two mid-size urban universities (18,000 students each) in 13 course sections (8 traditional classroom, 5 fully online; n = 296 students), first year through senior level, representing five departments revealed many potential advantages as well as conceptual and practical challenges in embracing e-portfolio pedagogy.
Utilizing qualitative feedback gathered from biweekly instructor focus groups over the course of a full academic year, individual instructor exit interviews, interviews with e-portfolio pedagogy experts, and student responses to open-ended survey questions, this study identified the following issues as primary focal areas affecting the pedagogical value of e-portfolios:
1. Tensions between the development function of e-portfolios to gauge student progress versus their serving as showcases that highlight the student’s best work.
2. The labor intensiveness of constructing and evaluating e-portfolios.
3. Maximizing student, instructor, and institutional buy-in to e-portfolios as objects for assessment (documenting competencies), pedagogical tools (demonstrating and reflecting).
Overall, the PREP Project revealed that many advantages attributed to e-portfolios in the scholarly literature (a) relied on theoretical assumptions inconsistently supported in actual implementation, and (b) achieved pedagogical and professional benefits realizable only with significant restructuring of instructor workloads, student support infrastructure, and institutional commitment to the centrality of e-portfolios.
Developmental vs. Showcase e-Portfolios
According to Chang, Liang, Tseng, and Tseng (2013), e-portfolios are “digitalized tools for collecting and presenting students’ learning processes and outcomes systematically (p. 187).” While serving as tools for both pedagogical and professional assessment, they have been evaluated and implemented in classroom environments for both developmental and showcase purposes. For purposes of development, e-portfolios are utilized as a pedagogical tool for assessing progress of learning competencies. However, when they are not being used as an academic assessment tool to evaluate progress of competencies, they can also be used for showcase purposes. As a showcase, the e-portfolio functions more as a professional résumé for students transitioning from academic to professional areas. Participants in the PREP Project investigated the use of student e-portfolios from varying angles, including developmental and showcase purposes.
As a pedagogical tool, e-portfolios can serve as a reasonable means of assessment for demonstrable student competencies. As evidence for achievement of competencies, e-portfolios play a crucial role in practicing strategic knowledge management (Chang et al., 2013). Beyond serving simply as repositories of work products, e-portfolios can become sites of knowledge amassment, “a common name for knowledge storage and accumulation” so that “newly acquired knowledge is integrated with existing knowledge and then stored in a form that can be comprehended easily and retrieved in the future” (Chang et al., 2014, p.188). Chang et al. (2014) describe the e-portfolio creation process as related to “data collection, acquisition, revision, organization, presentation, storage, and accumulation” (p. 187).
Trevitt, Macduff, and Steed (2013) note the tensions between conceiving of e-portfolios developmentally and conceiving of them as showcases. Developmental e-portfolios chronicle the process of learning by showing the various stages of work in progress as well as the student’s commentary on strategic choices made along the way. Showcase e-portfolios provide evidence of completed achievement: finished, polished work products that render their process of development relatively invisible. Developmental e-portfolios can serve as reflective documents for the students (Oakley, Pegrum, & Johnston, 2014) as well as pedagogical tools for instructors to diagnose areas of need in the learning process. Showcase e-portfolios are directed at broader audiences, primarily broadcasting the student’s level of achieved competencies to attract the attention of prospective employers.
Given the drastic differences between the contents, purposes, and audiences of developmental versus showcase e-portfolios, any decision to implement e-portfolios throughout an institution must incorporate a way to reconcile these divergent types of e-portfolios. Developmental e-portfolios are generally unsuitable for display to prospective employers or other non-academic constituents. Even if students gain proficiency in constructing such e-portfolios, they may still require substantial mentoring in developing a polished showcase e-portfolio as a professional presentation. Institutions cannot assume that e-portfolios evaluated to fulfill academic assessment requirements will be transferable to the professional realm.
Labor and Workload Issues
The main alteration in the project’s design and execution was the high attrition rate (83%) from the cohort at one of the participating universities, leaving only one participating instructor from that institution. Semi-structured exit interviews were conducted with the participants who withdrew. To increase candor of responses, the interviews were conducted by the PREP Project assistant rather than the principal investigator who supervised the project. These exit interviews used the following protocol, with participants verbally encouraged to add comments and expand on answers.
1. What interested you about e-portfolios initially that made you consider them as part of your teaching?
2. What factors influenced your decision not to continue with the Pilot Rollout of E-Portfolios Project?
3. If you could implement e-portfolios for you current courses, what benefits would you hope to achieve? How might e-portfolios help you teach and/or help your students to learn?
4. What factors would enable you to fully participate in the implementation of e-portfolios in your courses?
5. What recommendations would you offer about recruiting and retaining instructors to use e-portfolios in their pedagogy?
Overall, the participants who withdrew expressed no reservations regarding the nature of the PREP Project or about e-portfolios per se. The non-participants unanimously endorsed the use of e-portfolios.
Every instructor who withdrew cited time constraints as the reason. Interestingly, the respondents expressed no concerns about the time required to learn about, implement, and evaluate e-portfolios. In every case, the respondents blamed their heavy course loads, other institutional commitments, and workloads from other jobs as the primary obstacles. Each of the instructors who could not complete the PREP Project was teaching at least five sections of courses during that semester and faced multiple preparations for teaching different subjects. Some non-completing instructors were teaching as many as seven sections with additional expectations of university service. One instructor taught the equivalent of a 125 percent load for a full-time faculty member while also a full-time doctoral student at another university and parenting young children. Another instructor held two other jobs in addition to a teaching load higher than most tenure-track faculty members at that university. Every section these instructors taught was fully enrolled, with at least 20-25 students per section.
E-Portfolio Platform Selection
Based on (1) ongoing discussions within the cohort’s learning community meetings during a three month timespan, (2) webinar and in-person presentations from Taskstream, Digication, and Acclaim commercial e-portfolio platform representatives, (3) orientations to Google Sites and WordPress from the Digital Media Commons technical consultants at one of the participating universities, and (4) presentations from instructors who use the e-portfolio tools embedded in the learning management systems, the project cohort assembled a preliminary list of 27 factors to consider when selecting an e-portfolio platform. This list was formulated prior to the actual implementation of e-portfolios in coursework but after the instructors themselves tested each of the prospective platforms. These considerations (listed in no particular order) consist of:
1. Technical learning curve, intuitive interface for instructors
2. Technical support for instructors
3. Learning Management System (LMS) compatibility and integration (Blackboard, Moodle, Canvas, etc.)
4. Peer-to-peer sharing and evaluation of content
5. Interface with evaluation rubrics
6. Handles multiple drafts of content
7. Data gathering, analytics, and filtering
8. Comparative and cumulative data across courses and institutions (for norming and benchmarking)
9. Technical learning curve, intuitive interface for students
10. Technical support for students
11. Student control of access to e-portfolio (private; share with class, instructor, peers)
12. Student control of content (formats, multiple e-portfolios, etc.)
13. Student can transfer content to different platform
14. Student can continue access and construction beyond graduation
15. Can embed work products in a variety of media (audio, video, etc.)
16. Mobile and tablet functionality
17. Maximum single file size upload
18. Total size of e-portfolio storage capacity
19. Product cost per student
20. Ancillary costs (e.g., domain name, web hosting, etc.)
21. Cost for multiple user subscriptions
22. Cost for institutional license
23. Cost to train faculty
24. Demands on IT and support staff
25. OS compatibility (PC/Mac; Android/iOS)
26. Host server reliability
27. Stability of company offering platform
Content analysis of qualitative instructor feedback from the surveys and interviews revealed three criteria most crucial in selecting suitable e-portfolio platforms: usability, cost, and level of availability.
Ease of Use
Throughout all input provided by instructor participants, concerns about the user friendliness of e-portfolios—for students and instructors—dominated the feedback. By far, observations about the learning curve involved in e-portfolio adoption constituted the most frequent theme. Instructor and student participants in the PREP Project reported few difficulties with using the e-portfolio platforms or with supportive technologies (such as the technological infrastructure at either university). This relatively seamless integration of e-portfolios into coursework may stem from availability of a wide array of support services unrepresentative of typical support services throughout the University of North Carolina system. In addition to direct training and ongoing help from the commercial platform vendors (Taskstream, Digication, and Acclaim), participants engaged in robust and ongoing hands-on orientation and practice. Section 4B details those activities and resources.
Sheer ease of use, however, may not mitigate other expenses associated with training and support. Mere competency in using the e-portfolio platform is not sufficient to produce a high quality end product. Users also need to master the strategic and technological skills to present the contents of the e-portfolio in a compelling fashion. More concretely, e-portfolio users must master not simply the vessel of presentation (the e-portfolio platform), but also the acumen to use the e-portfolio as a tool for crafting and adapting their academic and pre-professional story to appropriate audiences (e.g., institutional e-portfolio evaluators, prospective employers, etc.).
Cost Considerations
All commercial e-portfolio platforms carry direct cost; users must “pay to play.” The e-portfolio market is highly competitive and brimming with vendors eager to land large, multi-institutional contracts. Two of the largest and most established vendors, Digication and Taskstream, essentially match each other on the cost per student. Adopted on the scale of the entire sixteen-campus UNC system, both vendors could use this economy of scale to provide accounts approaching $10-$14US per student per year. This cost represents an ongoing expense not currently included or anticipated in institutional budget plans. Digication adds value to its e-portfolio package by providing free faculty and alumni accounts. With Taskstream, students would need to purchase an individual account to retain access to the system after graduation. Those individual accounts currently run $42US annually. Most UNC system campuses already are Taskstream clients (due to it being the required platform for teacher credentialing in K-12 education), so possibly the long-term access rates could be negotiable. Acclaim is currently bundled with the textbook for the basic Communication Studies course in General Education. Its cost is highly scalable depending on the size of the adoption, but currently it costs approximately the same per student as the major commercial e-portfolio platforms.
Any e-portfolio platform carries substantial hidden costs on several levels. These costs may escape notice, but they may emerge over the long run as the highest ongoing expenditures. Since most of these costs are associated with labor not as visible as publicly observable teaching, they may constitute indirect costs that are not easily recoverable in any material benefits of e-portfolio adoption (Schwartzman & Carlone, 2010). The following hidden or indirect costs may be associated with a large-scale e-portfolio adoption. The list is intended as representative, not comprehensive.
- Time and labor involved in mentoring faculty in e-portfolio pedagogy
- Effort involved in norming and validating assessment rubrics
- Platform orientation and technical support (potential additions to existing IT personnel)
- Pedagogical support (tutoring, etc.) of students in e-portfolio construction
Balancing Sharability With Privacy
Understood as a gateway from college to career, e-portfolios can maximize a student’s exposure to prospective employers. In this sense, the e-portfolio provides an accessible source of proof that a student can exhibit the competencies claimed on a résumé or in an interview. An e-portfolio also could serve a “foot in the door” function, providing initial professional exposure that could trigger further interest from employers and eventual entry into a desired career field. To fulfill their professional promise, e-portfolios must be capable of sharing widely but controllably. Students need e-portfolio platforms that permit creation and circulation of multiple e-portfolios tailored to different target audiences. Targeted self-presentation implies that the totally open access characteristic of public web pages may not provide the finely grained levels of availability needed for adapting content to multiple, distinct types of viewers.
E-Portfolio Platform Selection: To Mandate or Not to Mandate?
For internal assessment and institutional research purpose, a single platform would seem desirable to allow for data collection, evaluation, and retention. A top-down mandate of a single e-portfolio platform, however, would likely generate strong resistance and seriously harm instructor buy-in to e-portfolios generally. One size, or in this context, one platform, of e-portfolio does not fit all. Any thorough and critical examination of an e-portfolio rollout must go beyond employing and studying a single platform. Such a study, while informative, primarily tests the features of that particular platform without necessarily yielding more generalizable results regarding the feasibility and utility of e-portfolio pedagogy per se.
At the universities that participated in the current study, different platforms and types of e-portfolios have tended to emerge organically as each academic program or department experiments with what best suits its needs and those of its students. If a uniform platform were selected, it would have the advantage of enabling students to easily collect work products from various courses and departments to create an aggregate e-portfolio. With multiple platforms, students face the burdensome task of re-creating their e-portfolio every time they take a course in a department that uses a different platform. (Education majors would face a mandatory transition to Taskstream anyway, as that is the state-mandated platform for teacher credentialing.)
One problem with a uniform platform, however, is that it limits digital literacy skill acquisition because students are not cross-trained in multiple e-portfolio environments. Second, each academic program has selected its preferred platform for its own reasons. An external mandate would likely be seen as an intrusion and perhaps as an infringement on academic freedom if it forces abandonment of a platform in which a program has deeply invested effort and training. A single platform might be feasible if it were chosen collaboratively, with active involvement across disciplines.
Factors Affecting Successful Implementation
Qualitative feedback from instructors of the 13 course sections included in the PREP Project was gathered in weekly learning community meetings over the course of a full academic year. In addition, each instructor was interviewed individually at the conclusion of the project. This feedback converged on several themes concerning the determinants of successful deployment of e-portfolios. The following considerations emerged as factors that could affect the feasibility and success of e-portfolios on a large scale.
Technical Factors
Technical issues pertain to any technological aspects of e-portfolio development both inside and outside of the classroom. Technological aspects encompass not only the usability and functionality of hardware and software, but the implications of implementing these tools. Aside from the issues listed above related to platform selection and operation, the following concerns arose:
1. Levels of privacy and access need to be adjustable by students to protect privacy while maximizing availability of e-portfolios to appropriate audiences.
2. Usage of e-portfolios needs to be scaled in proportion to an institution’s capacity to provide technical support. What happens if widespread e-portfolio usage creates more demand for services (e.g., technical support, e-portfolio construction tutoring, storage, bandwidth) than can be provided in order for students to fulfill course requirements?
3. Adoption of a freeware e-portfolio platform may require additional tools to enable data analytics and proper storage.
4. E-portfolio platforms must seamlessly integrate various media, including embedding video and multimedia content.
5. Platform should permit content creation and viewing across different types of electronic devices and browsers.
Instructor Factors
The realm of instructor-related issues comprises any particular enabler or constraint on more aggressive implementation of e-portfolios in teaching. In this study, those factors can include anything e-portfolio related that causes the participating instructor to change any pedagogical methodology to accommodate the needs of this project.
1. Justifying the use of e-portfolios to students (especially in course subjects where e-portfolios are not commonplace) will be necessary for student buy-in.
2. Rubrics may need to be re-normed for each course or subject matter to align with desired instructional goals and priorities.
3. Evaluative rubrics not designed natively for e-portfolios (such as the American Association of Colleges and Universities VALUE rubrics) may need to be revised or supplemented to account for issues related to e-portfolio strategy and design.
4. Instructor evaluations may suffer if students transfer frustrations with e-portfolios toward the instructor who requires them. This backlash increases in likelihood the less that e-portfolios have been institutionalized within a subject area.
5. Workload adjustments may be needed to accommodate startup learning curves and the labor-intensive nature of e-portfolio evaluations. The labor required for e-portfolios may challenge drives to maximize efficiency, as e-portfolios are far less efficient evaluation tools than standardized (especially quantitative) tests.
6. e-Portfolios need to be recognized as a desired and challenging pedagogical method whose use is acknowledged in institutional reward systems.
7. Instructors cannot reasonably be expected to provide major technical support in addition to teaching subject matter content.
Student Factors
The student dimension encompasses any experience in relation to the e-portfolio project that affected any student participant’s work products, ability to complete course assignments successfully, or satisfaction with the e-portfolio experience. This area can consist of any e-portfolio project related issue reported by students and/or faculty about students.
1. How user-friendly is the e-portfolio construction, editing, and sharing process?
2. Strategizing and creating e-portfolios requires digital literacy skills that may call for additional tutoring and practice.
3. The e-portfolio platform needs to build on tools the student already knows and uses, thereby shortening the acclimatization process.
4. The cost model for commercial e-portfolio platforms requires consideration. Should the cost be passed on to students (e.g., as an added technology fee), borne by the institution, etc.?
5. How long and under what conditions do students own the rights to their e-portfolios?
6. How long after graduation can students access and edit their e-portfolios?
7. How transferable are e-portfolios across different e-portfolio platforms?
8. What campus and e-portfolio platform resources are available for help?
9. How can users avoid plagiarism and illegal use of copyrighted material?
10. How well can the e-portfolio document the “whole life” of the student beyond conventional classroom assignments and work products?
11. Giving students the option to change platforms if they choose not to use the original platform selected. How does evaluation proceed if students select platforms that are not equally robust in their features?
12. To what extent are students involved as stakeholders in selecting e-portfolio platforms being considered for institution-wide or program-wide adoption?
Institutional Factors
The institutional dimension focuses on the roles e-portfolios could be expected to play within the educational organization. These factors might refer to the organizational culture of educational institutions per se or to the effects e-portfolios have on relationships, policies, and processes indigenous to a particular institutional environment.
1. To what extent is the institution legally liable for the contents of student e-portfolios, including copyrighted or offensive content?
2. Who owns the student’s e-portfolio? Do the same ownership policies extend beyond graduation?
3. How do e-portfolios fit within the institution’s (or each department’s) vision and mission? How do e-portfolios contribute to the fruition of the institution’s values?
4. How has each institution embedded e-portfolios into its unique culture of teaching and learning?
Notably, Kuh and colleagues (2005) found that e-portfolios provided a way for students to provide observable ways to link their work products with the values of the institution. This opportunity to rationalize their study in light of the institution’s overall mission can enable students to make more sense of their curricular and co-curricular activities, thus making learning more intentional and more likely to be completed.
Attitudinal and Structural Factors
The attitudinal realm includes any issue that emerged throughout the duration of this project that was a result of differing attitudes amongst participating faculty members with little to no prior experience implementing e-portfolios in the classroom. The structural aspects deal with the interface between e-portfolios and official policies or procedures at the institution.
1. How do instructors distinguish e-portfolios from the manifold instances of various educational fads?
2. How should e-portfolios be presented to students as something more than another exercise in assessment for its own sake?
3. How will instructors improve their teaching and improve student learning via e-portfolios?
4. Are the pedagogical benefits unique to e-portfolios, or could they be achieved through other means?
5. How can the institution cope with (faculty and student) digital divides in technology access and in technology competence (especially digital literacy deficits)?
6. How is student achievement in creating high quality e-portfolios recognized and rewarded?
7. How is the effectiveness of instructor pedagogy with e-portfolios assessed?
8. In what ways do e-portfolios “count” for the professional development of instructors? As equivalent to a course redesign? As service? As a new course?
The issue of documenting the pedagogical effectiveness of e-portfolios poses an especially important challenge. Many documented effects of e-portfolios, such as improved student perceptions of self-efficacy (Lopez-Fernandez & Rodriguez-Illera, 2009), may not align clearly with competencies traditionally measured by standardized quantitative assessments and used for institutional benchmarking.
Conclusion
Overall, the PREP Project demonstrated that broad-scale implementation of e-portfolios represents far more than simply adopting an online repository of student work products for conducting qualitative assessment. As the scope and depth of the issues overviewed in this discussion illustrate, e-portfolio adoption at the institutional or cross-institutional level may require significant alterations in pedagogical methods and institutional cultures (Buzzetto-More & Alade, 2008). The participating instructors in this project reinforced the concern expressed by Fong and colleagues (2013) that adequate resources and facilities for supporting e-portfolios must already be in place before e-portfolios are promoted and presented to students. The promotion of e-portfolios should not outrun the capacity of the technical and personnel infrastructure to enable their use. The PREP Project found that e-portfolios hold great potential for qualitative assessment, but they offer no magic bullet for solving the challenges associated with assessment. Careful consideration of the factors discussed in this report can make the different between e-portfolios becoming a superfluous burden or a genuine educational asset.
Ultimately the fate of e-portfolios on any campus relies on their use developing and broadening from the grassroots efforts of students and faculty. If e-portfolios can prove to facilitate more thorough teaching and deeper learning, they should be welcomed as an important pedagogical innovation. This pedagogical value appears to offer the greatest incentive for broad-based adoption by faculty (Cummings & Maddux, 2010). If understood as another labor-intensive, unfunded mandate from above, e-portfolios will face half-hearted reception or outright resistance. Similarly, if e-portfolios are treated as another layer of labor atop existing curricular requirements and not as an integral part of competency-based learning throughout the institution, students may be deterred from participating (Habron, 2015). Clearly a sustainable, long-term implementation of e-portfolios on a large scale requires thorough commitment and follow-through at all levels throughout an educational institution.
References
Buzzetto-More, N., & Alade, A. (2008). The pentagonal e-portfolio model for selecting, adopting, building, and implementing an e-portfolio. Journal of Information Technology Education, 7, 184-208.
Chang, C., Liang, C., Tseng, K., & Tseng, J. (2014). Using e-portfolios to elevate knowledge amassment among university students. Computers and Education, 72, 187-195.
Chang, C., Tseng, K., Liang, C., & Chen, T. (2013). Using e-portfolios to facilitate university students’ knowledge management performance: E-portfolio vs. non-portfolio. Computers and Education, 69, 216-224.
Cummings, R., & Maddux, C. D. (2010). The use of e-portfolios as a component of assessment and accreditation in higher education. In N. Buzzetto-More (Ed.), The e-portfolio paradigm: Informing, educating, assessing, and managing with e-portfolios (pp. 207-224). Santa Rosa, CA: Informing Science Press.
Fong, R. W., Lee, J. C., Chang, C., Zhang, Z., Ngai, A. C., & Lim, C. (2013). Digital teaching portfolio in higher education: Examining colleagues’ perceptions to inform implementation strategies. Internet & Higher Education, 20, 60-68.
Habron, G. (2015). Integrating eportfolios into sustainability education. International Journal of ePortfolio, 5(2), 123-134. Retrieved from http://www.theijep.com ISSN 2157-622X
Kuh, G. D., Schuh, J. H., Kinzie, J., & Whitt, E. J. (2005). Student success in college: Creating conditions that matter. Washington, DC: American Association for Higher Education.
Lopez-Fernandez, O., & Rodriguez-Illera, J. L. (2009). Investigating university students’ adaptation to a digital learner course portfolio. Computers and Education, 52(3), 608- 616.
Oakley, G., Pegrum, M., & Johnston, S. (2014). Introducing e-portfolios to pre-service teachers as tools for reflection and growth: lessons learnt. Asia-Pacific Journal of Teacher Education, 42(1), 36-50.
Schwartzman, R., & Carlone, D. (2010). Online teaching as virtual work in the new (political) economy. In S. D. Long (Ed.), Communication, relationships, and practices in virtual work (pp. 46-67). Hershey, PA: IGI Global. doi:10.4018/978-1-61520-979-8.ch003
Trevitt, C., Macduff, A., & Steed, A. (2013). [e]portfolios for learning and as evidence of achievement: Scoping the academic practice development agenda ahead. Internet and Higher Education, 20, 69-78.