E-Coaching Tip 44 (Summer 2007) How Do I Know what Learners Have Learned? Planning Assessment from the Beginning!

Advanced Tip -- Developing an Assessment Plan -- A Focus on Work Products

The usual steps in designing a course are to develop the course objectives, then the learning units and activities, and finally the assessment(s). This development strategy can sometimes weaken assessment by turning it into an evaluation of the learning activities rather than the learning objectives.

How might you strengthen your assessment plans?

Consider developing the objectives/performance goals concurrently with designing the assessment(s). You might even begin with the performance goal itself, answering the question, "What do learners want to be able to do -- with confidence -- once they have completed the course?"

You may wish to use something like the following to map your assessments directly to each objective:

Objective/Performance Goal

Assessment Type

Assessment Item

Example #1:

Learners will know the stages of implementing change in an organization.

Individual project -- with the final product being a plan with audio/presentation elements

Learners will develop a communication plan for a company's new strategic direction.

Example #2:

Learners will develop a performance improvement plan for a business unit

Group Project with a presentation and written plan

Develop a performance improvement plan for your assigned unit. The plan should include all elements which have been discussed in our weekly course discussions. And so forth:.

Now let's take a look at two issues that need special attention in online courses: the assessment of online collaboration and the security of online assessments.

Assessing Online Collaboration

Assessment of collaborative work is complex. The traditional method of providing one summative grade based on the quality of the demonstrable end-product of a collaborative group ignores the multiple aspects of a community-led activity. Relying solely on a group grade for assessment often contributes to the dissatisfaction of some learners participating in the collaborative process. Some participants feel that it can reward those who have not fully participated and punish the learners who actually developed the end-product. By adding peer assessment, a better picture of the participation of an individual can be determined.

Peer assessment is an evaluation method in which all team members assess the contributions, skills and behaviors of each individual team member as they relate to group work and project completion. Peers have a unique perspective of significant aspects of each other's behavior that occur outside the purview of the instructor. This means the individual group members can more accurately assess each team member's contributions. This is particularly true in an online learning environment where the instructor does not have the opportunity to observe group interaction outside the course site. (Note: New collaborative tools such as the live classroom tool that can archive meetings might be useful for faculty in the future.)

Here's one example of a peer assessment that could be used for an online team project:

Team Member Evaluation Form

Using your best, objective and fair professional analysis, complete the following evaluation form concerning your team member's performance on your team project. For questions 1 through 8 use the following meanings for the numbers:

1 = poor (did practically nothing)
2 = fair (did as little as possible)
3 = about right (about the right amount)
4 = good (performance was better than average)
5 = excellent (performance was super! beyond the call of duty!)

The LEVEL of effort this team member gave toward the conference was:	1 2 3 4 5
The QUALITY of that effort was:	1 2 3 4 5
How much INPUT did this team member contribute to the team discussions?	1 2 3 4 5
How much INPUT did this team member contribute to the team's plan?	1 2 3 4 5
How much INPUT did this team member contribute to the team's presentation?	1 2 3 4 5
How would you rate this team member's level of cooperation?	1 2 3 4 5
How would you rate this team member's level of time on the project?	1 2 3 4 5
The level of POSITIVE impact this team member's work had on the total project was:	1 2 3 4 5
The level of quality of the resources this team member contributed was:	1 2 3 4 5
How would you rate this team member's completion of his/her role responsibilities?	1 2 3 4 5
This team member met team deadlines	Rarely Sometimes Mostly Always
Relative to what they were supposed to do, HOW would you rate this team member's OVERALL work and contribution to this (project, conference, discussion, presentation)?	Well Somewhat Somewhat Well Below Below Above Above

Peer assessment can positively affect accountability and responsibility within groups. However, there still may be individuals who do not participate in an online project. Peer assessment cannot remedy the obstinate lurker. It is only a deterrent. But it can ensure that that lurker is not rewarded by receiving the same grade as team members who actually produced the end result.

Dealing with the Issue of Security

One characteristic of online assessment that can be particularly unnerving when assessing online is the issue of security. Reasons for concern include reduced visibility of learners who are online rather than face-to-face and the fewer interactions that may typically result. This concern needs to be addressed both from the perspective of learners as well as from that of instructors and institutions.

Learners are primarily concerned with the privacy of test answers and results being maintained, and the reliability of the assessment environment. To help address these concerns, an instructor needs to be scrupulous about following obvious security practices, such as guarding his or her password and only providing access to assessment information on a need to know basis. Another good practice is to have a back-up plan in the event the online environment fails at a critical point in the assessment. Some possibilities for an effective back-up plan might be the use of email or fax. Or the assessment might be able to be changed and rescheduled?

From an instructor's perspective, the primary concern surrounding security in the online environment is academic honesty. This is best addressed by clearly communicating expectations and policies in the course documents and introductions, such as the syllabus and assessment overviews. Some online faculty use timed assessments. These can be effective in ensuring that students do prepare ahead-of time for online exams. Some software tools prevent students from using electronic copy and paste of test material; some software prevents students from using a printer, which can also help in minimizing cheating.

However, many online faculty now minimize their dependency on security by using the automated quizzing and grading features within CMS systems for practice and review purposes, and relying on the more substantive work products such as papers, presentations and other productive work for grading purposes.

Just the Tip of the Iceberg -- Going Deeper

Another good resource on assessment is an article by Thomas Angelo (1999) titled "Doing assessment as if learning matters most." Angelo asks the question, "Why hasn't' all the talk about assessment led to more learning improvements?" One possible reason he suggests is that most assessment initiatives have been implemented without a clear vision of what "higher" or "deeper" learning is and just how the assessment strategies that we use can begin to promote such deeper or higher learning. This question of just what is deeper learning or higher learning and how to promote it could generate a lively discussion among faculty and is perhaps a discussion we should have with our students as well. A discussion forum asking our students just that question for our particular course might be very worthwhile!

Angelo provides a list of 10 guidelines for guiding our assessment practices. For now, here is just one: Engage actively -- intellectually and emotionally-- in our learners' academic work. This means, we think, that faculty get into and share the mental models that our learners are grappling with and that we "grow with them as they grow with us." This type of engagement with our learners is one way of implementing "cognitive presence" that we discussed in earlier tips.

Brainstorming and Course Reviews

Do you have questions about your assessment plan or "products" for learning? Remember that you can contact us -- Judith Boettcher or Rita-Marie Conrad at eCoach@designingforlearning.info -- for brainstorming, conversation or a more general review of your course. Course reviews help you make the most of your time while doing the most for your students!

References

Note: This e-coaching tip was derived from Assessing Learners Online by Oosterhof, Conrad & Ely, Pearson Prentice Hall, 2008.

Angelo, T. A. (1999). Doing assessment as if learning matters most. AAHE Bulletin (American Association for Higher Education) http://frontpage.uwsuper.edu/scholars/assess.pdf. Accessed May 16 2007.

Ecoaching Table of Contents