 |
 |
| [back] | [Appendix] |
James R. Coliz and Lester Diamond
Project Report
We have completed the assessment project entitled "Assessment Techniques Appropriate For One-week 3 Credit courses" for which we received an Assessment Grant during Fall 1993. As a result of that project, we have determined that there are certain assessment techniques that are helpful in this format, although the one week format requires careful attention to their administration. We remain concerned about the educational viability of the one week class format; yet we find that assessments properly administered in the future will allow us to determine the success of particular courses on an individual class by class basis.
The use of one-week courses as a format for delivering three credit Courses is rapidly expanding in the School of Information Studies (IST). The one-week format has been in place for several years in our Summer Institute, but until recently this format has not been employed to satisfy the curricular needs of our regular degree granting programs. Recently, however, in an attempt to meet the needs of our part time students, we have begun to offer established full semester courses in a compressed one week format. This new emphasis on the one-week format raises two particularly critical issues. First, is it possible to offer a oneweek course with a learning experience comparable to an equivalent full semester course? Second, can we institutionalize a technique that would allow instructors to ascertain the quality of learning experiences of future one-week format courses?
In a semester course, material is presented in doses of one-fourteenth
or one
twenty eighth of a course's content. Between each dose the student
has time to digest the material just presented and to prepare
for the next dose. This digestion time allows the student to become
comfortable with the material and to identify areas with which
the student remains uncomfortable. These areas of discomfort may
then be addressed at the next session, or sooner. The student
may also develop some intuition about the material. A framework
can be incubated within the student's mind upon which the student
can "hang" new concepts as they are presented.
In a one-week course, the material is presented as through an IV. It drips continuously, and the student has no opportunity to metabolize the new material prior to the presentation of new material. In this new format the student may need extra assistance to integrate all of the material into a cohesive body of understanding.
This study proposed using assessment techniques to resolve two questions related to this course structure:
These questions can be subdivided as follows:
The actual assessment project was carried out the week of January 3 through January 8, 1994, as part of the regularly scheduled Advanced Telecommunications Management (IST763) course taught in Endicott/Binghamton, New York. This course is a required course in the M.S in Telecommunications and Network Management degree and an elective in other professional masters' level degree programs in the School of Information Studies. The course was scheduled for 8 hours per day for 6 days in one week with two Saturday follow-up courses. This format results in an increase in the normal contact hours for a 3 credit course, but it requires students to assimilate in a week the same material normally assimilated during an entire semester. Students enrolled in the course are professionals, mostly mid-level managers or engineers, employed full time in a wide variety of companies in the Southern Tier. Many of them are currently working in the telecommunications area; some are the telecommunications managers in their firms. In general, these students are known to be highly motivated and demanding.
This course not only had the drawbacks of the compressed time frame, but it was unclear to us at the start of the course to what extent student employment responsibilities would distract them from the educational process. More significantly, we were also concerned with the specific content of the course -- it is a concept-rich core course where understanding of the intellectual structure is critical to success in subsequent courses. There must be an assessment system in place to ensure that students have achieved a sufficient level of understanding each time such a course is taught in this format.
For this project, the course instructor, Assistant Professor Lester Diamond, taught the course and supervised the in-class activities. However, we also had a graduate student, Ms. Jennifer Meyer, present during the entire process to produce, administer, process, and facilitate the assessments. The Director of the M.S. Program in Telecommunications and Network Management, James Coliz, was also present to supervise the project work outside of the classroom. A wide variety of assessment types were attempted during the week, and their value to the educational process was assessed by determining the saliency of the information gained, the ability of the instrument to supply the information in a useful time frame, and the overall student reaction to the process.
During the project, assessment timing was not based upon natural breaks in subject matter, but on issues related to the flow of the course. The instructor was concerned that in an all day session, he must not allow assessments to delay the pace of the presentation. He agreed that an assessment at the beginning of each day, before lunch, and at the end of the day would be the maximum he could effectively schedule. In addition, he agreed to schedule a separate speaker assessment after each of 4 guest speakers.
In general, we found students thought that this was too many assessments and that it was not reasonable to implement that number of assessments without student resentment. Their informal comments during break showed they begrudged the time these required and we found decreasing cooperation with the process as the week progressed. At one point, as the GA entered the room with assessment forms, there were several cries of "Oh no, not another assessment." The instructor did not provide academic credit for completing the assessments and we found those that were complex or took more thanjust brief responses were ignored by most students and at times derided. The possibility of giving credit was considered, but had been rejected due to difficulties in fairly grading the process.
In some ways, the factors unique to this course also made giving assessments difficult. During the week a major blizzard moved through the area at times officially closing the roads as county road crews tried to keep emergency routes open. Students had to twice be let out early on direction of local emergency management orders, to permit them travel time home. On these days, obviously, end of class assessments were not only inappropriate from a time perspective, but also it was clear that students had other more practical concerns.
In the one-week class format, not only are the students mentally exhausted after each day's sessions, but the instructor too is generally ready for a break. The instructor, however, must take the evening to review the next day's material and ake any changes in presentation that may be necessary due to the course work that has just transpired. As a result, detailed content-based assessments, i.e., directed paraphrase, analytic memo, etc., required more effort in analysis than we felt reasonable for the instructor. It was also not acceptable to rely on the GA for this type of analysis unless the GA were well versed in the course material and pedagogy. Finally, these assessments took much more mental effort than the students were willing to expend on the project and tended to be done haphazardly or not at all. We used informal student feedback to ascertain that the lack of cooperation was lack of interest in the assessment rather than lack of understanding of the material.
This course is a new course and initially we believed
these assessments would help guide and direct the creation of
the course. We now find that there was a serious conflict between
the flexibility of the new course -- it was impossible to determine
exactly how long each segment would take -- and the ability to
have assessments given at set times of the day. Often the day
would end at points where an assessment was not appropriate, and
often places where an assessment would be valuable were in mid
session and could not be analyzed quickly enough to provide corrective
feedback. Moreover, not knowing at what point in the presentation
of material an assessment would take place meant that assessments
prepared in advance were often not appropriate and needed to be
discarded.
While it was initially proposed by us to give assessments at break
times and at the end of the day, we found these to be especially
disruptive times. Before a break, students were thinking about
the break -- not only a cup of coffee, but the opportunity to
call back to work and take care of problems that had arisen. An
assessment right before a break or at the end of the day was therefore
not taken seriowly, and was rushed through or ignored. When an
assessment was given right after a break or first thing in the
morning, it was difficult to get students to return to their seats.
Most important, assessments at these times were most resented
- as they were perceived as having "eaten into" their
break time.
We did not get assessment results that varied in any
significant manner from the informal comments that the students
provided before class, during breaks, during lunch, and after
class. This group of students was quite vocal in discussing both
how they felt the class was going and what they either did or
did not understand. We thought that perhaps the formal assessments
would provide a voice to those few students who were not quite
so vocal, However, while this may have happened, the comments
reflected in the formal assessments were no different from those
received informally. Major benefit of the one week format is that
the students do have this greatly increased informal contact with
the instructor, therefore, in order to be perceived as worthwhile,
assessments need to provide information that would not arise normally
in that environment.
Process Oriented Assessments
Process oriented assessments were not useful to the instructor in terms of restructuring the class. By the time a redirection of the teaching methods or general content could be accomplished, the course was already completed. Therefore, these did not seem to give useful information. However, in informal discussion we found that students felt this type of input was valuable and seemed to appreciate the opportunity to complete them if done only occasionally.
This activity, just before lunch on the first day of class, asked
students to state and rank three goals of theirs in taking the
class. These goals were reviewed during lunch and were discussed
with the students upon return. In that discussion, the instructor
discussed how their goals were be addressed in the course.
In general, it would have been impossible to significantly alter
the format to reflect differences in student goals and instructor
goals due to the short time format and the requirements of a core
curriculum course. In fact, this being a management course, it
would have been impossible to address the widely expressed goal
of "learning how the technology works." However, it
was felt by the instructor and students during informal discussions
that it better prepared them for what would be happening during
the week.
In general this form seemed to be especially valuable in:
This activity, given at the end of the first day, asked students to briefly answer two questions related to how they expected the class to meet their needs in the next five days.
The instructor's assessment of this project was that it served no useful purpose. Students did not seem to take it very seriously in part because there were a variety of end of class activities including picking up readings. Half the students simply said it would meet their objectives and had no concerns about it. All other responses were scattered and inconclusive. Clearly one assessment on the goals of the course would have been sufficient, from the student perspective
This memo assignment, which they took home and prepared overnight in addition to other homework, also was almost useless during the course of the class. Given at the end of the week, it required students to write a recommendation to a coworker about enrolling in a one-week class. Only 13 students (of 25) actually took the time to complete the assessment. Students informally indicated that they had better things to do with their time.
Subsequent review of the memos indicates great division of opinion on the value of these courses. About 1/3 of responding students felt it was too rushed and they were too worried about outside commitments. About 1/3 would like to take another one-week course. The remaining 1/3 were noncommittal, commenting on various strengths and weaknesses of the process. It is almost impossible to know what the remaining half of the class thought -- but they did not think it was important enough to write out the memo.
We did note that this one week class seemed to work from an educational standpoint. Students did seem focused upon the material even by the end of the 6th day. Attendance remained nearly perfect during the entire period in spite of a major blizzard. Discussion of the material was strong throughout. Note that the assessments at the end of the course were not useful in determining student engagement -- less than half of the students were cooperating with the assessments by the end of the course and this could have been due to lack of attention to the course. However, this would be inconsistent with the strong discussion evident throughout the course. Informal discussions with students showed a general lack of respect for the assessments and showed students had good comprehension of the course material. In addition, traditional evaluation instruments, including course projects, case studies, and final papers were well executed.
This assessment, given early in the week, asked them to respond overnight by explaining a main concept from the lecture for two different audiences. Twenty-two students responded. Very little useful information was collected, however. First, they were collected the following day and could not quickly be evaluated, nor could they be evaluated by the GA who was not familiar with the nuances of the material. Therefore, the next time the instructor could respond to them was the following afternoon, almost 8 class hours after the material had been discussed. It was by then inappropriate to respond during the class.
Nor did these assessmenta give us good data upon which to judge learning. Students had other homework that evening and many had also reported to work to take care of workplace priorities. Therefore, it was clear that very little thought went into theae papers, and often they were little more than random thoughts scribbled down at the last minute.
This assessment, given shortly after a discussion of a detailed procurement process, asked for the "muddiest point." This was exceptionally useful in that students tended to respond in one sentence or less. These could be skimmed almost on the spot and responded to immediately. Moreover, in this case, there was a central tendency around three different responses - includin.g no muddy point. The two remaining points could be addressed immediately.
Students and instructor agreed that these were quick, easy, and useful. They are highly recommended for use in future courses.
In a version of the applications cards, students were asked to list two real world advantages of using a particular procurement process. These also could be read quickly and evaluated by the instructor almost on the fly and therefore could lead to a quick response. In general, the data was widely diaparate but the instructor could detect interesting or unusual replies and discuss these with the class to help deepen their understanding of the topic. By giving this assessment early in the day, the instructor found most students did make some effort to comply.
Nonetheless, only about 50% of students gave insightful answers with the rest giving either obvious or frivolous replies. Even this relatively simple exercise probably involved more thought than some students were willing to invest The 50% good answers were so useful, however, that this too is recommended for future use due to its ease of use and success in fostering additional discussion.
This assessment was given the morning following a major in class analysis of a case study. Three questions were asked relating to the students' ability to summarize the main points, comment on unanswered questions, and connect the case study to other ideas in the course. Two questions asked students to comment on their degree of preparation for the assignment and their recommendation on its use in future classes. This assignment was a data nightmare. The question on what questions remained unanswered and the question on the use of the case in future classes were summarized rather quickly by the GA for feedback later that day. However, the other material had to be studied and reviewed by the instructor overnight and was not helpful during the course period.
The answers have been helpful subsequently in evaluating the success of the course. Evaluation by the instructor showed that students did recall main points with significant clustering around those points. Student attempts to connect theae the case study with other ideas in the course resulted in scattered answers, although all the answers were reasonable and innovative attempts to place material in context.
After each speaker, an assessment form was completed which combined the minute paper and applications cards approaches to the content. In addition this assessment form included scale questions on clarity and effectiveness of the speaker. Students did seem to take these seriously even though they did get a break period upon completion of the form. The value, however, was not in allowing the professor to provide feedback to the student but in reinforcing learning by forcing the students to evaluate and write down the important parts of the presentation.
In each case, there were many different answers with no discernable central tendency, making classification difficult, but they seemed in general plausible. These assessments also have the practical benefit of helping decide which speakers to invite to future classes.
Answers From
Formalized Assessments
In retrospect, the assessment project did not answer many of the questions we initially tried to solve. Each of the initial research questions will be reflected upon herein:
Q. Does the student feel overwhelmed by the presentation of a large amount of information in this particular format?
A. No assessment vehicle adequately addressed this question. We could not separate assessment burnout from information overload. We believe from quality of student analysis shown in class discussion and in graded formal assignments that students were integrating the material successfully. We also found that informal discussion indicated students felt it was NOT overwhelming them. We have no reason to believe otherwise.
Q. Do students find themselves longing for more traditional classroom experiences?
A. The analytic memo, discussed previously, related to course format tried to answer this question directly. Poor response rate probably reflected assessment burnout. However, of those responding, 1/3 of responses representing 1/6 of the entire class did seem to prefer a traditional format. Informal discussions indicated that about 1/3 of the class thought it was good to keep their mind focused on the material. Most students, however, didn't seem to care. This also may account for the poor response rate.
Q. Is the student retaining a significant portion of the material as it is being presented?
A. Formal assessments were very poor at making this determination on the fly but retrospective analysis did provide reason to believe material was being successfully retained. The RSQC2 assessment shows that students did recollect main points and were able to draw upon their understandings of the course material to find hooks into the current topic. Directed paraphrase and application cards were performed haphazardly, however, the responses that in the professor's judgement did reflect sincere effort tended to show accurate understanding of the material. Muddiest point responses, given directly after a discussion, indicated students were listening to the material. As discussed, assessment burnout and the need for students to work on "real" homework made these assessments too low a priority for students to be reliable.
Q. Can the student utilize the material in a meaningful way?
A. Again, assessments did not provide this information due to burnout. Application cards, specifically aimed at this question, were treated frivolously by half the students. The best way to assess this question was through graded projects and exercises.
Q. Is the student developing an understanding of the intellectual framework?
A. We did not find any assessment vehicle that addressed this question. We did find that this framework was in fact often a topic of discussion at the breaks. Those discussions indicated that often students were not developing their frameworks in class, but needed the more reflective break time to discuss their understandings with other students and the faculty. In fact, without carefully orchestrated break-time interaction, it did appear to the faculty that a sufficient framework would not have been developed by the students.
Q. How can results from assessments be evaluated and used to inform the presentation of course material in a very short time?
A. The assistance of a GA in keeping this type of course "on track" appears to be of utmost important from many different perspectives. However, for assessments the GA could not only prepare the assessment vehicles while the class was in-session, but was able to categorize answers on the fly for the instructor. Conventional wisdom would indicate that assessments that cannot be responded to in an immediate time frame are of little value. Virtually all content rich analytical type of assessments did not meet this criterion. To respond on the fly, the instructor must use assessments that can be read and understood very quickly in real time in front of the class.
Only a few vehicles meet this condition. However, with the presence of a GA, a additional set of vehicles (those that can be analyzed in a 60 minute time frame) become feasible.
Q. What assessment techniques are available or may be developed to test material understanding without taking much time from class? The technique should not cause stress to a student or be seen as testing for individual evaluation.
A. We found that assessments that required intellectual engagement did cause burnout. Assessments in this format must be quick and intellectually easy for the student. An alternative, of course, would be to use a challenging assessment vehicle and assign no out of class work. That was, and should be, rejected as educationally unacceptable.
Q. How can the degree of development of a knowledge framework be ascertained?
A. Only through one on one discussion with the instructor or classmates.
Q. What types of data are useful to the instructor for immediate changes in classroom presentation? Ideally, misunderstandings in the morning can be corrected and adjusted for by mid-afternoon at the latest.
A. Muddiest point assessment was most useful as the data could be quickly extracted and the response could be in real time. Thus misunderstandings that were uncovered in this fashion could be corrected within the class. The goal ranking assignment also allowed real time response that clarified student expectations of the course. All others were useful after the course was completed --but not during the time the course was in session.
Recommendations
We recommend the following use of assessments based upon this experience:
|
|
|
| [back] | [Appendix] |