Jim Kenney

Pepperdine University
Omet Cadre 7

 
 
 
 
 


 

Action Research: Cycle 3

Introduction
My Action

 

Introduction

For my previous cycle, I administered my organization's first synchronous online course. I had chosen an existing and highly successful course, "Troubleshooting Autoclaves," and adapted it for live online delivery. Since this was a first attempt, I was uncertain whether a synchronous approach would work for highly technical topics, and for students dispersed across a wide geographical area. Fortunately, the results of the course were very encouraging. In fact, the online results were slightly better then previous classroom sessions.

After reflecting on the feedback and data for cycle 2, I realized there were a number of improvements that had the potential of improving the outcome of the course. I also saw the need to gather more data to further understand the impact of using a synchronous approach. Were the results of the first session just a fluke? Would making changes to the course result in better learning? To answer these questions, I decided to make improvements and deliver a second course for cycle 3.

My first significant change was the instructor. I decided to utilize an instructor with a more dynamic delivery style. My cycle 2 instructor understood the course content very well, but seemed to lack the energy and enthusiasm necessary to motivate online students. My instructor for cycle 3, on the other hand, had a very friendly and outgoing approach. He was also eager to give the online approach a try, so he was highly motivated. Although my cycle 3 instructor was not as familiar with the technical content of the course, he had previously taught two face-to-face sessions. By using this new instructor, I reasoned the level of participation would improve. I did arrange for the same Subject Matter Expert to attend cycle 3. This would ensure all technical questions could be covered.

Based on my analysis of cycle 2, I felt the need to make the next class more learner-centered. This would improve involvement and result in greater learning. Because of the technical content, I decided the first three sessions would remain instructor led. I did add more questions to improve dialog and encourage participation. For the last two sessions, I planned on having students break into teams (one team per site) and develop strategies for several troubleshooting scenarios. The teams would then present their scenarios and strategies to the class during our online sessions. This would help me evaluate if a learner-driven activity works within a synchronous course using our WebEx tool.

I also changed the schedule for this course. I decided to shorten the number of classes from six to five sessions. I also lengthened each session from one to one-and-a-half hours. The goal was to make each session feel less hurried and allow for a better flow of content.

Another big difference was the total number of sites involved. For cycle 2, we had students from three sites. For cycle 3, we had students from five sites, representing three different time zones. The sites represented included: California, Colorado, Rhode Island, Puerto Rico, and Washington. We planned for having smaller class size, again hoping for improved participation.

Final improvements for cycle 3 involved administration. Students would be better informed prior to the course of the challenges within synchronous online learning; including the need for a quiet learning environment free from distractions. An effort was also made to send course materials sooner to each student. This would allow for a longer preparation period and time to review content.

My Action

As with the previous cycle, I handled all pre-course logistics. This included finalizing dates, revising materials, sending out course announcements, and enrolling students. This all proved to be very time-consuming for me. However, I again felt our focus was to ensure the method of instruction was successful. I did not want any distractions concerning the logistics. If I turned the logistics over to support staff, I risked having someone unfamiliar with this new approach mishandle the administration.

After I received notification of those interested in enrolling, I sent out a welcome email with links to a pre-course performance survey and pretest. In the email, I provided information on what to expect and how to prepare for online learning. I emphasized the importance of securing a quiet environment free from distractions. I recommended using headsets as a way to control the noise. Finally, I mentioned they needed self-study and course preparation time. This was a strong emphasis since self-study was lacking in cycle 2.

The performance survey had students rate their current experience level for the topic. This provided a valuable picture of the experience level of the students. The pretest tested the knowledge of the students, and would be compared with post-test results upon course completion. As soon as the students completed the performance survey and pretest, I emailed course materials and a detailed course schedule.

Each class lasted one-and-a-half hours, a was held at 8:30 AM (Pacific) each morning. We met on Tuesday, Wednesday and Friday the first week, and Tuesday and Wednesday for the second week. For the most part, the schedule worked fine. Adding a half-hour each day was a very positive change. The extra time seemed to make students feel more relaxed and yet remained short enough to maintain their involvement.

Another significant change for this cycle, was the support provided to the instructor. For my previous cycle, the instructor was located in Rhode Island, while the Subject Matter Expert and I were in California. For cycle 3, we were all located in the same room. I was able to assist the instructor in the course and WebEx administration, while the Subject Matter Expert was instantly available to answer questions and assist with content. Whereas in cycle 2 we had a lag-time in supporting the instructor, for this cycle the instructor could simply signal to us for quick assistance. We did have one day where the Subject Matter Expert was unable to attend. Since there were questions the instructor was unable to answer, he simple postponed answering the questions until the Subject Matter Expert returned.

For the first week, most of the instruction consisted of using Power Point summary slides. Polling and whiteboards were utilized to encourage participation and sharing across sites. Most of the content was highly technical, so participation was much greater among the more experienced students.

For the second week, each site was assigned troubleshooting scenarios. Students were asked to discuss the scenarios with others at their site before class, and review how they would tackle and troubleshoot the problem. During class, individuals from each site would present their scenarios and lead the class discussion regarding the best strategy. The instructor handed over the control of WebEx to the students so they could control the presentation. Because of the number of scenarios, nearly every student had an opportunity to present.

We ended the last day by taking the post-test and having the students fill-out a post-course feedback survey. In the end, twelve students completed the course. We decided, as with out previous class, not to have a passing score for post-test. All we required was participation and completion for the exams. My objective was to motivate students to learn the content, and not to focus on specific test scores.

When the course started, we had seventeen enrolled. Because of some work demands, five students had to quit the course in the second day. Although attendance for all 5 days was stressed, several students had to miss days due to emergency work requests. Because of the nature of these emergencies, we allowed for the absences.

Evidence & Evaluation

As with cycle 2, learners in the cycle 3 session demonstrated learning through a variety of methods including pre and post-tests, knowledge checks, polling questions, whiteboard activities, troubleshooting scenarios and class discussion. The course was evaluated based on my personal analysis and reflection (located at my ARP Blog), post-course feedback surveys, and post-course interviews.

The following includes photo artifacts and result summaries for cycle 3 data collection:

Photos of cycle 3: Photos were taken of our control room in California, including shots of our instructor and Subject Matter Expert. You will also find several shots of our WebEx tool in use. The layout and use of one control room proved to work extremely well and provided a very supportive environment for the instructor.

Pre & Post-test Results. The results of the pre & post-tests showed slightly lower scores then the previous online session (cycle 2), and previous classroom sessions (based on comparing the percent improvement from pre-test to post-test scores). Since the differences in results are minimal, I would argue that synchronous online delivery is comparable to face-to-face instruction in terms of knowledge learned.

The results seem to indicate that changes made to this cycle did not result in greater learning. However, a possibly reason for a lack of improvement could be the attendance. Since several students missed sessions due to emergencies, it is likely the absences negatively impacted learning. Also, as noted in the interviews, students had little time for self-study and course preparation. Although this was also an issue for cycle 2, it seemed to be even more significant for this course.

Post-Course Feedback Survey. This survey was distributed at the end of the class to gather the initial reaction of the students about the class. On the positive side, students overall seemed pleased with the course and gave high marks for the instructor.

Areas of concern include the learning environment (nothing surprising here), and the accuracy of the materials. Although it is not noted specifically, the level of difficulty of the content most likely impacted some of the ratings, especially for those new to the subject matter. We also found a few mistakes in the course materials, which also influenced the feedback.

Finally, the exams received a low ratings due to the clarity of the questions. This may also be due to the exam content being very technical.

Post-Course Interviews. The result of the interviews were very similar to cycle 2. Again, students liked the online experience. Most preferred the online delivery over traditional instruction. Students also provided positive feedback for the materials, technology and instruction.

There were several other themes that surfaced in cycle 3. First, several noted they felt more comfortable in a online session. This seemed to encourage participation from those how are more reserved in a face-to-face classroom. Also, several called out how the particularly enjoyed learning from other students representing other sites.

Once again, a primary concern was the learning environment. Some students took the course in noisy environments with interruptions. There were also those who found the content too technical. However, even those who had difficulty with content still gave the class very positive feedback. A final concern was with the lack of self-study time. Since the synchronous class time was designed for review, discussion, and questions, having student miss self-study time should negatively impact learning.

Personal analysis (ARP Blog). After each session, I provided a summary of the results from my perspective. My observations were very similar to the other feedback collected in a number of areas. I felt the technology and schedule both worked very well. I also observed how the learning environment was a problem for some. I felt attendance was a major issue. Arrangements should have been made for students to attend all sessions. To accomplish this, all emergencies should be covered by others.

I was also concerned over the preparedness of the instructor. Even though this was not identified in other feedback, I personally felt the instructor needed more subject expertise to effectively facilitate the sessions. Although the Subject Matter Expert was available most of the time providing support, I still felt the course needed an instructor who had more confidence. and could adequately respond to technical questions.

A final observation was the effectiveness of the troubleshooting scenarios. Turning the instruction over to the students seemed to energize everyone. As I noted in my blog, I only regret not adding more learner-centered activities during the first week of instruction.

Reflection

There are many factors that make a course succeed or fail. The complexity of instruction is still amazing to me. Adding an electronic component to live instruction seems to add even more complexity.

My hope in cycle 3 was to make the right changes to the curriculum and logistics that would result in at least a moderate improvement in results. Instead, I encountered many challenges, some new, some remaining from my previous cycle, but all having an impact on the results. This is definitely a journey that will take many tries to refine the process and to reach the full potential of synchronous instruction.

I learned that a synchronous instructor needs to be a very balanced trainer; at least for optimum results. An energetic and non threatening style is important. However, and equally important quality is a keen understanding of the subject matter being instructed. An effective synchronous online instructor needs both qualities to fully engage the learner.

Of course, the ideal online instructor has another area of expertise. He or she is able to multi-task and use technology in a optimum manner. My difficulty in cycle 2, but even more so in cycle 3 has been finding multi-skilled instructor. I felt I was the one keeping the class afloat. And yet, I more than anyone lacked the technical expertise for the topic. What I offered was the ability to keep the instructor on track. It was exhausting at times.

There are of course other factors for me to consider other than instructors alone. The method of instruction is also key. I ventured into a learner-centered approach that worked very well. The learners took control and work through a series of scenarios while applying their own solutions. Was this perfect? No, it had its limitations. But, the energy generated by the exercise was refreshing. Why didn't I apply a similar strategy throughout? I felt much of the initial content was very process focused, and needed an instructor as a guide. However, I know much of that knowledge could have been "discovered" by learners on their own, and with potentially better results. Is our organization ready for that degree of learner responsibility? In time, I believe it will. For now, the blended approach of instruction seems appropriate.

Another concern is with the ownership of the learning activity. The learner and supervisors must take responsibility for the schedule and preparation needed for effective learning. This is an area of control that I must give back to the student. There is little value of a student attending two-thirds of a training program. How to enforce - or motivate this ownership is challenging for me. I want students to attend a session self-motivated. I do not want to bury a student with a load of requirements. I need to learn the balance here.

If the learner need to take ownership of their learning, what about the ownership of the program? A new issue that perplexes me, not originally within the scope of this research, is how to sustain the global online program. As a leader I was able to initiate and bring this effort to completion. What happens if I leave this effort? What would have happened if I had gotten sick mid-way through the course? In both cases, I am sure the program would come to a halt. Therefore, much needs to be done to create an infrastructure that keeps the program moving forward. One greatest assets of this approach, global reach, also opens-up logistical challenges that did not exist at the site level. What is my role as a leader in this area? It is something I have thought much about. Realistically, the training organizations with our company may need to evolve further into a more global approach and structure in order to sustain live distance learning. The truth is, many of the resources needed for this type of global effort do not exist.

Despite this challenge, what is the motivation of this synchronous approach for me personally? It is simple - I see the tremendous potential of this technology. For the first time, we brought together 5 sites and student from a variety of backgrounds. It is amazing that the technology could support such a project. But it did! With the right approach, leadership, and expertise, I am convinced that this type of instruction could change the way my organization spreads knowledge across sites.

As a final thought for this cycle, I said it best in my ARP blog, "Despite these and other "bumps" in the road, I see a tremendous potential for synchronous instruction for a variety of topics." As my instructor pointed out several times, this is the future of training.


Next Steps

The next step is to develop additional synchronous topics. We already have another course being developed that would benefit from an online approach. Having additional cycles using new topics would provide very useful data, and offer a great comparison to our current topic.

In addition to the new topic, I plan to introduce a asynchronous component to the curriculum. My original plan was to add asynchronous instruction to this cycle 3, however, changes begin implemented to our intranet forced a delay until later this summer. For technical topics, I believe an asynchronous tool would really benefit our student and eventually lead to ongoing communities of practice.