Tuesday, August 24, 2004

Successful Implementation of a Novel Internet Hybrid Surgery Curriculum An interesting article (Successful Implementation of a Novel Internet Hybrid Surgery Curriculum; The Early Phase Outcome of Thoracic Surgery Prerequisite Curriculum E-Learning Project) published in Ann Surg (2004;240: 499-509) details the use of the Internet for education. Specifically a prematriculation curriculum for thoracic surgery residents. The authors conclude that the project was "exciting and successful." The authors spent $340,000 dollars producing the content. Yes that's not a mistake - $340,000 dollars. 138 residents were randomized to 2 groups, one receiving the educational materials, the other did not. Outcomes included the results of an inservice examination and subjective evalutions by thoracic surgery faculty. The authors report a positive correlation between exam performance and use of the E-materials and a significant difference for faculty evaluations. My interpretation: Even though this generation of residents grew up with Nintendo, 32% (22/69) did not click once ie. did not use the materials in the 24 month evaluation period. Interestingly, 12 self-reported that they used the materials, but did not (that is, the authors reviewed server logs and found that that they never logged in.) Thus 12% lied. Onle 31 residents used the E-materials more than 20 times. Considering that there were 75 total segments, most of the material went unused and unviewed. These 31 were busy, however, as the average number of sessions per resident was 144. If we assume that the only good to come out of this was these 31 residents, then the authors spend close to $11,000 to educate them. Luckily this was not a government grant. Now let's examine the results. There was NO difference in the test scores between those who used the E-materials (experimental group) and the control group. NO DIFFERENCE. There was a crude positive correlation between the number of sessions used by the residents and their percent correct on the examination. Close examination of Figure 1 reveals that this correlation is due to 3 outliers, otherwise there would be NO correlation. And the Faculty assessments. Faculty were asked to rate the residents on a 5 point "Expectation" scale. Statistically significant differences were found for resident overall knowledge (3.54 control vs 3.65 experimental group) and resident application of knowledge (3.55 control vs 3.62 experimental). It appears that the authors have taken an ordinal scale and used the wrong statistics to generate this result so that this conclusion is dubious. (What is the real difference between a 3.55 vs 3.62 expectation scale? [ Since the statistical tests used were incorrect, the authors should probably review section XII: Research methodology and professional info, it was accessed only 68 times suggesting by the residents suggesting that statisitical prowess won't be the new generations thing either]. All in all, I don't really see what these authors are so excited about. 2:59:01 PMGoogle It!