Individualized feedback during simulated laparoscopic training: a mixed methods study

Introduction

Learning is a complex process and is of particular importance in medical disciplines, such as surgery, since it also concerns patient safety. Finding a balance between doctors’ training and patients’ safety demands a structured, and preferably evidence-based, approach to surgical education. Such an approach needs, among other things, to include; training of technical skills, continuous assessment of skills and also attention to non-technical skills.¹ The learning environment can facilitate or obstruct the learning process by influencing these factors.²

Education of surgeons is rooted in apprenticeship methods developed over a century ago, as originally championed by William Halsted in 1904.³ The Halsted “see one, do one, teach one” surgical training method is, however, associated with unacceptable cost from a patient safety perspective. Although this approach has fostered excellent surgeons, we can suspect that some of this excellence came at the expense of the patients. There is now ample evidence to suggest that more structured training programs can, in fact, enhance patient safety.^4-6 The teacher, not the least, may have a profound influence on the learning process.² Teachers’ skills, assessment of skills and feedback are factors that influence the learning process and therefore the education.^1,7

In gynecology and general surgery, the laparoscopic approach is nowadays the first choice in the majority of, at least benign, abdominal procedures. This development has been driven by the lesser surgical trauma inflicted on the patients, faster post-operative recovery, shorter absence from work and also the fact that it is cosmetically more pleasing than traditional, open, surgery.⁸ In laparoscopy, the learning process for novices is, however, longer than in open surgery.⁹ Particular abilities are needed in laparoscopy. For example, the ability to convert the two-dimensional video-image to a three-dimensional mental picture.¹⁰ Therefore the curriculum in laparoscopic education demands a different set of tools compared to that of traditional open surgery. Simulators are important tools for improving the technical skills needed in laparoscopic surgery.^11-13 A recent review by Thomas¹⁴, concludes that surgical simulation should be tailored to the individual needs and include feedback by tutors. Feedback has been defined as: “Specific information about the comparison between a trainees´ observed performance and a standard, given with the intent to improve the trainees´ performance”.¹⁵ Feedback ought to be informative and non-judgmental. As an integral part of the learning process, such feedback allows the student to remain on course in reaching a goal.¹⁶ Feedback can be both positive and negative, but needs to be constructive. Feedback may, however, result in decreased self-efficacy, negative self-reactions, and decreased interest in performing the task at hand.¹⁷

While teacher feedback has a long-standing tradition in surgical training, it has in most settings been unstructured and informal.¹⁸. Inasmuch as it has been evaluated, some studies suggest that instructor feedback improves the laparoscopic result.^19-22 Others, however, report that an independent approach, without feedback, in simulated laparoscopy, might actually better facilitate learning.²³ Moreover, one study suggests different perceptions of feedback among teachers and students. For example, whereas 86% of surgeons felt that feedback was given often/always immediately after the activity, only 12.5% of residents agreed.²⁴ Previous studies of feedback in laparoscopic simulator training, suggest that feedback is best given according to the individuals’ needs.^7,25. However, little is known on how individual feedback is best structured.

Several non-technical factors, like self-efficacy, also appear to contribute to learning in laparoscopy and simulated laparoscopy. Self-efficacy; one´s belief in one´s ability to succeed in specific situations,²⁶ has been reported to be relevant for learning and to correlate with simulated laparoscopic performance among surgical residents.²⁷ According to Banduras´ social cognitive theory; successful performance enhances, whereas repeated failures reduce, perceived self-efficacy.²⁸ Efficacy beliefs are essential in the development of motor skills and how well they are executed under pressure. Belief in one´s abilities to learn the patterns of the action and successfully deliver them, contribute independently to better performance.²⁸ The learning environment can also contribute to perceived self-efficacy.²⁹ Some previous studies suggest thatfeedback improves self-efficacy and provides a feeling of proficiency. However, feedback may also be a source of anxiety.⁷ The experience of flow is another non-technical factor that appears to contribute to simulated and real-life laparoscopic learning.³⁰“When in flow an individual operates at full capacity”.³¹ Experiencing flow is to move or progress smoothly with unbroken continuity, with concentration and complete absorption in what you do. Flow depends on perceived action-capacities and opportunities. The balance between capacity and opportunity is, however, delicate. If a challenge begins to exceed your skills, you eventually become anxious. On the other hand, if your skills begin to exceed the challenges in the task at hand, you relax and run the risk of becoming bored.³¹ Thus, it is reasonable to assume that enhancing self-efficacy and flow in the operating room will potentially improve the surgical result and thereby patient safety. Additionally, assessments of these factors may be useful for evaluating feedback. In light of the controversies regarding feedback described above, the aim of this study was to explore the potential benefits of individualized feedback on the surgical result, self-efficacy and flow in a standardized setting involving simulated laparoscopy. In addition, we investigated students’ perceptions of the learning process in a laparoscopic simulator with or without feedback during this learning processto collect information on how, when and how much feedback ought to be given.

Methods

This was a mixed methods study investigating the value of individualized feedback on the simulated surgical experience and on self-efficacy and flow. Additionally we explored students’ perception of learning in a simulator with and without instructor feedback.Data was collected from group interviews, questionnaires and from the simulator performance score.

Setting

This study was conducted at the Center for Advanced Medical Simulation and Training (CAMST), Karolinska University Hospital, Stockholm, Sweden and was approved by the regional research ethics committee in Stockholm in September 2012.

Participants

The study participants were medical students assigned to one of the OBGYN undergraduate courses, year 5, at Karolinska University Hospital, Stockholm, Sweden, during the fall of 2012. Participation in the study was voluntary. Laparoscopic simulator training was not part of the participants’ academic curriculum, but was offered as an extra-curricular activity within the study. We chose medical students since they had no experience of laparoscopy and were novices to simulated laparoscopy. We obtained oral and written informed consent from each participant. Simulator results, interviews and questionnaires were de-identified upon transcription.

Sixteen medical students (eight females) were included in the study. The median age was 25 years (range 24-29). Ten of the students had one prior experience of simulator training, more than one year prior to participation in the present study. Gender, age, left-handedness and prior experience of computer games and simulator training were distributed equally in the two groups.

Study design

The participants were randomized into two groups (simulator training with or without instructor feedback), balanced for gender. A mixed study method³² was used in order to evaluate learning with feedback by quantitative and qualitative methods. The surgical instruments pathways were measured in the simulated operation. We chose to measure this particular parameter since the complication risk is dependent on handling of the instruments, the length of the pathway of the instruments and the duration of surgery.^{11, 33, 34} Furthermore, a visual analogue scale and a validated questionnaire evaluated students’ experience of flow and self-efficacy.^35,36 Finally students’ views on the learning process in the surgical simulator with and without feedback were captured by group interviews.

All participants had individual discussions with an instructor, (the same instructor for every participant), regarding a patient case with an extra-uterine pregnancy in preparation for the simulated surgery. The training session in the simulator started with the instructor performing a laparoscopic tubal occlusion, to demonstrate the simulator and its instruments. Each student then performed one laparoscopic tubal occlusion to get familiar with the simulator. The instructor continued the demonstration with a salpingectomy, the actual patient case. Each student subsequently performed three laparoscopic salpingectomies. The students randomized to the feedback group received feedback throughout the procedures. Feedback was given continuously, individualized by reinforcing and correcting each step. Feedback was also provided after each procedure, based on the parameter scores of each student. The control group performed the salpingectomies without feedback. Hence, they did not receive the feedback given by the simulator by the parameter scores or feedback given by the instructor. Feedback was evaluated by the parameter score in the simulator, by self-efficacy scores, by flow scores and furthermore by focus group interviews.

Quantitative data: simulated laparoscopy

For laparoscopic gynecological simulation we used the LapSimGyn^® VR simulator (Surgical Science AB, Gothenburg, Sweden). The software runs on a Xeon-1.8 GHz processor using the Microsoft Windows^® XP (Microsoft Corporation, Redmond, WA, USA) operating system. The computer is equipped with 256 MB internal RAM, a NVIDIA Quadro2 EX graphics card (NVIDIA Corporation, Santa Clara, CA, USA), a 15-inch monitor and a virtual laparoscopic interface manufactured by Immersion Inc. (San Jose, CA, USA). Construct validity (to differentiate an expert from a novice)³⁷and transfer of skills to the operating theatre have been established for LapSim^®.^{5,12, 38} Simulator tasks performed were “tubal occlusion” and “salpingectomy”. In the salpingectomy, the subject uses both hands to manipulate the instruments. When the task starts the system presents an ectopic pregnancy in the right fallopian tube. The students were instructed to use the bipolar grasper first, followed by the diathermy scissors. All students performed the surgery from the lateral/right side of the ectopic pregnancy, i.e. operating primarily with their right instrument, while the left instrument was practically static. The simulator calculates scores for e.g. instrument pathways, bleeding amount, tissue damage. The simulator parameter analyzed in the present study was right instrument path length (m).

Self-efficacy and flow

Self-efficacy was self-assessed before and after the simulator training session using a 3-item questionnaire where each item was rated on a 7-grade Likert-type scale.³⁶

1. "I am confident that I can handle the most difficult parts of the tasks during the simulator training/ future simulator training."

2. "I will comprehend the meaning of the simulator tasks/ future simulator tasks."

3. "I am confident I will succeed in future simulator tasks."

The flow experience was self-rated immediately after the session. Two components of flow, Pleasure (4 items) and Concentration (4 items), were measured using 0 to 10 visual analogue scales. An overall flow experience index was calculated according to the results from Ghani and Despande,³⁵ by the mean value of the Pleasure and Concentration components. The maximum score possible to achieve was 20.

Analysis and statistics of quantitative data

Data analyses were carried out using JMP^® version 9.0.0 (SAS Institute Inc., Cary, NC, USA) for Mac OS X^® version 10.5.7 (Apple Inc., Cupertino, CA, USA). Mann-Whitney U was used to evaluate differences in self-efficacy, flow scores and surgical performance between two groups. Wilcoxon matched paired signed rank sum test was used to compare self-efficacy scores before and after training and surgical performance between sessions. A p-value <0.05 was considered statistically significant.

Qualitative data: focus group interview

Focus group interviews³⁹ were used to capture students’ perceptions of simulator training, the learning process and more specifically on feedback. The interviews were carried out with each group individually. For logistical reasons the focus sessions were held one to seven weeks following training. This delay was equal between the two groups. Participants in the focus interviews were asked to describe their experience in general terms. We asked open questions in order to stimulate reflection. Examples of questions that were asked were; “What did you experience?”, “What did you learn?”, “Did you get enough support?”, "Did the situation appear authentic?” The interviews were recorded and later transcribed.

Analysis of qualitative data

Interview transcripts were analyzed using an inductive, thematic approach.⁴⁰ Transcripts were read several times in order to get to know the material. We went beyond the manifest content that was explicitly stated and interpreted the latent content.⁴¹ Data was coded inductively and subsequently grouped into themes.²²

Results

The primary findings in the study include a significantly shorter instrument pathway in the group receiving feedback. Based on the interviews, we conclude that the whole study group felt that feedback was essential. In contrast to the control group, the feedback group was more concentrated on the task compared to the control group and captured the whole procedure. The results are presented below with the quantitative simulator results first, followed by the results of the analyses of flow and self-efficacy and finally the results from the qualitative analyses, the students’ perceptions subdivided by different themes.

Findings from the quantitative analysis: simulator performance

Each student performed three simulated laparoscopic salpingectomies.Right instrument path length was significantly shorter in the feedback group, in the first (median 5.2 m; IQR: 4.5-6.5) and last (median 3.9 m; IQR: 3.3-4.9) sessions as compared to the control group (median 7.7 m; IQR: 5.8-12.3) and (median 5.9 m; IQR: 5.0-8.1), p<0.05 (Figure 1). In addition, all students improved their performances significantly between the first and last session. The simulator performance was similar between male and female students.

Figure 1

Simulator performance comparing groups. *p ＜ 0.05

Flow and self-efficacy

Flow experience had a median score of 16.4 (range11.0-18.8) in the whole sample. No differences in flow experience or self-efficacy scores were observed between the feedback and control groups. Overall self-efficacy scores improved with training in both groups (Figure 2). Males scored significantly higher than females on self-efficacy item 3:”I am confident I will succeed in future simulator tasks”, both before and after training (Figure 3).

Figure 2

Self-efficacy score before and after simulator training. *p ＜ 0.05

Figure 3

Self-efficacy score, item 3; “I am confident I will succeed in future simulator tasks”, comparing males and females before and after training. *p ＜ 0.05

Findings from the qualitative analysis: students’ perceptions of the simulator training

Five to six students participated in each of the focus groups. Both male and female students were represented at each session. Three themes were identified: feedback, learning and experience/simulator training (Table 1).  The themes are described in more detail below.

Feedback

This theme focused on the students’ perceptions of feedback. The control group felt they lacked support and feedback. “I definitely lacked support” (Male 1, control group). “You need an observer (Female 1, Male 2, control group)”.

The feedback group appreciated advice on how to think and how to do. They felt that each session in the simulator went smoother thanks to the feedback. “I received feedback when I performed well and when and how I needed to improve my performance. Each subsequent session went smoother” (Female 1, feedback group). In addition to from spontaneous feedback from the instructor, they also received feedback by asking questions. Some students expressed that they became passive and awaited initiative or acknowledgement from the instructor. Others expressed that the instructor was too passive and did not provide enough feedback. These students also asked for a more structured feedback.

Learning

What students learned from the simulator training constituted the second theme. The control group expressed that they appreciated an opportunity to handle the laparoscopic instruments, “I got a feeling for how to handle the instruments” (Female 1, Male 3, control group). They appreciated the procedure illustrations beforehand. They expressed that learning would have been enhanced if simulator training had been part of the mandatory curriculum and the exam.

The feedback group expressed that they better understood the surgical procedure. “I came to understand the full procedure” (Male 1, feedback group). They felt that the training fitted well within the context of the medical curriculum.

Experience of simulator training

The third theme focused on the experience of the simulator training. The control group expressed that simulator training was fun and even exciting. They became engaged in the activity, although not as much as if the training session had been part of the mandatory course. One student expressed; “I had to stop the bleeding!” (Male 4, control group)and another “It was exciting!” (Male 1, control group)Compared to the control group, the feedback group seemed to be more concentrated, but also more anxious. Several students expressed that they felt “a pressure to do well” (Female 2, male 2, feedback group) and to “do the right thing for the patient.” (Male 3, feedback group)

Both groups perceived the session in the simulator as authentic, particularly the feedback group. The connection to the patient case was important for both groups.

Both groups expressed high ambitions to succeed and also the importance of support and being seen. The students in the feedback group expressed a more focused attitude towards the training, were more anxious not to succeed and experienced the situation as more realistic compared to the control group.

Discussion

We here report beneficial effects of feedback on the performance in simulated laparoscopy. Our findings suggest that simulator training enhances self-efficacy and flow experiences. Participants in the control group expressed that they had more fun, whereas the feedback group was more concentrated on the task. Both groups had high ambitions to succeed and also expressed the importance of getting support and being acknowledged.

Simulator performance

The feedback group used a significantly shorter path (i.e. moved the instruments less) compared to the control group, indicating a positive effect of feedback. Both groups, however, improved, regardless of feedback, which is in line with previous studies^37,42 indicating an independent training effect of the simulator. The performances of the individuals in the feedback group were also less varied compared to the individuals in the control (Figure 1). This indicates that feedback is especially important for those students/trainees experiencing difficulties, perhaps in part due to a lesser visuospatial ability.⁴³ Previous studies suggest that such individuals benefit the most from simulator training.³⁰ This finding underlines the importance of individualizing feedback.

Self-efficacy

In the current study, self-efficacy scores were similar between the two groups. Interestingly, these scores were significantly higher for males than for females concerning the item “I am confident I will succeed in future simulator tasks” both before and after training. Males did, however, not perform better than females in the simulator. Gender appears to affect the learning process and therefore needs to be considered during the construction of a curriculum.⁴⁴ Self-efficacy scores increased for the whole group following training in the simulator, supporting the notion that simulator training has a positive effect on self-efficacy, as previously reported.³⁰

Flow

Both groups in this study scored relatively high on overall flow, suggesting that they approved of the simulator and its authenticity, regardless of feedback. The feedback group stated a high level of concentration, whereas the independent group stated more enjoyment. According to Ghani and Deshpande,³⁵ concentration and enjoyment are key characteristics of flow, which can explain why no differences in the overall flow experience were observed between the groups and the lack of correlation with simulator performance. Whereas feedback appeared to enhance concentration and focus, despite a higher anxiety level, the independent approach seemed to enhance enjoyment and creativity.

Learning

The students in this study stated that the authenticity of the situation, i.e. face validity (the simulator’s ability to resemblance the real tool/ situation), was important for the learning process. Introducing the simulator session in the context of a patient case seemed to be an important factor.

The feedback group stated that they experienced a more realistic situation as compared to the control group. The feedback group was focused on the patient and the whole procedure, whereas the control group focused on procedural details, one at a time.

Feedback

The instructor provided guidance and feedback to the feedback group during the simulator procedure, depending on the individual needs of each student. The parameter scores in the simulator also provided support for feedback on the performance. The control group received neither. The aim was to use constructive individualized feedback that is feasible and that can easily be transferred to the operating room. In a few cases, help was provided to allow the students of the control group to move on with the procedure, however, no feedback was given concerning these students techniques. This approach was chosen to somewhat mimic traditional surgical training and also to contrast the structured feedback.

The control group stated that they suffered from lack of feedback. On the other hand, some of the students that received feedback felt that it made them more passive or, as stated in some cases, that the instructor was too passive. The students in the present study wished for predetermined structures for both the instructor and the training. This is in line with the conclusions drawn by Evans et al. who report that the learning objectives need to be clear.⁴⁵

Experience of simulator training

Students in the feedback group expressed more anxiety as compared to the control group. They felt more pressure to succeed. This can be explained by the “mentor effect”. Paskins et al. reported an increased anxiety level among medical students, during training with a whole body simulator, under teacher supervision.⁴⁶ The students in the feedback group expressed an understanding of the whole procedure, whereas the control group was more focused on details, like instrument handling. The students in the control group felt more freedom to explore the instruments and their own capacity as well as the surgical approach. This indicates that students can benefit from an independent approach to learning. However, you cannot ignore the fact that technical skills improved more in the group that received feedback in this study. A mixture of feedback and independence might be the golden choice as indicated by Strandygaard et al.²⁵ Moreover, feedback and being supported were factors that all the students felt facilitated learning.

Limitations of the study

A limitation of this study is the relatively small study sample. The sample consisted of medical students and not residents in OBGYN, which is the main target-group for training and learning in gynecological laparoscopy. The reason for choosing novices was the advantage of their inexperience of simulators and lack of prior experience of laparoscopy, thus providing a homogenous test group. Another limitation is the way feedback was given. We did not use a validated tool, like for example Non-technical Skills for surgeons.⁴⁷ A recent Danish study evaluated this tool adjusted for Danish circumstances in relations to feedback style.⁴⁸ They found the tool useful, albeit still in need of improvement, for mentorship to reach its full potential. In the present study, we wanted the students’ demands to guide the feedback style and to objectify it by using the parameter score in the simulator. We encourage future studies to use a tool/ checklist, but still allow the feedback to be adapted to the individual student and the specific situation.

Another limitation of the present study is the fact that we investigate the effect of feedback during simulated and not real laparoscopy and the findings can therefore not be generalized to the clinical situation. The simulator, however, provided a standardized patient scenario and procedure, which is impossible to obtain in a clinical setting. Different approaches to feedback in the operating room therefore need to be explored among residents in OBGYN.

Conclusions

This study suggests that feedback enhances simulated laparoscopic learning. Students appear to benefit from support if it is carefully matched to their different needs. Compared to previous work, this study adds value as it highlights and meets the individual needs of feedback and explores students’ views on feedback while learning laparoscopy. The next step is to optimize the amount and form of feedback and to transfer standardized and individualized feedback from the simulated setting to the operating room.

Acknowledgments

The authors thank Mr. Armand Sadeghi, Department for Biomedical Engineering and Center for Advanced Medical Simulation and Training, at Karolinska University Hospital, Stockholm, Sweden for excellent technical assistance. This study was supported by research grants from the Stockholm County Council (FoUU and ALF).

Conflict of Interest

The authors declare that they have no conflict of interest.

References

1. Grantcharov TP and Reznick RK. Teaching procedural skills. BMJ. 2008; 336: 1129-1131.
   Full Text PubMed
2. Biggs J. What the Student Does: teaching for enhanced learning. Journal of the Higher Education Research and Development Society of Australia. 1999;18(1):57-75
3. Cameron JL. William Stewart Halsted. Our surgical heritage. Ann Surg. 1997; 225: 445-458.
   PubMed
4. Scott DJ and Dunnington GL. The new ACS/APDS Skills Curriculum: moving the learning curve out of the operating room. J Gastrointest Surg. 2008; 12: 213-221.
   Full Text PubMed
5. Wakefield M. To err is human: An Institute of Medicine report. Professional Psychology: Research and Practice. 2000; 31: 243-244.
   Full Text
6. Hafford ML, Van Sickle KR, Willis RE, Wilson TD, Gugliuzza K, Brown KM and Scott DJ. Ensuring competency: are fundamentals of laparoscopic surgery training and certification necessary for practicing surgeons and operating room personnel? Surg Endosc. 2013; 27: 118-126.
   Full Text PubMed
7. Zigmont JJ, Kappus LJ and Sudikoff SN. Theoretical foundations of learning through simulation. Semin Perinatol. 2011; 35: 47-51.
   Full Text PubMed
8. Bongard F, Dubecz S and Klein S. Complications of therapeutic laparoscopy. Curr Probl Surg. 1994; 31: 857-924.
   Full Text PubMed
9. Aggarwal R, Moorthy K and Darzi A. Laparoscopic skills training and assessment. Br J Surg. 2004; 91: 1549-1558.
   Full Text PubMed
10. Hanna GB, Shimi SM and Cuschieri A. Randomised study of influence of two-dimensional versus three-dimensional imaging on performance of laparoscopic cholecystectomy. Lancet. 1998; 351: 248-251.
    Full Text PubMed
11. Seymour NE, Gallagher AG, Roman SA, O'Brien MK, Bansal VK, Andersen DK and Satava RM. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002; 236: 458-463.
    Full Text PubMed
12. Larsen CR, Soerensen JL, Grantcharov TP, Dalsgaard T, Schouenborg L, Ottosen C, Schroeder TV and Ottesen BS. Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. BMJ. 2009; 338: 1802.
    Full Text PubMed
13. Ahlborg L, Hedman L, Nisell H, Felländer-Tsai L and Enochsson L. Simulator training and non-technical factors improve laparoscopic performance among OBGYN trainees. Acta Obstet Gynecol Scand. 2013; 92: 1194-1201.
    Full Text PubMed
14. Thomas MP. The role of simulation in the development of technical competence during surgical training: a literature review. Int J Med Educ. 2013; 4: 48-58.
    Full Text
15. van de Ridder JM, Stokking KM, McGaghie WC and ten Cate OT. What is feedback in clinical education? Med Educ. 2008; 42: 189-197.
    Full Text PubMed
16. Ende J. Feedback in clinical medical education. JAMA. 1983; 250: 777-781.
    Full Text PubMed
17. Wulf G, Shea C and Lewthwaite R. Motor skill learning and performance: a review of influential factors. Med Educ. 2010; 44: 75-84.
    Full Text PubMed
18. Patel VM, Warren O, Ahmed K, Humphris P, Abbasi S, Ashrafian H, Darzi A and Athanasiou T. How can we build mentorship in surgeons of the future? ANZ J Surg. 2011; 81: 418-424.
    Full Text PubMed
19. Choy I, Fecso A, Kwong J, Jackson T and Okrainec A. Remote evaluation of laparoscopic performance using the global operative assessment of laparoscopic skills. Surg Endosc. 2013; 27: 378-383.
    Full Text PubMed
20. Grantcharov TP, Schulze S and Kristiansen VB. The impact of objective assessment and constructive feedback on improvement of laparoscopic performance in the operating room. Surg Endosc. 2007; 21: 2240-2243.
    Full Text PubMed
21. Mahmood T and Darzi A. The learning curve for a colonoscopy simulator in the absence of any feedback: no feedback, no learning. Surg Endosc. 2004; 18: 1224-1230.
    Full Text PubMed
22. Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D and Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005; 27: 10-28.
    Full Text PubMed
23. Snyder CW, Vandromme MJ, Tyra SL and Hawn MT. Proficiency-based laparoscopic and endoscopic training with virtual reality simulators: a comparison of proctored and independent approaches. J Surg Educ. 2009; 66: 201-207.
    Full Text PubMed
24. Sender Liberman A, Liberman M, Steinert Y, McLeod P and Meterissian S. Surgery residents and attending surgeons have different perceptions of feedback. Med Teach. 2005; 27: 470-472.
    Full Text PubMed
25. Strandbygaard J, Bjerrum F, Maagaard M, Winkel P, Larsen CR, Ringsted C, Gluud C, Grantcharov T, Ottesen B and Sorensen JL. Instructor feedback versus no instructor feedback on performance in a laparoscopic virtual reality simulator: a randomized trial. Ann Surg. 2013; 257: 839-844.
    Full Text PubMed
26. Kneebone R. Evaluating clinical simulations for learning procedural skills: a theory-based approach. Acad Med. 2005; 80: 549-553.
    Full Text PubMed
27. Maschuw K, Schlosser K, Kupietz E, Slater EP, Weyers P and Hassan I. Do soft skills predict surgical performance?: a single-center randomized controlled trial evaluating predictors of skill acquisition in virtual reality laparoscopy. World J Surg. 2011; 35: 480-486.
    Full Text PubMed
28. Bandura A. Self-Efficacy: The Exercise of Control. New York, NY: WH Freeman and Company, Worth Publishers; 1997.
29. Bandura A and Locke EA. Negative self-efficacy and goal effects revisited. J Appl Psychol. 2003; 88: 87-99.
    Full Text PubMed
30. Ahlborg L, Hedman L, Rasmussen C, Fellander-Tsai L, Enochsson L. Non-technical factors influence laparoscopic simulator performance among OBGYN residents. Gynecological Surgery. 2012;9(4):415-420.
31. Nakamura J, Csikszentmihalyi M. The concept of flow. In: Handbook of positive psychology. Oxford: Oxford university press; 2002.
32. Schifferdecker KE and Reed VA. Using mixed methods research in medical education: basic guidelines for researchers. Med Educ. 2009; 43: 637-644.
    Full Text PubMed
33. Minekus JP, Rozing PM, Nelissen R and Dankelman J. Identifying error pathways during elbow and knee replacements. Clin Orthop Relat Res. 2005; 121-127.
    PubMed
34. Scholes RL, Browning L, Sztendur EM and Denehy L. Duration of anaesthesia, type of surgery, respiratory co-morbidity, predicted VO2max and smoking predict postoperative pulmonary complications after upper abdominal surgery: an observational study. Aust J Physiother. 2009; 55: 191-198.
    Full Text PubMed
35. Ghani JA and Deshpande SP. Task Characteristics and the Experience of Optimal Flow in Human—Computer Interaction. The Journal of Psychology. 1994; 128: 381-391.
    Full Text
36. Pintrich PR, Smith DAF, Garcia T and Mckeachie WJ. Reliability and Predictive Validity of the Motivated Strategies for Learning Questionnaire (Mslq). Educational and Psychological Measurement. 1993; 53: 801-813.
    Full Text
37. Larsen CR, Grantcharov T, Aggarwal R, Tully A, Sørensen JL, Dalsgaard T and Ottesen B. Objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator. Surg Endosc. 2006; 20: 1460-1466.
    Full Text PubMed
38. Cosman PH, Hugh TJ, Shearer CJ, Merrett ND, Biankin AV, Cartmill JA. Skills acquired on virtual reality laparoscopic simulators transfer into the operating room in a blinded, randomised, controlled trial. Stud Health Technol Inform. 2007;125:76-81.
39. Patton MQ. Qualitative Research. Encyclopaedia of Statistics in Behavioral Science. Thousand Oaks: SAGE Publications; 2005.
40. Elo S and Kyngäs H. The qualitative content analysis process. J Adv Nurs. 2008; 62: 107-115.
    Full Text PubMed
41. Graneheim UH and Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004; 24: 105-112.
    Full Text PubMed
42. Ahlborg L, Hedman L, Murkes D, Westman B, Kjellin A, Felländer-Tsai L and Enochsson L. Visuospatial ability correlates with performance in simulated gynecological laparoscopy. Eur J Obstet Gynecol Reprod Biol. 2011; 157: 73-77.
    Full Text PubMed
43. Van Herzeele I, O'Donoghue KG, Aggarwal R, Vermassen F, Darzi A and Cheshire NJ. Visuospatial and psychomotor aptitude predicts endovascular performance of inexperienced individuals on a virtual reality simulator. J Vasc Surg. 2010; 51: 1035-1042.
    Full Text PubMed
44. Schlickum M, Felländer-Tsai L, Hedman L and Henningsohn L. Endourological simulator performance in female but not male medical students predicts written examination results in basic surgery. Scand J Urol. 2013; 47: 38-42.
    Full Text PubMed
45. Evans LV, Gusberg RJ. Creating a surgery clerkship in a changing environment: reality, simulation, and the rules of engagement. Yale J Biol Med. 2012;85(1):143-152.
46. Paskins Z and Peile E. Final year medical students' views on simulation-based teaching: a comparison with the Best Evidence Medical Education Systematic Review. Med Teach. 2010; 32: 569-577.
    Full Text PubMed
47. Crossley J, Marriott J, Purdie H and Beard JD. Prospective observational study to evaluate NOTSS (Non-Technical Skills for Surgeons) for assessing trainees' non-technical performance in the operating theatre. Br J Surg. 2011; 98: 1010-1020.
    Full Text PubMed
48. Spanager L, Dieckmann P, Beier-Holgersen R, Rosenberg J and Oestergaard D. Comprehensive feedback on trainee surgeons' non-technical skills. Int J Med Educ. 2015; 6: 4-11.
    Full Text PubMed