ORIGINAL RESEARCH | https://doi.org/10.5005/jp-journals-10024-2700 |
A Comparative Study of the Examination Pattern of Panoramic Radiographs Using Eye-tracking Software
1,5Department of Orthodontics, Faculty of Dentistry, King Abdulaziz University, Jeddah, Kingdom of Saudi Arabia
2Department of Oral Diagnostic Sciences, Faculty of Dentistry, King Abdulaziz University, Jeddah, Kingdom of Saudi Arabia
3Department of Oral and Maxillofacial Surgery, Faculty of Dentistry, King Abdulaziz University, Jeddah, Kingdom of Saudi Arabia
4Department of Diagnostic Sciences, Faculty of Dentistry, Kuwait University, Kuwait City, Kuwait
Corresponding Author: Fatima M Jadu, Department of Oral Diagnostic Sciences, Faculty of Dentistry, King Abdulaziz University, Jeddah, Kingdom of Saudi Arabia, Phone: +966 254 00000, e-mail: fjadu@kau.edu.sa
How to cite this article Bahaziq A, Jadu FM, Jan AM, et al. A Comparative Study of the Examination Pattern of Panoramic Radiographs Using Eye-tracking Software. J Contemp Dent Pract 2019;20(12):1436–1441.
Source of support: This project was funded by the Deanship of Scientific Research (DSR) at King Abdulaziz University, Jeddah, under grant no. (D1435-387-165). The authors, therefore, acknowledge with thanks the DSR for the financial support.
Conflict of interest: None
ABSTRACT
Aim: To analyze the differences between novice and expert orthodontists examining panoramic radiographs with incidental findings of varying difficulty. The null hypothesis states no difference in the examination pattern of panoramic radiographs between novice and expert orthodontists.
Materials and methods: Expert and novice orthodontic observers were asked to examine six panoramic radiographs with incidental findings of varying difficulty. The eye-tracking software recorded and analyzed their eye movements during the examination. After examining each radiograph, the observers were asked questions about the radiograph. All these collected data were analyzed to compare the performance of the two sets of observers.
Results: The total number of observers was 72 in the novice group and 64 in the expert group. There was only one statistically significant finding between the two groups of observers, which was the end time. Expert orthodontists recorded longer panoramic radiograph examination times.
Conclusion: There is no significant difference in the abilities of expert and novice orthodontists to examine panoramic radiographs and identify incidental findings.
Clinical significance: Orthodontists might benefit from additional education and continued training in examining and reporting radiographs commonly utilized by this specific group of dental specialists, such as panoramic and cephalometric radiographs.
Keywords: Area of interest, Eye-tracking, Orthodontists, Panoramic radiograph.
INTRODUCTION
Panoramic radiographs are an important imaging tool frequently used by dentists and dental specialists for diagnosis, treatment planning, and follow-up of dental patients. Two-dimensional (2D) radiographs are commonly used by dentists and dental specialists to simultaneously assess both jaws and their dentition, in addition to surrounding structures, such as the cervical spine, maxillary sinus, and temporomandibular joints (TMJ).1 Orthodontists routinely utilize panoramic radiographs to examine the developing dentition, the position and orientation of crowns and roots, and to assess the presence or absence of impactions or pathological conditions.
Interpreting panoramic radiographs is challenging for several reasons. First, panoramic radiographs are inherently 2D and thus suffer from overlapping structures. Second, they suffer from distortion (unequal magnification) and ghost images that further complicate the problem of overlapping structures. Third, panoramic radiographs provide images of a complex anatomic area and observers must be well versed in not only the anatomy of the facial and dental structures but also the radiographic anatomy and their specific appearance in panoramic radiographs.
Incidental findings are common in panoramic radiographs because of the large area that they cover. Most of these findings are insignificant, but some are serious findings that require immediate action.2 The available literature is bereft of data on the efficiency of dentists and dental specialists, including orthodontists, in detecting incidental findings in panoramic radiographs. Some have even commented on the likelihood that orthodontists might be distracted by points of interest in a radiograph and miss significant incidental findings.3 The available research has also linked the efficiency of evaluating radiographs with clinical experience.4 The more years of experience, the less time the clinician requires to examine a radiograph and the more accurate the interpretation of the radiographic findings.5,6
Eye-tracking records and analyzes the gaze pattern of observers during a given task. The data then provide clues as to which areas of the screen are examined and thus which areas have contributed to the decision-making process. Eye-tracking has been used extensively in medical education to understand how experts perform.7 This has been especially helpful in the field of radiology and the interpretation of chest radiographs and mammograms.8–10 More recently, orthodontists have used eye-tracking technology to determine if there is any difference between groups of observers for the esthetic component in the different malocclusions.11
In this study, we examined the differences in the gaze pattern between novice and expert orthodontists examining panoramic radiographs with incidental findings of varying difficulty. The null hypothesis states no difference in the examination pattern of panoramic radiographs between novice and expert orthodontists.
MATERIALS AND METHODS
The research ethics board at the Faculty of Dentistry of King Abdulaziz University, where the study took place, approved the study protocol. An informed consent was obtained from the observers. No informed consent was required for the use of anonymized panoramic radiographs.
Observers
Sample size calculations, based on a 95% confidence interval (CI), estimated the need for a total of 100 observers divided into two main groups, expert and novice orthodontists. The original number of observers was 146. However, 10 participants were excluded, 6 because they moved their heads or the screen during the experiment and 4 because the calibration step failed despite repeated attempts. The actual sample size was 136 orthodontists, 72 in the novice group and 64 in the expert group. Experts were qualified orthodontists (graduates of a clinical postgraduate orthodontic program regardless of the years of experience), and the novices were orthodontic residents (currently enrolled in a clinical orthodontic postgraduate program regardless of the level of training). Demographic data of the observers are presented in Table 1. Most of the recruitment process and data collection took place during an orthodontic conference that was held at the same academic institution. This facilitated the availability of a significant number of novice and expert orthodontists who were willing to participate in the study. No incentive was offered and all subjects were free to stop the experiment and withdraw from the study at any stage.
Panoramic Radiographs
Six digital panoramic radiographs were selected from the library of radiographs made for orthodontic purposes. The radiographs were chosen based on the presence of a single incidental finding. The inclusion criteria also included panoramic radiographs of diagnostic quality in terms of density and contrast with no positioning errors. Then, an independent panel of one senior radiologist and one senior orthodontist reviewed the radiographs and divided them into three categories based on the difficulty of the incidental findings. The panoramic radiographs were divided into those with easy (defined as obvious), intermediate, and difficult (defined as subtle) incidental findings (Table 2). Three radiographs are shown demonstrating an easy, an intermediate, and a difficult incidental finding (Fig. 1).
| Variable | Novice orthodontist | Expert orthodontists |
|---|---|---|
| Gender | ||
| Female | 20 | 36 |
| Male | 52 | 28 |
| Level of experience | ||
| Junior | 33 | 26 |
| Senior | 39 | 38 |
Junior novice defined as resident in first and second years of orthodontic clinical training program Senior novice defined as resident in last 3 years of orthodontic clinical training program Junior expert defined as having 5 or less years of orthodontic clinical experience Senior expert defined as having more than 5 years of orthodontic clinical experience
Data Collection
The eye-tracking camera was mounted to the base of a 15.6-inch laptop computer screen (Latitude E6530, Dell Corporation, Round Rock, TX, USA). The screen has a display resolution of 1,600 by 900 pixels. The software used to track the observer’s gaze pattern was RED-m SMI (Sensomotoric Instruments, Teltow, Germany). Care was taken to conduct the experiments in a dimly lit room with no distractions in the observers’ field of view. The laptop was set so that the screen was perpendicular to the floor. All the steps of the experiment and the instructions were explained thoroughly to each observer before commencement. The observers were then asked to sit comfortably at an eye-to-screen distance of between 50 cm and 75 cm. Next, the observer’s head position was adjusted so that the eye-tracking camera picked up the position of the eyes consistently. Instructions were given to the observers to stabilize the head position for the duration of the experiment (Fig. 2). Finally, a calibration step was undertaken using a five-point calibration image to ensure proper alignment of the gaze pattern relative to the image. The accuracy (gaze deviation) was set at 0.5°, which, at an eye-to-screen distance of 65 cm, corresponds to a spatial resolution of 5 mm.
The panoramic radiographs were displayed in a random order. There were no time constraints, and the observers were allowed to examine the radiographs for as long as they wished. The average time spent by the observers during the experiment lasted between 10 minutes and 15 minutes. The principal investigator administered all the experiments. The observers did all the data entry, but the principal investigator was available close by to provide help and technical assistance. First, demographic data about the observer were collected, including gender and level of experience or years of experience. Next, the observers were allowed to examine one panoramic radiograph at a time. Following each radiographic image, the observer was required to answer a number of questions regarding the presence of an incidental finding (or not), its location, disease category, and radiographic interpretation.
| Incidental findings difficulty level | Location | Interpretation |
|---|---|---|
| Easy | Maxillary midline | Compound odontoma |
| Right posterior maxilla | Dentigerous cyst | |
| Intermediate | Left condyle | Trauma |
| Left condylar head | Benign tumor | |
| Difficult | Left posterior mandible | Fibrous dysplasia |
| Right maxillary sinus | Malignant tumor |
Figs 1A to C: Three panoramic radiographs demonstrating three incidental findings of varying difficulty: (A) Easy incidental findings (dentigerous cyst in the right maxilla associated with tooth 15); (B) Intermediate incidental finding (benign tumor involving the left mandibular condyle); (C) Difficult incidental finding (fibrous dysplasia involving the left mandible)
Fig. 2: Photograph of expert orthodontists while conducting the examination
The eye-tracking data that were collected for each panoramic radiograph included end time and scan path length for the entire radiograph. The definition of these terms can be found in Table 3.
The data collected for each observer by the eye-tracking device were automatically saved from the experiment software to the analysis software BeGaze (Sensomotoric Instruments, Teltow, Germany). The BeGaze software was then able to outline the incidental findings in the radiographic images as areas of interests (AOI), and with further analysis, it was possible to produce the data outlined in Table 4.
After completion of the experiment, data collected from all observers were exported to Microsoft Excel for Mac 2011 version 14.0.0, where they were grouped according to the observer category. STATA Version 13.0 (StataCorp, College Station, Texas, USA) was then used for the statistical analysis, where the significance level was set at p < 0.05 and the confidence level at 95%. The interpretive skills of the observers (i.e., their answers to the questions that followed each radiograph) were reported as frequencies and percentages. The association between these responses and the level of experience of the observers was further tested using the Chi-square test and the Fisher’s exact test. The detection skills of the observers (i.e., the results of the eye-tracking data) were reported as medians and interquartile ranges (IQR). The association between the eye-tracking data results and the level of observer experience was tested using the nonparametric Kruskal–Wallis test.
| KPI | Unit | Description |
|---|---|---|
| End time | Seconds | Time needed to examine the entire radiograph |
| Scan path length | Centimeters | Successive fixations produce a scan path along the area of the radiograph. A scan path length is a measure of that path |
| KPI | Unit | Description |
|---|---|---|
| Entry time | Seconds | Mean time from the start of the image examination to first identification of the AOI |
| Number of fixations | Count | The number of fixations in the AOI |
| Fixation duration | Seconds | Total time spent in AOI |
| Number of revisits | Count | The number of revisits to the AOI once the observer leaves the AOI the first time |
RESULTS
A total of 136 observers took part in this study. Participants were divided into the subgroups, as shown in Table 5.
There were no statistically significant differences between the two groups of observers in terms of detecting the findings regardless of their difficulty level and in terms of placing them in the correct disease category. However, two trends were noted. First, observers—generally and regardless of their level of experience—were better at perceiving the findings than at interpreting them. Second, both the detection and interpretation skills declined as the difficulty of the incidental findings increased (Fig. 3).
Eye-tracking Results for Entire Panoramic Radiograph
There was a statistically significant difference between the expert and novice groups for the end time (p = 0.01). The observers in the expert group recorded longer examination times (Fig. 4); this was especially true for panoramic radiographs with findings considered intermediate in difficulty. The observers in the expert group also recorded the longest scan path length. However, this was not significantly different from the scan path length recorded by the observers in the novice group (Fig. 5).
| Level of experience | n | Percent |
|---|---|---|
| Novices | 72 | 53 |
| Experts | 64 | 47 |
| Total | 136 | 100 |
n, number of observers
Eye-tracking Results for AOI in Each Panoramic Radiograph
There were no statistically significant differences among the novice and expert groups with regards to the AOI results, regardless of the level of difficulty of the incidental findings. However, several trends were noted. The entry time to the AOI was longer for observers from the expert group (Fig. 6). The number of fixations was almost equal for both groups of observers (Fig. 7). However, the fixation duration was longer for observers in the expert group (Fig. 8). The number of revisits was also greater for observers in the expert group (Fig. 9).
These results represent our analysis of the differences between novice and expert orthodontists in terms of their ability to examine panoramic radiographs and detect and interpret incidental findings of variable difficulty.
DISCUSSION
The goal of the present study was to understand the differences in examination patterns of panoramic radiographs between novice and expert orthodontists utilizing the eye-tracking technology. The eye tracking technology relies on the proprietary hardware and software used in experiments to register eye movement.12 Analysis of the eye movement data has found many applications in fields such as cognitive psychology, marketing, and medical imaging.12 In medical imaging, eye-tracking studies have improved our understanding of how radiologists detect and interpret positive findings.12 This in turn has improved the speed and accuracy with which diagnostic images are interpreted.12
One of the first eye-tracking studies published was by Carmody et al.13 They studied the detection of lung nodules on chest radiographs and found that the scanning strategies of radiologists significantly affected the nodule detection rate.13 Several eye-tracking studies of chest radiographs followed and confirmed that experts were faster and more accurate than novice observers.14 Mammography eye-tracking studies also demonstrated that experts were faster at detecting lesions and required less time to make a decision regarding the lesion.14,15 The results of these studies have significantly influenced the teaching methods and training strategies used to examine diagnostic images, yet the literature is bereft of similar studies that investigate diagnostic images commonly used by dentist and dental specialists.
Fig. 3: Detection and interpretation results of both types of observers according to the difficulty of the incidental findings
Fig. 4: Results of the panoramic examination time for the novice and experts observers in relation to the difficulty level of the incidental findings
Fig. 5: Results of the scan path length for the novice and expert groups in relation to the difficulty level of the incidental findings
Fig. 6: Results of the area of interest entry time for the novice and expert groups in relation to the difficulty level of the incidental findings
Fig. 7: Results of the area of interest fixation counts for the novice and expert groups in relation to the difficulty level of the incidental findings
Fig. 8: Results of the area of interest fixation duration for the novice and expert groups in relation to the difficulty level of the incidental findings
Fig. 9: Results of area of interest revisit counts for the novice and expert groups in relation to the difficulty level of the incidental findings
The results demonstrated very little difference between the two groups of observers. This is contrary to the findings of Turgeon et al., who compared the search pattern of panoramic radiographs between dental students and oral and maxillofacial radiologists.6 Their results demonstrated a significant difference between the two groups of observers, with radiologists being faster at detecting incidental findings.6 We suspect that our results were different because the two groups of observers in the current study were closely related, unlike the vastly different observers in the Turgeon et al. study.
Only one statistically significant result was recorded for expert observers, and that is their significantly longer panoramic radiograph examination time (end time). This result is contrary to Wood, who compared experts and novices seeking radiological training.16 Novices in the Wood study were found to comment on and search for absent findings rather than focusing on significant positive findings.16 This explains why the experts in the same study were faster and required less time to examine and report radiographs.16 One explanation for the longer expert end time in the current study is the long scan path length noted for these observers. Another explanation is our hypothesis that the expert observers are usually more focused on management and treatment and might start the mental process of treatment planning while examining the radiograph, thus requiring a longer end time.
Wood divides the radiographic examination process into three timed phases, 30% for the search, 25% for recognition, and 45% for decision making.16 In the current study, a marked difference was noted between the ability to detect an incidental finding and the ability to interpret it. Both groups of observers had better detection abilities than interpretation skills. Wood reported that observers usually perform poorly in the interpretation of a radiograph if patterns of examination are not ordinary.16 Perhaps the observers in the current study did not follow a systematic approach when examining the panoramic radiographs. This finding leads us to support the conclusions of Kuhlberg and Norton that orthodontists require adequate education and continued training in examining and reporting radiographs.3 These authors also concluded that orthodontists should collaborate with oral and maxillofacial radiologists and seek their consultation when possible.3
Limitations of this study include that the level of experience or years of experience were not correlated with the results. Future studies with larger sample sizes should take this into account. It would also be interesting to conduct this same experiment to compare between orthodontists and radiologists. Investigating the examination pattern of other diagnostic images such as cephalometric radiographs and cone beam CT images would also be a valuable future direction.
CONCLUSION
There is no significant difference in the abilities of expert and novice orthodontists to examine panoramic radiographs and identify incidental findings of varying difficulty.
CLINICAL SIGNIFICANCE
Orthodontists might benefit from additional education and continued training in examining and reporting radiographs commonly utilized by this specific group of dental specialists, such as panoramic and cephalometric radiographs.
REFERENCES
1. Jadu FMJA. Incidental findings on panoramic radiographs for pre-extraction assessment of third molars. Asian J Sci Tech 2015;6(6):1539–1543.
2. Bondemark L, Jeppsson M, Lindh-Ingildsen L, et al. Incidental findings of pathology and abnormality in pretreatment orthodontic panoramic radiographs. Angle Orthod 2006;76(1):98–102.
3. Kuhlberg AJ, Norton LA. Pathologic findings in orthodontic radiographic images. Am J Orthod Dentofacial Orthop 2003;123(2):182–184. DOI: 10.1067/mod.2003.4.
4. Grunheid THD, Miller JR, Larson BE. Visual scan behavior of new and experienced clinicians assessing panoramic radiographs. World Fed Orthod 2013;2(1):e3–e7. DOI: 10.1016/j.ejwf.2012.12.002.
5. Dreiseitl S, Pivec M, Binder M. Differences in examination characteristics of pigmented skin lesions: results of an eye tracking study. Artif Intell Med 2012;54(3):201–205. DOI: 10.1016/j.artmed.2011.11.004.
6. Turgeon DP, Lam EW. Influence of experience and training on dental students’ examination performance regarding panoramic images. J Dent Educ 2016;80(2):156–164.
7. Kundel HL, La Follette PS,Jr. Visual search patterns and experience with radiological images. Radiology 1972;103(3):523–528. DOI: 10.1148/103.3.523.
8. Nodine CF, Kundel HL, Mello-Thoms C, et al. How experience and training influence mammography expertise. Acad Radiol 1999;6(10):575–585. DOI: 10.1016/S1076-6332(99)80252-9.
9. Kundel HL, Nodine CF, Conant EF, et al. Holistic component of image perception in mammogram interpretation: gaze-tracking study. Radiology 2007;242(2):396–402. DOI: 10.1148/radiol.2422051997.
10. Kundel HL. Visual search and lung nodule detection on CT scans. Radiology 2015;274(1):14–16. DOI: 10.1148/radiol.14142247.
11. Leandro de Oliveira W, Saga AY, Ignacio SA, et al. Comparative study between different groups of esthetic component of the index of orthodontic treatment need and eye tracking. Am J Orthod Dentofacial Orthop 2019;156(1):67–74. DOI: 10.1016/j.ajodo.2018.07.026.
12. Harezlak K, Kasprowski P. Application of eye tracking in medicine: a survey, research issues and challenges. Comput Med Imaging Graph 2018;65:176–190. DOI: 10.1016/j.compmedimag.2017.04.006.
13. Carmody DP, Nodine CF, Kundel HL. Finding lung nodules with and without comparative visual scanning. Percept Psychophys 1981;29(6):594–598. DOI: 10.3758/BF03207377.
14. Kelly BS, Rainford LA, Darcy SP, et al. The Development of expertise in radiology: in chest radiograph interpretation, “Expert” search pattern may predate “Expert” levels of diagnostic accuracy for pneumothorax identification. Radiology 2016;280(1):252–260. DOI: 10.1148/radiol.2016150409.
15. Kundel HL, Nodine CF, Krupinski EA, et al. Using gaze-tracking data and mixture distribution analysis to support a holistic model for the detection of cancers on mammograms. Acad Radiol 2008;15(7):881–886. DOI: 10.1016/j.acra.2008.01.023.
16. Wood BP. Visual expertise. Radiology 1999;211(1):1–3. DOI: 10.1148/radiology.211.1.r99ap431.
________________________
© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by-nc/4.0/), which permits unrestricted use, distribution, and non-commercial reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.