ORIGINAL RESEARCH

Application of Computer-Aided
Diagnosis on Breast Ultrasonography
Evaluation of Diagnostic Performances and Agreement
of Radiologists According to Different Levels of Experience
Eun Cho, MD, Eun-Kyung Kim, MD, PhD , Mi Kyung Song, MS, Jung Hyun Yoon, MD, PhD

Objectives—To investigate the feasibility of a computer-aided diagnosis (CAD) sys-

tem (S-Detect; Samsung Medison, Co, Ltd, Seoul, Korea) for breast ultrasonography
(US), according to radiologists with various degrees of experience in breast imaging.
Methods—From December 2015 to March 2016, 119 breast masses in 116 women
were included. Ultrasonographic images of the breast masses were retrospectively
reviewed and analyzed by 2 radiologists specializing in breast imaging (7 and 1 years
of experience, respectively) and S-Detect, according to the individual ultrasono-
graphic descriptors from the fifth edition of the American College of Radiology
Breast Imaging Reporting and Data System and final assessment categories. Diag-
nostic performance and the interobserver agreement among the radiologists and S-
Detect was calculated and compared.
Results—Among the 119 breast masses, 54 (45.4%) were malignant, and 65
(54.6%) were benign. Compared to the radiologists, S-Detect had higher specificity
(90.8% compared to 49.2% and 55.4%) and positive predictive value (PPV; 86.7%
compared to 60.7% and 63.8%) (all P < .001). Both radiologists had significantly
improved specificity, PPV, and accuracy when using S-Detect compared to US alone
Received November 22, 2016, from the Depart- (all P < .001). The area under the receiving operating characteristic curves of the
ment of Radiology, Severance Hospital, Re- both radiologists did not show a significant improvement when applying S-Detect
search Institute of Radiological Science (E.C., compared to US alone (all P > .05). Moderate agreement was seen in final assess-
E.-K.K., J.H.Y.), and Department of Research ments made by each radiologist and S-Detect (j 5 0.40 and 0.45, respectively).
Affairs, Biostatistics Collaboration Unit
(M.K.S.), Yonsei University College of Medicine, Conclusions—S-Detect is a clinically feasible diagnostic tool that can be used to
Seoul, Korea. Manuscript accepted for improve the specificity, PPV, and accuracy of breast US, with a moderate degree of
publication April 10, 2017. agreement in final assessments, regardless of the experience of the radiologist.
This study was supported by the research
fund of Samsung Medison Co, Ltd. Key Words—breast; Breast Imaging Reporting and Data System; computer-
Address correspondence to Jung Hyun aided diagnosis; neoplasm; ultrasonography; ultrasound equipment and
Yoon, MD, PhD, Department of Radiology, products
Severance Hospital, Research Institute of
Radiological Science, Yonsei University College
of Medicine, 50-1 Yonse-ro, Seodaemun-gu,
Seoul 03-722, Korea.

O
ver the recent decades, breast ultrasonography (US) has
E-mail: [email protected] been widely available in everyday practice. Based on that
trend, the American College of Radiology Breast Imaging
Abbreviations
AUC, area under the receiver operating Reporting and Data System (BI-RADS) was released and recently
characteristic curve; BI-RADS, Breast Imag- updated.1 The ultrasonographic BI-RADS provides a sonographic lexi-
ing Reporting and Data System; CAD, com- con and final assessment categories, which have excellent diagnostic
puter-aided diagnosis; NPV, negative performances when applied to breast US in the differential diagnosis
predictive value; PPV, positive predictive
value; US, ultrasonography of breast masses and suggest standardized management for patients.2,3
In spite of the excellent performances reported by using the
doi:10.1002/jum.14332 ultrasonographic BI-RADS, the final assessments made for breast

C 2017 by the American Institute of Ultrasound in Medicine | J Ultrasound Med 2017; 00:00–00 | 0278-4297 | www.aium.org
V
Cho et al—Application of Computer-Aided Diagnosis on Breast Ultrasonography

masses by different performers are known to vary signifi- excision, or a combination thereof were included in this
cantly,4,5 mostly because of the multiple BI-RADS ultra- study. Among them, 7 women were excluded for non-
sonographic descriptors used for describing breast mass lesions seen on breast US (n 5 4), since BI-RADS
lesions and the subjectiveness of US. To increase the descriptors are hard to apply on these lesions, and lack
diagnostic accuracy of breast US, several additional ultra- of grayscale images of the targeted breast mass stored in
sonographic techniques have been developed and the US machine for S-Detect analysis (n 5 3). Only elas-
applied in clinical practice, such as elastography,6 auto- tograms and Doppler US images were stored for the 3
mated breast US,7,8 and computer-aided diagnosis cases with images not fit for the S-Detect analysis.
(CAD) systems.9–13 Among these additional imaging Finally, 119 breast masses in 116 women were included
modalities, CAD systems have been reported to enable in this study.
efficient interpretation, in which consistent improved
accuracy can be expected.14,15 S-Detect (Samsung Medi- Ultrasonographic Examinations
son, Co, Ltd, Seoul, Korea), a recently developed com- Ultrasonographic examinations were performed with a
mercially available imaging analysis program for breast 3–12-MHz linear transducer (RS80A with Prestige;
US, analyzes the morphologic features of a target breast Samsung Medison, Co, Ltd). Two staff radiologists
mass according to the BI-RADS ultrasonographic (J.H.Y. and E.-K.K.) with 7 and 19 years of experience in
descriptors and provides a final assessment, which is breast imaging, respectively, were involved in image
based on analysis of the ultrasonographic features.16 acquisition. Bilateral whole-breast ultrasonographic
This program may be useful in clinical practice, since it examinations were routinely performed, according to
can be used to provide a second opinion on tumor char- scanning protocols including representative transverse
acterization and guidance in deciding on patient treat- and longitudinal images of the breast masses with and
ment. In a recent study analyzing the diagnostic without calipers used for size measurements. The same
performances of S-Detect and a dedicated breast radiol- radiologist who had performed breast US proceeded
ogist,16 S-Detect had equivalent diagnostic performance with US-guided core needle biopsy. Both radiologists
and moderate agreement with the radiologist. The use- had access to the mammographic images obtained
fulness of CAD for breast US can differ according to the before US, images of ultrasonographic examinations per-
level of experience of the radiologist, as reported in a formed in the past, and medical records containing clini-
recent study,17 but the previous study used in-house cal information of the patients.
software that is not commercially available, which had its Image Review and Application of S-Detect
limitations in that the analytic results could differ accord- Ultrasonographic images of the 119 breast masses were
ing to different algorithms or models used for analysis. retrospectively reviewed and documented for data analy-
Therefore, the purpose of this study was to investigate sis by 2 breast radiologists (J.H.Y., radiologist 1; and
the clinical feasibility of S-Detect when applied to breast E.C., radiologist 2) with 7 and 1 years of breast imaging
US by comparing the diagnostic performances and experience, respectively. All observers were blinded to
agreement between S-Detect and radiologists with vari- the clinical information and pathologic results of each
ous degrees of experience in breast imaging. mass during image review. Breast masses were analyzed
according to the descriptors and final assessment catego-
Materials and Methods ries used in the fifth edition of the BI-RADS and final
assessments (Table 1).1 Calcifications were not analyzed
This study was a prospective study, and it was approved because of the limited data analytic ability of S-Detect
by the Institutional Review Board of Severance Hospital. for calcifications.16 The radiologists individually chose
Informed consent was obtained from all patients for the most appropriate term to describe each lesion for
study inclusion. each descriptor and made a final assessment accordingly.
The ultrasonographic BI-RADS final assessment catego-
Patients ries were made as follows: category 2 (benign), 3 (prob-
From December 2015 to March 2016, 126 breast ably benign), 4a (low suspicion of malignancy), 4b
masses in 123 consecutive women who were scheduled (moderate suspicion of malignancy), 4c (high suspicion
for breast US-guided core needle biopsy, surgical of malignancy), and 5 (highly suggestive of malignancy).

2 J Ultrasound Med 2017; 00:00–00

 Cho et al—Application of Computer-Aided Diagnosis on Breast Ultrasonography

After image review by the radiologists, S-Detect was Data and Statistical Analyses
applied to the same image the radiologists used for gray- Histopathologic results from US-guided core needle
scale ultrasonographic feature analysis. A region of inter- biopsy, vacuum-assisted excision, or surgery were
est was either automatically or manually drawn along the regarded as the reference standards. Patients with a
border of the mass by S-Detect. Analytic results of S- diagnosis of high-risk lesions, such as atypical ductal
Detect, including the BI-RADS lesion descriptors and hyperplasia, intraductal papilloma, radial scar, and lobu-
final assessments, were immediately displayed and re- lar carcinoma in situ, were recommended to have further
corded for data analysis (Figure 1). After being informed surgical treatment, for which the final pathologic diagno-
of the final assessment made by S-Detect, each radiolog- sis was considered. Final histopathological diagnoses of
ist gave a final assessment for each breast mass, integrat- high-risk lesions were considered benign for the statisti-
ing the analytic results of S-Detect. cal analysis.
Because the final assessment data from S-Detect
were in a dichotomized form, “possibly benign” and
“possibly malignant,” final assessments from the radiol-
Table 1. Ultrasonographic Descriptors Used for Image Analysis ogists based on the BI-RADS were also divided into 2
Characteristic Descriptors groups for statistical analysis: positive assessments con-
Shape Oval, round, irregular
sisted of categories 4a to 5, and negative assessments
Orientation Parallel, nonparallel consisted of categories 2 and 3. Diagnostic performances
Margin Circumscribed, not circumscribed: of the individual radiologists, S-Detect, and the integra-
indistinct, angular, microlobulated, tion of S-Detect with each radiologist were analyzed and
speculated
Echo pattern Anechoic, hyperechoic, complex
compared, including sensitivity, specificity, positive pre-
cystic and solid, hypoechoic, dictive value (PPV), negative predictive value (NPV),
isoechoic, heterogeneous and accuracy. The generalized estimating equation
Posterior acoustic No posterior features, method was used for comparing the diagnostic perform-
features enhancement, shadowing,
combined
ances between radiologists and S-Detect. The area under
the receiver operating characteristic curve (AUC) was

Figure 1. Representative image of setting the region of interest for S-Detect analysis in a 52-year-old woman with a diagnosis of cancer in her left
breast. The region of interest was set automatically along the margin of the breast mass for analysis (green line). After the region of interest was
set, the ultrasonographic features were automatically analyzed by S-Detect, and a final assessment was automatically visualized.

J Ultrasound Med 2017; 00:00–00 3

Cho et al—Application of Computer-Aided Diagnosis on Breast Ultrasonography

obtained and compared by the Delong method. j statis- excision, and 103 (86.6%) by US-guided core needle
tics were calculated to assess the agreement for ultraso- biopsy. Fourteen (11.8%) lesions were diagnosed as
nographic descriptors and final assessments among the benign according to ultrasonographic features without
radiologists, S-Detect, and the integration of each radiol- additional biopsy, among which 8 were considered
ogist and S-Detect. Estimation of the overall j was based benign, since they showed benign ultrasonographic fea-
on a study by Landis and Koch18: less than 0 indicated tures along with stability during imaging follow-up for
poor agreement; 0.00 to 0.20 indicated slight agreement; greater than 2 years, 1 being a cyst, and 5 proven as gal-
0.21 to 0.40 indicated fair agreement; 0.41 to 0.60 indi- actoceles on aspiration. The mean size of the malignant
cated moderate agreement; 0.61 to 0.80 indicated sub- masses was significantly larger than that of the benign
stantial agreement; and 0.81 to 1.00 indicated almost masses: 17.0 mm (range, 4.0–45.0 mm) and 11.0 mm
perfect agreement. The weighted least squares approach (range, 4.0–60.0 mm), respectively (P < .001).
of Barnhart and Williamson19 was used to compare j
coefficients. All statistical analyses in this study were Diagnostic Performances of the Radiologists
performed with SAS version 9.2 software (SAS Inc, Versus S-Detect
Cary, NC), P < .05 was considered to have statistical The distribution of breast masses with histopathologic
significance. diagnoses analyzed by the radiologists and S-Detect is
summarized in Table 2. There were no significant differ-
Results ences in S-Detect results between benign and malignant
masses for both radiologists (P > .05).
Of the 119 breast masses included in this study, 65 Table 3 summarizes the diagnostic performances of
(54.6%) lesions were benign, and 54 (45.4%) lesions the radiologists and S-Detect. The sensitivity and NPV
were malignant. The mean age 6 SD of the 116 women were significantly higher for both radiologists compared
included in this study was 48.5 6 12.2 years (range, 20– to S-Detect, whereas the specificity, PPV, and accuracy
83 years). The mean size of the breast masses was were higher for S-Detect (all P < .05). The AUCs of the
16.9 6 10.7 mm (range, 4–60 mm). Among the 119 radiologists were significantly higher than that of S-
breast masses, 39 (33.3%) lesions were diagnosed by Detect (0.887 and 0.901 compared to 0.815, respec-
surgery, 8 (6.7%) by US-guided vacuum-assisted tively; P 5 .023 and .004). When the results of S-Detect

Table 2. Distribution of Breast Masses Analyzed by the Radiologists and S-Detect

Radiologist 1 Radiologist 2
Final Assessment S-Detect Benign Malignant Total Pa Benign Malignant Total Pa

2 Total 4 (100.0) 0 (0.0) 4 NA 0 (0.0) 0 (0.0) 0 NA

Probably benign 4 (100.0) 0 (0.0) 4 0 (0.0) 0 (0.0) 0
Possibly malignant 0 (0.0) 0 (0.0) 0 0 (0. 0) 0 (0.0) 0
3 Total 28 (100.0) 3 (100.0) 31 NA 36 (100.0) 3 (100.0) 39 NA
Probably benign 28 (100.0) 3 (100.0) 31 36 (100.0) 3 (100.0) 39
Possibly malignant 0 (0.0) 0 (0.0) 0 0 (0.0) 0 (0.0) 0
4a Total 26 (100.0) 7 (100.0) 33 .052 23 (100.0) 9 (100.0) 32 .121
Probably benign 24 (92.3) 4 (57.1) 28 21 (91.3) 6 (66.7) 27
Possibly malignant 2 (7.7) 3 (42.9) 5 2 (8.7) 3 (33.3) 5
4b Total 1 (100.0) 0 (0.0) 1 NA 4 (100.0) 8 (100.0) 12 .548
Probably benign 1 (100.0) 0 (0.0) 1 2 (50.0) 2 (25.0) 4
Possibly malignant 0 (0.0) 0 (0.0) 0 2 (50.0) 6 (75.0) 8
4c Total 4 (100.0) 24 (100.0) 28 .574 2 (100.0) 16 (100.0) 18 >.999
Probably benign 2 (50.0) 7 (29.2) 9 0 (0.0) 3 (18.7) 3
Possibly malignant 2 (50.0) 17 (70.8) 19 2 (100.0) 13 (81.3) 15
5 Total 2 (100.0) 20 (100.0) 22 >.999 0 (0.0) 18 (100.0) 18 NA
Probably benign 0 (0.0) 1 (5.0) 1 0 (0.0) 1 (5.6) 1
Possibly malignant 2 (100.0) 19 (95.0) 21 0 (0.0) 17 (94.4) 17

Data are presented as number (percent) where applicable. NA indicates not applicable.
a
Comparison between each radiologist and S-Detect in each BI-RADS category analyzed by the radiologist.

4 J Ultrasound Med 2017; 00:00–00

 Cho et al—Application of Computer-Aided Diagnosis on Breast Ultrasonography

were integrated, the specificity, PPV, and accuracy Figure 2. Receiver operating characteristic curves for the radiologists,
showed significant improvement compared to the per- S-Detect, and integration of the radiologists and S-Detect.

formances of the individual radiologists alone (all

P < .05). The sensitivity of radiologist 2 integrated with
the result of S-Detect was lower than that of radiologist
2 alone (P < .05). There was no significant difference
between the sensitivity of radiologist 1 integrated with
the result of S-Detect and that of radiologist 1 alone
(P 5 .170). The AUCs of the radiologists integrated
with S-Detect did not show significant differences when
compared to the AUCs of the radiologists alone (0.895
compared to 0.887 and 0.901 compared to 0.901; all
P > .05; Figure 2).
Agreement for Ultrasonographic Descriptors and
Final Assessment Between the Radiologists and S-
Detect
Agreement between the radiologists and S-Detect for
ultrasonographic descriptors and final assessments are
summarized in Table 4. Between the radiologists, agree-
ments were substantial for shape (j 5 0.72), orientation
(j 5 0.72), and posterior acoustic features (j 5 0.76)
and fair for margin (j 5 0.35) and echo pattern
(j 5 0.33). Moderate agreement (j 5 0.57) was seen
for final assessments between the radiologists. Agree-
ment between radiologist 1 and S-Detect for the ultraso-
nographic descriptors ranged from fair to moderate
(j 5 0.23–0.55). Agreement between radiologist 2 and

Table 3. Diagnostic Performances of the Radiologists With and Without Integration of S-Detect

Diagnostic Radiologist Radiologist

Performance Radiologist 1 Radiologist 2 S-Detect Pa Pb 1 1 S-Detect Pc 2 1 S-Detect Pd
Sensitivity, % 94.4 94.4 72.2 <.001 <.001 87.0 .170 83.3 .040
(88.3–100.0) (88.3–100.0) (60.3–84.2) (78.1–96.0) (73.4–93.3)
Specificity, % 49.2 55.4 90.8 <.001 <.001 86.2 <.001 87.7 <.001
(37.1–61.4) (43.3–67.5) (83.7–97.8) (77.8–94.6) (79.7–95.7)
PPV, % 60.7 63.8 86.7 <.001 <.001 83.9 <.001 84.9 <.001
(50.3–71.2) (53.2–74.3) (76.7–96.6) (74.3–93.6) (75.3–94.5)
NPV, % 91.4 92.3 79.7 .015 .006 88.9 >.999 86.4 .595
(82.2–100.0) (83.9–100.0) (70.6–88.9) (81.1–96.7) (78.1–94.6)
Accuracy, % 69.8 73.1 82.4 .014 .060 86.6 <.001 85.7 .015
(61.5–78.0) (65.1–81.1) (75.5–89.2) (80.4–92.7) (79.4–92.0)
AUC 0.887 0.901 0.815 .023 .004 0.895 >.999 0.901 >.999
(0.826–0.947) (0.846–0.956) (0.745–0.885) (0.835– (0.844–0.958)
0.956)

Values in parentheses are 95% confidence intervals.

a
Comparison between radiologist 1 and S-Detect.
b
Comparison between radiologist 2 and S-Detect.
c
Comparison between radiologist 1 and radiologist 1 1 S-Detect.
d
Comparison between radiologist 2 and radiologist 2 1 S-Detect.

J Ultrasound Med 2017; 00:00–00 5

Cho et al—Application of Computer-Aided Diagnosis on Breast Ultrasonography

S-Detect ranged from fair to substantial (j 5 0.31– levels of experience perform breast US, and the useful-
0.68). When the BI-RADS final assessments of the radi- ness of S-Detect may be different according to the level
ologists were dichotomously divided into benign (cate- of experience: for example, radiologists with less experi-
gories 2 and 3) and malignant (categories 4a–5), fair and ence may benefit more by using S-Detect.
moderate agreements were seen between the radiologists In our study, S-Detect showed significantly higher
and S-Detect (j 5 0.40 and 0.45, respectively). specificity, PPV, and accuracy compared to the radiolog-
Interobserver agreements for the shape, orientation, ists (all P < .001). When integrating the results of S-
and posterior features between the radiologists were sig- Detect, it led to significant improvements in specificity,
nificantly higher than those between the radiologists and PPV, and accuracy in both radiologists, similar to the
S-Detect (P < .05). For the final assessment categories, results of previous studies for CAD systems. In addition,
there were no significant differences in the agreement 24 of 26 (92.3%) and 21 of 23 (91.3%) benign breast
between the radiologists versus that between the individ- masses initially assessed as category 4a by the radiologists
ual radiologists and S-Detect (P > .05). were categorized as probably benign by S-Detect (Table
2). Based on our results, S-Detect could be used as an
Discussion additional tool with breast US regardless of the level of
experience the radiologist has and may be used to reduce
The ultrasonographic BI-RADS lexicon is widely used the number of unnecessary biopsies of benign breast
for breast lesion descriptions, but because of the subjec- masses. Although the specificity, PPV, and accuracy were
tive tendency of US, observer variability is inevitable, improved, the AUCs of the radiologists integrated with S-
and it can lead to inconsistent diagnoses among per- Detect had no significant differences compared to those
formers.2,5 Computer-aided diagnosis systems were of the radiologists alone (all P > .05). Both of the radiol-
recently applied to overcome the observer variability of ogists already had very high AUC values; therefore, little
breast US,20 as well as to improve the diagnostic per- was left to improve, which may have been the cause for
formances.14,15 S-Detect applies a novel feature extrac- no differences in AUCs after the integration of S-Detect.
tion technique and support vector machine classifier that Several reports have been published on applying dif-
categorizes breast masses into benign or malignant ferent types of CAD to breast US.16,17,22 These studies
according to the suggested feature combinations inte- commonly reported that the CAD systems enable
grated according to the contents of BI-RADS ultrasono- improvement in diagnostic performances of breast US,
graphic descriptors.21 In a recent study, Kim et al16 especially specificity and accuracy. Shen et al22 suggested
reported that S-Detect had significantly higher specific- that computer-aided classification systems could be help-
ity, PPV, accuracy, and AUC compared to a dedicated ful in assessing indeterminate category 4 cases. Wang
breast radiologist, with fair to substantial agreement in et al17 concluded that the inclusion of CAD was more
ultrasonographic feature analysis for breast masses. helpful for junior radiologists than the seniors, with
However, in clinical practice, radiologists with different greater improvement in the diagnostic performances in

Table 4. Interobserver Agreement for BI-RADS Sonographic Descriptors and Final Assessment Categories Among the Radiologists and
S-Detect

j, Radiologist 1 j, Radiologist 1 j, Radiologist 2

BI-RADS Lexicon vs Radiologist 2 vs S-Detect Vs S-Detect Pa Pb Pc

Shape 0.72 (0.62–0.83) 0.44 (0.30–0.58) 0.44 (0.30–0.58) <.001 <.001 .996
Orientation 0.72 (0.58–0.85) 0.55 (0.39–0.71) 0.68 (0.54–0.83) .024 .694 .053
Margin 0.35 (0.24–0.46) 0.28 (0.15–0.39) 0.31 (0.19–0.43) .316 .582 .656
Echo pattern 0.33 (0.18–0.48) 0.23 (0.03–0.44) 0.35 (0.19–0.51) .355 .883 .311
Posterior acoustic features 0.76 (0.66–0.86) 0.35 (0.22–0.49) 0.46 (0.32–0.61) <.001 .001 .037
Final assessment 0.57 (0.41–0.73) 0.40 (0.28–0.53) 0.45 (0.33–0.58) .050 .223 .392

Values in parentheses are 95% confidence intervals.

a
Comparison between radiologist 1 versus radiologist 2 and radiologist 1 versus S-Detect.
b
Comparison between radiologist 1 versus radiologist 2 and radiologist 2 versus S-Detect.
c
Comparison between radiologist 1 versus S-Detect and radiologist 2 versus S-Detect.

6 J Ultrasound Med 2017; 00:00–00

 Cho et al—Application of Computer-Aided Diagnosis on Breast Ultrasonography

the junior group. In our study, the specificity, PPV, and References
accuracy of both radiologists were improved, supporting
the idea that S-Detect can be relied on to provide a sec- 1. D’Orsi CJ, Sickles EA, Mendelson EB, et al. ACR BI-RADSV Atlas:
R

ond opinion that can be used to decide the next step for Breast Imaging Reporting and Data System. 5th ed. Reston, VA: Ameri-
patient treatment. In addition, S-Detect is a commer- can College of Radiology; 2013.
cially available, easy-to-use program that enables imme- 2. Lazarus E, Mainiero MB, Schepps B, Koelliker SL, Livingston LS. BI-
diate visualization of analytic results during real-time RADS lexicon for US and mammography: interobserver variability
ultrasonographic examinations, which can easily be and positive predictive value. Radiology 2006; 239:385–391.
incorporated in daily practice. 3. Yoon JH, Kim MJ, Moon HJ, Kwak JY, Kim EK. Subcategorization
In our study, agreement for the BI-RADS ultraso- of ultrasonographic BI-RADS category 4: positive predictive value
nographic descriptors between the radiologists and S- and clinical factors affecting it. Ultrasound Med Biol 2011; 37:
Detect was fair to substantial. For final assessments, the 693–699.
agreement between the radiologists and the agreement 4. Hong AS, Rosen EL, Soo MS, Baker JA. BI-RADS for sonography:
between each radiologist and S-Detect was moderate, positive and negative predictive values of sonographic features. AJR
without significant differences. The level of agreement Am J Roentgenol 2005; 184:1260–1265.
for final assessments between each radiologist and S- 5. Lee HJ, Kim EK, Kim MJ, et al. Observer variability of Breast Imaging
Detect were in a similar range as the agreement bet- Reporting and Data System (BI-RADS) for breast ultrasound. Eur J
ween the radiologists in this study (j 5 0.57), and also Radiol 2008; 65:293–298.
the agreement among radiologists reported in the liter- 6. Au FW, Ghai S, Moshonov H, et al. Diagnostic performance of quan-
ature (j 5 0.28–0.65).2,5,23 In addition, as S-Detect titative shear wave elastography in the evaluation of solid breast
provides the final assessment in a dichotomized form masses: determination of the most discriminatory parameter. AJR Am
of possibly benign and possibly malignant, we consider J Roentgenol 2014; 203:W328–W336.
this factor to have affected the low levels of agreement 7. Kim EJ, Kim SH, Kang BJ, Kim YJ. Interobserver agreement on the
among the radiologists. Further investigation with tech- interpretation of automated whole breast ultrasonography. Ultrasonog-
nical improvements is anticipated for a more sophisti- raphy 2014; 33:252–258.
cated algorithm using the multiple final-assessment BI- 8. Shin HJ, Kim HH, Cha JH, Park JH, Lee KE, Kim JH. Automated
RADS ultrasonographic categories. ultrasound of the breast for diagnosis: interobserver agreement on
There were a few limitations to our study. First, lesion detection and characterization. AJR Am J Roentgenol 2011; 197:
present study included a relatively small number of cases 747–754.
(n 5 119). Second, the presence of calcifications, which 9. Chang RF, Wu WJ, Moon WK, Chen DR. Automatic ultrasound
is a strong predictor for breast malignancy,24,25 was not segmentation and morphology based diagnosis of solid breast tumors.
included in the analyses because of the limited analytic Breast Cancer Res Treat 2005; 89:179–185.
ability of S-Detect for microcalcifications. Similarly, non- 10. Chen CM, Chou YH, Han KC, et al. Breast lesions on sonograms:
mass lesions were not included in this study, which may computer-aided diagnosis with nearly setting-independent features
have affected the results. Third, representative still and artificial neural networks. Radiology 2003; 226:504–514.
images stored for analysis were used during image analy- 11. Drukker K, Giger ML, Vyborny CJ, Mendelson EB. Computerized
sis by the radiologists and S-Detect. Variability also exists detection and classification of cancer on breast ultrasound. Acad Radiol
in selecting representative images of a breast mass. 2004; 11:526–535.
Fourth, 60 cases (50.4%) of breast masses included this 12. Kim KG, Cho SW, Min SJ, Kim JH, Min BG, Bae KT. Computerized
study were diagnosed by core needle biopsy only. The scheme for assessing ultrasonographic features of breast masses. Acad
false-negative rate of core needle biopsy was less than Radiol 2005; 12:58–66.
2% in the literature.26 Therefore, this factor may have 13. Sehgal CM, Cary TW, Kangas SA, et al. Computer-based margin
had little effect on our results. analysis of breast sonography for differentiating malignant and benign
In conclusion, S-Detect is a clinically feasible diag- masses. J Ultrasound Med 2004; 23:1201–1209.
nostic tool that can be used to improve the specificity, 14. Horsch K, Giger ML, Vyborny CJ, Lan L, Mendelson EB, Hendrick
PPV, and accuracy of breast US, with a moderate degree RE. Classification of breast lesions with multimodality computer-
of agreement in final assessments, regardless of the expe- aided diagnosis: observer study results on an independent clinical data
rience of the radiologist. set. Radiology 2006; 240:357–368.

J Ultrasound Med 2017; 00:00–00 7

Cho et al—Application of Computer-Aided Diagnosis on Breast Ultrasonography

15. Horsch K, Giger ML, Vyborny CJ, Venta LA. Performance of 21. Lee JH, Seong YK, Chang CH, et al. Computer-aided lesion diagnosis
computer-aided diagnosis in the interpretation of lesions on breast in B-mode ultrasound by border irregularity and multiple sonographic
sonography. Acad Radiol 2004; 11:272–280. features. In: SPIE Medical Imaging. Bellingham, WA: International
16. Kim K, Song MK, Kim EK, Yoon JH. Clinical application of S-Detect Society for Optics and Photonics; 2013:86701–86707.
to breast masses on ultrasonography: a study evaluating the diagnostic 22. Shen WC, Chang RF, Moon WK. Computer aided classification sys-
performance and agreement with a dedicated breast radiologist. Ultra- tem for breast ultrasound based on Breast Imaging Reporting and
sonography 2017; 36:3–9. Data System (BI-RADS). Ultrasound Med Biol 2007; 33:1688–1698.
17. Wang Y, Jiang S, Wang H, et al. CAD algorithms for solid breast 23. Park CS, Lee JH, Yim HW, et al. Observer agreement using the ACR
masses discrimination: evaluation of the accuracy and interobserver Breast Imaging Reporting and Data System (BI-RADS)–ultrasound,
variability. Ultrasound Med Biol 2010; 36:1273–1281. first edition (2003). Korean J Radiol 2007; 8:397–402.
18. Landis JR, Koch GG. The measurement of observer agreement for 24. Sickles EA. Mammographic features of “early” breast cancer. AJR Am J
categorical data. Biometrics 1977; 33:159–174. Roentgenol 1984; 143:461–464.
19. Barnhart HX, Williamson JM. Weighted least-squares approach for 25. Sickles EA. Breast calcifications: mammographic evaluation. Radiology
comparing correlated kappa. Biometrics 2002; 58:1012–1019. 1986; 160:289–293.
20. Singh S, Maxwell J, Baker JA, Nicholas JL, Lo JY. Computer-aided 26. Schueller G, Jaromi S, Ponhold L, et al. US-guided 14-gauge core-nee-
classification of breast masses: performance and interobserver variabil- dle breast biopsy: results of a validation study in 1352 cases. Radiology
ity of expert radiologists versus residents. Radiology 2011; 258:73–80. 2008; 248:406–413.

8 J Ultrasound Med 2017; 00:00–00