Hard-to-heal wounds include pressure ulcers (PU),1 diabetic foot ulcers (DFU)2,3 and venous leg ulcers (VLU).4,5 The increase in the prevalence of hard-to-heal wounds is an urgent issue that must be resolved in order to relieve pressure on the health-care economy. These wounds are increasingly treated by a range of health professionals in outpatient clinics or in the community, under which circumstances, the use of computer technology and cloud applications using mobile devices, such as smartphones and tablets is becoming more important.
Telemedicine is an essential part of the support system for clinical settings outside the hospital.6 Recently, the importance of the model for assessment of telemedicine (MAST) has been emphasised.7,8 Telecare and telehealth are under a state of rapid development. Of these a fundamental technology is an eHealth-supported wound assessment system using artificial intelligence (AI). However, at present, no device capable of automatically evaluating wounds has been developed, although the segmentation of wounds with considerable accuracy is possible.
Remarkable progress has been achieved in the field of AI technology in recent years. Deep learning, another term for AI, is widely used in research domains, such as computer vision, natural language processing, speech analysis and automatic driving. The use of deep learning as a machine learning and pattern recognition tool is also becoming important in the field of medical imaging analyses, as is evident from the recent increasing number of studies and reports on this topic.9 The major areas of application of eHealth medical imaging analyses involve segmentation, classification and abnormality detection in images generated from a wide spectrum of clinical imaging modalities. As a result, many novel imaging analysis methods using convolutional neural networks (CNNs) have been reported.10,11
Artificial intelligence (AI)
First, we determined the relationship of three terminologies regarding the definition of AI, machine learning and deep learning (Fig 1). AI is the largest field, and machine learning is under the umbrella of AI. Furthermore, deep learning is one type of machine learning. A machine learning algorithm is the ‘neural network’, which is a self-learning programme modelling human neurons and synapses (Fig 2). Neurotransmitters are released from presynaptic neurons, and information is transmitted to postsynaptic neurons by binding to the receptor.
How the mechanism of how the synapse functions is both chemically and biomedically known
Neurotransmitters, such as acetylcholine, are transmitted to the next postsynaptic cell. However, representing the synapse mechanism from a computational point of view, allows us to accumulate information gathered at the synapse and to classify the signal by 1, 0 by the threshold in postsynaptic cell. In other words, synapses have the function of converting a continuous signal into a digitised signal of 1s and 0s. That is, the neurotransmitters at the synapse correspond to the digitised signals of 1 and 0. The sigmoid function plays the role of this synapse. A neural network simulates this synapse mechanism on a computer and is the initial stage neural network using the sigmoid function.
A network composed of many layers in which neurons are connected to each other can perform deep learning and thereby automatically learn itself without the need for a coder to programme according to logical relationships and rules.
Deep learning
Deep learning is based on artificial neural networks and it attempts to mimic the way the human brain works. Deep learning is a branch of machine learning, in other words, deep learning is formatted as a network with a multi-layered algorithm of the middle layer in a ‘neural network’ (Fig 3).
By increasing the number of layers, we can increase the precision and versatility of feature quantities and improve the prediction accuracy. According to the Deep Learning textbook, deep learning is a kind of machine learning that achieves great power and flexibility by learning to represent the world as a nested hierarchy of concepts, with each concept defined in relation to simpler concepts and more abstract representations computed in terms of less abstract ones.12
Deep-belief networks, stacked auto-encoders and CNNs are the three main networks used in the field of deep learning.13 Applications based on deep learning include image recognition, face recognition, image classification, video classification, visual tracking, speech recognition, natural language processing and automatic driving.10 Deep learning has been applied to fields such as computer vision, automatic speech recognition, natural language processing, audio recognition and bioinformatics, where it has been shown to produce phenomenal results in various tasks. Improvements in computer capabilities and the development of various deep networks have recently led to some major advancements in the field of deep learning. In recent years, CNN-based methods have gained more popularity in vision systems as well as in the domain of medical imaging analyses. The origin of CNN is said to be ‘neocognitron’, as reported by Fukushima.14 Deep learning algorithms can be applied to both supervised and unsupervised learning tasks.
Most medical imaging analyses use supervised learning, as it is difficult to create algorithms with unsupervised learning by itself. Without supervised data which was annotated, even with exposure to a large amount of data during unsupervised learning, the rate of detection will be low.
However, reports of wound image segmentation performed using CNNs through supervised learning in the field of wound healing is gradually increasing, though the numbers are still relatively low.11 This may be because there are few, if any, researchers who understand the two fields simultaneously.
Image segmentation and recognition using CNNs
Sometimes a deep neural network is called a ‘multi-layered perceptron’. A perceptron is a single layer classifier (or regressor) with a binary threshold output using a specific way of training the weights. This is quite a primitive machine learning algorithm.
Perceptrons are an algorithm for the supervised learning of binary classifiers. They tend to recognise visual patterns directly from raw image pixels. In some cases, a slight amount of pre-processing is performed before images are fed to CNNs. The first CNN model (LeNet-5) was proposed in 1998 for recognising handwritten characters (Fig 4).15
A breakthrough was made concerning the accuracy of deep learning in 2012. In the image recognition contest ‘ImageNet Large Scale Visual Recognition Challenge’, Hinton's group from the University of Toronto presented a CNN engaged in supervised learning with eight layers. The error rate of their system was 15.3%, which beat the winning record of one year earlier of 25.7%.16 This result intrigued many researchers around the world. Since then, the image recognition approach has shifted away from the method of using a ‘feature amount’ to using a ‘neural network’.
Image segmentation using CNNs is currently being actively carried out by computer scientists. Many studies on computerised methods for evaluating wounds have been conducted.17 CNNs are also being used in research on wound assessment, reflecting the general trend in the field of computer science.18,19,20 However, few studies have involved health professionals engaged in wound assessment, and few journals in the wound healing field have published reports on the use of CNN for image segmentation.
Image segmentation of wounds is still in the preliminary stage of research. We therefore used four different CNN architectures, including U-Net, to perform wound segmentation in order to identify the most accurate architecture.
Aim
In order to build a good wound segmentation CNN model, four architectures were prepared. Each CNN learned using supervised data of sacral PUs. The aims of this study were as follows:
- To clarify which of the CNN architectures was the most accurate in the segmentation of sacral PUs
- To clarify whether or not healthy tissue could be discriminated from ulcer regions in DFUs using the most accurate CNN educated by a PU dataset
- To clarify whether or not healthy tissue could be discriminated from ulcer regions in VLUs using the most accurate CNN learned by a PU dataset.
Methods
Wound dataset
Images of PUs (n=400), DFUs (n=20) and VLUs (n=20) were extracted from a digital picture database of wounds from the past decade. The 440 cases were patients who had previously been treated at Kyorin University Hospital and all were associated with diagnosis names and images from medical records. The wounds were photographed under controlled illumination conditions using a EOS 7D and EOS 7D mark 2 digital camera (Canon Inc, Japan). The images were acquired with flashlight at a distance of approximately 30–40cm from the wound plane.
The collection of the data was approved by the institutional review board (IRB) of Kyorin University Hospital Informed consent, including for the use of photographs, was obtained from patients. Patients could withdraw from the study at any time.
The plastic surgeon divided healthy skin and PU segments one at a time using the Adobe Photoshop Elements software programme (Adobe, US) (Fig 5). The ground truth data consisted of two pieces, namely the data that punctured (segmented) the ulcer part of the original picture and the original picture data. This was read into the CNN as teacher data (supervised data). The number of images that could actually be used for the analysis was 396, as four images were excluded from the evaluation due to errors in the data storage format. The 396 images were divided into supervised learning data (356 images) and test data (40 images).
CNN architectures
The four architectures used for the CNN models and experiments were:
- SegNet (Fig 6)21
- LinkNet (Fig 7)22
- U-Net (Fig 8)23
- U-Net with the VGG16 Encoder Pre-Trained on ImageNet (Unet_VGG16) (Fig 9, Table 1).24
Table 1. Parameters of Analysis
| Architecture | Activation | Optimizer | Learning rate | Kernal initialiser | Loss function | Batch size | Image size |
|---|---|---|---|---|---|---|---|
| SegNet | ReLU | Adam | 0.001 | glorot_uniform | BCE dice loss | 16 | 256 x 256 |
| LinkNet | eLU | Adam | 0.001 | glorot_uniform+he_norm | BCE dice loss | 16 | 256 x 256 |
| U-Net | eLU | Adam | 0.001 | he_normal | BCE dice loss | 16 | 256 x 256 |
| U-Net with VGG16 Encoder Pre-Trained on ImageNet (Unet_VGG16) | eLU | Adam | 0.001 | he_normal | BCE dice loss | 16 | 256 x 256 |
ReLU—rectified linear unit; eLU—linear unit; BCE—binary cross-entropy
Performance analysis: the evaluation metrics for image segmentation
Image segmentation was performed using different key performance measures, including the area under the receiver operating characteristic curve (AUC), dice similarity coefficient (Dice), sensitivity, specificity, Matthews correlation coefficient (MCC) and accuracy for the segmentation of PUs according to the specific region.
Experimental setup
We used three Nvidia GeForce GTX 1080 graphic processing units (NVIDIA Corp, US) with an Intel CoreTM i9-7900X CPU at 3.30GHz (Intel Corp, US) and a memory of 128GB RAM to increase the speed of the parameter-learning and tested the performance of the learned model on a computer, MacBook Pro (Apple Inc, US) with a 2.7GHz Intel CoreTM i5 with 8GB memory, 1867MHz DDR3 and a 256GB SSD.
Experiment 1
The four CNNs were trained using the supervised learning data of 356 images. The accuracy of image segmentation of each of the four CNNs were evaluated in 40 test PU cases.
Experiments 2 and 3
The U-Net was found to have been the most practical in the first experiment (as shown in the results section) the same U-Net CNNs learned by the 356 PUs image dataset was used in the second experiment, where we analysed whether or not healthy tissue could be discriminated from ulcer regions in 20 DFUs.
The third experiment, carried out in the same manner as the second, analysed whether or not healthy tissue could be discriminated from ulcer regions in 20 VLUs using the most accurate CNN (U-Net CNN) educated by a PU dataset.
Results
The parameters of activation, optimiser, learning rate, kernel initialiser loss, loss function and batch size were adjusted for each architecture (Table 1).
The architecture with the highest accuracy among the four architectures was Unet_VGG16 (AUC 0.998) (Fig 11), while U-Net was the second-most accurate (Table 2). The ROI (region of interest) indicated by the four architectures on the photograph showed hardly any difference by visual observation (Fig 12). However, U-Net was 20 times faster than U-Net_VGG16 in terms of the calculation processing speed (Table 2).
Table 2. Comparison of the receiver operating characteristics area under curve (ROCAUC) between the four tested architectures
| Time (seconds) | ||||||||
|---|---|---|---|---|---|---|---|---|
| Architecture | AUC | Sensitivity | Specificity | Dice | MCC | Accuracy | GPU | CPU |
| SegNet | 0.994 | 0.909 | 0.982 | 0.874 | 0.87 | 0.976 | 9.64 | 83 |
| LinkNet | 0.987 | 0.989 | 0.989 | 0.825 | 0.838 | 0.972 | 2.01 | 5 |
| U-Net | 0.997 | 0.993 | 0.993 | 0.936 | 0.937 | 0.988 | 2.61 | 11 |
| U-Net_VGG16 | 0.998 | 0.992 | 0.992 | 0.947 | 0.946 | 0.989 | 15.79 | 250 |
AUC—area under curve; MCC—Matthews correlation coefficent; GPU—graphic processing unit; CPU—central processing unit
The evaluation metrics when DFU detection, carried out using a CNN trained with the PU dataset, demonstrated high accuracy (AUC: 0.981) (Fig 13) and a high specificity (0.988) (Table 3). However, the sensitivity (0.857) was worse than that for PUs. The DFU ROI was slightly inferior to the ROI at the time of PU by visual observation (Fig 14).
Table 3. Evaluation metrics when diabetic foot ulcer detection was made using convolutional neural networks learned using pressure ulcers
| Time (seconds) | ||||||||
|---|---|---|---|---|---|---|---|---|
| Architecture | AUC | Sensitivity | Specificity | Dice | MCC | Accuracy | GPU | CPU |
| U-Net | 0.982 | 0.858 | 0.988 | 0.850 | 0.846 | 0.97821 | 2.18 | 7.09 |
AUC—area under curve; MCC—Matthews correlation coefficient; GPU—graphic processing unit; CPU—central processing unit
The evaluation metrics when VLU detection was carried out using the CNN trained using PUs demonstrated high accuracy (AUC: 0.994) (Fig 15), specificity (0.880) and sensitivity (0.989) (Table 4).
The VLU ROI indicated by the CNN trained with the PU dataset was not different from the ROI at the time of PU indicated by the CNN trained with the PU dataset (Fig 16).
Discussion
Wound segmentation
Wound segmentation is conventionally performed using various methods, such as the k-means and fuzzy c-means (FCM) algorithms, spectral clustering (SC) and fuzzy spectral clustering (FSC).25 However, few reports have described wound detection or wound classification performed using a CNN.
Recently, Goyal et al. performed wound segmentation for DFUs using the four architectures of FCN-AlexNet, FCN-32s, FCN-16s and FCN-8s, and FCN-16s had the best accuracy.11 The fully CNN model is the basic architecture among CNNs.
Among the four architectures used to conduct the experiments, we expected to achieve the best results with U-Net composed of encoders and decoders. The reason for this is that in the Kaggle data science competition in 2015, U-Net provided the best results in the field of image segmentation/recognition. U-Net is now widely used for image segmentation tasks and biomedical image segmentation.23
In addition, we added U-Net_VGG16 which uses pre-trained VGG16 as the encoder.24 Typically, the neural network is initialised with weights from a network pre-trained on a large data set, such as ImageNet, and thus shows a better performance than those trained from scratch on a small dataset. This was the model used by the Visual Geometry Group (VGG) of Oxford University when they won the ImageNet competition in 2014.
SegNet and LinkNet are CNN models that have shown good research results and are now being used frequently in automatic operations.21,22 In automatic driving, objects in the image are detected by the segmentation of the object. SegNet and LinkNet are able to recognise animated images quickly, which is a skill similar to those needed for the segmentation of medical images. These two architectures were developed to perform the segmentation of animated images on a network used for automatic driving, their processing speeds are generally fast.
On comparing the four architectures of this study, the architecture with the highest accuracy was U-Net_VGG16; however, U-Net was considered to be the most practical, as its segmentation speed was faster than that of U-Net_VGG16, which took a long time to carry out image segmentation. When comparing PC performance in image segmentation, whether a CPU (central processing unit) or GPU is installed, plays an important role since GPU has a faster image processing performance. The time spent on computation with a PC equipped with GPU and with a CPU was compared. It was found that the processing speed depends on the computer's CPU and GPU capabilities.
These architectures were therefore included in the present study for comparisons with U-Net. While the results were good for both the U-Net and U-Net_VGG16 CNNs, U-Net was ultimately selected for further experiments, as its performance did not depend on the computer capability.
There are no useful insights to be gained from comparing the results of previous wound segmentation reports with those of this present study as the wound datasets and analysis methods are different. The results of this study are excellent as absolute values for two reasons:
- Supervised data was used which were created by clinically-experienced plastic surgeons
- The experiments performed used the four architectures currently receiving the most attention in the field of image segmentation.
Validity of photographic data for wound assessments
Segmentation using CNNs was performed using digital photographic data. At present, two approaches are available for measuring the ulcer area during wound segmentation: invasive contact type and non-invasive non-contact type.
Tracking is one method of contacting in which a transparent sheet is placed on a wound and then the wound is measured. In the past, the Visitrack (Smith & Nephew, UK) measuring device was used as a tracking device. Clinicians tend to consider direct tracking superior to photograph-based evaluations. The validity of non-contact photographic data for wound segmentation was reported by Chang et al.,26 who found that the photographic method was an accurate alternative to the tracking measuring device for measuring the wound area. No significant differences in the tracking method were noted in the wound area measured in this study.
Evolution and future potential utility of CNN segmentation for wound assessment
Accurate measurement of the wound area can be an important predictor of wound healing in non-healing wounds.27,28 In critical limb ischaemia (CLI), debridement is contraindicated because the wound is not stable until it has been demarcated, so accurately assessing the boundary between a wound and healthy tissue is fundamental.27,28 Regarding granulation, Iizaka reported29 that: ‘The assessment of granulation tissue colour using a digital image analysis will be useful as an objective monitoring tool to determine granulation tissue quality or surrogate outcomes of pressure ulcer healing’. A number of wound assessment tools have been developed in Japan, such as the National Pressure Ulcer Advisory Panel (NPUAP) staging classification30 and DESIGN-R (a PU assessment tool of the Japanese Society of Pressure Ulcers).31 However, none has provided a numerical value that can be represented by a continuous absolute value, instead only describing the pathological findings scored at the time of the observation.
In a Japanese clinical trial assessing wound healing devices such as negative pressure wound therapy (NPWT), performed under the jurisdiction of the Pharmaceuticals and Medical Devices Agency (PMDA) of Japan, it was recommended that a third-party committee be established to confirm the wound assessment in order to guarantee the objectivity of the assessment. Furthermore, the wound assessment of CLI is described in the joint statement of the FDA and PMDA.32 According to this report, ‘The evaluation of digitally photographed wounds is analysed by a central digital photometric laboratory (wound core-laboratory).’ At present, wound core-laboratory is an essential condition in clinical study, however, this will become unnecessary if objective evaluation of a wound using CNN becomes possible. Incorporating objectivity into wound evaluation is a goal for health professionals. Progress in the field of AI in recent years and the development of image segmentation by CNNs will help establish an objective wound assessment method within a few years.
Limitations
Because the dataset of this study was created by the wounds of Japanese patients, it is not possible to say whether or not similar wound segmentation results would be obtained in assessments of patients of other races.
Conclusion
CNNs for wound detection showed good segmentation accuracy when using the U-Net_VGG16 architecture. However, in terms of the calculation processing, U-Net was considered to be the most practical.
CNNs trained using a PU dataset were also useful for the accurate segmentation of DFUs and could be applied to the segmentation of other wounds. CNNs for wound detection are expected to bring major changes to the Japanese health-care system. This study concerning CNNs is the first step in perfecting AI wound assessment. These CNNs can be used to make remote, AI-based diagnoses via the cloud. The implementation of this system will be possible as soon as it can be formatted to work with smartphone apps. If it is able to perform the accurate segmentation of the wound by using CNNs, even if the AI learned with less supervised data, it may be possible to diagnose skin tumours and other skin lesions.
An eHealth-supported wound assessment system that uses AI has the possibility to significantly change the medical treatment of chronic wounds in clinical settings outside the hospital.