HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY

NGUYEN THI THANH NHAN

INTERACTIVE AND MULTI-ORGAN
BASED PLANT SPECIES
IDENTIFICATION

Major: Computer Science
Code: 9480101

INTERACTIVE AND MULTI-ORGAN BASED PLANT
SPECIES IDENTIFICATION

SUPERVISORS:
1. Assoc. Prof. Dr. Le Thi Lan
2. Assoc. Prof. Dr. Hoang Van Sam

Hanoi − 2020


HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY

Nguyen Thi Thanh Nhan

INTERACTIVE AND MULTI-ORGAN
BASED PLANT SPECIES
IDENTIFICATION

Major: Computer Science
Code: 9480101


Nguyen Thi Thanh Nhan

SUPERVISORS

i


ACKNOWLEDGEMENT
First of all, I would like to thank my supervisors Assoc. Prof. Dr. Le Thi
Lan at The International Research Institute MICA - Hanoi University of Science and
Technology, Assoc. Prof. Dr. Hoang Van Sam at Vietnam National University of
Forestry for their inspiration, guidance, and advice. Their guidance helped me all the
time of research and writing this dissertation.
Besides my advisors, I would like to thank Dr. Vu Hai, Assoc. Prof. Dr. Tran
Thi Thanh Hai for their great discussion. Special thanks to my friends/colleagues in
MICA, Hanoi University of Science and Technology: Hoang Van Nam, Nguyen Hong
Quan, Nguyen Van Toi, Duong Nam Duong, Le Van Tuan, Nguyen Huy Hoang, Do
Thanh Binh for their technical supports. They have assisted me a lot in my research
process as well as they are co-authored in the published papers.
Moreover, I would like to thank reviewers of scientific conferences, journals and
protection council, reviewers, they help me with many useful comments.
I would like to express a since gratitude to the Management Board of MICA Institute. I would like to thank the Thai Nguyen University of Information and Communication Technology, Thai Nguyen over the years both at my career work and outside
of the work.
As a Ph.D. student of the 911 program, I would like to thank this program for
financial support. I also gratefully acknowledge the financial support for attending
the conferences from the Collaborative Research Program for Common Regional Issue (CRC) funded by ASEAN University Network (Aun-Seed/Net), under the grant
reference HUST/CRC/1501 and NAFOSTED (grant number 106.06-2018.23).
Special thanks to my family, to my parents-in-law who took care of my family and
created favorable conditions for me to study. I also would like to thank my beloved

LIST OF TABLES

x

LIST OF FIGURES

xiv

INTRODUCTION

1

1 LITERATURE REVIEW
1.1 Plant identification . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1.1 Manual plant identification . . . . . . . . . . . . . . . . . . .
1.1.2 Plant identification based on semi-automatic graphic tool . .
1.1.3 Automated plant identification . . . . . . . . . . . . . . . .
1.2 Automatic plant identification from images of single organ . . . . .
1.2.1 Introducing the plant organs . . . . . . . . . . . . . . . . . .
1.2.2 General model of image-based plant identification . . . . . .
1.2.3 Preprocessing techniques for images of plant . . . . . . . . .
1.2.4 Feature extraction . . . . . . . . . . . . . . . . . . . . . . .
1.2.4.1 Hand-designed features . . . . . . . . . . . . . . . .
1.2.4.2 Deeply-learned features . . . . . . . . . . . . . . .
1.2.5 Classification methods . . . . . . . . . . . . . . . . . . . . .
1.3 Plant identification from images of multiple organs . . . . . . . . .
1.3.1 Early fusion techniques for plant identification from images
multiple organs . . . . . . . . . . . . . . . . . . . . . . . . .
1.3.2 Late fusion techniques for plant identification from images
multiple organs . . . . . . . . . . . . . . . . . . . . . . . . .

12
12
13
13
16
18
20
20
22
26
28
30
31
33
35
44


2 LEAF-BASED PLANT IDENTIFICATION METHOD BASED ON
KERNEL DESCRIPTOR
2.1 The framework of leaf-based plant identification method . . . . . . . .
2.2 Interactive segmentation . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.1 Pixel-level features extraction . . . . . . . . . . . . . . . . . . .
2.3.2 Patch-level features extraction . . . . . . . . . . . . . . . . . . .
2.3.2.1 Generate a set of patches from an image with adaptive
size . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.2.2 Compute patch-level feature . . . . . . . . . . . . . . .
2.3.3 Image-level features extraction . . . . . . . . . . . . . . . . . . .
2.4 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . .

56
56
57
57
58
58
60
61
66

67
67
69
75
77
78
79
79
87

4 A FRAMEWORK FOR AUTOMATIC PLANT IDENTIFICATION
WITHOUT DEDICATED DATASET AND A CASE STUDY FOR
BUILDING IMAGE-BASED PLANT RETRIEVAL
88
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4.2 Challenges of building automatic plant identification systems . . . . . . 88

iv



No. Abbreviation Meaning
1

AB

Ada Boost

2

ANN

Artificial Neural Network

3

Br

Branch

4

CBF

Classification Base Fusion

5

CNN

Convolution Neural Network


11

FC

Fully Connected

12

Fl

Flower

13

FN

False Negative

14

FP

False Positive

15

GPU

Graphics Processing Unit


21

Le

Leaf

22

L-SVM

Linear Support Vector Machine

23

MCDCNN

Multi Column Deep Convolutional Neural Networks

24

NB

Naive Bayes

25

NNB

Nearest NeighBor


vi


31

RAM

Random Acess Memory

32

ReLU

Rectified Linear Unit

33

RHF

Robust Hybrid Fusion

34

RF

Random Forest

35


SVM-RBF

Support Vector Machine-Radial Basic Function kernel

41

TP

True Positive

42

TN

True Negative

vii


MATH SYMBOLS
No. Symbol

Meaning

1

Summation - sum of all values in range of series

2


sign(x)

The sign function that determines the sign. Equals 1 if x ≥ 0, −1
if x < 0

8

∈

Is member of

9

max

The function takes the largest number from a list

10

arctan(x)

It returns the angle whose tangent is a given number

11

cos(θ)

Function of calculating cosine value of angle θ

12


argmax(x)

It indicates the element that reaches its maximum value

18

⊗

The Kronecker product

19

xT

Transposition of vector x

20

Product of all values in range of series

21

q

The query-image set

22

si (Ik )

kp

The position kernel

28

m(z)
˜

The normalized gradient magnitude

viii


LIST OF TABLES

Table 1.1

Example dichotomous key for leaves . . . . . . . . . . . . . . . .

11

Table 1.2

Methods of plant identification based on hand-designed features .

21

Table 1.3 A summary of available crowdsourcing systems for plant information collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Table 3.1 An example of test phase results and the retrieved plant list determination using the proposed approach. . . . . . . . . . . . . . . . .

72

Table 3.2

The collected dataset of 50 species with four organs . . . . . . .

78

Table 3.3 Single organ plant identification accuracies with two schemes: (1)
An CNN for each organ; (2) An CNN for all organs. The best result is
in bold. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

80

Table 3.4 Obtained accuracy at rank-1, rank-5 when combining each pair of
organs with different fusion schemes in case of using AlexNet. The best
result for each pair of organ is in bold. . . . . . . . . . . . . . . . . . .

81

Table 3.5 Obtained accuracy at rank-1, rank-5 when combining each pair of
organs with different fusion schemes in case of using ResNet. The best
result is in bold. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

81

ix


Plant images dataset built by crowdsourcing data collection tools.

91

Table 4.3

Dataset used for evaluating organ detection method. . . . . . . .

94

Table 4.4 The organ detection performance of the GoogLeNet with different
weight initialization. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

95

Table 4.5

Confusion matrix for plant organ detection obtained (%) . . . . .

96

Table 4.6 Precision, Recall and F-measure for organ detection with LifeCLEF2015 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

97

Table 4.7 Confusion matrix for detection 6 organs of 100 Vietnam species
on VnDataset2 (%) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Table 4.8

Four Vietnamese medicinal species databases . . . . . . . . . . . 102

of the same species vary significantly due to the growth stage. . . . . .

5

Figure 5
Challenges of plant identification. (a) Viewpoint variation; (b)
Occlusion; (c) Clutter; (d) Lighting variation; (e) color variation of same
species . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5

Figure 6

Confusion matrix for two-class classification. . . . . . . . . . . .

6

Figure 7

A general framework of plant identification. . . . . . . . . . . .

8

Figure 1.1

Botany students identifying plants using manual approach . . .

11

Figure 1.2 (a) Main graphical interface of IDAO; (b), (c), (d) Graphical icons


Figure 1.8

The entire views for Acer pseudoplatanus L. . . . . . . . . . . .

17

Figure 1.9

Fundamental steps for image-based plant species identification .

17

Figure 1.10 Accuracy of plant identificaiton based on leaf images on complex
background in the ImageCLEF 2012 [18] . . . . . . . . . . . . . . . . .

19

xi


Figure 1.11 Feature visualization of convolutional net trained on ImageNet
from [58] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23

Figure 1.12 Architecture of a Convolutional Neural Network. . . . . . . . . .

23


Figure 2.1

The complex background leaf image plant identification framework. 46

Figure 2.2

The interactive segmentation scheme. . . . . . . . . . . . . . . .

47

Figure 2.3 Standardize the direction of leaf. (a): leaf image after segmentation; (b): Convert to binary image; (c): Define leaf boundary using
Canny filter; (d): Standardized image direction. . . . . . . . . . . . . .

49

Figure 2.4 Examples of leafscan and leaf, the first row are raw images, the
second row are images after applying corresponding pre-processing techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

50

Figure 2.5 An example of the uniform patch in the original KDES and the
adaptive patch in our method. (a,b) two images of the same leaf with
different sizes are divided using uniform patch; (b,c): two images of the
same leaf with different sizes are divided using adaptive patch. . . . . .

52

Figure 2.6 An example of patches and cells in an image and how to convert
adaptive cells. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .



Figure 2.12 Detailed scores obtained for Leaf Scan [1], my team’s name is Mica. 64
Figure 2.13 Detailed scores obtained for all organs [1], my team’s name is Mica. 65
Figure 3.1 An example of a two plant species that are similar in leaf but
different in flower (left) and those are similar in leaf and different in fruits. 68
Figure 3.2

The framework for multi-organ plant identification

. . . . . . .

68

Figure 3.3

Explanation for positive and negative samples. . . . . . . . . . .

71

Figure 3.4 Illustration of positive and negative samples definition. With
a pair of images from leaf (a) and flower (c) of the species #326, the
corresponding confidence score of all species in the dataset (e.g., 50)
when using leaf and flower image are shown in (b). . . . . . . . . . . .

71

Figure 3.5 In RHF method, each species has an SVM model based on its
positive and negative samples . . . . . . . . . . . . . . . . . . . . . . .

73

column shows top 5 species returned by the classifier. The third column
is the corresponding confidence score for each species. The name of
species in the groundtruth is Robinia pseudoacacia L. . . . . . . . .

82

xiii


Figure 3.12 Cumulative Match Characteristic curve obtained by the proposed
method with AlexNet (Scheme 1 for single organ identification) . . . .

84

Figure 3.13 Cumulative Match Characteristic curve obtained by the proposed
method with ResNet (Scheme 1 for single organ identification) . . . . .

85

Figure 3.14 Cumulative Match Characteristic curve obtained by the propsed
method with AlexNet (Scheme 2 for single organ identification) . . . .

85

Figure 3.15 Cumulative Match Characteristic curve obtained by the proposed
method with ResNet (Scheme 2 for single organ identification) . . . . .

86

Figure 4.1


97

Figure 4.7 Detection results of the GoogLeNet with different classification
methods at the first rank (k=1) . . . . . . . . . . . . . . . . . . . . . .

98

Figure 4.8 Results obtained by the proposed GoogLeNet and the method
in [7] for six organs . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

99

Figure 4.9

Architecture of Vietnamese medicinal plant search system [124]

100

Figure 4.10 Snapshots of VnMed; a) list of species for a group of diseases; b)
a detail information for one species; c) a query image for plant identification; d) top five returned results. . . . . . . . . . . . . . . . . . . . . 100
Figure 4.11 Data distribution of 596 Vietnamese medicinal plants. . . . . . . 103
Figure 4.12 Illustration of image-based plant retrieval in VnMed. . . . . . . 104

xiv


INTRODUCTION
Motivation
Plants play an important part in ecosystem. They provide oxygen, food, fuel,

The main aim of this thesis is to overcome the second limitation of the automatic
plant identification (low recognition accuracy) by proposing novel and robust methods
for plant recognition. For this, we first focus on improving the recognition accuracy
of plant identification based on images of one sole organ. Among different organs of
the plant, we select leaf as this organ is the most widely used in the literature [10].
However, according to [10], most analyzed images in the previous studies were taken
under simplified conditions (e.g., one mature leaf per image on a plain background).
Towards real-life application, the plant identification methods should be experimented
with more realistic images (e.g., having a complex background, and been taken in
different lighting conditions).
Second, taking into consideration that using one sole organ for plant identification
is not always relevant because one organ cannot fully reflect all information of a plant
due to the large inter-class similarity and the large intra-class variation. Therefore,
multi-organ plant identification is also studied in this thesis. In this thesis, multiorgan plant identification will be formulated as a late fusion problem: the multi-organ
plant results will be determined based on those obtained from single-organ. Therefore,
the thesis will focus on fusion schemes.
Finally, the last objective of the thesis is to build an application of Vietnamese
medicinal plant retrieval based on plant identification. By this application, the knowledge that previously only belongs to botanists can be now popular for the community.
To this end, the concrete objectives are:
❼ Develop a new method for leaf-based plant identification that is able to recognize

the plants of interest even in complex background images;

2


❼ Propose a fusion scheme in multiple-organ plant identification;
❼ Develop a image-based plant search module in Vietnamese medicinal plant re-

trieval application.


Figure 3 One observation of a plant
The automatic plant identification has to face different challenges. The first challenge is the large inter-class similarity and the large intra-class variation. Figure 4(a)
illustrates the case of the large inter-class similarity (leaves of two distinct species are
very similar) while Figure 4(b) shows an example of the large intra-class variation
(leaves of the same species vary significantly due to the growth stage). The second
challenge is the background of the plant images is usually complex especially for NaturalBackground images. Data imbalance is the third challenge of automatic plant
identification as the distribution of plant species on the planet is diverse. The fourth
challenge is the high number of species. To the best of our knowledge, the biggest image
dataset of LifeCLEF 2017 contains more than 1.8M images of 10,000 plant species [3].
Finally, plants images are usually captured by different users with different acquisition
protocols. Therefore, they have lighting and viewpoint variations and may contain
occlusions, clutter, and object deformations. These issues are illustrated in Figure 5
with several species.

4


Figure 4 (a) Example of large inter-class similarity: leaves of two distinct species are
very similar; (b) example of large intra-class variation: leaves of the same species vary
significantly due to the growth stage.

Figure 5 Challenges of plant identification. (a) Viewpoint variation; (b) Occlusion;
(c) Clutter; (d) Lighting variation; (e) color variation of same species

5


Evaluation metrics
In plant identification, for each query containing one or multiple images of one or


Figure 6 Confusion matrix for two-class classification.

6


P recision =

TP
TP + FP

(2)

❼ Recall

Recall =

TP
TP + FN

(3)

❼ F-measure

F − measure =

2 × P resision × Recall
P recison + Recall

(4)

ru,p,n

(5)

where U is the number of users having at least one image in the test set, Pu is the
number of individual plants observed by the u-th user (within the test set), Nu,p
is the number of pictures of the p-th plant observation of the u-th user, ru,p,n is
the rank of the correct species within the ranked list of images returned by the
identification method. This metric allows compensating the long-tail distribution
effects occurring in social data (most users provide much less data, only a few
people provide huge quantities of data). The value of S ranges from 0 to 1. The
greater the value of S is, the better the identification method is.

Contributions
The dissertation has three main contributions as follows:
❼ Contribution 1: A complex background leaf-based plant identification method

has been proposed. The proposed method takes the advantages of segmentation
with a few interactions from the user to determine the leaf region. The features
are then extracted on this region by the representative power of Kernel Descriptor (KDES). The experimental results obtained on different benchmark datasets
have shown that the proposed method outperforms state of the art hand-crafted
feature-based methods.
7


❼ Contribution 2: One fusion scheme for two-organ based plant identification

has been introduced. The fusion is an integration between a product rule and a
classification-based approach.
❼ Contribution 3: Finally, an image-based plant searching module has been de-

❼ Chapter 3: This chapter focuses on multi-organ plant identification. We propose

a method named RHF (Robust Hybrid Fusion) for determining the result of
two-organ identification based on those of single-organ ones.
❼ Chapter 4: In this chapter, we propose a method of organ detection and an

application for Vietnamese medicinal plant retrieval system based on this method.
❼ Conclusion: We give some conclusions and discuss the limitations of the proposed

method. Research directions are also described for future works.

9