MINISTRY OF NATIONAL DEFENC E
MILITARY TECHNICAL ACADEMY

HA DAI DUONG

APPROACHES TO VISUAL FEATURE EXTRACTION
AND FIRE DETECTION BASED ON DIGITAL IMAGES
Majored: Mathematical foundations for Informatics
Code: 62 46 01 10

ABSTRACT OF PHD THESIS OF MATHEMATICS
HA NOI - 2014

THIS THESIS IS COMPLETED AT
MILITARY TECHNICAL ACADEMY - MINISTRY OF NATIONAL DEFENCE

Automatic fire detection has been interested for a long time
because fire causes large scale damage to humans and our properties.
Until now, some kinds of automatic detection devices, such as smoke
detectors, flame or radiation detectors, or gas detectors, were
invented. Although these traditional fire detection devices have
proven its usefulness, they have some limitations; they are generally
limited to indoors and require a close proximity to the fire; most of
them can not provide additional information about fire circumstance.
Recently, a new approach to automatic fire detection based on
computer vision has lager attractive from researchers; it offers some
advantages over the traditional detectors and can be used as
complement for existing systems. This technique can detect the fire
from a distance in large open spaces, and give more useful
information about fire circumstance such as size, location, growth
rate of fire, and in particularly it is potential to alarm early.
This research concentrated on early fire detection based on
computer vision. Firstly, some techniques that have been used for in
the literature of automatic fire detection are reviewed. Secondly,
some of visual features of fire region for early fire detection are
examined in detail, which include a model of fire-color pixel, a
model of temporal change detection, a model of textural analysis and
a model of flickering verification; and a novel model of spatial
structure of fire region. Finally, three models of fire detection based
on computer vision at the early state of fire are presented: a model of
early fire detection in general use-case (EVFD), a model of early fire
detection in weak-light environment (EVFD_WLE), and a model of
early fire detection in general use-case using SVM (EVFD_SVM).
CHAPTER 1. INTRODUCTION
1.1 Automated fire detection problems
Automatic fire detection has been interested for a long time

“Approaches to visual feature extraction and fire detection based on
digital images” with the main interest in the problem of vision-based
fire detection at the early state of fire. Main question and also be
motivation for this research is can vision-based fire detection give a
fire alarm as soon as possible at the early state of fire?. This thesis
wants to find out the answers for that question in some different use-
case such as general conditions, weak-light environment, camera is
dynamic. The objectives of this research include the following
issues: 1) Firstly, some techniques that have been used for fire
detection based on computer vision are reviewed. 2) Secondly, some
of visual features of fire region such as color, texture, temporal
change, flicker and spatial structure are examined in detail so that
reducing the computational complexity of algorithm. 3) Thirdly,
some models of early fire detection based on computer vision are
developed. The development of each model relies on the analysis of
the use-case such as for buildings and office surveillance, for

3
warehouse with weak light environment, etc. It is also applying
intelligent classification to make the models more suitable and
accurate.
1.3 Contributions
This thesis makes the following contributions:
1. Develop and propose some methods of visual features of fire
region extraction. Develop four new methods of pixel or fire region
segmentation, these include a method of fire-color pixel based on
Bayes classification in RGB space, a method of temporal change
detection, a method of textural analysis and a method of flickering
verification; and propose a novel approach to spatial structure of fire
region by using top and rings features.

fire region segmentation and proposes a novel model of spatial
structure of fire region. Chapter 4, Early fire detection based on
computer vision, presents three models of fire detection based on
computer vision: early fire detection in general use-case, early fire
detection in weak-light environment, and early fire detection in
general use-case using SVM. Chapter 5, Conclusions and
Discussions, states the conclusions, presents the contributions and
summarizes the results obtained throughout the thesis and
recommendations future research of problem.
CHAPTER 2. FIRE DETECTION BASED ON COMPUTER VISION: A REVIEW
2.1 Introduction
Automatic fire detection has been interested for a long time due
to its large scale damage to humans and our properties. Heat or
thermal detectors are the oldest type of automatic detection device
originating from the mid-19th century. Since then, other kind of
automatic detection devices; smoke detectors, flame or radiation
detectors, or gas detectors for examples have been being developed.
Although these devices have proven its usefulness in some
conditions, they have some limitations. They are generally limited to
indoors and require a close proximity to the fire. Most of them can
not provide additional information about fire circumstances and may
take a long time to raise alarm.
Fire detection based on computer vision can be marked by the
research of Healey G. et al. in the early 1990s. Since then, various
approaches to this issue were proposed. The general scheme of fire
detection based on computer vision is a combination of two
components: the analysis of visual features and the classification
techniques. The visual features include color, temporal changes,
spatial variance, texture and flickering. The classification techniques
are used to classify a pixel as fire or as non-fire, or to distinguish a

3
: (255 )*
C S R
R S R
  
,
and the fire-color model is defined as:
1 2 3
1 ( ) ( ) ( )
( , )
0
C
if R and R and R
Fire x y
Otherwise





where R, G, B are red, green and blue components of pixel at (x,y)
respectively; S is the saturation component in HSI color space; S
T

and R
T
are two experimental factors. Several other works detect fire-
color pixel using more complex model such as Gaussian mixture
model. In this model, with a given pixel, if its color value is inside
one of distribution then it is considered as a fire-color pixel. Denote

0
i i i i
Tr
if i d R G B R G B v
Fire x y
Otherwise
i
  






in which R
i
, G
i
, B
i
are the mean of red, green blue components of
Gaussian distribution i-th; v
i
is its standard deviation.
2.2.2 The temporal changes
Color model alone is not enough to identify fire pixel or fire
region. There are many objects that share the same color as fire. An
important visual feature to distinguish between fire and fire-like
objects is the temporal change of fire. To analyze temporal changes,
it may cause by flame, almost proposals assume that the camera is

Flames of an uncontrolled fire have varying colors even within a
small area since spatial color difference analysis focuses on this
characteristic. Using range filters, variance/histogram analysis, or
spatial wavelet analysis, the spatial color variations in pixel values is
analyzed to distinguish between fire and fire-like object. Using
wavelet analysis, Toreyin et al. compute a value, v, to estimate
spatial variations as follows:
2
2 2
,
1
( , ) ( , ) ( , )
lh hl hh
x y
v s x y s x y s x y
M N
  



where s
lh
(x,y) is the low-high sub-image, s
hl
(x,y) is the high-low sub-
image, and s
hh
(x,y) is the high-high sub-image of the wavelet
transform, respectively, and MN is the number of pixels potential
fire region. If the decision parameter, v, exceeds a threshold, then it

The development of application based on computer vision for
fire detection, which can raise alarm quickly and accurately, is
essential. However, vision-based fire detection is not a completely
solved problem as in most computer vision problems. The visual
features of flames of an uncontrolled fire depend on the distance,
illumination and burning materials. In addition, cameras are not color
and/or spectral measurement devices, they have sensors with
different algorithms for color and illumination balancing, and
therefore they may produce different images and video for the same
scene. For the above reasons, the research of vision-based fire
detection is necessary.
In general, most proposed methods in vision-based fire detection
returns good results in some conditions of use-case, and may give
bad results in other conditions. In particularly, current vision-based
fire detection methods are not adequate attention to alarm early so
that research of vision-based fire detection is necessary, and using
this technique for early fire detection is an important issue.
CHAPTER 3. VISUAL FEATURE EXTRACTION FOR FIRE DETECTION
This chapter presents the examining in detail some of visual
features of fire region for early fire detection; and then develop four
new models of pixel or fire region segmentation, these include a
model of fire-color pixel, a model of temporal change detection, a
model of textural analysis and a model of flickering verification; and
propose a novel model of spatial structure of fire region.

8
3.1 A new approach to color extraction
3.1.1 Chromatic analysis
The model of fire-color is usually used in the first step of the
process and is crucial to the final result. The general idea of most

2
(v) are two
discriminatory functions based on Bayesian classification for fire and
non-fire classes of pixel p; if g
1
(v)>g
2
(v) then p belong to fire class,
otherwise p belong to non-fire class. Denote 
1
is set of fire class
samples, 
2
is set of non-fire class samples, Bayessian
discriminatory functions are defined as follows:
1 1 1 1
( )
T T
vg v v W w cv
  

2 2 2 2
( )
T T
vg v v W w cv
  

in which
1 1
1

ln | | ( )
2 2
T
c C mm C P

    
,
2 2 2 2 2
1
2
1 1
ln | | ( )
2 2
T
c C mm C P

   
where m
1
is mean and C
1
is covariance matrix of 
1
and m
2
is mean
and C
2
is covariance matrix of 
2

The results of training with prepared samples of group 2:
g
1
= 42e-3*R*R+.13e-2*R*G 56e-3*R*B 14e-2*G*G+.12e-2*G*B
41e-3*B*B+.46e-1*R 44e-1*G+.63e-1*B-17.
g
2
= 98e-3*R*R+.26e-2*R*G 10e-2*R*B 35e-2*G*G
+.46e-2*G*B 20e-2*B*B+.37e-1*R 16e-1*G 37e-2*B-12
The results of training with prepared samples of group 3:
g
1
= 19e-2*R*R+.38e-2*R*G 10e-2*R*B 40e-2*G*G+.44e-2*G*B
17e-2*B*B+.29*R 16*G+.17e-1*B-28.
g
2
= 43e-2*R*R+.13e-1*R*G 42e-2*R*B 1287e-1*G*G
+.11e-1*G*B 29e-2*B*B 27e-1*R+.16*G 87e-1*B-11
0
50000
100000
150000
200000
250000
Chen Horng Toreyin Celick Borger Color F

Figure 1. The number of misclassified pixels in comparison with ColorF
3.1.3 Experiments
For comparing and testing, the author perform the experiment of
color segmentation with the model proposed by T. Celik et al., the

     
 

 
 
 
     

in which, w and h are the width and height of A and B; A(x,y), B(x,y)
are the intensity of pixel at (x,y) on A and B respectively.
Formally, assuming that I and J are two consecutive frames; M
and N are the number of row and column of dividing grid. The
scheme of temporal change detection between two consecutive
frames, I and J, in this approach is described as follows:
1. Divide I and J into MN regions, denote I
k
and J
k
for k = 1,
, MN (Figure 2);
2. Calculate correlation-coefficient between k-th region I
k
and
corresponding region J
k
, and then assign CC(I
k
,J
k
) to CH

Figure 2. The scheme of partition of two frames for temporal analysis
Method
Time performance per frame (Milliseconds)
Frames difference
23.7
Background subtraction
38.8
CMCC
24.7
Table 1. The comparison of time performance

Figure 3. An example results of three temporal change detection techniques

Figure 4. The ROC curve of temporal changes detection
Figure 4 shows the ROC (Receiver Operating Characteristic)
curve of temporal changes detection for threshold T. Rely on this
evaluation, when the threshold T = 0.025 then true positive fraction
equal to 95% and false positive fraction is 6%.
a)
b)
d)
c)
e)

12
3.2.2 Textural analysis
Intuitively, fire has unique visual signatures such as color and
texture. The textural features of a fire region includes average values
of red, green, blue components, skewness of red component
histogram, and surface coarseness are used. Denote PFR as potential

PFR, m
2
and m
3
are the variance and the third moment of p’(r) the
skewness of p’(r),
4 3
2
3
2
x m m

, is consider a textural feature. Call
p(r) as normalized histogram of gray level in PFR, variance,
2
5
1
0
( )*( )
L
r
p r r mx


 

and third moment
3
6
1

and v is used to indicate candidate region contained fire or not by
applying Bayes classifier. Let
( )
FR
g v
and
( )
NR
g v
are decision
functions for fire and non-fire, the textural model for a potential fire
region PFR is defined as:
1 ( ( ) ( ))
( )
0
FR NR
if g v g v
TextureF PFR
Otherwise




Figure 5. The number of misclassified pixels in comparison with TextureF

13
In comparison with the model ColorF and Chen, the total

0
c
if b b b
b
otherwise

 





,
1 2 0
1
0
c
if r r r
r
otherwise

 






where a
1




3.3 A novel approach to spatial structure extraction
This section presents a novel model for fire-region verification.
The spatial structure of fire region is considered in term of rings and
top features of fire region.
3.3.1 Rings feature of fire region
Assuming

is the set of pixels in image I that satisfied:
{( , ) : ( , ) }
x y I Fire x y True
   
in which, Fire(x,y) is a model of

14
fire-color. Using fuzzy clustering technique, Fuzzy C-Mean, to
cluster

(in RGB space) into
K
class, an example with K = 3 are
shown in Figure 7.

Figure 7. An example of rings feature of a fire region
Let
(1)

,

 
, the set

has
spatial ring feature if
K
partition
(1)

,
(2)

, ,
( )
K

of


satisfied:
 
( ) ( 1) ( ) ( 1)
4
( ) :
1, , 1
i i i i
p M O p M
i K
 
          

| | {| |: ( , ) , ( , ) }x z max i k M i j N k j
      
. With
( , ), ( , )
A x y B z y
are chosen above, a part of

, denote
AB

, that lay
above of AB is determined as follows:
{ ( , ) : }
AB
P i j j y
   
.
Secondly, choose a point
( , )
AB
C a b

such that
b y

for

( , )
AB
M x y

if
t
e
     

 



in which,
01

and
02

were chosen by experiment. If triangle ABC
contains most of the pixels in the upper part of fire region, it is clear
that the value of
1

and
2

are not large. Figure 8 describes
structure of the flame with the triangle ABC which described above
3.3.3 Experiments
In these experiments 563 images are used, these are divided into
three categories: A) images containing a single fire region at the
early stages of fire, 157 images, B) images containing complex when
the fire broke out, 185 images, C) images do not contain the fire, but

3.4 Summary
The visual features of fire region play an important role in
vision-based fire detection. In this study, five visual features of fire
region are examined in detail; four new models of pixel or fire region
segmentation are developed, these include a model of fire-color pixel
([9]), a model of temporal change detection ([1], [2]), a model of

16
textural analysis and a model of flickering verification ([1], [2], [4]);
and a novel model of spatial structure of fire region ([7]).
CHAPTER 4. EARLY FIRE DETECTION BASED ON COMPUTER VISION
This chapter presents three models of fire detection based on
computer vision: early fire detection in general use-case, early fire
detection in weak-light environment, and early fire detection in
general use-case using SVM.
4.1 Early fire detection in the normal use-case
4.1.1 General use-case
This section presents an approach to the problem of early fire
detection based on computer vision for using in general use-case
conditions with assumptions: camera is static; burning material is
popular such as paper, wood, etc.; and circumstance of fire is not too
far from camera.
4.1.2 The algorithm EVFD
The model is a combination of temporal analysis using
correlation coefficient, color analysis based on RGB color space, and
flickering analysis as shown in Figure 9.

Figure 9. The scheme of EVFD algorithm
The detail of the EVFD algorithm: two consecutive frames,
previous frame I, current frame J have size of mn are inputs. The

CH k q
I k a i q b j J k a i q b j
 
   
        

        
 
   

b. Establish the change map based on correlation-coefficient,
CM, as follows:
For x =1 to m
For y =1 to n
1 ( \ 1, \ 1)
( , )
0
if CH x M y N T
CM x y
otherwise
  




;
3. Establish the potential fire region, PFR, based on the color clue by
using ColorF(x,y) model from the change map CM:
a. Detect potential fire mask PFM:
For x =1 to m

18
First frame detected has fire
Video
First frame
has fire
Chen
Gunay
EVFD
Video 1 - Indoor 1 with fire
2
2
3
2
Video 2 - Indoor 2 with fire
2
2
3
3
Video 3 - Indoor 3 with fire
104
2
2
105
Video 4 - Indoor 4 no fire
-
-
12
-
Video 5 - Outdoor 1 with fire
2

150
149
142
Video 2 - Indoor 2 with fire
150
142
149
101
Video 3 - Indoor 3 with fire
23
88
149
5
Video 4 - Indoor 4 no fire
0
0
98
0
Video 5 - Outdoor 1 with fire
150
150
149
74
Video 6 - Outdoor 2 with fire
150
150
149
100
Video 7 - Outdoor 3 with fire
27

4.2.1 The weak-light environment
Assuming p(r) is a normalized gray-level histogram of image I,
( )
r
p r n n

where r[0,…, L-1], and L is the number of gray level in
images, n
r
is the total number of pixels with gray level r and n is the

19
total number of pixels in I. In weak light environment average gray
level of I,
1
0
* ( )
L
r
M r p r




, is small, and uniformity of I,
1
0
( )* ( )
L
r

using ColorF to establish the potential fire region. Finally, analyses
spatial structure to confirm potential fire region as fire or non-fire
region. The scheme of EVFD_WLE is shown in Figure 10.

Figure 10. The scheme of EVFD_WLE algorithm
The detail of the EVFD_WLE algorithm: Input image I with the
size of mn. The output is a Boolean variance, A, it is TRUE to
notice that I contain fire and FALSE when J do not contain fire.
The algorithm EVFD_WLE
 Input: image I;
 Output: Boolean variable A;
o A = TRUE if I contain fire
o A = FALSE if I do not contain fire
1. A = FALSE;
2. Verification of the environment light:
a. Calculate intensity histogram p(r)
b. Calculate M and U
For r =1 to L
M = M + r  p(r); U = U + p(r)  p(r);
c. If M M
0
and UU
0
then goto 3, otherwise goto 6;
1. WLE verification
2. The color detection using ColorF
3. Spatial structure verification
Input: Frames
Output: Fire alarm


 
01
and 
1
 
01
then goto 5, otherwise goto 6;
5. Analyze the rings feature of 
a. Cluster  into K class using Fuzzy C-Mean algorithm
b. Verify rings feature of 
( )
Rings r
 

c. A= Rings
6. Return A;
4.2.3 Experiments
These experiments use 7 videos which consist of 4 videos with
fire in weak-light environment, 2 videos indoor, and one video
outdoor with the resolution is 320240. In this study, the values of
M
0
, U
0
, 
10
, 
20
are assigned 50, 0.05, 0.55 and 0.55 respectively. The
performance of proposed method here EVFD_WLE is computed and

121
124
108
Video 04 - WLE, camera is dynamic
150
70
41
44
58
Video 05 - Indoor, camera is static
150
0
129
149
0
Video 06 - Indoor, camera is static
150
0
130
149
0
Video 07 - Outdoor, camera is static
150
0
126
149
0
Table 7. The number of frames detected in comparison with EVFD_WLE
4.3 Early fire detection in the normal use-case using SVM
This section presents an approach to the problem of early fire

ClM(x,y) = ColorF(x,y) (on I);
b. Compute the potential fire mask, FM as
For x =1 to m
1. Pixel - based processing
2. Region - based processing
3. SVM classifier
Input: Frames
Output: Fire alarm

22
For y =1 to n
1 | ( , ) ( , ) | ( , ) 1
( , )
0
if I x y J x y and ClM x
otherw
y
FM x y
ise
  





c. Remove small components and recover fire mask FM:
For x =1 to m
For y =1 to n
If (FM(x,y)=0) and FM(x+x,y+y)=1 then FM(x,y)=1
3. Region-based processing

First
frame
detected
Number
frames
with fire
Number
frames
detected
False
detected
(%)
Fire in room 1
150
2
3
150
120
20(-)
Fire in room 2
150
2
3
150
136
9(-)
Fire outdoor 1
50
2
3

98(-)
Moving object
150
-
-
0
0
0(+)
Traffic 1
150
-
-
0
0
0(+)
Traffic 2
150
-
-
0
0
0(+)
Table 8. The performance of EVFD_SVM algorithm
4.3.3 Experiments
To evaluate, a set of 10 videos which consists of 7 fire videos
and 3 non-fire videos are used, and the video resolution is 320240.
The performance of algorithm is shown in Table 8. The third column
in Table 8 is the order of the first frame in video sequences that
contains fire, and the fourth column is the order of the first frame in
video sequences in which fire was detected. For the video from 1 to

of early fire detection in general use-case using SVM - EVFD_SVM.
5.1 Summary of contributions
Following the aim of the research, contributions that had done in
this work can be summaries as follows:
1. Develop and propose some methodsl of visual features of fire
region extraction. Develop four new methods of pixel or fire region
segmentation, these include a method of fire-color pixel based on
Bayes classification in RGB space, a method of temporal change
detection, a method of textural analysis and a method of flickering