451
CHAPTER
26
Neural Networks (and more!)
Traditional DSP is based on algorithms, changing data from one form to another through step-by-
step procedures. Most of these techniques also need parameters to operate. For example:
recursive filters use recursion coefficients, feature detection can be implemented by correlation
and thresholds, an image display depends on the brightness and contrast settings, etc.
Algorithms describe what is to be done, while parameters provide a benchmark to judge the data.
The proper selection of parameters is often more important than the algorithm itself. Neural
networks take this idea to the extreme by using very simple algorithms, but many highly
optimized parameters. This is a revolutionary departure from the traditional mainstays of science
and engineering: mathematical logic and theorizing followed by experimentation. Neural networks
replace these problem solving strategies with trial & error, pragmatic solutions, and a "this works
better than that" methodology. This chapter presents a variety of issues regarding parameter
selection in both neural networks and more traditional DSP algorithms.
Target Detection
Scientists and engineers often need to know if a particular object or condition
is present. For instance, geophysicists explore the earth for oil, physicians
examine patients for disease, astronomers search the universe for extra-
terrestrial intelligence, etc. These problems usually involve comparing the
acquired data against a threshold. If the threshold is exceeded, the target (the
object or condition being sought) is deemed present.
For example, suppose you invent a device for detecting cancer in humans. The
apparatus is waved over a patient, and a number between 0 and 30 pops up on
the video screen. Low numbers correspond to healthy subjects, while high
numbers indicate that cancerous tissue is present. You find that the device
works quite well, but isn't perfect and occasionally makes an error. The
question is: how do you use this system to the benefit of the patient being
examined?
The Scientist and Engineer's Guide to Digital Signal Processing452

If we stopped the analysis at this point, we would be making one of the most
common (and serious) errors in target detection. Another source of information
must usually be taken into account to make the curve in (d) meaningful. This
is the relative number of targets versus nontargets in the population to be
tested. For instance, we may find that only one in one-thousand people have
the cancer we are trying to detect. To include this in the analysis, the
amplitude of the nontarget pdf in (c) is adjusted so that the area under the curve
is 0.999. Likewise, the amplitude of the target pdf is adjusted to make the area
under the curve be 0.001. Figure (d) is then calculated as before to give the
probability that a patient has cancer.
Neglecting this information is a serious error because it greatly affects how the
test results are interpreted. In other words, the curve in figure (d) is drastically
altered when the prevalence information is included. For instance, if the
fraction of the population having cancer is 0.001, a test result of 15
corresponds to only a 0.025% probability that this patient has cancer. This is
very different from the 25% probability found by relying on the output of the
machine alone.
This method of converting the output value into a probability can be useful
for understanding the problem, but it is not the main way that target
detection is accomplished. Most applications require a yes/no decision on
Chapter 26- Neural Networks (and more!) 453
Parameter value
0 5 10 15 20 25 30
0.00
0.20
0.40
0.60
0.80
1.00
d. Separation

to some parameter value. From these histograms, the probability distribution functions of the two groups can be
estimated, as shown in (c). Using only this information, the curve in (d) can be calculated, giving the probability
that a target has been found, based on a specific value of the parameter.
Parameter value
0 5 10 15 20 25 30
0.00
0.04
0.08
0.12
0.16
0.20
non-
target
target
c. pdfs
probability of being target pdf
Number of occurencesNumber of occurences
the presence of a target, since yes will result in one action and no will result
in another. This is done by comparing the output value of the test to a
threshold. If the output is above the threshold, the test is said to be positive,
indicating that the target is present. If the output is below the threshold, the
test is said to be negative, indicating that the target is not present. In our
cancer example, a negative test result means that the patient is told they are
healthy, and sent home. When the test result is positive, additional tests will
be performed, such as obtaining a sample of the tissue by insertion of a biopsy
needle.
Since the target and nontarget distributions overlap, some test results will
not be correct. That is, some patients sent home will actually have cancer,
and some patients sent for additional tests will be healthy. In the jargon of
target detection, a correct classification is called true, while an incorrect

positive test result (i.e., a true-positive). The other 20% that have cancer will
be incorrectly reported as a negative (i.e., a false-negative). As shown in the
ROC curve in (b), this threshold results in a point on the curve at: %
nontargets positive = 13%, and % targets positive = 80%.
The more efficient the detection process, the more the ROC curve will bend
toward the upper-left corner of the graph. Pure guessing results in a straight
line at a 45E diagonal. Setting the threshold relatively low, as shown in (a),
results in nearly all the target signals being detected. This comes at the price
of many false alarms (false-positives). As illustrated in (d), setting the
threshold relatively high provides the reverse situation: few false alarms, but
many missed targets.
These analysis techniques are useful in understanding the consequences of
threshold selection, but the final decision is based on what some human will
accept. Suppose you initially set the threshold of the cancer detection
apparatus to some value you feel is appropriate. After many patients have
been screened with the system, you speak with a dozen or so patients that
have been subjected to false-positives. Hearing how your system has
unnecessarily disrupted the lives of these people affects you deeply,
motivating you to increase the threshold. Eventually you encounter a
Chapter 26- Neural Networks (and more!) 455
Parameter value
0 5 10 15 20 25 30
0.00
0.04
0.08
0.12
0.16
0.20
threshold
target

0.08
0.12
0.16
0.20
threshold
target
target
non-
Threshold on pdf Point on ROC
a.
b.
c.
d.
% nontargets positive
0 20 40 60 80 100
0
20
40
60
80
100
worse
better
% nontargets positive
0 20 40 60 80 100
0
20
40
60
80

pdf
target
nontarget
FIGURE 26-3
Example of a two-parameter space. The
target (Î) and nontarget (~) groups are
completely separate in two-dimensions;
however, they overlap in each individual
parameter. This overlap is shown by the
one-dimensional pdfs along each of the
parameter axes.
parameter 1
situation that makes you feel even worse: you speak with a patient who is
terminally ill with a cancer that your system failed to detect. You respond to
this difficult experience by greatly lowering the threshold. As time goes on
and these events are repeated many times, the threshold gradually moves to an
equilibrium value. That is, the false-positive rate multiplied by a significance
factor (lowering the threshold) is balanced by the false-negative rate multiplied
by another significance factor (raising the threshold).
This analysis can be extended to devices that provide more than one output.
For example, suppose that a cancer detection system operates by taking an x-
ray image of the subject, followed by automated image analysis algorithms to
identify tumors. The algorithms identify suspicious regions, and then measure
key characteristics to aid in the evaluation. For instance, suppose we measure
the diameter of the suspect region (parameter 1) and its brightness in the image
(parameter 2). Further suppose that our research indicates that tumors are
generally larger and brighter than normal tissue. As a first try, we could go
through the previously presented ROC analysis for each parameter, and find an
acceptable threshold for each. We could then classify a test as positive only
if it met both criteria: parameter 1 greater than some threshold and parameter

In the jargon of the field, this type of coordinate system is called a
parameter space. For example, the two-dimensional plane in this example
could be called a diameter-brightness space. The idea is that targets will
occupy one region of the parameter space, while nontargets will occupy
another. Separation between the two regions may be as simple as a straight
line, or as complicated as closed regions with irregular borders. Figure 26-
4 shows the next level of complexity, a three-parameter space being
represented on the x, y and z axes. For example, this might correspond to
a cancer detection system that measures diameter, brightness, and some
third parameter, say, edge sharpness. Just as in the two-dimensional case,
the important idea is that the members of the target and nontarget groups
will (hopefully) occupy different regions of the space, allowing the two to
be separated. In three dimensions, regions are separated by planes and
curved surfaces. The term hyperspace (over, above, or beyond normal
space) is often used to describe parameter spaces with more than three
dimensions. Mathematically, hyperspaces are no different from one, two
and three-dimensional spaces; however, they have the practical problem of
not being able to be displayed in a graphical form in our three-dimensional
universe.

The threshold selected for a single parameter problem cannot (usually) be
classified as right or wrong. This is because each threshold value results in
a unique combination of false-positives and false-negatives, i.e., some point
along the ROC curve. This is trading one goal for another, and has no
absolutely correct answer. On the other hand, parameter spaces with two or
The Scientist and Engineer's Guide to Digital Signal Processing458
more parameters can definitely have wrong divisions between regions. For
instance, imagine increasing the number of data points in Fig. 26-3, revealing
a small overlap between the target and nontarget groups. It would be possible
to move the threshold line between the groups to trade the number of false-

neural network.
Neural Network Architecture
Humans and other animals process information with neural networks. These
are formed from trillions of neurons (nerve cells) exchanging brief electrical
pulses called action potentials. Computer algorithms that mimic these
biological structures are formally called artificial neural networks to
distinguish them from the squishy things inside of animals. However, most
scientists and engineers are not this formal and use the term neural network to
include both biological and nonbiological systems.
Chapter 26- Neural Networks (and more!) 459
input layer
hidden layer
output layer
(passive nodes)
(active nodes)
(active nodes)
X1
2
X1
1
X1
3
X1
4
X1
5
X1
6
X1
7

Neural network architecture. This is the
most common structure for neural
networks: three layers with full inter-
connection. The input layer nodes are
passive, doing nothing but relaying the
values from their single input to their
multiple outputs. In comparison, the
nodes of the hidden and output layers
are active, modifying the signals in
accordance with Fig. 26-6. The action
of this neural network is determined by
the weights applied in the hidden and
output nodes.
Neural network research is motivated by two desires: to obtain a better
understanding of the human brain, and to develop computers that can deal with
abstract and poorly defined problems. For example, conventional computers
have trouble understanding speech and recognizing people's faces. In
comparison, humans do extremely well at these tasks.
Many different neural network structures have been tried, some based on
imitating what a biologist sees under the microscope, some based on a more
mathematical analysis of the problem. The most commonly used structure is
shown in Fig. 26-5. This neural network is formed in three layers, called the
input layer, hidden layer, and output layer. Each layer consists of one or
more nodes, represented in this diagram by the small circles. The lines
between the nodes indicate the flow of information from one node to the next.
In this particular type of neural network, the information flows only from the
input to the output (that is, from left-to-right). Other types of neural networks
have more intricate connections, such as feedback paths.
The nodes of the input layer are passive, meaning they do not modify the
data. They receive a single value on their input, and duplicate the value to

4
is duplicated and applied to the next layer. The active nodes of the output
layer combine and modify the data to produce the two output values of this
network, and .X3
1
X3
2
Neural networks can have any number of layers, and any number of nodes per
layer. Most applications use the three layer structure with a maximum of a few
hundred input nodes. The hidden layer is usually about 10% the size of the
input layer. In the case of target detection, the output layer only needs a single
node. The output of this node is thresholded to provide a positive or negative
indication of the target's presence or absence in the input data.
Table 26-1 is a program to carry out the flow diagram of Fig. 26-5. The key
point is that this architecture is very simple and very generalized. This same
flow diagram can be used for many problems, regardless of their particular
quirks. The ability of the neural network to provide useful data manipulation
lies in the proper selection of the weights. This is a dramatic departure from
conventional information processing where solutions are described in step-by-
step procedures.
As an example, imagine a neural network for recognizing objects in a sonar
signal. Suppose that 1000 samples from the signal are stored in a computer.
How does the computer determine if these data represent a submarine,
whale, undersea mountain, or nothing at all? Conventional DSP would
approach this problem with mathematics and algorithms, such as correlation
and frequency spectrum analysis. With a neural network, the 1000 samples
are simply fed into the input layer, resulting in values popping from the
output layer. By selecting the proper weights, the output can be configured
to report a wide range of information. For instance, there might be outputs
for: submarine (yes/no), whale (yes/no), undersea mountain (yes/no), etc.

6
w
7
FIGURE 26-6
Neural network active node. This is a
flow diagram of the active nodes used in
the hidden and output layers of the neural
network. Each input is multiplied by a
weight (the w
N
values), and then summed.
This produces a single value that is passed
through an "s" shaped nonlinear function
called a sigmoid. The sigmoid function is
shown in more detail in Fig. 26-7.
100 'NEURAL NETWORK (FOR THE FLOW DIAGRAM IN FIG. 26-5)
110 '
120 DIM X1[15] 'holds the input values
130 DIM X2[4] 'holds the values exiting the hidden layer
140 DIM X3[2] 'holds the values exiting the output layer
150 DIM WH[4,15] 'holds the hidden layer weights
160 DIM WO[2,4] 'holds the output layer weights
170 '
180 GOSUB XXXX 'mythical subroutine to load X1[ ] with the input data
190 GOSUB XXXX 'mythical subroutine to load the weights, WH[ , ] & W0[ , ]
200 '
210 ' 'FIND THE HIDDEN NODE VALUES, X2[ ]
220 FOR J% = 1 TO 4 'loop for each hidden layer node
230 ACC = 0 'clear the accumulator variable, ACC
240 FOR I% = 1 TO 15 'weight and sum each input node

x
-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7
0.0
0.1
0.2
0.3
b. First derivative
s`(x)
s(x)
FIGURE 26-7
The sigmoid function and its derivative. Equations 26-1 and 26-2 generate these curves.
EQUATION 26-1
The sigmoid function. This is used in
neural networks as a smooth threshold.
This function is graphed in Fig. 26-7a.
s (x) '
1
1%e
&x
EQUATION 26-2
First derivative of the sigmoid function.
This is calculated by using the value of
the sigmoid function itself.
s N(x) ' s (x) [1 & s (x) ]
Figure 26-7a shows a closer look at the sigmoid function, mathematically
described by the equation:
The exact shape of the sigmoid is not important, only that it is a smooth
threshold. For comparison, a simple threshold produces a value of one
when , and a value of zero when . The sigmoid performs this samex > 0 x < 0
basic thresholding function, but is also differentiable, as shown in Fig. 26-7b.