Tuyển tập công trình Hội nghị Cơ điện tử toàn quốc lần thứ 6 729
Mã bài: 157
Hệ thống tạo ảnh toàn nét và ứng dụng thời gian thực
trong các hệ robot cấp độ micro
All-In-Focus imaging and real-time microrobotic applications
Nguyễn Chánh Nghiệm
Trường ĐH Cần Thơ, e-Mail:
Văn Phạm Đan Thủy
Trường ĐH Cần Thơ, e-Mail:
Kenichi Ohara and Tatsuo Arai
Osaka University
Tóm tắt
Trong khoa học sự sống, việc quan sát và thao tác các vật thể vi sinh diễn ra rất thường xuyên và mang tính
lập lại trong đó việc điều chỉnh lấy nét là một yêu cầu tiên quyết. Nhiều giải thuật lấy nét tự động đã được đề
xuất để giúp thao tác viên giảm thiểu thời gian điều chỉnh lấy nét. Những giải thuật này cũng có thể được áp
dụng để tự động hóa các khâu vi cảm biến hay thao tác các vi vật thể như đo độ cứng của tế bào, gắp thả, hay
giữ cố định các vật thể di động. Bài nghiên cứu này đề xuất ứng dụng giải thuật tạo ảnh toàn nét để giúp tự
động hóa thao tác các vi vật thể trong khi có thể quan sát chúng được rõ nét trong thời gian thực. Thí nghiệm
gắp thả các vi vật thể với kích thước khác nhau được thực hiện để kiểm tra tính khả dụng của một hệ vi thao
tác tự động thời gian thực.
Abstract:
In life sciences, observing and manipulating various microbiological objects may be performed frequently
and repeatedly in which object focusing is the preliminary task of the operator. In order to reduce the manual
focusing time, various autofocus algorithms have been proposed. These algorithms can also be implemented to
automate microsensing and micromanipulation tasks such as measurement of cell stiffness, pick-and-place of
various microobjects, immobilization of moving objects, etc. This paper proposes the All-In-Focus algorithm
to automate micromanipulation of microobjects while they can be observed clearly in real-time. Pick-and-
place of single microobjects with different sizes is performed to demonstrate the effectiveness of a real-time
micromanipulation system.
an in-focus area in an image, a Micro VR camera
system had been developed to provide real-time
all-in-focus image which is a composite image
created by merging all in-focus areas from various
images of the observed object taken at different
730 Chanh Nghiem Nguyen, Dan Thuy Van Pham, Kenichi Ohara and Tatsuo Arai
VCM2012
focal distances [4]. This algorithm can thus be
called All-In-Focus (AIF) algorithm and is
classified into derivative-based category. The
system also provides a depth image in real time so
that 3D positions of microobjects can be obtained
to facilitate automated micromanipulation, e.g.,
automated grasping and transporting an 8 μm
microsphere [5].

The real-time micro VR camera system estimates
the depth from in-focus pixels extracted from a
series of images taken along z-direction. It is,
therefore, independent on the shape of the object.
There are, however, a few problems towards
obtaining accurate 3D information from this
imaging system. For example, there is a trade-off
between the frame rate and the accuracy of the
system. In order to achieve real-time detection,
fewer images are used to create the AIF image
which increases the resolution error. To capture
images at different focal position, an actuator is
used to move the lens in the optical axis. Vibration
from the actuator may also reduce the quality of

3D positions of transparent end-effector tips of
common microtools, as well as glass micropipettes,
and other micro biological cells. This helps the
All-In-Focus imaging system a versatile 3D
imaging system that can be integrated into a
micromanipulation system to provides not only
real-time extended depth of field with the AIF
image but also the 3D positions of transparent
microobjects to handle them automatically.

Fig. 2 Illustration of All-In-Focus algorithm

2. System overview
2.1 All-In-Focus imaging system
The All-In-Focus imaging system is developed
based on the Micro VR camera system [4] and
consists of a piezo actuator and its controller, a
processing unit to create the AIF and HEIGHT
image, and a high-speed camera attached to the
camera port of the microscope (Fig. 1). The piezo
actuator can move the objective lens cyclically up
and down over a
SWING
distance up to 100 µm
along the optical z-axis. When the system is
running, the high-speed camera (Photron
Tuyển tập công trình Hội nghị Cơ điện tử toàn quốc lần thứ 6 731
Mã bài: 157
Focuscope FV-100C) captures images at different
focal planes at the rate of 1000 frames per second.

the optical axis of the microscope. The
( , )
X Y

plane lies on the object plane and its X-axis and Y-
axis align with the horizontal x-axis and vertical y-
axis of the AIF image, respectively. The
relationship between the distance in
( , )
X Y
plane
and in the number of pixels of the AIF image is
obtained by measuring the pixel size of an AIF
image of a scalar.

Let
{20,40,60,80,100}
SWING  be the distance
over which the piezo actuator moves objective lens.
This distance is normalized into a gray scale from
0 to 255 in the HEIGHT image. Therefore, the z-
coordinate of a pixel at position
( , )
x y
can be
estimated from the corresponding pixel value
( , )
H x y
in the HEIGHT image as


frequency of scanning or the frame rate of the AIF
imaging system as
 
30
_ frames per second
frame rate
FRAME

(3)Fig. 5 Two-fingered microhand for dexterous
micromanipulation applications

The highest and lowest frame rate of the AIF
imaging system is 30 and 5 frames per second,
respectively (Eq. 3). With the lowest frame rate
when
6
FRAME

and with
20
SWING

(μm) the
best resolution of the system becomes
0.1
d
 

microobjects in a large workspace.

3. Measuring microobject position in 3D
3.1 Measuring 3D positions of end-effectors
Having an elongated shape, a few lines can be
detected along the microfinger in its AIF image.
The 2D position of the fingertip can be thus
obtained from these detected lines. The z-position
of the fingertip is estimated from the HEIGHT
image using the information of the detected lines.
The process is as follows.

(a) (b)
Fig. 6 (a) Microfingers and 55 μm microsphere.
(b) Detected lines superimposed on detected
microfingers Fig. 7 Line grouping using middle position of
lower endpoints of detected lines in x-
direction

3.1.1 Line detection
The two microfingers are set in the vertical
direction and inclined toward each other (Fig. 6).
Due to the shallow depth of field, only part of the
microfinger can be in focus. The curvature of the
surface of the microfinger functions as the surface
of a lens. Therefore, the middle region of this local
area will be brighter when it is in focus. This

classified as left-microfinger group if its lower
endpoint’s x-coordinate is smaller than
x_midpoint; otherwise, it belongs to the right-
m i c r o f i n g e r g r o u p .

3.1.3 Line-type pattern matching for fingertip
identification in 2D
The AIF imaging system needs at least 30 images
to create the AIF image in real-time at 30 frames
per second. The system can provide good AIF
observation of the microobject even when it is
moving. However, line detection for identifying
two microfingers of the microhand becomes more
difficult if it moves in high-speed. The edges along
the microfinger may form broken line segments
due to the limited processing speed of the AIF
imaging system hardware.
Tuyển tập công trình Hội nghị Cơ điện tử toàn quốc lần thứ 6 733
Mã bài: 157

Because the microhand is set in a vertical direction
in the image and three regions with different
intensity levels are observed for each microfinger
in the AIF image, the image intensity can change
either “from bright to dark” or “from dark to
bright” when going across a detected line from left
to right. This detected line is defined to be type
“0” and type “1”, respectively. Let L
1
, L

L
4

Line type 0 1 0 1 Table 2 Line-type patterns of 3 detected lines
Missed
line
Line type
l
1
l
2
l
3

L
1

1 0 1
L
2

0 0 1
L
3

0 1 1
L



tiptip
yx ,


tiptip
yx ,


yx,
fitted line
90

(a) (b)
Fig. 9 Pixel values from HEIGHT image along
inner line on left microfinger (a) and right
microfinger (b) at initial setup. Fitted line is
calculated from 80 points It is also possible that a line-type pattern of four
detected lines does not match with that in Table 1.
This can happen when the microhand is moving in
fast motion so that the two broken lines can be
found on the finger border (right finger in Fig. 8a).
In addition, a line can also be found from the ghost
of the microfinger border (left finger in Fig. 8b)
due to limitations of the AIF processing speed of
the hardware. In these cases, the line-type pattern

directly estimated from the gray value
( , )
H x y
tip tip
of the pixel at location
( , )
x y
tip tip
in
the HEIGHT image using Eq. 1. However, the
HEIGHT image is very noisy. Therefore, more
information is required to obtain accurate z-
position of the tip. In this paper, the angle of
734 Chanh Nghiem Nguyen, Dan Thuy Van Pham, Kenichi Ohara and Tatsuo Arai
VCM2012
inclination of the microfinger is utilized to obtain
accurate depth information of the fingertip.

Given the positions of the pixels which lie on a
line detected from the microfinger in the AIF
image, the pixel values in the HEIGHT image at
these positions are collected. A line is fitted from
the values of 80 pixels along the tip’s part of the
detected line. The angle of inclination of the fitted
line estimates the inclination angle of the
microfinger to the object plane. Figure 9 shows the
values of the HEIGHT image’s pixels along the
inner lines of the left microfinger and the right
microfinger. Because of the limited SWING range
of the AIF imaging system, only the upper part of


The inclination angle and depth information can be
obtained from either the border lines or the inner
lines. However, it is observed that the inner lines
are clearer and less broken especially when the
microfinger is in fast motion. For this reason, the
inner lines of a microfinger are used to estimate its
tip’s position in z-axis. If two inner lines can be
found for a microfinger after Line-Type Pattern
Matching, the z-position of the fingertip is
estimated from the fitted line with the smaller
regression error.

Since microfingers and micropipettes can be
fabricated similarly by pulling a glass rod or tube,
they may have similar elongated shapes. Thus, the
proposed method can also be applied to measure
the 3D position of a micropipette. However, a
micropipette may have less-invasive rounded
shape. Therefore, the method should be modified
to identify the position of the tip in the 2D AIF
image. Unlike the tip of a sharp microfinger, the
x-coordinate of the rounded tip of a micropipette
(pointing in y-direction) should be determined as
the average of the x-coordinates of the upper
endpoints of the detected lines on the micropipette.

3.2 Measuring 3D positions of target objects
The AIF imaging system can also be used to find
the 3D position of micro transparent objects.





(4)where
C
is the contour or the boundary of the
object in the AIF image and
C
n
is the number of
pixel points on the contour
C
.

In this paper, a glass microsphere is used as the
target object. The microsphere is transparent and
qualifies our assumption. Thus, its 2D contour in
the AIF image is detected as a circle using Hough
gradient algorithm [13].

4. Experimental methods
The performance of the AIF system depends on
the parameter
SWING
and
.


0
20
40
60
80
100
120
140
1
10
19
28
37
46
55
64
73
82
91
100
109
118
127
136
145
154
163
172
181

4.1.1 Depth measurement of the target object
Figure 10 shows the histogram of the gray values
of the pixels on the circular contour around a 55
μm microsphere in the HEIGHT image. Most of
the pixels (88%) have the gray value of 119 and
127. The standard deviation of these pixel values
is about 4.0. This corresponds to about 1.24 μm
which is about the same as the resolution of the
AIF imaging system at the chosen settings.
Therefore, the average gray value of all the pixels
along the detected circle in the HEIGHT image
can be used to find the z-coordinate of the center
of that microsphere using Eq. 4.

In order to evaluate the linearity against z-position
of the object, a microsphere was moved 60 μm in
z-direction with a step-distance of 2 μm. The plot
of measured z-position of the microsphere versus
its displacement is shown in Fig. 11. A high
linearity can be observed from the dotted trend
line.

4.1.2 Depth measurement of the microhand
A linear displacement of 30 μm in z-direction was
sent to the microhand and the measured z-position
of the moving microhand is shown in Fig. 12.
Good linearity of the measured data can also be
observed from the trend lines.

15

and right microfinger f2

4.3 Pick-and-place of different-sized
microspheres
As an application of the AIF imaging system,
pick-and-place task was performed to single
microspheres by using a two-fingered microhand
[6]. The microspheres are suspended in the water
on a glass plate to resemble biological cells in their
culture medium. The 3D positions of the two
microfingers of the microhand and of a
microsphere estimated from the AIF imaging
system helped automate the pick-and-place task.

Because the microhand was developed to have a
multi-scale manipulability, microspheres of 96 μm,
55 μm, and 20 μm in diameter were used. This is
736 Chanh Nghiem Nguyen, Dan Thuy Van Pham, Kenichi Ohara and Tatsuo Arai
VCM2012
also the size range of our currently interested
objects; for example, lung epithelial cells whose
stiffness was measured [8] were about 20 μm in
diameter.

In this experiment, the microhand is placed over
100 μm from a target microsphere in the 2D object
plane. It is manually brought to about the same z-
level of the microsphere and coarsely focused so
that both the microhand and the target object are
within the scanning range of the AIF imaging

distance
z

that is about the object
diameter (Fig. 13d).

Step 3:The microsphere is transported
100 μ
m
x
 
away from its position (Fig.
13e).

Step 4:The microsphere is moved down the same
distance
z

by the microhand and is
released (Fig. 13f).
5. Results and discussion
5.1 Real-time tracking of the microhand
The microhand was tracked for 500 image frames
in this experiment. The success rate was about
93.2%. The average computation time for
searching the microhand was about 14.5 ms. The
tracking frame rate was about 21 frames per
second. Thus, real-time tracking was achieved.

During tracking, the performance of LTPM was

FRAME
= 2, the resolution of the system was
about 1.3 μm which may not be suitable for a
perfect spherical object such as a 20 μm
microsphere. Since the experiment was performed
to evaluate the method of obtaining 3D
information from the AIF imaging system, no
treatment to the microfingers was performed to
overcome adhesion problem that might have
contributed to the decrease of the success rate.

The success rate might also attribute to the
vibration generated by the piezo actuator when
grasping smaller microspheres. In the case of a
microsphere, it can slide out of the two
microfingers while being grasped if large vibration
occurs. In the case of grasping a biological cell,
vibration may not affect much at the grasping step
since cells are generally adhesive. However,
releasing a cell will be more difficult. Using a
fingertip to push a cell which is adhered to the
other microfinger may help to successfully release
the cell.

Table 3 Pick-and-place performance for
microspheres of different sizes
Microsphere 96 μm

55 μm 20 μm
Success rate 90% 80% 74%

also be improved by increasing the value of
parameter
;
FRAME
however, this adjustment
lowers the frame rate and affects the real-time
performance of AIF imaging directly.

6. Conclusion
This paper presents the AIF imaging system which
is used to extend the depth of focus when
observing microobjects. In addition, it also
provides 3D information of microobjects being
observed. Thus, 3D position measuring techniques
have been proposed for both the end-effector and
the target object so that handling microobjects can
be automated.

As a potential tool for micromanipulation, a two-
fingered microhand was used in the experiment.
Line-Type Pattern Matching was proposed to
detect the 3D positions of the tips of the
microfingers.

Multisized microspheres were used as target
objects in the pick-and-place experiment and their
z-coordinates could be estimated with Contour-
Depth Averaging.

As AIF observation of microobjects and their 3D

Time Micro Observation Technique for Tele-
Micro-Operation, in IEEE/RSJ International
Conference on Intelligent Robots and
Systems, vol. 1, pp. 647–652, 2000
[5] Ohara K, Ohba K, Tanikawa T, Hiraki M,
Wakatsuki S, and Mizukawa M: Hands Free
Micro Operation for Protein Crystal Analysis,
in IEEE/RSJ International Conference on
Intelligent Robots and Systems, vol. 2, pp.
1728–1733, 2004
[6] Avci E, Ohara K, Takubo T, Mae Y, Arai T:
A new multi-scale micromanipulation system
with dexterous motion. In: Int symp micro-
nanomechatronics human science, pp. 444–
449, 2009
[7] Inoue K, Tanikawa T, Arai T: Micro-
manipulation system with a two-fingered
micro-hand and its potential application in
bioscience. J Biotechnol, vol. 133, no. 2, pp.
219–224, 2008
[8] Kawakami D, Ohara K, Takubo T, Mae Y,
Ichikawa A, Tanikawa T, Arai T: Cell
stiffness measurement using two-fingered
microhand. ROBIO, pp. 1019–1024, 2010
[9] Inoue K, Nishi D, Takubo T, Arai T:
Measurement of mechanical properties of
living cells using micro fingers and AFM
cantilever. In: Int symp micro-
nanomechatronics human science, pp. 1–6,
2006