tnt J Adv Manuf Technol (t996) 11:206-213 © 1996 Springer-Verlag London Limited

The international Journal of

Rdvanced manuPactuPin[I Technology

Vision-based Automated Inspection System in Computer Integrated Manufacturing F. L. Chen and C. T. Su Department of Industrial Engineering, National Tsing Hua University, Hsinchu, Taiwan, Republic of China

The industrial application of an automated inspection system that aims to enhance the efficiency and flexibility of a computer integrated manufacturing system (CIMS) is proposed in this paper. A machine-vision-based approach is adopted to utilise its advantages of measurement flexibility, high resolution, and non-destruction. With a closed-loop feedback control architecture, the inspection procedures are fully integrated with the production process to detect and to recover abnormal operations in a real-time mode. Implementation of such a system at a physical shop floor for socket inspection is presented as an example. Keywords: Automated inspection; Computer integrated manufacturing; Machine vision; Pattern recognition; Quality control

1, Introduction Computer integrated manufacturing (CIM) is widely accepted as a fundamental strategy employed for enterprises competing in the world market. The implementation of CIM systems, despite a considerable number of successful cases having been reported in many countries, remains a difficult task. A CIM system is typically a highly computerised system that consolidates all production and management functions in a company to form a more efficient unit, generally in an untended mode. The product quality and productivity can consequently be increased. In order to ensure a high quality product, it is essential to integrate an automated inspection system with manufacturing. Most current inspection systems achieve this quality control function by detecting deviations from specifications and relaying this information to a central computer. Fully automating the manufacturing system, however, requires that the inspection systems should possess the

Correspondence and offprint requests to: Fei-Long Chert, Department of Industrial Engineering, National Tsing Hua University, Hsinchu, Taiwan, Republic of China.

capability both to determine the possible causes of failures in manufacturing and to recommend solutions for these failures. The industrial application of an automated inspection system aimed to elevate the efficiency and flexibility of an automated manufacturing system is reported in this paper. A machinevision-based approach is adopted here which utilises its advantages of measurement flexibility, high resolution, and non-destruction. A closed-loop feedback control architecture is formed by fully integrating the inspection procedures with the production process. A defective part, once detected, is collected by a material handling device. An intelligent diagnostic algorithm based on statistics and heuristics is simultaneously applied to analyse the consecutive inspection results in order to deduce possible causes of defects. This system therefore not only eliminates defects but also recovers normal operations in a real-time mode. This proposed automated inspection system has been implemented on a physical shop floor for automatic inspection of wrench sockets of varied types and sizes. The machine vision system uses two cameras that perform multiple functions: 1. The first camera recognises the shape of the socket, and measures its dimensions and concentricity. 2. The second camera determines the amount of die wear. This system, with other automated devices, assists in the formation of a truly automated manufacturing system. The experimental results show that the machine vision system has satisfactory performance and flexibility in the factory environment. Even though this system is specially designed for socket inspection, the same methodologies and algorithms can be easily modified and applied for inspection of other components. Details of the inspection algorithms are discussed in this paper.

2.

Automated Quality Control

Quality control (QC) cannot be neglected during development of a fully automated system. The purpose of quality control lies in detecting and eliminating insufficient quality in manufacturing a product through the application of inspection. Quality

Vision-based Automated Inspection System

4os e

Stations

1 Computer System

Corrective Actions

]

Fig. 1. Architecture of an automated quality control system. control is traditionally accomplished by experienced operators who inspect the manufacturing components to ensure that specified design standards are satisfied. Primary limitations of manual QC inspections include their inflexibility, inconsistent results, and time consumption. In order to form a highly automated manufacturing system, such as a CIM system, several approaches to tile inspection function have emerged as potential quality control techniques for the future. These approaches use advanced computer-based sensor technologies along with sophisticated software development to automatically acquire and analyse inspection data [1]. The direct advantage of such approaches is their capability of becoming integrated with other automated equipment to form a truly automated production system. Other potential benefits include high speed of inspection, measurement flexibility, on-line inspection capability, and the possibility of 100% inspection to ensure 100% acceptable quality. Figure 1 depicts the basic architecture for a fully automated quality inspection system in a typical CIM environment. The key component in this figure is a central computer system that incorporates all operations within this quality control system. The computer system should also be capable of communicating with the higher level supervisory computer to form a fully integrated computer system. In order to compensate for the high cost of implementation, the automated inspection stations in a CIM system are generally designed with flexibility of inspecting products of various types. The specifications of these products, i.e. geometric features or dimensions, must therefore be distributed to the corresponding stations throughout the central computer system. The sensor measurement data obtained are then compared with these specifications to determine the acceptance or rejection of the products. The occurrence of a defective part does not invariably imply that an abnormal production process error has happened. It may be just a random error. The inspection results should therefore be relayed to the central computer for enduring statistical analysis and report generation. If the presence of a process error is revealed during analysis, corrective action can be invoked to eliminate it.

3. Machine Vision Inspection Several automated inspection techniques are currently available that can be classified into two categories, e.g. contact

207

type and non-contact type. Inspection methods of the contact type employ a probe to touch a measured surface to sense the departure of the surface element from a reference datum. The geometrical and dimensional elements of the part can then be calculated and evaluated from the measured data. A coordinate measuring machine (CMM) represents a general example of such an inspection method [2]. Non-contact inspection methods are applied when no physical contact with the product is permitted. Rather than using a touch probe, the non-contact methods employ optical or non-optical, i.e. ultrasonic or electrical, sensors to collect the product information. Non-contact inspection methods have the advantage of greater processing speed, implementation flexibility, and no damage to the part surface. Machine vision, among many non-contact inspection techniques, has received increasing attention over the past several years. Machine vision can be defined as the automatic analysis of images applied toward interpretation of a scene [3]. A basic machine vision system is composed of one or more cameras, digitising hardware, and a digital computer. The camera takes images of an object; these images then become digitised and stored in the computer memory by a device called a "frame grabber" for further analysis and interpretation. One of the primary applications of machine vision lies in inspection. The effectiveness of machine vision has made it possible to replace sampling inspection with 100% inspection. Other typical applications include part identification, orientation, and location. All functions can be integrated into a feedback control system for real-time control. Compared with other sensor technologies, machine vision systems have the advantage of flexibility because their applications are less restricted by the geometrical configuration of parts being inspected. Machine vision systems can be easily integrated with other automated machining or handling equipment as all image interpretations and analyses are processed within the computer system. This integration makes them suitable for inclusion in a CIM environment. The machine vision systems, however, are sensitive to environmental conditions such as lighting, dust and vibration (the relative motion of camera and inspected parts). To prevent disturbance from lighting and dust, the machine vision system can be implemented in a separate environment, such as being enclosed in a black-box, so that better images can be acquired. To solve the vibration problem, damping devices are generally used to eliminate, or to diminish, the amplitude of vibration. In the following section, the implementation of an automated inspection function employing machine vision is introduced.

4. Implementation Example The principal applications of machine vision in industry include pattern recognition, dimensional measurement, defect detection, and PCB inspection [4]. Basic research in machine vision is abundant, but its applications in machine shops is limited owing to the hostile manufacturing environment. The application of a machine vision system in a machine shop to perform part identification and precision measurement functions is introduced here. The primary products manufac-

208

F.-L. Chen and C. T. Su

raw m a t e r i a l

t_ 7-

Forging

f

1

-inspection

i .....

Inspection No., Discard

Discard

.

No

T Heat reatment External Machining I -internal Machining l

[__

l

Plating

Packing

Fig. 2. Production sequence of sockets. tured in this shop are cylindrical wrench sockets of various sizes. Figure 2 shows the production sequence of these sockets. First, a long steel pipe is cut into small workpieces and each workpiece then goes through a forging process so as to generate different shapes at two ends. One end of the sockets is invariably a square shape with a specific size for the wrench to fit. Figure 3 shows an example image of this end. The other end may have varied shapes. Some common examples are illustrated in Fig. 4. Following this step, these workpieces are transported via a conveyor to a turning centre for external and internal machining. The workpieces have to be sampled for inspection between the forging and machining processes. The features to be inspected include dimensions, concentricity, and the presence of die wear. A failure of any item to satisfy the specification requires that the forging process be interrupted for adjustment or to replace the dies. Following the machining process, the workpieces are sampled again for the second inspection. Only visual observation is performed at this time. The approved parts then go through heat treatment and plating procedures, and, finally, packing. The sequence shown in Fig. 2 represents a standard manufacturing routine of many socket production factories.

Fig. 3. An example image of a socket.

@ @ 0 Fig. 4. Examples of socket shape.

For a production sequence of this type, most labour-intensive operations performed in the shop have been successfully replaced by highly computer-controlled equipment. The inspection operations are the only ones not yet automated. The primary reason for this is the complexity of the inspection process. As mentioned earlier, the inspection station following the forging process verifies three items, e.g. dimensions, concentricity, and the presence of die wear. Conventionally, the dimensions are inspected using go and no-go gauges; concentricity is verified by a specially designed hand tool, whereas the detection of die wear relies on visual observation. An automated inspection device to replace human labour must be able to verify all these features. As new automated machines are able to manufacture products of several sizes and shapes, the selected inspection device must also possess the flexibility to measure various parts. Among existing methods, machine vision is probably one of the most feasible solutions because of its low cost, good measurement capability and flexibility. For this reason, in this research we attempted to implement the on-line inspection concept in a socket production line using machine vision techniques. The implementation of such a system to achieve the quality control function is discussed in detail in the following parts of the paper. The proposed inspection station for automated quality control is composed of a machine vision system, a mechanical arm, and an X Y table. The machine vision system includes two CCD cameras, a frame grabber board installed in a PC/ 486 computer, which functions as the central computer of the entire inspection system. The system flow of the inspection function is illustrated in Fig. 5. The sockets generated by the forging process are first sampled and transported to the XY table via the mechanical arm. A camera then captures the image of this socket. Following some image processing procedures such as noise reduction, enhancement, and segmentation, pattern recognition algorithms are applied to identify the shape of the socket. The reference information, including dimensions and tolerances, of the shape are retrieved from a prepared database. Inspection algorithms are then employed to calculate the dimensions and concentricity of the sampled part. The acceptance or rejection of the part can be determined

Vision-based A u t o m a t e d Inspection System

work well when applied on the factory floor, which generally lacks a well-controlled environment, and therefore the algorithms need to be modified to fit the real conditions. Figure 6 shows the inspection sequence developed in this research. Major steps involved in this sequence are discussed in the following sections.

Image Capturingl ~ P r o ~ ~

Image Processing ~

ISpeeification~_.~ , _ 1 LDatabase~ ---~pattern Recognition-~

5.1.1

Dimension&~

Warning -

+

t 'ns'°eti°nJ

1 Department

Warning Next Process Fig. 5. Systemflowof automatedinspectionfunction. by a comparison of the inspection results with the reference data. As the socket has various shapes at two ends, the above procedures have to be repeated once the socket has been turned around by the mechanical arm. The part, having passed dimensional and concentricity inspection, is fed by movement of the X Y table into the next station where the second camera is located. In this station, the sampled part is checked again for die wear inspection. Die wear is observed from accumulated experience to occur normally around the corners. Only the inside corners of the socket therefore need to be tested. As the previous camera is arranged to capture the entire image of the socket, employing the second camera to take corner images of the socket can significantly enhance the measurement resolution. A part that passes die wear inspection is transported to the turning process. Otherwise, a warning message is initiated. Following a sampled part's completion of inspection operations, the inspection results are recorded into a database for future use. Additionally, statistical reports can be generated anytime at the request of the quality control department.

Locate the Outside Circle

The first step in socket shape recognition is to locate the position of the external circle. This task can be done by general edge detection algorithms, such as the Sobel operator. In this research we adopted a simplified edge-detection method that defines a threshold value for the variation of grey level. If the variation between one pixel and neighbouring pixels is greater than the threshold, this pixel is labelled as an edge pixel. This detection routine is easily demonstrated as follows: if [gi - gi+Ai[ ?> Tthen gi is labelled as edge, otherwise gi is not labelled as edge T: threshold value Ai = 1..... k k is a gauge number

(1)

After the edge points are detected, a four-dimensional chain code representation is applied to connect these points. Following this step, the centre (xc,y,) and radius of the

Locate the circle

IDetect edge o f [ inner p o l y g o n

I

IPlot frO) c u r v e [____

5. Inspection Algorithms The algorithms for pattern recognition and measurement of dimensions, concentricity, and die wear are discussed briefly in the following sections.

5.1

209

~

Smooth

r(O) curve

l

1

Pattern Recognition

For socket inspection, the most fundamental task is to recognise the shape of the inner hole. There are many existing algorithms developed for shape recognition. These algorithms may perform perfectly in the laboratory, but they may not

Identify pattern ( e d g e no.)

Fig. 6, Procedureof pattern recognition.

210

F.-L. Chen and C. T. Su

outside circle are estimated by applying the algorithm of Thomas and Chan [5]: 1,'2W 1 - - l ~ l W 2

x~

(2)

Ult/'2 -- U2~ 1 UlW2

y~ -

-- U2W 1

(3)

U1122 - - U 2 U 1

Pc =

-~

For a regular part, the geometrical centre should be near the circular centre. The distance between these two centres denotes eccentricity. If this distance is larger than a predefined value, it is concluded that the part is defective. After having identified the geometrical centre, we can construct the distance-versus-angle function r(0) as follows: r(o,) = .j((x; - gxc) 2 +

-

(7)

where 0i = tan-l((yi - gyc)/(x~ - g~c))

Xi i

l 11/2

+ XY2-i

2ycXYii

+ Ny2]I

(4)

where

The following step is to distribute the edge data into equal intervals. In this research, the whole circumference was equally divided into 360 intervals. If there is more than one datum in an interval, the average value of these data is taken. Finally, the relationship plot can be generated like the examples shown in Fig. 7.

5.1.4 Smoothr(O) Curve

vl = 2(~jx,~ffi Y, - N~i xcvi) Uz = 2(~x,~y~ - N~ffi x~yi)

\

i

\

i

i

i

i

i

xz, EY~ - Eyi + Ey,~,y~-

w2 :

i

i

i

x~y,

i

where N is the number of circumference points estimated.

5.1.2 Detect Edges of Inner Shape To detect the edges of the inner shape, the same procedure described above is applied. As a socket is symmetrical in shape, it is reasonable to start the search procedure from the centre point obtained in the previous step.

After the distance-versus-angle function r(0) is generated, shape recognition is easily achieved by counting the number of peaks and valleys of the curve. As there may exist much local noise, the correct number of peaks and troughs cannot be counted without smoothing the relationship curve. For example, Fig. 8 shows the distance-versus-angle plot of the inner polygon of a socket in a noisy environment. In order to recognise the shape, we need to count the number of peaks or troughs. It seems extremely difficult to do so from Fig. 8. To solve this problem, in this research we developed a smoothing algorithm using time series analysis to eliminate noise. The analysis of a time series that exhibits "long-term" variation of the mean depends on whether one seeks to measure the trend, or to remove the trend in order to analyse the local fluctuation. A common procedure to treat a trend is to use a linear filter that converts one time series {xi}, into another {Yl}, by a linear operation:

Yt = ~ a,x,+,.

(8)

r=--q

5.1.3 Plot r(O) Curve In order to recognise varied socket shapes, a so-called distanceversus-angle signature, r(0), was adopted in this research. A signature is a one-dimensional representation of a boundary. After being transferred from two dimensions into one dimension, the boundary becomes much easier to describe and to analyse. The distance-versus-angle signature is generated by calculating the distance from the centroid to the boundary and plotting it as a function of angle. To generate the r(0) function, the geometrical centre, (gxc,gyc), of the inner shape is calculated and applied as the centroid: N

gxc = .Y__,x,

(5)

i=1 N

gyc

= £

Y,

(6)

i=1

where N is the number of collecting edge points. More edge points ensure high accuracy of the calculated centroid.

where {at} are a set of weights. In order to smooth local fluctuation and to estimate the local mean, we choose the weights so that £ar = 1; then the operation is referred to as a moving average. Moving averages are generally symmetric with s = q and aj = a_j. The simplest example of a symmetric smoothing filter is the simple moving average for which ar = t/(2q + 1) for r = - q ..... q, and Sm(xt) = the smoothed value of xt

Sm(xt)

1

+q

2q + 1 ~ Xt+r

(9)

r = --q

Another approach is to fit a polynomial curve to a portion of the data instead of the whole series. For example, one could fit a cubic polynomial to the first (2q + 1) data points, with q--> 2. Then we use the polynomial to calculate the interpolated value at t = q + 1, and repeat the operation. Suppose that there are N data denoted Xl,...,XN. For a symmetric filter, we calculate the smoothed value from t = q + t to t = N - q, and 2q data are lost. Consider Fig. 8,

Vision-based Automated Inspection System

211

9.86-

8.91" 8.59-

7367.~ 7.127,00 • 6.68"

r

60

L

1~0

120

......

240

,

,

300

360

8

I~0) 10.67I0.42" 16.16"

9.919.65" 9.4&

9,14g.gS~,63" 8.3"7-

r(0) 7.62 +L49 7.36 7.23 7.10 6.97 6.84 6,7I 6.59 6,46 6.33 60

120

180

240

300

360

r(o)

7.01 e ~9 6.77 ~65 652

6.28 6.t6 e~ 592

Fig. 7. Distance-versus-angle plots of socket images. which has a sine function with a random fluctuation, as an example. This curve is smoothed with a symmetric linear filter, for which q = 12 and there are 24 lost data points. The result is given in Fig. 9. Even though noise is significantly diminished, it is still unsatisfactory. A modified method is developed to improve it. Assuming that the domain of the

curve is [-%oo] and that there exists some period p, we expand the observation data to N + 2q by adding 2q data. By the same operation from (q + 1)th data up to (N + q)th data, the plot with the original N observed data is smoothed out, as shown in Fig. 10. The peaks and troughs can then be counted without confusion.

212

F.-L. Chen and C. 7". Su .185

Iv

,

~LB3 ,884

.842 .IB21

h,

- .E28 -.841

Fig. 11. Die wear measurement in socket corners.

-,862

ax + by + c' = 0. The distance between these edges then becomes

-.IB83 -.184

~ .80

.05

.1

.15

,20

-J-.25 X

.31B

.35

.48

.45

,58

le 3

d

Fig. 8. Distance-versus-angle plot in a noisy environment,

Ic - c'[ \/(a 2 + b 2)

(10)

If the boundary is composed of circular arcs, the centres and radii of the arcs can be estimated using Eqs (2)-(4).

183

.l&2

.B90

5.3 Amount of Die Wear

.868 ,845 .823 >

,8B1 -.022 -.844 -.066 -.888 -.111

.83

.1

,15

,

,

,

k

,20

.2fi X

.30

,35

.4(I

.45

.50

18 3

.40

.45

,50

10 3

Fig. 9. Original moving average. .112

IB 3

.890

d

I

'

.00

.85

'

'

'

.15

.20

.

.

.

.

.B68 , B45 .B23

- .822

-.844 -,866

-. 088

Die wear, which normally occurs around corners, induces geometrical variations in socket corners. The amount of die wear can therefore be calculated by verifying every corner of the socket (see Fig. 11). A second camera is employed here to enhance measurement resolution. As the camera captures only one corner at a time, the XY table has to be relocated several times so that every corner can be inspected. In this system, the first camera cannot only recognise the socket shape but can also determine the position and location of the socket. The X Y table can therefore relocate according to the determined position and orientation. For each corner, the amount of wear is tested along the diagonal direction. If the summation of wear of any two symmetric corners exceeds a controlled value, this socket is classified as being defective, and a warning signal is given to call for a replacement die. The automated inspection system described above has been implemented on the factory floor using a Dipix P360 frame grabber board, which can connect up to four cameras at once, for image acquisition and processing. The computer can function as the central control unit of several inspection stations as more than one board can be simultaneously installed on the PC/486 computer. The total processing time in this case, including image processing and material handling, is approximately two minutes. This system, when compared with a human worker, shows a markedly more consistent inspection capability.

-.111 .1

.25 X

.30

.35

Fig. 10. Modified moving average. 5.2

Dimensional Measurement

The dimensions of an equilateral polygon are calculated with the edge equations of the polygon being determined with a least-square-error method. Two opposite edges are assumed here to follow the equations a x + b y + c = O and

6.

Conclusions

Automatic inspection plays an important role in the formation of a computer integrated manufacturing system. The development of an automated inspection system for quality control using machine vision is reported in this paper. The quality control function was achieved by inspecting the parts so as to detect the presence of improper production processes and relaying this information to the central computer. Implemen-

Vision-based Automated Inspection System tation and further modification of this system on the factory floor is currently in progress. The machine vision system, from experimental results has attained satisfactory performance and flexibility in the factory environment. Additionally, the cost of implementation is less than most sensor-based techniques. From these observations, it is believed that machine vision possesses great potential to assist the formation of a truly automated quality control system.

Acknowledgement The funding of this research was provided by the National Science Council of the Republic of China under grant No. NSC82-0422-E-007-262.

213

References 1, M. Shiraishi, "Scope of in-process measurement, monitoring and control techniques in machining processes - Part 2: In-process techniques for workpieees", Precision Engineering, 11(1), pp. 27-37, 1989. 2. H. Kunzmann and F. W~ildele, "Performance of CMMs", Annals of the CIRP, 37(2), pp. 633-640, 1988. 3. A. Barr and E. A. Feigenbaum, The Handbook of Artificial Intelligence, William Kaufmann, Los Altos, California, 1982. 4. R. M. Haralick and L. G. Shapiro, Computer and Robot Vision, Vol. I, Addison-Wesley Publishing Company, Reading, Massachusetts, 1992. 5. S. M. Thomas and Y. T. Chan, "A simple approach for the estimation of circular arc center and its radius", Computer Vision, Graphics, and Image Processing, 45, 1989.