Machine Vision and Applications (2012) 23:111–121 DOI 10.1007/s00138-011-0338-8

ORIGINAL PAPER

Machine vision system for automated spectroscopy Eranga Ukwatta · Jagath Samarabandu · Mike Hall

Received: 25 November 2009 / Revised: 16 January 2011 / Accepted: 12 April 2011 / Published online: 7 May 2011 © Springer-Verlag 2011

Abstract This paper describes a novel system based on the machine vision and machine learning techniques for fully automated, real-time identification of constituent elements in a sample specimen using laser-induced breakdown spectroscopy (LIBS) images. The proposed system is developed as a compact spectrum analyzer for rapid element detection using a commercially available video camera. We proposed a correlation-based pattern matching algorithm for analyzing single element spectra. However, the use of a high-speed laser and presence of numerous imperfections in the experimental setup require advanced techniques for analyzing multi-element spectra. We cast the element detection problem as a multi-label classification problem that uses support vector machines and artificial neural networks for multi-element classification. The proposed algorithms were evaluated using actual LIBS images. The machine learning approaches yielded correct identification of elements to an accuracy of 99%. Our system is useful in instances where a qualitative analysis is sufficient over a quantitative element analysis. Keywords Laser-induced breakdown spectroscopy · LIBS · Element identification · Machine vision · SVM · Multi-label classification E. Ukwatta (B) · J. Samarabandu Department of Electrical and Computer Engineering, The University of Western Ontario, London, ON N6A 3K7, Canada e-mail: [email protected] J. Samarabandu e-mail: [email protected] M. Hall Checkfluid Inc., 116-4096 Meadowbrook Drive, London, ON N6L 1G4, Canada e-mail: [email protected]

1 Introduction Rapid element identification in an in-situ environment is indispensable for numerous industrial and military applications. LIBS is one such emerging technique used in determining the element composition of a sample specimen. LIBS is a method of atomic emission spectroscopy [5] that uses a highpowered laser to excite microscopic amount of solid, liquid, or gaseous material, to spectrally analyze the plasma. The spectral composition of the plasma depends on the constituent atomic species of the ablated material. Elements can be uniquely identified based on the spatial and temporal resolutions of their emissions. The analysis can range from a simple identification of elements to a more detailed determination of absolute masses or relative concentrations. LIBS technique offers fast and real-time analysis with no sample preparation thus giving rise to numerous applications in industry, such as chemical and biological hazards analysis, radioactive contamination analysis, blood and tissue analysis, dental analysis, space exploration, explosive residues analysis, aerosol analysis, and many others [13,16,20]. Although the LIBS method has been in existence for approximately 45 years, interest was centered solely on the basic physics of plasma formation prior to 1980 [5,11]. However, during the last three decades, LIBS has developed rapidly as an analytical technique and has endured a dramatic transformation in terms of hardware, software, and areas of application [11]. This is largely a result of significant technological developments in lasers, detectors, and spectrographs used in LIBS instruments, as well as emerging needs for new methods of analyzing materials under conditions unachievable by conventional analytical techniques. Industrial equipment that employ LIBS systems tend to be expensive and non-portable as they utilize high- powered

123

112

E. Ukwatta et al.

(a) Emission signal

(b) Spectral image

Fig. 1 Emission spectra of a calibrated neon light source

lasers, built-in spectroscopes, and computers for post processing. In addition, a growing interest for in situ element identification demands automated and portable systems [11]. In this paper, we describe a fully automated system for detecting constituent elements of a sample specimen using machine vision and machine learning techniques developed using a video camera. The use of a commercial camera reduces the cost of the system and permits a more compact and portable system. In particular, the development of our system is motivated by the requirement for a man-portable system, for in situ metal impurity detection on machine lubricants flowing through oil valves, even though the algorithms described in the paper can easily be generalized to identify any kind of material as well. We used a CMOS sensor and a diffraction grating setup to transform the classical one-dimensional frequency spectra into two-dimensional space. Resulting images contain intensity peaks (blobs) that correspond to the spectral peaks of a plasma emission. Figure 1 shows the frequency spectrum and the corresponding spectral image obtained by exposing the grating-camera setup to a neon light source. The intensities of the blobs are dependent on the amount of material excited. When the amount is less, only prominent spectral peaks are observable. However, we need to examine only the presence of intensity peaks for qualitative analysis of elements. Afterwards, the regions of interests (ROIs) are extracted from the images based on the knowledge of the diffraction grating. We proposed two algorithms for processing the LIBS images in real time within an embedded system of a camera: (1) a pattern matching technique based on the correlation with known emission patterns to detect single emission spectra, (2) multi-label classification method detecting multiple elements simultaneously. The proposed algorithms for metal identification were evaluated using an actual experimental setup. The robustness of the algorithms were also evaluated on the effect of noise using a simulation setup.

123

The key contributions of our work include: – A novel compact spectrum analyzer design using a video camera with pixel-addressable capability and a highspeed laser, – automatic identification of a single element from a predetermined set of elements, as well as identifying multiple elements with overlapping spectral peaks, and – use of multi-label classification approach to metal spectroscopy that enables multi-element analysis. The rest of the paper is organized as follows: We will begin Sect. 2 with an overview of the existing methods. Our experimental setup is described in Sect. 3. Section 4 discusses the algorithms used to detect elements. In Sect. 5, we show that using a setup such as ours, detection results to an accuracy of 99% could be obtained. The paper is concluded in Sect. 6, along with an examination of future work.

2 Related work The existing literature in the realm of LIBS is concerned with improving the detection of spectroscopes. There is algorithmic progress in addition to improvements in hardware instrumentation. An increased interest in LIBS in recent times has focused on developing compact, low-powered systems. These developments are promoted by an increased interest for applications such as space exploration [21] and military applications [29]. Here, we investigate the literary background of LIBS under two considerations: hardware instrumentation and detection algorithms. LIBS systems not so long ago employed either photo multiplier tubes (PMT), intensified photodiode array (IPDA), or charged coupled devices (CCD) as detectors [5]. However, in such setups, the detector was used in vertical binning mode with the intensities along added to

Machine vision system

produce an intensity at each wavelength. The generated spectrum is one dimensional and limited in bandwidth. Using an Echelle spectrograph, the two-dimensional capabilities of the detector may be utilized to increase the spectral range significantly [11]. Our setup comprises of a diffraction grating array and a CMOS sensor to exploit the two-dimensional capability of the detector. The diffraction grating array resembles the operation of the Echelle spectrogram by diffracting light into several orders, thereby permitting two-dimensional spectral analysis. Nd:YAG lasers are the most widely used lasers in LIBS applications [11]. They generate high-energy pulses and operate at a pulse rate of 10–20 Hz. However, a low-powered Microchip laser is used in our setup, as the cost is significantly lower, its size is notably smaller, and it may be used in more rugged environments. The chosen Microchip laser operate at 1 KHz which is approximately about hundred times faster than a Nd:YAG laser. The use of a Microchip laser increases probing efficiency, even though it complicates the spectral image. When the laser operates at 1 KHz, with a frame rate as low as 30 frames/s, an average composite spectrum of multiple elements is observed at the CMOS sensor (see Fig. 4). This makes the element detection task more challenging. In the area of LIBS literature, correlation-based spectral matching technique is the ‘workhorse’ for element detection. An unknown sample is identified by correlating its spectrum with a library of spectra. Lentjes et al. [14] presented such a single- and multiple-shot analysis technique. They correlated multiple single-shot spectra with each reference spectrum to obtain a histogram of correlation values. If these distributions of correlation have no overlap across different reference spectra, they can declare a detection. Gornushkin et al. [9] used both linear and rank correlation methods to identify samples with similar chemical composition. The correlation plot corresponding to the highest correlation probability is taken as the matching library spectra. A lesser distinctive technique than correlation analysis is peak detection. For example, Barbini et al. [1] used peak detection for soil analysis. Ferrero et al. [6] proposed an alternative approach based on the algebraic determination of unknown spectral coordinates with respect to a spectral library base. We applied a simple pattern matching algorithm for element detection [26]. However, this method is not sufficient for multi-element detection due to the imperfections in the experimental setup, such as the presence of multiple metals, impurities, unknown emissions, missing blobs, high CCD noise, changes in illumination, camera tremor, and variations in the point spread function. We observed that the threshold used to declare a detection of elements requires adjustment whenever the experimental setup is slightly changed (e.g., camera positioning and alignment) so that calibrations are often necessary.

113

Samek et al. [22] used principle component analysis (PCA) and Mahalanobis distance measures in a discriminant analysis to match spectra of unknown samples to a library spectra of calibration samples. The material is classified as the closest match, or no match at all based on the likelihood of the spectrum matching. Sirven et al. [24] applied PCA and neural networks to analyze three chromium-doped soils to discriminate the soils. The potential of neural networks for qualitative classification of three different soils was investigated and compared with PCA. Neural networks were shown to be more efficient, particularly in the case of real world, noisy and variable spectra, with correct identification rate of 100%. However, the techniques mentioned so far for detecting elements, use commercially available spectrometers to gather the spectra to a computer to perfom the analysis. Owing to the use of sophisticated hardware and instrumentation, these spectrometers are capable of obtaining a noise-free spectrum. Because we build a spectrum analyzer using simple hardware, more robust techniques are required for analyzing blob patterns. We proposed a machine learning technique for the element detection task to adapt to the non-linearities in our setup [27]. In the single-label classification problems, classes are considered to be mutually exclusive. However, in spectral analysis, a spectrum could contain emissions from multiple elements such that the disjointness of the labels is no longer valid. The multi-label classification method [25] is increasingly used in applications such as text classification, semantic scene classification, music categorization, and protein function classification where a sample may belong to more than one class. For example, Boutell et al. [3] used multi-label classification on semantic scene classification, where a photograph could simultaneously belong to more than one conceptual class, such as beaches and sunsets.

3 Experimental setup The schematic diagram of our actual experimental setup is shown in Fig. 2. The LIBS system comprises a low-powered laser that ablates a microscopic amount of material. The generated plasma contains signature wavelengths that correspond to its constituent elements. Afterwards, it uses an array of optical devices and a diffraction grating array to spatially resolve the spectrum. The bandwidth of the diffraction grating array is shown in Table 1. We use multiple diffraction arrays to capture a wider bandwidth for detecting more elements. A CMOS sensor is used at the detection stage. The captured video stream is further processed within the camera. Figure 3 shows such acquired LIBS images for Hg, He, Ne, Na, Cu, and Al. We show that, using markers and computing homographies, correct alignment of the scene and image planes can be achieved to detect elements. After

123

114

E. Ukwatta et al.

Fig. 2 Schematic diagrams of the LIBS prototype

Data transfer to computer

Delay generator

Mirror

Microchip laser @1KHz

Elphel 353 camera

Focusing lens

Data processing in camra NIST

CMOS

Sample specimen

Diffraction grating

Optical fiber

Range (nm)

*

*

Table 1 Bandwidth of diffraction grating array Minimum (nm)

Maximum (nm)

526

674

148

0.123

387

487

100

0.092

(a)

Resolution (nm)

350

455

105

0.0838

250

350

100

0.102

*

(b) (c) * - Element ablated Fig. 4 Laser pulses and exposure time. The waveforms are as follows: a 1 KHz laser pulses, b camera exposure, c signal of a given pixel of the sensor

(a) Hg

(b) He

(c) Ne

(d) Na

(e) Cu

(f) Al

Fig. 3 LIBS images of actual experiment

the alignment is made the ROIs are extracted from LIBS images. The algorithm that resides within the camera detects intensity patterns and declares the elements present in the sample. The camera is aware of the reduced NIST atomic spectra database [19]. Since we have the ground truth, which is provided by the NIST database, and information about the ablated elements, we are able to evaluate the accuracy of the detection algorithm.

123

Unlike a typical LIBS setup, our setup allows a highspeed, low-powered laser to be used to excite microscopic amounts of elements. As illustrated in Fig. 4, our laser operates approximately 33 times faster than that of the frame rate of the camera. Note that there would be multiple laser pulses that fall into one cycle of the exposure time of the camera. Therefore, the observed image at the sensor is time averaged. This enables the LIBS signal to be stronger, since the photon energies of the single-shot LIBS emissions are very weak. However, due to this composite element spectrum obtained, the task of detection of elements becomes more challenging. 3.1 Pattern alignment For every video frame received, the image processing module finds the matching patterns from the pattern repository. However, the observed image and the canonical point locations in the repository need to be registered first, as the image plane and the grating may not in general be parallel. As a

Machine vision system

result, there is perspective distortion between the grating and a canonical rectangle. This distortion can be modeled by a planar homography [10,17] using at least four correspondence points. We use four calibration markers that indicate beginning and end points of the spectral coverage provided by the primary gratings. The homographies are computed using the direct linear transformation algorithm [10,27]. A homography is computed at the beginning of the experiment to enable correct wavelength to spatial mapping and to account for slight changes in camera viewpoint. Once the homography is calculated, the location of a point in the pattern repository in the captured image may accurately be computed. In addition, since the relative intensity and spread of each blob are known, we may locally replicate a blob which is similar to what is created by ablation. In other words, using the captured image, we may query for the presence of each pattern in the repository. 3.2 Extracting the regions of interest We achieve higher frame rates by limiting processing of the full-frame video to a specified region. Our method involves the extraction of ROIs from raw image frames of the camera sensor based on the knowledge of the diffraction grating that was used. The main reason to increase the processing speed is to maintain a higher frame rate on the camera, so that the camera would accumulate fewer element ablations for a single LIBS image. The use of raw image data eliminates the need for extra processing steps (e.g: de-mosaicing, gamma correction). Further analysis could also be carried out by coupling the spectral response characteristics of the sensor with Bayer color pattern [23]. We consider two methods of obtaining the regions from a full frame image. One method is the extraction of individual blobs in the image, whilst the second method obtains full rectangular stripes of the image. We exploit the behavior that all blobs fall into finite number of spectral lines in the second method. Table 2 shows the computational times of blob extraction for both methods. Note that by extracting the

115 Table 2 Computational-time comparison for blob extraction Method

Region type

Average time (μs)

Rectangular stripes

400 × 20

3,722

Individual blobs

12 blobs , 20 × 20

21,883

20 blobs , 20 × 20

36,109

500 × 20

4,296

rectangular stripes, the process may be sped up by approximately four times. Yet one more step we perform is to subtract the background noise from the image. This could be achieved by subtracting the background image which was taken when no material is ablated, from the ROIs on the current image frame. For our work, we use Elphel 353 model network camera [7] which has an embedded computer, running open source (GNU GPL) software on a Linux operating system and a general purpose reconfigurable FPGA with HDL code not bound by commercial intellectual property licensing. The camera is equipped with an Aptina MT9P001 3MPix CMOS monochrome sensor, an Axis ETRAX FS processor, and a 64 MB DDR SDRAM. Unlike typical commercial cameras, we possess complete access to the camera internals and camera firmware, thus permitting us to develop our own programming code to execute on the camera.

3.3 Quantum efficiency of the sensor Figure 5a shows the variation of the quantum efficiency of our CMOS sensor as a function of the wavelength of incident light [18]. Quantum efficiency has an implication on element detection capability, which potentially limits the detection of elements whose emissions lie only at frequencies of lower quantum efficiency. We correct for the variations in the quantum efficiency by multiplying the curve (Fig. 5a) with an inverse curve to obtain an approximate uniform distribu-

Fig. 5 a Quantum efficiency of the monochrome sensor, b effect of quantum efficiency on blob patterns: blob with low relative intensity (1,100) has high image intensity than a blob with high relative intensity

123

116

E. Ukwatta et al.

tion in the calibration phase of the spectrometer. However, it should be noted that the noise is being amplified as well.

4 Element detection algorithms Two approaches are considered for blob-pattern analysis: (1) a simple cross-correlation-based pattern matching technique detecting a single element at an instance and (2) a machine learning-based technique for multi-element identification. We describe each of these methods in detail in the following sections. 4.1 Simple pattern matching Assume that there are K patterns and each pattern pk , k = 0, . . . , K − 1 contains n k blobs (or points). Similarity between a blob and the corresponding image region is performed by computing the scalar product and normalizing by the squared sum of the kernel g(s, t). Let us assume there is a blob present at (x0 , y0 ) location of the unknown image pattern δ(x, y). Then, the similarity score is obtained as following: Similarity score =

ws 

wt 

g(s, t)δ(x0 + s, y0 + t),

s=−ws t=−wt

(1) 2 2 −( s 2 + t 2 )k 2σs 2σt

where g(s, t) = αk e , s ∈ [−ws , ws ], and t ∈ [−wt , wt ]. αk is the coefficient which accounts for quantum efficiency. The procedure for pattern matching technique is shown in Algorithm 1. Algorithm 1 Pattern matching algorithm Ensure: image exists  image is a frame from video stream 1: Detect markers. 2: Compute H. 3: Extract rectangular ROIs 4: for k ← 0, N − 1 patterns do 5: for i ← 0, n k points do 6: Calculate similarity scor e between thresholded image region and blob (eq. 1) 7: end for 8: scor e ← average similarity. 9: if scor e > threshold then 10: Declare a match. 11: end if 12: end for 13: Print k of matched patterns

At the beginning, the markers are detected and homographies are computed. After the accurate alignment is obtained, ROIs are obtained from raw images of the sensor. Assuming for simplicity, that the point spread is an anisotropic Gaussian

123

function, this algorithm computes a score which is related to the correlation value. This ensures that low intensity blobs pertaining to insignificant spectral responses have less influence on the score. Finally, patterns are declared based on a user-defined threshold. 4.2 Multi-label classification Traditional single-label classification involves learning from a set of examples that are associated with a single label l from a set of disjoint labels L = {l1 , l2 , . . . , ln }, |L| > 1 [3]. If |L| = 2, then the learning problem is deduced to a binary classification problem, while |L| > 2 is regarded as a multi-class classification problem. However, in multilabel classification, each training sample x is associated with a subset Px ⊂ L of possible labels. Thus, unlike multi-class learning, samples are not mutually exclusive as multiple labels may be associated with a single sample. A straight forward method to achieve our goal of correctly classifying the element signatures is to consider the signatures of elements corresponding to multiple elements as a new class and build a model for the new class. However, this method presents itself to be impractical in practice, as the number of classes required grows exponentially with respect to the number of elements that need to be analyzed. The most common approach to multi-label classification is binary relevance learning. This involves learning |L| binary classifier for each possible label, using all examples x with / Px as li ∈ Px as positive examples and all those with li ∈ negative examples. For classifying a new instance, all binary predictions are obtained and then the set of labels corresponding to positive relevance classification is associated with the instance. This scenario is commonly used for evaluating algorithms on scene classification [3,12]. Support vector machines (SVM) is the most widely used technique in the multi-label classification [3,8,12] literature. The basic SVM facilitates binary class problems, in which the data are separated by a hyperplane defined by a number of support vectors [28]. If SVM cannot separate two classes, it uses a kernel function to map input data into a high-dimensional space where it is possible to create a hyperplane which permits linear separation of classes. A significant advantage of the SVM approach is that the complexity of the resulting classifier is characterized by the number of support vectors rather than the dimensionality of the transformed space. In our approach, we decompose the element detection problem into a set of binary classification problems and develop a classifier for each class (each element) which considers each training sample as positive to which it belongs and as negative to all others. In other words, we apply one versus rest classification approach. To determine labels of a test sample, the binary classifiers thus developed are run individually on the test sample and every label for which the

Machine vision system

output of the classifier exceeds a predetermined threshold is selected as a label of the sample. We use the radial basis function (RBF) as the kernel. The RBF kernel nonlinearly maps samples into a higher dimensional space, such that it, unlike the linear kernel, may handle the case in which the relation between class labels and attributes is nonlinear. We use LIBSVM [4] library for the SVM implementation. Next, we applied an artificial neural network (ANN) [2] for the multi-label classification task. We recognize that it appears natural to use a neural network with one output node per class for multi-label classification. In other words, more than one output of the ANN may be active in a multi-element detection task where active outputs corresponding to active elements in the specimen. As opposed to the SVM multilabel classification method, only a single classifier is used in this case. The training process for an ANN is also relatively straightforward where a rearrangement of training data is not required. Here, a multi-layer perceptron with one hidden layer is initially considered. The main criteria for selecting the most suitable classifier are accuracy of the classification, complexity and computational time, as our method requires real-time performance. The training step in a machine learning task could be considered as a calibration step in the element spectroscopy implementation, where samples of known elements with known compositions are propelled through an oil-pipe of which measurements are subsequently taken. The machine learning approach for element spectroscopy poses a problem that requires finding a distinctive feature set. The task of finding a distinctive feature set is important as it allows us to use a relatively simple classifier and also simplifies the work of the classifier itself. Our approach occupies a feature set which is based on an average similarity score (as computed in Algorithm 1) obtained by correlating a pattern template present in the NIST [30] database with each signature to be classified. Note that we only need to consider signatures resulting from single elements. Since the expected regions of the blobs are already known, we only search for the peaks in those regions. This method saves significant amount of computational cost compared to running a peak detection on the whole image frame to find the intensity peaks. Here, there are equal number of binary SVM classifiers to the number of elements being classified. Each classifier is trained separately by showing positive or negative examples: positive when the sample contains the corresponding element, and negative when the sample does not contain the corresponding element. For example, Ag classifier and Cu classifier would consider the composite spectra of Ag and Cu to be a positive sample, even though the H g classifier would consider the same spectra to be a negative sample. As for the ANN, we train only a single ANN classifier with one output per element using the back-propagation learning technique with the same training data set.

117

Assume that there are K patterns and each pattern pk , k = 0, . . . , K −1 contains n k blobs (or points). These are read at the beginning of the algorithm. Similarity between a blob and the corresponding image region is obtained by computing the scalar product, and normalizing by the squared sum of the kernel g(s, t). Algorithm 2 Multi-label classification Ensure: image exists  image is a frame from video stream 1: Detect markers. 2: Compute H. 3: Extract rectangular ROIs 4: for k ← 0, K − 1 patterns do 5: for i ← 1, n k points do 6: Calculate similarity scor e between thresholded image region and blob 7: end for 8: scor e ← average similarity. 9: end for 10: Form the feature vector 11: Declared L binary classifier 12: for j ← 1, |L| classes do 13: Match the new instance against all 14: end for 15: Take the union and Print k of matched patterns

Similar to Algorithm 1, homographies are computed, the ROIs are extracted and the similarity scores are calculated at the beginning of the Algorithm 2. The feature set is then formed after calculating the average similarity of the unknown pattern with the library patterns. The new sample is then compared against all the binary classifiers and the union of result is taken as the matched patterns. Note that Algorithm 2 is more computationally complex than Algorithm 1, even though Algorithm 2 provides better accuracy of detection (see Table 3).

5 Results We conducted experiments using the experimental setup by applying pattern matching algorithm and the multi-label classification algorithms. We use the Accuracy as the basis for evaluating the performance of the algorithms: Accuracy =

TP + TN . TP + FP + FN + TN

(2)

TP is the number of true positives (correctly predicted positive examples), FP is the number of false positives (positive predictions that are incorrect), FN is the number of false negatives (positive examples that are incorrectly predicted negative) and TN is the number of true negatives (correctly predicted negative examples). In our case, a true positive is is considered as the correct identification of an element from its blob pattern. Since known element samples are ablated for

123

118 Table 3 Results of experiments using actual data from the prototype spectrometer

E. Ukwatta et al.

Metal No. of training No. of testing Pattern matching samples samples Average Accuracy detection rate (%) error

SVM

Average Accuracy Average Accuracy detection rate (%) detection rate (%) error error

Hg

30

30

23.3 ± 3.8 92.72

0±0

100.00

0±0

Na

30

30

27.5 ± 4.7 91.41

3.5 ± 0.3

98.91

3.4 ± 0.3

98.93

Ne

50

50

13.6 ± 2.8 95.75

0±0

100.00

0±0

100.00

He

20

20

21.5 ± 3.5 93.28

1.8 ± 0.4

99.43

0±0

100.00

Al

20

20

46.3 ± 5.9 85.53

5.1 ± 0.7

98.41

4.1 ± 1.0

98.72

Zr

40

40

44.7 ± 3.8 86.03

4.3 ± 0.9

98.66

4.4 ± 0.6

98.62

Ag

30

30

37.0 ± 4.5 88.43

6.2 ± 1.2

98.07

4.7 ± 0.9

98.53

Pb

50

50

54.9 ± 7.1 82.84

3.6 ± 0.6

98.87

3.4 ± 0.8

98.93

Cu

20

20

40.1 ± 5.9 87.47

4.0 ± 0.8

98.75

3.0 ± 0.6

99.06

Ni

30

30

35.2 ± 3.3 89.00

6.4 ± 1.1

98.00

2.5 ± 0.5

99.22

experiments, we have the true pattern (ground truth) at our disposal for this comparison. This enables us to cross-check the detection and verify that the pattern has correctly been classified.

5.1 Actual experimental results The pattern matching technique and the classification algorithms are first evaluated with single emission images. We used the single-emission spectra of mercury (Hg), sodium (Na), neon (Ne), helium (He), aluminum (Al), zirconium (Zr), silver (Ag), lead (Pb), copper (Cu), and nickel (Ni), for our experiments. For Hg, Na, Ne, and He, the spectra were obtained using known sources of light. For Al, Zr, Ag, Pb, Cu, and Ni, LIBS emission spectra were used. The experiments were conducted using an image size of 1, 280 × 1, 024 pixels on a calibrated experimental setup. The video stream of the camera is recorded at 5 frames/s. ROIs were extracted from the images for further processing. To mitigate the effect of broadband noise and other artifacts in the experiment setup, we performed background subtraction: i.e. the current LIBS image with emission pattern is subtracted by the image with no element ablation. We selected a specific number of training and testing data randomly from a pool of images as shown in Columns 1 and 2 of Table 3, respectively. Algorithms were evaluated for their accuracy for 20 such selections of training and testing data. We maintained a fixed threshold value of 0.34 for the pattern matching approach. The threshold value was chosen by selecting the value that yielded the lowest misclassification rate using an experiment which varied the threshold value against the misclassification rate. Since we utilize ten elements in the experiment, ten features were used such as ‘the similarity score with Hg’, ‘similarity score with Na’,

123

ANN

100.00

’similarity score with Ne’ etc. for multi-label classification. The features were obtained by correlating each signature to be classified with the pattern templates obtained from NIST database (as computed in Algorithm 1). In the SVM approach, we employed ten binary classifiers for each metal class to be classified. When a binary classifier is trained, for example for Hg, 30 samples are considered to be positive and the remaining 290 samples are considered to be negative. In the ANN approach, we use a multi-layer perceptron consisting of single hidden layer and ten output layer nodes representing elements. Table 3 tabulates the performance results of the simple pattern matching technique, SVM, and ANN on actual experimental data. As observed, the classification algorithms outperformed the pattern matching technique for all instances. SVM yielded slightly higher accuracy than ANN for most of the elements. Blob patterns that correspond to emissions from sources of light provided higher accuracy rates of detection for all three techniques. This may be due to the high intensity of blobs and low background noise in these images. However, the image patterns obtained from the LIBS technique have high broadband noise, emission patterns resulting from ablation of other impurities (usually air), and low intensity profiles. The execution times were ≈ 12 ms, ≈ 127 ms, ≈ 109 ms for the pattern matching algorithm, SVM algorithm, and ANN algorithm respectively, to process a single image frame within the Elphel 353 camera [7] using a non-optimized C++ implementation.

5.2 Results for simulated images In the absence of actual multi-element spectra images, we used the simulation setup [27] to evaluate the performance of

Machine vision system Table 4 Number of training and testing samples

Table 5 Comparison of algorithms for synthetic data set

119

Metal

Ag

Pb

Cu

Fe

Cr

Ni

Ag, Pb

Pb, Cu

Ag, Cu

Ag, Pb, Cu

Total 1,900

Training

220

210

260

160

180

170

150

140

210

200

Testing

450

350

300

350

400

350

400

450

500

250

3,800

Total

670

560

560

510

580

520

550

590

710

450

5,700

Metal

Pattern matching Average detection error

ANN Accuracy rate (%)

SVM

Average detection error

Accuracy rate (%)

Average detection error

Accuracy rate (%)

Ag

321.6 ± 47.9

79.90

8.3 ± 3.1

99.48

5.7 ± 2.8

99.64

Pb

250.5 ± 24.1

82.72

5.6 ± 2.6

99.61

4.3 ± 2.2

99.70

Cu

228.2 ± 29.8

84.78

9.7 ± 3.5

99.35

7.2 ± 2.6

99.52

Fe

34.3 ± 13.9

90.20

2.2 ± 2.8

99.37

0.8 ± 0.4

99.77

Cr

49.1 ± 14.6

87.72

0.9 ± 0.6

99.78

1.2 ± 0.7

99.70

Ni

23.3 ± 20.7

93.34

2.5 ± 0.4

99.28

2.5 ± 0.5

99.88

the algorithms for multi-element spectra and their robustness against noise. This setup uses a computer monitor to display simulated patterns that are observed by the video camera. The recorded images are processed in a computer using the algorithms. The key difference between simulation data and actual experimental data is the high level of broadband noise present in the actual LIBS images. Results for multi-label classification. We used Ag, Cu, Pb, Cr, Fe, and Ni as metals and the combined compounds of Ag + Pb, Ag + Cu, Pb + Cu, and Ag + Cu + Pb for experiments. Similar to the actual experiments, we use the similarity score with Ag, similarity score with Cu etc., as features. Training data are obtained by running the system for a period of time and manually labeling the data according to the time-stamp of the display computer. Six features are used based on similarity score as computed in Algorithm 2 to build the classifiers. Sub divisions of the training and testing data are shown in Table 4, which are used to train the classifiers. We used six binary SVM classifiers for SVM approach and a multi-layer neural network with six output nodes for ANN approach. We found the optimal number of hidden neurons as 15 for this experiment by selecting the number of hidden neurons that gives the lowest misclassified samples. Table 5 shows a comparison of the proposed multilabel classification algorithms to pattern matching technique. An optimal threshold value of 0.21 was chosen for the pattern matching technique. Pattern matching technique yielded a high misclassification rate compared to multilabel classification approach. Clearly, the machine learning approaches outperformed the pattern matching technique for single-emission spectra as well as multi-element spectra. For multi-element spectra machine learning approaches

outperformed the pattern matching technique by a higher margin. Effect of noise. We validated the algorithms for their robustness to noise. The presence of noise is considered in the spectral position as well as in the point spread of pattern blobs. As for the location noise, we assumed that the blobs may deviate from their position largely along the dispersion lines. Therefore, we considered the variations of blobs only along dispersion lines, thereby adding noise to the x-coordinate only. For the pattern matching algorithm, the experiments were conducted with a fixed value for the threshold and maintained other experimental conditions unchanged. Similarly, we evaluated the classification algorithms (both SVM and ANN) in the presence of noise. The results of the algorithms obtained for different levels of location noise are shown in Fig. 6a. We added random noise drawn from a normal distribution with zero mean. For each algorithm, ten experiments were performed and obtained 1,200 samples for each consideration. As expected, we observed a decrease in accuracy as well as an increase in the variation of accuracy with increased levels of noise. Accuracy decreases with a higher margin at higher levels of noise. Experimental results also exhibit relatively high standard deviations for high noise levels. Figure 6b plots the experimental results obtained over a varying noise levels in the point spread. Here, we introduced zero-mean Gaussian noise to the spread of the blobs. Similar to the experiments for location noise, 10 experiments were conducted for each level of noise and obtained 1,200 samples for each consideration. As observed, the multi-label classification methods provided robustness to noise compared to pattern matching technique for both types of noise.

123

120

E. Ukwatta et al. 3

3

10

2

10

1

10

Correlation SVM ANN

0

10

Misclassified samples

Misclassified samples

10

2

10

1

10

Correlation SVM ANN

0

0

2

4

6

8

10

10

0

10

20

30

40

Noise level (location noise)

Noise level (spread noise)

(a) Location noise

(b) Spread noise

50

Fig. 6 Performance of the pattern matching technique with varying levels of noise. x axis of a is the standard deviation of the noise distribution in pixels. x axis of b is the standard deviation varied as a percentage of maximum spread of the blobs

Both machine learning approaches (SVM and ANN) yielded promising results so that, the selection of a particular method is influenced largely by the computational resources required by each respective method.

6 Conclusions and future work We presented a novel spectrum analyzer design based on machine vision and machine learning techniques for qualitative analysis of elements. The main contribution of this work is a real-time, fully automated element detection technique which is developed using a commercially available video camera. Other contributions of this paper include multi-label classification approach for element detection and techniques for implementing sufficiently capable pattern recognition algorithms in a resource constrained environment. With the proposed algorithms capable of processing only small and relevant portions of an image, we are able to significantly reduce the computing burden with the use of small, but powerful embedded processors. We were able to obtain accuracy rates exceeding 99% that are comparable to techniques used in metal spectroscopy literature. The design seeks to maintain the fundamental advantages of LIBS, while achieving a costeffective configuration capable of accurate characterization of the element constituents of materials. The method is useful in instances in which a qualitative analysis of elements is sufficient over a quantitative analysis of elements. The proposed system is merely a first step towards developing a machine vision based multi-element detection system on LIBS spectra. Even though, this technology is developed with a specific application in mind, the proposed technology has much broader applications in machine vision based sensors in a wide variety of domains including manufacturing, automation, and surveillance. In this work, we evaluated our approach only for metal identification. However, the system could be generalized to detect various solid, liquid, and gaseous materials (e.g:

123

type of alloys [15]) as well. In this paper, we considered only qualitative multiple-shot LIBS element detection. However, our approach could be extended to quantitative analysis as well. Estimating the quantity of material involves determining particles size based on the relative strength of spectral peaks, as well as counting the frequency occurrence over a certain period of time. Investigating the ability to estimate material content based on the spectral peak level over time is yet another future direction. The algorithms were implemented using an non-optimized C++ code within the camera. This causes our system to be constrained by the available processing power of the embedded CPU as well as the available memory. With respect to hardware, future work lies in developing the proposed techniques at an FPGA level in the embedded processor of the camera. This would enable us to perform the entire processing within the camera itself, and to present detected elements as an output of a LCD display. Acknowledgments We would like to thank Andrey N. Filippov for the help given on the camera firmware. We gratefully acknowledge the financial support received from the Ontario Centres of Excellence (OCE), the University of Western Ontario and the Natural Sciences and Engineering Research Council of Canada (NSERC).

References 1. Barbini, R., Colao, F., Fantoni, R., Lazic, V., Palucci, A., Capitelli, F., van der Steen, H.J.L.: Laser induced breakdown spectroscopy for semi-quantitative elemental analysis in soils and marine sediments. In: Proceedings of EARSeL-SIG-Workshop LIDAR, vol. 1, pp. 122–129 (2000) 2. Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1995) 3. Boutell, M., Shen, X., Luo, J., Brown, C.: Learning multi-label semantic scene classification. Pattern Recognit. 37(9), 1757–1771 (2004) 4. Chang, C.C., Lin, C.: LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/ libsvm (2001)

Machine vision system 5. Cremers, D.A., radziemski, L.J.: Handbook of Laser-Induced Breakdown Spectroscopy, chap. 1–6. pp. 1–168. Cambridge University Press, Cambridge (2006) 6. Ferrero, A., Lucena, P., Herrera, R.G., Dona, A., Fernandez- Reyes, R., Laserna, J.J.: Libraries for spectrum identification: method of normalized coordinates versus linear correlation. Spectrochim. Acta B 63(3), 383–388 (2008) 7. Filippov, A.: Reconfigurable high resolution network camera. In: IEEE Field-Programmable Custom Computing Machines, vol. 9, pp. 267–277 (2003) 8. Godbole, S., Sarawagi, S.: Discriminative methods for multilabeled classification. In: Conference on Knowledge Discovery and Data Mining, pp. 22–30 (2004) 9. Gornushkin, I.B., Smith, B.W., Nasajpour, H., Winefordner, J.D.: Identification of solid materials by correlation analysis using a microscopic laser-induced plasma spectrometer. Anal. Chem. 71(22), 5157–5164 (1999) 10. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2003) 11. Hunter, A.J.R., Piper, L.G.: Laser-Induced Breakdown Spectroscopy—Fundamentals and Applications, chap. 1–18, pp. 1–614. Cambridge Univeristy Press, Cambridge (2006) 12. Lauser, B., Hotho, A.: Automatic multi-label subject indexing in a multilingual environment. In: European Conference in Research and Advanced Technology for Digital Libraries, pp. 140–151 (2003) 13. Lee, W.B., Wu, J., Lee, Y.I., Sneddon, J.: Recent applications of laser-induced breakdown spectrometry: A review of material approaches. Appl. Spectrosc. Rev. 39(1), 27–97 (2004) 14. Lentjes, M., Dickmann, K., Meijer, J.: Calculation and optimization of sample identification by laser induced breakdown spectroscopy via correlation analysis. Spectrochim. Acta B 62(1), 56–62 (2007) 15. Lopez, A.J., de Castro, M.D.L.: Rank correlation of laserinduced breakdown spectroscopic data for the identification of alloys used in jewelry manufacture. Spectrochim. Acta B 58(7), 1291–1299 (2003) 16. Lucia, F.C.D., Gottfried, J.L., Munson, C.A., Miziolek, A.W.: Multivariate analysis of standoff laser-induced breakdown spectroscopy spectra for classification of explosive-containing residues. Appl. Optics 37(41), G112–G121 (2008) 17. Ma, Y., Soatto, S., Kosecka, J., Sastry, S.S.: An Invitation to 3-D Vision. Springer, Berlin (2004) 18. Micron Technology, Inc.: MT9M001: 1/2-Inch Megapixel Digital Image Sensor Features (2004). http://www.aptina.com/products/ image_sensors/mt9t001p12stc/#overview 19. Ralchenko, Y.: A guide to internet atomic databases for hot plasmas. J. Quant. Spectrosc. Radiat. Transf. 99(1–3), 499–510 (2006) 20. Rusak, D.A., Castle, B.C., Smith, B.W., Winefordner, J.D.: Fundamentals and applications of laser-induced breakdown spectroscopy. Crit. Rev. Anal. Chem. 27(4), 257–290 (1997) 21. Sall, B., Lacour, J.L., Vors, E., Fichet, P., Maurice, S., Cremers, D.A., Wiens, R.C.: Laser-induced breakdown spectroscopy for mars surface analysis: capabilities at stand-off distances and detection of chlorine and sulfur elements. Spectrochim. Acta B 59(9), 1413–1422 (2004) 22. Samek, O., Krzysanek, V., Beddows, D.C.S., Telle, H.H., Kaiser, J., Lika, M.: Material identification using laser spectroscopy and pattern recognition algorithms. In: International Conference on Computer Analysis of Images and Patterns, pp. 443–450 (2001) 23. Scheirer, W.J., Kirkbride, S.R., Boult, T.E.: Inexpensive spectrometer color camera technology. In: IEEE Workshop on Application of Computer Vision, pp. 1–8 (2008) 24. Sirven, J.B., Bousquet, B., Canioni, L., Sarger, L., Tellier, S., Potin-Gautier, M., Hecho, I.L.: Qualitative and quantitative investigation of chromium-polluted soils by laser-induced breakdown

121

25. 26.

27.

28. 29.

30.

spectroscopy combined with neural networks analysis. Anal. Bioanal. Chem. 385(2), 252–262 (2006) Tsoumakas, G., Katakis, I.: Multi-label classification: an overview. Int. J. Data Warehouse. Min. 3(3), 1–13 (2007) Ukwatta, E., Rodrigo, R., Samarabandu, J., Hall, M.: Vision based spectroscopy simulation. In: Conference on Computer and Robot Vision, pp. 363–368 (2008) Ukwatta, E., Samarabandu, J.: Vision based metal spectral analysis using multi-label classification. In: Conference on Computer and Robot Vision, pp. 132–139 (2009) Vapnik, V.: The Nature of Statistical Learning Theory. Springer, London (1995) Wainner, R.T., Harmon, R.S., Miziolek, A.W., McNesby, K.L., French, P.D.: Analysis of environmental lead contamination: comparison of LIBS field and laboratory instruments. Spectrochim. Acta B 56(6), 777–793 (2001) Watson, P.R., Hove, M.A.V., Hermann, K.: NIST Surface Structure Database, vol. Version 5.0. National Institute of Standards and Technology, Gaithersburg (2004)

Author Biographies Eranga Ukwatta completed his B. Sc. (Eng) with first class honours in Electronic and Telecommunication Engineering at the University of Moratuwa, Sri Lanka in 2005. He subsequently joined Dialog Telekom Plc. as a Telecommunication Engineer. He obtained his M.E.Sc. in Electrical and Computer Engineering from The University of Western Ontario in 2009. Currently, he is a PhD student in Biomedical Engineering at The University of Western Ontario. His research interests are medical image processing and analysis.

Jagath Samarabandu received B. Sc. (Eng) in Electronic and Telecommunication with first class honours from the University of Moratuwa, Sri Lanka in 1982. He was awarded the Fulbright scholarship in 1987 for postgraduate study. He received M.S and PhD degrees in Electrical Engineering from State University of New York at Buffalo in 1990 and 1994, respectively. He held a post-doctoral position in the Dept. of Biological Sciences at SUNY-Buffalo until 1997 and joined Life Imaging Systems Inc. as a senior software engineer in 1997. He joined the Dept. of Electrical and Computer engineering at the University of Western Ontario as an assistant professor. Currently, he is an associate professor in the Dept. of Electrical and Computer Engineering at the University of Western Ontario. His research interests include computer vision, image understanding, and pattern recognition.

Mike Hall is the president of Checkfluid. His research interests include laser, spectrometer, and oil quality sensor design and development.

123