DISCUSSIONS APPLICATION
OF T H E I N F O R M A T I O N - T H E O R Y
TO MEASUREMENT-TECHNIQUE A,
CONCEPTS
PROBLEMS
S. R i v k i n
UDC 681.2.001.1:519.92
Since 1957 the journal Measurement Techniques has systematically carried articles in which the information-theory apparatus is applied to m e a s u r e m e n t - t e c h n i q u e problems [1-9]. This borrowing is aimed mainly at establishing for measurement means and processes suitable characteristics which take into account the probability nature of errors and of the measured quantity [7]. In fact, the concept of information is so general that the establishment of information criteria seemed at first natural for evaluating the quality of measurements and instruments. It would appear that the majority of authors were guided by this consideration in writing the a b o v e - m e n t i o n e d works. However, it should be borne in mind that the existing information theory was developed for solving c o m m u n i cation problems and is organically connected with them. Therefore, it is necessary to e x a m i n e in greater detail to what extent the measurement process is identical to the condition examined in the statistical theory of c o m m u n i c a tions.. In the theory of communications there are two kinds of problems, according to which the information theory was developed [10]. The first kind is related to designing coding and receiving systems which meet conditions for attaining a m a x i m u m transmission speed with m i n i m u m errors in reproducing messages. The general possibilities in this respect were investigated by Shannon and his collaborators and constitute the content of information theory proper [11, 12], The other fundamental problem, consisting of the best reproduction of a transmitted signal in the presence of noise at the input or inside the receiving system, was solved by Wiener and Kolmogorov, whose works laid the foundations of the modern theory of filtration and forecasting. The c o m m o n ground in the approach to these two problems consists of the assumption that the recieved signal or the transmitted message and noise effect belong to a given aggregate of random processes or sequences. It should be noted that the measurement of quantities which change with time in the presence of noise corresponds to the second kind of problems. By using the knowledge of the signal and noise statistical characteristics, it becomes easy to c a l c u l a t e the filtering-device parameters for reproducing the input signal with a m i n i m u m dispersion of errors. This is a reasonable requirement for recording instruments. The o p t i m i z a t i o n of the properties of such instruments is a promising trend for their improvement. Let us c o m p a r e the measurement process to the transmission of messages in communication systems for which the information, theory concepts were developed. Communication systems are divided in their theoretical study into the following elements: sources of information, coding devices, c o m m u n i c a t i o n channels, receiving and decoding devices, and the addressee [10, 12]. The determining factors consist of the properties of the communication source and the nature of noise in the communication channel. The source of information is c h a r a c t e r i z e d by the statistical properties of the continuous signals or discrete sequences produced by it. Stipulations for stationary ergodic sequences or functions are provided in the information theory. Long segments of signals which normally occur at the output of information sources serve as good approximations to the above quantities. The properties of ergodicity and of the stationary state, for instance, of discrete information sources, which are widely used in communication lines, often depend on the fact that the transmitted ruesages are expressed in one of the natural languages. The statistical distribution of letters in such messages is adequately approximated by a stationary ergodic Markov chain [11]. However, there are no groundin measurement techniques for assuming a priori the ergodicity and stationary state of measured quantities' sequences. Therefore, the requirements stipulated by the information theory for the measured object severely restrict the c i r c l e of m e a s u r e m e n t - t e c h nique problems to which this theory is applicable. It is also important to stress that the messages transmitted in communications systems belong to finite sets l i m ited by the number of possible states, for instance, by the letters of an alphabet in a discrete case, or by the frequency bandwidth in the case of continuous signals. Messages which are close to each other in their selected distinguishing criteria are formed into subsets. Coded equivalents representative of the e l e m e n t a r y message groups thus formed Translated from I z m e r i t e l ' n a y a Tekhnika, No. 2, pp. 63-65, February, 1968.
Original article submitted
April 17, 1967.
229
are then subject to transmission. Such a simplification is not being extensively used in transmitting measured information, since the assignment of a measured quantity to a fixed interval is normally the final aim of measurements. The above-mentioned discrepancy between the measurement process and the transmission of messages in communication systems is responsible for the fact that the information-theory concepts are applied mainly to analyzing existing instruments, and not to designing new ones, Let us examine certain information methods used for analyzing measurement means and results, and described in [2-8]. The particular feature of information criteria consists of the facility provided by them for using information theory methods for obtaining integral measurement equipment characteristics which depend on an aggregate of several parameters of an instrument. It is desirable to have such characteristics when it is necessary to account for several contradictory factors. Thus, in measuring dynamic quantities, an instrument's carrying capacity provides a quantitative evaluation which includes the recording precision and the time required for separate measurement [2, 6, 8]. At the same time the universality of information indexes is often the reason which makes their application inadvisable. This is sufficiently obvious when the information criterion combines instrument parameters which do not affect each other. This occurs when it is necessary that an aggregate of such apparent physical characteristics as the measurement range of an instrument and the type of its random error components' distribution or individual parameters of this distribution should be expressed quantitatively by a single abstract value, for instance, information precision [2] or information content transmitted by the instrument. Information-theory concepts were also used for evaluating the distribution of the instruments' random errors. The information index suggested in [3, 4] is based on the entropy of the error-distribution law. For the purpose of this article it is sufficient to stress the main property of the discrete-distribution entropy. It follows directly from the definition of entropy that H = - - %~ P i log P i i
(where H is entropy, Pi is the probability of an elementary message with index i appearing; the summation with respect to i entails all the possible messages) that entropy is a function of probabilities only and it does not account for the distribution of random quantities over the scale of their possible values, Therefore, the information determined as a value equal to the entropy of the experiment before its complete realization depends on the interrelationship of the probabilities of the elementary events appearing and not on their specific values. In metrological pracrice, the event of greatest interest in evaluating the distribution laws consists precisely of the combination of the error's numerical value with the probability of their appearance. It is a little more complicated to consider the application of the entropy criterion to continuous distributions which were dealt with in [3, 4]. In the theory of continuous-signal transmission, the information content transmitted over a channel is determined as the difference of the input-signal entropies before and after the signal's reception. It should be borne in mind that the continuous signal is assumed to belong to a stationary ergodic ensemble. If an additive noise independent of the signal exists in the channel, the information content is determined as the difference between the input-signal and noise entropies. For stationary ergodic-signal sources, the right-hand side of the relationship H=lim-----nn~o~
...
S
P ( x l , x~ . . . . .
x.)
xiogP(xl,
x 2.....
xr~) d x l dx2 . . .
dxn,
which determines the entropy of a continuous distribution, is convergent,and the entropy can be calculated from the formula H = - - S P (x) log P (x) dx, where P(x) is the distribution density of random quantity x. The entropy of continuous distribution is not invariant with respect to the transformation of the variable, i. e., it changes with a transition to a new variable which has a functional relationship to the old one. Therefore, in the case of continuous signals the information theory operates with the concepts of information content and carrying capacity, i.e., with quantities which are not affected by the transformation of the independent variable, A consideration of only the error entropy is not sufficient for describing the information conditions. Moreover, the computation
230
of an arbitrary random quantity's distribution moment from the formula H=--
i P(x) l o g P ( x ) d x
loses c o m p l e t e l y the information meaning of entropy as a characteristic of the probability density of typical ergodic ensemble signals. It should also be pointed out that the information-theory methods do not deal at a l l with the values of the signal and noise m a t h e m a t i c a l expectation, since it does not affect the transmitted4nformation content. Therefore, the suggestion of evaluating the distribution law of random errors by their m a t h e m a t i c a l expectation and the moment of the form H =--
j" P (x) log P (x) dx
cannot be considered o b j e c t i v e l y related to the notions of the existing information theory. The use of the entropy value as a distribution p a r a m e t e r of a random error has no advantage as compared with the error's r o o t - m e a n - s q u a r e evaluation. The entropy value of the error is one of the i n c o m p l e t e characteristics of the distribution law, whereas, the knowledge of the type of distribution law provides an unambiguous linear relationship between the root-mean-square and the entropy values of the error. In conclusion the author should like to note once more the particular features of the penetration of the inform a t i o n - t h e o r y concepts into metrology. Above all, the theoretical structure of measuring means should not be identified with a c o m m u n i c a t i o n system's theoretical model, which is the subject studied in the information theory. Therefore, it is necessary to substantiate the selection of measurement-technique problems which correspond to the notions of the information theory. Such problems should include the analysis of the carrying c a p a c i t y and noise stability of dynamic measuring equipment transducers. This analysis should be similar to the one carried out in the information theory for various types of modulation. In this connection works [6, 8] should be mentioned. The information content which is a measure of a free selection can be used directly for optimizing the organization of aggregate measurements, for instance, in selecting the aggregate of the studied object's tested parameters, or in cases which are related in their content to the problem of counterfeit coins [13], However, the information criteria suggested at the present time cannot always be used as a convenient unambiguous characteristic for the quality of measurement means and processes. The apparent objectivity of the information approach is often substantiated by references to the information theory concepts' universality, which, however, does not apply to the quantitative aspect of Shannon's information theory. LITERATURE 1, 2, 9
3. 4. 5. 6. 7, 8. 9. 10.
11. 12. 13.
CITED
k. G, Dubitskii, I z m e r i t e l ' . Tekh., No. 7 (1963). P . V . Novitskii, I z m e r i t e l ' . Tekh., No. 1 (1962). P . V . Novitskii, I z m e r i t e l ' . Tekh., No. 7 (1966). P . V . Novitskii, N. A. Nazarov, V. Ya. Ivanova, and G. A. Kondrashova, I z m e r i t e l ' . Tekh., No. 9 (1966). S . M . Persin, Izrneritel'. Tekh., No. 7 (1964). B. M. Pushnoi, I z m e r i t e l ' . Tekh., No. 7 (1968). V. I. Rabinovich and M. P. Tsapenko, I z m e r i t e l ' . Tekh., Nos. 4, 6, and 10 (1963). Yu. V. Sychevskii and V. V. Kafarov, I z m e r i t e l ' . Tekh., No. 2 (1965). Rezo Tar'yan, I z m e r i t e l ' . Tekh., No. 2 (1957). R. Fano, Information Transmission. Statistical Theory of Communications [Russian translation], Izd. Mir, Moscow (1965). C. E, Shannon, M a t h e m a t i c a l Theory of Communications, in Coll.: Works on the Theory of Information and Cybernetics [Russian translation], IIL, Moscow (1963). C. E, Shannon, Communications in the Presence of Noise (Ibid,), A . M . Yaglom and N. M. Yaglom, Probability and Information [in Russian], FM, Moscow (1960).
Editorial Note. The works of Sovietauthors criticized in this article obviously do not consist of the borrowing of terms from the information theory, but they represent an a t t e m p t to develop in practice various information criteria for the quality of measuring instruments and measurements.
231
It is hardly possible to agree with A. S. Rivkin's treatment of several propositions of the theory of information and the information theory of measuring devices, In fact, several of Shannon's theorems (1, 3-9, 12, and 14) deal with the coding theory, but the remaining theorems (2, ]0, 11, 13, and 15-23) form the basis of the theory of genera l i z e d evaluations of information-transmitting devices. The editorial board considers it advisable to discuss the problems which have thus been raised, and invites the readers of this journal to send their contributions on the subject.
232
