An Empirical Investigation of the Reliability and Stability of Selected Activity and Attitude Measures

Citation:

Edgar Pessemier and Albert Bruno (1971) ,"An Empirical Investigation of the Reliability and Stability of Selected Activity and Attitude Measures", in SV - Proceedings of the Second Annual Conference of the Association for Consumer Research, eds. David M. Gardner, College Park, MD : Association for Consumer Research, Pages: 389-403.

Proceedings of the Second Annual Conference of the Association for Consumer Research, 1971     Pages 389-403

AN EMPIRICAL INVESTIGATION OF THE RELIABILITY AND STABILITY OF SELECTED ACTIVITY AND ATTITUDE MEASURES

Edgar Pessemier, Purdue University

Albert Bruno, University of Santa Clara

[The authors are indebted to Douglas Tigert for providing the data for a portion of this study. We also wish to acknowledge the analytical assistance provided by Stephen Arnold at the University of Toronto.]

Since 1965, a substantial body of data and research findings have been generated which employ market-related measures of consumers' activities and attitudes. This work has consciously attempted to expand the set of descriptors of consumer characteristics to fill the void between the economist's demographic profiles and the psychologist's personality inventories. Even though the developmental time has been short, it is impossible to review all the work in which this more problem-specific point of view was adopted. Interested readers are directed to a paper by Hustad and Pessemier which summarizes much of the history and current status of the research on activity and attitude measures (3).

In this paper we will concentrate on a variety of studies conducted by researchers at B.B.D.O., Purdue University, the University of Chicago, Lever Brothers, and the University of Toronto (1, 2, 5, 6, 7, 9, 10, 11). By abstracting some of their findings and modestly extending some of the analyses which these researchers performed, we hope to draw specific conclusions which will have generalized implications for all attitude and activity research. In particular, we will look at a few mundane but significant questions relating to reliability and validity. To do so, selected common aspects of eight research efforts will be examined. These surveys involved ten investigators and nearly 8,000 subjects from male and female, student and non-student and American and Canadian populations sampled in the interval from 1963 through 1970.

The research efforts noted in Table l represent important contributions to the expanding collection of data dealing with consumer activities, attitudes, interests, opinions and values. Since these areas of measurement are largely unexplored, it is not surprising that few well-worn paths have appeared in the few years this work covers. However, a large enough number of common criterion, cross-study variable sets and within-instrument controls for bias and consistency were used to provide a real opportunity to examine some aspects of validity and reliability. By far the greatest commonality exists among activity,interest and opinion variable sets across the separate surveys. Matched sets of items have been extracted for various pairs of surveys. These will be the principal vehicle for examining structural stability or cross-validation properties of the constructs employed in the separate surveys. Test-retest and consistency substudies address the reliability issue. Finally, because the instruments employed have typically been of considerable length, some findings about order bias and fatigue will be reported.

RELIABILITY

Reference to Table 1 makes it clear that both the activity,interest and opinion (AIO) sections and the total instruments tended to be lengthy. Several of the studies required six or more hours per respondent. Naturally, this time was spread out over several sessions but fatigue during any given session could not be ignored as a source of potential measurement error. In the studies by Summers, Baumgarten and King, the question of order of presentation and fatigue were examined in some detail. The summary results appear in Table 2. These results support the general conclusion that carefully organized and motivated surveys can collect very large amounts of data without serious degradation of the measures appearing in the latter part of the instruments. It must be noted however, that the questions which were susceptible to the greatest change when their positions tn the instruments were varied were those that are affected by the subject's mood.

TABLE 1

SUMMARY DESCRIPTIONS OF THE EIGHT SURVEYS EMPLOYED IN THIS STUDY

TABLE 2

THE PERCENTAGE OF QUESTIONS WHOSE RESPONSES VARIED SIGNIFICANTLY WHEN THE ORDER OF APPEARANCE FOR THE QUESTIONS WAS MODIFIED

In three of the studies, a sizable number of questions were repeated at various points in the questionnaire. These questions were used to test the reliability of individual responses. The tests for consistency involved the examination of the percentage of direct matches in responses and/or the percentage of near matches. A new match was a response that was within the interval + 1 response category from the original response. As Table 3 indicates, the consistency of responses to the questions (mainly attitudinal) were high enough to yield reliable data, especially when multiple items are used to measure a particular construct. As we will see, not only were multiple items used but they exhibited strong, persistent within-construct behavior when subjected to cross-validation.

Finally, two test-retest measures of the reliability of individual responses have been reported for the studies we are examining. The most extensive efforts have been reported by Tigert (8). He correlated and cross tabulated two sets of responses to the same 150 AIO variables. The retest questionnaire was administered to 280 respondents after a period of somewhat more than six months. The distribution of reliability coefficients across respondents is shown in Table 4. The reliability coefficients for sum scores of high-loading items extracted in both studies are shown in Table 5.

TABLE 3

THE PERCENTAGE OF PERFECT MATCHES OR NEAR MATCHES ON REPEATED QUESTIONS WITHIN THREE QUESTIONNAIRES

TABLE 4

FREQUENCY DISTRIBUTION OF TEST-RETEST RELIABILITY COEFFICIENTS FOR 150 AIO QUESTIONS

TABLE 5

TEST-RETEST RELIABILITY COEFFICIENTS FOR SUM SCORES OF PRINCIPAL ITEMS LOADING ON THE 12 COMMON CONSTRUCTS OF "PSYCHOGRAPHIC" DIMENSIONS

In a more limited test-retest examination of the reliability of individual responses, DeBruicker has reported on the responses of 972 women in the 1970 Lafayette, Indiana study. Here, ten questions from various sections of the questionnaire were readministered one to two weeks after the initial test. The retest reliability coefficients for the individual questions, like those found by Tigert, were all significant above the one in one million level. The subject retest reliability coefficients computed for the 1970 study appear below.

TABLE 6

FREQUENCY DISTRIBUTION OF THE TEST-RETEST RELIABILITY COEFFICIENTS FOR SUBJECT-RESPONSES TO TEN QUESTIONS IN THE 1970 LAFAYETTE STUDY

TABLE 7

COMPARISONS OF SIX ATTITUDE, INTEREST, AND OPINION DATA SETS - COMMON TO FOUR SURVEY INSTRUMENTS

TABLE 8

SIGNIFICANT CORRELATIONS AMONG THE VARIABLES WITHIN A STUDY: ALL VARIABLE COMMON TO THE FOUR STUDIES: T/W = TIGERT/WELLS (CHICAGO); W/B = WILSON (B.B.D.G.); T/P = TIGERT/PESSEMIER (PURDUE) AND L = ROHLOFF/BRUNO (LEVER).

Note that over 96% of the respondents showed test-retest reliability coefficients significant above the one in one million levels.

The general conclusion to be drawn from the results cited above is that research activity and attitude variables are sufficiently reliable for most marketing applications. It is also clear that additional development is needed. For example, instruments may be improved by adding, dropping or rephrasing some of the items associated with different constructs. In addition, sets of new items which fall outside the purely demographic and psychological domains may be needed to complete this collection of useful measures.

Questions about the additions and deletions of variables, are directly concerned with validity, particularly content, construct, concurrent and predictive validity. Here, one must face questions of criterion and predictive power. Although a thorough examination of concurrent and predictive validity is well beyond the scope of this paper, important data on validity can be presented. In particular, cross validation of the structural properties of the constructs is possible. It will be reassuring, if the association of variables within and across studies remains consistent and stable.

STABILITY

Construct stability across different populations and across time is important in developing a theoretical base for attitude and activity research. In its absence a.researcher would be confronted with several serious problems:

1) The distribution of the measured levels of the constructs based on attitudes and activities (say self confidence, new-product process and the like) could vary markedly from population to population and with the passage of time. Assuming a construct satisfies the researcher's criterion for predictive validity, the construct's value would be increased if the measured characteristic is widely applicable and persistent.

2) If the items used to develop the constructs of interest do not retain the same general relationships to each other, the existence and/or definition of the construct comes into question.

For these reasons, it will be valuable to examine several of the constructs that appeared in the separate studies. At the outset it is important to observe that the overlap in both questions and constructs from study to study is usually modest. Table 7 makes this fact clear for those studies in which we would directly reanalyze matched data sets. On the whole, the analyses indicate similar amounts of variance could be extracted and that most of the constructs and their constituent variables remain largely unchanged across different populations, points-in-time and instrumental contexts.

Although it is impractical to display all the factor analytic results here for the data in Table 7, it is possible to indicate the character and stability of those few variables common to all four studies. The four-way correlation matrix in Table 8 shows how these variables were related to each other within each study and the extent to which the relationships were similar across the studies.

In Table 8, a full one third of the 84 entries involved cases in which the correlation between a pair of variables is significant and of like sign for all four studies. The probability-of the latter outcome occurring by chance is extremely small (about one in a million). In addition, only five percent (2/42) of all possible entries were inconsistent (significant sign reversals). Clearly, the relations among the items described above are sufficiently stable for the purposes of most theoretical and practical applications. There is no reason to believe that the set of common questions which have been examined exhibit any more or less stable correlational properties than the other AIO items in each study.

A similar set of issues can be examined by looking at how a specific construct or set of constructs emerge across studies in which a suitable number of common variables were present. If the construct is a valid characteristic of consumers and the component variables are reliable measures, analyses should yield similar factor structures. A representative set of results for three studies is displayed in Table 9.

Several important aspects of the above illustration should be noted. J In the first two studies a separation of risk avoidance into "negative attitudes to change" and the "uncertainty avoidance" had been hypothesized. In the 1967 study, the decomposition did not develop but it did appear in the 1968 study. In the 1970 study, the decomposition was not expected but was produced by the analysis. In the case of the 1967 and 1968 studies, additional variables loaded on the factors beyond the original hypothesized set and several variables that were expected to load heavily (greater than .350), failed to do so. These modest departures of the results of the analysis from prior expectations are hardly surprising in light of the exploratory nature of the constructs and the variables used to measure them. The important differences in the respondent populations and time at which the studies were administered introduced other sources of instability. Though perhaps no more surprising, it is encouraging to find the central features of the constructs remain essentially unchanged.

To emphasize further the generality of consumer activity and attitude measures and wide applicability of the above observations, two additional sets of illustrative cross study comparisons will be presented. First, the analysis of data from a 1970 Canadian national sample of 1849 females will be compared to a similar analysis of data from a 1 % 9 U. S. national sample of 829 females. The basic data presented includes the top-loading variables and their correlations from the analysis of a matched subset of 41 questions taken from each study. In both cases, the subset of questions were embedded in a larger set of about 300 attitude and activity questions.

In the U. S. study, 33 of the 41 common variables yielded 14 factors accounting for 60% of the variance. In the Canadian study 34 of the variables yielded 15 factors accounting for 56% of the variance. Ten of the factors in both studies had the same highest loading variables and appear to measure the same constructs. Summary data for these ten common constructs appear in Table 10.

To further re-enforce the comparative analysis of activity and attitude variables, Table 11 displays the analysis of a set of 47 variables common to the Rohloff:Lever and the Tigert:Canadian studies. In the first study, 42 variables loaded (above .38) on 16 factors which accounted for 55% of the variance. In the second study, 34 variables loaded on 14 factors which accounted for 54% of the variance. Summary data on the nine factors that had the same high loading variables appear in Table 11.

SUMMARY AND CONCLUSIONS

It is important to note both the nature of the comparative evidence and the conclusions that have been presented about the development of marketing-related activity and attitude measures.

1) The variables studies are respondent related, not product or media specific.

2) Although they deal with both cognitive and affective constructs, they are rarely linked immediately to specific choice behavior.

3) The wide range of variables employed and the constructs to which they relate appear to be sufficiently reliable for both practical and theoretical purposes.

4) Both the constructs and the component variables have been used in diffuse exploratory studies. This approach has produced usefuL varietY.

5) The time has arrived when it would be profitable to begin to standardize some activity and attitude measures. This is especially true for those measures with high a priori marketing relevance but low specificity in individual brand and media choice contexts. In this regard, it would be helpful to develop several taxonomies of activity and attitudes measures. The work by Wind concerning AIO variables is illustrative (12).

6) It will be useful to begin Q detailed examination of the reliability of product class and media specific activity and attitude measures.

7) Finally, the principal gap in this review should be closed by undertaking a relatively extensive examination of the concurrent and predictive validity of marketing/consumer specific activity and attitude variables and related constructs.

Elsewhere, one of the authors had argued that both general purpose and product/media/brand specific activity and attitude measures are necessary components of many applied marketing research studies (4). That viewpoint remains unchanged. We also argue that standardized, general purpose activity and attitude measures such as the AIO variables examined here offer a degree of efficiency and cross product-class and cross media-class comparisons not otherwise attainable. Unless further development and predictive testing are conducted under relatively rigorous conditions, these measures will not achieve their proper theoretical and applied stature.

TABLE 9

ILLUSTRATIVE FACTOR LOADINGS FOR SELECTED ATTITUDES TOWARD CHANGE AND UNCERTAINTY: A THREE-STUDY CROSS VALIDATION

TABLE 10

COMPARATIVE ANALYSIS OF 41 COMMON VARIABLES IN TIGERT/WELLS: U.S. AND TIGERT: CANADA STUDIES (300 AIO VARIABLES PER STUDY) FEMALE RESPONDENTS

TABLE 11

COMPARATIVE ANALYSIS OF 47 COMMON VARIABLES IN THE ROHLOFF:LEVER AND TIGERT:CANADA STUDIES (ABOUT 50 AND 300 AIO VARIABLES PER STUDY) FEMALE RESPONDENTS

REFERENCES

Baumgarten, Steven, "A Study of Fashion Adoption Among Male College Students," unpublished doctoral dissertation, Krannert Graduate School of Industrial Administration, Purdue University, June, 1971.

Bruno, Albert V., "An Empirical Model for the Evaluation of Television Vehicles," unpublished doctoral dissertation, Krannert Graduate School of Industrial Administration, Purdue University, August, 1971. The data base is a portion of a large-scale study conducted by Lever Brothers, Inc., (January 1970).

Hustad, Thomas P. and Edgar A. Pessemier, "A Review of Current Developments in the Use of 'Attitude and Activity' measures in Consumer Market Research," Marketing Science Institute, March 1971.

Hustad, Thomas P. and Edgar A. Pessemier, "Industry's Use of Life Style Analysis: Measures, Segmenting Consumer Markets with Activity and Attitude presented at the 54th International Marketing Congress, American Marketing Association, San Francisco, California, April 12-15, 1971.

Pessemier, E. A., DeBruicker, F. S., and T. P. Hustad, 'The Purdue Consumer Behavior Research Project," unpublished working paper, Krannert Graduate School of Industrial Administration, Purdue University, August, 1971.

Summers, John 0., "The Identity of the Women's Clothing Fashion Transmitter,' unpublished doctoral dissertation, Krannert Graduate School of Industrial Administration, Purdue University, 1968.

Tigert, Douglas J., 'Consumer Typologies and Market Behavior," unpublished doctoral dissertation, Krannert Graduate School of Industrial Administration, Purdue University, 1966.

Tigert, Douglas J., "Psychographics: A Test-Retest Reliability Analysis,' Marketing Involvement in Society and the Economy, Proceedings of the American Marketing Association, August 1969, pp. 310-315.

Tigert, Douglas J., The author has published several research efforts relating to this data base; it consists of the responses of 1800 English-speaking Canadian women to a variety of questions relating to market-related behavior (See Burke Trendtape, June 1970).

Wells, William D. and Douglas J. Tigert, "A Consumer Attitude Inventory, unpublished working paper presented at the June 1968 Conference of the American Marketing Association, The Graduate School of Business, the University of Chicago.

Wilson, Clark L., "Homemaker Living Patterns and Marketplace Behavior--A Psychometric Approach," New Ideas for Successful Marketing. Proceedings of the 1966 World Congress, American Marketing Association, 1966, pp. 305-331.

Wind, Yoram, "Life Style Analysis: A New Approach," paper presented at the 54th International Marketing Congress of the American Marketing Association, April 12-15, 1971.

----------------------------------------