Methodological Advance in Consumer Research

Naresh K. Malhotra, Georgia Institute of Technology
ABSTRACT - This paper provides discussant comments on the papers by Holbrook and Moore (1983), Murphy (1983), and Dillon and Madden (1983) presented in the session on Methodological Advances in Consumer Research. After discussing each paper individually, an attempt is made to integrate the three papers from the perspective of the analysis of categorical data.
[ to cite ]:
Naresh K. Malhotra (1984) ,"Methodological Advance in Consumer Research", in NA - Advances in Consumer Research Volume 11, eds. Thomas C. Kinnear, Provo, UT : Association for Consumer Research, Pages: 63-65.

Advances in Consumer Research Volume 11, 1984      Pages 63-65


Naresh K. Malhotra, Georgia Institute of Technology


This paper provides discussant comments on the papers by Holbrook and Moore (1983), Murphy (1983), and Dillon and Madden (1983) presented in the session on Methodological Advances in Consumer Research. After discussing each paper individually, an attempt is made to integrate the three papers from the perspective of the analysis of categorical data.


As Olson (1981) remarked in his presidential address, the field of consumer behavior is rapidly emerging as a science. Along with theory development, methodological advances in consumer research are taking place at a fast pace. Hence, a session on this topic in the 1983 ACR conference is only appropriate. The purpose of our paper is to offer commentaries on the three papers presented in this session. The session comprised of papers by Holbrook and Moore (1983) on the pick-any procedure, Murphy (1983) on uninformed response, and Dillon and Madden (1983) on inter-judge reliability and latent class models. All the three papers are well written and are recommended to the readers. Specific comments on the papers follow.


Consumer researchers are constantly faced with the tasks of handling missing data and reducing the demands placed on the respondents while collecting primary data. In this respect, Levine's (1979) Pick-any procedure seems to be a welcome addition to the arsenal of multidimensional scaling techniques. The basic input into the pick-any procedure consists of binary information indicating whether each respondent's preference set includes or excludes each alternative in the choice set. Therefore, this procedure allows respondents to ignore the unfamiliar alternatives and does not require detailed (interval or ratio scaled) information on the chosen alternatives. Hence, as compared to the conventional method of multidimensionally-scaled correlations, the pick-any method is less taxing on the respondents and also adjusts for missing data in a natural way. Given that the pick-any procedure uses only binary level input, it might not be expected to perform as well as the multidimensionally-scaled correlations method which typically uses as input interval scaled data on all the alternatives in the choice set, assuming of course that good quality interval scaled data could be obtained on all the choice alternatives. However, one might expect the performance of the pick-any procedure to be comparable to, or even better than, that of multidimensionally-scaled correlations under the following conditions.

1. Most of the respondents are not familiar with a large number of alternatives in the choice set. This is not an unlikely situation given that size of the evoked set is small (Malhotra, Pinson and Jain 1980).

2. Obtaining interval scaled information on all the choice alternatives imposes too burdensome a task on the respondents resulting in fatigue, boredom, loss of interest and, hence, poor quality data. This could, for example, occur if a large number of alternatives were being rated on a large number of attributes.

3. Interval scaled data such as similarity ratings are obtained on all the choice alternatives. However, the proportion of missing data is large. This could happen either by design due to the researcher wishing to minimize the load on the respondents or due to unusual circumstances.

In the empirical comparison by Holbrook and Moore (1983), none of the foregoing conditions are met. In fact, the choice alternatives, consisting of different breeds of dogs, were specifically selected so as to be familiar to the respondents. The respondent evaluation task was kept to a reasonable load. Furthermore, an attempt was made to obtain complete information from the respondents, that is, there was no missing data due to the design adopted. Hence, it is not surprising that in the Holbrook and Moore (1983) study the performance of multidimensionally-scaled correlations was superior to that of the pick-any procedure. However, the comparison of model fits for the pick-any procedure and multidimensionally-scaled correlations should be tempered by a realization that the comparison, as conducted by the authors, is not appropriate in the strict sense. For the multidimensionally-scaled correlations method, the input as well as the output is intervally scaled and, as such, Pearson's product moment correlations between the two can be obtained. However, for the pick-any procedure, while the output is intervally scaled, the input consists of only binary data. Hence, Pearson's product moment correlations between the two are not appropriate. Rather, the fit for the pick-any procedure should be examined via point biserial correlations. Another consideration which should be borne in mind is that for the pick-any procedure the preference data was dichotomized by arbitrarily assigning each respondent's three highest-rated dogs with scores of one, with zeros being assigned to the other dogs. While the authors tried other cut-off points as well, these cut-offs were uniformly applied to all the respondents. The realistic possibility of different cutoff points (evoked set sizes) for different respondents could not be taken into account by Holbrook and Moore (1983).

A more direct comparison between the two techniques could have been made in terms of recovering a known underlying structure. For this purpose Monte Carlo technique would have been appropriate. Overall, the study by Holbrook and Moore is interesting and does provide useful information on the relative performance of the pick-any procedure.


The problem of uninformed response in survey research is one which deserves more attention from consumer researchers. One approach to examining this problem has been to study subjects' response to fictitious and obscure issues by using a five point agree-disagree scale (Hawkins and Coney 1981; Murphy 1983). However, the use of this approach raises several issues. We disagree with Hawkins and Coney (1981, p. 372) that "any response about the past performance of a fictitious entity must be uninformed". Rather we tend to agree with Murphy (1983) that in responding to a fictitious entity on a five point agree-disagree scale, the selection of the scale mid-point is equivalent to a neutral, "don't know", or "no opinion" response. Also, it appears reasonable that in some cases, either an agree or disagree response in reacting to a fictitious or obscure issue may be appropriate rather than uninformed.

In the specific study conducted by Murphy (1983) only agreements with the statement "The National Bureau of Consumer Complaints (NBCC) provides an effective means for consumers who have purchased a defective product to obtain relief" should be scored as uninformed responses. Even under this conservative criteria, the percentage of uninformed responses varies from 30% to 42% in the four treatment conditions employed by Murphy. This rate of uninformed response is considerably high as to cause concern. Consider the finding that 302 of respondents expressed agreement with the positive statement about NBCC even when they were informed that NBCC did not exist and provided with a "don't know" option. Could it be that this result is due to a lack of involvement of the student subjects with the task and as such reflects indifference rather than uninformed resPonse?

We would also like to point out that the chi-square analysis reported by Murphy (1983) in tables 2 and 3 is not appropriate as several cells have a size which is too small, less than the recommended norm of 5 (Mendenhall and Scheaffer 1973). A more appropriate procedure would have been to pool the "strongly agree" and "moderately agree" responses and examine the percentage agreement or uninformed response as a function of the treatment conditions as indicated in Table 1. Given the cell sizes, logit or probit analysis could then be used to examine the treatment effects as illustrated in Malhotra (1982a, 1982b, 1983, 1984) and Malhotra, Jain and Lagakos (1982).



A final comment on this topic relates to the possibility that subjects respond to fictitious or obscure issues not by merely flipping mental coins, as the concept of uninformed response might suggest. Rather, subjects lacking opinions about the particular issues referred to in a question may construct answers by drawing on an underlying disposition not specific to the issue but relevant to it (Schuman and Presser 1980). Thus attitude towards National Bureau of Consumer Complaints may be formed based on attitude toward consumerist or consumer advocate organizations.


There are a variety of situations in which a consumer researcher is faced with the task of assessing the reliability of judges who have coded cognitive responses obtained from consumers. Such situations arise not only in the context of assessing effects of persuasive communications as argued by Dillon and Madden (1983), but also in the analysis of protocols in information processing studies (Biehal and Chakravarti 1983); use of projective techniques, for example, measurement of integrative ability by the Paragraph Completion Test or Impression Formation Test (Goldstein and Blackman 1978) etc. Hence, an illustration of the applicability of latent class models to the problems of assessing inter-judge reliability makes a useful contribution.

The study by Dillon and Madden (1983) is a good application in this respect. The authors give a clear exposition of the latent class approach and also discuss its advantages over the more traditional measures of inter-judge reliability such as Kappa and weighted Kappa and its various modifications and extensions (Cohen 1960, 1968; Light 1971, Fleiss 1971; Landis and Koch 1977).

However, the authors do not discuss the limitations of the latent class approach. Among other limitations, it should be pointed out that the size of the cross-classification table increases exponentially with increase in the number of judges and/or categories thereby increasing the complexity of the analysis. Also, problems could arise if some of the judges do not use all of the categories or do not rate all the subjects. Also, selecting an appropriate model may not be straight forward when one model fits the data slightly better but the other is more parsimonious.

The enthusiasm of Dillon and Madden (1983) to use inter-judge reliability to assess the quality of data should be tempered by due caution. High inter-judge agreement is desirable but high agreement alone is not sufficient to insure the quality of the data that are collected. Evidence of reliability and validity of the data should also be reported (Mitchell 1979). In this context it should be emphasized that reliability and inter-judge agreement are not the same. It is possible to have low inter-judge agreement and a high reliability (correlation) coefficient, and vice versa (Tinsley and Weiss 1975). For example, if two judges rated the complexity of subjects' verbal responses on a seven point scale such that one judge consistently assigned a lower complexity by one scale unit, the judges' ratings would correlate perfectly. However, there would be no agreement between these judges.


Having commented on each of the three papers individually, some general comments are also in order. The common theme binding the three papers may be described as the analysis of categorical data. In the case of the pick-any procedure, the nature of data being analyzed is binary. For uninformed response, the input data may be binary consisting of agreement and nonagreement or categorical comprising the agree, neutral or disagree responses. In the latent class approach, the basic data consists of assignments by judges to different categories. The analysis of categorical data is becoming increasingly important in consumer research. Hence it is appropriate that the papers by Holbrook and Moore (1983), Murphy (1983) and Dillon and Madden (1983) be viewed in this light and that the analysis of categorical data be highlighted as one of the methodological advances in consumer research.


Biehal, Gabriel and Chakravarti, Dipankar (1982), "Experiences with the Bettman-Park verbal-protocol Coding Scheme", Journal of Consumer Research, 8, 442-448.

Cohen, F. (1960), "A Coefficient of Agreement for Nominal Scales," Educational and Psychological Measurement, 20, 37-47.

Cohen, F. (1960), "Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit, Psychological Bulletin, 70, 213-20.

Dillon, William R. and Madden, Thomas J. (1983), "Assessing Inter-Judge Reliability: A Probabilistic Latent Class Approach", in Advances in Consumer Research, vol. 11, ed. Thomas C. Kinnear, Ann Arbor, MI: Association for Consumer Research

Fleiss, J.L. (1971), "Measuring Nominal Scale Agreement Among Many Raters," Psychological Bulletin, 76, 378-382.

Goldstein, Kenneth, M. and Blackman, Sheldon (1978), Cognitive Style. New York: John Wiley and Sons.

Hawkins, D.I. and Coney, K.A. (1981), "Uninformed Response Error in Survey Research," Journal of Marketing Research, 18, 370-4.

Holbrook, Morris B. and Moore, William L. (1983), "The Pick-Any Procedure Versus Multidimensionally-Scaled Correlations: An Empirical Comparison of Two Techniques for Forming Preference Spaces," in Advances in Consumer Re- search, vol. 11, ed. Thomas C. Kinnear, Ann Arbor, MI: Association for Consumer Research.

Landis, J.R. and Koch, G.G. (1976b), "A One-Way Components of Variance Model for Categorical Data," Biometrics, 33, 617-79.

Levin, Joel H. (1979), "Joint-Space Analysis of 'Pick-Any' data: Analysis of Choices from an Unconstrained Set of alternatives," Psychometrika, 44, 85-92.

Light, R.J. (1971), "Measures of Response Agreement for Qualitative Data: Some Generalizations and Alternatives," Psychological Bulletin, 76, 365-377.

Malhotra, Naresh K. (1982a), "Information Load and Consumer Decision Making," Journal of Consumer Research, 8, 419-430.

Malhotra, Naresh K. (1982b), "Structural Reliability and Stability of Nonmetric Conjoint Analysis," Journal of Marketing Research, 19, 199-207.

Malhotra, Naresh K. (1983), "A Comparison of the Predictive Validity of Procedures for Analyzing Binary Data," Journal of Business and Economic Statistics, 1,326-336.

Malhotra, Naresh K., Jain Arun K. and Lagakos, Stephen (1982), "The Information Load Controversy: An Alternative Viewpoint," Journal of Marketing, 27-37.

Malhotra, Naresh K., Pinson C., and Jain, Arun K. (1980), "Accommodative Cognitive Style Differences in Consumer Reduction of Alternatives," in Advances in Consumer Research, vol. 7, ed. Jerry C. Olson, Ann Arbor: Association for Consumer Research. 541-546.

Mendenhall,William and Scheaffer, Richard (1973), Mathematical Statistics with Applications, North Scituate, MA: Duxbury Press.

Mitchell, S.K. (1979), "Interobserver Agreement, Reliability, and Generalizability of Data Collected in Observational Studies," Psychological Bulletin, 16, 376-90.

Murphy, John M. (1983), "Methodological Problems Related to the Use of Fictitious or Obscure Issues to Investigate Uninformed Response in Survey Research," in Advances in Consumer Research, vol. 11, ed. Thomas C. Kinnear, Ann Arbor, MI: Association for Consumer Research.

Olson, Jerry C. (1981), "Presidential Address - 1981: Toward a Science of Consumer Behavior," in Advances in Consumer Research, vol. 9, ed. Andrew Mitchell, Ann Arbor, MI: Association for Consumer Research.

Schuman, H. and Presser, S. (1980), 'Public Opinion and Public Ignorance: The Fine Line Between Attitudes and Nonattitudes," American Journal of Sociology, 85, 1214-12225.

Tinsley, M.E. and Weiss, D.J. (1975), "Interrater Reliability and Agreement of Subjective Judgments, " Journal of Counseling Psychology. 22. 358-376.