The Impact of Data Collection Procedure on Choice Rule

ABSTRACT - In a laboratory study involving choice among 16 alternative "brands" of typewriters, the effect of data collection procedure on choice rule was investigated. Protocol analysis was used to identify the choice rule used by 57 MBA students in two conditions. Subjects who were permitted to sort the information cards, as is the common practice in conjoint analysis studies, tended to adopt a lexicographic (by attribute) rule, whereas subjects who were not permitted to manipulate the cards tended to adopt a conjunctive (by brand) rule.

Citation:

Richard W. Olshavsky and Franklin Acito (1980) ,"The Impact of Data Collection Procedure on Choice Rule", in NA - Advances in Consumer Research Volume 07, eds. Jerry C. Olson, Ann Abor, MI : Association for Consumer Research, Pages: 729-732.

Advances in Consumer Research Volume 7, 1980     Pages 729-732

THE IMPACT OF DATA COLLECTION PROCEDURE ON CHOICE RULE

Richard W. Olshavsky, Indiana University

Franklin Acito, Indiana University

ABSTRACT -

In a laboratory study involving choice among 16 alternative "brands" of typewriters, the effect of data collection procedure on choice rule was investigated. Protocol analysis was used to identify the choice rule used by 57 MBA students in two conditions. Subjects who were permitted to sort the information cards, as is the common practice in conjoint analysis studies, tended to adopt a lexicographic (by attribute) rule, whereas subjects who were not permitted to manipulate the cards tended to adopt a conjunctive (by brand) rule.

INTRODUCTION

Research on consumer decision making over the past 15 years has shown that brand choice can and does occur by a variety of choice rules. Bettman (1979) lists ten rules: affect referral, linear compensatory, general information integration, conjunctive, disjunctive, lexicographic, sequential elimination, elimination by aspects, lexicographic semiorder, and additive difference. Wright and Barbour (1977) add yet another rule to this list, attribute dominance. Bettman (1979) further indicates that the selection of choice rule depends upon both characteristics of individual decision makers and factors relevant to the decision task and situation. Research has begun to identify those factors which determine which of the choice rules will be used in a particular task and situation.

Some of the task and situational factors which have already been identified as determinants of choice rule include: (1) Number of alternatives and amount of information per alternative. When presented with a large number of alternatives, subjects were observed to use noncompensatory choice rules (e.g., conjunctive) which serve to reduce the number of alternatives to a manageable size; then compensatory choice rules (e.g., additive difference) are applied to the few remaining alternatives (Lussier and Olshavsky 1974; Payne 1976); (2) Format. The type of information acquisition strategy used, which has direct implications for choice rule, has been found to be influenced by the format of the information (Bettman and Kakkar 1977; Tversky 1969); (3) Time. When placed under time pressure, subjects were observed to use different, simpler choice strategies (Wright 1974); (4) Attribute complexity. Research by Park (1978) suggests that subjects use simplification strategies if several values per attribute are presented.

Another situational factor which may have a significant influence on choice rules but which has not yet been investigated is the data collection procedure adopted by the experimenter, in laboratory or field studies. The procedures used in studies of brand choice vary considerably. In some studies, e.g., those involving conjoint and multidimensional scaling (Green and Wind 1973; Green and Srinivasan 1978), subjects are presented with cards and explicitly instructed to sort the cards into piles. In other studies subjects are instructed to obtain information from a display board, e.g., information monitoring studies (Jaccoby, Szybillo, and Busato-Schach 1977; Payne 1976). And in some cases, e.g., protocol analysis (Lussier and Olshavsky 1974; Bettman 1970) subjects are presented with cards and instructed that they may not manipulate the stimulus materials or they cannot do so by virtue of the nature of the products involved. (In actual brand choice situations, stimulus materials may or may not be manipulable. For example, brochures describing alternative life insurance companies or condominium apartments are manipulable while large, durable goods, like refrigerators or typewriters, are not.)

The purpose of this study was to investigate the influence of data collection procedure on choice rule. Specifically, it was hypothesized that when subjects could freely sort stimulus materials, then they would most likely adopt a choice rule involving processing on a single attribute at a time (by attribute). When subjects could not manipulate the stimulus materials, then they would adopt a rule involving consideration of each alternative as a whole (by brand). Processing by attribute would be inhibited when the freedom to sort the cards is not present.

METHOD

The results reported here are based on two of a series of experiments involving protocol analysis of choice rules. The first of the two studies involved 37 MBA students while the second involved 20 MBA students. Laboratory conditions and procedures were identical in both studies, except for manipulation of the task description.

The experimental task involved expressing preferences regarding alternative typewriter profiles. Typewriters were selected as a good example of a multiattribute product of relevance to students. Only students who expressed some familiarity and experience with typewriters were asked to participate; subjects were compensated for their participation.

Profiles describing typewriter alternatives were printed on 5" x 7" cards, with each card listing five attributes. The five attributes were each offered at two levels: price ($180, $210), character size (pica, elite), platen release key (yes, no), tabulator (yes, no), and warranty period (2 years, 5 years). The attributes had been identified by earlier research as being important to students in the selection of typewriters; the levels were chosen so that realistic attribute combinations would result.

A subset of 16 of the 32 possible typewriter combinations was selected according to an orthogonal array design (Plackett and Burman 1946) and used as the analysis set. The remaining subset of 16 profiles was used as the validation set as described later.

Experimental Procedure

Prior to the typewriter selection task, each subject was given a "warm-up" exercise which involved a similar selection procedure, but different product category (portable, black and white television). The purpose of the warm-up task was to familiarize respondents with the thought verbalization process required by protocol analysis. After the warm-up task, subjects were asked to assume that they were about to purchase a portable electric typewriter for their personal use and that the 16 cards described the alternatives available to them. The five attributes and their levels were then explained,

The "Fixed" Condition.  Thirty-seven subjects of the first study were assigned to this condition. The 16 alternatives were shuffled (for each subject) and arranged in a 4 x 4 array fashion on a table and the subjects were asked to examine the "typewriters" and to select the one machine, from the set of machines offered, that would be purchased. Subjects were instructed that they could not touch or in any way manipulate the cards. Subjects were further instructed to verbalize all their thoughts as the typewriters were considered. The experimenter, who was seated opposite the subject, constantly monitored the protocol and reminded the subject to articulate his/her thoughts whenever necessary. Each session was tape recorded with the subject's knowledge.

After the subject made a choice, that alternative was eliminated from the set and he/she was asked to imagine that the chosen alternative was no longer available for sale. The subject was instructed to again make a "purchase'' decision from the remaining alternatives. This procedure was repeated until a preference ordering for all 16 alternatives was obtained.

To provide data needed to compare the predictive ability of the models resulting from the protocol analysis, the experimental procedure was repeated immediately using the 16 typewriter profiles not used in the first task. Instructions to subjects and experimental conditions for this second set of 16 profiles were the same as previously described. The data obtained during the first half of the interview could then be used to predict the rank order of the profiles considered and the choice rule used in the second half of the interview.

The "Movable" Condition.  The stimulus materials and procedures were identical for the movable condition except that the deck of 16 alternatives was simply handed to the subject instead of being placed on the table. No further instructions concerning the sorting procedure to be used were given.

Coding Procedure

The tape recorded protocols were first transcribed and then broken into a sequence of task relevant statements as is typically done with protocol data (Newell and Simon 1972; Payne 1976; Payne, Braunstein, and Carroll 1978). On the basis of their protocols, subjects were then classified by the authors according to the type of choice strategy they used. In the interest of consistency and to ensure reliable categorization, the same definitions and coding criteria used by previous researchers were adopted (Lussier and Olshavsky 1974; Payne 1976; Wright and Barbour 1977). Space limitations permit a description of only a few of the choice rules used as a basis for classification. Among these examples, the additive, additive difference, attribute dominance, and conjunctive rules all involve processing by brand. The lexicographic rule is an example of processing by attribute.

Additive Model (ADD).  According to the additive model (Rappoport and Wallsten 1972; Coombs 1964) the overall worth of each alternative is first assessed and then a final ordinal comparison on the basis of this index is made. A subject was classified as using an ADD rule if explicit reference were made to several attributes, followed by an overall worth assessment for several alternatives, followed by a preference statement tied to this worth index. The only permissible reference to differences among alternatives was on the worth dj-mansions.

Additive Difference Model (ADD DIFF).  This choice rule involves pairwise comparisons of alternatives in distance terms along separate dimensions, followed by a summation across differences (Tversky 1969). A subject was classified as using an ADD DIFF rule if explicit references were made to the magnitude of the differences between two alternatives on several attributes and the summation of these differences resulted in a preference for one alternative.

Attribute Dominance Model (ATT DOM).  According to this model, pairwise comparisons are made between alternatives but only on an ordinal basis on several attributes (Coombs 1964; Wright and Barbour 1977). Preference for a single alternative is based on a simple counting of the number of times one alternative dominates another. Subjects were classified as using an ATT DOM rule if explicit reference were made to pairwise comparisons on several attributes followed by preference based on the number of "votes" each alternative received.

Conjunctive Model (CONJ).  According to the conjunctive model, whether a multidimensional alternative x = (x_i, ... x_i, ..., x_n) surpasses some standard y = (y₁, ... y_i, ..., y_n) depends on x_i exceeding y_i for all i (Coombs 1964). A subject was classified as using a CONJ model if an explicit reference were made to comparisons of an alternative's attributes to one or more cutoff levels.

Lexicographic Model (LEX).  According to the LEX model attributes are first ordered according to importance (Coombs 1964; Fishburn 1974). [According to Tversky (1972) the LEX model is similar to the elimination-by-aspects model.] Then all alternatives are evaluated on this attribute; those which do not exceed the cutoff limit are excluded. If more than one alternative remains, the second most important attribute is selected and the remaining alternatives are evaluated along this attribute. The procedure is repeated until all but one alternative is eliminated. A subject was classified as using this model if there were explicit reference to an ordering of attributes and if alternatives were evaluated according to the importance ranking specified.

RESULTS

"Fixed" Condition

Analysis of the protocols for subjects in the "fixed" condition (from the analysis sample only of each of the 37 subjects), revealed two basic decision models in this condition, both of which were based on a modified conjunctive decision rule (by brand). The first, Model I, involved the following process. The subject first would scan the available typewriter alternatives to determine the range of choices. Next, a set of directional preferences were formed: e.g., "I prefer elite type because you can squeeze more words per page." This would usually be done for several of the attributes; in a few cases respondents initially expressed indifference about particular attributes. Using the attribute information, the subject formed an "ideal" profile and then searched the available alternatives for one which matched his or her ideal (i.e., a conjunctive test). When no exact match was available, the subject compromised on the least important attribute. This processing strategy was repeated until all choices had been made. This model then was essentially the application of a sequence of continuously modified conjunctive tests. A single, fixed priority of attribute preferences was usually established that could be ascertained from the protocol data. Model I characterized 19 subjects.

Model II, which characterized 18 subjects also involved the sequential application of a modified conjunctive rule, but involved an interaction effect in which attribute importance changed during the processing. The subjects using Model II began just as before, establishing a set of attribute preferences and using a conjunctive model to select typewriter profiles. During the course of the selection process, usually around the midpoint of the profile set, the subject would establish a revised set of attribute preferences and begin to use the conjunctive model based upon this revised set. This usually, but not always, occurred when the subject's most important attribute was no longer available. For example, a subject might strongly prefer pica type. After the eight alternatives involving pica type were selected, only elite models remained. Some subjects would then revise their priorities, saying, for example: "If I can't have a pica type, then I really don't care as much about features and warranty any more. Price is now most important." In all but six cases, the pattern was to give price the highest importance during the evaluation of the less preferred profiles.

"Movable" Condition

Analysis of the protocols of each of the 20 subjects in the "movable" condition (again from the analysis sample only), revealed that 12 out of the 20 subjects used a decision rule which most closely approximated the lexicographic rule (by attribute). According to the lexicographic rule the most important attribute is identified and then the alternatives are sorted on that attribute. Then the second most important attribute is selected and the alternatives are sorted on that basis, and so on until all alternatives are completely ranked. This decision rule is referred to here as Model III. However, most subjects departed from a pure lexicographic model in as much as they did not always sort the cards on the basis of the most important attribute. For instance, Subject 5 initially sorted on the basis of price, yet this attribute was not relevant to this subject; apparently he sorted on this basis simply because it appeared to him to be a convenient basis. In some cases subjects encountered difficulty in making a choice given this lack of correspondence between the attribute importance for sorting and the attribute importance for choice.

The remaining eight subjects used a choice rule which could not be described easily, for it appeared to be a hybrid rule involving a conjunctive rule, additive difference, or attribute dominance rule combined with an initial lexicographic sorting rule. These models are referred to collectively as Model IV.

Validation of Choice Rule Descriptions

The identification of the choice rules and the criteria used by subjects on the basis of protocol data is an inexact process and varies in difficulty from subject to subject. Two procedures were used to check the initial rules and criteria identified during the analysis of the protocols based on the first set of profiles.

The first procedure involved prediction of the rank order of the second set (validation set) of typewriter profiles. Prediction of rank order will not uniquely identify a decision rule, since it is well known that correlational methods are often insensitive to the type of decision heuristic used (Bettman 1979). In fact, for the two decision rules identified in this study (the conjunctive with sequential relaxing of the least important criterion and the lexicographic) predictions of rank order of profiles was independent of decision rule. Once a given order of attribute importance and directionality of attribute preference was established, then identical rank orders of the 16 typewriter profiles would result. The first validation procedure, therefore, was used to check the consistency of attribute importances identified in the protocol analysis of the first set of profiles, as well as the validity of either the lexicographic or conjunctive rule.

A computer model programmed in FORTRAN was constructed to generate predictions of rank order and conversational output to indicate details of processing. Inputs to the program included type of choice model and each subject's preference order for each attribute and the preferred value for each attribute. Because of the paramorphism described earlier, the program handled Models I and III identically. The only modification required to represent Model II subjects was to determine, after each selection, if any brands remained which had the preferred value on the most important attribute. If none remained, then the subject's second set of preference orderings was substituted and processing continued with the revised set of choice criteria. The output of the program was the rank order of the 16 profiles for each subject. The program was used to predict the rank order of the 16 "analysis" profiles and the 16 "validation" profiles, in each case only using information obtained from the analysis set. (No separate computer models were constructed for Model IV subjects; predictions, presented in Table 1 for these subjects were generated with the same computer model as was used for Model I and III.)

Table 1 shows the Spearman correlation coefficients for the analysis and prediction sets for the "fixed" and "movable" conditions. Very high predictive validity was observed under both conditions. The Wilcoxon matched-pair test showed that the predictions for the analysis set were significantly better than those for the validation set (p < .01) for the "fixed" condition. For the "movable" condition, no significant differences between the analysis set and validation set predictions were observed.

The second validation task involved an examination of the protocols used by subjects during the ranking of the validation set of profiles. In each case, the decision rules used by subjects in determining their first choice were classified as being conjunctive, lexicographic or some other model. Again, classification was somewhat difficult, since subjects at times appeared to be using more than one rule. The primary criterion was the strategy initially used in evaluating the typewriter profiles. Table 2 shows the results for subjects in the two conditions. Again, there was substantial evidence that the subjects in the "fixed" condition were using a type of conjunctive rule (by brand) while those in the "movable" condition tended to use a type of lexicographic rule (by attribute).

DISCUSSION

The results of this study strongly suggest that data collection procedure does influence the type of choice rule adopted. When subjects can manipulate the cards which contain the "profiles" of alternative brands there is a strong tendency to sort these in a lexicographic (by attribute) fashion. When subjects were unable to manipulate the cards, they adopted a conjunctive (by brand) type rule. Several important qualifications should he stated however. First, even though a relatively simple task environment was selected for study (i.e., five attributes, two values per attribute) and relatively sophisticated subjects were used (i.e., MBA students), the choice rule used frequently did not fit any of the "textbook" choice rules perfectly. The choice rule actually used changes over time and often takes the form of a hybrid of several "pure" strategies. This complicates the protocol analysis task which at best is in need of reliability and validity checks in addition to those performed here. Second, the simplistic character of the task environment used here limits the generalizability of the findings. Further research is needed with more complex task environments. And, third, further research is needed to test the results observed here under longer periods of delay between the analysis set and the validation set.

TABLE 1

SPEARMAN RANK CORRELATION BETWEEN OBSERVED AND PREDICTED RANKS

TABLE 2

TYPE OF DECISION RULE FOUND IN VALIDATION SET PROTOCOLS BY EXPERIMENTAL CONDITION

REFERENCES

Bettman, James R. (1970), "Information Processing Models of Consumer Behavior," Journal of Marketing Research, 7 (August), 370-376.

Bettman, James R. (1979), An Information Processing Theory of Consumer Choice, Reading, Mass.: Addison-Wesley Publishing Company.

Bettman, James R., and Kakkar, Pradeep (1977), "Effects of Information Presentation Format on Consumer Information Acquisition Strategies," Journal of Consumer Research, 3 (March), 322-340.

Coombs, C. H. (1964), A Theory of Data, N.Y.: Wiley.

Fishburn, P. C. (1974), "Lexicographic Orders, Utilities, and Decision Rules: A Survey," Management Science, 20, 1442-1471.

Green, Paul E., and Srinivasan, V. (1978), "Conjoint Analyses in Consumer Research: Issues and Outlook," Journal of Consumer Research, 5 (September), 103-123.

Green, Paul E., and Wind, Yoram (1973), Multiattribute Decisions in Marketing: A Measurement Approach, Hinsdale, Ill.: Dryden.

Jacoby, Jacob, Szybillo, George J., and Busato-Schach, Jacqueline (1977), "Information Acquisition Behavior in Brand Choice Situations," Journal of Consumer Research, 3 (March), 209-216.

Lussier, Denis and Olshavsky, Richard W. (1974), "An Information Processing Approach to Individual Brand Choice Behavior," Annual Conference of the Operations Research Society of America/The Institute of Management Science, San Juan, De Puerto Rico, Fall. (In press, Journal of Consumer Research, September 1979).

Newell, Allen add Simon, Herbert (1972), Human Problem Solving, Englewood Cliffs, NJ: Prentice-Hall.

Park, C. Whan (1978), "A Seven-Point Scale and a Decision-Maker's Simplifying Choice Strategy: An Operationalized Satisfying Plus Model," Organizational Behavior and Human Performance, 21, 252-271.

Payne, John W. (1976), "Task Complexity and Contingent Processing in Decision Making: An Information Search and Protocol Analysis," Organizational Behavior and Human Performance, 16 (August), 366-387.

Payne, John W., Braunstein, L., and Carroll, J. S. (1978), "Exploring Predecisional Behavior: An Alternative Approach to Decision Research," Organizational Behavior and Human Performance, 22, 17-44.

Payne, John W., Ragsdale, Easton E. K. (1978), "Verbal Protocols and Direct Observation of Supermarket Shopping Behavior: Some Findings and a Discussion of Methods," in H. Keith Hunt, Advances in Consumer Research, Vol. 5, Proceedings of the Eighth Annual Conference, ACR.

Plackett, R. L. and Burman, J. P. (1946), "The Design of Optimum Multi-Factorial Experiments," Biometrika, 33, 305-325.

Rappoport A., and Wallsten, T. S. (1972), "Individual Decision Behavior," Annual Review of Psychology, 23, 131-179.

Tversky, A. (1972), "Elimination by Aspects: A Theory of Choice," Psychological Review, 79, 281-299.

Tversky, A. (1969), "Intransitivity of Preferences," Psychological Review, 76 (January), 31-48.

Wright, Peter (1974), "The Harassed Decision Maker: Time Pressure, Distractions, and the Use of Evidence," Journal of Applied Psychology, 59, 555-561.

Wright, Peter, and Barbour, Fredrick (1977), "Phased Decision Strategies: Sequels to an Initial Screening," in M. K. Start and M, Zeleny (eds.), Multiple Criteria Decision Making: TIMS Studies in the Management Sciences, Vol. 6, Amsterdam: North-Holland.

----------------------------------------