Reliable and Valid Measurement of Memory Content and Structure As a Function of Brand Usage Patterns

ABSTRACT - The measurement of product knowledge has been examined by a number of researchers using a variety of methods. Differences in usage patterns of consumers may provide an explanation for low overall measurement reliability in past research. This study examines the content and structure of loyal and non-loyal consumers' memory for information about soft drinks using a test-retest procedure. For loyal brand users in a category, cognitive structure is found to be unidimensional. The recall of brands and brand information is focused on the loyal brand, resulting in greater consistency of measurement between occasions. Conversely, the cognitive structure of non-loyals is multidimensional. Information that is most accessible in memory varies to a greater extent between occasions, moderating measurement reliability.

Citation:

Karen Finlay (1996) ,"Reliable and Valid Measurement of Memory Content and Structure As a Function of Brand Usage Patterns", in NA - Advances in Consumer Research Volume 23, eds. Kim P. Corfman and John G. Lynch Jr., Provo, UT : Association for Consumer Research, Pages: 282-288.

Advances in Consumer Research Volume 23, 1996      Pages 282-288

RELIABLE AND VALID MEASUREMENT OF MEMORY CONTENT AND STRUCTURE AS A FUNCTION OF BRAND USAGE PATTERNS

Karen Finlay, University of Guelph

ABSTRACT -

The measurement of product knowledge has been examined by a number of researchers using a variety of methods. Differences in usage patterns of consumers may provide an explanation for low overall measurement reliability in past research. This study examines the content and structure of loyal and non-loyal consumers' memory for information about soft drinks using a test-retest procedure. For loyal brand users in a category, cognitive structure is found to be unidimensional. The recall of brands and brand information is focused on the loyal brand, resulting in greater consistency of measurement between occasions. Conversely, the cognitive structure of non-loyals is multidimensional. Information that is most accessible in memory varies to a greater extent between occasions, moderating measurement reliability.

The content and structure of product category information in memory is a topic of growing interest to researchers (Brucks 1986, Gutman 1980, Hirschman and Douglas 1981, Kanwar, Olson and Sims 1980, Mitchell and Dacin 1995). Knowledge content refers to the amount and type of information stored in memory. Knowledge structure refers to the organizational properties of information in memory - properties describing the configuration of nodes representing concepts in memory and the links connecting these nodes. Analysis of knowledge content and structure attempts to delineate the concepts stored in memory about a domain and the way these concepts are configured or organized in memory. The content and structural properties of stored information may affect how consumers search for information and how they process and store new information.

Although several approaches have been used to measure memory content and structure, few attempts have been made to assess the reliability of methods used to measure constructs, convergent validity using the same or different methods, or the discriminant validity of constructs conceived to be distinct. Typical constructs involved in the measurement of memory may not be stable overtime, since new knowledge may be acquired. Nevertheless conceptual variables can be identified and are expected to be constant within a short time horizon, facilitating the worthwhile examination of the reliability and validity of methods used in the literature to measure constructs.

The few studies that have examined reliability and validity issues (Kanwar, Olson and Sims 1980, Olson and Muderrisoglu 1979) did not find strong evidence of measurement reliability or construct validity. Researchers have argued that the probabilistic process of information retrieval from memory and problems of information interference inhibit the potential for strong results. It may be, however, that additional factors moderate reliability and validity. The present research seeks to: identify reliable methods of measuring memory content and structure; examine construct validity of measures of content and structure; and explore the systematic impact of product usage history in limiting the potential for methods to reliably measure constructs.

The reliability and validity of three methods are tested in this study: card sorting (Gutman 1980, Hirschman and Douglas 1980, Marks 1985), free elicitation (Kanwar, Olson and Sims 1980, Mitchell and Dacin 1995, Olson and Muderrisoglu 1979) and category membership elicitation (Rosch and Mervis 1975). These methodologies will be described in detail in the procedures for the study. Table 1 describes the set of seven properties of the content and structure of knowledge measured in this research. Measures were chosen which quantify the size of the knowledge structure, measure the fragmented versus unified nature of it organization, and measure the extent to which information within the structure is categorized or grouped. It was judged these measures would be useful in identifying optimal communication strategies and positionings for brands within the domain. A discussion of which specific properties are of conceptual interest to the study of knowledge content and structure, however, is beyond the scope of this paper. These seven constructs include structural properties examined previously by Scott, Osgood and Peterson (1979). The reliability of the three methodologies is tested using the seven proposed measures of knowledge content and structure for the product class of soft drinks.

It is argued that the reliability and validity of measurement of the content and structure of product consumer product knowledge depends partially on whether a consumer is a loyal brand user (defined in this study similarly to Jacoby and Chestnut's (1978) "hard-core criterion", i.e. use 1 brand 80% or more of occasions) or a non-loyal brand user. An individual may use a variety of brands of soft drinks and hence, may be more receptive to perceiving and processing information about product alternatives and integrating it with information stored in existing knowledge structures. Consequently, the cognitive structure of such an individual may be rich and diverse. The likelihood that a non-loyal user will access the same product information from one occasion to the next is anticipated to be low, however, thereby negatively impacting measurement reliability for all methods and the validity of constructs measured by the same or alternate methods.

A loyal user of a single brand, on the other hand, may have learned overtime to tune out information unrelated to the habitually-used brand. A loyal user may therefore have limited knowledge of the category, stored in a more unidimensional, rigid structure. From one retrieval occasion to the next, a loyal user should be more likely to access the same product information, enhancing measurement reliability and validity. In this study, undergraduate commerce students provided measures of cognitive content and structure using a test-retest design. Differences in reliability, and the convergent and discriminant validity of constructs measured using the same and different methods are hypothesized and assessed separately for both loyal and non-loyal brand users.

METHOD

Choice of the soft drink category for the study was driven by the following considerations: i) the category must be familiar to an undergraduate university sample; ii) the category must be divisible into a variety of sub-categories; iii) a number of different brands must be available within the category; and iv) subsamples of brand loyal and non-loyal consumers should be available.

A considerable amount of time and effort was required to obtain all the measures necessary to assess each subject's knowledge content and structure in two separate sessions (combined duration: 2 hours). Consequently, the number of subjects in the study was necessarily small, consistent with sample sizes employed by other researchers in the paradigm (Kanwar Olson and Sims 1980, Mitchell and Dacin 1995). Twenty individuals attending an undergraduate business course at a large, urban, university participated for course credit. Two sessions were held two weeks apart within a classroom situation. The two-week interval was chosen to allow subjects' memory of first session responses to dissipate, while limiting the amount of new information that might be naturally acquired about the product class as a result of advertising exposure or product usage. Subjects were aware that a second research session was required, but were told that this research was unrelated to that of the first session.

Procedures

The following set of procedures were used to measure the seven content and structure constructs using the three methodologies. At the first session subjects were given one-and-a-half minutes to "list all the brands of soft drinks they could think of". They were then given a set of blank cards and asked to write one of the brands they recalled on each card. Subjects were asked to sort the cards into broad natural groupings and to list the brands from each group on a sheet along with an indication of why brands were grouped together (card sorting). Subjects were then asked to re-shuffle the cards and form different groupings.

Following the card sorting, free elicitation was taken for four of the recalled brands - the first two brands the individual had recalled and the last two brands recalled. Subjects were asked to write the name of these brands at the top of a sheet and then write everything that came to mind when they thought of the brand. Free elicitation was administered after card sorting as it was feared that memory associations recalled about brands using elicitation might influence the number of natural brand groupings formed.

The second session was exactly the same as the first session until the point of the free brand elicitation. For the second session, the elicitation task was broadened in two respects: 1) the elicitation task was performed for all brands recalled by the subject ("everything that comes to mind when you think of the brand") [Due to time constraints, free elicitation was not taken for all brands in both sessions. Instead, priority was given to obtaining elicitation for the first 2 and last 2 brands in an effort to determine whether measurement was more consistent for brands that were more versus less accessible in memory.]; and 2) after free elicitation for all brands, category membership for each brand was elicited. The subjects were told that the bottom of the free elicitation page contained five lines and they were asked to write the name of a category to which the brand belonged on each line. The following example was provided: "For example, if the brand were Cheer laundry detergent, you might write laundry detergents, lower-priced detergents, and detergents that can be used in cold water". They were told to write categories to which the brand naturally belonged and not to be concerned with filling all five lines. After all elicitations, subjects were asked to use a 7-point scale to indicate how typical or good an example the various brands were of the various categories of soft drinks they had formed at the first session (questionnaires individualized for each subject). This provided a measure of brand typicality, hypothesized to be unrelated to measures of knowledge content and structure, for use in tests of discriminant validity of knowledge measures. A given brand may be judged a more or less typical instance of a brand grouping within a domain. Perceived typicality of a specific brand should be unrelated to the amount of information contained in a domain or how all brands within the domain are organized. The session concluded by asking usage and demographic information. On the basis of usage data, subjects were classified as loyal users if they used the same brand more than 80% of usage occasions; otherwise they were non-loyal users in the current study.

Hypotheses and Results

Measurement Reliability. It was hypothesized that reliability coefficients would be higher among loyal users than among non-loyal users for all methods employed in a test-retest procedure for all measures. Since loyal users are hypothesized to store less information in memory, the likelihood that paths to the same memory items will be activated between occasions should be higher for loyal users. Furthermore, information most accessible at any given point in time should be related to the loyal brand. Consequently, the probability that the same information will be recalled between occasions is anticipated to be greater for loyal than for non-loyal users, enhancing consistency of measurement.

Loyal and non-loyal users were equally reliable in the number of brands accessed or elicited between sessions as indicated by correlation coefficients (.774 loyals, .879 non-loyals versus, difference p<.23, Table 2). Loyal users, however, tended to recall more of the same brands between sessions (87.0% versus 74.5% same brands recalled for non-loyals, T=-1.91, p<.07). Brands accessed by loyal users tended to come from categories to which the loyal brand also belonged (54.4% of all brands for loyal users versus only 37.9% for non-loyal users, based on brand used most often, T=3.03, p<.01). Loyal users appear more focused in their recall of brands. The loyal brand was the first brand accessed 66.7% of the time and the first or second brand 83.3% of the time. Taken together, these results provide support for the hypothesis that the retrieval of brands by loyal users is cued by the loyal brand or influenced by the categories to which the loyal brand belongs, resulting in greater consistency in brands recalled between sessions.

The reliability of the free elicitation method was assessed based on memory item elicitation cued by the first two and the last two brands recalled by respondents at both sessions. For loyal users, it was anticipated that reliability in terms of the number of elicited memory items and the consistency with which items were elicited would be greater for the first two brands recalled than for the last two brands recalled. The loyal brand, with which the loyal user is highly familiar, was expected to be one of the first two brands recalled in both of the sessions. Consequently, memory items elicited in response to brand names recalled should be relatively consistent between the two sessions for the first two brands recalled. On the other hand, brands recalled in the last two positions by a loyal user were expected to vary to a greater extent between sessions. Consequently, lower consistency of items elicited about the last two brands between sessions should result for loyal users. Since brands recalled by non-loyal users in either the first two or the last two positions were expected to be equally random, no difference in the reliability of memory items elicited in response to the first two brand names versus the last two brand names was expected.

Overall, the free elicitation method appears to be a reliable instrument to measure the quantity of items stored in memory among both loyal and non-loyal users (.689 correlation coefficient among loyals, .745 correlation coefficient among non-loyals). Although loyal and non-loyal users were similarly consistent in the number of memory items elicited for the first two brands recalled (.799 correlation loyals versus .762 non-loyals, p<.42), loyal users demonstrated lower reliability in the number of memory items elicited per brand for the last two brands recalled than they did for the first two brands recalled (.421 versus .799 respectively, p<.05), as hypothesized.

TABLE 1

MEASURES OF MEMORY CONTENT AND STRUCTURE

Loyal users additionally recalled fewer of the same memory items for the last two accessed brands between sessions than they did for the first two brands accessed (35.8% versus 61.5% respectively, T=2.18, p<.04, Table 2). Although loyal users appear highly consistent in the recall of memory items for the first two brands recalled, outside of items relating to the loyal brand, loyal users appear less stable in number and consistency of memory items recalled. The examination of recall for the first two versus the last two brands recalled supports the hypothesis that high relative accessibility of the loyal brand (recalled in first or second position at least four out of five times) heavily influences the extent to which the same items are recalled overall between sessions among loyal users.

The reliability of the card sorting method was assessed for the following measures: number of categories, average number of brands per category, number of memory items recalled, unity, domain dimensionality and image comparability. Reliability, when assessed in terms of the correlation in these measures between card sorting sessions was generally lower among non-loyal users for several measures, particularly number of categories (.681 loyals, .158 non-loyals, difference: p<.05), number of memory items (.648 loyals, .186 non-loyals, difference: p<.08), unity (.607 loyals, .099 non-loyals, difference: p<.05) and image comparability (.506 loyals, .016 non-loyals, difference: p<.05, Table 2). The assessment of the reliability of the card sorting method using reliability coefficients also confirmed generally stronger reliability of measurement among loyal versus non-loyal users. Among loyal users, the reliability coefficients for 3 out of 7 measures of memory content and structure were above the .7 level, while only 1 measure out of 7 was above this level for non-loyal users (Table 3).

Overall, measurement reliability using card sorting is stronger for loyal users but below acceptable levels for some measures (particularly domain dimensionality and image comparability), even among this group. Reliability of measure using card sorting is alarmingly low for non-loyal users for several measures (reliability coefficients: number of categories .22, number of memory items .39, unity .13 and image comparability .07. Future research should consider limitations in measurement reliability using card sorting.

TABLE 2

RELIABILITY RESULTS AND MEANS BY SUBSEGMENTS

Free elicitation, on the other hand, did produce acceptable reliability results overall among both loyal and non-loyal users, as discussed earlier, for the two measures of memory content tested (# of brands accessed and memory item elicitation). Free elicitation, therefore, appears a strong method for measuring information content stored in memory. Future research should confirm if its strength extends to the reliable measurement of properties of memory structure.

Convergent Validity. It was hypothesized that strong convergence would result for variables measured by different methods among loyal users. Values in the validity diagonal (same variables measured by different methods) of the multitrait-multimethod matrices in Table 3 confirmed that reasonable convergence (.56 to .75 inter-item correlations) between methods was obtained among loyal users for all but two measures, domain dimensionality (.24) and image comparability (.14). Domain dimensionality and image comparability had been the two measures not reliably measured among loyal users using the card sorting method. Low reliability of measurement for one method would explain low convergence results between methods.

Further evidence of convergent validity would be provided if inter-item correlations obtained between the same variable measured by different methods were higher than the inter-item correlations between that variable and any other variable measured by the same method or by a different method. For loyal users, values in the validity diagonal (same variables measured by different methods) tended to be higher than values in adjacent columns or rows (i.e. when compared to a different variable measured by a different method), and higher than correlations between different variables using the same method. These results provide further evidence of convergent validity among loyal users. Among non-loyal users, similar analyses do not provide evidence of convergent validity. Because reliability of measurement was so low for this group, however, it is impossible to know whether lack of convergence actually exists or is a by-product of low reliability.

TABLE 3

MULTITRAIT-MULTIMETHOD MATRIX

Additional evidence of convergent validity can be found by examining results for specific measures expected to positively and negatively converge. Positive convergence had been expected among measures describing the number of categories of brands reported, the number of memory items and domain dimensionality. As more categories are formed in a domain, more memory associations should be stored which describe or rationalize the basis for categorization. Similarly, as the number of categories and the number of memory items in a domain increase, so should domain dimensionality, or the number of combinations of memory items used to describe a domain. Among loyal users, strong positive convergence resulted among these 3 measures, with an average inter-item correlation of .70 (.62 to .80 range) when the same method was used and an average inter-item correlation of .61 (.52 to .66 range) when different methods were used (Table 3). Comparable average correlations were much lower for non-loyal users (.61 same method; .12 different methods).

It was hypothesized that negative convergence would result between domain dimensionality and unity and between domain dimensionality and image comparability, particularly among loyal users. Unity and image comparability describe the unidimensionality of cognitive structure, while domain dimensionality describes its fragmentation. Using the same method, inter-item correlations among loyal users between domain dimensionality and unity and domain dimensionality and image comparability were .00 and .11 respectively, while using different methods they were -.46 and .05. While not providing strong evidence of an inverse relationship between variables, evidence of lack of convergence is encouraging.

Discriminant Validity. Evidence of discriminant validity was expected in the form lack of strong positive or strong negative intercorrelations between mean brand typicality and each of three properties of memory structure - unity, domain dimensionality and image comparability, when measured by the same or different method. Brand typicality was therefore expected to be unrelated to measures of knowledge structure. Inter-item correlations of brand typicality with these three properties of cognitive structure among loyal users were .05 -.13 and .09 when the same method was used, and .08, -.59 and .11 when different methods were used (Table 3). Evidence of independence is therefore found in five out of six cases.

CONCLUSIONS AND IMPLICATIONS

Several conclusions can be drawn from the reliability and validity analysis performed on data from this study. First, it appears, that loyal users can be more reliably measured than non-loyal users using either card sorting or elicitation methods. Based on the measures analyzed using free elicitation in this study, this method more reliably measures constructs than does card sorting. The reliability of the card sorting method was unacceptably low for some measures, even among loyal users.

Card sorting, as executed by researchers previously in the literature and in this study, does not appear to be a reliable or valid method to measure knowledge content and structure, particularly among users of a variety of brands in a category. The card sorting method, where all brands are sorted into exclusive groupings may force subjects to form categories that may not naturally exist within their cognitive structure, thereby inhibiting the potential for reliable measurement. A more natural brand grouping task for respondents may be one where a group of brands can be chosen from the pool of accessed brands without the need to group and label all remaining brands in the pool each time a group is chosen. This finding is also of relevance for market researchers in the field who frequently use card sorting as a facilitating technique in qualitative research.

Despite the fact that measurement reliability using card sorting was limited, convergence between card sorting and elicitation methods was found among loyal users for measures expected to be positively related. Preliminary discriminant validity results were also promising among loyal users. Taken together, the pattern of convergent and discriminant validity establishes support for construct validity, at least among loyal users. Contributions of this research fall in two areas. First, it has provided a better understanding of how individual usage differences can affect the reliability of measurement of memory content and structure constructs. Non-loyal users appear to less consistently retrieve the same information and the same amount of information between occasions than do loyal users. If an individual uses multiple brands in a category, the probability that information about the same brand will have been most recently activated in memory between occasions will be reduced. When asked to recall information about a category, the entry point into the network of stored information is likely to be the most recently used brand or the brand for which an ad was most recently seen and processed. Activation will spread to adjacent items in memory from the first activated brand into the stored structure of information. If the most recently activated brand is different between occasions for non-loyal users, the subset of information recalled from the domain is more likely to differ.

Loyal users, on the other hand, will be more likely to have recently activated the same brand between occasions. The most recently activated brand will tend to be the loyal brand and loyal users may have been less likely to attend to, process, or recently activate information in memory about an alternate brand between occasions.

The second contribution of this research is the identification of methods of measurement that more reliably measure memory content and structure. When both loyal and non-loyal users are sampled, free elicitation appears to perform better than card sorting. Card sorting, as executed in this study with respect to the soft drink category, appears to be a less reliable method, especially among non-loyal users.

FUTURE RESEARCH

This study began the process of identifying reliable and valid methods of measuring knowledge content and structure. As with previous studies in this area, sample size was relatively low and this should be addressed in future research. Nevertheless, it appears that the loyal/non-loyal user dichotomy appears important to the determination of differences in knowledge content and structure for the soft drink category. More finite subsegments of category users should be examined in future research to obtain a better understanding of the potential influence of usage patterns on reliable construct measurement. It can be hypothesized, for example, that several homogenous subsegments of consumers exist for the soft drink category. Consumers loyal to a particular brand may be knowledgeable about a variety of other brands, but choose one brand on the basis of conscious preference. Alternatively, the loyalty of a second segment may be based purely on habit, with little awareness and knowledge of alternative brands in a product class. Non-loyal users may consciously choose from a preferred set of brands, or their choice may be random, based on little category knowledge. Finally, a subset of users might technically be classified as non-loyal, but subcategory loyal (regularly use brands from only one subcategory, e.g. diet colas). This latter group could be hypothesized to store information unidimensionally, similar to loyal users of a single brand. Identification of subsets of consumers with homogenous cognitive content and structure for a product class would aid in the refinement of communication strategies and further the examination of the moderating effect of usage patterns on reliability and validity of measurement.

REFERENCES

Brucks, Merrie (1986), "A Typology of Consumer Knowledge Content", Advances in Consumer Research, 13, 58-63.

Gutman, Jonathan (1980), "A Means-End Model for Facilitating Analyses of Product Markets Based on Consumer Judgements), Advances in Consumer Research, 8, 116-121.

Hirschman, Elizabeth C. and Susan P. Douglas (1981), "Hierarchical Cognitive Content: Towards A Measurement Methodology", Advances in Consumer Research, 8, 100-105.

Jacoby, Jacob and Robert W. Chestnut (1978), Brand Loyalty Measurement and Management, New York, NY: John Wiley and Sons.

Kanwar, Rajesh, Jerry Olson, and Laura Sims (1980), "Toward Conceptualizing and Measuring Cognitive Structure", Advances in Consumer Research, 8, 122-127.

Mitchell, Andrew A. and Peter A. Dacin (1995), "Differences by Expertise in the

Content and Organization of Knowledge for a Product Class", Working paper, Faculty of Management Studies, University of Toronto.

Olson, Jerry C. and Aydin Muderrisoglu (1979), "The Stability of Response Obtained by Free Elicitation: Implications for Measuring Attribute Salience and Memory Structure", Advances in Consumer Research, 6, 269-275.

Rosch, Eleanor and Carolyn B. Mervis (1975), "Family Resemblances: Studies in the Internal Structure of Categories", Cognitive Psychology, 1, 573-605.

Scott, William A., D. Wayne Osgood, and Christopher Peterson (1979), Cognitive Structure. Washington, D.C.: V.H. Winston & Sons.

----------------------------------------