Cue Modality: Video and Audio Effects on Recall

Carolyn L. Costley, Texas A & M University
Duane DeWald, Texas A & M University
ABSTRACT - We present theoretical justification for a cue modality hypothesis. The hypothesis asserts that the sensory mode of retrieval cues triggers recall for same-modality information. Results from an experiment using video and audio presentations suggest that cue modality does influence recall for video material but not for audio. Pictures, furthermore, were recalled better than words regardless of cue. We suggest implications for marketing communications and recommend that future studies address other modalities and other attention conditions.
[ to cite ]:
Carolyn L. Costley and Duane DeWald (1991) ,"Cue Modality: Video and Audio Effects on Recall", in NA - Advances in Consumer Research Volume 18, eds. Rebecca H. Holman and Michael R. Solomon, Provo, UT : Association for Consumer Research, Pages: 819-825.

Advances in Consumer Research Volume 18, 1991      Pages 819-825

CUE MODALITY: VIDEO AND AUDIO EFFECTS ON RECALL

Carolyn L. Costley, Texas A & M University

Duane DeWald, Texas A & M University

ABSTRACT -

We present theoretical justification for a cue modality hypothesis. The hypothesis asserts that the sensory mode of retrieval cues triggers recall for same-modality information. Results from an experiment using video and audio presentations suggest that cue modality does influence recall for video material but not for audio. Pictures, furthermore, were recalled better than words regardless of cue. We suggest implications for marketing communications and recommend that future studies address other modalities and other attention conditions.

Most communicators want their audiences to remember the material they presented. What one remembers from a communication, however, may be context-dependent. We argue that the modality [Modality refers to any sensory mode: visual, audio, olfactory, tactile, or taste.] of a cue is one context factor that influences retrieval processes and potentially affects what one remembers. In other words, pictures trigger memory for pictures, sounds trigger memory for sounds, etc. We review research that led to this cue modality hypothesis, present test results, and discuss communication implications.

Modality effects can be particularly important to advertisers who regularly communicate via multiple modalities (primarily audio and visual, but also tactile and olfactory (scratch-and-sniff)). Advertisers desire that their audiences remember information such as brand name and product attributes, especially at decision times such as point-of-purchase or exposure to a competitor's ad (a preliminary decision).

Understanding memory-related factors can be important for designing and for evaluating marketing communications. It is essential to study and understand things that affect consumers' memories because what they remember can be critical to their decisions. While advertisements provide much information about different brands, rarely do consumers remember everything when they choose a brand. Marketing scholars have posited an influential role for memory in consumer decision processes and have called for more research on this aspect of consumer decision making (Bettman, 1986; Lynch & Srull, 1982).

We propose that the sensory mode of communication devices such as advertisements can influence consumers' memory for them. The modality of an initial stimulus (such as an ad) and the modality of a subsequent cue (such as another ad) may interactively affect memory accessibility. The reported research investigated visual and audio modes inherent in video communications. The following sections describe retrieval cue effects, perceptual processing modes and the cue modality hypothesis.

RETRIEVAL CUE EFFECTS

Encoding and retrieval are mental processes important to any discussion of memory accessibility. Information, from an ad for instance, goes into memory via encoding and is remembered via retrieval. Between encoding and retrieving, information is stored in memory. Storage form and structure and the encoding and retrieval processes interact to affect what people remember. Either encoding or retrieval processes can affect information accessibility. in the memory store. A great deal of empirical evidence supports the notion that retrieval is cue-dependent. This section reviews the principle literature on retrieval cue effects.

Tulving and Pearlstone (1966) distinguished between "available" memory information and "accessible" memory information. All events and information that have ever been encoded are available for recall in the memory store (Lachman, Lachman, & Butterfield, 1979). When it is hard for a person to find (access) certain information, forgetting results. Accessibility depends both on availability and on retrieval cues. Retrieval cues can affect the amount of information recalled by affecting information accessibility. That is, cued recall often is better than uncued recall (Tulving & Pearlstone, 1966; Tulving & Psotka, 1971).

Retrieval cues not only facilitate the level of recall but also can affect recall content. This may occur in either of two ways. Retrieval cues may bias recall content by facilitating access to information associated with the cue (Anderson & Pichert, 1978). Alternatively, retrieval cues may bias recall content by inhibiting access to other items (Rundus, 1973). The second is known as the part-list cuing effect (Slamecka, 1968, 1969).

Encoding Specificity Hypothesis

The encoding specificity hypothesis asserts that retrieval probability depends on the compatibility between stored information and information in the retrieval environment (i.e., cues) (Tulving, 1983). The hypothesis is based on the notion that a memory trace and the retrieval environment interact to influence retrieval (Tulving & Thomson, 1973). Hard-to-recall information often becomes accessible in the appropriate retrieval environment (Tulving & Pearlstone, 1966). Furthermore, according to the hypothesis, associations between the retrieval environment and the encoding environment facilitate retrieval only if information was encoded with respect to that association.

Godden and Baddeley (1975) examined the interaction between encoding and retrieval environments. extending investigations to the surrounding context. They studied under sea and dry land environments. Recall was better when the encoding and retrieval environments were the same. This research is particularly interesting because it suggests that people encode the contextual environment and that a similar retrieval environment affects recall.

Little is known about the perceptual system's role in information storage and retrieval (Tulving, 1972). There is evidence that modality information is encoded (Lehman, 1982) suggesting that modality context or modality specific processes in the encoding and retrieval environments may affect recall. The next section focuses on modality issues.

PERCEPTUAL PROCESSING MODES

There is evidence that people use unique mental processing modes for encoding and retrieving different sensory types of information (Brooks, 1968; Segal & Fusella, 1970; Glenberg, 1984). That is, visual, audio, tactile, taste, and olfactory information may be processed through different mental systems.

Brooks (1968) observed that people took longer to complete multiple process tasks that involved the same modality than they did mixed modality tasks. Brooks speculated that this indicated competition for modality specific processing capacity. Such competition supports the existence of modality specific information processing systems.

Segal and Fusella (1970) also found support for distinct processing modes corresponding to perceptual processes. Subjects first participated in a signal detection task. They indicated whether the signal was a sound, a picture, or nothing. Then subjects imagined either a visual image (e.g., a volcano) or an auditory image (e.g., the sound of a typewriter). A signal (a sound, a picture, or nothing) was presented while they were imaging. Detection of the signal was poorer when the image and the signal were in the same sensory mode than when they differed. The results indicate that imagery and perception are similar processes and that distinct processing modes may correspond to perceptual processes.

Greater competition for processing capacity between same-modality stimuli than between different-modality stimuli (demonstrated by the above two studies) suggests that unique processing systems exist and correspond to perceptual systems. Applying the encoding specificity hypothesis to this phenomenon we speculate that the processing systems employed in the encoding and retrieval contexts will interact to affect recall. That is, information encoded by one system may be retrieved better by the same system than by another.

Encoding

Modality of encoded information can also influence its accessibility. Paivio (1979) augmented the multiple processing systems idea when he proposed similar multiple storage codes [Code refers to the symbolic representation of a sensory modality.]. He specifically dealt with a visual code and a linguistic code. His theory suggests that the form of a stimulus (e.g., pictorial or verbal) triggers a processing system which stores information in a corresponding unique code. Paivio's dual-code theory maintains that the storage code for picture stimuli is relatively more accessible than the storage code for verbal stimuli. The empirical evidence that people remember pictures better than words (Edell & Staelin, 1983; Childers & Houston, 1984 are examples in marketing literature) is commonly referred to as a picture superiority effect.

Other researchers insist that a single storage code can accommodate the data explained by dual codes (Anderson, 1976, 1978). If memory networks use a single code, then modality information may reside in an associated node. A single code system is consistent with our cue modality hypothesis provided that people encode modality information.

Research results attest that information about presentation modality is processed, stored, and available in memory (Lehman, 1982; Lehman & Mellinger, 1984; Lehman, Mikesell, & Doherty, 1985; Murdock & Walker, 1969; Nilsson, 1973). Lehman and her colleagues provide evidence that modality processing is automatic according to specified criteria (Hasher & Zacks, 1979). Other research indicates that modality information may be useful during retrieval (Murdock & Walker, 1969; Nilsson, 1973).

Hasher and Zacks (1979) contend that automatically encoded information's cognitive function is to guide retrieval. If so then information modality may guide retrieval. Modality, however, may be a useful guide only under special circumstances. Nilsson (1973) found that people can organize recall by modality when instructed to do so and that they will spontaneously organize recall by modality when no other organizational basis exists.

CUE MODALITY HYPOTHESIS

The encoding specificity hypothesis and the existence of multiple processing modes, taken together, suggest a processing modality effect on recall. If context includes the operative processing mode, then recall should be better when the retrieval mode is the same as the encoding mode than when they differ.

The cue modality hypothesis posits that the perceptual mode of new information triggers a corresponding processing system and that information originally encoded through that same system will be more accessible than information encoded via another mode. It asserts, for example, that a visual retrieval cue stimulates the visual processing system, instigating quicker access to visually processed material (picture memories) than to audio material. In this case, an individual would more likely recall a picture stimulus than an auditory stimulus. or would recall it faster.

AN EXPERIMENT

We tested the cue-modality hypothesis using visual and audio stimuli presented on video tape. We hypothesized that people would more likely remember pictured items when presented with picture cues than when presented with spoken cues. Similarly, we hypothesized that people would more likely remember spoken items when presented with spoken cues than when presented with picture cues. We also tested for picture superiority, hypothesizing that pictorial information would be recalled better regardless of cue modality.

Pretests were necessary to determine category membership for the target-cue pairs, picture recognizability and naming, and encoding task. Category membership was pretested because items from the same category as target items were thought to be appropriate cues via associated links in memory (Glass and Holyoak 1986). We selected 25 categories from Battig and Montague (1969) and generated more following their procedures. [The final list of categories and items is available from the authors.]

Of the two items selected from each category, the most representative was assigned to the cue list. We did not want a high ranking target item to be recalled because it was strongly associated with the category instead of remembered from the target list. Furthermore, higher ranking items should stimulate the category node in memory and be more likely to cue other category members.

Having selected target and cue category representatives, we collected the items and videotaped them individually against the same background. Thirty-six people viewed the tape and named the items. We used the items that consistently received the same name and we used that name for the spoken stimuli.

Finally, we pretested incidental and intentional encoding task conditions. We opted for the intentional condition ("try to remember each item") because recall exhibited neither a lower bound (i.e., recall was significantly different from zero) nor an upper bound.

FIGURE A

EXPERIMENTAL DESIGN

Experimental Design

Target (2) and cue (2) modalities created the within-subject conditions. Using videotaped lists of familiar items (e.g., farm animals, furniture, fruit), we presented two codes (video and audio) of memory information and two codes of cue information. Each subject received all four modality conditions (see Figure A for the within-subject design). List content was counter-balanced across experimental conditions making four between-subject conditions. The cow/horse pair, for example, was picture-picture, picture-word, word-picture, and word-word for different subjects. Each subject was exposed to "cow" and "horse" only once.

Randomly ordered target lists contained 34 items while each cue list contained 28 items [Only 28 categories met all of our pretest criteria. Desiring to compare recall to the similar Costley and Brucks study (1990), we included 34 items in the target list so that recall would be from the same sized set as in their study.], half pictures and half words, 7 items in each treatment condition. The first and last items were not treatment items. They were used to absorb primacy and recency effects and to illustrate category membership in the recall instructions. Video items remained on the screen-for about 5 seconds, audio items for as long as it took to speak the-word. Five seconds between each item resulted in an even-paced presentation

Recall was the dependent variable. If recall for the video-video pairs and for the audio-audio 0 pairs exceeded recall for the mixed target-cue pairs (video-audio pairs and audio-video pairs) the cue modality hypothesis would be supported. We expected some interaction between cue modality effects on recall and picture superiority effects on 0 recall. Referring to the cells in Figure A, we f expected recall performance to be ordered: 1 - 2 - 4 (or 4 - 2) - 3. Video recall given a video cue was expected to be better than recall in any of the other conditions. Audio recall given a video cue was expected to be worse-than recall in any of the other conditions.

One hundred and one students used the 5-second intervals to "learn and remember" each item in the target list. About an hour later the students viewed the cue items. They wrote down the name of X the item from the first tape that was most similar to 0 each item on the second tape. There were about 5 seconds in which to respond between items.

Analysis and Results

Multivariate, repeated measures analysis of variance was performed on the data. The multivariate procedure is recommended by LaTour and Miniard (1983) because it produces accurate estimates of Type I error (the probability of rejecting a true null hypothesis). The dependent variables in the model were recall totals (ranging from 0-7) for each of the within subject treatments.

TABLE 1

RECALL MEANS IN EACH WITHIN SUBJECT TREATMENT CONDITION

Respondents recalled between 0 and 6 of the 7 possible items in each of the video target conditions and between 0 and 5 items in each of the audio target conditions. Table 1 displays mean recall scores for each treatment condition.

Not only are the recall means in the predicted order, but both the cue modality and the picture superiority effects are significant. Figure B graphically depicts the results. The multivariate test criteria (Wilks' criterion, Pillai's trace, Hotelling-Lawley trace, and Roy's maximum root criterion) indicated that modality match between target and cue significantly affected recall (F(197) = 16.08, p =.0001). The slope of the lines in the graph represent the cue modality effect. We observed a picture superiority effect (F(197) = 102.43, p =.0001) clearly indicated by the distance between the lines in the figure. Video representations in the target list were recalled better than audio regardless of the cue.

The interaction (target modality x modality match), however, is also significant (F(1 97) = 5.32, p = .0232). One can see in the graph that the cue modality effect is "steeper" for video material than for audio material (indicated by the slope of the lines). Indeed audio-audio recall was not significantly different from audio-video recall (p = .2089).

DISCUSSION

Since we observed a "cue modality effect" in one modality but not in the other, we can not claim support for the processing modality theory. Figure C may help us speculate about explanations. Recallability of a target item is a function of its accessibility via memory links. Regardless of its modality, the cue item provides the link to the category label (B). Links A and B were pretested and held constant. Differences in recallability, therefore, are functions of link C. We hypothesized that link C would be strengthened when the target item was pictured (picture superiority) and when the target and the cue were of the same modality. Support required that both modalities exhibit modality-match enhanced recall.

What we observed was a synergy between pictorial stimuli, i.e., enhanced recallability when the target item was pictured and further enhancement when the cue was also pictured. One explanation is that category membership (links A and B) provides a sufficient organizational basis for recall- and that pictorial cues carry the same kind of recall enhancing traits as do pictorial targets (picture superiority of retrieval and encoding). The characteristic (saliency, amount of information) that causes the effect is unclear.

It is interesting to compare these results to a nearly identical study that used print stimuli (Costley and Brucks 1990). The earlier study did not find a statistically significant cue modality effect. Mean recall in each treatment condition was almost the same between studies. The only difference is that recall was better in our video-video condition than in the former study's picture-picture condition. This suggests that some quality of video presentation (e.g., spatial information or color) enhances recall compared to line drawings. It further suggests that audio presentations do not provide a similar benefit over printed words.

If our results are due to picture saliency (retrieval cue picture superiority) then we should have observed enhanced recall for words given a video cue. We did not. If, however, both retrieval cue picture superiority and our hypothesized target cue modality match acted to enhance recall then we would get results similar to those observed. While an audio cue may have enhanced audio recall, if a video cue also enhanced audio recall, then there might be no statistically significant difference between audio recall conditions. One must use a third modality cue and a no cue control group in order to test this.

Alternative explanations are that video provides better category information than audio (link B) or that video processing is holistic and faster than audio processing. The audio cue may have indeed triggered associated processing modes but video processing may be faster than audio processing. Our recall time constraint may have capped any audio-audio enhancement over audio video recall. It may be worth replicating this study with longer exposures.

Finally, this study was conducted under high involvement conditions. Respondents' attention to the stimuli was high because they were specifically asked to try to learn and remember the list items. Cue modality may influence recall even more when there are fewer other influences. Lower involvement conditions that are typically associated with television viewing may be riper for cue modality influence and remain untested.

FIGURE B

RECALL RESULTS AS A FUNCTION OF TARGET AND CUE MODALITIES

FIGURE C

CONCEPTUAL MODEL

Implications

Current results are not directly generalizable to consumer behavior applications. However, these results build theoretical support for an ongoing stream of research that has implications for planning and evaluating marketing communications. Our cue modality effects suggest developing promotional campaigns as cohesive packages. Greater effectiveness may be achieved when both the original processing and cuing modalities are considered. For example, when point-of-purchase displays are used to cue prior television ads (similar to Keller 1987), pictures should elicit better video recall than plain text or audio.

Cue modality may also be relevant for evaluating ad effectiveness and misleading potential. If retrieval cue modality indeed biases recall, then cue modality must be considered in the evaluation and control functions. An ad may be memorable in some contexts but not in others. Similarly, failure to recall key points may ultimately be misleading.

SUMMARY

In this article, we presented theoretical justification for a cue modality hypothesis. A preliminary study did not confirm it, but lends it merit and we recommend additional research. The hypothesis asserts that the sensory mode of retrieval cues triggers recall for same-modality information. If true, then sounds in the retrieval environment (i.e., audio cues) will trigger recall for audio ads more so than for pictorial ads and vice versa. We observed that video cues enhanced recall for video targets compared to audio cues. We failed to observe a similar modality match effect for audio targets. The findings provide the theoretical foundation for continued research on marketing stimuli and consumer behavior.

REFERENCES

Anderson, John R. (1976), Language, Memory, and Thought, Hillsdale, NJ: Erlbaum.

Anderson, John R. (1978), "Arguments Concerning Representations for Mental Imagery," Psychological Review, 85, 249-277.

Anderson, Richard C. and James W. Pichert (1978), "Recall of Previously Unrecallable Information Following a Shift in Perspective," Journal of Verbal Learning and Verbal Behavior, 17, 1-12.

Battig, William F. and William E. Montague (1969), "Category Norms for Verbal Items in 56 Categories: A Replication and Extension of the Connecticut Category Norms," Journal of Experimental Psychology Monograph, 80 (3), Part 2, 146.

Bettman, James R. (1986), "Consumer Psychology," Annual Review of Psychology, 37, 257-289.

Brooks, Lee R. (1968), "Spatial and Verbal Components of the Act of Recall," Canadian Journal of Psychology, 22, 349-368.

Childers, Terry L. and Michael J. Houston (1984), "Conditions for a Picture-Superiority Effect on Consumer Memory," Journal of Consumer Research, 11 (September), 643-654.

Childers, Terry L., Michael J. Houston, and Susan E. Heckler (1985), "Measurement of Individual Differences in Visual Versus Verbal Information Processing," Journal of Consumer Research, 12 (September), 125-134.

Costley, Carolyn L. and Merrie Brucks (1990), "Perceptual Processing Modes and Advertising Effectiveness," working paper, Marketing Department, Texas A&M University, College Station, TX 77843-4112.

Edell, Julie A. and Richard Staelin (1983), 'The Information Processing of Pictures in Print Advertisements," Journal of Consumer Research, 10, 45-61.

Glenberg, Arthur M. (1984), "A Retrieval Account of the Long-Term Modality Effect," Journal of Experimental Psychology: Learning, Memory and Cognition, 10 (1), 16-31.

Godden, D.R. and A.D. Baddeley (1975), "Context-Dependent Memory in Two Natural Environments: On Land and Underwater," British Journal of Psychology, 66, 325-332.

Hasher, Lynn and Rose T. Zacks (1979), "Automatic and Effortful Processes in Memory," Journal of Experimental Psychology: General, 108 (3), 356-388.

Houston, Michael J., Terry L. Childers, and Susan E. Heckler (1987), "Picture-Word Consistency and the Elaborative Processing of Advertisements," Journal of Marketing Research, 24, 359-369.

Keller, Kevin Lane (1987), "Memory Factors in Advertising: The Effect of Advertising Retrieval Cues on Brand Evaluations," Journal of Consumer Research,14 (December), 316-333.

Lachman, Roy, Janet L. Lachman, and Earl C. Butterfield (1979), Cognitive Psychology and Information Processing, Hillsdale, NJ: Erlbaum.

LaTour, Stephen A. and Paul W. Miniard (1983), 'The Misuse of Repeated Measures Analysis in Marketing Research," Journal of Marketing Research, 20 (February), 45-57.

Lehman, Elyse Brauch (1982), "Memory for Modality: Evidence for an Automatic Process," Memory & Cognition, 10, 554-564.

Lehman, Elyse Brauch and Jeanne C. Mellinger (1984), "Effects of Aging on Memory for Presentation Modality," Developmental Psychology, 20 (6), 1210-1217.

Lehman, Elyse Brauch, James W. Mikesell, and Suzanne C. Doherty (1985), "Long-Term Retention of Information about Presentation Modality by Children and Adults," Memory & Cognition, 13, 21-28.

Lynch, John G. and Thomas K. Srull (1982), "Memory and Attentional Factors in Consumer Choice: Concepts and Research Methods," Journal of Consumer Research, 9, 13-37.

Murdock, Bennet B. and Keith D. Walker (1969), "Modality Effects in Free Recall," Journal of Verbal Learning and Verbal Behavior, 8, 665-676.

Nilsson, Lars-Goran (1973), "Organization by Modality in Short-Term Memory," Journal of Experimental Psychology, 100, 246-253.

Paivio, Allan (1979), Imagery and Verbal Processes, Hillsdale, NJ: Erlbaum.

Rundus, Dewey (1973), "Negative Effects of Using List Items as Recall Cues," Journal of Verbal Learning and Verbal Behavior, 12, 43-50.

Segal, Sydney Joelson and Vincent Fusella (1970), "Influence of Imaged Pictures and Sounds on Detection of Visual and Auditory Signals," Journal of Experimental Psychology, 83, 458-464.

Slamecka, Norman J. (1968), "An Examination of Trace Storage in Free Recall," Journal of Experimental Psychology, 76, 504-513.

Slamecka, Norman J. (1969), "Testing for Associative Storage in Multitrial Free Recall," Journal of Experimental Psychology, 81, 557-560.

Tulving, Endel (1972), "Episodic and Semantic Memory" in Endel Tulving and Wayne Donaldson (eds.) Organization of Memory, New York: Academic Press, 448.

Tulving, Endel (1983), "Elements of Episodic Memory, New York: Oxford University Press.

Tulving, Endel and Zena Pearlstone (1966), "Availability Versus Accessibility of Information in Memory for Words," Journal of Verbal Learning and Verbal Behavior, 5, 381-391.

Tulving, Endel and Joseph Psotka (1971), "Retroactive Inhibition in Free Recall: Inaccessibility of Information Available in the Memory Store," Journal of Experimental Psychology, 87 (1), 108.

Tulving, Endel and Donald M. Thomson (1973), "Encoding Specificity and Retrieval Processes in Episodic Memory," Psychological Review, 80, 352-373.

----------------------------------------