Channing Bete Company(R) Types of Products and Services

Adult Effectiveness Study

Methodology

  • In Phase II of the study, various titles in Classic illustration, realistic illustration, and story formats were tested with adults reading at or above the seventh-grade level.
  • Quantitative data were collected through testing done before and after subjects had read a given booklet.
  • Researchers pre-tested and post-tested each participant to precisely measure an increase in understanding of the information in each booklet and intent to act as a result of that information.
  • Qualitative data were collected from active participants in structured discussion groups, as opposed to using questionnaires alone.
  • The study was national in scope, encompassing five regions: the Northeast, the Southeast, the Midwest, the Southwest, and the Northwest.
  • Over 1,000 adults took part in the study. Represented were African Americans, Latinos, Caucasians, Asians, and Native Americans, each making up approximately one-fifth of the total group.
  • The study was conducted using a representative sampling of booklets divided equally among four major booklet formats: Classic illustration, realistic illustration, story format, and low-literacy.
  • In Phase I of the study, various titles in low-literacy and story formats were tested with adults reading below the seventh-grade level.

Adult Study Results

Quantitative Results

 

Phase I
Low-literacy and story formats tested with adults who read below the seventh-grade level.

Phase II
Classic and realistic illustration, and story formats tested with adults who read above the seventh-grade level.

Transfer of knowledge
Number of participants who said they learned new information from the booklets

84%

82%

Reading ease
Number of participants who found the booklets easy to read

96%

97%

Personal relevance
Number of participants who found the booklets interesting to read

96%

95%

Intent to act
Among participants expressing little or no intention to take a specific action prior to reading the booklet, the intent to act after reading the booklet

increased by
38%

increased by
56%

Qualitative Results

Discussion groups provided confirmation of statistical results and added personal context to the quantitative data:

"I am interested in this booklet because the story about two families could be about me."

"The story made me get into it, understand the person's situation. It forced me to think maybe I should
go and get this done."

"I knew some about lead, but I have more information now. It made me think I should have my
children checked."

"I'd recommend all of them. It's important information."

"The pictures got right to the point and explained what they were talking about."

"The thoughts that went through my mind are things that I'm going through with stress and
what I need to do about that."

Adult Study Executive Summary

Download PDF pdf icon

Executive Summary

Report on the National Product Effectiveness Study with Adult Readers

Study conducted by John Snow, Inc., & World Education, Inc.

Channing Bete Company (R) www.channing-bete.com


One Community Place
South Deerfield, MA 01373-7328
Phone: 1-800-477-4776
Fax: 1-800-329-2939

©1998 Channing Bete Company, Inc. All rights reserved.
No part of this publication may be reproduced without written permission of the publisher.


Report on the National Product Effectiveness Study with Adult Readers

Abstract Objectives

This independent study assessed the effectiveness of Channing Bete Company (CBC) educational materials in terms of:

  • knowledge transfer
  • intention to act/attitude change
  • reading ease
  • personal relevance.

Methods

The study analyzed quantitative and qualitative data collected in person from a national sample of over 1,000 adults. Study participants were divided into two groups depending on reading ability in order to measure effectiveness of CBC educational materials. Each group was asked to evaluate materials appropriate to the reading level of its members. Within each group, significant representation from 5 major racial subgroups was ensured. (Racial subgroups were: Caucasian; African American; Hispanic; Asian/Pacific Islander; Native American.)

Results

Quantitative and qualitative results showed that CBC materials are effective in terms of transfer of information and influence on intention to act. Study participants also found CBC educational materials to be easy to read and relevant to their own lives. Results were consistent across all formats tested: Materials aimed at lower-level readers were found effective by lower-level readers; and products aimed at average readers were found effective with average readers.

Conclusions

CBC publications in all tested formats are an effective means of communicating information to adults. The publications effectively influence readers' intention to take actions suggested by the publications. The publications are easily read by appropriate audiences, and the readers regard the content as relevant and useful to their lives.

Return to top

Introduction

Channing Bete Company, Inc., is the publisher of Scriptographic(R) booklets, i.e., 8- and 16-page booklets designed to communicate information through a combination of simple text and supporting illustrations. CBC has over 50 years of experience providing health, safety and other information on hundreds of topics. Materials are published in a variety of styles and formats; CBC also has a line of educational materials aimed specifically at readers with very limited reading skills.

Many of CBC's customers in the public and private sectors are increasingly concerned about measuring and understanding the effectiveness of their communication efforts. The reasons for this concern vary. For example, among government customers, it often grows out of the need for support in making grant applications; or, the need may arise out of "performance budgeting" requirements in which the level of funding for a program is tied to measurable results. In the medical field, the need to measure effectiveness is related to efforts to improve patient satisfaction, compliance, and outcomes.

In addition, many programs that might utilize CBC materials must first go through a process of "pretesting" materials prior to distribution, to ensure that the materials are understood by and appropriate for target audiences.

In late 1996-early 1997, John Snow, Inc., an internationally known independent public health research firm, and the Literacy Division of World Education, Inc., a nonprofit organization committed to furthering adult literacy, conducted an extensive national quantitative and qualitative evaluation of CBC health-education booklets. These organizations brought broad experience and outstanding qualifications to this study. For example, JSI has partnered with organizations such as the U.S. Department of Health and Human Services' Office of Population Affairs, and Health Resources and Services Administration; WEI has sponsored a national conference on health and literacy and is in partnership with Harvard University in the National Center for Adult Learning and Literacy. The JSI/WEI team thus approached the study with wide expertise in the fields of public health and literacy.

This study sought to obtain data about the effectiveness of the booklets based on the following criteria:

  • knowledge transfer -- how much the reader learns from the booklets
  • intention to act -- how the booklets affect the reader's attitudes or intention to act in a positive (i.e., healthier or safer) way
  • reading ease -- how easy the booklets are to read and understand
  • personal relevance -- how relevant the reader perceives the information to be to his or her life.

Return to top

Methods

Data collection for the study was conducted between December 1996 and March 1997. It consisted of two phases: In Phase I, 501 "lower-literacy" adults evaluated the effectiveness of the Scriptographic formats aimed at lower-literacy adults. Participation for Phase I was limited to adults with a reading ability below the 7th-grade level.

In Phase II, 591 "average-reader" adults evaluated the Classic and realistic illustrated formats. "Average reader" was defined as reading at or above the 7th-grade level.

The reading levels of prospective participants were determined by their performance on standard literacy tests, such as the Test of Adult Basic Education (TABE). Where such test results were not available, the prospective participant's reading level was estimated at two grade levels below last grade completed in a U.S. school.

In both phases, "adult" was defined as age 18 and over. Seventy-five percent of the study participants were between the ages of 18 and 44; 25 percent were 45 or older.

Participants were also selected in order to achieve wide representation from major racial subgroups: African American, Asian/Pacific Islander, Hispanic, Native American, and Caucasian. Of the 1,092 participants in both phases, there were at least 170 from each of these subgroups.

In Phase I, 66 percent of the participants had an annual household income of less than $20,000, and 82 percent had an annual household income of less than $30,000.

In Phase II, 60 percent had an annual household income under $30,000; 82 percent were under $50,000.

The study was conducted using a representative sampling of booklets, divided equally among the 4 major booklet formats. Because the use of instructional design and reading level is consistent within each format, the JSI/WEI team concluded that study results for each format sample could be extrapolated across all titles in that format.

Data collection consisted of two major components: quantitative data collection and qualitative data collection about selected CBC booklets. Information was gathered from data-collection groups consisting of 8 to 14 participants. Ninety-one of these groups met in numerous locations representing all geographical regions of the continental United States. Over the course of a 2.5-hour session, each lower-literacy group evaluated two booklets, while each average-reader group evaluated three booklets.

Quantitative data collection for each of the Scriptographic booklets under study was achieved by the following method:

1. Participants filled out a pre-survey for each booklet, which yielded:

  • demographic data: gender, age, income, race, native language, level of acculturation
  • information about the participants' existing knowledge of the topic covered in the test booklet
  • intention to act/attitude: the participants' current behaviors and attitudes regarding the topic of the booklet being evaluated.

2. Participants read the selected booklets.

3. Participants completed a post-survey for each booklet, which revealed:

  • level of knowledge acquired about the topic
  • change in intention to act/attitude toward the topic
  • participants' impressions of the booklet's relevance to their lives.

Quantitative data was analyzed using SPSS (Statistical Package for Social Studies) software.

Qualitative information was gathered via focused discussion groups that followed the quantitative data collection. The groups were designed to allow participants to share verbally their impressions and opinions about the booklets they had read. Trained facilitators elicited responses on specific issues and product attributes. The discussions were tape-recorded, and the qualitative data was then analyzed using Q.S.R. Non-Numerical Unstructured Data Indexing, Searching and Theorizing software.

Results

Overall, the study proved that Scriptographic booklets are an effective means of communicating information and influencing attitudes and intention to change behavior. The study also found that readers find Scriptographic booklets to be easy to read and relevant to their lives.

Key findings of the quantitative data collection and analysis are shown in Table 1. (All figures are averages of results generated by all the groups in each phase.)

Return to top

Table 1: Key findings of quantitative data collection and analysis


 

Phase I
(lower-literacy adults)

Phase II
(average-reader adults)

Knowledge Transfer

84% reported learning new information from the booklets.
Post-survey measures showed that participants' knowledge of the topics covered in the booklets increased by 23.5%.

82% reported learning new information from the booklets.
Post-survey measures showed that participants' knowledge of the topics covered in the booklets increased by 31.5%.

Intention to Act

Among participants expressing little or no intention to take a specific action prior to reading a booklet, there was a 38% increase in reported intention to act after reading the booklets.
Gain was most significant among participants reporting very little or no prior knowledge of the topics.

Among participants expressing little or no intention to take a specific action prior to reading a booklet, there was a 56% increase in reported intention to act after reading the booklets.
Gain was most significant among participants reporting very little or no prior knowledge of the topics.

Ease of Reading

96% found the booklets easy to read.
85% felt illustrations aided understanding.
77.5% found that the amount of information in the booklets was "just right."

97% found the booklets easy to read.
80% felt illustrations aided understanding.
82% found that the amount of information in the booklets was "just right."

Personal Relevance

95.5% found the booklets interesting to read.
89.5% found the booklets helpful in explaining the topics.

95% found the booklets interesting to read.
92% found the booklets helpful in explaining the topics.

The qualitative portion of the study provided participants an opportunity to expand on their responses in the pre- and post-surveys. Results of this qualitative data collection confirm the information gathered during quantitative data collection: Scriptographic booklets are effective at communicating information and influencing intention to act. The booklets are also easy to read, interesting, and relevant to readers.

Overall, and throughout the qualitative data collection, participants were overwhelmingly positive about the educational materials under study. It was common for participants to ask if there were extra copies of the booklets available for their personal use. As one participant noted:

"After I read them, I thought: 'I hope I can take these booklets home.'" (Phase II participant, race unknown)

Other remarks noted during the sessions reflect the effectiveness of the booklets in terms of transfer of knowledge:

"I learned a lot of good information from these booklets." (Phase I participant, race unknown)

"I thought I knew more. Reading this book helped me understand better." (Phase II participant, Hispanic)

Participants' comments also reflected the impact of the booklets on the readers' intention to act:

"For me maybe next week I will bring my children to the clinic to see what they need." (Phase I participant, Asian)

"I'm going to talk to someone that's close to me about this." (Phase I participant, African American)

Participants' comments also reflected the opinion demonstrated in the pre- and post-surveys that the booklets are relevant to the readers' lives:

"I am interested in this booklet because the story about two families could be about me." (Phase I participant, Asian)

"It reminded me of my family with my grandmother. We're going through the same thing, and that hit home." (Phase II participant, Native American)

Finally, comments also supported the fact that Scriptographic booklets are easy to read:

"They were easy to read and didn't talk down to you." (Phase I participant, Hispanic)

"Even though I was familiar with a lot of the information, I think that the way that it was laid out and the comprehensiveness was good. It was very user-friendly." (Phase II participant, Caucasian)

The qualitative portion of the study also provided an opportunity to evaluate readers' responses to various formats. While the study did not specifically seek to compare formats, it did yield a variety of opinions on preferred presentation of information. Some readers responded more positively to the fact-based presentation of the Classic and realistic illustrated formats:

"I'd rather have just the facts. It's easier to read." (Phase I participant, African American)

"I like the facts. Bring me the facts." (Phase II participant, Native American)

Other readers favored the narrative approach of the story format:

"This is a story and it makes it easier to understand." (Phase I participant, race unknown)

"I liked how they gave you a little story. It made it personal and more interesting." (Phase II participant, race unknown)

These findings suggest that CBC's different formats offer important options for more effectively communicating with readers who differ in their preferred styles of acquiring information.

Return to top

Conclusions

The study conducted by John Snow, Inc., and World Education, Inc., provides an independent statistical and qualitative analysis of the overall effectiveness of CBC educational materials. The study was structured to isolate the effects of the booklets apart from other elements of the comprehensive educational programs in which they are typically used: specific environmental factors, larger program objectives and components, and so on.

The study demonstrates that CBC materials are a highly effective means of transferring knowledge and influencing readers' intention to act in positive ways. The study also confirms that the materials are regarded as easy to read and relevant to the lives of readers.

The general findings of the study are consistent across all the booklet formats studied. Statistical data revealed similar results for all racial subgroups and across all other demographic variables that were analyzed.

The study did reveal a broad range of results in the participants' intention to act in positive ways. Generally speaking, the amount of increase reported in the intention to act was proportional to the amount of knowledge reported in pre-surveys: Those readers who had the least prior knowledge of a given topic reported the greatest increase in their intention to act; those with the greatest prior knowledge had the smallest increase.

These findings are consistent with the stages of change behavioral model, which holds that people pass through several distinct stages on their way toward behavior change. Different types and levels of intervention are required at each of these stages. As people progress along the continuum of change, it may become more difficult to measure the smaller, incremental gains in knowledge and changes in attitudes that accumulate to move individuals into new stages of change. In the later stages, repeated and multifaceted exposure to information and motivation are required to overcome resistance and continue progress toward the ultimate goal of adoption of a new behavior. Thus, the impact of a booklet will be greatest on those who possess the lowest preexisting level of information and motivation.

The Channing Bete Company, Inc., recognizes the importance of providing effective educational materials to its customers and readers. The study discussed above is part of CBC's ongoing efforts to test and ensure the effectiveness of its educational materials.

Return to top