Challenges, problems, and triumphs
-- from a manufacturer's perspective.
Analyzing the CHA Attitude & Usage Study
The rationale and the science behind the
by Rob Bostick (June 1, 2009)
(Note: Rob is the CEO of Judikins and a member of the CHA
Board of Directors.)
A task force was formed by the CHA board of directors to look
into the accuracy of the Attitude & Usage Study and
suggest alternatives should we find the study to be flawed. In
college I minored in math with an emphasis on statistics, so I felt
I could be of help to this task force. I also wanted to be sure that
the smaller companies were counted, something that the proposed
alternatives might miss.
The Study is done by IPSOS, which is a very reputable
survey firm. They send out a questionnaire to around 10,000
representative households and ask them if they craft. The
approximately 60% that say they do are then asked to complete
additional surveys. From the data IPSOS gets back from these
households, they extrapolate to the entire population of the U.S.
Overall the task force found that CHA's Attitude & Usage
Study is a good representation of the craft industry in the U.S.
It is not perfect, but for the money, it is about as good as we can
get. Starting last year, to check on the accuracy of the Study,
we ask our survey participants to name the stores where they bought
their crafts. For the first time we separated the categories such as
"chain store" down to individual stores such as Michaels
and Jo-Ann. One of the reasons we know that the industry is truly
$27 billion is that our extrapolated sample of how much was spent in
Michaels stores was very close to their reported sales.
The task force also asked IPSOS to report the confidence limits
for all the various categories. Basically we wanted to know not just
what was the dollar figure spent on a particular category, but how
confident, from the sample data, we were that the extrapolated
values were a good representation of the entire U.S. craft industry.
If you ask 6,000 crafters what hobbies they participate in, the
most popular crafts have more participants and thus give us more
data points. The smaller crafts have fewer participants and thus the
data collected on them are much less reliable. Thus for the crafts
that have fewer participants, we are much less sure that our
estimates are correct. In many cases the error of these estimates
(one standard deviation from the mean) is larger than the estimate
itself. So if the estimate for Quilling is $5 million but the
confidence limit is $20 million, we really can't say that Quilling
is up or down.
I would love to tell everyone that the numbers are gospel, but
with the size of our Study they are only best estimates. For
the big categories like the size of the industry or for the larger
hobbies like scrapbooking, they are very good estimates. For the
smaller crafts, our extrapolations may be off by so much we really
can't say if they are up or down from year to year. Only a larger
study could tell us this and to get the accuracy the members of
these smaller "industries" would like, it may take 10 or
100 times the number of data points (at 10 or 100 times the cost).
We have debated how much CHA should report. I know how much
everyone wants to see the numbers for their craft, but the less
popular the craft, the fewer the data points we get and thus the
less accurate our data is about trends occurring in that craft. In
the future CHA plans to lump many "industries" into one
category and publish the confidence limits for that category, like
only reporting a Paper Crafts category that would include many
crafts. This would give our members much more accurate trend data
for that broad category, but wouldn't address the ups and downs of
any particular craft inside of that category. Our members may balk
at this but it would give them trend data they could trust.