Creative Leisure News - "Vinny Da Vendorr"

Challenges, problems, and triumphs -- from a manufacturer's perspective.

Analyzing the CHA Attitude & Usage Study

The rationale and the science behind the numbers..

by Rob Bostick (June 1, 2009)

(Note: Rob is the CEO of Judikins and a member of the CHA Board of Directors.)

A task force was formed by the CHA board of directors to look into the accuracy of the Attitude & Usage Study and suggest alternatives should we find the study to be flawed. In college I minored in math with an emphasis on statistics, so I felt I could be of help to this task force. I also wanted to be sure that the smaller companies were counted, something that the proposed alternatives might miss.

The Study is done by IPSOS, which is a very reputable survey firm. They send out a questionnaire to around 10,000 representative households and ask them if they craft. The approximately 60% that say they do are then asked to complete additional surveys. From the data IPSOS gets back from these households, they extrapolate to the entire population of the U.S.

Overall the task force found that CHA's Attitude & Usage Study is a good representation of the craft industry in the U.S. It is not perfect, but for the money, it is about as good as we can get. Starting last year, to check on the accuracy of the Study, we ask our survey participants to name the stores where they bought their crafts. For the first time we separated the categories such as "chain store" down to individual stores such as Michaels and Jo-Ann. One of the reasons we know that the industry is truly $27 billion is that our extrapolated sample of how much was spent in Michaels stores was very close to their reported sales.

The task force also asked IPSOS to report the confidence limits for all the various categories. Basically we wanted to know not just what was the dollar figure spent on a particular category, but how confident, from the sample data, we were that the extrapolated values were a good representation of the entire U.S. craft industry.

If you ask 6,000 crafters what hobbies they participate in, the most popular crafts have more participants and thus give us more data points. The smaller crafts have fewer participants and thus the data collected on them are much less reliable. Thus for the crafts that have fewer participants, we are much less sure that our estimates are correct. In many cases the error of these estimates (one standard deviation from the mean) is larger than the estimate itself. So if the estimate for Quilling is $5 million but the confidence limit is $20 million, we really can't say that Quilling is up or down.

I would love to tell everyone that the numbers are gospel, but with the size of our Study they are only best estimates. For the big categories like the size of the industry or for the larger hobbies like scrapbooking, they are very good estimates. For the smaller crafts, our extrapolations may be off by so much we really can't say if they are up or down from year to year. Only a larger study could tell us this and to get the accuracy the members of these smaller "industries" would like, it may take 10 or 100 times the number of data points (at 10 or 100 times the cost).

We have debated how much CHA should report. I know how much everyone wants to see the numbers for their craft, but the less popular the craft, the fewer the data points we get and thus the less accurate our data is about trends occurring in that craft. In the future CHA plans to lump many "industries" into one category and publish the confidence limits for that category, like only reporting a Paper Crafts category that would include many crafts. This would give our members much more accurate trend data for that broad category, but wouldn't address the ups and downs of any particular craft inside of that category. Our members may balk at this but it would give them trend data they could trust.

xxx