SnapStream Blog

Note: Zack’s opinions don’t necessarily match those of SnapStream.
Before I worked for SnapStream, I read computer and car magazines as the gospel. If it was in print, I simply believed it. When a reviewer told me what the best card/software/car was, I had total faith that I had been told the whole story.

But what I didn’t know is that there is a secret, sometimes seedy, underworld to reviews.

My eyes are open now from experience dealing with reviewers, seeing the resulting articles and reading blog posts or reviews that were done without our involvement.

So why the post now? I just read a blog post by Gary Krakow discussing the new gPC that Walmart is hawking for $200. (side note: if there is an English major out there that can confirm “hawk: to peddle” instead of “hock: to pawn” was right in this sentence, it would be appreciated) It’s your standard review, but the last 20% of the article is a rant about (what I presume to be) a missing flash plugin for firefox. This, I’m sure, was a big sticking point for the author. But at the same time those words could have been used more effectively to help me decide if the machine is worth trying myself. The last five paragraphs could have been summed up into “as with other linux machines, the software is still not as user friendly as many Windows users expect or need.” Then, there is space left to actually tell me if the machine’s intended purpose is met by its capacities. For example, is it a competent open office and web surfing machine? Would a low income family benefit more from having it rather than no computer at all? Would a tech geek with limited linux experience be able to use the machine as a second box in the den or office?

I don’t mean to pick on Mr. Krakow specifically and I’m certainly not implying that there was any nefarious scheme behind his review. In fact, I’m sure he’s a geek that I would get along splendidly with. That being said, in amongst the well written reviews in magazines and on blogs, there are also plenty of misinformed or misleading ones as well as a handful that are marred by bias, payola or worse.

On the whole, you can assume that a reviewer or blogger is doing so for the benefit of others. They are trying to tell a useful story that the unwashed masses can digest.

Keep in mind, though, that there are pitfalls which you should watch out for.

• Missing the big picture: This is our case in point. A review should answer the question the article purports it to. If it’s a high level overview, it should tell me what worth the product has at achieving its target. If it’s a review of one feature in detail, it should give me all of the information (say scientific benchmarks) and a summary (for the non-technical) that covers the whole feature. If a cheap pc doesn’t play doom at 5000 frames a second, they shouldn’t harp on it for 17 paragraphs. Look out for any item, praise or criticism, that eats up space at the expense of a full review.
• Important details that aren’t included: a related item to the big picture. If something is missing, what is it? Also, why is it not there? For instance, say the pc does everything else so well that the only thing to write about is how the flash plugin isn’t loaded. If that’s the case, it means the machine is pretty good generally even though the review has a negative tone.
• User Error: If the author says something went wrong, there should be clear proof that they eliminated themselves as the culprit or at least called tech support. This is very true in blog posts, but still true in even the most “trusted” of mainstream reviewers.
• Swag Bias: What did the reviewer get to keep after the review? It’s common for software vendors to supply pre-loaded PCs (usually nice) that show the software in the best light. Did the reviewer get to keep it?
• Fanboi Bias: Does the author have a tendency to like or dislike certain things? Can you find patterns in their writing? A linux user may be disoriented reviewing a windows pc just like a race car driver would feel weird driving an SUV. Look at the writer’s background to understand where they are coming from.
• Writing that sounds like marketing: Does the review read like an advertisement? How much time went into the review? It’s ok to be skeptical if the tone is a little too happy or there aren’t any negatives listed.
• Forgetting that nothing is free: Before upgrading the turbocharger on my old WRX, I read an article that said “this turbo comes on strong at 4500rpm.” Yes, indeed, it did. But below that speed it was like driving a dump truck powered by a hamster wheel. Look for the trade offs when splendid new things come along. HDTV looks great but is 20mb/s. H.264 makes tiny files but requires a ton of cpu. Dodge Vipers are fast but get like 2 miles a gallon. Nothing is free, period. If the author forgets to tell you what the trade off is, remember that you will probably run into it.
• Blogs and “reader reviews” are more negative: Let’s say I bought a new widget and it was the best thing since sliced bread. Am I going to go online and discuss the virtues of it or play with my shiny new toy? Conversely, if I have a broke widget that I can’t play with, what else might I do than sit down, blood boiling, and write a scathing review? Take bad comments with a grain of salt.

In conclusion, I’m surely not saying “don’t believe reviews.” Just understand what went into its production before making a final decision.
Take your time when reading a review and then do the research that wasn’t done. Talk to real people one on one to get an accurate description. Ask questions about your own prospective use cases.

And remember, Skeptical cat is fraught with skepticism.

About the Author:

Zack Price, a long time member of the SnapStream engineering team, is a notoriously cheap geek. He’s recently described how to ditch cable and move to over the air HD. When not at SnapStream, Zack enjoys time at home with his wife Candice, dog Watson, cat Mittens (whose breath smells like cat food) and two iPhones.

This entry was posted on Friday, November 2nd, 2007 at 6:35 pm by zack and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.