Finding the diamonds in the rough in the “blogosphere”

Please note that this post is from 2009. Evaluate with care and in light of later events.

I’ve been giving a lot of thought over the weekend to the problem of finding good content buried amidst all the noise on the Internet, especially when it comes to blog articles from lesser-known sources. (This is true for readers looking for quality content, but it’s also true for authors seeking readers.) Although I’ve been struggling with this problem for a while, this recent post from Louis Gray really resonated with me:

In essence, the incentives, for the most part, do not tilt in favor of writing unique stories or doing the required research necessary to get a full story, to get quotes from a source, or find data points that back up analysis.

There are going to be pockets of the Web that harbor original ideas, a focus on quality and data, and there are going to be other places where copying, scraping, and shortcuts are going to rule the day.

via louisgray.com: Growing Grumblings on Tech News Don’t Address Incentives.

I run into this problem when, as a researcher, I seek useful analysis on topics I am investigating. Google tends to turn up sources with high “PageRank,” which reflects a certain “wisdom of the crowds” when it comes to breaking stories, but stumbles when looking for more specific content, or insightful analysis. I often have to wade through what Mike Arrington of TechCrunch calls “fast-food content” from content mills of various sorts (including, often, well-known blogs).

As a result, I turn to sources I know (like SSRN or blogs I already know who are not quite on target) and spend inordinate amounts of time painstakingly, manually finding decent sources of information (which usually means big blogs). Usually I miss the odd, insightful posts from “little guy” blogs.

Note that turning up top blogs in niche is not too hard (Alltop is a good place to start). Many of them are quite good (the crowd is good for something), especially for getting the pulse of a niche.

But what happens when you want something beyond the latest and greatest happenings? What about all those little blogs out there that occasionally produce brilliant content, but are never going to compete (and probably aren’t trying to) with TechCrunch (or even louisgray.com)?

Regular journalism (Salon, for example) can be good for this (especially in politics), although the lack of links to sources makes it difficult to use such pieces as launching points for more research. (More academic articles are better for this, but can be long, complex, and specialized.)

Some ideas, thoughts, and sources:

• I encourage well-known bloggers and big blogs to link to lesser-known, but insightful posts on blogs — even if the small guy is not an up-and-coming, next-big-thing discovery. (It would be great if journalists could do the same, but since traditional news outlets seem unable to move forward to embrace the Web, I don’t hold out much hope for this.)
• Digg and Reddit can sometimes turn up good content, but this is another case of crowdsourcing not always producing good results. Content that rises tends to fall into certain patterns and appeal to a certain demographic or mindset. Good, but uninteresting-to-the-masses articles tend to get classified with spam.
• StumbleUpon can be better than Reddit or Digg at allowing niche content, but because it tends to feel random, it’s use for specific research is limited — I’ve never had much luck searching it for useful content.
• Google (and its competitors) should consider finding a way to “de-rank” content mills in some fashion. (Yes, I know they provide a good deal of revenue via advertising, so perhaps this will never happen.) Meanwhile, Google Blog Search and Google Scholar are useful, if imperfect, tools.
• Web tools like AllTop and PostRank are useful tools, although both tend to highlight top blogs in a niche, not top posts (which is still very useful).
• “Bog rolls” are still useful sources, even if their use is dying off, but again tend to turn up niche blogs, not specific content.
• Academic sources like SSRN, or PubMed, are useful for certain kinds of specific research, but they can be too specialized and too in-depth.

Any other ideas? I’m still looking.