Finding the diamonds in the rough in the "blogosphere"

Image via Wikipedia

I’ve been giv­ing a lot of thought over the week­end to the prob­lem of find­ing good con­tent buried amidst all the noise on the Internet, espe­cially when it comes to blog arti­cles from lesser-known sources. (This is true for read­ers look­ing for qual­ity con­tent, but it’s also true for authors seek­ing read­ers.) Although I’ve been strug­gling with this prob­lem for a while, this recent post from Louis Gray really res­onated with me:

In essence, the incen­tives, for the most part, do not tilt in favor of writ­ing unique sto­ries or doing the required research nec­es­sary to get a full story, to get quotes from a source, or find data points that back up analysis.

There are going to be pock­ets of the Web that har­bor orig­i­nal ideas, a focus on qual­ity and data, and there are going to be other places where copy­ing, scrap­ing, and short­cuts are going to rule the day.

via louisgray.com: Growing Grumblings on Tech News Don’t Address Incentives.

I run into this prob­lem when, as a researcher, I seek use­ful analy­sis on top­ics I am inves­ti­gat­ing. Google tends to turn up sources with high “PageRank,” which reflects a cer­tain “wis­dom of the crowds” when it comes to break­ing sto­ries, but stum­bles when look­ing for more spe­cific con­tent, or insight­ful analy­sis. I often have to wade through what Mike Arrington of TechCrunch calls “fast-food con­tent” from con­tent mills of var­i­ous sorts (includ­ing, often, well-known blogs).

As a result, I turn to sources I know (like SSRN or blogs I already know who are not quite on tar­get) and spend inor­di­nate amounts of time painstak­ingly, man­u­ally find­ing decent sources of infor­ma­tion (which usu­ally means big blogs). Usually I miss the odd, insight­ful posts from “lit­tle guy” blogs.

Note that turn­ing up top blogs in niche is not too hard (Alltop is a good place to start). Many of them are quite good (the crowd is good for some­thing), espe­cially for get­ting the pulse of a niche.

But what hap­pens when you want some­thing beyond the lat­est and great­est hap­pen­ings? What about all those lit­tle blogs out there that occa­sion­ally pro­duce bril­liant con­tent, but are never going to com­pete (and prob­a­bly aren’t try­ing to) with TechCrunch (or even louisgray.com)?

Regular jour­nal­ism (Salon, for exam­ple) can be good for this (espe­cially in pol­i­tics), although the lack of links to sources makes it dif­fi­cult to use such pieces as launch­ing points for more research. (More aca­d­e­mic arti­cles are bet­ter for this, but can be long, com­plex, and specialized.)

Some ideas, thoughts, and sources:

• I encour­age well-known blog­gers and big blogs to link to lesser-known, but insight­ful posts on blogs — even if the small guy is not an up-and-coming, next-big-thing dis­cov­ery. (It would be great if jour­nal­ists could do the same, but since tra­di­tional news out­lets seem unable to move for­ward to embrace the Web, I don’t hold out much hope for this.)
• Digg and Reddit can some­times turn up good con­tent, but this is another case of crowd­sourc­ing not always pro­duc­ing good results. Content that rises tends to fall into cer­tain pat­terns and appeal to a cer­tain demo­graphic or mind­set. Good, but uninteresting-to-the-masses arti­cles tend to get clas­si­fied with spam.
• StumbleUpon can be bet­ter than Reddit or Digg at allow­ing niche con­tent, but because it tends to feel ran­dom, it’s use for spe­cific research is lim­ited — I’ve never had much luck search­ing it for use­ful content.
• Google (and its com­peti­tors) should con­sider find­ing a way to “de-rank” con­tent mills in some fash­ion. (Yes, I know they pro­vide a good deal of rev­enue via adver­tis­ing, so per­haps this will never hap­pen.) Meanwhile, Google Blog Search and Google Scholar are use­ful, if imper­fect, tools.
• Web tools like AllTop and PostRank are use­ful tools, although both tend to high­light top blogs in a niche, not top posts (which is still very useful).
• “Bog rolls” are still use­ful sources, even if their use is dying off, but again tend to turn up niche blogs, not spe­cific content.
• Academic sources like SSRN, or PubMed, are use­ful for cer­tain kinds of spe­cific research, but they can be too spe­cial­ized and too in-depth.

Any other ideas? I’m still looking.