By Scott Clark – Web Marketing Consultant – 7/05

Ever feel like there is too much information to deal with? Well, it stands to reason. Extrapolate a 2003 study from UC Berkeley and you get a total of 6,917,529,027,641,081,856 bytes of data produced annually in all forms on Earth. That’s almost 7 exabytes – or around a gigabyte for each person on earth every year!

When you look at it this way, it’s not whoever has the most data who wins in this information economy, but whoever can make sense out of their own domain of knowledge in time to move through windows of opportunity or avoid disruptive surprises.

If it were possible to get your arms around just the World Wide Web (800 terabytes or 80 times the size of all print collections in the US Library of Congress,) you might be able to find key indicators that can have powerful positive influence on your day-to-day operations. There are industry events, legal decisions, and regulatory changes dispersed through the web like needles in a gigantic moving haystack. The bookmarks of today are the page-not-founds of tomorrow.

If you spent all day on the web you might get what you need – for today. Tomorrow morning, your answers may be obsolete, and you’re back to square one. But there is hope, thanks to strategic use of Internet agents.

I define web agents as saved and scheduled queries which compile, sort and send results to you without intervention. They must only be set up once and are consistent in the information they send. They can be your eyes and ears on the web and blogging community, and they work for free.

The most basic type of agent is an email alert. These are old-school in that their delivery depends on it making it to your in-box and through your SPAM filters. Email alerts are available from many sources, such as Reuters and CNN, or even eBay. They are incredibly easy to set up, but can be tricky to stop or change. I find them of limited use, but Google News Alerts does the best job of any I’ve tried.

If you know in advance the URL of information you want, you may want to use a “page monitor.” These “visit” a site on your behalf and send you an email when the contents change. This is very useful for watching your competitions’ websites or press releases. For less predictably-located information you can use an “inverted search engine” such as googlealerts.com or feedster.com. Still, these make their way to you in the mass of other email you read, making them less useful.

Email notification methods are rapidly being overtaken by RSS, which stands for either “Really Simple Syndication” or “Rich Site Summary.” It offers many advantages over email alerts such as privacy and subscription control. To find RSS feeds (sources) you can use a search engine like feedster which has indexed over 150 million RSS items, or look for the orange RSS icons found on many web pages. Feedster in particular, will send alerts of new posts by author, subject, and more, and it will even combine your feed search results into a new “feed-feed” to scan quickly.

Feeds are usually viewed using software called an aggregator. These come in all flavors, including plug-ins for Microsoft Outlook and Eudora, as well as “live bookmarks” in the Firefox browser. Microsoft announced last month that RSS support would be built into the next version of Windows and Macintosh users have enjoyed it for a while.

Subscribing is done via simple click or cut and paste. You can assign expirations to keep your screen clear of old news. In some ways, it’s like a Tivo for business intelligence.

My current favorite combined system is Klipfolio by Serence, a free tool for assembling feeds to meet your own needs. There is a page monitor, feed monitor, and automated Google query system for web or news searches. You can browse an incredible “Klip farm” with thousands of data sources of all types. I have around forty precision feeds, page monitors and news searches, and find it quite manageable for a morning scan, and have never been so well informed.

A warning: Once you start using RSS or Klipfolio you will not be able to live without them. The first time you are able to make a competitive move based on a feed summary sent to your cell phone, you’ll be hooked. You’ll want to install it on your entire staff’s computers, and design queries for each person’s roles. “How’d you know that?” is something I love to hear, especially from a potential customer. “I try stay up to date” is my polite answer. Of course it’s possible to react too quickly or too often, so sometimes you can use multiple feeds to triangulate on the truth and see what’s really going on.

Albert Einstein once said “If I have seen farther than others, it is because I was standing on the shoulders of giants.” I hope that some of these ideas will help you see a little farther ahead for your own business.