It's hard to argue with the proposition that the World Wide Web
is the largest repository of information that has ever existed.
In just over a decade, the Web has moved from a university curiosity
to a fundamental research, marketing and communications vehicle
that impinges upon the everyday life of most people in the developed
world. But there's a catch, of course. As the amount of information
on the Web grows, that information becomes ever harder to keep track
of and use.
This vast amount of freely available information is spread over
billions of Web pages, each with its own independent structure and
format. So how do you find the information you're looking for in
a useful format -- and do it quickly and easily without breaking
the bank?
Search Isn't Enough
Search engines are a big help, but they can do only part of the
work, and they are hard-pressed to keep up with daily changes. For
all the power of Google and its kin, all that search engines can
do is locate information and point to it. They go only two or three
levels deep into a Web site to find information and then return
URLs. They also find and return meta descriptions and meta keywords
embedded in Web pages, but these may well be inaccurate.
Consider that even when you use a search engine to locate data,
you still have to do the following tasks to capture the information
you need:
- Scan the content until you find the information.
- Mark the information (usually by highlighting with a mouse).
- Copy the information.
- Switch to another application (such as a spreadsheet, database
or word processor).
- Paste the information into that application.
A better solution, especially for companies that are aiming to
exploit a broad swath of data about markets or competitors, lies
with Web harvesting tools.
Web harvesting software automatically extracts information from
the Web and picks up where search engines leave off, doing the work
the search engine can't. Extraction tools automate the reading,
copying and pasting necessary to collect information for analysis,
and they have proved useful for pulling together information on
competitors, prices and financial data of all types.
Harvesting Techniques
There are three ways we can extract more useful information from
the Web.
The first technique, Web content harvesting, is concerned directly
with the specific content of documents or their descriptions, such
as HTML files, images or e-mail messages. Since most text documents
are relatively unstructured (at least as far as machine interpretation
is concerned), one common approach is to exploit what's already
known about the general structure of documents and map this to some
data model.
Another approach to Web content harvesting involves trying to
improve on the content searches that tools like search engines perform.
This type of content harvesting goes beyond keyword extraction and
the production of simple statistics relating to words and phrases
in documents.
Another technique, Web structure harvesting, takes advantage of
the fact that Web pages can reveal more information than just their
obvious content. Links from other sources that point to a particular
Web page indicate the popularity of that page, while links within
a Web page that point to other resources may indicate the richness
or variety of topics covered in that page. This is like analyzing
bibliographical citations -- a paper that's often cited in bibliographies
and other papers is usually considered to be important.
The third technique, Web usage harvesting, uses data recorded
by Web servers about user interactions to help understand user behavior
and evaluate the effectiveness of the Web structure.
General access-pattern tracking analyzes Web logs to understand
access patterns and trends in order to identify structural issues
and resource groupings.
Customized usage tracking analyzes individual trends so that Web
sites can be personalized to specific users. Over time, based on
access patterns, a site can be dynamically customized for a user
in terms of the information displayed, the depth of the site structure
and the format of the resources presented.
Also Known As . . .
Over the past decade, the terminology used to describe Web harvesting
has undergone several changes. In 1996, researcher Oren Etzioni
wrote a paper called "The World Wide Web: Quagmire or Gold
Mine?" which was published in the journal Communications of
the ACM. Etzioni defined Web mining as the use of data mining techniques
to automatically discover and extract information from Web documents
and services.
In the late 1990s, Richard Hackathorn coined the term Web farming
to describe a discipline combining aspects of data warehousing,
Web data mining and knowledge-base creation.
Around the turn of the millennium, Web harvesting began to replace
Web mining as the fashionable buzzphrase, although it can mean different
things to different people. Web harvesting can be synonymous with
Web mining, Web farming and Web scraping, but it can have other
meanings as well. One widespread usage of the term refers specifically
to the searching of Web pages for e-mail addresses for resale and
use in commercial solicitations (i.e. spam).
The Web site of the Medical University of South Carolina defines
Web harvesting as "the process of downloading RSS feeds and
consolidating them for display."
Another related term is Web scraping, an obvious derivation from
the 1980s catchphrase "screen scraping," where PC- or
mini-based applications accessing mainframe systems emulated 3270
or VT100 terminals. Such applications were quick and cheap but not
always reliable. Similarly, Web scraping applications process a
Web page's HTML to extract meaningful data, often from live data
feeds or by manipulating specific applications. Web scrapers are
also cheap and useful but of questionable reliability.
Kay is a Computerworld contributing writer in Worcester,
Mass. Contact him at russkay@charter.net.
|
