different direction for electronic publishers – how indexing
can increase functionality
Rhind-Tutt, April 2001
past three years, there has been an increasing trend toward extremely
large databases. Companies like Gale, Bell & Howell, EBSCO and
Wilson now have merged what used to be independent collections of journals
into "mega files" with as many as six thousand titles.
Internet services such as Google, AltaVista, and Lycos now index
as many 1.4
billion pages, and are moving to add further large collections of
texts, such as newsgroup archives.
At the same, time there's been a move away from precision searching,
along with increased pressure to create a single user interface to
all of this material. Worried that their users are unable to do Boolean
searching or understand how to navigate multiple search screens, librarians
are requesting ever-simpler interfaces. Vendors have responded. Almost
all of the systems listed above open their services with a single entry
box for searching. The user is simply required to key in a word. Behind-the-scenes
technologies such as natural language mapping and relevance ranking
serve to improve the search results.
The combination of ever-larger files and ever-simpler queries lead
us to systems intended for the average user and the average query.
Such systems miss the fundamental purpose of certain queries - for
example when scholars are looking for rare, "off the beaten track" information
that has not been found before. Such systems provide little incentive
to publishers or to librarians to incorporate new information and
new ways of finding it.
It is not my intent to disparage these kinds of systems - several of
which proved useful in writing this article. It is to suggest that
such systems are only part of the picture. We need also to look at
systems that provide richer research and analysis of data, where the
goal is to do more than simply retrieve articles in response to keyword
This article describes an alternative approach to electronic databases.
Instead of creating a system that attempts to answer general questions
from many different users on many subjects, this approach focuses on
enabling a particular group of users to answer in-depth questions in
a specific discipline. Instead of relying on automation to reduce the
need for human intervention, this approach requires substantial intellectual
effort. The examples I use are restricted to the humanities and the
social sciences, but they could equally well be used in other disciplines.
About a year ago, Pat Lawry, Eileen Lawrence and I left Chadwyck-Healey
following its acquisition by Bell and Howell. Our experience sets had
been varied. Eileen knew a wide range of library systems from her time
with what was then Ameritech. As a librarian and a professional indexer,
Pat had a rich understanding of how indexing could be used to improve
a system. I'd had experience managing the InfoTrac product line, as
well as seven years experience with SilverPlatter Information. In this
last role I had seen how the indexing and controlled vocabularies used
by files like MEDLINE could enrich systems and make them considerably
As we surveyed the electronic publishing scene, several things were
The vast majority of systems were designed to provide "pretty good" retrieval
of articles corresponding to a wide range of subjects. These systems excelled
for specific keyword searches. They did less well in orchestrating the results
of queries. And very few had the ability to combine concepts (contrasted with
combining words) to restrict searches.
The increasing size of the databases meant that traditional notions of selection
and quality were driven by the editors of the original works, rather than by
the needs of the aggregated database. With the exception of purposefully built
encyclopedias such as Grove's Dictionary of Music, most systems relied on their
size to overcome deficiencies of omission and expected the user to select which
article to read from a list of candidates.
The absence of precision in search results precluded certain kinds of secondary
analysis. For example, a question such as, "Which author was the first person
to use the word abortion?" could not be asked unless the system was capable
of generating a precise list of authors.
Very few database publishers were investing in indexing. Instead they were relying
on keyword searching, in some cases supplemented by basic date, title, author
and (sometimes) subject fields. This puzzled us, because the larger the file
the more important indexing becomes.
Indexing tended to be source-work-centric rather than user-centric. In other
words, the system would consider a web page or a work or a chapter as the answer
to all queries. Instead of listing authors, events, characters, or specific
items, the system always assumed that the user wanted a document. Questions
such as, "Give
me all battles in which more than 200 people were killed," or "List
all events that correspond to these criteria," could not be asked.
Several full-text databases required the user to ask questions, rather than presenting
the user with choices. A number had a browse list of subjects, but many users
wanted a browse list of authors, journal titles, places, media types, and more.
This tied directly to the deficiencies of indexing. With multiple index fields
and controlled vocabularies, it is relatively easy to present the user with the
complete contents of the database in an organized, tabular form. However, if
the contents have been indexed only by subject, then there is no easy way to
display, say, the geographic locations contained in the database.
The rigorous construction so evident in bibliographic journal databases like
MEDLINE was not being applied to most new databases, especially in the case of
electronic texts. This presented an opportunity, especially since the development
of processing power and software tools over the past few years has enabled large,
multi-field, relational databases to be built more cheaply than before.
In all of our research there was one system that was conspicuously different
- The Internet Movie Database (www.imdb.com). This system enables searching
for characters, for movies, for actors/actresses, rather than just for movies.
has browse tables, enabling you to view the contents of the database by country,
by date of release, by language, and more. It allows the user to ask sophisticated
questions such as "Give me all movies that star Dustin Hoffman and Tom Cruise".
This may seem like a simple question, but it requires the system to have standardized
descriptions of actors, and to understand the relationship between actor and
Developing a product
In July 2000 we decided to begin a new publishing company, Alexander Street Press.
Our goal was to develop databases that would address the opportunities above.
We began by asking scholars and researchers what kinds of questions they wanted
a database to answer. Our first product was decided quickly - it was to be North
American Women's Letters and Diaries. We chose this partly in response to market
demand, but also because the letters and diaries lent themselves to extensive
indexing. Although these writings contained valuable information, it was extremely
hard to locate what you wanted to find.
One of our early decisions was on the issue of what were to be the basic elements
of the database. These are the items that would be returned to users in response
to queries. We decided on three files - one for authors, one for sources, and
one for documents. We defined a document as a month of diary entries or a letter.
I'm going to dwell on this for a moment, because it's an excellent example of
the kind of decision that is critical in creating a different kind of application.
Rather than choosing a month of diary entries as the basic unit, we could have
chosen a chapter of a book or a single day's entry in a diary.
The former - a whole chapter - would have dramatically reduced the value of
the database, because there would have been no way for us to make chapters
correspond to dates. This in turn would have meant that users could not perform
searches such as, "Give me everything written in May 1835," or any
other questions having to do with dates.
The latter - a day's writing as the result unit - would have meant that even
the simplest search would yield excessively high numbers of hits. The average
size of a daily diary entry is a few lines, so the user would be forced to wade
through hundreds of tiny entries.
Our choice of a month of diary entries or a letter as the basic element for the
document file allowed us to preserve the integrity of dates. It also allowed
us to separate materials written after the initial entry, so that searches would
yield items written contemporaneously.
The initial specification called for some 40 fields to be available for searching
each letter and diary. Although we didn't anticipate having quite this number
of fields, it quickly became apparent that scholars wanted them. Some of the
fields were obvious requirements. Others were added only after we understood
how users would want to search the database. For example, the ability to restrict
document searches to letters sent to men was of interest to sociologists exploring
the distinctions of gender.
We noticed early on that having so many fields enabled us to calculate meta-data
that was useful in itself. For example, using the Date Written and Date of Birth
fields, we were able to calculate the Age at Time of Writing field. This provides
a rich avenue for new questions to be asked. For example, the ability to search
by Marital Status and combine that with the Age of Writer allows the user to
see the attitudes expressed by young women when they were first married.
The data creation was done using a team of seven professional indexers. Pat Lawry
and Laura Gosling, librarians experienced in the development of thesauri, created
controlled vocabularies, standardizing names, places, subjects and more.
The net result of this work is a relational database with three major files
and more than eighty fields. Some forty of these fields are searchable by the
user. The other fields serve to enhance the display of materials. Each file
within the database - author, source and document - is independently searchable.
user is able to ask questions such as, "Give me all publications in the
database from the Pennsylvania Historical Society" and retrieve a list of
source works. One can also, for example, ask for a list of all authors in the
database born in Pittsburgh who were married and had more than 5 children. Searching
is also possible by document ("Give me everything written on April 5th,
A different direction?
It's fair to ask whether this kind of database is really so different from the
multitude of files already being used by libraries. I believe the answer is yes.
Each of the four products we are producing - North American Women's Letters and
Diaries, Civil War Letters and Diaries, Early Encounters in North America in
North America, and American Film Scripts Online - has more fields by an order
of magnitude and tailored definitions of what constitutes a document, and each
product enables a different level of query than just about anything else available.
From a technical standpoint, each of these databases is fully relational, containing
multiple, separate, interlinked files. For example, in the case of the Civil
War database, there is an Events file that enables the user to ask questions
such as, "Give me a list of all battles with more than 300 casualties." One
can also search to find documents pertaining to those battles. In American
Film Scripts Online there is a character file, enabling sociologists to search
retrieve a list of all African American characters who played in films from
1950-1960, for example.
From a patron standpoint, the level of query moves research significantly further
than otherwise would be possible. The intellectual effort that we expend in categorizing
the material enables users to examine hypotheses much more quickly than before.
Using the available fields, the user is able instantaneously to extract the materials
for answering the following questions:
Does a woman who has many children have a different attitude about the death
of her child as compared with a woman who has few children?
Were the encounters between the Jesuits and the Huron more violent than those
between the Franciscans and the Huron?
How did attitudes toward slavery among women on plantations evolve in the years
What metaphors are common in first-time encounters between European explorers
and Native Americans?
The system's indexing allows scholars to conduct their research more quickly
and at a greater depth than ever before. This level of indexing also enables
a layperson to understand and use the database in a richer fashion, because it
shows the contents of the database in ways that would otherwise remain hidden.
(actual list continues)
For example, the Table of Contents listed above - one of five available
TOCs - enables the user to view the contents of the database organized
events (key events in the life span of a woman), inspiring users to view
materials in ways that they might not have thought of independently.
The cost of indexing at this level is extremely high. One might ask whether
the utility we provide is worth all of the investment. My view is that it
on the application you're trying to create.
The products we've created reflect the natural role of librarians. Some library
users rarely talk to the librarian, while top-level research has always relied
on librarian involvement. Highly indexed databases allow for the same scenario
- some users will do a keyword search, find a few documents, and be satisfied,
while top-level researchers will benefit from the increased utility. Our
databases offer both simple and advanced search screens, to support both
We already have large, expensive systems capable of giving us the journal
articles we need. What we lack are systems that provide scholars and laypeople
of exploring, analyzing, and discovering information. For the novice, such
new systems can provide easy ways to find what they're looking for, and for
researcher they can lead to unique, new understandings that were not possible
in print alone.