Simultaneous searching refers to a process in which a user submits a query to numerous information resources. The resources can be heterogeneous in many aspects: they can reside in various places, offer information in various formats, draw on various technologies, hold various types of materials, and more. The user's query is broadcast to each resource, and results are returned to the user. The development of software products that offer such simultaneous searching relies on the fact that each information resource has its own search engine. The simultaneous searching product transmits the user's query to that search engine and directs it to perform the actual search. When the simultaneous searching software receives the results of the search, it displays them to the user.
Simultaneous searching is also known as integrated searching, metasearching, cross-database searching, parallel searching, broadcast searching, and federated searching.  MetaLib, the library portal from Ex Libris, provides such simultaneous searching with its Universal Gateway component. In this paper, we shall refer to these systems as metasearch systems.
Let's take a look at an example of a metasearch process that a user carries out via MetaLib or a similar product.
A student is interested in the works of Henrik Ibsen. Since the student knows that Ibsen is Norwegian, she submits a search query in several Norwegian resources that she knows about, such as the catalog of the National Library of Norway, the catalog of the University of Oslo, and several archives maintained by the National Library of Norway-the television, radio, and newspaper archives. The student submits the query author = Henrik Ibsen to all these information resources. She then receives the results. If they are displayed by resource, she can easily pick out the results that seem most relevant. Let's say that one result from the television archive is a program about the play Peer Gynt, written by Ibsen. Looking at this record, the student decides that she can focus solely on the work Peer Gynt rather than all of Ibsen's works. She then uses additional functions of the system to submit a second query, title = Peer Gynt, to the same information resources. This time she receives different results, including the Peer Gynt Suite, composed by Edvard Grieg - a result from the radio archive that she did not obtain earlier. However, the Ibsen play A Doll's House, from the catalog of the University of Oslo, did not come back this time, although it was on the previous result list.
Let's take the process one step farther and ask another question: How did the student know of the resources relevant to her research? Of course, she could have been knowledgeable in this area and aware of pertinent resources. If she was just starting out, however, perhaps she concentrated on the default resources that her library has set, on the basis of her group affiliation, as a component of its gateway. Alternatively, she might have requested resources relevant to her subject, a specific geographic region, a certain type of material, and so on, thus creating a personal searching scope maintained by the system and available for reuse. In MetaLib, such functionality is provided through the Information Gateway component.
Most researchers today deal with content residing in a wide range of materials. For example, our student might want to access materials such as the script of the play in book form or PDF file, literary analyses of the play, various recordings of the suite, the score of the suite, or a video or poster of a specific performance. The immediate search result is typically a bibliographic record or other form of metadata describing the actual material. From the end user's perspective, the bibliographic records serve only as a means of obtaining the material itself. Users do not want to be bothered with technical issues such as the format of the material they seek and the software that they need to access it - the library OPAC, Adobe® Acrobat® Reader®, Microsoft® Word or PowerPoint®, MP3, the MrSid viewer, or any other software that handles specific types of files.
To provide users with convenient access to materials contained in a range of resources, multiple software products need to be integrated, and they should offer a seamless interface to users. The first type of information typically presented to users as a search result is a description of the material - the metadata - such as a bibliographic record representing a video. Ideally, the user should see the material on her screen - in this case, a video - without having to concern herself about how to find the actual material and how to view it.
The link from the bibliographic record to the actual material can be direct, an explicit URL embedded in the metadata, as in the MARC 856 field of a bibliographic record in library catalogs. However, in many instances, the system must perform calculations to create the link - for example, when the bibliographic record resides in one information repository, such as an abstracting and indexing database, but the actual material resides elsewhere, such as in an e-Journal repository or the library's printed collection. The user expects to reach the actual material nevertheless. A library can make this possible by configuring a context-sensitive linking server, such as the Ex Libris SFX server (Van de Sompel & Beit-Arie, 2001), that links the user to the actual material as a part of a set of extended services and onward navigation options. Such links include the appropriate copy of an article, the holdings in the user's library OPAC or any other relevant OPAC, the institution's document delivery service, citation information, a periodical directory, Internet searches, and information about the book in Internet bookstores or content-based services such as those offered by Syndetics. The software determines the list of links on the basis of the information in the specific bibliographic record and the institution's subscriptions and policies as predefined by the librarians.
The process of finding relevant materials for research falls, therefore, into two stages. First is the resource discovery phase, when the user locates the resources most relevant to the specific search. Next comes the information discovery phase, when the search is executed in the various information resources and the results are retrieved. Institutions strive to provide their members - students, staff, and researchers - with high quality resources that offer information of real value. It is up to the librarians to determine what constitutes the institution's collections, both physical and virtual, and set the collections' boundaries. Every member of the institution should be able to define a personal scope that derives from the institution's scope.
Once the user sets the scope of the search and submits a query, the information discovery phase begins. The metasearch system delivers the query to the selected information resources and returns the results to the user. The process requires that the system 'understands' the expectations of the resources regarding the form of the query, on the one hand, and the nature of the results, on the other. It is up to the system to convert the unified query and adapt it to the requirements of each searched resource, deliver the query in the form appropriate to each resource, receive the results, and manipulate them so that they comply with the system's unified format.
The first question, therefore, is which resources are available and which of those are appropriate for the institution. No software can replace librarians when it comes to an understanding of the scholarly information arena; only they can select the resources that are appropriate and affordable for their institution. However, the selection of a resource is just the first step. Information about the resource, resource metadata, is necessary as well. The metasearch software needs to obtain descriptive metadata about the resource, such as its coverage and the types of materials that it offers, and makes it available to end users so that they can make a knowledgeable decision about the relevance of the resource to their needs. Furthermore, the system needs technical metadata regarding its impending interaction with the resource.
Resource metadata can be made available in several ways:
|·||Resources can offer their metadata to any metasearch system that attempts to access them for the purpose of information retrieval.|
|·||A central repository can offer resource metadata to any metasearch system.|
|·||Metasearch systems can maintain their own repository of resource metadata.|
The first method - that a resource describes itself when relevant - seems the best. Resources provide the most accurate information about themselves, information that other repositories need not replicate. As a matter of fact, the Z39.50 Explain function was based on this premise. The idea was that when external software needed to access an information resource, the software would extract the details of the impending interaction from the resource on the fly and use the information to formulate the exact steps of the interaction. Apparently, few vendors implemented the Z39.50 Explain function, and those who did implemented it in a variety of forms. The Semantic Web approach takes the idea one step farther. With this approach, a typical metasearch process involves an interaction between agents that exchange requests and information to construct the final product, which is the information requested by the end user. This is the vision, but today's Web does not allow for such interaction between agents, and, therefore, an automated interaction between the metasearch system and a resource's own search engine cannot be achieved at the present time (Sadeh & Walker, 2003).
The second method - building and maintaining a central repository - is under discussion by the new NISO metasearch committee, MetaSearch Initiative, which was formed in early 2003. Maintaining a central repository would assure the availability of resource metadata but would pose new challenges. First, a decision would need to be made about which kinds of resources such a repository would store. Then a format for the resource metadata would need to be specified, as well as protocols dictating the manner in which resource metadata find their way to and from the repository. Finally, a decision would have to be made about who is responsible for storing information in the repository and keeping it updated - the repository, by means of harvesting programs, or the resource itself. Another undertaking similar to that of the NISO committee is the Information Environment (IE) Service Registry pilot project, driven by MIMAS, in the UK, in collaboration with UKOLN and the University of Liverpool. The purpose of the project is to provide a registry of IE collections and services and examine the feasibility of such a registry in terms of discovery, access, maintenance, sustainability, ownership, and scalability. The information science community is watching these initiatives with interest to see whether such repositories become comprehensive and robust enough to provide services as necessary.
The third method is one that various current metasearch products have already implemented. Each such product holds the metadata, both descriptive and technical, of all the resources that it can access. Products differ in the amount of descriptive metadata that they release to the end user and the way in which they display it. They also differ in the degree to which they implement the search interaction and hence vary in the amount of technical metadata that they store. The method whereby each metasearch system maintains information about the resources has many drawbacks. The most obvious one is that every vendor of a metasearch system has to configure and maintain the resource metadata. Handling such a repository requires considerable effort and therefore depends on the capabilities of the individual vendor.
MetaLib, like other products, provides a repository that includes the metadata of all the resources that it can access. However, the metadata are not maintained as part of the software but stored in the MetaLib Knowledge Base, a repository of resource data and rules. The software itself does not include any information that relies on specific resources: it extracts the information from the Knowledge Base. This information enables the user to select the resources and the MetaLib Information Gateway to perform the actual search and retrieval. If, in the future, one of the first two options regarding the origin of the resource metadata materializes, MetaLib will only need to extract the required metadata from another repository.
The MetaLib Knowledge Base is a proprietary repository provided to institutions along with the MetaLib software. The Knowledge Base holds two types of metadata about resources:
|·||Descriptive metadata, such as the resource's name, coverage, language, data types, and publisher. The user sees this information and, with it, can make a sensible selection of resources. It is the same information that enables the system to create resource lists based on the user's specifications and display them in a comprehensive way. In short, this information serves the resource discovery phase described earlier.|
|·||Technical metadata, such as the type of protocol that the resource supports, the cataloging format it uses, and the physical and logical structure of the records that it retrieves. We can describe this information as rules that define the flow, interface, and manner of searching and that the software uses for searching, retrieving the results, and manipulating them - that is, for the information discovery phase.|
The resource metadata in the MetaLib Knowledge Base can be divided into global metadata and local metadata:
|·||Global metadata are that part of the resource metadata that is universal and does not depend on the implementation of MetaLib at a specific institution. These metadata include the name of the resource owner, the coverage, and the interfacing rules.|
|·||Local metadata are institution-specific; they relate to the way in which the resource is used in the institution's environment and presented to the institution's members. Such metadata include elements of authentication vis_à-vis the provider of the resource, the authorization rules that apply to it within the institution, and the categorization information that the institution uses to enable the software to offer the resource in specific contexts. For instance, one institution might categorize a certain resource under Medicine, whereas an institution with a different orientation might categorize it under Social Studies.|
Ex Libris maintains a master Knowledge Base, which is copied to every MetaLib installation. Automated routines ensure that the Knowledge Base at each installation is updated as necessary. Institutions localize the relevant metadata and add configurations to local resources.
The process of searching and retrieving in a heterogeneous environment is far from trivial. Each resource has its own expectations regarding the form and manner in which it receives queries; even if the resource supports a standard interface, such as the Z39.50 protocol, the metasearch system needs to make further adjustments so that the resource's engine will interpret the query correctly.
The types of information that the Knowledge Base maintains to enable the system to search include the following examples:
|·||Access mode: What kind of interfacing protocol does the resource employ? Is it a structured, documented interface, such as Z39.50, the PubMed Entrez protocol, or a proprietary XML gateway? Or is it an unstructured HTTP protocol that dictates the use of HTML parsing techniques to access the resource?|
|·||Password control: How does the user access a specific, licensed resource? Are a user ID and password required, which the metasearch system delivers when the connection is established? Should the software redirect the query via a proxy to grant the user access?|
|·||URL creation: If a URL needs to be formulated to hold the specific query, what should the structure of the URL be?|
|·||Character conversion: What character set does the system use at the resource end? Does the character set comply with that of the end user?|
|·||Query optimization: How should the query be structured?|
|1.||What is the exact syntax that the resource's system expects?|
|2.||How should fields be mapped to the fields of the resource; for example, to which field should the system map the "author" field selected by the user for a specific query?|
|3.||How does the system expect to receive an author's name? Should it be|
<last name><,><first name>;
<last name>< ><first initial>; or in some other format?
|·||Normalization: What should the system do when the search engine at the resource end does not support a specific type of search? For instance, what rules should be applied if the user looks for a specific subject but a certain resource does not support a search by subject?|
Once the information is there, the metasearch system can indeed adapt a single, unified query to the requirements of the specific resource, as in the following example.
The user submits a query for title = dreams and author = Schredl, Michael in the following resources:
|·||Library of Congress (Z39.50 access to Endeavor's Voyager ILS)|
|·||NLM PubMed (the Entrez HTTP protocol)|
|·||HighWire Press® (HTML parsing)|
|·||Ovid MEDLINE® (Z39.50 access via the SilverPlatter ERL platform)|
|·||University of East Anglia (XML access to the Ex Libris ALEPH ILS)|
Even when looking at one brick of the process structure - the query syntax - we can clearly see the differences between the resources:
|·||The Library of Congress expects this query string: 1=Schredl, Michael AND 4=dreams|
|·||PubMed expects this query string: term=dreams+AND+Schredl+M|
|·||HighWire expects to see the encoded form of the following URL: author1=Schredl,+Michael&author2=&title=dreams|
|·||Ovid's MEDLINE via ERL, although accessed by the same protocol (Z39.50), expects this query string: 1003=Schredl-M* AND 4=dreams(Note the phrasing of the author's name.)|
|·||The ALEPH system at UEA expects the following encoded request: wau=(Schredl, Michael) AND wti=(dreams)|
Up to now we have discussed only the flow from the user to the resource. However, now that the query has been processed, the metasearch system needs to get back to the user with search results. Typically the interaction between the metasearch system and the resource consists of two phases. The first occurs after the search has been invoked: the resource returns the number of hits and some kind of reference to the result set. This phase is important because it gives the user some information about the search and enables the user to refine the query before browsing through the results. For instance, if a user sees that there are thousands of hits, she can modify the query to be more specific and thus reduce the number of results. The second phase consists of retrieval: The metasearch system retrieves the number of hits along with the first few records for each resource. This information is shown to the user instantly, even though the query might result in hundreds or thousands of hits. Some systems, including MetaLib, allow for further retrieval upon request.
Why do the systems provide such limited retrieval initially? First, retrieval depends on the use of networks, which are still not as rapid as one would like. Retrieving hundreds or thousands of records over a network is an extremely time-consuming process, and users are not likely to wait until it is completed. Second, people have difficulty handling immense result sets; after seeing the number of hits for each resource, users are likely to refine their query to obtain fewer hits. Once retrieved from the resource, each result is converted to a unified format before the user sees it. The rules that define the manipulation of the retrieved data are part of the resource metadata, which, in MetaLib, is stored in the Knowledge Base. These rules include information about the logical format, the cataloging format, the script, and the structure of certain fields, such as the citation field. For further processing to take place, the metasearch system must be able to apply these rules and convert all retrieved records, regardless of their origin.
Such additional processing can include the unified display of the records to end users; the merging of result lists from heterogeneous resources into one list; the comparison of records to eliminate duplicates; the creation of an OpenURL to allow context-sensitive reference linking; and the saving of records in whatever format is required. Consequently, functionality that might have been missing from the native interface of the resource, such as the provision of an OpenURL, is added to the same set of records by the metasearch system. However, the display of result lists is not as straightforward as might be expected. Users are well acquainted with Web search engines and therefore have solid expectations regarding the display. They would like their results ranked, merged into one list, and filtered for a selected resource. Furthermore, they would like to be able to sort results by various attributes, such as title, author, and date.
Given that only the first results are retrieved from the various resources, these expectations are not so easily satisfied. When the result sets are small, all records are in the system's cache memory and so the metasearch system can offer the expected functionality in a comprehensive manner. However, the larger the number of hits, the greater the value of merging, sorting, de-duplication, and ranking - and the more difficult these features are to provide. Consider, for instance, the merging of the lists. How should it be done? The number of hits may vary considerably from resource to resource. Would it be appropriate to merge the two hits received from one resource with the dozens or hundreds of hits received from another resource? And if so, in which order? Every resource returns results in a different sorting order - by date (ascending or descending), title, relevance, or another attribute of which the users are not necessarily aware. Because only the first records are retrieved, the issue of merging the results needs careful consideration.
Other issues are the sorting capability and relevance ranking that users expect to find when looking at results, even when the resource itself does not support such functionality. Does it make sense to rank and sort only those results that have been retrieved? Let's say that the metasearch system applies certain relevance-ranking algorithms to all retrieved records and sequences them accordingly in the display to the user. This display can be rather misleading, because the 'best' hits are not necessarily those that were retrieved first. It could well be that if the user asks for more hits better results will be retrieved. A similar problem applies to sorting: even though a system might enable the user to sort the records according to various parameters, this sorting would apply only to the set already retrieved.
MetaLib handles these issues by always allowing the user to see the results for each resource. If the resource supports sorting, the user can request that the result list be sorted. Then MetaLib submits the search to this resource again, asking that the entire set of results be arranged in the order requested by the user. Hence, the user indeed receives the first records of the whole set. MetaLib also enables users to explicitly request and obtain a merged set at any point. Such a set is already de-duplicated and sortable. Institutions are likely to limit the number of records that can be merged, to avoid lengthy waiting periods caused by the retrieval of large result sets.
End-users may wonder why other searching systems, primarily the Web search engines, are able to provide them with large sets that are merged and ranked. The reason is that these systems use a different type of technology to provide the users with search results. Metasearch systems are based on 'just-in-time' processing. The system does not maintain any indexes of its information landscape locally; only when the information is required does the system access the various resources to obtain the results. The approach of Web search engines is based on 'just-in-case' technology. Huge efforts are invested in preparing the information prior to users' requests so that when the information is needed, it is obtained immediately. Google, for example, holds indexes for the entire World Wide Web, including not only pointers to sites but also information that enables the search engine to evaluate the relevance ranking of a site. When the user searches with Google, only the indexes are scanned - and the information that Google initially displays on the screen is not from the sites themselves but from this vast repository of indexes. The search engine provides the actual access to a certain Web location only when the user selects it from the list. Needless to say, huge computing power and disk space along with sophisticated technologies for harvesting, evaluating, and maintaining the information are necessary for such powerful tools.
The use of local repositories of indexes in the library environment started some time ago. As opposed to union catalogs, which actually replicate the information that is located in local catalogs, repositories such as MetaIndex from Ex Libris hold only the indexes to the bibliographic materials that are kept in the resources. An example is the MetaIndex implementation at the Cooperative Library Network Berlin-Brandenburg (KOBV), which preceded the metasearch systems a few years back: At KOBV, MetaIndex enables each of the consortium members to maintain its library system and cataloging conventions while the consortium provides a single search interface for end users. MetaIndex has now become a resource available to MetaLib at KOBV, along with other resources. No doubt that a local repository of indexes has many advantages. Information that is gathered and processed prior to queries can be organized, evaluated, and de-duplicated and therefore can be accessible to end-users in a rapid and comprehensive manner. However, maintaining such a repository has a major drawback: the repository is another system, with hardware and software, to create and maintain, and personnel must be available to take care of it.
Considering libraries' budget constraints and limitations in the technical expertise available to them, a combination of just-in-case and just-in-time approaches would be optimal for metasearch systems. Local repositories would be useful in the following cases:
|·||When no searching mechanism exists at the resource end. This situation is typical of various types of local repositories, such as those that hold research papers written by institution members or spreadsheets relevant to institutional activities; but it could obviously apply to any other data that have not yet been made available to the public.|
|·||When the information is scattered. A local repository may be worthwhile if several resources that are mutually compliant form a single resource of value to the institution. For example, a worldwide organization that has dozens of branches, each of which holds regionally relevant information, wants to provide a simultaneous search capability that will cover all the local information. Creating an index such as MetaIndex would be preferable to requiring users to search all the repositories simultaneously.|
|·||When the interface is not reliable. Some institutions want to provide access to resources that are not always online or do not offer reliable networking for accessing them. In such cases, an institution might be better off harvesting the information and keeping it as a local repository.|
|·||When preprocessing is important. Preprocessing tasks such as relevance ranking and the elimination of duplicate records can be of value for some institutions. However, a component like MetaIndex can provide a solution only if the search scope is defined and limited. For instance, at KOBV, the consortium catalogs represent a limited search scope; as a result, the mathematics department of the consortium was able to develop a sophisticated de_duplication algorithm that permitted the construction of a comprehensive MetaIndex component.|
MetaIndex from Ex Libris is created through the harvesting of information from other repositories. One of the harvesting mechanisms is the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). The use of such harvesting protocols can facilitate the gathering of data and is applicable to a wide range of resources that are now becoming OAI compliant. Furthermore, MetaIndex itself can become OAI compliant, thus serving as both a resource for MetaLib and an OAI-compliant resource that enables other systems to harvest the data from it.
The promise of a truly integrated environment in a heterogeneous world may not yet be a reality, but with the active involvement of all the stakeholders, significant progress has been made. Just a few years back, metasearch systems seemed like a dream; today they are already a building block in the information resource environment serving the academic and research community.
Ex Libris. http://www.aleph.co.il/
Information Environment (IE) Service Registry. http://www.mimas.ac.uk/iesr/
MIMAS - Manchester Information & Associated Services. http://www.mimas.ac.uk/
NISO MetaSearch Initiative. http://www.niso.org/committees/metasearch-info.html
The Open Archives Initiative Protocol for Metadata Harvesting.
SemanticWeb.org - The Semantic Web Community Portal. http://www.semanticweb.org/
University of Liverpool. http://www.liv.ac.uk/
LIBER Quarterly, Volume 13 (2003), No. 3/4