The Archive Index

____

Description

The index file is a database of information about tools and documents in the archive. It was created to allow more efficient and structured access to the information contained in the archive. For each indexed tool or document in the archive, the index contains such entries as file name, authors, and a short abstract or description. The index file is physically located in the file /u/coast2/ftp-admin/archive_index/index.v2/index.v2

Index Structure

The structure of an index entry is relatively simple. It is composed of a series of fields separated by a #. The fields defined are, in order, file name, entry title, list of authors (separated by the commas) , type of entry (doc or tool), user id of last person to modify the entry, date of last modification of entry, list of keywords (separated by the % character), url for entry, and description.

Index Database Server

The index database server is the program that is responsible for retreiving and updating information from the index file. All accesses and modifications to the index database should be performed through this database server. The database server program is in /u/coast2/ftp-admin/archive_index/indexdaemon.pl.

Modifying the index file

Before adding an entry to the index file, check to see that the entry is not already there by searching the author, filename and title. Remember that just looking for the filename is not enough! The key_find tool is an efficient way of searching for an entry. It is not possible to partially modify an entry in the Index file. To modify an entry you must replace the entire record, re-entering all fields. To add or replace an entry you must run the key_add program.

The following rules must be followed for the addition of index entries:

The file name of the entry must exist! Multiple files cannot be specified. Rather, if more than one file are related then you should create a directory and reference that directory.
Try to keep the author name consistent! Please! The authprint tool is a good way of searching for the proper name of an author.

Once you have modified the index, you should run the pindex command to generate the /pub/aux/ index files and the WWW hyperlink pages. Also, you should run the index_integrity program. It will detect problems with the index (such as when the filename in the file#filename entry does not exist).

Tools for manipulating the index

The `key_find` tool

The key_find tool is a byproduct of the search utility built for the gopher and WWW servers. The idea is that you can run the program, give it a search type and a search keyword and it'll find and present index entries for you. It's a lot easier and cleaner that grep.

The program can be found in /u/coast2/ftp-admin/bin/key_find. The -k <keyword> option is required and specifies the keyword to search for. The option -t can be used to specify a more narrow (and a lot faster) search criteria, but is not required. Specifically you can indicate one of the following search types:

fname: search on file names only
aname: search on author names only
title : search on title only
keywords: search the keyword field only
url: search the WWW URLs only
desc: search the abstract/description only
global: search for match in any of these fields

For example, type the following command:

   'key_find -t fname -k "Artificial Life"'

to search for index entries that have the words Artificial Life in the title.

The `key_add` tool

The key_add tool is the program that os responsible for adding an entry to the archive. The program can be found in /u/coast2/ftp-admin/bin/key_add. The program is completely interactive and should be straightforward to use.

The `pindex` tool

Once an entry to the index file has been added, the pindex command, located in the /u/coast2/ftp-admin/bin directory, will generate the /pub/aux/ index files and the WWW hyperlink pages. The tool does this by calling the following programs:

Execute the process_index command with the -d and -t flags to generate the files /u/coast3/ftp/pub/aux/documents.index, /u/coast3/ftp/pub/aux/documents.abstract, /u/coast3/ftp/pub/aux/tools.index, and /u/coast3/ftp/pub/aux/tools.abstract. The permission bits for these files are set to 664.
Executes the update_abstracts command to update the file sizes on abstracts and index files.
Executes the extract_www command to automagically generates, from the index file, a series of hypertext pages that allow for easy navigation through the thousands of entries in the index.

The pindex program can be found in /u/coast2/ftp-admin/bin/pindex.

The `process_index` tool

The process_index program generates human readable versions of the index file. This program is typically called from the pindex program.

The program takes one of two parameters. If a parameter -d is given, then the program will only process those index records that have a the type field equal to "document". If the flag provided is -t then the program will operate only on those records that have the type fields equal to "tool".

The program will read each index records, discarding those records that don't match the type specified by the parameter, and writes a human readable version of each record onto three files as follows:

The file /u/coast2/ftp-admin/index.authors will contain at one line of information per author per index record entry. Hence, if a paper or tool is written by two authors, the file will contain two entries for the item; one for each author. The resultant list is sorted by author name. The line contains the name of the author, the title of the tool or paper and the full path name (relative to /ftp) of the file.

The file /u/coast2/ftp-admin/index.desc will contain one human readable, multi-line record per index record.

The file /u/coast2/ftp-admin/index.verified will contain one line per index record. The information that will be stored in this line is the file name and the name of the archive maintainer that last modified the index entry.