Chapter 11: Integrating domains.

11.1 -- Introduction: While discussing this chapter, Dr. Subrahmanian recently noted that "Hermes domain integration is a non-trivial task." Indeed, appending new Hermes data domains remains, perhaps, the most time consuming task as the domain author must thoroughly explore the target software, envision an appropriate access function library, and then fabricate the required interface layer which also provides any necessary on-the-fly type conversions. There is no easy way to learn the process; Just do it. The budding domain author should look for an existing domain within the Hermes source library which hopefully is similar to the target domain and then use it as an example while performing the following basic steps:

Familiarize yourself with the target domain software and documentation; Determine the best method for data access (such as direct API calls, TCP/IP socket messages, or perhaps SQL server calls) and identify the target domain functions.
Declare the target domain, functions and necessary type definitions in the domain catalog.
Write the target domain's call function prototype for the Hermes search engine and updating the appropriate C header files.
Write the necessary target domain data communication, and conversion, layer. This functions as a conduit between the Hermes search engine and the actual domain data source.
Update the primary target declaration and interface production files to allow compilation and linking for all Hermes variants from one unified makefile.

11.2 -- Target domain software exploration: This section begs for common sense but must be covered nonetheless; The better you understand the target domain software, the easier it is to design and write the Hermes interface.

11.2.1 -- Software languages vary extensively, many will be implemented in C or C++ (Ingres and ObjectStore), some may be written Lisp (US Army route planners), while still others might be based from Pascal. Some companies, such as university Ingres, publish the application source code while others such as Oracle, provide a pre-compiler which renders the necessary target function calls (without revealing the underlying protected code). The domain integrator should be well versed in C, Hermes base code language, and whatever programming language the target software interface might require.

11.2.2 -- Author selected target domain functions can often be a subset of the available target software features -- You need not implement every function; For example, most database systems provide the mechanism for data table projections and joins in addition to the basic select function. Yet, for Hermes sake, simply implementing the select function (and select all) proves adequate.

11.3 -- Target domain declaration: As discussed extensively in chapter six, for every software domain, there exists a series of declarations in the domain catalog file (hermes.dom) which identify the domain name, irregular data types (if necessary), and all supported domain function calls. Add new domains by appending the appropriate declarations to the domain catalog file which define the target domain parameters.

NOTE: You can check the domain declaration syntax simply by launching one of the Hermes interfaces; If something proves wrong, the system should render an error.

11.4 -- Writing the target domain call function, a common front-end for all domains: Hermes search engines access domain interfaces through an array of function pointers -- Thus for every domain there exists a similar call function which takes two Hermes TERM pointers as input and returns a string character pointer (STR). Header files, such as c-ingres.h listed in Fig. 11-2, exist in the /source/call2 directory for every domain, declaring individual call function prototypes (such as callIngres). The /source/call2/domcall.h header file, in-turn, includes individual domain headers and defines the domain name / function pointer association array sAllCalls[] (see Fig 11-1).

11.4.1 -- Writing call function prototypes: Create the appropriate c-domainname.h header file in the /source/call2 directory and then append the new file, domain, and call function names to the domcall.h header file (#include and sAllCalls array entries).

NOTE: GROUP versus Non-GROUP builds. The GROUP macro definition allows us to conditionally compile the same code base, to use or omit certain special (un-publishable) domains at build time, through a simple compiler switch rather than require two separate code sources. "Group" specifies those domains destined for internal usage only and not distributable with the base code set. Thus our "export" Hermes package omits source code for the face, video, dow_jones, and nonlin domains, requiring compilation without the "g" (for group) target specifier (discussed below). With this in mind, "outsider" Hermes source code users may want to maintain this partition of distributable and private domains by defining new domains under the domcall.h header file _GROUP section and commenting out (or removing) the face, video, dow_jones, and nonlin entries. Others may simply wish to eliminate _GROUP references altogether and compile as one code base.

11.4.2 -- Writing the call function definitions: Typical call function steps include parsing the first argument into domain function arguments (invoking data type conversions as necessary), creating a solution file TERM for the second argument, prompting domain function execution (invoking return data type conversions as necessary), and returning the appropriate return result string pointer (NULL equates to success while returned strings cite the appropriate error message). To define the new domain call function interface (prototyped in 11.4.1), create the appropriate c-domainname.c source file in the /source/call2 directory and "stub" the appropriate TERM retrieval, and domain interface calls as necessary (see Ingres example below) to fulfill the domain function request.

callIngres

/source/call2/c-ingres.c

checkRemoteDomainFunction

TERM

getDomainFunctionName

Ingres:select

callIngres

callSelect

TERMS

Hermes

getTerm

callSelect

TERM

getString

The remaining callSelect function statements, listed in Fig 11-6 above, demonstrate calls to the underlying domain communications layer which perform the actual data retrieval and necessary result conversions. Here we find Hermes invoking an Ingres session with beginIngres, ensuring a fresh Ingres answer file with isIngresRelation, invoking the data selection through selectIngresRelation, generating a standard Hermes answer file via convertIngresRelationToHermesFile, destroying the scratch Ingres answer file through destroyIngresRelation, and finally terminating the Ingres session with a call to endIngres.

NOTE: Some domains (such as the math, string, and time domains) appear trivial, only requiring a few lines of code to implement. As such, not all domains require an underlying communications and data conversion layer; Rather, code for these may well appear directly in the main callDomainName function.

11.5 -- Writing the target domain communication and conversion layer: Once the domain interface calls are identified (see Call function definition in section 11.4), the main task becomes implementation using the available target domain API function library, TCP/IP sockets, or perhaps direct SQL messaging. As seen above, the communications layer consists typically of functions such as beginIngres, isIngresRelation, selectIngresRelation, convertIngresRelationToHermesFile, destroyIngresRelation, and endIngres, which serve to open new domain data sessions, invoke desired data processes (retrieve raw answers), convert resulting answers to Hermes standard format data files, and cleanly shut down target domain sessions.

11.5.1 -- Create the target domain source code directory (d-DomainName). NOTE: The current directory structure denotes domain directories with a "d-" prefix (highlighted in Fig. 11-7).

11.5.2 -- Create the target domain prototype header file in the new d-DomainName directory. Fig 11-8 shows the /source/d-ingres/ingres.c.h domain prototypes referenced from the Ingres callSelect function (Fig 11-6).

11.5.3 -- Create the target domain interface source code file (the Ingres domain interface lies in /source/d-ingres/ingres.c).

"Stub" out the necessary functions (prototyped above).
Generate the makefile which includes the \source\make.inc compiler macro file (defines common paths and target information). NOTE: The master Hermes makefile (/source/interfaces/makefile) treats every domain as a separate compiler target and calls the individual domain makefiles to rebuild the target domain objects when the objects are missing or outdated. Thus, every domain directory contains a makefile which draws path and target information from the /source/make.inc compiler macro file; In this respect, Hermes developers can recompile domain objects individually (invoke the compiler with "make" from the individual directories), or invoke the complete build process from the /source/interfaces directory with a call such as "make sxhermes" or "make sgxhermes" (which builds the Solaris X-Hermes plain and GROUP versions respectively).
Implement the interface functions using the available target domain resource tools such as pre-compilers, or domain API function calls. Any error checking (particularly regarding data type conversions and / or mismatches) should occur within these interface functions; The current implementation allows functions to return a string pointer (describing the error) when a problem arises.

NOTE: The /source/util directory contains the Hermes utility function implementations. Including the util.h header file in the target domain source files will allow reuse of all available Hermes utility functions (don't forget to include the directory in the target domain makefile).

11.6 -- Updating the Hermes master production files: As noted above, the primary interface makefile (/source/interfaces/makefile) obtains (includes) path and compiler target macros from the /source/make.inc file; This allows us to update the compilation settings for all interface targets (and sub-objects) from one common file. To append new domains to build process, insert the appropriate file and path references to the call function, domain object, dependency, and link object sections defined in the /source/make.inc file.

11.6.1 -- Call function make.inc file entries: Near the source/make.inc file top we define the COM_CALL2_O and GRP_CALL2_O domain call function object compiler macros (see Fig. 11-9). The main Hermes makefile (/source/interfaces/makefile) directs the compiler to include the common call function objects (COM_CALL2_O) for all builds. When building the "Group" versions (such as "make gxhermes" vice "make xhermes"), however, the file directs the compiler to include the group function call objects (GRP_CALL2_O) as well. Thus, when adding new target domains to the build system:

Append new group domain call function object paths to the GRP_CALL2_O macro definition.
Append new common domain call function object paths to the COM_CALL2_O macro definition.

11.6.2 -- Domain object make.inc file entries: For every domain, there exists a compiler macro which cites required object paths for the domain interface; Fig. 11-10, above, lists a few of the current entries. To address a new domain, define another macro (D-DomainName = ) with the appropriate object paths.

11.6.3 -- Domain dependency make.inc file entries: The main Hermes makefile (/source/interfaces/makefile) directs the compiler to check the common call function objects (COM_DEPS) currency for all builds (prompting recompilation as necessary). When building the "Group" versions (such as "make gxhermes" vice "make xhermes"), however, the file directs the compiler to check the group function call objects (GRP_DEPS) as well. Thus, when adding new target domains to the build system:

Append new group domain call function object paths to the GRP_DEPS macro definition.
Append new common domain call function object paths to the COM_DEPS macro definition.

11.6.4 -- Hermes link-object make.inc file entries: The main Hermes makefile (/source/interfaces/makefile) directs the compiler to link the common call function objects (COM_LOBJS) to the interface executable for all builds. When building the "Group" versions (commands such as "make gxhermes" vice "make xhermes"), however, the file directs the compiler to also link the group function call objects (GRP_LOBJS). Thus, when adding new target domains to the build system:

Append new group domain object macro (defined in 11.6.2 above) to the GRP_LOBJS macro definition.
Append new common domain object macro (defined in 11.6.2 above) to the COM_LOBJS macro definition.

Web Accessibility