A Python module for placing requests in ILLiad
12-31-12
This post describes a Python module for creating requests in ILLiad, the interlibrary loan software used in libraries.
Many libraries use ILLiad as the software system for document delivery and interlibrary loan services. As a developer working with this system, you might find a need to create ILLiad requests programmatically from another system. This other system could be your library catalog or OpenURL resolver or just a standalone script that processes batches of requests. However ILLiad doesn't support an API for creating requests. In response to this, at Brown University Libraries we have developed a Python module that serves as a programming interface to ILLiad. The code for the module is available on Github for downloading, forking, and inspection. It relies on two Python libraries, requests for creating HTTP requests and pyquery, which provides a nice syntax for parsing HTML documents.
Request workflow
This is the basic worfklow for the module:
- authenticate or verify the user in your external system.
- establish an ILLiad session on behalf of the user. Retain the returned session cookie for further requests.
- pass an OpenURL to ILLiad for the item the user is requesting.
- parse the response, which is an HTML form with populated values from the OpenURL. If you were doing this manually from the ILLiad user interface, this would be the pre-populated form that the user sees and either enhances with more information or clicks submit to process.
- post the values returned by the step above to the ILLiad server.
- parse the response. This response will contain either the transaction number for the request or an error message describing what when wrong.
- log the user out.
Example in code
How does the module work?
ILLiad ships with a set of web forms that will respond to OpenURL requests and pre-populate forms with the appropriate data. The module will open ILLiad web pages on a user's behalf, parse the responses, and post values to the ILLiad server.
The module does rely on screen scraping the HTML returned by the ILLiad application but experience has shown that this method is quite stable and robust enough to be used in production systems. Versions of this module have been in place at Brown for four years or more and have processed over 10,000 user requests during the last six months. One of the common problems encountered when relying on screen scraping to provide functionality is that the HTML can change without notice. In this case the ILLiad software is managed by the library so the chances of it changing without notice is small. As an extra measure to protect against unforeseen HTML changes, we have place the HTML pages that are used with this module on a different web path than the user pages (something like http://illiad.school.edu/api-pages/) so that we can update the user pages without changing the markup that this module relies on.
The module as implemented relies on the RemoteAuth in ILLiad. This allows users to be authenticated via an HTTP header and can be used with systems like Shibboleth or CAS. The module will pass the appropriate header for authentication and save the session cookie for further requests. This eliminates the need to store the user's ILLiad credentials in a local database, which could be seen as a security risk. If this is not a concern for your project, you could use a modified version of this module without enabling the RemoteAuth functionality. Leave a comment below if you would like some help in getting started with that.
Example in video
The screencast below shows an example of this module being integrated into the library's OpenURL resolver. The user in this example authenticates with the campus Shibboleth system and places a request in ILLiad directly from the resolver interface. There is no need to visit ILLiad to place the request.
If you are interested in learning more about this project or run into problems when getting started with the module, please leave a comment below.
This post describes a Python module for creating requests in ILLiad, the interlibrary loan software used in libraries.
Many libraries use ILLiad as the software system for document delivery and interlibrary loan services. As a developer working with this system, you might find a need to create ILLiad requests programmatically from another system. This other system could be your library catalog or OpenURL resolver or just a standalone script that processes batches of requests. However ILLiad doesn't support an API for creating requests. In response to this, at Brown University Libraries we have developed a Python module that serves as a programming interface to ILLiad. The code for the module is available on Github for downloading, forking, and inspection. It relies on two Python libraries, requests for creating HTTP requests and pyquery, which provides a nice syntax for parsing HTML documents.
Request workflow
This is the basic worfklow for the module:
- authenticate or verify the user in your external system.
- establish an ILLiad session on behalf of the user. Retain the returned session cookie for further requests.
- pass an OpenURL to ILLiad for the item the user is requesting.
- parse the response, which is an HTML form with populated values from the OpenURL. If you were doing this manually from the ILLiad user interface, this would be the pre-populated form that the user sees and either enhances with more information or clicks submit to process.
- post the values returned by the step above to the ILLiad server.
- parse the response. This response will contain either the transaction number for the request or an error message describing what when wrong.
- log the user out.
Example in code
How does the module work?
ILLiad ships with a set of web forms that will respond to OpenURL requests and pre-populate forms with the appropriate data. The module will open ILLiad web pages on a user's behalf, parse the responses, and post values to the ILLiad server.
The module does rely on screen scraping the HTML returned by the ILLiad application but experience has shown that this method is quite stable and robust enough to be used in production systems. Versions of this module have been in place at Brown for four years or more and have processed over 10,000 user requests during the last six months. One of the common problems encountered when relying on screen scraping to provide functionality is that the HTML can change without notice. In this case the ILLiad software is managed by the library so the chances of it changing without notice is small. As an extra measure to protect against unforeseen HTML changes, we have place the HTML pages that are used with this module on a different web path than the user pages (something like http://illiad.school.edu/api-pages/) so that we can update the user pages without changing the markup that this module relies on.
The module as implemented relies on the RemoteAuth in ILLiad. This allows users to be authenticated via an HTTP header and can be used with systems like Shibboleth or CAS. The module will pass the appropriate header for authentication and save the session cookie for further requests. This eliminates the need to store the user's ILLiad credentials in a local database, which could be seen as a security risk. If this is not a concern for your project, you could use a modified version of this module without enabling the RemoteAuth functionality. Leave a comment below if you would like some help in getting started with that.
Example in video
The screencast below shows an example of this module being integrated into the library's OpenURL resolver. The user in this example authenticates with the campus Shibboleth system and places a request in ILLiad directly from the resolver interface. There is no need to visit ILLiad to place the request.
If you are interested in learning more about this project or run into problems when getting started with the module, please leave a comment below.