OAI2::Harvester - Agent for harvesting from an Open Archives 1.0,1.1 or 2.0 compatible repository
OAI2::Harvester provides the front-end to the OAI-PERL library for harvesting from OAI repositories. Direct use of the other OAI-PERL modules for harvesting should be avoided.
To harvest from an OAI-compliant repository first create the OAI2::Harvester interface using the -baseURL option. It is recommended that you request an Identify from the Repository and use the repository method to update the Identify object used by the harvester.
When making OAI requests the underlying OAI2::UserAgent module will take care of automatic redirection (error code 302) and retry-after (error code 503).
Flow control is handled transparently by OAI2::Harvester.
All arguments are prefixed with a dash ('-').
You should refer to the Open Archives Protocol version 2.0 and other OAI documentation, available from http://www.openarchives.org/.
In the examples I use arXiv.org's, and cogprints OAI interfaces. To avoid causing annoyance to their server administrators please contact them before performing testing or large downloads (or use other, less loaded, servers).
use OAI2::Harvester;
my $h = new OAI2::Harvester(-baseURL=>'http://arXiv.org/oai2');
my $response = $h->repository($h->Identify)
if( $response->is_error ) {
print "Error requesting Identify:\n",
$response->code . " " . $response->message, "\n";
exit;
}
# Note: repositoryVersion will always be 2.0, $r->version returns
# the actual version the repository is running
print "Repository supports protocol version ", $response->version, "\n";
# Version 1.x repositories don't support metadataPrefix,
# but OAI-PERL will drop the prefix automatically
# if an Identify was requested first (as above)
$response = $h->ListIdentifiers(
-metadataPrefix=>'oai_dc',
-from=>'2001-02-03',
-until=>'2001-04-10'
);
if( $response->is_error ) {
die("Error harvesting: " . $response->message . "\n");
}
print "responseDate => ", $response->responseDate, "\n",
"requestURL => ", $response->requestURL, "\n";
while( my $id = $response->next ) {
if( $id->is_error ) {
print "Error: ", $id->code, " (", $id->message, ")\n";
last;
}
print "identifier => ", $id->identifier;
# Only available from OAI 2.0 repositories
print " (", $id->datestamp, ")" if $id->datestamp;
print " (", $id->status, ")" if $id->status;
print "\n";
# Only available from OAI 2.0 repositories
for( $id->setSpecs ) {
print "\t", $_, "\n";
}
}
$response = $h->ListRecords(-metadataPrefix=>'oai_dc')
if( $response->is_error ) {
print "Error: ", $response->code,
" (", $response->message, ")\n";
exit();
}
while( my $rec = $response->next ) {
if( $rec->is_error ) {
die $rec->message;
}
print $rec->identifier, "\t",
$rec->datestamp, "\n",
$rec->metadata, "\n";
}
Any other options are passed to the OAI2::UserAgent module, and from there to the LWP::UserAgent module.
The -resume argument controls whether resumptionToken flow control is handled internally. By default this is 1. If flow control is not handled by the library programs MUST check the resumptionToken method to establish whether there are more records.
repository([repo])These methods either return either a:
If there was a problem making the HTTP request.
Or, a module derived from OAI2::Response, corresponding to the method name (e.g. GetRecord will return OAI2::GetRecord).
Use $r->is_success to determine whether an error occurred.
Use $r->code and $r->message to obtain the error code and a human-readable message. OAI level errors can be retrieved using the $r->errors method.
If the response contained a OAI2::ResumptionToken this can be retrieved using the $r->resumptionToken method.
These modules have been written by Tim Brody <tdb01r@ecs.soton.ac.uk>.
You can find links to this and other OAI tools (perl, C++, java) at: http://www.openarchives.org/tools/tools.html.
9/7/2002
Bug fixes, extended documentation.
5/6/2002
Updated documentation to v2.
31/10/2001
Updated documentation to reflect new construction and flow-control mechanism.
18/10/2001
Added ability to create OAI responses, as well as read them.
17/10/2001
Initial version.