Bibliographic Wilderness

Up too late, and for some reason I can’t stop thinking of ideas for improving my interfaces. That I don’t have time to do, first having to get my interfaces barely decent before I can add this stuff! So blogging.

Lib X improvements ideas (some of which might already be in libx without me knowing it; haven’t had time to explore LibX as much as I’d like):

* SFX has an interface to return a very quick response indicating ‘yes’ or ‘no’ to full text present for ISSN and date. How about LibX finds any OpenURLs on a page (COinS or an actual full OpenURL to your link resolver, recognized because LibX already knows your resolver base url), and asks SFX for coverage on each one, and indicates that to the user  if full text is present or not. SFX doens’t do ISBNs, but I could build my own ISBN index (as part of umlaut).

* EZProxy.  Every single page looked at, Lib X should check the EZProxy API to see if it’s proxyable. If it is, give the user a highlighted message “Off campus and no access? Click here to proxy please.”

OPAC:

On every search (or appropriate searches), in a sidebar or top bar, offer the first few hits from my federated search product; and the first few hits from OCLC worldcat.  Similar to how google offers Google Books and Scholar content sometimes on an ordinary google search. APIs for both can let me show the first few hits right on the page. “Click here to see more results like this”. The heading/note for the Worldcat search needs to somehow make it clear that Worldcat can be used to find more hits in a topical type search, OR to find a known item not found in our catalog to make an ILL request. (If only I could figure out how to make worldcat.org reliably show my link resolver link on hits! Maybe I need to get LibX to do that too).

[Sadly, I don't think I can show federated search results to an unauthenticated user. But I thinkI can tell if a user is already signed into our Single Sign On using shibboleth, and show it to them if they are. And offer them a login button if they are not, to see results.  If they're on campus, I can show them results straightaway.]

On 0-hit searches give even more prominence to Worldcat for finding the known item or topics you wanted.

On a 0-hit search that was for ISBN (often forwarded by LibX etc., occasionally typed in manually), offer a link to make an ILL request for that item.  Pre-fill the ILL request with details, using Worldcat API or Bowker ISBN API to look up complete metadata for the ISBN given. Maybe better show this metadata to the user with the ILL button, before they’ve clicked on it, to confirm this is what they wanted.  This may be applicable to other types of searches that are 0-hit, where there is reason to think it was a known item search, and enough clues in the search terms to try and identify the item.

Naturally, a 0 hit search on ISBN (or other known item for which an isbn or other identifier can be found) should use xISBN/thingISBN to offer alternate versions that may be in our catalog too.

Man, we could make this stuff work so much better for users, using pretty simple stuff, no serious R&D needed, just some time spent doing it.

An important reminder that this stuff really does matter, really seriously.  Do you know precisely what personally identifiable information your library’s systems keep on on patron information consumption? I don’t.   This story (while the information was not, so far as I know, from library systems) reminds me again that this really does matter, and patron privacy should be non-negotiable, not even for enhanced reccommendation features etc.  Every library should have regular privacy audits of their system to be aware of exactly what personally identifiable logs are kept being kept by what system, and that they are being kept intentionally by choice, not accidentally, only what is neccesary to condut business. Figuring this out for our chaotic ecology of often proprietary closed-source software is not trivial, but it matters.  It should go without saying that every library should have patron confidentiality policies that, to the extent allowable by law, prohibit library staff from reporting on patron activities to law enforcement.

http://www.boingboing.net/2008/05/26/uk-set-to-deport-mas.html

“UK set to deport Master’s student whose Master’s degree research led him to look up Al Qaeda info - ratted out by Nottingham University

Posted by Cory Doctorow, May 26, 2008 7:19 PM | permalink

Academics at the UK’s Nottingham University were arrested as terrorists for downloading Al Qaeda documents from a US government server in the course of research into a Master’s degree convering terrorist tactics. The two UK-born profs were released, but the student faces deportation to Algeria under the Terrorism Act, where he believes he will be tortured. The university — which encouraged its staffers to rat out people they thought were involved in researching terrorism — refuses to acknowledge that anything is wrong with any of this.”

The original article:
http://education.guardian.co.uk/higher/news/story/0,,2282045,00.html

To explain my current frustration with Internet Archive a bit more, here’s what I want to do.

What we want to do

I have library interfaces that display detail pages for known items: My catalog, and my link resolver. When my known item is a book, I’d like to do a search on the IA corpus, and see if a publically available digitized full text is available, and tell my users about it. (Check for a publically available audio book version too while we’re at it, why not). Maybe search on identifiers if available, but I know most of the time I’ll be searching on title/author keyword, and experimenting against indexdata’s OCA index, that works Good Enough ™, so good. This would be a really great feature. Users shouldn’t have to know to go to the IA search to find IA content, and shouldn’t have to do a second search, I want to put it right in my own searches.

Brewster suggests an access method

So how do I do this? At code4lib after Brewster Kahle’s keynote, I asked him this, and he sent me a nifty url to a search of IA that would let me limit to collections (like just full text ones, perhaps), and actually returned some XML with a very basic set of metadata on the items (title, collection it came from, format, a url to get associated assets including more metadata). Okay, I said, I wish that was actually documented somewhere, but that’s good enough that’ll work, so good!

The Case of the Disappearing Interface

So, several months later, I (or actually Jason Ronallo interning for me) finally get around to writing the functionality that’s going to use this access—and the access has been removed! First of all, if I already had spent time writing to this interface (at Brewster’s reccommendation!) only to find it disappear–I’d be really mad, and that’s an indication of a lack of proper communication between IA and developers like me. So I guess it’s fortunate that I didn’t, and it disappeared before I started. But still, how do I do what I need now?

It ain’t in the OL

Perhaps IA thinks that the OpenLibrary APIs are sufficient, and that’s why they got rid of their other search. Indications are that the IA is “getting” to some extent with OpenLibrary that they actually need to understand and provide for machine-access needs, if they want developers like us to interface with it. But the OpenLibrary is not good enough for me, because the OpenLibrary database doesn’t in fact include all Internet Archive hosted free digital content! It is only intended to include Internet Archive content that has MARC, but Alexis Rossi of the OpenLibrary project estimates that there are upwards of 100,000 (~25% of entire corpus) digital texts hosted by the Internet Archive which do not have MARC, and thus aren’t included in the OpenLibrary db and there are no plans to include them. Well, gee, I want to search over these too! Not to mention audio book versions–I don’t know if any of these are included in the OpenLibrary database. Why would I want to leave them out of my search to put them in front of my users just because they don’t have MARC?

So I’m (or Jason is) basically reduced to a screen scrape of the Internet Archive HTML search results. An html page which doesn’t even have quite sufficient metadata to do everything I could do with the previous XML hit list, even once I do succesfully screen scrape. And I don’t think this ‘advanced’ search lets me limit to a set of _multiple_ collections like the last one did–I’m going to have to do several searches to search several collections (putting more load on the IA servers while we’re at it). But that’s what I’m going to do, because I see little other choice. And I hate screen scraping. It is a testament to how valuable I believe the IA content is to my users, that I’m even willing to go that route–I wouldn’t do it for just anyone.

A little attention and support could go a long way

So this is an example of why I get the feeling that the Internet Archive is uninterested in developers like me and our attempts to expose Internet Archive content to our users–even as they complain, mystified, about why more developers aren’t hooking into IA content! It’s no mystery to me. I don’t think anyone at the IA even realizes what a problem they caused by eliminating this XML search, because I don’t think anyone is even paying attention to what people like us are trying to do–while complaining that we’re not doing more.

And, yeah, sure, I could harvest the whole darn IA with OAI-PMH, find just the pieces I want in it (full text and possibly audio book collections), and index them locally. Maybe eventually I’ll do that. IA does give me permission. But if IA actually gave me a machine searchable interface to their own indexes so I don’t have to figure out how to do that, it’d go a lot faster, easier, and more locally maintainable for me. I’d really rather not run my own local index of of IA content–experiences with the IndexData OpenContent OCA index shows that such things really do require attention and effort to keep them up and working (that OCA index keeps breaking and going down and being heavily out of date), attention and effort that are in short supply where I am.

If IA wants to encourage people to write hooks into their system, a little bit of support on their end (which would require caring about what we are trying to do in the first place, instead of just telling us what they think we should be trying to do) would go a long way–you get enough infrastructure there for us, we can and want to put all sorts of cool things on top of it.

But you’ve got to get enough actually supported infrastructure there to make it easy enough to do that we can find time to do it, cause we’re busy and underfunded, like everyone. You don’t need to write our applications for us, we really will write open source apps on top of the services you provide, but you’ve got to give us those services—creating our own index on top of your OAI-PMH feed (which incidentally, is also, according to colleagues who have tried to use it, a bear to deal with that often doesn’t behave as expected) is apparently just a bit too much for us out here in library developer world to realistically accomplish succesfully with our resources.

Update 26 May: The XML search is back! Thanks Internet Archive!  Thanks Jason for finding it somehow! I haven’t been checking my email or the lists over the long weekend, so I’m not sure how Jason discovered it was back, or if it’s been advertised, or if it’s a response to this complaint, or what. But thanks!

Something retrospectively obvious just occurred to me about the Google Book Search/OCLC partnership. If it means what I think it means, does that mean I don’t need to worry about the GBS API anymore (and it’s restriction to client-side javascript only access, bah!)—-I can just search the worldcat API instead?

I guess it probably doesn’t mean that _every_ book GBS book will be in worldcat though, so not quite. But it might be one option. Wonder how long until most/all full-text-digitized GBS books are in worldcat?

Wonder if there’s any way to limit a worldcat api search to only GBS books, or to only records which represent publically accessible fultext digitizations?

I think it’s kind of sad that Worldcat is apparently going to include GBS digitized books before it includes Internet Archive hosted books (which there are still no public plans for). I blame both parties. I’ve got nothing against GBS books being in Worldcat. I am suspicious of OCLC being more interested in wheeling-and-dealing with getting exclusive access to things from Google in return for access to things I don’t think it’s OCLC’s place to control access to in the first place—instead of prioritizing increasing access to IA’s already public access content which could use better access. I also remain frustrated with IA’s lack of attention to machine accessibility of it’s content and metadata, or interest in meeting the needs of those who would like to provide hooks into it. Although for all I know IA did approach OCLC for a partnership and were rebuffed. I am frustrated with both organizations lack of transparency. Google’s lack of transparency I am not frustrated with, only because it’s what I expect from a for-profit megacorp like Google.

The Internet Archive has in fact been trying to get OCLC to give their members ‘permission’ (that I don’t  believe is OCLC’s to give or withhold in the first place, but anyway) to share all their records with IA.  Which OCLC has markedly with-held.  Now OCLC hasn’t given members blanket ‘permission’ to share any held records with Google I don’t think, just records for titles scanned for GBS. (Unless OCLC is really going to share their entire corpus? THAT would make me angry, not that they’re sharing it, but that they are privileging Google over competitors and alternatives in this sharing). I believe the IA is now generally getting MARC records for digitized books that go into IA from libraries too. So that’s fine. But still.  The IA, unlike Google, doesn’t have anything to offer in return, because most of what the IA has they give permission to anyone to use, unlike Google, they can’t barter permission or access that would otherwise be withheld. So, good for IA. (I’m still frustrated with IA’s lack of attention to meeting the needs of those who want to write software that hooks into their corpus. But at least it’s not lack of permission that’s the barrier).

So I just stumbled accross a great set of projects I didn’t even know about.

The most visible is http://www.openstreetmap.org/ , a very nice free mapping system ala Google Maps or Yahoo Maps etc, but with all the data user-contributed and open access. (it doesn’t have driving directions though). It’s out of the UK, but had pretty complete street maps for Baltimore MD, at least I didn’t check more.

But that’s not what got me excited. What got me excited is http://www.geonames.org/, an open access gazetteer/geocoder/etc. Check it out, enter a place name, get lattitude and longtitude. Also in their db is hieararchical relationships between place names (ie, topopnyms) (locations in cities in states in countries), alternate language names for a place, and a bunch more.

And they’ve got a great set of APIs, and also their entire dataset is freely downloadable if you need to do something the API is incapable of. Enjoy the open access!

This is a great additional option to commercial companies geocoding APIs, like Yahoo and Google.

It’s got a pretty good free text query that determines ranked matching places–that is, place name geocoding. As you can guess, I immediately tried it out with a few LCSH geographic headings. It works pretty well, although not perfectly.

Remove punctuation from “Detroit (Mich.)” turning into “Detroit Mich”, and the first hit is exactly what you want. “Illinois and Michigan Canal”? It’s in there, man! And is matched by that string. Ah, but “Illinois and Michigan Canal (Ill.)”, another heading for the same place in my catalog does not hit, not even if you remove the punctuation, not even if you change our weird “Ill.” abbreviation to the postal code IL. So maybe a good heuristic is to remove the parenthetical qualifier and try again if at first you don’t succeed.

So anyhow, I can imagine all sorts of cool interfaces we can do with geocoded LCSH (and possibly other place vocabularies, if we have them). In the “place” facet of your fancy facetted search, have a button to show all the places on a map. But wait, there’s more! Let the user draw a bounding box on the map to limit the search to just the places within it. We could do that! And lots more.

I wonder if Ed Summers would feel like using the geonames.org db to add lattitude and longitude to all of the LCSH geographic terms in his SKOS vocabulary? It’s do-able. And would be SUPER cool.

One frustrating thing for me in general is that I can think of all sorts of cool experimental features, but we don’t even have the basics in place yet. I can’t add that to my “cool new facetted discovery tool” until I _have_ a cool new facetted discovery tool. We’ve got to catch up to “barely decent what is expected in 2008″ before we can add actual forward-looking things. This was a frustrating with my Umlaut link resolver too–I did spend a lot of time getting it up and running as a really good link resolver platform, but it doesn’t have very many really forward looking features yet. But it’s a great platform for them, it should be easy to add them now that the infrastructure is there—but I don’t have the time, I’m busy getting the next set of services (metasearch) up to “barely decent what is expected in 2008″. Although for Umlaut, I’m hoping Jason Ronallo, who is doing an internship for his school developing Umlaut this summer, can crank out some impressive stuff.

I have up to now rigorously avoided posting any content to this blog that was personal, political, or not professionally relevant. But sometimes, I guess I can’t resist it anymore. I will resist posting any commentary to the link to this article that I can not resist publicizing

Some Detainees Are Drugged For Deportation
Immigrants Sedated Without Medical Reason
by Amy Goldstein and Dana Priest | Washington Post Staff Writers
Page A1; May 14, 2008

The U.S. government has injected hundreds of foreigners it has deported with dangerous psychotropic drugs against their will to keep them sedated during the trip back to their home country, according to medical records, internal documents and interviews with people who have been drugged.

http://www.washingtonpost.com/wp-srv/nation/specials/immigration/cwc_d4p1.html

So google and Yahoo both sometimes offer “related” searches, in a nice AJAXy popup.

I don’t have time to find an example to show you, but I think most of you have seen it with Google at least. The firefox google opensearch toolbar for instance. I put in “library” and in a popup it suggests “library of congress; librarything; library thing; library journal” etc. Maybe that wasn’t the best example, but sometimes this is useful.

It strikes me that it would be really nice to have a similar feature in our various library search functions (including catalog and federated search?). First thought is, gee, can I just use the Yahoo and/or Google apis to do this? But I seriously doubt that would be consistent with either of their Terms of Service, to use this service for something that has nothing to do with google/yahoo and isn’t going to lead to a search of google/yahoo, but instead use these suggestions for search of our own content.

So, that gets me thinking, how do you do this? Obviously Google and Yahoo are coming up with these suggestions by analyzing their own data—either their corpus of indexed stuff, their query logs, or likely a combination of both. Anyone know if there are any public basic algorithms for doing this kind of thing? Anyone have enough “information retrieval” knowledge to hazzard a guess as to what sorts of algorithms are used for this? How would we go about adding this to our own apps?

Update: It also occurs to me that this would be ANOTHER natural service for OCLC to provide. To provide “related search” suggestions well, you need a good corpus and some data mining. OCLC has a giant corpus of not only book metadata, but search query history from their database offerings.  An OCLC “search suggestion” API where you give it a query, and it gives you search suggetsions, which you are licensed to use in any search your library has? I’d reccomend my library pay for that, if the price was right.  Natural service from OCLC.

Eh, this comment was long enough I might as well post it here too, revised and expanded a bit. (I’ve been flagging on the blogging lately). Karen Schneider thinks about “tagging in a workflow context“

Tagging in library catalogs hasn’t worked yet for a number of reasons…

Karen goes on to discuss much of the ‘when’ of tagging, but I still think the ‘why’ of tagging is more relevant. Why would a user spend their valuable time adding tags to books in your library catalog?

This is worth pulling out into a post of it’s own. Thanks to Dorothea Salo for the comments on the post where I broached this issue sort of in passing. Good to know that I’m indeed not alone in worrying about this stuff.

But there are actually a few different (but related) issues Dorothea has identified here, some of which aren’t a problem for my projects at all, others of which are. Let’s analyze them out:

1. Some faculty are unwilling to publish open access.

This might be a problem, but despite this problem there’s plenty of free-web publicly accessible scholarly content available. (I use this phrase because the specific licensing might be unclear, but an unauthenticated user can get it on the web.) I’m thinking specifically about so-called preprint/postprint public accessible versions of articles that also appear in not-open-access journals. There’s lots of it. This is in fact what motivates my desires in the first place.

2. Some repository software doesn’t allow control of access to the level desired by repository managers.

This might be a problem too, but despite it, most supposed “open access” repositories do contain material that the repository does not in fact make available to the general unauthenticated public! So the software might not be flexible enough, but it is often restricting access to contents in it anyway. And including metadata for those restricted items in the general OAI-PMH feed, without any predictable machine-readable way to tell that it is in fact restricted content.

So it’s in fact the ability of many repositories now to restrict content that brings me to my issue:

3. I have no way to identify the universe of actually publically accessible ‘open access’ scholarly content.

Even if I created an aggregate index of OAI-PMH feeds from all “open access” repositories—it would include content which is not viewable by an unauthenticted user! What I want to do in my software is, I have a known-item citation, I want to tell the user if there’s a publically-viewable copy of this citation online. I have no way to find/identify such a copy though! I have no way to weed out the stuff that isn’t really publically accessible. I don’t want to send the user to something they cant’ access—some repositories listed in DOAR actually have the majority of their items (in the OAI-PMH feed) not available to the unauthenticated off-campus user!

So 1 and 2 might be issues in general, but aren’t what’s providing the roadblock for me. 3 is. There are a couple other issues worth nothing, one that is an inconvenience (but not a roadblock) for my project, one that is not.

4. Difficulty of identifying articles in repositories matching a citation.

When I experimentally tried doing a search against OAISter (before I realized that OAISter didn’t even limit itself to so-called open access repositories; and before I realized that even open access repositories weren’t)—I had to do a search based just on title and author keywords. It would be better if I could search based on an identifier (DOI or pmid) when present—or based on structured publication data for the actual publication of the pre/postprint: ISSN, vol, issue, page number. But these things aren’t available in the OAI-PMH feed, and in fact probably aren’t even in most repositories metadata. Most repository metadata doesn’t try to connect a pre or post-print to the actual published version in any way.

This is annoying, but I found that author/title keyword search worked good enough to be useful even without this, so it wasn’t a roadblock.

5. Might be publically accessible, but is it open access?

This gets at what the SPARC/DOAJ initiative is trying to solve. Okay, I’m a reader, I can look at this article online on the free-web, but what am I allowed to do with it? Am I allowed to reproduce it? This matters to readers and is a real issue, but doesn’t in fact matter to my project. All I care about is if I can show them the full text on the public web—once I can do that, I can worry about helping them understanding the license and their access rights, but first I need to help them discover the article in the first place!

Freedom Summer of Code is a summer-of-code-style distributed collaboration for technology projects benefiting radical/progressive movements. Exciting idea.

http://www.fsdaily.com/Community/Announcing_Freedom_Summer_of_Code

(en) Riseup Labs is excited to announce the Freedom Summer of Code! We aim to advance critical movement technology projects and tools that benefit a wide-variety of radical social justice organizations and movements.

[...]

Modeled after the successful Google Summer of Code, the Freedom Summer of Code adds a radical social justice twist. We will be working with select tech activist organizations to generate interesting ideas as well as help people develop several projects over the next three months.

The [Freedom Summer of Code -> https://we.riseup.net/fsoc] aims to advance critical movement technology projects and tools that benefit a wide-variety of radical social justice organizations and movements; inspire developers to become more interested in directly participating in social-justice tech organizations; contributes back, for the benefit of all, to the free software world which sustains us while simultaneously honoring individual’s labor; increases the social ownership and democratic control over information, ideas, technology, and the means of communication; empowers organizations and individuals to use technology in struggles for liberation. We are developing software that is geared specifically to the needs of network organizing and democratic collaboration, providing new services that greatl enhance your security and privacy.

Consider this is a call-out!
—————————-

To get started, think about how you would like to participate. Regardless of your technical skills, we need your help and have numerous ways to plug into FSoC:

* We want your proposals, dream big! Submit any and all politically important technology project proposals for and by the radical tech community! Individuals, or organizations, can submit ideas for what they would like to see done during FSoC. We will collect these proposals and put them online for potential programmers to check out.

* Interested organizations should sign up: we want your organization to join FSoC, to not just submit project ideas, but also be an organizational contact person who can act as a facilitator if your project/organization is chosen.

* Do you want to participate? Come apply to the program, submit an individual project proposal, and when the time comes, you can pick projects that you are interested in working on.

* We also need facilitators, you are encouraged to apply to help individual participants through the process.

To learn about what kinds of things we are looking for, how to submit a proposal, to sign up as participating organization/facilitator or to apply as a participant in the first FSoC, come visit the [Freedom Summer of Code site -> https://we.riseup.net/fsoc].

[...]