ORE software libraries from Foresite

The Foresite [1] project is pleased to announce the initial code of two software libraries for constructing, parsing, manipulating and serialising OAI-ORE [2] Resource Maps. These libraries are being written in Java and Python, and can be used generically to provide advanced functionality to OAI-ORE aware applications, and are compliant with the latest release (0.9) of the specification. The software is open source, released under a BSD licence, and is available from a Google Code repository:

http://code.google.com/p/foresite-toolkit/

You will find that the implementations are not absolutely complete yet, and are lacking good documentation for this early release, but we will be continuing to develop this software throughout the project and hope that it will be of use to the community immediately and beyond the end of the project.

Both libraries support parsing and serialising in: ATOM, RDF/XML, N3, N-Triples, Turtle and RDFa

Foresite is a JISC [3] funded project which aims to produce a demonstrator and test of the OAI-ORE standard by creating Resource Maps of journals and their contents held in JSTOR [4], and delivering them as ATOM documents via the SWORD [5] interface to DSpace [6]. DSpace will ingest these resource maps, and convert them into repository items which reference content which continues to reside in JSTOR. The Python library is being used to generate the resource maps from JSTOR and the Java library is being used to provide all the ingest, transformation and dissemination support required in DSpace.

Please feel free to download and play with the source code, and let us have your feedback via the Google group:

foresite@googlegroups.com

Richard Jones & Rob Sanderson

[1] Foresite project page: http://foresite.cheshire3.org/
[2] OAI-ORE specification: http://www.openarchives.org/ore/0.9/toc
[3] Joint Information Systems Committee (JISC): http://www.jisc.ac.uk/
[4] JSTOR: http://www.jstor.org/
[5] Simple Web Service Offering Repository Deposit (SWORD):
http://www.ukoln.ac.uk/repositories/digirep/index/SWORD
[6] DSpace: http://www.dspace.org/

Exciting news from the pages of the Chronicles

Some of you will already know this, but for the benefit of those that don't but wanted to know, here is some job related news on my part.

With the recent launch of Spiral, I have felt free to consider again my place in the world, the work I do on Open Source and Open Access, and my general future, knowing that if I were to leave Imperial College, I would not be leaving having achieved nothing visible.

I have, therefore, decided to make a move from the academic into the commercial sector, and have taken up a position with HP Labs to work with DSpace especially in the context of India, where it has become extremely popular. So towards the end of next month you will see the "About Me" section of this blog get updated, and I may vanish off the radar for a week or two while I get myself up and running in this new post.

I'm greatly looking forward to working with the DSpace folks both in HP Labs Bristol, Bangalore and Vermont!

DSpace 1.5 Alpha with experimental binary distribution

The DSpace 1.5 Alpha has now been released and we encourage you to download this exciting new release of DSpace and try it out.

There are big changes in this code base, both in terms of functionality and organisation. First, we are now using Maven to manage our build process, and have carved the application into a set of core modules which can be used to assemble your desired DSpace instance. For example, the JSP UI and the Manakin UI are now available as separate UI modules, and you may build either or both of these. We are taking an important step down the road, here, to allowing for community developments to be more easily created, and also more easily shared. You should be able, with a little tinkering, to provide separate code packages which can be dropped in alongside the dspace core modules, and built along with them. There are many stages to go through before this process is complete or perfect, so we encourage you to try out this new mechanism, and to let us know how you get on, or what changes you would make. Oh, and please do share your modules with the community! Props to Mark Diggory and the MIT guys for this restructuring work.

The second big and most exciting thing is that Manakin is now part of our standard distribution, and we want to see it taking over from the JSP UI over the next few major releases. A big hand for Scott Phillips and the Texas A&M guys for getting this code into the distribution; they have worked really hard.

In addition to this, we have an Event System which should help us start to decouple tightly integrated parts of the repository, from Richard Rodgers and the guys at MIT. Browsing is now done with a heavily configurable system written initially by myself, but with significant assistance from Graham Triggs at BioMed Central. Tim Donohue's much desired Configurable Submission system is now integrated with both JSP and Manakin interfaces and is part of the release too.

Further to this we have a bunch of other functionality including: IP Authentication, better metadata and schema registry import, move items from one collection to another, metadata export, configurable multilingualism support, Google and html sitemap generator, Community and Sub-Communities as OAI Sets, and Item metadata in XHTML head <meta> elements.

All in all, a good looking release. There will be a testathon organised shortly which will be announced on the mailing lists, so that we can run this up to beta and then into final release as soon as possible. There's lots to test, so please lend a hand.

We are also experimenting with a binary release, which can be downloaded from the same page as the source release. We are interested in how people get on with this, so let us know on the mailing lists.

Come and get it:

http://sourceforge.net/project/showfiles.php?group_id=19984

DSpace User Group 2007, Rome

Last week was the annual DSpace User Group Meeting, this year held in Rome, hosted by the Food and Agriculture Organization of the United Nations:

http://www.aepic.it/conf/index.php?cf=11

These guys have an interest in DSpace for sharing knowledge throught the developing world, and kindly offered to run the user group this year. The FAO building is set at the east end of the incredible Circus Maximus, and just 5 minutes up the road from the Colosseum. And we could see all of this from the 8th floor terrace cafe where lunch and coffee was served every day.

The presentations for this event are mostly available online, at:

http://www.aepic.it/conf/program.php?cf=11

If there are presenters reading this whose papers are not yet online, please contact the conference organisers so they can make it available.

I felt that this year the balance between technical and non-technical presentations was struck particularly well. While there were streams of non-technical presentations, there were highly technical tracks for the developers among us to attend. Specifically worth a mention was Scott Phillips' Introduction to Manakin, which is something we will all need to get to grips with in the long run, and something which I knew woefully little about. After that session, though, I'm confident about getting stuck in.

The quality of the work going on with DSpace is definitely reaching a high degree of maturity, with increasingly many developments leveraging the latest features of DSpace in new and innovative ways. For me this suggests that our platform has approached a critical point where we must, as a community, find a way to make these developments easier to share and easier to adopt and easier to write.

So thanks from me to the organisers. It was great to see the usual suspects again, but equally great was it to put faces to names from the mailing lists and IRC. See you all next year!

vive la revolucion

Today I'm happy to see a major hardward manufacturer teaming up with a major Linux distro, and doing so in a nice visible place like the BBC:

http://news.bbc.co.uk/1/hi/business/6610901.stm

I've been a Linux user for some years now, but when I first made the switch from the competition it was still a very difficult thing to do, even as a professional computer geek. Ubuntu seems pretty good, and hopefully it will help encourage non-expert users to have it installed before they even get their laptop home.

Ridiculously well integrated IDE and Application

This week I have been playing around a lot with Eclipse, which is an Integrated Development Environment platform, geared principally (but by no means exclusively) towards Java development. I've been an Eclipse user for some time, but principally for its ability to mediate nicely with my version control server (SubVersion), and the pleasantness of using the graphical editing tools. I was aware, though, of the huge potential it has for rapid application development, and finally got around to putting some time aside to investigate.

The product of my attempts have been documented here, on the DSpace wiki:

HOWTO Integrate DSpace with Eclipse and Tomcat

Along the way I found all sorts of goodies, like the Database Explorer tools, that allow me to execute SQL directly from files open in the editor onto my running database, and a variety of graphical and semi-graphical tools for editing my files. The full power of the source code analysis for a properly set-up project is staggering, as are the refactoring tools.

The real kicker, though, the feature that makes this effort all worthwhile is that I can now run DSpace (or any other web application) from within Eclipse. It looks after controlling Tomcat, and ties in the Tomcat debugger to Eclipse, so that (and this is the really cool bit) I can set points in my application source for execution to halt while I examine the state of the machine. So, I load a web page, which invokes code within which I have set "breakpoints", and Eclipse immediately takes over, opens up a debugging environment, and allows me to step through the code, line by line if I like, examining the in-memory objects all the way. Awesome. In the original sense of the word.

I've seen other people use similar functionality (for example in Visual Studio), so I'm glad I've replicated it in Eclipse. Of course, now I'll discover that everyone has been doing this for years, and I'm the last to catch on. But if I'm not the last to catch on, or at least to figure out how to get DSpace working in this environment (non-trivial) then I strongly encourage you to give this a go.

Google Summer of Code Go-Ahead

At the start of last week Google announced the students who had been funded under this year's Summer of Code project. DSpace is pleased to be able to say that it has 5 students being funded to do development for the platform over the next 3 to 4 months. The details of the projects are as follows:


DSpace Content Integrity Service
- Student: Jiahui Wang
- Mentor: Jim Rutherford
- Mentor Backup: Scott Phillips


Portable citations: moving citations between DSpace and bibliographic software managers
- Student: Jodi Schneider
- Mentor: Stuart Lewis
- Mentor Backups: Claudia Jurgen, Christophe Dupriez


DSpace Versioning
- Student: Robert Graham
- Mentor: Robert Tansley
- Mentor Backups: Mark Diggory + Scott Phillips


Statistics
- Student: Federico Paparoni
- Mentor: Richard Jones
- Mentor Backup: Stuart Lewis


Visualization Artifacts for Manakin/DSpace
- Student: Brian Eoff
- Mentor: Scott Phillips
- Mentor Backup: Mark Diggory

An overview of DSpace involvement is available here:

http://code.google.com/soc/dspace/about.html

We are currently involved in the preliminary discussions with the students over their work, and design and development will start in earnest during May. Watch this space for updates.

Google Summer of Code

DSpace is pleased to announce that it has been accepted as a mentoring organisation in Google's Summer of Code 2007.

http://code.google.com/soc/dspace/about.html

The following DSpace developers are officially mentoring for the period:

Robert Tansley (Google)
Jim Rutherford (HP)
Richard Jones (Imperial College)
Stuart Lewis (University of Aberystwyth)
Claudia Jürgen (Universität Dortmund)
Scott Phillips (Texas A&M University)

I am very excited about it, and we have the workings of plenty of ideas for the developments that can be undertaken.

AAP PR campaign: opinion

The last week or so have seen an explosion of discussion over the hiring by the Americal Association of Publishers of a well known PR firm whose director is known as the 'pit bull' of the PR community. Others have done the details, so I won't go over them here. Instead check out the coverage at Peter Suber's blog.

It's already been pretty heavily commented, so I wasn't going to add anything, but I've not yet seen the words of warning that immediately sprung to my mind when I read about this. Most commentary has been of the "they know they're backed into the corner, and they're fooling nobody" line. While I agree that those of us on the other side of the fence are not fooled by this, it is not us that they are concerned with. If Microsoft want to outdoo Apple, they don't market to Apple employees, saying "we're better than you, so just give up". Whether we know or not that this is just FUD is irrelevant - it is the people who ultimately make the decisions that are the targets of a campaign like this, and those people are our practicing academics, and, to a degree, members of the public.

We are all aware that people will believe the most ridiculous things if they're told them the right way, and being a top academic does not change that (I've seen some "interesting" opinions on OA from very senior staff). The battle is between the links twixt us and the academics and the links twixt publishers and the academics. If the AAP can convince their authors that OA is bad/wrong/immoral/censorship then we have a serious problem on our hands.

An analagous situation might be the Linux vs Windows argument, which has been raging for some time in this PR zone. Linux might be in the right, but at ever step of the way Microsoft have yet more cards to play to maintain their stranglehold monopoly. I don't think we've seen the end of this; in fact, I would say that we are only just over the Fuseki, and the middle-game is now underway. We cannot allow ourselves to relax in the knowledge that the publisher's have admitted that we're right, and a threat, because we've always known that

How does a loose community (by necessity) such as the Open Access community combat a well directed organisation which is seriously motivated to see itself previal? If you know the answer to that, then it won't just be this dispute which we can solve.

EPrints 3.0 Released

The release of EPrints.org 3.0 was announced yesterday at the Open Repositories 2007 conference in San Antonio

http://www.ecs.soton.ac.uk/about/news/1148

DSpace developer though I am, the Open Source voice in me reminds us that diversity in software and breadth of choice is part of the point, so congratulations to the EPrints.org team on their latest release.

Mind you, despite the press release saying "EPrints is already the world’s leading software for producing open access institutional repositories", the Registry of Open Access Repositories (ROAR) lists (at time of writing) 218 EPrints.org repositories and 223 DSpace ones. I'm just saying ;)

Calgary Rejects DSpace for E-Theses Archives

The Rejection of D-Space : Selecting Theses Database Software at the University of Calgary Archives

This article outlines reasons that the University of Calgary Archives chose not to implement DSpace to manage their e-theses, although they are still using it for the Institutional Repository system.

I'd like to include some responses to the significan issues that Calgary encountered:

• Searching Issues: I think that the main problem here is a lack of information about the customisations available in both the search and the browse. The existing browse has a very primitive customisation capability, but the search indexes are extremely flexible, and probably the addition of search.index.<x> = dc.date.accessioned would be sufficient to construct accurate searches or reports on specific years. The admission of responsibility on the DSpace part is that the search interface is not the most flexible, and it is not possible to construct searches like this without knowing exactly what you are doing. This is a very serious issue, because the report cites lack of sufficient search indices as a major concern, and yet a quick peek into the dspace.cfg file, and a quick tweak of search/advanced.jsp and all the problems are solved.



• Browse by date: just to blow my own trumpet for a moment, when the browse code I've been working on is finished, it will be possible to index by date, and browse only items within a specific period. So if date.submitted = 1994, then you can browse on all date.submitted where it is 1994 and so on



• Counting results: using the mechanisms above it would be possible to get the results counted correctly. Even if not, recourse back to generating SQL queries to do the counts would be much more desireable than counting by hand, and since our code base is open, you are free to write in the functionality



• Cannot do X: much of the content of this article is about how the UI cannot do X (allow you to search for more than 3 terms at a time | search on the relevant fields). DSpace is an Open Source product, where words like "cannot" should not be used unless you really have looked into it. The underlying search engine can do all of the things required for Calgary, and all it requires is the alteration of the UI to support it



• Reporting Issued: as has already been noted at time of writing on the DSpace lists, DSpace is not a reporting tool. I understand Calgary's pain here, as it is part of my remit to report on the content and activity of our repository. I will be using a combination of the log file analysis, the web server logs, and potentially the Minho stats add-on. I would welcome a suite of reporting tools for the platform, though



• Sorting search results: That DSpace can't is a complete nuisance, and I agree that it should be able to do so. Lucene, which is the search engine upon which DSpace relies for this functionality, does support result searching, so if anyone wants to have a go at adding in the feature, I expect you would be very popular



• Public and Private Display: This is also a potential shortcoming of DSpace, and was certainly not a design goal in the initial case. DSpace was originally intended to help you achieve open access, and so does not do so well with the ideas of public and private views. To fix this here at Imperial we have one public repository and X private repositories which deal content to the public one on demand (and yes, one of those private repositories is an under-construction e-theses repository built on DSpace).



• scalability: I feel that Calgary didn't do their research regarding scalability here. They worry that adding 600 new items annually might be a problem. Cambridge have 200,000 records in their public repository, and we have 75,000 records in our private repository, and while there are scalability bumps, they are being ironed out, and they don't show up until you get into the tens of thousands of records at all. I blame this on the over cited (so I won't) DSpace Scalability Issues page on the wiki being people's only source of information.

So what can we, as the DSpace community, learn from this. The first thing is probably about documentation - we are very good at documentation, but sometimes it can be hard to find what you are looking for. Could we do more, and if so what?

The second is about customisability of our UI. Manakin will hopefully lay to rest many of the problems that we come up against with our UI customisations, but we should also bear in mind that library administrators want graphical interfaces to modify their configuration in all senses. I can give you a concrete example of where having our config defined through the UI would be useful: At Imperial College if we want to change the configuration and restart the application server to pick up the changes, we need to go through an official "Change Request" process. If this could be done through the UI, though, this would become an administrative task, and would not require the extra bureaucracy! In addition, it will make it easier for non-technical folks to understand what options are available to them.

Third is about the nature of Open Source. DSpace has many known problems or discrepencies between what the UI will allow and what the underlying application will actually support. The DSpace core is much more powerful than the UI would have you believe, and those of us who spend most of our time "under the hood" can testify to the things that you can make it do if you know what you are doing. The problem at Calgary was that they didn't appear to understand that in order to make the system work in their exact niche, it was going to require some modification. This is a different issue, in my mind, than not having the resources to undertake said modifications.

Nonetheless, if WebGencat meets their needs, and DSpace does not, then that is a success for the diversity of software products available and the evaluation process in this part of the market. We (the DSpace community) need to learn from the feedback in the report, and hopefully use it to make our system better.

Richard Poynder interviews Professor Tony Hey

Respected Open Access journalist and blogger Richard Poynder interviews Professor Tony Hey of Microsoft and The University of Southampton on his career, his move to Microsoft, his stance on Open Source and Open Access:

http://poynder.blogspot.com/2006/12/conversation-with-microsofts-tony-hey.html

This interview was probably kick-started at least in part because Southampton's world class Institutional Repository package EPrints.org can now be run on smoothly (I'm told) on Windows, and Microsoft's apparent interest in funding Open Source development work (but not GNU licenced work, only BSD or similar).

There are sentiments expressed in this interview which set my Open Source alarm bells ringing. Phrases like Microsoft "promise not to sue you" don't sit well with me, along with the idea that "if Microsoft doesn't patent these technologies, someone else will" doesn't strike me as getting to the nub of the argument. Or that the GPL "denies the existance of a software industry". These don't strike me as statements on behalf of a company that really wants to share, and my paranoia alarm says that things will only continue like this for as long as it suits them. Be on your guard.

As ever, a thorough treatment by Richard Poynder.