An Analytical Anniversary

Today is my anniversary.  I have been at Symplectic Ltd for one of your Earth "years".  And a very busy one it has been, what with writing repository integration tools for our research management system to deposit content into DSpace, EPrints and Fedora, plus supporting the integration into a number of other platforms.  I thought it would be fun to do a bit of a breakdown of the code that I've written from scratch in the last 12 months (which I'm counting as 233 working days).  I'm going to do an analysis of the following areas of productivity:

• lines of code
• lines of inline code commentary
• number of A4 pages of documentation (end user, administrator and technical)
• number of version control commits

Lets start from the bottom and work upwards.

Number of version control commits

Total: 700

Per day: 3

I tend to commit units of work, so this might suggest that I do 3 bits of functionality every day.  In reality I quite often also commit quick bug fixes (so that I can record in the commit log the fix details), or at the end of a day/week, when I want to know that my code is safe from hardware theft, nuclear disaster, etc.

Number of A4 pages of documentation

Total: 72

Per day: 0.31

Not everyone writes their documentation in A4 form any more, and it's true that some of my dox take the form of web pages, but as a commercial software house we tend to produce well formatted, nice end-user and administrator documentation.  In addition, I rather enjoy at a geek level a nice printable document that's well laid out, so I do my technical dox that way too.

The amount of documentation is relatively small, but it doesn't take into account a lot of informal documentation.  More importantly, though, at the back end of the first version of our Repository Tools software, the documentation is still in development.  I expect the number of pages to probably triple or quadruple over the next few weeks.

Lines of Code and Lines of Commentary

I wrote a script which analysed my outputs.  Ironically, it's written in Python, which isn't one of the languages that I use professionally, so it's not included in this analysis (and none of my personal programming projects are therefore included).  This analysis covers all of my final code on my anniversary (23rd March), and does not take into account prototyping or refactoring of any kind.  Note also that blank lines are not counted.

Line Counts:

XML (107 Files) :: Lines of Code: 17819; Lines of Inline Comments: 420

XML isn't really programming, but it was interesting to see how much I actually work with it.  This figure is not used in any of the below statistics.  Some of these are large metadata documents and some are configuration (maven build files, ant build files, web server config, etc).

XSLT (36 Files) :: Lines of Code: 8502; Lines of Inline Comments: 2762
JAVA (181 Files) :: Lines of Code: 22350; Lines of Inline Comments: 7565
JSP (16 Files) :: Lines of Code: 2847; Lines of Inline Comments: 1
PERL (58 Files) :: Lines of Code: 6506; Lines of Inline Comments: 1699
---------------
TOTAL (291 Files) :: Lines of Code: 40205; Lines of Inline Comments: 12027

I remember once being told that 30k lines of code a year was pretty reasonable for a developer.  I feel quite chuffed!

Lines of code/comments per day:

XSLT :: Lines of Code: 36; Lines of Inline Comments: 12
JAVA :: Lines of Code: 96; Lines of Inline Comments: 32
JSP :: Lines of Code: 12; Lines of Inline Comments: 0
PERL :: Lines of Code: 28; Lines of Inline Comments: 7
---------------
TOTAL :: Lines of Code: 173; Lines of Inline Comments: 52

It looks much less impressive when you look at it on a daily basis.  We just have to remember that this is 173 wonderful lines of code every day!

Comment to code ratio (comments/code):

XSLT :: 0.33
JAVA :: 0.34
JSP :: 0
PERL :: 0.26
---------------
TOTAL :: 0.30

It was interesting to see that my commenting ratio is fairly stable at about 30% of the overall codebase size.  I didn't plan that or anything.  This includes block comments for classes and methods, and inline programmer documentation.  The reason for the shortfall in Perl is suggested below.  Notice that I didn't write any comments in the JSPs because I only use this code for testing, and is less carefully curated code.

Some perl comments don't start with anything specific - they are block comments starting and ending with =xxx and =cut respectively, which is difficult to parse out for analysis easily. Therefore the Perl code line counts overestimate and the comment counts underestimate. More likely figures are, given a 0.33 comment to code ratio:

PERL (58 Files) :: Lines of Code: 5498; Lines of Inline Comments: 2707

Amount of testing code (testing/production):

9937 / 30268 = 0.33

This is the total amount of code that I wrote to test the other code that I wrote.  So nearly 10k lines of code are there purely to demonstrate that the other 30k lines of code are working.  I'm not going to suggest that this 33% is a linear relationship as the projects increase in size, but maybe we'll find out next year.  Incidentally, the test code that I analysed was the third version of my test framework, so in reality I wrote quite a few more lines of code (perhaps 3 or 4k) before reaching the final version used above.

Note that I'm a big fan of Behaviour Driven Development, and this does tend to cause testing code to be fairly extensive in its own right.

Number of new files per day:

XSLT :: 0.15
JAVA :: 0.78
JSP :: 0.07
PERL :: 0.25
---------------
TOTAL :: 1.25

In reality, of course, I create lots and lots of new files over a short period of time, and then nothing for ages.

Average file length:

Excluding blank lines: 179
Including blank lines: 211
Spaciousness (including/excluding): 1.18

What is spaciousness?  It's a measure of how I tend to space my code.  Everyone, I have noticed, is fairly different in this regard - I wonder what other people's spaciousness is?

Source Code

Do you want to have a go at this yourself?  Blogger doesn't make attaching files particularly easy, so you can get this from the nice folks at pastebin, who say this shouldn't ever time out: http://pastebin.com/GVkHd7tB.

Knowledge Exchange Workshop 16 - 17 January

I have just returned from a very interesting workshop organised by the Knowledge Exchange organisation, on the topic of Interoperability and Institutional Repositories. There were around 70 experts from the 4 countries involved in Knowledge Exchange (UK, Denmark, The Netherlands, and Germany) discussing the following broad topics in the context of interoperability:

• e-theses


• OAI-PMH


• Research Paper Metadata


• Usage Statistics


• Exchanging Research Information


• Author Identification

The findings of each group should be made public shortly, and I will be sure to post the location of any resources that I am aware of.

In the mean time I can present only the outline of the findings of the group I was in: Exchanging Research Information. This was focussed around the possibility for integration or interoperation between Current Research Information Systems (CRIS) and Open Access Repositories (OAR). There were representitives from both communities, and a large part of the meeting was for each of us to understand the other. The Common European Research Information Format was introduced to us, in light of the upcoming release of the latest revision.

It was initially felt, especially by the CRIS community, that interactions between CRIS and OARs would be very one way, and that the CRIS would simply make available the relevant information for the OARs. This doesn't strike me as being the definition of Interoperability, and so it was necessary for us to examine what was really the relationship between the data held by each system.

The approach that was taken was that a simple use case was analysed for the following features:

1) What information it would need to encompass
2) Where the information could be obtained
3) Where the information would be of interest

The use case is the traditional repository use case of "Deposit", although it was necessary to formulate this in a more general way as a "Publication Registration Process". This allowed us to successfully abstract away from where the User Interface for such a registration process lay, and thus to take away some of the arguments over whether this was the domain of the CRIS or the OAR.

Throughout the meeting, the discussion was very wide ranging, but out of it were extracted some important similarities and differences between CRIS and OARs. The most basic formulation of the key difference is as follows: The CRIS's primary interest is in high-quality, accurate metadata, while the OAR's primary interest is in content, and can live with a lower-quality metadata. This exposes two things: how the CRIS can be of benefit to the OAR, and how the problem domains do not overlap quite as much as it might first appear. My conclusion from this is that the interoperability we are talking about is actually about finding the layer at which these two systems domains can be stitched together for the benefit of the research community.

With this discussion under our belts, then, we enumerated first the information that CRIS are interested in, and then the information that OARs are interested in. The following list is not exhaustive, but gives an example of the differing perspectives:

• CRIS
  
  
  
  • project information
  
  
  • bibliographic metadata
  
  
  • researcher role
  
  
  • scientific impact
  
  
  
  


• OAR
  
  
  
  • bibliographic metadata
  
  
  • administrative metadata (technical, preservation, etc)
  
  
  • collection/group information
  
  
  • full-text / content
  
  
  • {persistent} identifier
  
  
  
  

The resulting analysis of the use case showed that information needed to come from all corners to achieve this process, including the special case of author information, which may come to the process from yet another system, albeit via the CRIS.

The general consensus of the meeting is that a working group needs to look closely at the interactions going on in this and other use cases, and specify some set of interfaces and content models that can allow for interchange of the relevant data. This should be followed by a reference implementation and service. It was proposed that the basis for a project looking at these issues might consider e-theses and other grey literature, as they may prove to be the easiest place to start.

It was good to see plenty of crossover between this and other strands. The bibliographic metadata obviously mattered to the Research Paper Metadata group, while starting with e-theses and grey literature will matter to the E-Theses group. That author names may have to come from some third-party system may well be connected to the Author Identifier group, and since interoperability is of essence, you can barely go any distance before considering at least the base problems which OAI-PMH addresses.

All in, an interesting meeting, and I'm looking forward to seeing the reports that will be published by the group moderators in due time.

Richard Poynder interviews Professor Tony Hey

Respected Open Access journalist and blogger Richard Poynder interviews Professor Tony Hey of Microsoft and The University of Southampton on his career, his move to Microsoft, his stance on Open Source and Open Access:

http://poynder.blogspot.com/2006/12/conversation-with-microsofts-tony-hey.html

This interview was probably kick-started at least in part because Southampton's world class Institutional Repository package EPrints.org can now be run on smoothly (I'm told) on Windows, and Microsoft's apparent interest in funding Open Source development work (but not GNU licenced work, only BSD or similar).

There are sentiments expressed in this interview which set my Open Source alarm bells ringing. Phrases like Microsoft "promise not to sue you" don't sit well with me, along with the idea that "if Microsoft doesn't patent these technologies, someone else will" doesn't strike me as getting to the nub of the argument. Or that the GPL "denies the existance of a software industry". These don't strike me as statements on behalf of a company that really wants to share, and my paranoia alarm says that things will only continue like this for as long as it suits them. Be on your guard.

As ever, a thorough treatment by Richard Poynder.