January 09, 2018

Reading Lists: finding out what users really want

By Elizabeth Simpson and Kirsty Whitehead

In summer 2016 it was decided that the Library would look to upgrade its reading list system. The in-house system we had developed was at the end of its life and we wanted to switch to a product provided by a commercial provider. Members of the Academic Liaison team were on the project group and their primary responsibility was to engage our university community in the process of selecting the new system.

Gathering user requirements
We were fortunate to be able to pull feedback from a variety of sources. In addition to the knowledge that we already have through our work with academic departments, we were able to draw on existing feedback from our subject emails, CRM, and most importantly, the Understanding Academics project that had recently started. Rather than going back out and asking people to comment again, we started by synthesising the rich commentary we already had. The aim was to be able to clearly articulate what features people would like to see in the new reading list system. 

What did we do?
Initial work had already been done to code up the Understanding Academics interviews. During the summer of 2016 we looked at the 30+ pages of comments related to reading lists, and the first step we took was to analyse the comments in more detail and identify the main themes that emerged. We also made a note of our initial thoughts and ideas about how we could take this forward, and created a summary of them. Below is a snapshot of the document:

1.

Theme
Number of comments
Summary of comments
Take forward....
Alternatives to EARL
1
Currently using paperpile to generate lists

Clunky
20
Needs to be more user friendly for staff and student users. Issues raised:
  • multiple screens, slowness, too many clicks, it’s ugly, difficulty moving items within lists and between lists (esp if scans are attached), only works properly with IE.
  • rolling over could be smoother or automatic
  • should be able to hide specific items for future use.
  • difficulty finding items which are already in stock
Invite staff/student reps to the supplier demos.
  • make it easier for staff to input the details of the item(s) they want to include (importing/bookmarklet) esp. for items on the catalogue.
  • provide more flexibility for editing lists
  • make it easier to find items already in the catalogue?
  • improve rollover (automatic if possible)

However, this didn’t provide clear enough criteria that we could use further on in the project in terms of assessing potential systems. We developed a second document that really drilled down and listed each requirement separately, stating the needs and goals of our academic staff and students. Below is a snapshot of the document, in total there were 39 requirements listed. This was also useful to share with other colleagues in the project team who came from Collections and the Electronic Learning Development Team (ELDT) team.

2.

Id
Epic Category
As
I Want....(to do)
So That (Goal)...
Would Like to Have
Type
HL Acceptance Criteria
REQ-001
EPIC-001
Create a new reading list
Academic staff
Import references from a list I have already created e.g. word, paperpile, google doc, Endnote or other reference management software i.e. batch import.
I don't have to spend hours/days to create the list in a new piece of software. And it is easy to include things that I know we already have in stock.
Must Have
Interface
Must be able to import files from a variety of locations in a variety of formats (eg RIS).
Must be able to identify resources which are in stock from imported list, and automatically create links to these on reading list.
REQ-002
EPIC-001
Create a new reading list
Academic staff
Import a list of references from our library catalogue e.g. list of starred items, rather than having to type everything into EARL individually and hope I've included the right information to find the item I want.
I don't have to spend hours/days to create the list in a new piece of software. And it is easy to include things that I know we already have in stock.
Must Have
Interface
Must be able to import files from a variety of locations in a variety of formats eg RIS.
Must be able to identify resources which are in stock from imported list, and automatically create links to these on reading list.


How did this feed into the selection process? We decided that we needed to involve academic staff and also students in the process of selecting a reading list system. Using the feedback we had gathered via the process above and also from our conversations with other institutions, we devised 3 scenarios for the invited suppliers. These 3 scenarios covered all the points from the user requirements spreadsheet.

Five people from the project team were the official scorers but the staff and students who attended were also given a scorecard with which to score the scenarios, and prompts were provided. There was also space for them to write any other further comments. The feedback from the university community was to act as a benchmark should we all have wildly different scores. As it happened, we all scored the suppliers in a very similar manner.

"What does EARL* even mean?” This was one of the comments about our old system (called EARL) from one of the Understanding Academics interviews. We used switching to the new system as a chance to update the name and simply call the system Reading Lists. This has elicited lots of positive feedback from academic staff and students, who refer to lists as ‘reading lists’ and were often confused by our use of ‘resource’ lists. The new name reflects this user preference, regardless of the fact that lists contain much more than just reading.

What’s next? 18 months later, as part of reflecting on how successful the project has been, we’ve been able to return to the user requirements spreadsheet and evaluate the system that we have now. Having looked at each requirement individually, 91% of them have been met either partially, fully or, in some cases, requirements have been exceeded. The few requirements which haven’t been met relate to referencing styles, which is work in progress, and restrictions in the library catalogue system. After a really busy few months combining the start of term and the switch to the new system, being able to go back to the user requirements spreadsheet and use this to evaluate the work that’s been done has been invaluable, especially in helping us to plan our next steps.

Indeed, the work that we’re doing around this shouldn’t stop now that the system is fully up and running. We need to sustain a dialogue with users of the system in order to remain aware of their experiences of using Reading Lists: what they like and don’t like about it, and how this should shape proposed developments in the system.

For instance, in November we ran a focus group to which we invited staff and students from all departments. We felt it was timely to do this since we had reached the end of Reading Lists’ first term, and it seemed a good way to keep the conversation open with the academic community who had been so engaged with the process right from the start. The session allowed to us to gather some useful feedback about how people are using the system, and most of it was very positive, and it also gave us the chance to talk to people about the proposed changes that we want to make to Reading Lists. We will continue to run regular focus groups so that we can continue this conversation, and we are also continually collecting feedback that we receive during Boards of Studies in our departments, anecdotally from staff and students and via the training sessions that we provide. One of the great things about having a new system means that we can finally be more responsive to the concerns that our user community have and we are keen to ensure that we continue to actively engage with them.
*Easy Access to Resource Lists

No comments:

Post a Comment