Survey Results
Home Run Reports Import Statistics Manage Profiles & Lists Reference
 
Project Description Project Status Survey Survey Results Survey Results w/ Data

Survey Results Summaries

This survey solicited responses from the ERIL listserv (Electronic Resources in Libraries) of 1,567 subscribers. The survey was not conducted in a formal manner and no conclusions should be drawn or inferred from the summary of results that follow. The purpose of the survey was anecdotal - to provide an introduction to the current practices of some electronic resources librarians working with e-resource usage statistics.

29 survey responses were received between 14 July and 6 August 2004 and are included in the following summary, except where noted. When percentages appear, they reflect portions of the total number of respondents to that question only.

For further information, please see the main ERUS page, or contact Caryn Anderson at caryn.anderson@simmons.edu.

Quick Links to Survey Results

This page contains only summaries of the results for each question. For results summaries that include the RAW DATA, please view the ERUS Survey Results with RAW DATA page.

Executive Summary

The Survey

This survey solicited responses from the ERIL listserv (Electronic Resources in Libraries) of 1,567 subscribers in support of a project to develop an integrated, user-friendly, web-accessible data repository for electronic resources usage statistics. 29 survey responses were received between 14 July and 6 August 2004 and and while the survey was not conducted in a formal manner and no conclusions should be drawn or inferred from the summary of results that follow, the responses provide a useful introduction to the current practices of some electronic resources librarians working with e-resource usage statistics.

The Respondents

The 29 respondents represented primarily academic institutions, though 2 corporate and 1 public library were also included. The academic institutions ranged from Associate Colleges and Specialized Institutions to Masters Colleges and Universities (I) and Doctoral/Research Universities (Extensive). The most responses came from these last two. The average size of the institutions represented is 13,654 (FTEs). Questions were asked about Data Collection, Data Analysis, Utilization of Statistics and feedback was solicited on a potential model for an integrated usage statistics system.

The Problems

The general sense from the results of this survey is that these 29 respondents are frustrated with the lack of standards in electronic resources usage statistics provision - both in the calculation of data and in the delivery of the reports. This is the most irritating problem, because it prohibits real comparisons of usages statistics across vendors. It also contributes to challenges in the extensive time and effort in collecting data as well as trying to explain the statistics to colleagues or subordinates.

The respondents are equally exasperated with the electronic resources vendors. They claim that vendors are not consistent in calculating their data, there are not a lot of easy to use interfaces, their data is not all COUNTER compliant (even when they say they are), the exported data formats are not consistent and often it is easier to just re-enter data manually, and the customization options for reports are not flexible enough to accommodate the needs of users who are importing the data into their own local applications which are necessary for interacting with other institutional applications (e.g. ILS, financial systems). In addition, vendors frequently change the way they calculate statistics and the data becomes inconsistent with itself over time. With some changes, data is not archived, and is therefore lost, and customer service has been poor for many respondents.

All of the above contribute to the other largest lament - the amount of time and complexity involved in gathering and analyzing usage statistics data. Because all the systems are so different, and because virtually all the data available needs some level of manipulation, and because the differences in data calculation methods and data units available requires alert and informed attention on the part of the data collector, the entire processing is deeply time-consuming. Add to this the combination of a lack of standards and questionable calculation methods by vendors and the emotional strain of constantly questioning the reliability of the data creates a stressful and somewhat unstable atmosphere around the whole endeavor.

To top it all off, once usage statistics have been collected, analyzed and presented to various personnel, many respondents report significant challenges in explaining the statistics to colleagues and getting them to actually believe the results.

The Solutions

The good news is that most of the respondents are pretty clear about what they want to see in an electronic resouces usage statistics system. They want standardization (compliance by all vendors), customization in small data units for easy integration with their local applications, automated data collection (or at least significantly reduced human intervention from the current environment) and delivery. They do not want to have to go and collect it all themselves.

As far as analysis goes, most of the respondents do not need/want much more than they types of data that the COUNTER standard recommends. But many already conduct additional cost-related analysis (e.g. cost per search) and if they don't, they would like to. They are also interested in having the option for more journal/title specific details and subject analysis of the statistics.

Virtually every respondent uses their statistics for making subscription decisions, while about half use them to improve marketing, promotion and library instruction. The statistics also provide a supportive role for budget justification in the case of half the respondents.

Most of the respondents seemed interested in the ERUS model and thought that it would serve a lot of purposes for them. Some were concerned about the ability to integrate such a system with their existing ILS and were wary about learning a new tool unless it was able to eliminate other programs they currently use. But overall, the concept of a single repository for all electronic resources usage statistics was attractive, and the added value of providing cost-related analysis, comparisons to peer institutions and subject analysis were interesting to many.

While this survey was not formal, it seems fairly clear that any usage statistics system ought to primarily focus on automating the data collection process and storing data in the smallest reasonable data units so that customized reporting is more feasible. Standardization, and compliance with the standards, are problems that can only be solved by advocacy with vendors and support of standardization efforts, but it seems possible that a tool can be developed to assist electronic resources librarians in managing usage statistics in the current, chaotic environment.

We are deeply grateful to those 29 who took the time and energy to complete the survey. Summaries of the responses to each question are listed below, with links to view the raw data. For further information, to be kept updated on the progress of the ERUS Project and/or to be included in plans for a pilot project, please e-mail Caryn Anderson at caryn.anderson@simmons.edu.

Top

Question Summaries

Demographic Data

1. Total number of students enrolled at your institution in FY04 (FTEs (full time equivalents), both undergraduate and graduate)

28 respondents provided data for this question. Range and average includes two non-academic institutions that listed their organization size.

Range: 400 – 34,694. Average size: 13,654

View the Data | Top

2. Your institution’s Carnegie Classification.

Two respondents were not academic institutions and one respondent did not provide data. The remaining 26 were distributed among Carnegie classifications as follows:

Carnegie Classification ERUS Total ERUS Percent U.S. Total U.S. Percent
Doctoral/Research Universities – Extensive 8 30.8% 151 3.8%
Doctoral/Research Universities – Intensive 2 7.7% 110 2.8%
Master’s Colleges and Universities I 6 23.1% 496 12.6%
Master’s Colleges and Universities II 3 11.5% 115 2.9%
Baccalaureate Colleges – Liberal Arts 2 7.7% 228 5.8%
Baccalaureate Colleges – General 1 3.8% 321 8.1%
Baccalaureate/Associate Colleges 1 3.8% 57 1.4%
Associate’s Colleges 1 3.8% 1669 42.3%
Specialized Institutions (Theological, Medical, Engineering/Technology, Business/Management, Art/Music/Design, Law, Teachers, Other) 2 7.7% 766 19.4%
Tribal Colleges and Universities 0 0% 28 .7%
TOTAL 26 100% 3941 100%

There may be a variety of reasons why the distribution of respondents does not parallel the distribution of institutions in the United States as a whole, but no conclusions can be drawn here. The distribution, however, does provide a useful context to keep in mind while reviewing the remainder of the results.

View the Data |Top

Data Collection

3. Who collects e-resource usage data? (e.g. on-site personnel? student worker? contract/temporary worker?)

Of the 29 responses to this question:

Electronic Resources Librarians (11 / 37.9 %) - Eleven respondents indicated that data was collected by individuals specifically identified as e-resource librarians (various titles), of whom two were additionally associated with technical services.

Professional Library Staff Other (9 / 31.0%) - Nine respondents noted a variety of library staff as assuming responsibility for collecting e-resources usage data including professionals from Acquisitions, Collection Management, Library Services, Reference, Systems, Technical Services, Periodicals and at least one Assistant Library Director.

Non-professional Staff (7 / 24.2%) - Seven responses state that primary responsibility for data collection fell to non-professional staff including student, temporary or clerical workers in cooperation with, or under the supervision of, other library staff.

Unclear, but amusing (2 / 6.9%) - The nature of the roles of two of the respondents was unclear as they responded with "me" and "The Automation Librarian."

One respondent indicated that data collection was also coordinated with the consortium they were members of.

View the Data | Top

4. How is it gathered? (e.g. downloaded from the sites? Notes taken from sites or e-mailes? Imported into spreadsheet or dateabase?)

Of the 29 respondents to this question, most identified data collection procedures as a "mish-mash" of downloading from vendor sites and copying from e-mails into spreadsheets and/or databases.

Some specifically noted the time and effort necessary to manipulate the data even when available in .csv formats. Two respondents said that they have found it easier to just print out reports from the vendor web sites and enter the data into their systems manually rather than fuss with the downloading and manipulation process. Two others mentioned integrating the vendor-provided data with statistics collected locally.

View the Data | Top

5. When is the data collected? (e.g. annually, each semester, when requested)

Primary Data Collection Frequency:

48.3% (14 respondents) - Monthly
17.2% (5 respondents) - Intermittently (2-6 times a year)
13.8% (4 respondents) - Upon Request
10.3% (3 respondents) - Annually
10.3% (3 respondents) - As Available (M, Q, or A as vendor provides)

While nearly half of the 29 respondents collect their usage data monthly, many of these indicated that actual reporting may only be annually, or that the monthly collection is only for certain types of resources (e.g. indexing and abstracting databases). Of those that collect data between 2-6 times a year (the next largest group), two admitted that they were supposed to be doing it monthly but just have not been able to manage it.

Over 20% collect data annually or follow a data collection schedule uniquely conformed to that of each different vendor. Most of these respondents indicated that they also collect data whenever it is requested for specific purposes. There was also a small group that primarily collected data only on an as needed basis.

View the Data | Top

6. What challenges do you encounter with data collection of usage statistics?

The volume of challenges identified was quite large as most respondents mentioned more than one. In order to reflect the fullest scope of the issues discussed, each separate issue identified by a respondent was assigned a category. Some compound issues were assigned to two categories.

Number of challenges listed by 29 respondents (grouped by issue, ordered by most prevalent):

Challenges Category Description
         
30 Vendor Vendor challenges include complaints about vendors changing their systems (for data calculation, presentation, or even access to the statistical reports) too frequently, without advising clients effectively and/or without archiving old data; poor customer service response; statistical reports that are confusing; interfaces that are difficult to use; and general issues of trustworthiness (not presenting COUNTER compliant data although claiming to do so, and doubts about honesty of calculation methods).
 
17 Definitions Definitions challenges include complaints about the sheer diversity of ways that data is calculated under different and similar terminology. The challenges counted here reflect the "what do you really mean by search?" types of questions.
 
7 Comparability Challenges counted here include those where the respondent specifically mentioned the inability to compare statistics between vendors. If a respondent mentioned terminology or definition problems but did not identify comparability their issues was included in the Definitions category only.
 
4 Timing Timing challenges were noted by those respondents for whom vendors do not provide data in time for the respondents scheduled reporting on statistics for their institution. (e.g. "My Section's statistics are due on the 7th of the month.")
 
4 Formatting Formatting challenges include issues of poor formatting options, formatting options not working correctly, or the fact that different vendors don't all have the same options available.
 
4 Substance Substance challenges include complaints about not enough statistics, not enough detail to the statistics or issues related to the cross-counting of resources available from multiple sources.
 
4 Procedures Procedures challenges counted here reflect those respondents that complained about the complexity of remembering all the various instructions for accessing, selecting, running, downloading and manipulating data.
 
4 Explanation Explanation challenges include complaints about how hard it is to explain all the diversity in statistics to colleagues or subordinates (e.g. "not being able to have a student do it..."), and also the poor descriptions by vendors of the precise nature of the data they are providing (some of these types of complaints were also counted as a vendor challenge).
 
3 Amount Amount challenges counted here include complaints about the sheer volume of work involved in collecting, manipulating, analyzing and reporting on e-resource usage statistics.

The results above highlight the well-known dissatisfaction with the lack of adherence to standards for defining, calculating and presenting data (Definitions) and the resulting inability to compare statistics between vendors (Comparability). The data also reflect an apparently deep frustration with the behavior of vendors (Vendors).

Vendors reviewing these results will find the raw responses to this question highly instructive.

View the Data | Top

7. How would you like to see the process improved? What do you envision the ideal data collection procedure would be?

As might be expected, the ideal systems for 28 respondents are too diverse to effectively group. Themes that emerge include a combination of standardization, customization, and a delivery vs. retrieval model for data collection.

Standardization dreams include compliance by all vendors, simplification of the COUNTER standard, standardization of the delivery format as well as the presentation format (e.g. While the COUNTER report may appear the same on the screen, the delivery form is not always the same. For example, some vendors send .csv files as attachments that can be easily opened in Excel, while others send the ".csv" data in the text of an e-mail which cannot be effectively imported into Excel. Or the .csv version is displayed in a browser and must be saved from there.)

Customization of output beyond time period and frequency covered is desired. Respondents wish to be able to select data elements desired and ignore those unwanted, and/or to select in distinct data units so as to avoid transposing data and further manipulation in the local target application. A note on Integration: It seems important to recognize that standardization of reporting is only good up to a point as many respondents must use local applications for managing usage statistics because they ultimately need/want to integrate the statistics with other local institutional applications (e.g. "become a part of ILS" "run reports against financials").

Delivery vs. retrieval model of data provision refers to the desire to reduce the amount of time accessing vendor web sites to retrieve data. In combination with the key themes above, the simultaneous delivery of all data from all vendors in a standard and customizable format that is ready for direct import into a local application seems to capture the ideal.

A review of the raw responses is recommended for a broad sense of the current dreams about usage statistics.

View the Data | Top

Data Analysis

8. What additional analysis do you do beyond what the vendors provide? (e.g. do you group the statistics? how?)

Of the 28 respondents to this question, over 20% are not currently doing any analysis beyond the COUNTER level, mostly due to time constraints. Of the other 80%, there is a broad range of analyses being conducted, but three types of analysis are by far the most common:

* Ranking databases and other e-resources by volume of searches or other indices
* Annual comparisons to track change over time for each vendor
* Cost-related analysis including cost per search (most common), cost per article/hit, cost per download, search per user, cost per search per user

Other types of analysis reported by the respondents include measuring turnaways per "seats," on- and off-campus access comparisons, reviewing titles, ranking and comparing by subject and/or major, comparing vendor-provided and locally-collected click-through stats, ARL E-metrics and weighting.

View the Data | Top

9. Do you summarize more than the sessions, session turnaways, searches, and items retrieved that the COUNTER standard recommends (e.g. do you track to deeper levels (journals), do you track stats for unique types of items retrieved when available?)

The responses to this question appear to indicate that either the COUNTER project is on the right track in what it is choosing to track and/or it is helping to shape habits of e-resource librarians. Of the 29 respondents, over 40% do not track any statistics beyond those covered by the basic COUNTER reports. An additional one third say their institution tracks journal related statistics at least periodically, although sometimes it is another department that does it or cooperates. For the most part then, the COUNTER standard serves over 75% of these respondents well.

Additional types of statistics collected and analyzed by the remaining six respondents include detailed item retrieved information, basic v. advanced search, comparisons to peer groups where available (JSTOR), and modes of delivery.

View the Data | Top

10. What time frames do you address? (e.g. monthly? daily? hourly?)

Virtually all respondents answered "Monthly."

View the Data | Top

11. Who are the summary statistics provided to? (e.g. library director? committee? senior institution personnel?)

Percentage (and number) of respondents who have indicated that usage statistic data are provided to the various types of personnel or media listed below:

65.5% (19 respondents) - Library Director or Dean
58.6% (17 respondents) - Other library staff (reference, subject librarians, etc.)
34.5% (10 respondents) - Committee (within library and/or across institution)
34.5% (10 respondents) - Annual Report and/or senior institution administration
31.0% (9 respondents) - Collections department
24.1% (7 respondents) - Available on intranet or other public internal location
10.3% (3 respondents) - Faculty

View the Data | Top

12. What challenges do you encounter with the analysis of usage statistics?

The primary challenges for analysis for the 27 who responded to this question are similar to those for collecting the data:

Comparability Across Vendors - over 40% mentioned the "apples and oranges" effect. They complained of the fact that it is fruitless to try to compare data across vendors which are collected in different ways under different definitions for terms and processes.
Vendor Internal Inconsistencies - nearly 30% noted problems within resources. The most common problem mentioned was vendors changing their methods of calculating data units such that numbers are not consistent over time and therefore not comparable even within the same database. In addition, it damages the perception of vendor's reliability.
Time - Over 20% complained of not having enough time to do analysis at all.

Other challenges include comparing on- and off-campus statistics, understanding and/or explaining the reports, inability to verify type of use (e.g. "Just because something moved off a shelf doesn't necessarily meant it was used or looked-at."), and the difficulty/impossibility in disaggregating statistics to identify subject/academic department usage.

View the Data | Top

13. What additional kinds of analysis do you wish you could easily do?

Only 22 individuals responded to this question. As with question 7 (ideal data collection) the desires are varied. The most common types of analysis respondents wished they could do easily and do more of were:

Financial Analysis - Linking usage statistics to financial data is clearly done by some respondents as evidenced by question 8, but it remains the most popular item for additional analysis desired (27.2% listed it). Cost per search, per item, per session, per user - all were mentioned.
More Journal/Title Specific Analysis - Interests ranged from wanting to isolate which database is most used for which journals, to simply more detail on usage of individual journals. In the case of journals, this survey was not specific enough about soliciting feedback about the differences between statistics for journal usage through full-text indexing and abstracting databases and e-journal subscriptions (often tracked separately by the serials department). This is a topic for further exploration.
Subject Analysis - Interests included distribution of usage by subject generally and also a desire to track by major or academic department.

Other desires mentioned include better information on abstract only databases to match the tracking of full text databases, getting more detailed data from proxy server/click throughs, and, once again, dreaming of a more automated system for collecting and analyzing electronic resources usage statistics.

View the Data | Top

Utilization

14. What are the statistics used for (e.g. deciding whether to drop/continue subscribing to a resource? Identifying resources to promote/market more? Comparing with in-house stats? scheduling support services/staff? making budget requests to administration?)

It seems very clear what the statistics are being used for. Virtually every respondent uses the statistics for making subscription decisions. These decisions include dropping or adding electronic subscriptions as well as print ones. Approximately half use statistics to uncover needs for increased marketing, promotion and library instruction. Half also use the statistics for budget justification.

Other uses include identifying subject coverage strengths and weaknesses and comparing with locally collected statistics. One respondent also uses statistics to help in scheduling library staff.

View the Data | Top

15. What kinds of decisions/actions (if any) can you point to in recent years that were directly affected by your statistics?

Due to an error, unfortunately no responses were captured for this question. Some independent responses, however, indicated that usage statistics are never the sole determining factor in any decision, which makes it therefore difficult to identify specific results.

View the Data | Top

16. What challenges do you encounter with utilizing your usage statistics?

23 responses were received for this question. While some respondents noted the now familiar refrain of problems of comparability and vendor behaviors like changing methods and general reliability, there were two issues that stode out noticably.

Time - Over 39% of the respondents indicated that time and the complicated procedures for collecting, analyzing and distributing the electronic resources usage statistics were their chief challenge in utilizing the statistics.
Explaining Usage Statistics to the Frontline - This issue turns out to be the most significant one. Over 43.4% outlined this problem in a variety of ways. Some mentioned general politics, some discussed difficulties in getting faculty to understand what the e-resources statistics say about their programs. Many mentioned general difficulties in explaining the statistical reporting of different vendors with all of their quirks. In some cases, the staff simply refuse to believe the statistics. This apparently occurs for at least two reasons: 1. the plethora of special conditions surrounding the statistics damages the credibility (e.g. "a liaison made the (valid, in my view) point that nearly every set of stats needed an asterisk next to it."); 2. inconsistencies (e.g. "if they don't jibe with what our reference folks think, they don't want to believe the stats.")

View the Data | Top

17. How do you wish your usage statistics could be utilized?

Only 19 responses were recorded for this question. Four respondents (21.0%) wished that faculty and staff would take more of an interest in how the statistics can help them improve their own work. Others longed for greater data reliability (15.7%), better integration for financial analysis and budget justification (15.7%), and the possibility of real subject analysis (15.7%).

View the Data | Top

ERUS Model

The following outline of the proposed ERUS model was reviewed by respondents in order to answer questions 18-20:

ERUS Model

Administration

Each institution will use a simple web form to input their initial profile, detailing institutional demographics, selecting resource subscriptions, etc. (similar to the set-up for a product such as Serials Solutions or SFX). An institutional representative can update the profile as appropriate (when demographics or contracts change).

The ERUS project will maintain a central e-resource statistics depository to enable review of products and statistics:

  • Indexed by subject – subjects also linked to degree programs to enable program specific queries
  • Indexed by resource types (e.g. bibliographic citation database, e-books, online reference (encyclopedia), and ARL E-metrics categories)

While the primary purpose of ERUS is to manage statistics, institutions can also choose to simplify subscription management by storing some subscription details in this central, web-based database (including contacts, URL, admin passwords, and basic contract info including price, usage restrictions, renewal dates, dbase updates, etc.).

Data

Usage statistics will be updated on a regular basis depending on the practices of each specific vendor. The following data will be available for analysis on a monthly basis, tracked to the database/journal/book level when possible:

  • Sessions
  • Session Turnaways
  • Searches
  • Items Retrieved

These statistics comply with the COUNTER standard and will be made available at the degree to which the various vendors provide the information (regardless of whether it is provided in the COUNTER report format or some other way). For vendors that offer further detailed statistics than the COUNTER standard, links to the appropriate sites, or e-mail contact information, will be provided.

Analysis

The above data categories can be analyzed by:

  • Time period
  • Subject (and/or academic program)
  • Resource type
  • Cost per search
  • Comparison to average of peer institutions

Response to model:

18. What do you like about the ERUS model?

Many of the 26 respondents were generally pleased with the overall idea. In addition, a few identified specific things they liked.

9 (34.6%) liked the comparison to peer institutions
5 (19.2%) were interested in the subject analysis and resource type identification component
3 (11.5%) specifically mentioned the cost per search functions, and
3 (11.5%) were most pleased with the whole repository concept

Others were also intrigued by the integrative nature of the project and the possibility that they might be relieved of some of their data collection responsibilities.

View the Data | Top

19. What would you not be likely to use from the ERUS model?

Only 20 responded. This may mean that the other nine would have used all of it, or simply chose not to respond. Of those that did respond, nine (45%) indicated that they would use all of it. The other 11 were concerned with whether it would integrate with their ILS or if the vendor lists would be restricted? And some were wary of learning and managing a new tool when they are managing with their own systems at an acceptable level - the ERUS system would have to do everything they are doing already and better.

View the Data | Top

20. What is missing? What else would you like the ERUS model to do?

"the word free" said one respondent. "I would hope that it's easier to set up than SFX," said another.

Two mentioned graphical features. Two asked for more administrative area for contacts, passwords, etc. while one suggested that less administrative functions would be better so as not to duplicate an ILS. Two wanted to ensure integration with existing ILS. Two wanted a breakout of various pricing strutures.

Other items and components the respondents would like the system to do include: created in open source technology, including a demographic component, tracking remote vs. in-library usage, including a statement of standards, and options for tracking consortium subscriptions separately.

View the Data | Top

For further information, please see the main ERUS page, or contact Caryn Anderson at caryn.anderson@simmons.edu.
Content Updated: 2 February 2005
caryn.anderson@simmons.edu 		Contact | Help   Simmons College Disclaimer
Creative Commons License This work is licensed under a Creative Commons License by Caryn L. Anderson. Valid XHTML 1.0!