Is part of your job is to encode finding aids in EAD? Then please ask if you can send a dozen of them to the researchers on this project!

Seeking EAD records from repositories that have implemented EAD

Standards have been entering the archival lexicon at a fast pace to ensure data reliability, enable data aggregation, and manage data over the long term. However, we have not yet examined the use of these standards across the archival community. As we move into the next phase of standards-creation, a broad look at current implementations will help to inform the next generation of these standards. To do this, Kathy Wisser (Simmons College) and Jackie Dean (UNC Chapel Hill) are conducting research on EAD tag usage in the encoding community.

This project is intended to inform the TS-EAD revision process of the standard, and results will be disseminated through traditional publication avenues.

We are seeking a sample of encoded finding aids from institutions that have implemented EAD. If you are willing to participate in this project, please submit via electronic mail 12 to 15 finding aids to eadtagresearch@gmail.com by December 15, 2010.

The goal of the project is to identify encoding behavior and not to evaluate the quality of the encoding or the content of the finding aid. We will be noting the presence and absence of elements and attributes and the way that elements are used within the context of an EAD instance.

All results will be anonymized; no institution-specific information will be linked to the results. Institutions willing to participate will be acknowledged.

In order to obtain an accurate account of the use of the standard, we are looking for EAD instances from as many institutions as possible. We hope you will consider contributing to this effort.

If you have any questions about the project, please contact:

• Kathy Wisser (Simmons College – wisser@simmons.edu)
• Jackie Dean (UNC Chapel Hill – jdean@email.unc.edu)

This post is from from: Spellbound Blog.

Support EAD Tagging Research

In my presentation at the Spring 2010 Mid-Atlantic Regional Archives Conference (MARAC), Whirlwind Tour of Visualization-Land,  I showed some screenshots of a tool called Gridworks. At the time, Gridworks was not available to the general public. The good news is that earlier this month Gridworks 1.0 was officially released and you can get Gridworks right now.

For those of you who didn’t see my presentation, Gridworks is tool you run locally on your computer via a web browser. It permits you to load ‘grid-shaped data’ for examination, filtering and data cleanup. That makes is sound so much less exciting than it is. The best way to get a sense of what you can do is to watch the Gridworks Videos.

What sort of data do I think there is in archives to be pumped into Gridworks? How about collection descriptive data and electronic record datasets? Since all the data is kept locally, you don’t need to worry about uploading your data to some anonymous server in order to work with it. It all stays safely on your local computer the whole time.

A quick list of things that Gridworks can do:

• Cluster data to find values that are almost the same so you can normalize your data (for example – NYC vs N.Y.C.)
• Create instant facetted browsing based on any column in your data
• Provide scatterplots of the values from any two numeric columns as well as a way to spot the most interesting combinations across many possible columns
• Reconcilliation and validation of values based on data from within Freebase.com
• Pull data from Freebase.com based on a matched column – such as the population of a country, if you have a column in your dataset with country specified
• Splitting data within a cell based on a specified delimiter
• Application of regular expressions and other simple code to data to create new columns

This list just scratches the surface, but it should give you a decent idea of the power of Gridworks. Even if the only feature you ever use is the one which lets you cluster and update your data to remove the ‘almost the same’ values, Gridworks can save you hours of painstaking data cleanup.

Why is data cleanup exciting? Because once you have nice clean data with all the attributes that are usefull to have for your data set – then you can start playing with the data in visualization tools! So go watch some Gridworks Videos, get Gridworks for yourself and start playing with data. It is free and it makes working with data fun!

This post is from from: Spellbound Blog.

Gridworks: Super Data Cleanup and Exploration Tool

Introduction to Metadata Standards

Michael Bolam, metadata librarian for digital production, is in charge of all the metadata for all the collections at the Digital Research Library at the University of Pittsburgh. He is not an archivist – but does know where the archives is at Pitt! He has put lots of archival material online through digitization and assignment of metadata.

The best definition he has found of metadata, good for all audiences: “Metadata consists of statements we make about resources to help us find, identify, use, manage, evaluate and preserve them” Marty Kurth – Head of Metadata Services, Cornell University Libraries

Reviewed examples of metadata for images, text documents and archival collections. There is also data related to the business of scanning and making content available – administrative/behind the scene. Standards let you take your data and use it for other purposes.

Overview of alphabet soup of metadata standards:

• MARC: bibliographic information in machine-readable form (a MAchine-Readable Cataloging record).
• Dublin Core: the goal of Dublin Core was to create a core set of metadata fields that could be used across platforms, across various disciplines.
• MARCXML: schema for representing MARC in XML. Makes it easy to convert to and from MARC without loosing any data. May have more data than you need. MARCXML is not very ‘human readable’. You need to recall all the code numbers for the different data elements. Can be exported from Archivist Toolkit.
• MODS: Metadata Object Description Schema – sort of a ‘MARCXML light’. Tries to be a step between MARCXML (robust & complicated) and Dublin Core (really simple). May result in compacting multiple MARCXML fields into single MODS fields. May loose some of the granularity of the data. The tags ARE human readable. The tag is the word ‘author’ – not a number. Also can be exported in Archivists Toolkit.
• ONIX: ONline Information eXchange – standard used by the book publishing industry. XML-based standard for making available intellectual property in published form, both physical & digital. Data created by the publisher. They use different ways of representing authors, keywords..etc in comparison to LOC and library cataloging.
• METS: Metadata Encoding & Transmission Standard. XML standard wrapper for describing divergent types of content within a digital library. The metadata for books, images, collections etc keep this data in different formats – METS lets you bring them together.
• OAI-PMH: Not a metadata standard – but rather a protocol for sharing metadata. Gives us a way to pull baseline information about a digital object out of a database and put it out somewhere where it can be harvested and used.

Examples of projects built on shared metadata:

• Worldcat.org: Has everything that is shared with OCLC. They do expose their records to google and yahoo harvesting.
• OAIster: Searches a harvested data set – it is not going live out on the web. The OAIster records are also available in Worldcat. Example: search for Pittsburgh City Photographer (that is a provider of data). Most digitization software will generate an OAIster harvestable version. In his example we see that address and location get compressed into Notes. This is because there is not always a place in Dublin Core that maps to the level of detail you collect at your local institution. http://www.oclc.org/us/en/oaister/default.htm – has the info about contributing your content for crawling.
• Archive Grid: The goal is to pull in finding aids from many sources. It is a service – requires some sort of subscription and payment to see the data. Uses Lucene for searching. The content in Archive Grid is now available in Worldcat. To participate – see http://www.oclc.org/us/en/archivegrid/default.htm

Google and Yahoo do index OAIster and WorldCat, so that is one path to being found in search engines.

MARC Records for Archival Materials in WorldCat Local

Jennifer MacDonald from the University of Delaware presented a cataloger’s perspective of a WorldCat Local environment. She is a “concerned enthusiast” with regard to metadata. The University of Delaware was the first institution to buy WorldCat Local. She ended up on the WorldCat Local Special collections and Archives Task Force. The task force made their final report in 2008 and got a response from OCLC in 2009. They did get some immediate changes based on their feedback – like moving the 520 “summary” data element higher in the display. For some problems the task force identified, such as Archival Materials that were not being identified properly (Internet Resource is the type for all OAI records), it is hard to tell if the issue has been fixed.

She showed some screenshots from WorldCat local to show what data elements are there and how they are organized. In the FirstSearch screenshot (only available at the school), Notes and General Info holds a mishmash of content from various data elements consolidated into single fields. The task force asked for the “Browse” feature but apparently this feature is dead. They got no response from OCLC to this request in their report.

If you use the University of Delaware instance of WorldCat Local to search for walter penn shipley and drill down to the detail record display for the Walter Penn Shipley Papers you will see what was shown during the session. This display is customizable at the institution level in WorldCat Local. Some data is shown. You see lots of Web 2.0 options to add your own data, but the display is missing some of the data from the original MARC record. The full MARC record is indexed for keyword search, but since some of it is not displayed, users may not be able to determine why a record was returned.

Fields missing from the WorldCat Local display:

• 351 – Organization and Arrangement of Materials
• 545 – biographical note
• 506 – restrictions on access
• 540 – Use of materials – with link to an askspec page: http://www.lib.udel.edu/cgi-bin/askspec.cgi
• 525 – preferred citation form – and this is where the manuscript number is
• 655 – some of the parts of the genre terms are missing
• 656 – occupation

OCLC says that they have not included all this because people don’t want this displayed. Given that local organization is already deciding what to show, the task force would prefer the option to displayable all data elements. Due to this missing data, Jennifer prefers the FirstSearch interface – but this option is not always available at all institutions. You should take advantage of the Web 2.0 features. Archivist can create an account on WorldCat Local and add data elements.

Questions and Answers

QUESTION: You talk about having the metadta in a format that is accessible to harvesting. What I have is a bunch of CDs with images on them that have a folder and descriptor structure. Is there a metadata harvester that can go in and pull that metadata out? New York Stock Exchange photographer sent these.

ANSWER (Michael): So the metadata you are looking to extract is the filename and descriptors? You could have someone write a little script and extract what you need. I would hand it to the guy I work with because he writes perl. If then you made that available via your website – then people could find it. To get it into a database – it is just a small script.

QUESTION: Are there any specifically useful webinars/seminars for becoming familiar with these formats for skillbuilding?

ANSWER (Michael): Tons on the web. The LoC websites are very useful. You may have heard the term ‘crosswalking’ – that is where you take one format and turn it into another. Looking at the crosswalks can make it much easier to understand how a format you understand maps to one you are trying to learn about. Shareable Metadata – metadata for you and me. Not online yet – but someone in the audience said the plan is to post the materials. There have been a couple of books and ALA publications. Most of the ones I know of are about 10 years old. Jaime: SAA has a good workshop series.

QUESTION: One of the first things you said was to take data out of EAD and you didn’t go into detail in that. Were you talking about DAO tagged items?

ANSWER (Michael): I was just talking about reusing data in a new environment. For example, we just started digitizing manuscripts and each item is becoming an individual digital object. The only metadata we have is in the EAD finding aid – so we are using that data to make descriptive data about the digital objects. We are going to create a MODS or METS record for every digital object. Jaime: We use EAD to make MODS records. She has been manually extracting EAD data as Dublin Core data for ContentDM.

My QUESTION: What format does OAIster want?

ANSWER (Michael): OAIster is just harvesting Dublin Core. You can share MODS and other metadata types and you may find other aggregators that are expecting their users to work in a more detailed environment. You may publish more data elements for other harvesters as well – but OAIster will only pull the Dublin Core data elements.

QUESTION: We are working on a digitization project to digitize local historical societies, museums and libraries. Might the catalogers be able to deal with MODS or will the loss of granularity be a problem?

ANSWER (Michael): I am not a MODS expert. MARC is very granular. Maybe look at the MARCXML – MODS crosswalk?

QUESTION: At the University of Delaware, do you have any other systems?

ANSWER (Jennifer): When we first got WorldCat Local you had to know the URL to get to the library. That changed fast! The patrons couldn’t find anything. Jaime: In WorldCat Local you cannot scope the search to specific sub-collections.

QUESTION: Thank you Jennifer for your remarks. Is there a problem with catalogers trying to ‘sneak’ data elements into other places – are standards in danger?

ANSWER (Jennifer): I would hope we wouldn’t move 524 data into a 500 field just to get it displayed. There is some danger of loosing the granularity by pushing everything to Dublin Core. I don’t know how real that danger is at this point.

QUESTION: A political question for Jennifer: Who has the clout to push for changes with OCLC?

ANSWER (Jennifer): I think leaning encouraging users to give feedback is important. We were told that users don’t want that “we have proven that users don’t want that”. Users need to make comments about their challenges in dealing with the interface. FROM AUDIENCE: The strongest is to say that you are looking at Sky River. FROM AUDIENCE: Make your data more discoverable outside the catalog world – internal websites and Google. Jaime: We are working hard to make MARC records to push access to our collections. The push is to make the data available in as many locations as possible.

QUESTION: Are these all different levels of subscriptions? Are they trying to push people to buy more subscriptions?

ANSWER (Jennifer): There is a sense that WorldCat Local is pushed at local public libraries. Yes – WorldCat Local is something they have to pay for. Michael: With Archive Grid you are going a step further – EVERYTHING in the finding aid is indexed. Every search I did in there returned thousands of records. Then I filtered by institution – and it never loaded. FROM AUDIENCE: I think they are revamping Archive Grid – but I don’t know how far they are in the process. Michael: I love the detail – you don’t have to dig through other data to find something useful. Depending on the institution – and how they are allowing their data to be harvested – you may see less information. Jaime: You have to actively work with OCLC to get Archive Grid to pick up your data.

QUESTION: We are tinkering with users adding tags – are you having any success with people adding tags?

ANSWER (Jaime): No – it isn’t something we have dealt with. WorldCat Local does let you add stuff like that.

QUESTION: Will OCLC provide that UGC (user generated content) back to the institution?

ANSWER: We wouldn’t know.

QUESTION: Have they provided access to the user studies?

ANSWER: Yes – but it is based on watching individuals use the tools.

Image Credit: Statue representing Research by Henry Hering from image of the interior of the Field Museum of Natural History interior.

As is the case with all my session summaries from MARAC, please accept my apologies in advance for any cases in which I misquote, overly simplify or miss points altogether in the post above. These sessions move fast and my main goal is to capture the core of the ideas presented and exchanged. Feel free to contact me about corrections to my summary either via comments on this post or via my contact form.

This post is from from: Spellbound Blog.

MARAC Spring 2010: Hurray for Archival Metadata (Session S2)

I have spent lot of time thinking about how to generate thematic overviews of groups of archival collections. My information visualization project, ArchivesZ, aims to provide ways of understanding aggregated archival description data, both from a single institution or across institutional boundaries. Now I find myself wondering if text mining with a tool like MALLET might generate smart topic groupings more elegantly than fighting with the wide range of non-standardized collection subjects.

Topic Modeling with MALLET

To get a sense of what MALLET generates, see the excerpt below from Blevins’s post:

With some tinkering, MALLET generated a list of thirty topics comprised of twenty words each, which I then labeled with a descriptive title. Below is a quick sample of what the program “thinks” are some of the topics in the diary:

• MIDWIFERY: birth deld safe morn receivd calld left cleverly pm labour fine reward arivd infant expected recd shee born patient
• CHURCH: meeting attended afternoon reverend worship foren mr famely performd vers attend public supper st service lecture discoarst administred supt
• DEATH: day yesterday informd morn years death ye hear expired expird weak dead las past heard days drowned departed evinn
• GARDENING: gardin sett worked clear beens corn warm planted matters cucumbers gatherd potatoes plants ou sowd door squash wed seeds

He goes on to explain that “MALLET also allows us to track those topics across the text.” What if, instead of text mining a diary, we pumped the descriptions of every archival collection from a single institution into MALLET. Of course we would need a good list of stop words including such common terms as archives, history, sources and records. But I wonder how the topics MALLET suggests would compare to the official subjects associated with each collection? Could this give us a broad overview of the topics covered by a specific repository and give us a new way to build paths to the collections based on topic?

Auto-Classification Using Castanet

Text miner Aditi Muralidharan also posted recently on this theme in Castanet: automatically generating a browsing structure for a collection and explains:

Castanet automatically carves a sub-structure from the hierarchical concept dictionary, WordNet (http://wordnet.princeton.edu), and matches items in the collection to one or many appropriate places within that hierarchy. Then, after some automated trimming and flattening, the result is a hierarchical browsing system.

I have heard of Castanet before via the Flamenco Search Interface Project. Apparently Muralidharan did a project using Castanet last summer to create a category system for Flickr Commons images based on the images’ tags which is then rendered using a Flamenco interface. I include a partial screen-shot below to give you a taste of what the navigation of images feels like a few levels down in the hierarchy. I love the classification of ‘Group Action’ then filtered by a sub-classification of ‘Commerce’. The first images shown are of ‘horse trading’ – with additional headings and images beneath them as well as additional filter options on the left.

What If?

What if we pulled all the English language archival descriptions from around the world as our original data set. If we used this data for topic modeling, our subjects clusters would be cross-institutional. Maybe we could map the local institution assigned subjects to the topic model generated topics for each collection and get a sort of automated crosswalk for finding related collections. If we used the local institution assigned subjects from the archival descriptions for Canasta style auto-classification, maybe we could generate a way to hierarchically browse collections topically.

Both MALLET and Flamenco are open source (I am not sure of the status of Castanet) and, as I discovered working on ArchivesZ, many institutions will share their archival description data for a good cause. So – is this a good cause? I need to tease these ideas out a bit more, but what do you all think of it at first blush? Feasible? Interesting? Worthwhile experiments?

Image Credits: MALLET logo from MALLET homepage. Images in screen shot from Flickr Commons with no known copyright.

This post is from from: Spellbound Blog.

Topic Modeling, Auto-Classification and Archival Description

Overview: Review of 100 virtual museum websites and multiple flickr groups plus surveys of amateur website creators, memory institutions and Arts & Humanities academics leads to new perspective on digitization and creation of collections online by dedicated enthusiasts.

Session Highlights

Areas of “Amateur” endeavor  have a long history of launching collections, such as:

• cabinet of curiosities
• foundation of astronomical research
• british flora and amateur botanists
• weather observations
• open source software movement

Being an amateur doesn’t necessarily mean being bad at what you do!

Within the realm of self-defined museums some common topics often emerge:

• ephemera (advertising, packaging, nostalgia)
• comics
• technology – especially old tech, there is a surprising trend of being fascinated by technology approximately 10 years older than the collector
• personal and “embarrassing” collections
• genealogy

For these self-defined museums the scope is self-defined – these are self-delineated collections. Virtual museums can document aspects of cultural heritage considered socially taboo or in some way too sensitive to collect. A great example of this is the Museum of Menstruation which claims to have been created 14 years ago and is currently trying to establish a public permenant display for the public.

Platforms have evolved over the life of the web, starting with static html, then blogs and now Flickr images as a mode of presentation.

This is a list of successful amateur collections online:

• Today’s Inspiration – illustration from the 40′s and 50′s
• JonWilliamson.com – advertising 1940s-1960s
• Pulp Fiction Flickr Group – 882 members who provide basic metadata and often label stuff within the image – currently contains 3,385 items.
• Curio Cabinet Flickr Group – 1,206 members and 5,537 items

Visual Arts Data Service (VADS) is a more traditional site created by a cultural heritage institution. It contains 100,000+ images copyright cleared for use in teaching, learning and research in the UK. VADS is a very detailed static source of images with metadata, but provides no interaction.

Amateurs do provide metadata, but it is intuitive metadata. It might not fit into rigid buckets of data, but that doesn’t meant that the metadata available isn’t useful.

What are the boundaries between amateur and professional? Work vs hobby?

Many of these amateur sites get much more traffic than most standard museum sites. More than 50% of museum digitized images are never visited.

Memory institutions are starting to put things into the wider online community:

• Smithsonian: photos in Smithsonian Flickr Commons
• Tate: The How We Are Now project invited the public to contribute photos to the How We Are Flickr Group. The images were streamed to screens within the How We Are: Photographing Britain exhibit and 40 photos were chosen to be included as the last set of photos in the physical exhibit.
• Victoria & Albert Museum: created a Flicrk group of photos taken at the V&A museum along with a long list of other V&A Flickr groups and streams
• Oxford University’s Great War Archive: contains 6,500 items contributed by the public and related to the First World War.
• Facebook and Twitter are being used more often for informing the community about their collections

Much of amateur research has been driven by advances in technology. A great example of this is the advent of affordable metal detectors led to dramatic changes in archaeology. The internet and Web 2.0 technology are arming a whole new generation of enthusists who can find one another and collaborate more easily than might ever have been dreamed of 20 years ago.

Next Steps & Conclusions

Future research will involve looking at the psychology of collection: archives vs collections. For now it is important to realize that institutions are not the only hosts of “worthwhile” digital objects. Pro-am (aka, pro-amateur) are doing better with using web 2.0 & getting more traffic.

What can memory institutions learn from this?

• interact with user communities
• use the ‘grand central stations’ of flickr, twitter, facebook
• usability of flickr is better than what most memory institutions build for themselves

My Thoughts

This session considers the ways cultural memory institution can take advantage of the web by looking at what the successful enthusiasts are achieving. This research-backed approach confirms what I would have expected. Libraries, museums and archives are leaving a lot on the table when it comes to putting their collections online. Sites run by non-professionals are doing an amazing job of drawing in new audiences, keeping people around and then initiating conversation within that audience.

The Flickr Commons is a big step forward, but it isn’t the only option. There are also varying opinions about how successful the crowdsourcing aspect of the Flickr Commons is for memory institutions. A lot of this goes back to to a core question “how do we know if we have succeeded?”. There is much to be said for setting out clear goals when launching online initiatives. Is your goal increased traffic to your site or crowdsourcing of metadata? A great example of an initiative whose goal is clearly collection of crowdsourced metadata is the German Federal Archives who chose to use the Wikimedia Commons for their photo metadata initiative.

If you are trying to extend your mission of providing access to materials to the public, then how do you measure success? Putting your materials in what Melissa called “grand central stations” (or what I have also heard termed “public crosswalks”) definitely increases the chances of serendipitous discovery by new individuals. That said, we can see from the successful blogs mentioned above that tackling a niche with enthusiasm and consistent posting can go a long way to building a following. JonWilliamson.com seems to have only launched back in November of 2008 with a post featuring a Scotch Tape Christmas ad from 1951. The author posted in May of 2009 that his images in Flickr had surpassed 100,000 views.

To conclude this post I leave you with a list of inspirational digitized collections online that were created by various cultural heritage institutions:

• Publishers’ Bindings Online – discussed in SAA2007′s Session: Publishers’ Bindings Online – Digitization, Collaboration, Standardization and Community Building, a multi-institutional project that includes galleries of topical images combined with an essay that gives the images context. Two of my favorites are:
  • From Domestic Goddesses to Suffragists: The Story of Women Told on Bookbindings, 1820-1920
  • Indians, the Frontier, and the West in American Bookbindings
• Calisphere – more than 150,000 digitized items organized for easy use by K-12 teachers. This is especially interesting in that it represents items already available in Online Archive of California, but organized in a way to make them easy to find and use with their target audience in mind.
• Yiddish Books Online – A project by the National Yiddish Book Center that uses the Internet Archive as a platform to host 11,000 digitized out-of-print Yiddish books. This project is a nice cross between a branded custom site and a grand-central station

Have a favorite online collection website? Please share it in the comments below.

As is the case with all my session summaries from DH2009, please accept my apologies in advance for any cases in which I misquote, overly simplify or miss points altogether in the post above. These sessions move fast and my main goal is to capture the core of the ideas presented and exchanged. Feel free to contact me about corrections to my summary either via comments on this post or via my contact form.

Image credit: http://www.flickr.com/photos/mms0131/ / CC BY-NC-ND 2.0

This post is from from: Spellbound Blog.

DH2009: Digital Curiosities and Amateur Collections

Mark Shelstad, head of Archives and Special Collections at University of Texas at San Antonio, sent me a link to the TARO (Texas Archival Resources Online) page for UTSA’s Archives and Special Collections finding aids in XML format.

With the current scripts, these are the fun tag stats:

• 1,684 total tags extracted
• 75% (1,266 tags) are associated with only one finding aid
• 3% (51 tags) are associated with 10 or more finding aids

Collection Size

235 out of tne 253 collections ended up with a collection size of 0.

Consider the encoding of the collection size in the Guide to the Women’s Overseas Service League Records, 1910-2007:

<physdesc label="Extent:" encodinganalog="300$a">
    77 linear feet (approximately 44,000 items)
</physdesc>

Contrast this with one of the examples where the size of the collection was extracted properly by the current script:

<physdesc label="Extent:" encodinganalog="300$a">
    <extent>8.4 linear feet</extent>
    (14 boxes)
</physdesc>

Sometimes it feels like a game of Where’s Waldo. In this case we are simply missing the set of <extent> tags  from the first example. Off I went to the EAD tag descriptions to find the guidelines for use of the <physdesc> tag, where I found this overview of the tag:

A wrapper element for bundling information about the appearance or construction of the described materials, such as their dimensions, a count of their quantity or statement about the space they occupy, and terms describing their genre, form, or function, as well as any other aspects of their appearance, such as color, substance, style, and technique or method of creation. The information may be presented as plain text, or it may be divided into the <dimension>, <extent>, <genreform>, and <physfacet> subelements.

Bad news for my script logic – both versions are valid! This is a great example of how valid encoding can still present challenges. While in this example it seems just as easy to parse the version with the <extent> tags as without, it will only be through examination of a much broader sample of data that we can determine how much of a problem we have on our hands with this scenario of size data included in the <physdesc> tags without enclosing <extent> or <dimension> tags.

Inclusive Dates

Twenty of the UTSA collections came through with no years. When I examined the data, I found an assortment of <unitdate> formats that my current script could not parse properly, including the examples below:

• 1917-1980 (bulk 1920-1945)
• 1876-1903, 1914-1919, 1940-2002
• 1940s, 1970s-1990s

Another encoding approach that could not be parsed was the one used for the finding aid of the Church Women United of San Antonio Records. In this case the <unitdate> tag is within the <unittitle> tag as seen here:

<unittitle label="Title:" encodinganalog="245">
Church Women United of San Antonio Records,
<unitdate label="Dates:" encodinganalog="245$a">1961-2005</unitdate>
</unittitle>

Among the finding aids for which I did extract a range of inclusive date years, I also found issues with values like 1950s-1990s. The current script interpreted this to represent 1950 through 1990, but I believe it would be more properly translated as representing 1950 through 1999.

General Code Fixes

The University of Texas at San Antonio’s finding aids have provided additional examples of the following data and encoding issues already identified in earlier data sets:

• Inconsistent repository titles (26 different variations of “The University of Texas at San Antonio Library”)
• Titles with embedded and tagged dates
• Carriage return and tab characters that need to be removed
• Emphasis within a title or abstract added via a tag (such as <emph render=”italic”>Storyletters</emph> seen in A Guide to the Storyletters Records, 1991-2000) which interrupts extraction of text at that point

Next Steps

This is the last data set I am analyzing before tackling actual updates to the ArchivesZ data extraction script. My next step is to review and prioritize my long to do list for updates to this script. Most of what I have found in my examination of the data sets are ways in which my script was not smart enough to handle valid variations in encoding and the tabs, carriage returns, formatting tags and special characters found throughout everyone’s XML. Yes, there are some cases in which the data itself is less than optimal (such as non-standardized repository titles) or the values challenging (so many ways to describe the size of a collection!), but overall I am optimistic about how much more I can improve the extraction script before I have to resort to hand correcting records in the database.

Thanks to everyone for your patience with these data analysis posts. Onward to programming!

This post is from from: Spellbound Blog.

ArchivesZ Data Challenges: University of Texas at San Antonio

• Titles with embedded tags or punctuation. Generally the script drops anything after it hits either, so rather than a title like William E. Towell Papers, 1941 – 1988, my database ended up only with “William E Towell Papers,” based on this encoding:  <titleproper>Inventory of the William E. Towell Papers, <date normal=”1941/1988″>1941 – 1988</date></titleproper>
• Need to handle a conversion factor for  a size of  “1 folder” (as found in the Inventory of the Biltmore Forest School Images, 1890 – 1988)
• My script chokes on the Inclusive Year format “1910 and 1931 – 1937″ (as found in the Inventory of the Alfred Cunningham Papers, 1910 and 1931 – 1937)
• The presence of a <lb/> character within the <extent> tag, used to force a line break, is preventing my script from extracting any size information at all (as found in the Inventory of the DeWitt Nelson Papers, 1940 – 1976)
• Within the <abstract> tag, my script drops everything after an <emph render=”doublequote”> tag (making for a very short abstract in the case of the Inventory of the Arthur Bernard Recknagel Auxiliary Photograph Collection, 1911 – 1947).

The most dramatic issue, seen across all the finding aids in this set, is that no subject data was extracted from any of the finding aids. My working theory for the moment is that this is due to the use of <list> and <item> tags as shown here:

<controlaccess>
<head>Subject Headings</head>
<list type="simple">
<item><genreform source="lcnaf" encodinganalog="655">Audiotapes</genreform></item>
<item><persname source="lcnaf" encodinganalog="600">Ainsworth, John H., 1909-</persname></item>
<item><subject source="lcnaf" encodinganalog="650">Businessmen -- United States</subject></item>

This is in contrast with this example of encoding from Syracuse University:

<controlaccess>
<head>Subject and Genre Headings</head>
<subject encodinganalog="650" source="local">Adult education</subject>
<persname encodinganalog="600" source="lcnaf">Adolphson, L. H.</persname>
<persname encodinganalog="600" source="lcnaf">Bradford, Leland Powers, 1905-</persname>

Or this sample from Oregon State University:

<controlaccess id="a12">
	 <controlaccess>
		  <persname encodinganalog="600" source="local" rules="aacr2"
		  role="subject">Aitken, Frances Alva, 1889-1970.</persname>
	 </controlaccess>
	 <controlaccess>
		  <corpname encodinganalog="610" source="local" role="subject"
		  rules="aacr2">Oregon Agricultural College. Class of 1910.</corpname>
		  <corpname source="lcnaf" encodinganalog="610" role="subject">Oregon
				Agricultural College--Students.</corpname>
	 </controlaccess>
	 <controlaccess>
		  <geogname source="lcsh" role="subject" encodinganalog="651">Corvallis
				(Or.)</geogname>
	 </controlaccess>
	 <controlaccess>
		  <subject encodinganalog="650" source="lcsh">Student
				activities--Oregon--Corvallis.</subject>
	 </controlaccess>

Both the Syracuse and OSU examples are handled by the current state of the data extract script.

Amanda pointed me to the NCEAD Best Practice Guidelines for EAD 2002. Down in Appendex G: How Do I Encode…, the second question down is “What if I have multi-part scope notes, biographical notes or subject headings?” followed by exactly the <list> and <item> tag usage as is being done for the Forest History Society finding aids. This format clearly should be handled.

So, no fun tag stats for this run – but I hope to fix my ruby script so that the Forest History Society finding aids can be incorporated into the data set I use for testing version 2 of ArchivesZ. My ruby script to do list is getting quite long!

This post is from from: Spellbound Blog.

ArchivesZ Data Challenges: Forest History Society

I really want a t-shirt with the Bit-Rot guy on it!

This post is from from: Spellbound Blog.

Another Thrilling Digital Adventure With Team Digital Preservation

These finding aids have helped me identify three types of ArchivesZ data challenges:

• strange characters
• broad composite subjects
• determination of accurate collection size

Strange and mysterious characters!

These finding aids use a special character in the place of the standard Library of Congress double dash which normally appears between subsections of the subject heading.

An example subject from the Utah Government XML looks like this:

Women—Suffrage—Utah.

Viewing the same subject in a pure text editor (such as vi):

Women&#8212;Suffrage&#8212;Utah.

By the time it gets into my database and is pulled out via a query in MySQL Query Browser it looks like this:

Women—Suffrage—Utah.

Rather than just stripping out all instances of &#8212;,  my plan is to replace them with the standard Library of Congress double dash. This will ensure that the existing code that breaks the subjects down to tags will still work.

Composite Subjects

When I say “composite subject” what I mean is a subject that includes multiple very disparate terms. Rather than the Library of Congress style subjects, all aspects of which relate to the collection in question, these composite subjects cover multiple subjects which are grouped together for convenience.

This is a list of some of the most popular subjects for the Utah Gov collections:

• Politics, Government, and Law
• Business, Industry, Labor, and Commerce
• Science, Technology, and Health
• Arts, Humanities, and Social Sciences

These subjects throw a monkey wrench into my theories about decomposing subjects based on commas. The collections to which these subjects are assigned likely fit in only one of the component themes. For example, the “Inventory of Publications from Department of Technology Services, 1993-2008″ is assigned the subject “Science, Technology, and Health”. If I divide this subject into 3 separate tags, the Science and Health tags would be quite misleading.

So that leaves me a bit trapped. If I want to divide subjects such as “Art, Cuban, 20th century”, as I discuss in my Syracuse University post, then I end up also dividing these umbrella subjects which separate such very divergent terms with commas.

This issue goes on my list of reasons to add a repository configuration file for use by the data extraction script.

Accurate Collection Size

In my quest to convert all sizes to linear feet – sizes such as these are challenging:

• 0.20 cubic foot and 1 microfilm reel
• 0.35 cubic foot and 2 microfilm reels

I also have situations of sizes be specified in multiple sections of the finding aid. The Inventory of ALERT Foundation records from Governor Bangerter, 1986-1991 has a collection level size of “0.50 cubic foot and 2 microfilm reels”, but further down in this finding aid I see this:

series: ALERT Foundation records

• box 1, folder 1: Documentary: “”Letters from our Children,”" Motion picture film reel, 16mm
• box 1, folder 2: Documentary: “”Letters from our Children,”" VHS videocassette
• box 1, folder 3: Documentary: “”Letters from our Children,”" VHS videocassette
• box 1, folder 4: Documentary: “”Letters from our Children,”" VHS videocassette

When they said 2 microfilm reels – do they really mean a 16mm motion picture film reel and a VHS videocassette? Is there 1 VHS videocassette or 3? How sizes are specified in a specific repository’s finding aids is another possible candidate for a repository level configuration script.

Tagging Statistics

Finally, here are a few tag stats:

• Only 31 tags (1.5% of all Utah Government tags) are associated with 10 or more collections
• 1404 tags (71.5%) are assigned to only a single collection
• 107 collections have been assigned only 1 tag
• 10 collections have no subjects

Of course these statistics are based on the current incarnation of the data extraction script. After I modify the script, there will be a greater number of tags and (hopefully) more overlap of tags across multiple collections. These types of statistics should help me gauge how well my data extraction logic is working.

This post is from from: Spellbound Blog.

ArchivesZ Data Challenges: Utah Government Archives & Records Service

• University Archives
  
• Public Policy Papers
• Manuscript Division
  
• Latin American Ephemera Collection
• Engineering Library
  

So onward to the data issues and what they mean for my ever growing ‘script fix to-do list’.

Repository Names

As we saw with the Oregon State University finding aids, the finding aids from Princeton University had a wide range of different values for repository names. In the list below we spot some issues. Some end in periods, some do not. One has extra space (probably a carriage return) in the middle. One does not include Princeton in the repository name. Once we have many repositories’ finding aids in ArchivesZ, a repository name of ‘Engineering Library’ does not tell the user enough about where those collections can be found.

Here is the list of repository titles my script extracted:

• Princeton University Library. Department of Rare Books and Special Collections.
• Engineering Library
• Princeton University Library
• Princeton University Library. Department of Rare                    Books and Special Collections.
• Princeton University Library.

My script can handle the extra period and the extra spaces, but the non-specific name would need to ultimately be fixed on the source side.

Collection Size

The current script assumes that there is only one extent value specified to express the size of the collection. Princeton’s finding aids showed me examples of multiple extent values. For example, the Christina Georgina Rossetti Collection has both a collection level size of 0.4 linear feet (1 archival box) as well as a 2nd extent specification corresponding to a specific folder with the value of (1 poem, 3 drawings, 1 photo, 1 incomplete article). The script must be modified to only consider the collection level size.

Complicated Titles

The current script logic apparently does not handle what I would call ‘complicated collection titles’. For example, I ended up with “Edward Livingston Papers, ” as the title for a collection with a full title of Edward Livingston Papers, 1683-1877 (bulk 1764-1836). This is the way that this title is encoded:
<unittitle encodinganalog="245$a" label="Title and dates: ">Edward Livingston Papers, <unitdate encodinganalog="245$f" normal="1683/1877" type="inclusive">1683-1877</unitdate> (bulk <unitdate encodinganalog="245$g" normal="1764/1836" type="bulk">1764-1836</unitdate>)</unittitle>

This post is from from: Spellbound Blog.

ArchivesZ Data Challenges: Princeton University