If my panel is selected, I intend this session to give me the chance to review all of the following:

1. What are the special design requirements of archival records?
2. What are the biggest challenges to publishing archival records online?
3. How can archivists, designers and developers collaborate to build successful web sites?
4. Why is metadata important?
5. How can search engine optimization (SEO) inform the design process?

All of this ties into what I have been pondering, writing about and researching for the past few years related to getting archival records online. So many people are doing such amazing work in this space. I want to show off the best of the best and give attendees some takeaways to help them build websites that make it easy to see the context of anything they find in their search.

While archival records have a very particular dependence on the effective communication of context – I also think that this is a lesson that can improve interface design across the board. These are issues that UI and IA folks are always going to be worrying about. SXSW is such a great opportunity for cross pollination. Conferences outside the normal archives, records management and library conference circuit give us a chance to bring fresh eyes and attention to the work being done in our corner of the world.

If you like the idea of this session, please take a few minutes to go sign up at the SXSW Panel Picker and give Archival Records Online: Context is King a thumbs up. You don’t need to be planning to attend in order to cast your vote, though after you start reading through all the great panel ideas you might change your mind!

This post is from from: Spellbound Blog.

SXSW Panel Proposal – Archival Records Online: Context is King

The Stephen Wirtz Gallery has a number of images from the 2004 exhibition available online and features this great summary of the original project:

Sultan and Mandel created the series Evidence with documentary photographs mined from image banks of government institutions, corporations, scientific research facilities, and police departments. An NEA grant gave the artists a persuasive edge in gaining access these resources, and images were selected for their mysterious and perplexing subject matter. The series was presented in an exhibition at the San Francisco Museum of Modern Art in 1977, and simultaneously collected in the book Evidence, which is recognized among the most important publications in the history of photography. Removed from their original contexts and repositioned without references to their sources, these images challenged the viewer to examine the conceptual concern of identifying meaning and authorship in the creation and consideration of the art photograph.

I used WorldCat to find the closest copy of Evidence and happily found a copy of the 1977 imprint at the Art Library at the University of Maryland, College Park. It had been a long time since I had looked at photographs on paper and bound in a book rather than on a computer monitor. I love the idea of re-purposing of archival image – but I was also fascinated to realize that the word ‘archive’ does not appear anywhere in the publication. Even the description above mentions ‘image banks’, not ‘archives’.

The organizations thanked at the start of the book included major corporations, U.S. federal agencies and a long list of highway, fire and police departments. Sultan and Mandel seemed to focus their research efforts in California and Washington, DC – perhaps due to a need to limit their travel. While today one would likely still need to travel to many archives to find images like those used in Evidence, there are so many images available online (at least for preview). How would someone approach a project like this now?

It is so easy to create a slide show or website featuring images from repositories from around the world. Even the images that have not been digitized have a decent chance of at least being mentioned in an online finding aid. The recently introduced Flickr Galleries make it easy to select up to 18 images from across Flickr – like my November Flickr Commons Photos of the Month Gallery. Also, much of the online culture of reuse encourages giving proper attribution for materials.

Part of Evidence’s power is the extraction of the images from their original context and their unexplained juxtaposition with one another. Finding and harvesting an image online would make it much harder to entirely strip that context away to leave the raw image behind. I can imagine a web-wide hunt for an image’s origin. While that might be fun (maybe an archives answer to the DARPA Network Challlenge?), it would not be the same as a sleek hardback book with 59 stark, unlabeled, black-and-white photos that sits on the shelf of an art library.

I find it poetic that Evidence’s photos are a perfect example of a ‘secondary value’ of archival records, even though the images were literally evidential records necessary for the carrying out of daily business. That said, I don’t believe that ‘possibly useful to future artists’ is a typical reason given for retaining and preserving archival records. We are just lucky that artists have been (and will almost certainly continue to be) innovative in their hunt for inspiration.

If you have the opportunity, I encourage you to sit quiety with a copy of Evidence. The images include landscapes, explosions, deep pits, plants, rocks, people, planes, machinery, wires and a car on fire. My laundry list of contents cannot begin to do the images justice – but I hope that they might wet your appetite.

This combination of gallery exhibition and book has inspired me to wonder about other similar projects that specifically leverage archival images for artistic purposes. Please list any that you are aware of in the comments (be they in gallery exhibitions or published volumes).

This post is from from: Spellbound Blog.

Archival Photographs as Art: A Part of Larry Sultan’s Legacy

I have spent much of the past year learning the details of Search Engine Optimization. Usually shortened to SEO, this simply refers to the use of techniques which improve the traffic sent to a website via organic search. Want your webpage to show up at the top of the list for a specific search in Google? You want to work on your SEO.

So when I look at new archives website, I can’t help but keep an eye open for how well the site is optimized for search engines.

I hope that UMBC will forgive me for nitpicking their new site. A lot of their choices are great for SEO,  but they also have room for improvement.

Things Done Well for SEO

• Home Page Title & Description: The site’s home page has a good meta description. This is the text displayed below the link on a search results page – as shown below:
• Unique Page Titles At Collection Level: Each photography collection homepage has a unique page title and a nice block of explanatory text. Google can only read words – so the more unique text on a page, the better the job Google can do in figuring out what your page is about. Example: Ardsley Park Album
• Good anchor text: (also known as link text) The words used in anchor text tells search engines information about the destination page. For example, the blue text below is anchor text.

Areas for SEO Improvement

• Unique Page Titles At Item Level: Individual images and documents all use a generic page title such as ‘UMBC | Digital Archive | Document Viewer’. Document Example: Accidental Death of an Anarchist Image Example: 10 year old Bootblack
• H1 Tags: In the HTML of each page, the dominant heading of the page should use the <h1> tag. This helps Google know the phrase you are targeting with this page. It is your 2nd best place to emphasize your content after the page title. In the case of the item pages, there seems to often be a headline type title at the top of the page – but it currently is not an demarcated with an <h1> tag.
• Think About Search Results and Indexing: Pages displaying results of internal searches on your site are not likely to be useful as indexed pages in Google. The thinking here is that they can dilute the focus on the item and collection level pages on your site if Google also has many search results pages in the index. If UMBC wanted their search pages to be indexed, then those pages’ URLs should be simplified and the search results pages need a page title that somehow includes the search criteria. There are two ways that I know of to disable this indexing – blocking via the site’s robots.txt file or via a robots meta tag in the header of the search results page. Both of these methods tell obliging search engines to not crawl certain parts of your site.

Final Thoughts

There are plenty of other things that UMBC could do to support this new website. They could create an XML sitemap of all their pages and submit it to Google (maybe they already have). They might re-title some of their pages based on using a tool like Google Insight to see what variations of a phrase is searched on most frequently. My goal here was to give you a taste of the sorts of things that catch my eye. Also, SEO is still more of an art than a science – so you will sometimes notice that what one SEO expert recommends is the opposite of what the next expert would tell you.

In many cases changes, such as the Unique Page Title at the Item Level mentioned above, may not even be possible due to software or programmer resource limitations. The trick is to take advantage of every option that is available. There are also trade-offs to be made. UMBC’s site provides some very slick interfaces for viewing the details of a group of documents, such as theater programs and other materials related to a theatrical production. The imlementation elegantly handles the situation of multiple scanned images which relate to a coherent set of documents. Sometimes you can’t have both your innovative UI and perfect SEO. Then it gets down to what your goals are for your website. Are you trying to make a specific community of existing users happy by providing them with tools they can use? Or does your mission focus more on reaching out to a broader audience?

There is no silver bullet to search engine optimization. It just takes knowledge of the available tools and techniques combined with a willingness to keep learning and experimenting. Like the ‘Do-It-Yourself-Woman‘ pictured above in the Nationaal Archief‘s photo I found out on the Flickr Commons, you too can learn the basics and do-it-yourself. A great starting point is Google’s free SEO Guide. Also, please remember that the best time to plan your SEO strategy is before you have built your site in the first place!

I would love to do research on how much progress archives websites can make in their organic search traffic after SEO improvements. My thinking is to take a snapshot of a month of analytics (the statistics that tell you how many people are visiting your website) and then apply some SEO inspired changes. After a suitable delay (it takes some time for SEO to do its job) we consider another month of analytics to determine any change in organic traffic.

Do you want me to do a quick review of your archives website to see if there is room for SEO improvement? Please contact me or add a comment to this post. I feel like there is a conference presentation in all this if we can find a good set of websites to optimize.

Finally, thank you to unsuspecting UMBC – your new website really is beautiful.

Image credit: Doe-het-zelf vrouw /Do-it-yourself-woman from Nationaal Archief on Flickr Commons.

This post is from from: Spellbound Blog.

SEO Evaluation of an Archival Website: Looking at UMBC’s Digital Collections

The first oral history project that combines old family pictures with the stories that go with them, Centropa has interviewed more than 1,350 elderly Jews living in Central and Eastern Europe, the former Soviet Union, and the Sephardic communities of Greece, Turkey and the Balkans. With a database of 25,000 digitized images, we are bringing Jewish history to life in ways never done before.

Their fleet of 140 individuals conducted extensive oral interviews and digitized thousands of old family photos. They are quite intent on clarifying that they do not create videos during their sessions with their interviewees. Instead, they record audio of their multi-hour sessions, transcribe these sessions and combine them with the digitized family photos to create their movies.

The juicy center of their website is found in the Centropa Movies which are alternately billed as a “library of rescued memories” and a “digital bridge back to a world destroyed”.  Their movies are also available via iTunes and on the CentropaOffice YouTube Channel. The movie I have included below tells the story of Judit Kinszki and focuses on her father Imre Kinszki, a budding photographer from Budapest, Hungary. From this movie’s Centropa Movie page you can also navigate to Judit Kinszki’s biography , the full family photo album and a study guide for this movie.

The amount of detail provided with each posted interview is really incredible. Biographies, detailed notes on each photo, the study guide, a family tree and a currently grayed out but promising link to “Discuss Movie”. This site has clearly given great thought to how to support teachers and has followed that vision through in the form of tons of supporting materials. Centropa has chosen the path of quality over quantity with the 17 movies currently posted.

Upon further reflection, I realize now that the movies are an outgrowth of the database of photographs and biographies. The detail was not added to support the videos – but rather the videos are the next step of evolution beyond the photos and interview transcripts.

In addition to the movies they offer a Recipe Archive, downloadable eBook versions of some of their interviews as well as Centropa Student, aimed at high schools in Europe, North America, and Israel. For those of you working on your own oral history projects, there is the Centropa Oral History Tool Kit, available in 5 languages. The Centropa Glossaries are less glossary and more a detailed list of people, social groups, events and terms that can be searched by country, type or keyword. Finally, don’t miss the ‘Narrated Stories and Introductions’ featured on the right sidebar on the Centropa Movies page, such as Maps, Central Europe and History or the Introduction to Centropa for US Students.

Reading Centropa’s claim that they are the first to combine the use of family photos and oral histories made me recall the University of Alaska Fairbank’s Project Jukebox. This project launched back in 1988 and aims to ” integrate oral history recordings with associated photographs, maps, and text.” The original was written using Hypercard!

They have a map showing all the communities in Alaska currently included as part of the project. A good example of an individual photo with accompanying narration is Harry Cook in his Garden from the Kiana Village History Project. No – it isn’t as elegantly assembled as the Centropa Movies, but the intention is much the same. They use old photos as a catalyst for helping individuals being interviewed and then combine the audio and images to improve end users’ understanding of the context of individual photos.

I have signed up with Centropa to be notified when they launch the promised ‘Add Your Family Photos’ feature. Until then I will keep scanning my own family’s photos, such as the one below featuring my grandfather (back row on the right), and working my way through all the Centropa Movies and their supporting materials.

This post is from from: Spellbound Blog.

Preserving Jewish Memory: Family Photos Join Oral History in Centropa Movies

The official title of Session 102 was We’re Not the Destination, We’re the Journey: Revealing Archival Collections at the Web’s Surface. If you attended this session or don’t want to read through the details, you can skip to the end and just read my thoughts on this session.

California Digital Library

The first presentation was by Lena Zentall of the California Digital Library (CDL). I believe it was titled something like “Untitled <snappy name here>”. CDL is increasing visibility of primary sources by targeting primary sources to specific audiences. Lena described how they view the URL as a line to reel in new audiences. She started with an overview of how archival content traditionally makes its way online.

Start with a box -> described by finding aids -> digital copies of finding aids put on line and cherry picked individual items are digitized to be featured online.

Two Audiences, Two Sites

CDL has taken a new approach. They have two sites for two very different audiences:

• Online Archive of California (OAC): presents both finding aids and digitized primary sources and targets archivists, historians & researchers
• Calisphere – only takes primary sources (for now) and targets k-12 teachers, lifelong learners, and undergraduates

Collections can have home in several places. For example, the items about the Chinese in California can be found in:

• OAC: Guide to the Chinese in California Virtual Collection
• Calisphere: As a subset of the California Cultures: Asian Americans collection, including the Chinese Exclusion Act
• Library of Congress American Memory: The Chinese in California, 1850-1925

Calisphere has created themed collections to highlight superstar digital objects. They pull images out of the finding aids and rearrange them for the target audience. These images are hand picked and associated with an essay. They pick striking objects with good metadata. This is what their audience wants – the teachers asked for it. Another example themed collection is the Goldrush Murder & Mayhem collection which includes this photo of the “old time San Francisco pickpocket” Jennie Hastings.

Hidden Gems: Untitled and No Metadata

The next part of the presentation discussed what happens to items that are untitled and associated with no metadata. Lena showed us the results when you searched the OAC images for for untitled. I found 12,315 items when I did this search. They really only live in the context of the finding aid. Of course the challenge is that people use words to find images. These hidden gems can be helped by inheriting the metadata of their parent container (such as collection level information) when there is nothing else.

3 Approaches

1. Digitize and release content to the web: low effort (after infrastructure is set up), very high return on investment. Over 40% of Calisphere traffic generated by google searches… but when users follow the link from google then they find the rich context.
2. Align with other aggregators: – low/medium effort, medium return. Calipshere content is also being pulled into aggregators. They can also pull back new data that is added by 3rd party partners – such as reading level added on a teacher site. These are three examples of Murder and Mayhem content in three different partner sites:
   • CLRN: Murder and Mayhem – (California Learning Resources Network)
   • EdZone: Murder & Mayhem
   • OER: Murder and Mayhem (Open Educational Resources) lets users add tags and search by keywords
3. Cherry-picking the best items: high effort, promising returns – but it is also harder to measure the returns

Finding New Audiences and New Volunteers

The next step is to reach beyond standard cultural and education venues and move into different ares of the internet. For example, the CDL added links to Wikipedia. The perception of those involved with this effort was that it was a very convoluted process with lots of mysterious rules. They were unsure if the links would remain in place. It sometimes seemed like a lot of work when the links might just be removed. They added 33 links and found 53 links made by others not affiliated with the CDL. On the plus side, links like this puts the digital objects in a very specific context. Traffic initiated from these Wikipedia entries is almost certainly individuals seeking detailed information in the specific topic they are researching.

The next frontier involves blogs. CDL digital items are now featured in blogs, but soon CDL will be creating a blog for Calisphere to tell the story behind individual pictures. The final stop for this talk was an inspirational blog: Mustaches of the Nineteenth Century. This blog was presented as a way to achieve the fame that primary sources dream about.

Library of Congress

The second presentation, by Helena Zinkham from the Library of Congress Prints and Photographs division, was titled “The New Friends for Old Photos – putting pictures in your path with the Flickr commons and Web 2.0″. This talk focused on the pilot project of putting Library of Congress photos on Flickr in the new Flickr Commons.

People who want photos don’t think of libraries or archives. They go to museums and stock photo agencies. Helena wants to help people realize that archives are a great source of images.

There has been increasing progress with hidden collections. Lots of digitization and work with metadata has been done to help items make their way online. But this begs the question of whether we are just creating new hidden collections in corners of the Internet that the average person will never come in contact with. Collections like ArchiveGrid, DLF Aquifer, and OAC. The descriptions need to get out of the catalogs – most people find content on the web.. we need to put the images on the web in the path of the users.

The Flickr commons satisfied Helena’s desire to pull people in from Flickr back to discover the catalog world of archives. Flickr can be considered a virtual reading room and platform for a virtual volunteer corp. Helena showed the example of the image Weavers at Work. The comments on this photo included:

• information that photo is of blind women weaving rugs
• the photographer’s great grandchild identified the photographer as Percy Byron
• the start of a discussion about what the cabinet or instrument might be shown to the far right of the photo

These commenters are new friends worth making!

Pros of Web 2.0

• make collection available
• gain information about collections – participatory description
• increase the visibility of specific photos
• win support for cultural heritage organizations

Risks of Web 2.0

• disrespect for collections (smart aleck chat)
• loss of meaning
• reduce revenue from photo sales
• excludes undigitized collections
• higher costs (more money and time)
• less chance for us to have fun as history detectives – other people are doing ‘our’ work

read powerhouse museums’ 3 month report about their experience. … Helena will post info about the nuts and bolts on the SAA site, but she also directed the audience to Powerhouse Museum’s Commons on Flickr First 3 Months Report.

Flickr Basics

Helena asked the session attendees who was familiar with flicker? Most of the room raised their hands. Who has accounts? Still good number. Who is adding archival content? A sprinkling of hands were raised.

Helena then explores Flickr basics and showed off the following neat search examples:

• A search for germany schaefer in Google finds Flickr photos (as well as Flickr photo comments). The LOC Germany Schaefer photo was returned 4th on my list when I did the search when writing up this post.
• A search for houston house search within Flickr co-mingles old and current photos

Logistics and Statistics

The LOC liked Flickr and felt it was a good fit because photographs are the main focus of the site. They did need one big change. Because LOC is not the owner or photographer (unlike most photo contributors), they needed a way to express that clearly. Flickr responded by creating The Commons. They also created a new rights statement of ‘no known copyright restrictions’ for members of The Commons to use. This is different from public domain. Flickr also appears (based on my hunt through the links) to permit each institutions in The Commons to link to their own explanation about what they mean by ‘no known copyright restrictions’. LOC deep links to a specific section of their Copyright and Other Restrictions page for Prints & Photographs. George Eastman House has a special George Eastman House & The Commons on Flickr page about copyright, as does the Brooklyn Museum.

Statistics from the first 6 moths on Flickr:

• 3,500 LOC photos posted
• 8 million views
• 30,000 favorites for 80% of the photos
• 14,000 Flickr members made LOC a contact
• 5,000 comments (3,300 people)
• 12,500 unique tags (59,000 total)
• 500 catalog records updated – Helena indicated that this could be considered a new kind of backlog, “but a backlog you can come to like”
• 20% increased traffic to p&p online catalog

There are 30,000 more photos from Bain News Service on the way, but they are only adding fifty photos a week. This number was recommended by Flickr as the largest they would want to push at any one time. This goes back to the tolerance of people who have Flickr in their friend photo stream. Fifty photos is about as many as people want to get at any one time. More than that and you increase the likelihood that people would remove you from their stream instead of be overwhelmed. They would have no chance to really look at more than that.

Contributors to The Commons can choose which features to enable. For example, the Portrait of Hine as small child standing by drum shows how george eastman house chooses to send people back to their institution for prints.

How much does it cost?

• a Flickr pro account costs $24.95 a year
• digitization costs
• time: daily moderation on the account – LOC checks every day for uncivil discourse which takes about 10 minutes
• 15-20 hours a week to pull data from comments to update metadata

Flickr Comments

One of the greatest parts of this presentation was the examination of ways in which flicker users contributed through comments. Here are some examples:

• Auto Polo: – comment includes link to an auto polo thread on the Jalopy Journal’s message board which includes newspaper images and an extended discussion.
• Sylvia Sweets Tea Room – includes a very extensive history of the business added by the daughter of the original proprietor
• Negro boy near Cincinnati, Ohio – the comments include a deep conversation about the title of the photo and the context of this title at the time it was taken (1942 or 1943).
• Jones Barn where dynamite was found – Flickr members found the context and news article to go with this photo
• Al Palzer – this photo’s original title was Al Palser – but the misspelling was pointed out in the comments. The comments also include a response from the LOC noting that the boxer’s name would be updated in the original catalog record.

Other Promotion Approaches

The Library of Congress has now started linking out from the LOC catalog entries to the Flickr image so that it is easy for users to discover any conversations associated with the Flickr version. Powerhouse museum has a Photo of the Day blog to highlight images from their collection. The Brooklyn Museum encourages people to upload photos of things happening in Brooklyn. Then and now photos can be taken – in this case see factory buildings in Lowell, Massachusetts in December 1940/January 1941 and then again in January of 2008.

The key to 2.0 is frequent, new content and interaction from archival staff. Helena is open to new ideas about how to use Flickr and closed with saying that Web 2.0 is right in our path.

Questions and Answers

Question: What is their view of the accuracy/inaccuracy user generated tags and comments?

Answer: Study done in the past comparing accuracy of official cataloging to comments – even if people make mistakes, but others will correct them.. LOC has a ‘hands off’ policy to not delete/change stuff unless it is defamatory or spam. Only 3 instances of this so far. LOC is citing the source as ‘Flickr commons’ and also include commenters’ sources – which are actually a lot more varied than you might expect (like the Jalopy Journal).

Question: Are you worried about an increase demand in staff time as you add more photos?

Answer: Yes.. there will be an increase in demand.. but the Flickr comments are there and since LOC is adding links back out to those records they are available for researchers even if they are not added to the original catalog record. Maybe they need more staff? depends on goals. Could work with expert teams and look for ‘formal trusted’ volunteers. A great example was the baseball history association who took photos and contributed expert information in a spreadsheet (if I heard correctly they gave LOC a spreadsheet identifying team, game, date and opponent for more than 3000 photos).

Question: Isn’t the link from the LOC catalog record to Flickr enough? Why update the LOC catalog records at all?

Answer: They are really only updating when it is a mistake (like Palser’s name mistake). Flickr also provides APIs and LOC pulls all the comments and tags into external database so that LOC can choose how to use the information over time.

Question: What are your thoughts and concerns about the longevity of Flickr as a platform?

Answer: What grows fast can die fast. Their perspective: Flickr is a copy.. and LOC has an extract of all the tags and comments – nothing lost if it disappears.

Question: Calipshere: how do they work with teachers to learn their needs and their satisfaction with the work that is done?

Answer: They hired Berkeley experts to talk to teachers about what they wanted. They used interviews and created personas to capture the audience needs. Targeting the K-12 audience was aimed at being a success by being clear about their audience. Teachers used to print out images, but now they do more with powerpoint and iPods plugged into TV in the classroom. The teachers say they are happy with the theme collections and they want more. They have an advisory board with teachers.. they use surveys and watch the bboards.

Question: Is there a crossover between Calishpere and OAC users?

Answer: They almost didn’t cross link to the finding aids from within Calisphere.. but they decided the information was so important. Reason for the upcoming blog – want to tell the story behind the photos.

Question: Do they have anlytics/evidence of pulling people back to their sites?

Answer: Yes.. they can see increases in usage from everything they have done.

Question: When you download the comments – are they dated so you can only look at the new ones? How hard was it to change the title in your catalog?

Answer: Everything is time/date stamped when you pull info out of Flickr. Quick and easy to update.. 10 minutes per picture to do the updates.. Flickr members are doing a great job with citations.

Question: Do you have advice about how to get historical society folks who are concerned about loosing the admission fee for people coming in to do research on board with these web 2.0 approaches?

Answer: You show them alternative revenue streams. In the museum world .. they realized that they weren’t making money from reproductions and a change is in process to let people use images for publishing.. all about improving the brand recognition. Helena: I would love ideas from people using Flickr.. and to hear from people who are dealing with multiple audiences.

Question: Have you had complaints? Any specifically from copyright holders?

Answer: Yes.. they have had complaints.. one “Why haven’t you cleaned up the photos?” LOC position is to provide the version they have.. and it is up to others to cleanup and do what they like with the photos. They also point out that instead of perfecting photos, they are spending money on providing access to more photos.

Question: Expectations of service. Are people expecting that if they ask a question about a photo that they will get an answer from a LOC representative?

Answer: Do you have to respond to everyone who asked to be a contact? No.. perhaps different expectations for institutions. They currently add a comment when they are updating the original catalog records. Might acknoledge big contributors (more than 10 photos) at the end of the pilot via a direct e-mail to individuals.

Question: Have people complained about rights – that is my grandmother.. don’t put it on the web?

Answer: No. They do have a policy in place. Most people are ‘pleased as punch’ to learn that their family heritage is alive and well. OAC: They haven’t had anyone ask to take the content down. In the case that people provide feedback for updates – since OAC is an aggregation of items from so many institutions – they have to pass corrections info along to original keeper of the metadata and leave it in their hands to do updates.

Question: Is there a fear that interest will decrease as more photos are added to the commons?

Answer: Bloggers in the web were in love with the idea that the photos would go into Flickr. There was a big peak at the start – but views and comments are still steady (but smaller) . The more additions.. more communities that will be touched. The Powerhouse Museum experienced a tripling of their traffic after posting images in the Flickr Commons.

Question: Have people come into the reading room because of the Flickr pilot?

Answer: Maybe? We don’t know. Lena said she did!

Question: Are we teaching the teachers how to teach with photos?

Answer: Calisphere has provided links to info about using primary sources and analysis tools.. resources for teachers. (Follow-up: Are they clicking those links? Good question!)

Question: Are you contacting the people who post negative comments?

Answer: Yes.. and most of them were more spam.

My Thoughts

Culture of Online Communities

There are a few different ideas I wanted to share related to the material from this presentation. First, I noticed that the online culture of both Flickr and Wikipedia were called out as having a clear impact. They are in fact two very different communities. In the case of the LOC and Flickr we heard that part of what seemed to keep the comments constructive and friendly was that Flickr’s users strive to keep a ‘play nice’ atmosphere in place. In contrast, we heard that Wikipedia was perceived as confusing and unpredictable when the CDL staff was updating pages to add links back to their primary sources. They never felt certain that the links they were working so hard to add wouldn’t be removed the next day.

These are just two examples of ways in which the archival community is beginning to bump into various online communities. We need to really understand the cultural rules for each of the communities in which we want to participate. Another excellent example of this was the revelation that LOC should only upload 50 new images a week into Flickr because of the way in which users view new images uploaded by their friends. It would be unfortunate for LOC to loose many of its Flickr friends because it overwhelmed their Flickr feeds with 1,000 images.

Personas: Targeting Real People

I was also very pleased to hear Lena discuss the creation of personas to define and target the audiences they want to serve. If you want to listen to a great presentation on personas – give a listen to the IA Summit 2008 presentation Data driven design research personas (2nd podcast down on the page) while going though the presentation slides up on slideshare. I promise it is a very accessible talk (ie, low on jargon and tech – high on real life examples) and very worth your time. It was one of the best sessions I saw at that conference.

Finding Images Without Words

While today it is generally true that people must use words to find images – someday people will be able to use images to find images. An example of this work in progress is an experimental service named retrievr. You can already use this tool to search for Flickr images either by uploading an image or by creating a sketch you want to match. Another interesting image search interface is found over on Xcavator.net. You pick a photo as your starting point – and then you can even trace a subsection of the image to be used for subsequent image matching. We are not there yet – but we will be someday. I can only image the number of Untitled images that will finally be found!

Vigilance

Your reward for reading this far is discovering my rationale for using the image I included at the top of this post. I think that many people are worried that we must be like the San Jose Vigilance Committee of 1906 – on our guard to stop people from stealing images from cultural heritage institutions when they are posted online. I would argue that the two projects described in this session show the benefits of a more open attitude. The Internet isn’t the wild west anymore. We should stop treating it that way. We don’t need Vigilance Committees online – we need ambassadors, interpreters and brave pioneers like Lena, Helena and the amazing teams of people who made the projects they described come to life.

Image credit: History San Jose Research Library via Calisphere.

As is the case with all my session summaries from SAA2008, please accept my apologies in advance for any cases in which I misquote, overly simplify or miss points altogether in the post above. These sessions move fast and my main goal is to capture the core of the ideas presented and exchanged. Feel free to contact me about corrections to my summary either via comments on this post or via my contact form.

This post is from from: Spellbound Blog.

SAA2008: Revealing Archival Collections at the Web’s Surface (Session 102)

Dan recently delivered the keynote speech at a workshop at the University of North Carolina at Chapel Hill. The workshop brought together dozens of historians to talk about how the 16 million archival documents of the Southern Historical Collection (SHC) should be put online. He devoted his keynote “to prodding the attendees into recognizing that the future of archives and research might not be like the past” and goes on in his post to explain:

The most memorable response from the audience was from an award-winning historian I know from my graduate school years, who said that during my talk she felt like “a crab being lowered into the warm water of the pot.” Behind the humor was the difficult fact that I was saying that her way of approaching an archive and understanding the past was about to be replaced by techniques that were new, unknown, and slightly scary.

This resistance to thinking in new ways about digital archives and research was reflected in the pre-workshop survey of historians. Extremely tellingly, the historians surveyed wanted the online version of the SHC to be simply a digital reproduction of the physical SHC.

Much of the stress of Dan’s article is on fear of new techniques of analysis. The choppy waters of text mining and pattern recognition threaten to wash away traditional methods of actually reading individual pages and “most historians just want to do their research they way they’ve always done it, by taking one letter out of the box at a time”.

I certainly like the idea of new technologically based ways of analyzing large sets of cultural heritage materials, but I also believe that reading individual letters will always be important. The trick is finding the right letter!

And of course – we still need the context. It isn’t as if when we digitize major collections like the SHC that we are going to scan and OCR each page without regard to which box it came out of. We can’t slice and dice archival records and manuscripts into their component parts to feed into text analysis with no way back to the originals.

I like to imagine the combination of all the new technology (be it digitization, cross collection searching, text mining or pattern recognition) as creating keys to different treasure chests. Humanities scholars are treasure hunters. Some will find their gems through careful reading of individual passages. Others will discover patterns spread across materials now co-existing virtually that before digitization would have been widely separated by space and time. Both methods will benefit from the digitization of materials and the creation of innovative search and text analysis tools. Both still require an understanding of a material’s origin. The importance of context isn’t going anywhere – we still need to know which box the letter came from (and in a perfect world, which page came before and which came after). I want scholars to still be able to read one page from the box – I just want them to be able to do it from home in the middle of the night if they are so inclined with their travel budget no worse for wear.

Dan ties his post together by pointing out that:

… in Chapel Hill I was the pirate with the strange garb and ways of behaving, and this is a good lesson for all boosters of digital methods within the humanities. We need to recognize that the digital humanities represent a scary, rule-breaking, swashbuckling movement for many historians and other scholars.

In my opinion, the core message should be that we just found more locked treasure chests – and for those who are interested, we have some new keys that just might open those locks. I enjoyed the Pirate metaphor (obviously) and I appreciate that there are real issues here relating to strong discomfort with the fast changing landscape of technology, but I have to believe that if we do something that prevents historians from being able to read one letter at a time we are abandoning the treasure chests that are already open for the new ones for which we haven’t yet found the right keys. I am greedy. I want all the treasure!

Image credit: key to anything by Stoker Studios via flickr

This post is from from: Spellbound Blog.

Of Pirates, Treasure Chests and Keys: Improving Access to Digitized Materials

On the Digital Preservation page of the Dioscuri website I found this paragraph on their goals:

To prevent a digital black hole, the Koninklijke Bibliotheek (KB), National Library of the Netherlands, and the Nationaal Archief of the Netherlands started a joint project to research and develop a solution. Both institutions have a large amount of traditional documents and are very familiar with preservation over the long term. However, the amount of digital material (publications, archival records, etc.) is increasing with a rapid pace. To manage them is already a challenge. But as cultural heritage organisations, more has to be done to keep those documents safe for hundreds of years at least.

They are nothing if not ambitious… they go on to state:

Although many people recognise the importance of having a digital preservation strategy based on emulation, it has never been taken into practice. Of course, many emulators already exist and showed the usefulness and advantages it offer. But none of them have been designed to be digital preservation proof. For this reason the National Library and Nationaal Archief of the Netherlands started a joint project on emulation.

The aim of the emulation project is to develop a new preservation strategy based on emulation.

Dioscuri is part of Planets (Preservation and Long-term Access via NETworked Services) – run by the Planets consortium and coordinated by the British Library. The Dioscuri team has created an open source emulator that can be ported to any hardware that can run a Java Virtual Machine (JVM). Individual hardware components are implemented via separate modules. These modules should make it possible to mimic many different hardware configurations without creating separate programs for every possible combination.

You can get a taste of the big thinking that is going into this work by reviewing the program overview and slide presentations from the first Emulation Expert Meeting (EEM) on digital preservation that took place on October 20th, 2006.

In the presentation given by Geoffrey Brown from Indiana University titled Virtualizing the CIC Floppy Disk Project: An Experiment in Preservation Using Emulation I found the following simple answer to the question ‘Why not just migrate?’:

• Loss of information — e.g. word edits

• Loss of fidelity — e.g. WordPerfect to Word isn’t very good

• Loss of authenticity — users of migrated document need access to original to verify authenticity

• Not always possible — closed proprietary formats

• Not always feasible — costs may be too high

• Emulation may necessary to enable migration

After reading through Emulation at the German National Library, presented by Tobias Steinke, I found my way to the kopal website. With their great tagline ‘Data into the future’, they state their goal is “…to develop a technological and organizational solution to ensure the long-term availability of electronic publications.” The real gem for me on that site is what they call the kopal demonstrator. This is a well thought out Flash application that explains the kopal project’s ‘procedures for archiving and accessing materials’ within the OAIS Reference Model framework. But it is more than that – if you are looking for a great way to get your (or someone else’s) head around digital archiving, software and related processes – definitely take a look. They even include a full Glossary.

I liked what I saw in Defining a preservation policy for a multimedia and software heritage collection, a pragmatic attempt from the Bibliothèque nationale de France, a presentation by Grégory Miura, but felt like I was missing some of the guts by just looking at the slides. I was pleased to discover what appears to be a related paper on the same topic presented at IFLA 2006 in Seoul titled: Pushing the boundaries of traditional heritage policy: Maintaining long-term access to multimedia content by introducing emulation and contextualization instead of accepting inevitable loss . Hurrah for NOT ‘accepting inevitable loss’.

Vincent Joguin’s presentation, Emulating emulators for long-term digital objects preservation: the need for a universal machine, discussed a virtual machine project named Olonys. If I understood the slides correctly, the idea behind Olonys is to create a “portable and efficient virtual processor”. This would provide an environment in which to run programs such as emulators, but isolate the programs running within it from the disparities between the original hardware and the actual current hardware. Another benefit to this approach is that only the virtual processor need be ported to new platforms rather than each individual program or emulator.

Hilde van Wijngaarden presented an Introduction to Planets at EEM. I also found another introductory level presentation that was given by Jeffrey van der Hoeven at wePreserve in September of 2007 titled Dioscuri: emulation for digital preservation.

The wePreserve site is a gold mine for presentations on these topics. They bill themselves as “the window on the synergistic activities of DigitalPreservationEurope (DPE), Cultural, Artistic and Scientific knowledge for Preservation, Access and Retrieval (CASPAR), and Preservation and Long-term Access through NETworked Services (PLANETS).” If you have time and curiosity on the subject of digital preservation, take a glance down their home page and click through to view some of the presentations.

On the site of The International Journal of Digital Curation there is a nice ten page paper that explains the most recent results of the Dioscuri project. Emulation for Digital Preservation in Practice: The Results was published in December 2007. I like being able to see slides from presentations (as linked to above), but without the notes or audio to go with them I am often left staring at really nice diagrams wondering what the author’s main point was. The paper is thorough and provides lots of great links to other reading, background and related projects.

There is a lot to dig into here. It is enough to make me wish I had a month (maybe a year?) to spend just following up on this topic alone. I found my struggle to interpret many of the Power Point slide decks that have no notes or audio very ironic. Here I was hunting for information about the preservation of born digital records and I kept finding that the records of the research provided didn’t give me the full picture. With no context beyond the text and images on the slides themselves, I was left to my own interpretation of their intended message. While I know that these presentations are not meant to be the official records of this research, I think that the effort obviously put into collecting and posting them makes it clear that others are as anxious as I to see this information.

The best digital preservation model in the world will only preserve what we choose to save. I know the famous claim on the web is that ‘content is king’ – but I would hazard to suggest that in the cultural heritage community ‘context is king’.

What does this have to do with Dioscuri and emulators? Just that as we solve the technical problems related to preservation and access, I believe that we will circle back around to realize that digital records need the same careful attention to appraisal, selection and preservation of context as ‘traditional’ records. I would like to believe that the huge hurdles we now face on the technical and process side of things will fade over time due to the immense efforts of dedicated and brilliant individuals. The next big hurdle is the same old hurdle – making sure the records we fight to preserve have enough context that they will mean anything to those in the future. We could end up with just as severe a ‘digital black hole’ due to poorly selected or poorly documented records as we could due to records that are trapped in a format we can no longer access. We need both sides of the coin to succeed in digital preservation.

Did I mention the part about ‘Hurray for open source emulator projects with ambitious goals for digital preservation’? Right. I just wanted to be clear about that.

Image Credit: The image included at the top of this post was taken from a screen shot of Dioscuri itself, the original version of which may be seen here.

This post is from from: Spellbound Blog.

Digital Preservation via Emulation – Dioscuri and the Prevention of Digital Black Holes

Introduction to Shareable Metadata and Interoperability

The first speaker, Sarah Shreeves, started the panel off with her presentation titled The Dynamics of Sharing: Introduction to Shareable Metadata and Interoperability (follow the link to view the full set of slides). Sarah is not an archivist, but she has extensive experience with metadata aggregation.

She began with the assumption that “we” (libraries/archives/museums/cultural organizations) cannot afford to think about our collections only in the context of our local community. There is no way to know where your metadata is going to end up – either grouped with other things or pulled out of your collection into single atomized items.

Why share content? It benefits our users, supports one-stop searching, brings together distributed collections and supports mashups . Sharing helps us and increases our exposure. We have to do this – we cannot assume that our users will come in through the front door. Lorcan Dempsey uses the phrase In The Flow to mean how to get your content “out” into the world where users will find it.

Keys to Shareability or Interoperability:

• You need the technical side (Z39.50, OAI PMH, RSS …etc)
• Organization commitment of resources (people, training, time, priority)
• Standards.. lots and lots of standards

There are two main ways to share metadata. The first is known as federated search. In this model a user searches from a single central location. That query is sent to distributed database and the answers are sent back for the central query source to assemble the results. Z39.50 and Search/Retrieval via URL ( SRU) are examples of technology used to perform federated searches.

The second way of sharing metadata is known as the metadata aggregation model. In this scenario, metadata is pulled from many places into a single location. This is what search engines, union catalogs, OAI PMH, RSS and Atom do. It provides an opportunity to massage and normalize the data. Once users find what they are looking for – they are often redirected to the original source of the item.

A major challenge of the metadata aggregation model is “the ability to perform a search over diverse sets of metadata records and obtain meaningful results” Priscilla Caplan (in Metadata Fundamentals for All Librarians). This is hard because we are not used to what metadata looks like outside our local context. Sarah then showed lots of different examples so the audience could see how different the metadata is.

Metadata is not monolithic. It can be a view projected from a single information object. It is possible to create multiple views appropriate for different uses. Each view will affect the granularity of description, choice of vocabularies, and choice of formats.

You can customize the format of your metadata depending on the context of how the metadata will be consumed. This might sound scary, hard and overwhelming – but Sarah is confident that we can do this in smart ways. She believes that we should be able to lobby for the features we need to support different views.

Sarah’s list of attributes of ‘shareable metadata’:

• is quality metadata
• promotes search interoperability
• is human understandable outside of its local context
• must be useful outside its local context – an aggregator can actually build services based on the data in the records provided – example was geographic data that can be used to put the items on a map
• preferably is machine processable – Subject clustering – machine created – but still needs lots of human intervention to make it work
• provides enough contextual information – the Theodore Roosevelt collection didn’t have a Roosevelt subject term because the title of the collection was assumed to be enough. She also mentioned a map that didn’t include the fact that it was a map in it’s metadata
• is consistent across a collection – ie, same date field, same controlled vocabulary.. this is within a single collection
• is coherent
• is true to its content but also its audience – different views for different perspectives
• conforms to standards – descriptive, technical, etc

There are some safe assumptions you can make. Users often get to your data through shared records – not through your front door. Users either don’t know about your collection or won’t remember. Shared records can lead users to local environments where the full context is available. Users are often entering through deep links that may bypass the introductory information that provides the larger context for a collection.

Implementing Shareable Metadata Practices

Jenn Riley, of the Inquiring Librarian Blog, gave the 2nd presentation: Implementing Shareable Metadata Practices in a Diverse University Environment. Jenn has a grand vision of what we are trying to achieve with all these efforts to share metadata. We needs lots of different ways to discover the data.. lots of environments.

We need machine-readable descriptive metadata, definitions of properties of shareable metadata in various communities (this is the focus of this session), and protocols and systems that use them for sharing that make it automatic. We also need online delivery of content too, but that is a big challenge and out of scope for this session.

Archives and digital libraries face different challenges in implementing standard practices related to shareable metadata. Archives are unique, making the notion of a single workflow model not possible. They are not a ‘homogeneous body’. Archives need to figure out how to support the expanding view of the mission to meet the needs of online users and make more services available. They need to find resources to provide appropriate description as well as technical implementation – and need time and money and skills in order to do this. On the other hand, digital library practice assumes content is digitized– that there will be ‘stuff’ at the end. Metadata-only workflows are not common. Digital libraries usually assume item-level description, but this is often not the case, and concepts of provenance and original order are largely foreign.

Communities need to agree on key definitions to bridge the gulf between digital libraries and archives. Digital libraries need to understand that Encoded Archival Description (EAD) is not a metadata format, EAD is a markup language.

The good news is that aggregations are not out to replace archives specific discovery systems. We don’t have to give up the local robust environment, we can and need to do both.

Key shareable metadata principles for archives:

• Context: need enough context so the user can figure out if the record is useful for them. At the same time – too much repeated info can cause issues too.
• Content: what is the appropriate granularity for shared records from archives — this choice needs to be done per usage and per audience.

Possible strategies include the creation of collection-level records only, creation of an aggregator that understands multi-level descriptions, the design of multi-level descriptions carefully for future item/file-level view, linking to digital objects from the lowest level of description in the finding aid and description at the item level.

Jenn then discussed the experiences at Indiana University’s Digital Library Program :

• They have a new EAD finding aid website
• the new system is more faithful to encoding with less ‘helpful’ fixed presentation
• mutual learning process about archival descriptive practices
• many decisions made about when encoding should be changed when systems should be changed
• results of this process: RE-ENGINEERING! New template, report card, better previewing capability — new template for EAD that supports new data we didn’t have before… report card built on schema-tron and encoder can preview how their encoding is really working and preview what the final product finding aid will look like
• some EAD files link to digital objects
• soon there will be item-level OAI records (Dublin Core and MODS) for digitized items linked from finding aids
• central Digital Library repository that allows EAD as the *master* metadata format
• new workflow that permits links from any level of a multi-level description in EAD

The more you put stuff online – the more you attract the sort of attention that gets you more money to put more stuff online. Jenn suggests lobbying of software vendors for a better support of EAD.. don’t settle for Dublin core. We need to discuss with our user communities about the need for an archives-specific aggregators and consider the multi-level description.

Libraries and archives are learning from one another. The item centric view can be too narrow – but it can help with re-engineering. More structure in finding aids can be a good thing. Archives can show libraries why expertise in descriptive practice is still necessary — maybe those who are running out of things to catalog on the library side can spend some time describing over on the archives side?

Archival Frameworks for Shareable Metadata

Kelcy Shepherd, Digital Interfaces Librarian at the University of Massachusetts Amherst , gave the final presentation of the session: “Archival Standards and Tools: A Framework for Shareable Metadata”.

The first framework Kelcy addressed was Describing Archives: A Content Standard (DACS). What about DACS is applicable to sharing metadata? It is compatible for use with controlled vocabularies. It can make sure that our access points will work well with access points from other metadata communities. Since DACS is output agnostic, you can create the data and then use that data to generate different views or formats. A single set of DACS based data can produce printed finding aids, EAD finding aids, MARC 21 or MODS records.

In order to produce each of these different views from a single original format, you must a crosswalk. A crosswalk maps individual elements from one data format to corresponding elements of another. Unfortunately, crosswalks come with their own challenges:

• granularity
• missing elements
• single element on one side that would need to be split into multiple elements on the other side

You need expertise in both standards addressed by a each crosswalk in order to do this well.

Next Kelcy discussed Encoded Archival Description (EAD). EAD is a data structure standard, machine readable format for encoding archival descriptions. It allows archivists to share the data across institutions. If you want to re-purpose a finding aids metadata, the data needs to be in a machine readable format. EAD gives you this. You can convert an EAD encoded finding aid into a Metadata Object Description Schema (MODS) document using an XSLT stylesheet and a crosswalk. The stylesheet may take a lot of work (especially for use across many finding aids), but there is a big payoff. Once the work is done a single stylesheet can be used across many many finding aids.

The Archivists’ Toolkit was cited as an example of a tool that can let you output multiple formats from a single set of data. It can produce EAD, MARC, MODS and Dublin Core records.

Tools can support efforts – but it all comes back to quality archival description. The best tool in the world will never make bad content into good content. If data is inconsistent – you have to manually go back and clean it up. I particularly liked Kelcy’s point about ensuring that your data doesn’t need the screen labels you to make sense. If you don’t consider this, when you export that data into a new format or view the data can loose it’s meaning.

Her concluding point was that if you don’t have the tech skills or support, work on your content.. use DACS… get your data in order and it will pay off later.

Questions and Answers

Question: How does this work when you are trying to share your metadata with communities that use different controlled vocabularies – thinking about the single EAD that generates MODS and MARC .etc etc…

Answer: Aggregators often they don’t use subject headings. This is nearly impossible to do in OAI – people use lots of different controlled vocabularies.. and sometimes no controlled vocabulary at all. There are experiments being done with subject clustering. Algorithms are used to cluster like things together – but it still requires human intervention to make sure the clusters make sense.

On the other hand – if you are using a standard vocabulary, there is work being done to map from one standard to another. An example of this is the OCLC Metadata switch project .

Question : What about social taggging?

Jenn : We are in no position to turn down metadata.

Sarah: DSpace has a concept of community. There is a way to let a community organically build their own controlled vocabulary as they go – new contributions are provided choices of terms that have been used before.

Bill talked about the article about Michaelson where they gave the same finding aids to 40 archivists to use LCSH for picking subject headings. The result was 0 consistency! Every single archivist picked different subject headings.

Jordan: PennTags is an example of an effort to combine social tagging with traditional classification. It shows tagging not as competition but as another way to get user generated descriptive information. It is an example of a way to ‘get into the flow’.

Sarah: Google will now use OAI PMH as a site map for indexing, but it throws away the metadata.

Jenn: Dlib – representing digital collections on wikipedia article.

Bill: PennTags is acting as an aggregation system to pull siloed information together.

Question: In some cases EAD data is flattened down for all items so that each item has all the context data and only one field is different on each? Is this an indication that the mapping have been better?

Answer: It is a problem – can be a problem…ultimately it is all about use and audience.

My Thoughts

I came away from this session with my head whirling with ideas. I was so pleased to hear people talk about concrete examples. We need more examples of challenges and real world benefits to further efforts to aggregate, publish and share archival content and it’s metadata. None of this is easy, but each project will give us new lessons and add to the growing set of best practices.

I truly believe that the sooner we tackle these thorny problems, the sooner we will start seeing the impact in improved access to archival records. The sooner we deal with it, the less we will be adding data that will have to be fixed later.

For anyone who has been following my blog – you will already know about my ArchivesZ project from last spring. One of the big struggles we had was figuring out how to make the subject term metadata ‘useful’ for aggregation and visualization. Another example of the challenges and benefits to shareable metadata is the SAA presentation about Publisher’s Bindings Online .

I had one last sentence in my notes from this session – an idea for a Facebook application that would let you feature your favorite archival image or record. This would be an amazing example of getting archival records ‘in the flow’ and showing up in surprising new places where no-one is ‘looking’ for records. Hey – maybe I should prod the Footnote people with this idea. It might be right up their alley!

As is the case with all my session summaries from SAA2007, please accept my apologies in advance for any cases in which I misquote, overly simplify or miss points altogether in the post above. These sessions move fast and my main goal is to capture the core of the ideas presented and exchanged. Feel free to contact me about corrections to my summary either via comments on this post or via my contact form.

This post is from from: Spellbound Blog.

SAA2007: Content Aggregation, Shareable Metadata and Access (Session 607)

Before I dive into my review of this book – I have to come clean. I must admit that I have lived and breathed the world of software development for years. I have, in fact, dreamt in code. That is NOT to say that I was programming in my dream, rather that the logic of the dream itself was rooted in the logic of the programming language I was learning at the time (they didn’t call it Oracle Bootcamp for nothing).

With that out of the way I can say that I loved this book. This book was so good that I somehow managed to read it cover to cover while taking two graduate school courses and working full time. Looking back, I am not sure when I managed to fit in all 416 pages of it (ok, there are some appendices and such at the end that I merely skimmed).

Rosenberg reports on the creation of an open source software tool named Chandler. He got permission to report on the project much as an embedded journalist does for a military unit. He went to meetings. He interviewed team members. He documented the ups and downs and real-world challenges of building a complex software tool based on a vision.

If you have even a shred of interest in the software systems that are generating records that archivists will need to preserve in the future – read this book. It is well written – and it might just scare you. If there is that much chaos in the creation of these software systems (and such frequent failure in the process), what does that mean for the archivist charged with the preservation of the data locked up inside these systems?

I have written about some of this before (see Understanding Born Digital Records: Journalists and Archivists with Parallel Challenges), but it stands repeating: If you think preserving records originating from standardized packages of off-the-shelf software is hard, then please consider that really understanding the meaning of all the data (and business rules surrounding its creation) in custom built software systems is harder still by a factor of 10 (or a 100).

It is interesting for me to feel so pessimistic about finding (or rebuilding) appropriate contextual information for electronic records. I am usually such an optimist. I suspect it is a case of knowing too much for my own good. I also think that so many attempts at preservation of archival electronic records are in their earliest stages – perhaps in that phase in which you think you have all the pieces of the puzzle. I am sure there are others who have gotten further down the path only to discover that their map to the data does not bear any resemblance to the actual records they find themselves in charge of describing and arranging. I know that in some cases everything is fine. The records being accessioned are well documented and thoroughly understood.

My fear is that in many cases we won’t know that we don’t have all the pieces we need to decipher the data until many years down the road leads me to an even darker place. While I may sound alarmist, I don’t think I am overstating the situation. This comes from my first hand experience in working with large custom built databases. Often (back in my life as a software consultant) I would be assigned to fix or add on to a program I had not written myself. This often feels like trying to crawl into someone else’s brain.

Imagine being told you must finish a 20 page paper tonight – but you don’t get to start from scratch and you have no access to the original author. You are provided a theoretically almost complete 18 page paper and piles of books with scraps of paper stuck in them. The citations are only partly done. The original assignment leaves room for original ideas – so you must discern the topic chosen by the original author by reading the paper itself. You decide that writing from scratch is foolish – but are then faced with figuring out what the person who originally was writing this was trying to say. You find 1/2 finished sentences here and there. It seems clear they meant to add entire paragraphs in some sections. The final thorn in your side is being forced to write in a voice that matches that of the original author – one that is likely odd sounding and awkward for you. About halfway through the evening you start wishing you had started from scratch – but now it is too late to start over, you just have to get it done.

So back to the archivist tasked with ensuring that future generations can make use of the electronic records in their care. The challenges are great. This sort of thing is hard even when you have the people who wrote the code sitting next to you available to answer questions and a working program with which to experiment. It just makes my head hurt to imagine piecing together the meaning of data in custom built databases long after the working software and programmers are well beyond reach.

Does this sound interesting or scary or relevant to your world? Dreaming in Code is really a great read. The people are interesting. The issues are interesting. The author does a good job of explaining the inner workings of the software world by following one real world example and grounding it in the landscape of the history of software creation. And he manages to include great analogies to explain things to those looking in curiously from outside of the software world. I hope you enjoy it as much as I did.

This post is from from: Spellbound Blog.

Book Review: Dreaming in Code (a book about why software is hard)

In the second half of class he moved on to address issues related to the freedom of information and struggling to gain access to born digital public records. Journalists are usually early in the food chain of those vying for access to and understanding of federal, state and local databases. They have many hurdles. They must learn what databases are being kept and figure out which ones are worth pursuing. Professor Chinoy relayed a number of stories about the energy and perseverance required to convince government officials to give access to the data they have collected. The rules vary from state to state (see the Maryland Public Information Act as an example) and journalists often must quote chapter and verse to prove that officials are breaking the law if they do not hand over the information. There are officials who deny that the software they use will even permit extractions of the data – or that there is no way to edit the records to remove confidential information. Some journalists find themselves hunting down the vendors of proprietary software to find out how to perform the extract they need. They then go back to the officials with that information in the hopes of proving that it can be done. I love this article linked to in Prof. Chinoy’s syllabus: The Top 38 Excuses Government Agencies Give for Not Being Able to Fulfill Your Data Request (And Suggestions on What You Should Say or Do).

After all that work – just getting your hands on the magic file of data is not enough. The data is of no use without the decoder ring of documentation and context.

I spent most of the 1990s designing and building custom databases, many for federal government agencies. There are an almost inconceivable number of person hours that go into the creation of most of these systems. Stakeholders from all over the organization destined to use the system participate in meetings and design reviews. Huge design documents are created and frequently updated … and adjustments to the logic are often made even after the system goes live (to fix bugs or add enhancements). The systems I am describing are built using complex relational databases with hundreds of tables. It is uncommon for any one person to really understand everything in it – even if they are on the IT team for the full development life cycle.

Sometimes you get lucky and the project includes people with amazing technical writing skills, but usually those talented people are aimed at writing documentation for users of the system. Those documents may or may not explain the business processes and context related to the data. They will rarely expose the relationship between a user’s actions on a screen and the data as it is stored in the underlying tables. Some decisions are only documented in the application code itself and that is not likely to be preserved along with the data.

Teams charged with the support of these systems and their users often create their own documents and databases to explain certain confusing aspects of the system and to track bugs and their fixes. A good analogy here would be to the internal files that archivists often maintain about a collection – the notes that are not shared with the researchers but instead help the archivists who work with the collection remember such things as where frequently requested documents are or what restrictions must be applied to certain documents.

So where does that leave those who are playing detective to understand the records in these systems? Trying to figure out what the data in the tables mean based on the understanding of end-users can be a fool’s errand – and that is if you even have access to actual users of the system in the first place. I don’t think there is any easy answer given the realities of how many unique systems of managing data are being used throughout the public sector.

Archivists often find themselves struggling with the same problems. They have to fight to acquire and then understand the records being stored in databases. I suspect they have even less chance of interacting with actual users of the original system that created the records – though I recall discussions in my appraisal class last term about all the benefits of working with the producers of records long before they are earmarked to head to the archives. Unfortunately, it appeared that this was often the exception rather than the rule – even if it is the preferred scenario.

The overly ambitious and optimistic part had the idea that what ‘we’ really need is a database that lists common commercial off-the-shelf (COTS) packages used by public agencies – along with information on how to extract and redact data from these packages. For those agencies using custom systems, we could include any information on what company or contractors did the work – that sort of thing can only help later. Or how about just a list of which agencies use what software? Does something like this exist? The records of what technology is purchased are public record – right? Definitely an interesting idea (for when I have all that spare time I dream about). I wonder if I set up a wiki for people to populate with this information if people would share what they already know.

I would like to imagine a future world in which all this stuff is online and you can login and download any public record you like at any time. You can get a taste of where we are on the path to achieving this dream on the archives side of things by exploring a single series of electronic records published on the US National Archives site. For example, look at the search screen for World War II Army Enlistment Records. It includes links to sample data, record group info and an FAQ. Once you make it to viewing a record – every field includes a link to explain the value. But even this extensive detail would not be enough for someone to just pick up these records and understand them – you still need to understand about World War II and Army enlistment. You still need the context of the events and this is where the FAQ comes in. Look at the information they provide – and then take a moment to imagine what it would take for a journalist to recreate a similar level of detailed information for new database records being created in a public agency today (especially when those records are guarded by officials who are leery about permitting access to the records in the first place).

This isn’t a new problem that has appeared with born digital records. Archivists and journalists have always sought the context of the information with which they are working. The new challenge is in the added obstacles that a cryptic database system can add on top of the already existing challenges of decrypting the meaning of the records.

Archivists and Journalists care about a lot of the same issues related to born digital records. How do we acquire the records people will care about? How do we understand what they mean in the context of why and how they were created? How do we enable access to the information? Where do we get the resources, time and information to support important work like this?

It is interesting for me find a new angle from which to examine rapid software development. I have spent so much of my time creating software based on the needs of a specific user community. Usually those who are paying for the software get to call the shots on the features that will be included. Certain industries do have detailed regulations designed to promote access by external observers (I am thinking of applications related to medical/pharmaceutical research and perhaps HAZMAT data) but they are definitely exceptions.

Many people are worrying about how we will make sure that the medium upon which we record our born digital records remains viable. I know that others are pondering how to make sure we have software that can actually read the data such that it isn’t just mysterious 1s and 0s. What I am addressing here is another aspect of preservation – the preservation of context. I know this too is being worried about by others, but while I suspect we can eventually come up with best practices for the IT folks to follow to ensure we can still access the data itself – it will ultimately be up to the many individuals carrying on their daily business in offices around the world to ensure that we can understand the information in the records. I suppose that isn’t new either – just another reason for journalists and archivists to make their voices heard while the people who can explain the relationships between the born digital records and the business processes that created them are still around to answer questions.

This post is from from: Spellbound Blog.

Understanding Born Digital Records: Journalists and Archivists with Parallel Challenges