THATCamp 2008: Text Mining and the Persian Carpet Effect

alarch: Drift of Harrachov mine (Flickr)I attended a THATCamp session on Text Mining. There were between 15 and 20 people in attendance. I have done my best to attribute ideas to their originators wherever possible – but please forgive the fact that I did not catch the names of everyone who was part of this session.

What Is Text Mining?

Text mining is an umbrella phrase that covers many different techniques and types of tools.

The CHNM NEH-funded text mining initiative defined text mining as needing to support these three research functions:

  • Locating or finding: improving on search
  • Extraction: once you find a set of interesting documents, how do you extract information in new (and hopefully faster) ways? How do you pull data from unstructured bulk into structured sets?
  • Analysis: support analyzing the data, discovery of patterns, answering questions

The group discussed that there were both macro and micro aspects to text mining. Sometimes you are trying to explore a collection. Sometimes you are trying to examine a single document in great detail. Still other situations call for using text mining to generate automated classification of content using established vocabularies. Different kinds of tools will be important during different phases of research.

Projects, Tools, Examples & Cool Ideas

Andrea Eastman-Mullins, from Alexander Street Press, mentioned the University of Chicago’s ARTFL Project and these two tools:

  • PhiloLogic: An XML/SGML based full-text search, retrieval and analysis tool
  • PhiloMine: a extension being developed for PhiloLogic to provide support for “a variety of machine learning, text mining, and document clustering tasks”.

Dan Cohen directed us to his post about Mapping What Americans Did on September 11 and to Twistori which text mines Twitter.

Other Projects & Examples:

Some neat ideas that were mentioned for ways text mining could be used (lots of other great ideas were discussed – these are the two that made it into my notes):

  • Train a tool with collections of content from individual time periods, then use the tool to assist in identification of originating time period for new documents. Also could use this same setup to identify shifts in patterns in text by comparing large data sets from specific date ranges
  • If you have a tool that has learned how to classify certain types of content well… then watch for when it breaks – this can give you interesting trails to things to investigate.

Barriers to Text Mining

All of the following were touched upon as being barriers or challenges to text mining:

  • access to raw text in gated collections (ie, collections which require payment to permit access to resources) such as JSTOR and Project MUSE and others.
  • tools that are too difficult for non-programmers to use
  • questions relating to the validity of text mining as a technique for drawing legitimate conclusions

Next Steps

These ideas were ones put forward as important to move forward the field of text mining in the humanities:

  • develop and share best practices for use when cultural heritage institutions make digitization and transcription deals with corporate entities
  • create frameworks that enable individuals to reproduce the work of others and provide transparency into the assumptions behind the research
  • create tools and techniques that smooth the path from digitization to transcription
  • develop focused, easy-to-use tools that bridge the gap between computer programmers and humanities researchers

My thoughts
During the session I drew a parallel between the information one can glean in the field of archeology from the air that cannot be realized on the ground. I discovered it has a name:

“Archaeologists call it the Persian carpet effect. Imagine you’re a mouse running across an elaborately decorated rug. The ground would merely be a blur of shapes and colors. You could spend your life going back and forth, studying an inch at a time, and never see the patterns. Like a mouse on a carpet, an archaeologist painstakingly excavating a site might easily miss the whole for the parts.” from Airborne Archaeology, Smithsonian magazine, December 2005 (emphasis mine)

While I don’t see any coffee table books in the near future of text mining (such as The Past from Above: Aerial Photographs of Archaeological Sites), I do think that this idea captures the promise that we have before us in the form of the text mining tools. Everyone in our session seemed to agree that these tools will empower people to do things that no individual could have done in a lifetime by hand. The digital world is producing terabytes of text. We will need text mining tools just to find our way in this blizzard of content. It is all well and good to know that each snowflake is unique – but tell that to the 21st century historian soon to be buried under the weight of blogs, tweets, wikis and all other manner of web content.

Image credit: Drift of Harrachov Mine by alarch via flickr

As is the case with all my session summaries from THATCamp 2008, please accept my apologies in advance for any cases in which I misquote, overly simplify or miss points altogether in the post above. These sessions move fast and my main goal is to capture the core of the ideas presented and exchanged. Feel free to contact me about corrections to my summary either via comments on this post or via my contact form.

http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/digg_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/reddit_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/delicious_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/blinklist_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/newsvine_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/technorati_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/google_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/facebook_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/yahoobuzz_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/sphinn_48.png http://www.spellboundblog.com/wp-content/plugins/sociofluid/images/twitter_48.png
Related Posts:

Posted on 1st June 2008
Under: digitization, historical research, information visualization, learning technology, software, text mining, THATCamp2008 | 5 Comments » | Print This Post Print This Post

5 Responses to “THATCamp 2008: Text Mining and the Persian Carpet Effect”

  1. To take away » Blog Archive » THATCamp 2008: Crowdsourced Transcription and Collaborative Annotation Says:

    [...] and take over the keyboard in order to show off their favorite sites. It was immediately after the Text Mining session, so our minds were already full of all the great things one could do with text once it is [...]

  2. THATCamp 2008: Crowdsourced Transcription and Collaborative Annotation - SpellboundBlog.com - spellbound by archival science and information technology in the digital age Says:

    [...] Matt « THATCamp 2008: Text Mining and the Persian Carpet Effect THATCamp 2008: Day 1 Dork Short Lightening Talks [...]

  3. Recent Links Tagged With "textmining" - JabberTags Says:

    [...] links >> textmining The Text Mining Can of Worms : Beyond Search Saved by xxc on Tue 14-10-2008 THATCamp 2008: Text Mining and the Persian Carpet Effect Saved by meximese on Mon 13-10-2008 The Text Mining Handbook: Advanced Approaches in Analyzing [...]

  4. IRC log from THATCamp 2008 | THATCamp Says:

    [...] [...]

  5. THATCamp News » Blog Archive Says:

    [...] [...]

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>