THATCamp Mallet Results

We have spent the last few weeks working to build a python script that would allow us to download and prep all of the THATCamp blog posts for topic modeling in MALLET (for those catching up, we detailed this process in a series of previous posts). As our last post detailed, we encountered a few more complications than expected due to foreign languages in the corpus of the text.  After some discussion, we worked through these issues and were able to add stoplists to the script for German, French, and Spanish.  Although this didn’t solve all of our issues and some terms do still show up (we didn’t realize there was Dutch too), it led to some interesting discussion about the methodology behind topic modeling.  Finally we were able to rerun the python script with the new stopwords and then feed this new data into MALLET.

MALLET, or MAchine Learning for Language Toolkit, is an open source java package that can be used for natural language processing.  We used the Programming Historian’s tutorial on MALLET.  Topic modeling is an important digital tool that analyzes a corpus of text and seeks to extract ‘topics’ or sets of words that are statistically relevant to each other.  The result is a particular number of word sets also known as “topics.” In our case we asked MALLET to return twenty topics based on our set of THATCamp blog posts. The topics returned by MALLET were:

  1. xa digital art history research university scholarship graduate field center publishing open today institute cultural knowledge professor online world
  2. university games pm humanities http digital september knowledge kansas saturday game conference state registration play information representation workshop boise
  3. thatcamp session sessions day participants free technology page unconference university nwe conference discussion information google propose hope event proposals
  4. people make time questions things idea access process ideas world work great making lot build add kind interesting nthe
  5. digital humanities data tools text projects research scholars omeka texts tool analysis scholarly archive reading online based book scholarship
  6. digital humanities session dh library libraries support projects open discussion librarians amp talk work journal sessions propose faculty list
  7. history public digital historical collections museum media project projects mobile online maps museums collection historians users sites site applications
  8. games zotero thinking place game code end cultural chnm hack year documentation humanists version number pretty application visualization set
  9. session open area data workshop tool knowledge teach interested bay prime bootcamp gis workshops reality night thatcampva virginia lab
  10. work interested students ways teaching post working talk writing blog love issues don conversation create collaborative thinking start discuss
  11. project web content information tools community resources archives experience research create learn creating learning share development materials specific provide
  12. xb xa del se humanidades digitales xad al madrid www mi este aires buenos digital personas taller cuba parte
  13. caption online id align width open attachment accessibility women read university american building accessible gender media november floor race
  14. data http org session www open twitter texas good wikipedia nhttp status wiki start commons drupal metadata people crowd
  15. xa workshop session omeka publishing http gt propose org workshops friday docs open hands amp doc studies topic discuss
  16. students digital learning technology education media college faculty humanities research game pedagogy student courses classroom assignments skills arts social
  17. xa oral digital humanities video event local application community offer interviews planning center education software jewish weekend college histories
  18. een het voor op te zijn deze met workshop kunnen om digitale bronnen data onderzoek historici nl wat worden
  19. social media technology studies arts performance museums xcf play participants cultural performing reading st email object platforms interaction technologies
  20. xa thatcamp org http thatcamps details read movement published planned access nthatcamp browse software follow break series google join

As you can see we have an impressive list of terms. Before we organize them in a meaningful way, we will briefly point out a common problem that scholars may confront when working with MALLET. As you may notice, we realized that we have quite a few errors such as ‘xa’ that appear in the results.  While we don’t have a great answer for why this is, we think it has to do with complex encoding issues related to moving content from a WordPress post that is stored in a MySQL database using Python. Each of these uses a different coding system and the error appears to be related to non-breaking spaces.  A little bit of Googling revealed that the non-breaking space character used by WordPress is ‘&nbsp’ which is different that the ASCII encoding of a non-breaking space ‘/xa0’.  When Python reads WordPress’s non-breaking space character ‘&nbsp’, it understands the space but encodes it as the UTF-8 version ‘/xa0’.  As second year fellow Spencer Roberts explained the issue is that meaning is lost in translation. He used this analogy: Python reads and understands the French word for “dog” then translates it and returns the English word.

In this case, what shows up in our results is not ‘/xa0’ but rather ‘xa’ because we had stripped out all of the non-alphanumeric characters prior to running the data through MALLET.  We think the errors such as ‘xa’ and ‘xb’ are because of these encoding issues.  Anyone interested in clarifying or continuing this discussion with us can do so in the comments.

Returning to our MALLET results, our next challenge was to present and analyze the large amount of data.  We drew from both Cameron Blevins and Robert K. Nelson in our approach and decided to group the topics by theme so that trends could be more easily identified.  We determined that there were about seven broad themes in the corpus of THATCamp blog posts from 2008 to present:

  1. Accessibility
  2. Building
  3. Community
  4. International
  5. Pedagogy
  6. Public Digital Humanities
  7. THATCamp Structure

Utilizing these larger categories, we were able to create several charts that demonstrate the changes over time with the THATCamps. The charts are available below; you’ll note that we have graphed them using percentages. The percentages that appear represent the number of times that topic occurred within the posts at that camp.

Chart of Topics Overall

Topics Overall

Chart of the topic Accessibility.

Topics relating to Accessibility

Graph of the Community Topic

Topics relating to Community

Graph of the THATCamp Structure Topics

Topics relating to THATCamp Structure

Graph of topics relating to Pedagogy

Topics relating to Pedagogy

Graph of topics relating to Public Digital Humanities

Topics relating to Public Digital Humanities

Graph of topics relating to building in the humanities.

Topics relating to Building

Topics relating to the international influence of THATCamp.

International Digital Humanities

We found these results to be particularly interesting. A larger overall conclusion is that THATCamp content emphasizes the various applications of digital technology to scholarship, from public uses to tool building or teaching. Since THATCamp was founded, it has become a more varied community. However close examination of the topic models this exercise produced reveals that a number of the same terms appear frequently across all of the topic models (“digital”, for instance, appears in 8 of the 20 topics). This references the way in which ideas are circulated throughout camps and unifies the community. It also reflects the subjects that are the focus of the community.

If you’re interested in the data, you can view the various files here:

Unexpected Challenges Result in Important and Informative Discussions: a transparent discussion about stripping content and stopwords

As described in previous posts, the first year Digital Fellows at CHNM have been working on a project under the Research division that involves collecting, cleaning, and analyzing data from a corpus of THATCamp content. Having overcome the hurdles of writing some python script and using MySQL to grab content from tables in the backend of a WordPress install, we moved on to the relatively straightforward process of running our stripped text files through MALLET.

As we opened the MALLET output files, excited to see the topic models it produced, we were confronted with a problem we didn’t reasonably anticipate and this turned into a rather important discussion about data and meaning.

As a bit of background: topic modeling involves a process of filtering “stopwords” from a data set. Frequently a list of stopwords includes function words, or terms that appear repeatedly in discourse, like: “a, an, the”. These are filtered out because they serve a grammatical purpose but have little lexical meaning. Additionally, errors, misspellings, and lines of code that were skipped in the previous steps can also be filtered out at this stage.

As we opened the file of keys produced by MALLET, we found that some terms appeared that raised questions about what should or should not be included in our analysis. In particular, the discussion centered around spelling errors and function words in Spanish and French.

The conversation that followed, reproduced below, was significant and as people look through the results of this project or consider their own efforts reproduce something like this elsewhere, we’d like to be transparent about the decisions we made and, perhaps, spur a discussion about how to address scenarios like this in the future.

_____________________________________________

Take a look at what MALLET spit out: there are some errors.
Stuff like “xe, xc, zijn, xb, en, la”.

Yeah, I saw that.
We can make a custom stoplist to remove those.
Make a list and we’ll add it.

Interesting keys though – did you see that “women” came up in #0?
I’m excited to see this once it’s all graphed.

There’s some stuff though that I’m not sure we want to remove: Is CAA an error or an abbreviation? “socal” – is that social misspelled or Southern California, abbreviated?
Hmm, that could be an organization..

Yeah, SoCal, as in Southern California.
There was a camp there.
This adds a larger question: do we remove misspellings?
For clarity?

I have mixed feelings.

Me too.

I think it’s appropriate to remove backend stuff-
tags and metadata, but content is not something we should modify.

I agree.
We don’t want to skew the results.
Some of it occurred when we stripped all the alphanumeric stuff out.
It took out apostrophes- causing words like “I’ve” to become “I ve”.

The errors in themselves are telling about the nature of THATCamps.
That the content is generated spontaneously
lends itself to deviations from appropriate spelling ect.

I agree

Look at #17 and 18.

Whoa, where did “humanidades” come from?
Oh, right! There are international conference posts in here too!

How do we handle this?
Its possible to strip out the camps that are not in English
or even to run analysis on them separately.
I don’t want to skew the results but this also throws things off.

I know.
I say we leave it. It shows a growing international influence.
We’ll be able to see the emergence of International THATCamps.

I’ve never run into this before.
It brings up some interesting issues -
I wonder if there is standard procedure for something like this.

What about things like “en” which is Spanish for “is”-
that would have been removed on an English stoplist.
And now function words in Spanish and French
seem to appear more frequently because
the English terms have been filtered out.
How do you do topic modeling with multiple languages?

What about special characters?
We’ve stripped stuff out, how about how that would affect the appearance of words?

We have to find a stoplist for each of the languages.
To strip out the function words of all of them.

Good call. This got complicated quick!

Agreed.

_____________________________________________

As outlined above, when we opened the text file with keys, new questions were raised about the relevance and complications of running a particular stoplist on a corpus of texts. Similarly, we were forced to rethink how we handle misspellings and unfamiliar abbreviations. In the end, we tracked down stoplists for Spanish (and French) terms, so that no function words in any language would skew the results of our analysis. We also carefully examined the keys to identify abbreviations and misspellings and decided that they are a significant contribution to the analysis.

A few questions remained for us: how might removing non-alphanumeric characters (a-z,A-Z,0-9) alter the meaning of special characters used in languages other than English? How have others responded to spelling errors? How significant are errors?

Hopefully a post of this nature will foster discussion and produce a stronger, more complete analysis on this and other documents.

Spring Semester in Research and a THATCamp Challenge

The spring semester is here and the first year DH fellows have begun our rotation into the Research division of CHNM.

To get the ball rolling, we spent a week working through the helpful tutorials at the Programming Historian. As someone new to DH, with admittedly limited technical skill and knowledge, these were immeasurably useful. Each tutorial breaks content into smaller, less intimidating units. These can be completed in succession or selected for a particular topic or skill. While there is useful content for anyone, we focused our attention on Python and Topic Modeling with the aim of solving our own programming dilemma.

Our central challenge was to extract content across the THATCamp WordPress site to enable us to do some text analysis.

Continue reading

Public Projects: Reflection

Our first semester at the Center for History and New Media has flown by. We spent the second half of the semester in the Public Projects Division which was a diverse and rewarding experience.

During this rotation we were able to tour the entire division and spend some time working with many of the division’s projects.  We spent a large chunk of time working with Omeka, testing plugins, themes, and other items that are in development.  One thing I took away from working with the Omeka team and attending the Sprint Planning meetings is how collaborative this division, and the center as a whole, is.  Between programmers, designers, testers, and content development– Omeka really is a team project that seeks to make collecting easier for museums and archives.  Through working with the software we also got some hands on experience with the amount of work it takes to build an archive and what kinds of issues come up when doing so.  We discussed and experienced issues such as the naming of pages and areas on a site, creating a strict vocabulary to make searching consistent, and developing content first hand.

We also spend time developing content for projects such as The Histories of the National Mall and Papers of the War Department.  The National Mall project allowed us to think about how the public utilizes mobile history sites when at a museum or a national park such as the Mall.  We spent a wonderful afternoon down on the mall testing the mobile first site (and enjoyed some excellent tacos from the local food truck tacos!).

Papers of the War Department was a different experience and we spent time both transcribing documents and tagging meta data for documents. Using the Scripto plugin for Omeka, we first tagged revisit documents with key words, names, places, and topics.  This element of the project required some knowledge and required a deeper engagement with the documents than transcribing did.  Transcribing the documents was challenging (seventeenth century handwriting is interesting) but we could all see the immense benefit to having the documents both transcribed and tagged on the site.

I think we are starting to really begin to understand the inner workings of the center and the projects and goals of each division.  Public Projects does several different things from software development to content based projects and I think we all benefited greatly from our tour around the division. Coincidently, the first year fellows were also taking Clio Wired I this semester and often what we did at the center overlapped with what we did in class making the experience even more valuable for us.  I think we all came away from this semester having learned a great deal and I feel much more aware of many of the issues facing scholars in Digital History centers as well as in academia in general.

Reflections: Year Two, Semester One

As the first term of 2013-14 closes, it seems appropriate to reflect on the experiences of the Digital History Fellows. Last year, our first cohort of DH Fellows spent the first semester meeting with Dan Cohen, learning the history of the center, discussing current projects, and thinking about how digital history is practiced. We spent our second semester working in each of the divisions for five weeks, and then decided in which division we would like to work in the second year. Although there was no specific requirement that we take positions spread across the three divisions, we were drawn in different directions. From the first days of the fellowship, Ben Hurwitz was most comfortable in Education and quickly entrenched himself at their community table. He now works on various educational projects, including the Popular Romance Project. Amanda Morton worked closely with Fred Gibbs before he relocated to New Mexico, which helped her transition into Research, where she works on Digital Humanities Now and related PressForward projects. Spencer Roberts was drifting toward Public Projects before the summer started, and settled in once the center received a grant to work with the National Park Service to revamp their War of 1812 site.

This year we welcomed three new members into the fellowship, bringing our total number to six. The second cohort follows a different schedule in their first year, so Amanda Regan, Anne Ladyem McDivitt, and Jannelle Legg stepped directly into the mix at RRCHNM, splitting their semester into seven-week blocks in Education and Public Projects. During those weeks, they have written reflective posts about the projects to which they’ve contributed, all of which can be found here. Next term, they will spend a block in Research before moving into a final seminar with Stephen Robertson.

Continue reading