Baby Steps in NLP

Having got the Bug model working again (which didn’t take long), I turned my attention to playing with natural language processing.

NLP is hard, and I’m not expecting any huge leaps in function from my tinkering. The intention is to do small bits of work in different areas, aiming to have enough function to support bootstrapping the next round of improvements.

I’m using a dump of Simple English Wikipedia as the main source of text to process.

Function Words

The first thing I worked on was scanning the text and identifying function words (the, is, it, etc — the most common words, and usually not in WordNet).

The procedure I ended up with was to take the 500 most common words and score them on a number of criteria:

• frequency
• rarely found at the end of a sentence
• short words
• does not appear to be a capitalised name
• can’t be reduced by stemming to a word defined in WordNet
• if it is in WordNet, has a small number of senses (some obvious function words, like ‘a’, do have entries in WordNet for various reasons)

Taking the highest scoring words yielded 72 function words. The list omits some words that occur more rarely, like between.

Phrases

The next thing I wanted to try was learning how words go together in phrases. When parsing general text, it’s not obvious where the boundaries of phrases are (function words and punctuation are useful, but not infallible). Conveniently though, from wikipedia I can easily extract article titles, headings, link text, and category names, all of which tend to be short and should be complete phrases.

On the first pass, I’m specifically looking for two-word phrases. I consider all possible dictionary entries for every word and build all the conceivable ‘phrase forms’ for every combination of word interpretations, generalising to the POS (Noun, Verb, Adverb, Adj) or a specific function word.

Every phrase form is recorded and scored for frequency of occurrence. That yielded 93 phrase forms. The ten highest scoring do include the ones you’d expect: Adj Noun, Noun Noun, Noun Verb, the Noun, but also Verb Verb, Adj Adj, and Adv Noun — which may not be sensible.

Next Steps

I have other projects to work on, but will try to keep tinkering as time allows. Morphology is probably the next area to look at, along with better handling of numbers and punctuation.