Having drifted off to other projects, I’m now finally getting back to thinking about AI. I’ve dumped all my random notes in one document and am trying to be more organised and put some serious time in.
One area I’ve been thinking about, on and off, is natural language processing. I have some corpora that came with Python NLTK, which are good, but on closer examination none of them were big lumps of fairly simple, straightforward English text.
The ideal thing would be articles from Simple English Wikipedia. So first, grab your articles — dumps of wikipedia are available from https://dumps.wikimedia.org/simplewiki/ The pages-articles-multistream.xml file is what I wanted.
This is a very large XML file. To read it, I can use a SAXParser — easy. The article text (which is what I actually want) is text content delimited by the <text> tag. It includes various kinds of wiki markup, like [[ links to other articles ]].
So the next problem is to strip out the markup to get to the basic English text. This proved tricky: there’s a lot of markup and its not straightforward. I went off to look for a Java library that would do the hard part for me, and found there were several projects that might do.
I got xwiki and played with it, but couldn’t get it to run in the Eclipse environment. So then I turned to sweble, which appears to be defunct, but I dug some jar files out from the depths of the internet. Surprisingly, this ran with very little trouble and had a pretty straightforward interface. It doesn’t do an ideal job of recognising all the formatting though.
For now, that’s good enough — I can extract a lot of good readable English text and ignore all the stuff which has leftover scraps of markup laying round.
Working Models of a Brain 