1

Data Request

I am looking for some data that contains English words along with the probability that a person will spell it incorrectly.

The probability depends on various factors. However, a simple measure would be frequency of wrong spellings divided by the total frequency of the word.

Alternatively, the data can also be of a word with a class denoting its likelihood to be mistaken.

Context

I want to create a basic NLP project in which I will generate a prediction - given a word, what is the probability of wrongly spelling it.

Region

An US/UK based English variant would be appreciated, although other combinations are also good.

License

Open data.

Format

A CSV based format would be appreciated. Some examples are given below.

Word,Frequency,FrequencyOfErrors
apple,100,2
asphyxiated,54,14
...

OR

Word,ProbabilityOfError
apple,0.02
asphyxiated,0.26
...

However, it is not necessary, and a general program-friendly format would suffice.

Authority

Data provided by some reliable source, for example, an academic body, would be good. Crowdsourced data would also be sufficient.

Requirements

There should be a large number of words - 20,000 or more.

The individual words should have moderately high frequencies of occurrence - 50 or more.

Non-answers

I found this topic which provides age of acquisition of various words. However, I believe there is very little correlation between making a spelling mistake and knowing the word.

pmcarpan
  • 111
  • 3
  • https://en.oxforddictionaries.com/spelling/common-misspellings doesnt anser ur queston bcause it has far to few words. However, you could dowload a word corpus (the Oxford one doesnt apear to be free, but others are) and perhaps glean the data you need from there. You could also contact Oxford directly and see if they have a longer and more numberical list of mispelled words (prolly not free). –  Nov 08 '18 at 20:00
  • I have looked at some word lists which only give word frequency. However, those do not give any "mistake frequency/hardness"; as a word like engineering may be highly used, but also prone to mistakes. Also, misspellings lists do not provide any frequency info(as far as I checked). Thank you. – pmcarpan Nov 09 '18 at 11:55
  • Not word lists, but word corpuses-- text as people use it (for example, a list of all reddit postings). The Oxford corpus is one such (it's just basically a bunch of text people have written), but it's not free. –  Nov 09 '18 at 23:48

0 Answers0