[
Lists Home |
Date Index |
Thread Index
]
The soundex and multiphone algorithms convert strings to sound equivalents:
a kind of multiphone algorithm is probably similar to what you are looking for.
These were created to allow hashed lookup up family names based on sounds,
and (in the case of soundex at least) work on the assumption that spelling mistakes
are more common later in words than earlier.
They work by mapping letters to a simplified alphabet (or to numbers): soundex
uses only 6:--for example, all vowels and h are ignored, and B and P are merged,
and then consecutive duplicates are removed.
Some spelling checkers work by using the resulting key as a hash into a large
sparse table, which contains the correct spelling of words with that rough sound.
I gather that the free "aspell" spell-checking software uses some version of
multiphone.
One more recent rub is that, because English spelling is so variable and because
there are so many foreign names and words, it may be more desirable to create
multiple keys per word. For example, so that the string "gh" gets mapped
to "F" and "".
I have been looking at this this week, actually, to see whether to put in a
"sounds like" search facility. I think each use case works best with slightly
modified algorithms. (One thing I found really interesting is that Thai
sometimes shares consonants between syllables: the same consonant
is held to terminate the previous syllable and start the next, in some cases.)
Cheers
Rick Jelliffe
----- Original Message -----
From: "David Megginson" <david@megginson.com>
To: "XML Developers List" <xml-dev@lists.xml.org>
Sent: Sunday, March 09, 2003 1:18 AM
Subject: [xml-dev] [OT] Looking for a text algorithm
> I'm looking for references to a specific kind of text algorithm -- the
> algorithm should generate a number (say, 32 or 64 bits) for any text
> string of any length, similar to a hash. However, it should be
> possible to compare the numbers for different strings to tell how
> close they are to each other. For example, the numbers for
>
> 1. To be or not to be.
>
> 2. Two bees or not two bees.
>
> 3. I don't know whether to be or not to be.
>
> should indicate that three strings are relatively close to each other
> (while a hash number would give no indication at all).
>
> I'm asking only out of interest, because I came up with a simple
> algorithm to do this while I was in the shower yesterday, and it would
> be fun to see how close it is to what the pros use for spam detection
> and so on.
>
> Note that I'm not looking for algorithms based on edit-distance,
> bag-of-words, and so on.
>
>
> Thanks in advance,
>
>
> David
>
> --
> David Megginson, david@megginson.com, http://www.megginson.com/
>
> -----------------------------------------------------------------
> The xml-dev list is sponsored by XML.org <http://www.xml.org>, an
> initiative of OASIS <http://www.oasis-open.org>
>
> The list archives are at http://lists.xml.org/archives/xml-dev/
>
> To subscribe or unsubscribe from this list use the subscription
> manager: <http://lists.xml.org/ob/adm.pl>
>
>
|
