Similar strings.

Question

Similar strings.

ben.brugman

SSChampion

Points: 13369
More actions
April 23, 2013 at 5:33 am

#273195

In the past week, I saw somewhere a string similarity function.
(I think I saw it in my 'spare' time when I am not focused on my work but still am reading about SQL-server to keep up. So did not note the link, the book or whatever.)
Depending on the longest string which was shared between two strings a value between 0 and 1 was given. This dependend on the length of the string(s) and the similarity. A tresshold was set to get a ?? good starting point??
One example were it was used was to compare chemical formula. And find the same formula but which was not written the same.
Does anybody recognise were I could have seen this?
Other (or same) techniques to find similar strings?
We want to use this technique to find questions which are similar but formulated differently. Does not have to be perfect, but is to be used a a help. (I know the Soundex and Difference functions).
Thanx in advance,
ben brugman

Viewing 7 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic. Login to reply

Eric M Russell SSC Guru Points: 125561 More actions · Answer 1

There there are plenty of user defined functions posted on the web that will parse a string into a resultset of keywords, compare two strings, etc. They may provide the functionality you're looking for, but understand that parsing keywords from varchar or text based columns tends to perform very poorly, because it leverages looping constructs and is not indexable.

However, perhaps you were reading an article on SQL Server 2012's new Semantic Search service.

... provides deep insight into unstructured documents stored in SQL Server databases by extracting and indexing statistically relevant key phrases. Then it also uses these key phrases to identify and index documents that are similar or related ...

http://msdn.microsoft.com/en-us/library/gg492075.aspx

https://www.simple-talk.com/sql/database-administration/exploring-semantic-search-key-term-relevance/

If you don't have SQL Server 2012, then look into the Full-Text Search service, which is robust and has been around for decades in several past releases of SQL Server.

http://msdn.microsoft.com/en-us/library/ms142571%28v=sql.105%29.aspx

"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho

Jack Corbett SSC Guru Points: 184394 More actions · Answer 2

You could also be thinking about the SOUNDEX function.

Jack Corbett
Consultant - Straight Path Solutions
Check out these links on how to get faster and more accurate answers:
Forum Etiquette: How to post data/code on a forum to get the best help
Need an Answer? Actually, No ... You Need a Question

Jeff Moden SSC Guru Points: 1004356 More actions · Answer 3

Jack Corbett (4/24/2013)
You could also be thinking about the SOUNDEX function.

Agreed but that's a terrible function. It stops scanning at the first non-alpha character to name just one serious disadvantage.

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

Jeff Moden SSC Guru Points: 1004356 More actions · Answer 4

ben.brugman (4/23/2013)
In the past week, I saw somewhere a string similarity function.
(I think I saw it in my 'spare' time when I am not focused on my work but still am reading about SQL-server to keep up. So did not note the link, the book or whatever.)
Depending on the longest string which was shared between two strings a value between 0 and 1 was given. This dependend on the length of the string(s) and the similarity. A tresshold was set to get a ?? good starting point??
One example were it was used was to compare chemical formula. And find the same formula but which was not written the same.
Does anybody recognise were I could have seen this?
Other (or same) techniques to find similar strings?
We want to use this technique to find questions which are similar but formulated differently. Does not have to be perfect, but is to be used a a help. (I know the Soundex and Difference functions).
Thanx in advance,
ben brugman

Perhaps you brushed against the following?

http://en.wikipedia.org/wiki/Levenshtein_distance

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

Jack Corbett SSC Guru Points: 184394 More actions · Answer 5

Jeff Moden (4/24/2013)
Jack Corbett (4/24/2013)
You could also be thinking about the SOUNDEX function.
Agreed but that's a terrible function. It stops scanning at the first non-alpha character to name just one serious disadvantage.

I didn't say it was any good, I just said it was out there. 😀

I've tried it and discarded it as not having any use.

Jack Corbett
Consultant - Straight Path Solutions
Check out these links on how to get faster and more accurate answers:
Forum Etiquette: How to post data/code on a forum to get the best help
Need an Answer? Actually, No ... You Need a Question

ben.brugman SSChampion Points: 13369 More actions · Answer 6

Jeff Moden (4/24/2013)
Perhaps you brushed against the following?
http://en.wikipedia.org/wiki/Levenshtein_distance

Thanks for the link (losing a 'h'):

http://en.wikipedia.org/wiki/Levenstein_distance]

This was not what I saw, but 'solves' the same problem. So maybe this is even better than what I saw. (Have to try it out).

As for the Soundex and difference, which I mentioned in my first message, they are not usefull for my situation.

Thanks for the link,

ben brugman