June 18, 2012 at 8:17 am
Hi,
I have 2 columns in a table which are both floats. In column A I have the value 5896, in Column B 13180.
If I divide Column A by Column B I get the result 0.449973288559872.
If I do the same in Excel or using a calculator I get 0.4473444613050076.
What makes the results different? I'm guessing something to do with Data Types and Lengths. As Excel's result is the answer I actually want (I already have a spreadsheet with all the data and calculations in, I'm just slowly replicating it into SQL), how would I go about getting the equivalent result in SQL.
Thanks
John
June 18, 2012 at 8:31 am
It would help if you posted the table definitions of the tables you are working with. When I run this:
select CONVERT(float, 5896)/CONVERT(float, 13180)
I get your calculator/Excel answer.
June 18, 2012 at 8:46 am
Thank you, but this was user error. I was actually using a similarly named column with similar values giving what looks like the figures I want just marginally different in the decimal places.
June 18, 2012 at 8:48 am
I'd be curious what is actually in the two columns. When I setup two variables as floats and enter the values you provide I get the answer you are looking for, not the answer you get. I have done this with both SQL Server 2005 and SQL Server 2008 R2.
June 18, 2012 at 8:49 am
john.walker 46407 (6/18/2012)
Thank you, but this was user error. I was actually using a similarly named column with similar values giving what looks like the figures I want just marginally different in the decimal places.
What?
Viewing 5 posts - 1 through 4 (of 4 total)
You must be logged in to reply to this topic. Login to reply