Good! I learn to post code in messages in the right way!

You code shows sql server behavior very well.

Unfortunately I need to have the same value in t-sql, not way...

well, it seems that the difference (tsql = 0 and vb = -1.2325951644078309E-32) is due to difference round between sql and vb.

who I can increase round to have the save...

Guys, this is driving me crazy!

T-SQL (all of the variables are FLOAT(53))

SELECT @Tmp = measureData FROM dbo.New_AMatrix WHERE x = @i AND y = @j-2

SELECT @tmp2...

Hi Michael,

VB and Excel returns the same value.

I'm currently coding T-SQL again to see where the problem lies.

in my modest opinion if it's a os/hw problem i should not have...

No problem Michael. i didn't mean you offense me.

I really appreciate the effort you all puts into this problem

I just want to understand what you all are trying to...

Hi Michael,

what do you mean with

First of all, I find it very unusal for the OP (Max) to have found the first two operands whose mantissas consist of almost all...

well. it seems that we have trust vb result for years and now we discover that it was not right!

i'll try this also with c++ to see what is the...

I do not have the same result in sql using float data type.

i got 0 back.

i checked also @@MAX_PRECISION is = to 38

regards,

max

I swear vb result is 3.85185988877447E-34 !! (how can i attach an image to a new post?)

SQL result is 0

I'm actually using FLOAT instead REAL.

Thank you

Regards,

Max

sorry i wrote some incorrect info in my first post.

the correct vars declare is

DECLARE @Tmp FLOAT

DECLARE @tmp2 FLOAT

DECLARE @NewValue FLOAT(53)

the @m is set to minus:

SELECT

here's the whole stored procedure.

sorry i copy the wrong variables declaration ( real instead of float )

yes, unfortunately i do need that precision as the request is to compute linear...