January 11, 2012 at 10:40 am
Greetings all. I'm using SQL Server 2008 R2 Express, and I've got a linked server to a DB2 mainframe. There is one particular field where I must use the DB2 HEX function to return a varchar value. I have been using OPENQUERY, so have been able to pull out what I need with the DB2 HEX function. I'll give one example to illustrate. If I do this...
SELECT
*
FROM OPENQUERY(LinkedServer,'
SELECT
ID,
Column1,
LENGTH(Column1),
HEX(Column1) AS HEX_VAL
FROM Schema.Table
WHERE ID = 1
WITH UR;
')
My results look like this
IDColumn1HEX_VAL LENGTH
1BC200000012700CD400000003673CD500000000850C42
Now, I know that the HEX_VAL string is simply a string of 14 character pieces (I'll call them "words"), from which I can extract a value by pulling out substrings. For example, CAST(SUBSTRING(HEX_VAL,3,11) AS DECIMAL(11,2))/100.0 = 127.00. No matter how many "words" are in the string, I can use a Tally table to reliably pull out what I need. My problem is that I have figured out how to configure my linked server so that I can use 4 part naming convention instead of OPENQUERY. This is great, however, there is no SQL Server equivalent to what the DB2 HEX function is doing. The closest I can muster is to convert to varbinary, which looks like this...
SELECT TOP 1
ID,
Column1,
VARBIN = CAST(Column1 AS VARBINARY),
LENGTH = LEN(Column1)
FROM LinkedServer..Schema.Table
WHERE ID = 1
with results...
IDColumn1VARBIN LENGTH
1B0x4200000012F80C4D00000003C5144E00000000650C42
So, I guess I have a couple of questions.
1) Why does the value of the field return 'B' when no conversion is happening, even though SQL Server knows the len is 42?
2) What is the difference between ...
C200000012700CD400000003673CD500000000850C and 0x4200000012F80C4D00000003C5144E00000000650C?
3) Is there a way to turn the second string into the first string?
4) Knowing that I can pull out all the values I need from the HEX string by doing...
DECLARE @string VARCHAR(100)
SELECT @string = 'C200000012700CD400000003673CD500000000850C'
SELECT
SUBSTRING(@string,CASE WHEN N = 3 THEN N ELSE N + 3 END,11)
FROM tally t
WHERE (t.n = 3
OR t.N%14 = 0)
AND t.N < LEN(@string)
-----------------------------------
results
00000012700
00000003673
00000000850
... is there a way to pull out the same values from the VARBIN string? Other than the length of the two strings, I don't see any reliable patterns. I sure would appreciate any insight into any aspect of my dilema, and, as always, I very much appreciate the time you all take to consider other's problems.
Greg
_________________________________________________________________________________________________
The glass is at one half capacity: nothing more, nothing less.
January 11, 2012 at 11:52 am
DB2 HEX appears to do a similar thing as using the optional style parameter of CONVERT when converting from binary to a string type:
SELECT
CONVERT(char(4), 0xDEADBEEF), -- 'Þ¾ï'
CONVERT(char(10), 0xDEADBEEF, 1), -- '0xDEADBEEF'
CONVERT(char(8), 0xDEADBEEF, 2) -- 'DEADBEEF'
Q1: Does the Column1 value really have length 42? (e.g. trailing non-spaces).
Q2: Hope this is now irrelevant. Possibly the DB2 binary storage format for the column.
Q3: Possibly. Would need to see the query plan and know the data types involved.
Q4: No answer needed now, I hope.
Paul White
SQLPerformance.com
SQLkiwi blog
@SQL_Kiwi
January 11, 2012 at 12:38 pm
Thank you Paul. I'm still a bit confused, but I did find where the 'B' is coming from, sort of...
DECLARE @string VARCHAR(100)
SELECT @string = 0x4200000012F80C4D00000003C5144E00000000650C
SELECT CONVERT(CHAR(10),@string)
The 'B' shows up no matter what I put in for the length parameter of CHAR. I still don't know what it means.
Greg
_________________________________________________________________________________________________
The glass is at one half capacity: nothing more, nothing less.
January 11, 2012 at 12:57 pm
Greg Snidow (1/11/2012)
The 'B' shows up no matter what I put in for the length parameter of CHAR. I still don't know what it means.
So 'B' is just ASCII character 66 = 0x42. The second byte of the binary is zero, and ASCII(0) is interpreted by SQL Server Management Studio as end-of-string when displaying grid view results. Thank you Drew.
Paul White
SQLPerformance.com
SQLkiwi blog
@SQL_Kiwi
January 11, 2012 at 1:44 pm
SQL Kiwi (1/11/2012)
Greg Snidow (1/11/2012)
The 'B' shows up no matter what I put in for the length parameter of CHAR. I still don't know what it means.So 'B' is just ASCII character 66 = 0x42. The second byte of the binary is zero, and ASCII(0) is interpreted by SQL Server as end-of-string.
ASCII(0) is interpreted as end-of-string by some software, but SQL Server is most definitely NOT one of them, although it appears that the grid view in SSMS is. If you calculate the Len() of your string it will return 10 (not 1) or if you switch your query to output "Results to Text" or if you use the FOR XML clause to produce xml results you will see that SQL Server is returning the full string.
Drew
J. Drew Allen
Business Intelligence Analyst
Philadelphia, PA
January 11, 2012 at 1:59 pm
drew.allen (1/11/2012)
ASCII(0) is interpreted as end-of-string by some software, but SQL Server is most definitely NOT one of them, although it appears that the grid view in SSMS is.
Yes, well done.
Paul White
SQLPerformance.com
SQLkiwi blog
@SQL_Kiwi
January 11, 2012 at 2:21 pm
Thank you both for the tips. After doing some research (google), I believe I am working with a packed decimal field. I suspected this was the case when I FTP'd a text extract, and the data looked like a bunch of non-language characters. I found some stuff on EBCDIC, and this looks like what I have. Any ideas how to deal with this? I saw a post from 2007 here about converting it when imported from a text file, but nothing about how to deal with it when getting it VIA linked server.
Greg
_________________________________________________________________________________________________
The glass is at one half capacity: nothing more, nothing less.
January 11, 2012 at 2:41 pm
Greg Snidow (1/11/2012)
Thank you both for the tips. After doing some research (google), I believe I am working with a packed decimal field. I suspected this was the case when I FTP'd a text extract, and the data looked like a bunch of non-language characters. I found some stuff on EBCDIC, and this looks like what I have. Any ideas how to deal with this? I saw a post from 2007 here about converting it when imported from a text file, but nothing about how to deal with it when getting it VIA linked server.
Just out of interest, what are you using to connect to DB2? I only ask because, although I have never used a DB2 linked server myself, I have used many other types, and it really ought not to be this hard. There's a bunch of stuff out there about connecting to DB2, this is just one promising link I found on a quick search: http://blogs.msdn.com/b/dotnetinterop/archive/2006/01/20/defining-a-db2-as-a-linked-server.aspx
Paul White
SQLPerformance.com
SQLkiwi blog
@SQL_Kiwi
January 11, 2012 at 3:46 pm
SQL Kiwi (1/11/2012)
Just out of interest, what are you using to connect to DB2?
Paul
I'm using the MSDASQL provider. I tried using the IBM provider that came with my IBM software, but I cannot get it to work with the 4 part naming convention, even with 'Level zero only' unchecked. I could only get data by using OPENQUERY, which is fine, but none of the OLAP functions work through ODBC connections to our mainframe. I don't know if there is a permissions issue there, but I need to use them. I found that by using the MS provider, I can use 4 part naming, and have the full power of SQL Server, which is my preference. I am 99% happy with it now, except for those darned packed decimal fields. A friend of mine, who is connecting to the same system with SAS, told me he has the same issue with the packed decimals. I am wondering if there is a configuration option I can set. Anyhow, I totally agree with you, this should not be this difficult. Thank you.
Greg
_________________________________________________________________________________________________
The glass is at one half capacity: nothing more, nothing less.
January 12, 2012 at 5:04 pm
Well my hopes are all but dashed now. I spoke to our mainframe guru, and she told me the column is a user defined data type. The first two bytes contain the length of the field in binary format, then the remaining bytes are an intermingled mix of signed packed decimal and alphanumeric data. I just don't think I can spar against that kind of data engineering. One last grasp at the proverbial straws though: is it possible to read the column bit by bit?
Greg
_________________________________________________________________________________________________
The glass is at one half capacity: nothing more, nothing less.
January 12, 2012 at 5:42 pm
Ok, so I am going to go ahead with the straw grabbing. I was messing around with Drew's suggestion to ouput to text, combined with casting various substrings as various data types, and I ended up with this...and I don't even know it it will display here...
------------------------------
øM
øM
d<C
ßæC
… M
Indeed, they are not displaying as they do in the output. However, I took one of them at random, this guy, '', and converted to ASCII...
SELECT ASCII('')
-----------
8
(1 row(s) affected)
So, if ASCII 8 = backspace, does anyone think this has any meaning, or is SQL Server just totally confused by the nature of this data? In my results pane, the character actually looks like a vertically aligned rectangle with a slightly ovoid white dot in the center. Could this be the seemingly confounding EBCDIC? At this point, I think I'm going to stick with OPENQUERY, so any further discussion would be academic. That is, of course, unless anyone thinks my errand is anything other than a fool's. Thanks.
Greg
_________________________________________________________________________________________________
The glass is at one half capacity: nothing more, nothing less.
January 13, 2012 at 5:01 am
What's the data type of Column1 one in DB2? It looks like a structure containing 6 columns:
Pos 1: CHAR(1) - 'B' = 0xC2 (EBCDIC) = 0x42 (ASCII)
Pos 2-7: NUMERIC(11) Packed decimal - 0x00000012700C (0xC = plus sign)
Pos 8: CHAR(1) - 'M' = 0xD4 (EBCDIC) = 0x4D (ASCII)
Pos 9: NUMERIC(11) Packed decimal - 0x00000003673C (0xC = plus sign)
Pos 15: CHAR(1) - 'N' = 0xD5 (EBCDIC) = 0x4E (ASCII)
Pos 16-21: NUMERIC(11) Packed decimal - 0x00000000850C (0xC = plus sign)
When using the four part naming convention the entire structure is translated from EBCDIC to ASCII byte by byte. For a translation table take a look at http://en.wikipedia.org/wiki/EBCDIC_37. Note that the least significant nible of the packed decimal values contains 0xC meaning positive. Other values may be 0xD (negative) or 0xF (unsigned) (see http://www.simotime.com/datapk01.htm).
One way to get the packed decimal values out this structure is to translate it back to EBCDIC, convert it to a hex-string and then extracting the values the way you already did using OPENQUERY. Here's some demo code. Note that the Ascii-Ebcdic table is not complete in function AsciiToEbcdic. I just put in enough codes to translate your sample data:
DECLARE @v-2 VARBINARY(100) = 0x4200000012F80C4D00000003C5144E00000000650C
SELECT
*
FROM
dbo.AsciiToEbcdic(@V) A2E
CROSS APPLY
dbo.GetPackedDecimal(A2E.Value, 2, 6) PD1
CROSS APPLY
dbo.GetPackedDecimal(A2E.Value, 9, 6) PD2
CROSS APPLY
dbo.GetPackedDecimal(A2E.Value, 16, 6) PD3
OPTION (MAXRECURSION 0)
Output:
0xC200000012700CD400000003673CD500000000850D 12700 3673 850
Functions:
IF OBJECT_ID('dbo.AsciiToEbcdic', 'IF') IS NOT NULL
DROP FUNCTION dbo.AsciiToEbcdic
GO
CREATE FUNCTION dbo.AsciiToEbcdic(@V VARBINARY(8000))
RETURNS TABLE
AS RETURN
WITH rCTE(I, Value) AS
(
SELECT 0 AS I, CAST('' AS VARBINARY(8000)) AS Value
UNION ALL
SELECT
I + 1, Value + A2E.EbcdicCode
FROM
rCTE
JOIN
( --Ascii - Ebcdic table: for all values look at http://en.wikipedia.org/wiki/EBCDIC_37
VALUES
(0x00, 0x00),
(0x03, 0x03),
(0x0C, 0x0C),
(0x0D, 0x0D),
(0x0F, 0x0F),
(0x12, 0x12),
(0x42, 0xC2),
(0x4D, 0xD4),
(0xC5, 0x67),
(0xF8, 0x70),
(0x14, 0x3C),
(0x4E, 0xD5),
(0x65, 0x85)
) A2E (AsciiCode, EbcdicCode) ON SUBSTRING(@V, I + 1, 1) = AsciiCode AND I < DATALENGTH(@V)
)
SELECT
Value
FROM
rCTE
WHERE
I = DATALENGTH(@V)
;
GO
IF OBJECT_ID('dbo.GetPackedDecimal', 'IF') IS NOT NULL
DROP FUNCTION dbo.GetPackedDecimal
GO
CREATE FUNCTION dbo.GetPackedDecimal(@V VARBINARY(100), @Pos INT, @Len INT)
RETURNS TABLE
AS RETURN
WITH Cte AS
(
SELECT
SUBSTRING(CONVERT(VARCHAR(100), @v-2, 2), 2 * @Pos - 1, 2 * @Len) R
)
SELECT
CASE RIGHT(R, 1)
WHEN 'D' THEN -- '0xC' = Positive, '0xF' = Unsigned
-1
ELSE
1
END * CAST(LEFT(R, DATALENGTH(R) - 1) AS DECIMAL(38)) Value
FROM
Cte
;
GO
January 13, 2012 at 8:41 am
Peter, thank you so much for taking your good time to work on my problem, I am truly humbled by your knowledge.
Peter Brinkhaus (1/13/2012)
What's the data type of Column1 one in DB2? It looks like a structure containing 6 columns:Pos 1: CHAR(1) - 'B' = 0xC2 (EBCDIC) = 0x42 (ASCII)
Pos 2-7: NUMERIC(11) Packed decimal - 0x00000012700C (0xC = plus sign)
Pos 8: CHAR(1) - 'M' = 0xD4 (EBCDIC) = 0x4D (ASCII)
Pos 9: NUMERIC(11) Packed decimal - 0x00000003673C (0xC = plus sign)
Pos 15: CHAR(1) - 'N' = 0xD5 (EBCDIC) = 0x4E (ASCII)
Pos 16-21: NUMERIC(11) Packed decimal - 0x00000000850C (0xC = plus sign)
Your break down makes perfect sense now, as I could not get how a single field could actually be an array.
When using the four part naming convention the entire structure is translated from EBCDIC to ASCII byte by byte.
The four part naming convention is causing me some trouble, as I can not get it to work with the IBM provider. If I use the MSDASQL provider, I can use it if I leave level zero only unchecked, and omit the catalog. The problem here is that I cannot see the objects. If I use the IBM provider, I can not get 4 part name to work regardless of the level zero value. There is somewhat of a language barrier between the dba's and me, because they do not know SQL Server, and I don't know DB2, so I don't even get what the catalog is. I scripted out my linked server, and tweeked it a bit for others to easily use, maybe you could Identify what I am doing wrong?
--==== Declare the local variables needed to make the connection. You will need to run this script
-- every time your mainframe password changes.
DECLARE @user VARCHAR(100),
@pwd VARCHAR(100),
@connection VARCHAR(100),
@local VARCHAR(100),
@LinkedServer VARCHAR(20),
@dsn VARCHAR(20)
--==== Set the values of the local variables
SELECT @user = '#######' --<<<< change this to your EID
SELECT @pwd = '#######' --<<<< change this to your mainframe password
SELECT @connection = 'Password=' + @pwd + ';Persist Security Info=True;User ID=' + @user
SELECT @local = '#######\' + @user --<<<< change to your domain
SELECT @LinkedServer = '#######' --<<<< change to your preference
SELECT @dsn = '#######' --<<<< change to your dsn
--==== Drop the linked server if it already exists.
IF EXISTS (SELECT srv.name
FROM sys.servers srv
WHERE srv.server_id != 0
AND srv.name = @LinkedServer)
EXEC master.dbo.sp_dropserver @server= @LinkedServer, @droplogins='droplogins';
EXEC master.dbo.sp_addlinkedserver
@server = @LinkedServer,
@srvproduct = 'DB2',
@provider = 'MSDASQL', -- can use 4 part name is catalog omitted, level zero only not checked
--@provider = 'IBMDADB2', -- 4 part name does not work, don't know what is catalog
@datasrc = @dsn,
@provstr = @connection;
EXEC master.dbo.sp_addlinkedsrvlogin
@rmtsrvname ='MIRROR',
@useself ='False',
@locallogin = @local,
@rmtuser = @user,
@rmtpassword = @pwd;
Again, thank you for your help.
Greg
_________________________________________________________________________________________________
The glass is at one half capacity: nothing more, nothing less.
January 13, 2012 at 10:12 am
I'm sorry I have to disappoint you, but I don't know DB2 either, so I can't help you on this.
January 13, 2012 at 10:54 am
Not even the least bit disappointed, indeed just the opposite is true, as not being able to see the tables is a minor issue. I built a full table of values, and it worked like a charm. My task now is to figure out how to make it so that I can use it in a select list on multiple records. Right now I am unable to do this. Thanks again for your help Peter.
Greg
_________________________________________________________________________________________________
The glass is at one half capacity: nothing more, nothing less.
Viewing 15 posts - 1 through 15 (of 19 total)
You must be logged in to reply to this topic. Login to reply